This reverts commit 63ce18cf.

It was the incorrect fix and causes a reference counting bug whenever
any driver module is removed from the system. Mike Galbraith
<efault@gmx.de> is looking for the real fix for his problem.
Signed-off-by: Greg Kroah-Hartman <gregkh@suse.de>