when we remove the gadget driver, it will already
do that for us.
Signed-off-by: Felipe Balbi <balbi@ti.com>