We are going to disable runtime_pm and we're
removing the driver, we must disable the device
now.
Signed-off-by: Felipe Balbi <balbi@ti.com>