we don't to gate clocks until our children are
done with their remove path.

Fixes: af310e96 (usb: dwc3: omap: use runtime API's to enable clocks)
Cc: <stable@vger.kernel.org> # v3.9+
Signed-off-by: Felipe Balbi <balbi@ti.com>