If the glue layer is removed first (core layer later),
it deletes the phy device first, then the core device.
But at core's removal, it still uses PHY's resources, it may
cause kernel's oops. It is much like the problem
Paul Zimmerman reported at:
http://marc.info/?l=linux-usb&m=136547502011472&w=2.

Besides, it is reasonable the PHY is deleted at last as
the controller is the PHY's user.
Signed-off-by: Peter Chen <peter.chen@freescale.com>
Cc: <stable@vger.kernel.org>
Signed-off-by: Felipe Balbi <balbi@ti.com>