This reverts commit 769907ae.

This change caused issues with people using USB gadget for serial
consoles.  In addition, with the other USB changes coming in, it
makes sense to revert this patch and apply the new set as it
becomes ready.
Signed-off-by: Andy Gross <andy.gross@linaro.org>