This reverts commit 0db213ea.

It introduces an issues with configuring the USB gadget hangs forever
on multiple Qualcomm and NXP i.MX SoC at least.

Cc: stable@vger.kernel.org
Fixes: 0db213ea ("usb: gadget: udc: core: Invoke usb_gadget_connect only when started")
Reported-by: Stephan Gerhold <stephan@gerhold.net>
Reported-by: Francesco Dolcini <francesco.dolcini@toradex.com>
Link: https://lore.kernel.org/all/ZF4BvgsOyoKxdPFF@francesco-nb.int.toradex.com/Signed-off-by: Francesco Dolcini <francesco.dolcini@toradex.com>
Link: https://lore.kernel.org/r/20230512131435.205464-3-francesco@dolcini.itSigned-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>