The interrupt may occur during the gadget deletion, it fixes the
below oops.

[ 2394.974604] configfs-gadget gadget: suspend
[ 2395.042578] configfs-gadget 5b130000.usb: unregistering UDC driver [g1]
[ 2395.382562] irq 229: nobody cared (try booting with the "irqpoll" option)
[ 2395.389362] CPU: 0 PID: 301 Comm: kworker/u12:6 Not tainted 5.8.0-rc3-next-20200703-00060-g2f13b83cbf30-dirty #456
[ 2395.399712] Hardware name: Freescale i.MX8QM MEK (DT)
[ 2395.404782] Workqueue: 2-0051 tcpm_state_machine_work
[ 2395.409832] Call trace:
[ 2395.412289]  dump_backtrace+0x0/0x1d0
[ 2395.415950]  show_stack+0x1c/0x28
[ 2395.419271]  dump_stack+0xbc/0x118
[ 2395.422678]  __report_bad_irq+0x50/0xe0
[ 2395.426513]  note_interrupt+0x2cc/0x38c
[ 2395.430355]  handle_irq_event_percpu+0x88/0x90
[ 2395.434800]  handle_irq_event+0x4c/0xe8
[ 2395.438640]  handle_fasteoi_irq+0xbc/0x168
[ 2395.442740]  generic_handle_irq+0x34/0x48
[ 2395.446752]  __handle_domain_irq+0x68/0xc0
[ 2395.450846]  gic_handle_irq+0x64/0x150
[ 2395.454596]  el1_irq+0xb8/0x180
[ 2395.457733]  __do_softirq+0xac/0x3b8
[ 2395.461310]  irq_exit+0xc0/0xe0
[ 2395.464448]  __handle_domain_irq+0x6c/0xc0
[ 2395.468540]  gic_handle_irq+0x64/0x150
[ 2395.472295]  el1_irq+0xb8/0x180
[ 2395.475436]  _raw_spin_unlock_irqrestore+0x14/0x48
[ 2395.480232]  usb_gadget_disconnect+0x120/0x140
[ 2395.484678]  usb_gadget_remove_driver+0xb4/0xd0
[ 2395.489208]  usb_del_gadget+0x6c/0xc8
[ 2395.492872]  cdns3_gadget_exit+0x5c/0x120
[ 2395.496882]  cdns3_role_stop+0x60/0x90
[ 2395.500634]  cdns3_role_set+0x64/0xd8
[ 2395.504301]  usb_role_switch_set_role.part.0+0x3c/0x90
[ 2395.509444]  usb_role_switch_set_role+0x20/0x30
[ 2395.513978]  tcpm_mux_set+0x60/0xf8
[ 2395.517470]  tcpm_reset_port+0xa4/0xf0
[ 2395.521222]  tcpm_detach.part.0+0x44/0x50
[ 2395.525227]  tcpm_state_machine_work+0x8b0/0x2360
[ 2395.529932]  process_one_work+0x1c8/0x470
[ 2395.533939]  worker_thread+0x50/0x420
[ 2395.537603]  kthread+0x148/0x168
[ 2395.540830]  ret_from_fork+0x10/0x18
[ 2395.544399] handlers:
[ 2395.546671] [<000000008dea28da>] cdns3_wakeup_irq
[ 2395.551375] [<000000009fee5c61>] cdns3_drd_irq threaded [<000000005148eaec>] cdns3_drd_thread_irq
[ 2395.560255] Disabling IRQ #229
[ 2395.563454] configfs-gadget gadget: unbind function 'Mass Storage Function'/000000000132f835
[ 2395.563657] configfs-gadget gadget: unbind
[ 2395.563917] udc 5b130000.usb: releasing '5b130000.usb'

Fixes: 7733f6c3 ("usb: cdns3: Add Cadence USB3 DRD Driver")
Cc: <stable@vger.kernel.org>
Acked-by: Roger Quadros <rogerq@ti.com>
Signed-off-by: Peter Chen <peter.chen@nxp.com>
Signed-off-by: Felipe Balbi <balbi@kernel.org>