Host might receive spectral events during interface
down sequence and this might create below errors.

failed to handle dma buf release event -22
failed to handle dma buf release event -22

Fix this by disabling spectral config during remove interface.

Tested-on: IPQ5018 hw1.0 AHB WLAN.HK.2.6.0.1-00861-QCAHKSWPL_SILICONZ-1
Signed-off-by: Tamizh Chelvam Raja <quic_tamizhr@quicinc.com>
Signed-off-by: Kalle Valo <quic_kvalo@quicinc.com>
Link: https://lore.kernel.org/r/20230328071150.29645-1-quic_tamizhr@quicinc.com