There's no need to stop works that have already been
stopped during disconnect, so don't.
Reviewed-by: Miriam Rachel Korenblit <miriam.rachel.korenblit@intel.com>
Link: https://msgid.link/20240506211034.f8434be19f56.I021afadc538508da3bc8f95c89f424ca62b94bef@changeidSigned-off-by: Johannes Berg <johannes.berg@intel.com>