It has to be removed from the driver, but completely
destroying it helps handle unplug of a device during
suspend since then the channel context handling etc.
doesn't have to happen later when it's removed.
Signed-off-by: Johannes Berg <johannes.berg@intel.com>