Set the card owner field to prevent the module from being removed from
underneath its users.
Signed-off-by: Axel Lin <axel.lin@ingics.com>
Signed-off-by: Mark Brown <broonie@kernel.org>