While running a stress test which enables/disables clkgating, we
occasionally hit device timeout. This patch avoids a subtle race condition
to address it.

Link: https://lore.kernel.org/r/20201117165839.1643377-3-jaegeuk@kernel.orgReviewed-by: Can Guo <cang@codeaurora.org>
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>