This reverts commit ac5e2fb4.

The commit caused a regression on Behringer UMC404HD (and likely
others).  As the change was meant only as a minor optimization, it's
better to revert it to address the regression.
Reported-and-tested-by: Michael Ralston <michael@ralston.id.au>
Cc: <stable@vger.kernel.org>
Link: https://lore.kernel.org/r/CAC2975JXkS1A5Tj9b02G_sy25ZWN-ys+tc9wmkoS=qPgKCogSg@mail.gmail.com
Link: https://lore.kernel.org/r/20230104150944.24918-1-tiwai@suse.deSigned-off-by: Takashi Iwai <tiwai@suse.de>