ALSA: hda: Set up BDL table at hw_params

So far the setup of BDL table is performed at the prepare stage, where
all PCM parameters have been already set up.  When something wrong
happens at it, we return -EINVAL; it's supposed to be a rare case
since the involved memory allocation is a small chunk of kmalloc for
the table.

However, when we receive too many small non-contiguous pages in highly
fragmented memories, it may overflow the max table size, resulting in
the same -EINVAL error from the prepare, too.  A bad scenario is that
user-space cannot know what went wrong (as it's an error from the
prepare stage) and -EINVAL, hence it may retry with the same
parameters, failing again repeatedly.

In this patch, we try to set up the BDL table at hw_params right after
the buffer allocation, and return -ENOMEM if it overflows.
This allows user-space knowing that it should reduce the buffer size
request accordingly and may retry with more fitting parameters.

Link: https://lore.kernel.org/r/20240221100607.6565-1-tiwai@suse.deSigned-off-by: Takashi Iwai <tiwai@suse.de>

sound/pci/hda/hda_controller.c

@@ -24,6 +24,7 @@
 
 #include <sound/core.h>
 #include <sound/initval.h>
+#include <sound/pcm_params.h>
 #include "hda_controller.h"
 #include "hda_local.h"
 
@@ -108,6 +109,7 @@ static int azx_pcm_hw_params(struct snd_pcm_substream *substream,
 	struct azx_pcm *apcm = snd_pcm_substream_chip(substream);
 	struct azx *chip = apcm->chip;
 	struct azx_dev *azx_dev = get_azx_dev(substream);
+	struct hdac_stream *hdas = azx_stream(azx_dev);
 	int ret = 0;
 
 	trace_azx_pcm_hw_params(chip, azx_dev);
@@ -117,9 +119,15 @@ static int azx_pcm_hw_params(struct snd_pcm_substream *substream,
 		goto unlock;
 	}
 
-	azx_dev->core.bufsize = 0;
-	azx_dev->core.period_bytes = 0;
-	azx_dev->core.format_val = 0;
+	/* Set up BDLEs here, return -ENOMEM if too many BDLEs are required */
+	hdas->bufsize = params_buffer_bytes(hw_params);
+	hdas->period_bytes = params_period_bytes(hw_params);
+	hdas->format_val = 0;
+	hdas->no_period_wakeup =
+		(hw_params->info & SNDRV_PCM_INFO_NO_PERIOD_WAKEUP) &&
+		(hw_params->flags & SNDRV_PCM_HW_PARAMS_NO_PERIOD_WAKEUP);
+	if (snd_hdac_stream_setup_periods(hdas) < 0)
+		ret = -ENOMEM;
 
 unlock:
 	dsp_unlock(azx_dev);