MDEV-30603: Wrong result with non-default JOIN_CACHE_LEVEL=[4|5] ...

JOIN_CACHE::alloc_buffer() used wrong logic when calculating the size
of all join buffers. Then, it computed the ratio by which
JOIN::shrink_join_buffers() should shrink the buffers.

shrink_join_buffers() ended up in a situation where buffers would not
fit into the total quota after shrinking, which resulted in negative
buffer sizes. Due to use of unsigned integers it would cause very large
buffers to be used instead.

Make JOIN_CACHE::alloc_buffer() use the same logic as
JOIN::shrink_join_buffers() when it calculates the total size of
all join buffers so far.

Also, add a safety check in JOIN::shrink_join_buffers()

This patch doesn't include a testcase, because the original test dataset
is too big and fragile. We have dbt3_s001.inc but I wasn't able to demonstrate
the issue with it.

sql/sql_join_cache.cc

@@ -910,7 +910,12 @@ int JOIN_CACHE::alloc_buffer()
   min_buff_size= get_min_join_buffer_size();
   buff_size= get_max_join_buffer_size(optimize_buff_size);
 
-  for (tab= start_tab; tab!= join_tab; 
+  /*
+    Compute the total buffer usage for all join buffers up to
+    and including the current one.
+  */
+  for (tab= first_linear_tab(join, WITHOUT_BUSH_ROOTS, WITHOUT_CONST_TABLES);
+       tab != join_tab;
        tab= next_linear_tab(join, tab, WITHOUT_BUSH_ROOTS))
   {
     cache= tab->cache;

sql/sql_select.cc

@@ -4443,6 +4443,15 @@ bool JOIN::shrink_join_buffers(JOIN_TAB *jt,
       }
       buff_size= cache->get_join_buffer_size();
       curr_space-= buff_size;
+      if (needed_space < buff_size)
+      {
+        /*
+          Safety: fail if we've exhausted available buffer space with
+          reduced join buffers.
+        */
+        DBUG_ASSERT(0);
+        return TRUE;
+      }
       needed_space-= buff_size;
     }
   }