MDEV-33392 Server crashes when using RANDOM_BYTES function and GROUP BY clause on a column with a negative value

Item_func_random_bytes did not set its NULL-ability flag.

Additionally pass `-ExecutionPolicy Bypass` to powershell.exe when
executing script

innodb_log_file_size_update(): Wait for buf_pool.done_flush_list
(that a flush batch has completed), not buf_pool.do_flush_list
(that the buf_flush_page_cleaner was woken up).

The condition variable buf_pool.done_flush_list is broadcast by the
buf_flush_page_cleaner() at the end of each batch, which is when the
log checkpoint can advance and where any log resizing may be completed.
The purpose of the condition variable buf_pool.do_flush_list is to
wake up the buf_flush_page_cleaner() thread because there is work to do.
If no thread is signaling that condition variable, this loop could
unnecessarily wait for up to 5 seconds too long for the log resizing
to be completed. By consuming signals it could also prevent the
buf_flush_page_cleaner() thread from waking up.

Tested by: Matthias Leich

Postfix for a6290a5b, in 10.11
where OS_DATA_FILE_NO_O_DIRECT gets used. Same #ifdef conditions
as other uses of OS_DATA_FILE_NO_O_DIRECT.

Noticed on aarch64-macos builder.

Add new options to mini-benchmark.sh so it is more useful, such as
`--log` to avoid output from `lscpu` and other commands being lost.

Also fix following errors:

- Fix if condition for the cycle count regression verdict. The condition
  checked if the cycle count was greater than 850 billion, but `$RESULT`
  is only 3 digits and represents the number of cycles in billions.

- Fix flamegraph width. The original width caused the flamegraph to be
  cut-off on most screen sizes in the browser.

- Fix ShellCheck warnings and suggestions.

- Fix condition to check if perf has permission to run on the system.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.

We can check only fields which take part in inserts.

Still not ready to support mariadb-plugin-rocks by distro.

pmem (despite its EOL on Intel) and liburing are available on Noble
(24.04). Its unclear why they where disabled in lunar/jammy. Too
unstable to change now so introduce to 24.04 and see how it goes
during testing.

Some old versions of "patch" (such as patch 2.5.9 on Microsoft Windows)
require that a file name header be present.
To ensure that the diff will be applied, let us add the header.

It is not sufficient to sleep for only 1 second to
ensure that the page cleaner has gone idle.
The timings could have been changed compared to earlier releases
by commit a635c406 (MDEV-27774).
Let us allow the non-debug test innodb.doublewrite to be skipped,
and remove the insufficient 1-second sleep.

This fixes up merge commit 9d20853c

A mix of path separators looks odd.

  InnoDB: Loading buffer pool(s) from C:\xampp\mysql\data/ib_buffer_pool

This was changed in cf552f58

Both forward slashes and backward slashes work on Windows. We do not
use \\?\ names.

So we improve the consistent look of it so it doesn't look like a bug.

Normalize, in this case, the path separator to \ for making the filename.

Reported thanks to Github user @celestinoxp.

Closes: https://github.com/ApacheFriends/xampp-build/issues/33
Reviewed by: Marko Mäkelä and Vladislav Vaintroub

use pkg-config to find pcre2, if possible

rename PCRE_INCLUDES to use PKG_CHECK_MODULES naming, PCRE_INCLUDE_DIRS

MDEV-32968  InnoDB fails to restore tablespace first page from doublewrite buffer when page is empty

recv_dblwr_t::find_first_page(): Free the allocated memory
to read the first 3 pages from tablespace.

innodb.doublewrite: Added sleep to ensure page cleaner thread
wake up from my_cond_wait

Revert "MDEV-32899 InnoDB is holding shared dict_sys.latch while waiting for FOREIGN KEY child table lock on DDL"

This reverts commit 569da6a7,
commit 768a7361, and
commit ba6bf7ad
because of a regression that was filed as MDEV-33104.

In commit a55b951e (MDEV-26827)
an error was introduced in a rarely executed code path of the
buf_flush_page_cleaner() thread. As a result, the function
buf_flush_LRU() could be invoked while not holding buf_pool.mutex.

Reviewed by: Debarun Banerjee

buf_flush_LRU(): Display a warning if no pages could be evicted and
no writes initiated.

buf_pool_t::need_LRU_eviction(): Renamed from buf_pool_t::ran_out().
Check if the amount of free pages is smaller than innodb_lru_scan_depth
instead of checking if it is 0.

buf_flush_page_cleaner(): For the final LRU flush after a checkpoint
flush, use a "budget" of innodb_io_capacity_max, like we do in the
case when we are not in "furious" checkpoint flushing.

Co-developed by: Debarun Banerjee
Reviewed by: Debarun Banerjee
Tested by: Matthias Leich

Noted by Susmeet Khaire - thanks.

The directio(3C) function on Solaris is supported on NFS and UFS
while the majority of users should be on ZFS, which is a copy-on-write
file system that implements transparent compression and therefore
cannot support unbuffered I/O.

Let us remove the call to directio() and simply treat
innodb_flush_method=O_DIRECT in the same way as the previous
default value innodb_flush_method=fsync on Solaris. Also, let us
remove some dead code around calls to os_file_set_nocache() on
platforms where fcntl(2) is not usable with O_DIRECT.

On IBM AIX, O_DIRECT is not documented for fcntl(2), only for open(2).

Upon further benchmarking, it turns out srw_mutex performs overall
slightly better with WaitOnAddress than CRITICAL_SECTION.

The parameter innodb_undo_log_truncate=ON enables a multi-phased logic:
1. Any "producers" (new starting transactions) are prohibited
from using the rollback segments that reside in the undo tablespace.
2. Any transactions that use any of the rollback segments must be
committed or aborted.
3. The purge of committed transaction history must process all the
rollback segments.
4. The undo tablespace is truncated and rebuilt.
5. The rollback segments are re-enabled for new transactions.

There was one flaw in this logic: The first step was not being invoked
as often as it could be, and therefore innodb_undo_log_truncate=ON
would have no chance to work during a heavy write workload.

Independent of innodb_undo_log_truncate, even after
commit 86767bcc
we are missing some chances to free processed undo log pages.
If we prohibited the creation of new transactions in one busy
rollback segment at a time, we would be eventually guaranteed
to be able to free such pages.

purge_sys_t::skipped_rseg: The current candidate rollback segment
for shrinking the history independent of innodb_undo_log_truncate.

purge_sys_t::iterator::free_history_rseg(): Renamed from
trx_purge_truncate_rseg_history(). Implement the logic
around purge_sys.m_skipped_rseg.

purge_sys_t::truncate_undo_space: Renamed from truncate.

purge_sys.truncate_undo_space.last: Changed the type to integer
to get rid of some pointer dereferencing and conditional branches.

purge_sys_t::truncating_tablespace(), purge_sys_t::undo_truncate_try():
Refactored from trx_purge_truncate_history().
Set purge_sys.truncate_undo_space.current if applicable,
or return an already set purge_sys.truncate_undo_space.current.

purge_coordinator_state::do_purge(): Invoke
purge_sys_t::truncating_tablespace() as part of the normal work loop,
to implement innodb_undo_log_truncate=ON as often as possible.

trx_purge_truncate_rseg_history(): Remove a redundant parameter.

trx_undo_truncate_start(): Replace dead code with a debug assertion.

Correctness tested by: Matthias Leich
Performance tested by: Axel Schwenke
Reviewed by: Debarun Banerjee

Adding GEOMETRY type user variables.

Since 0930eb86, system table creation
needed for spider init is delayed to the signal_ddl_recovery_done
callback. Since it is part of the init, failure should result in
spider deinit.

We also remove the call to spider_init_system_tables() from
spider_db_init(), as it was removed in the commit mentioned above and
accidentally restored in a merge.

MDEV-32968  InnoDB fails to restore tablespace first page from doublewrite buffer when page is empty

- InnoDB fails to find the space id from the page0 of
the tablespace. In that case, InnoDB can use
doublewrite buffer to recover the page0 and write
into the file.

- buf_dblwr_t::init_or_load_pages(): Loads only the pages
which are valid.(page lsn >= checkpoint). To do that,
InnoDB has to open the redo log before system
tablespace, read the latest checkpoint information.

recv_dblwr_t::find_first_page():
1) Iterate the doublewrite buffer pages and find the 0th page
2) Read the tablespace flags, space id from the 0th page.
3) Read the 1st, 2nd and 3rd page from tablespace file and
compare the space id with the space id which is stored
in doublewrite buffer.
4) If it matches then we can write into the file.
5) Return space which matches the pages from the file.

SysTablespace::read_lsn_and_check_flags(): Remove the
retry logic for validating the first page. After
restoring the first page from doublewrite buffer,
assign tablespace flags by reading the first page.

recv_recovery_read_max_checkpoint(): Reads the maximum
checkpoint information from log file

recv_recovery_from_checkpoint_start(): Avoid reading
the checkpoint header information from log file

Datafile::validate_first_page(): Throw error in case
of first page validation fails.

Upstream Debian Sid which will become Debian Trixie (13)
have dropped NCurses version 5 and changed dev package name just
libncurses-dev

Signed-off-by: Paul Szabo psz@maths.usyd.edu.au www.maths.usyd.edu.au/u/psz
School of Mathematics and Statistics University of Sydney Australia

Signed-off-by: Paul Szabo psz@maths.usyd.edu.au www.maths.usyd.edu.au/u/psz
School of Mathematics and Statistics University of Sydney Australia

Add INSERT ... SELECT to the list of commands that can be traced

Approved by Sergei Petrunia (sergey@mariadb.com)