trx_purge_truncate_rseg_history(): If all other conditions for
invoking trx_purge_remove_log_hdr() hold, but the state is
TRX_UNDO_CACHED instead of TRX_UNDO_TO_PURGE, detach and free it.

Tested by: Matthias Leich

buf_LRU_get_free_block(): Always wake up the page cleaner if needed
before exiting the inner loop.

srv_prepare_to_delete_redo_log_file():
Replace a debug assertion with a wait in debug builds.
Starting with commit 7e31a8e7
the debug assertion ut_ad(!os_aio_pending_writes())
could occasionally fail, while it would hold in core dumps of crashes.
The failure can be reproduced more easily by adding a sleep to the
write completion callback function, right before releasing to
write_slots.

srv_start(): Remove a bogus debug assertion
ut_ad(!os_aio_pending_writes()) that could fail in
mariadb-backup --prepare. In an rr replay trace, we had
buf_pool.flush_list.count==0 but write_slots->m_cache.m_pos==1
and buf_page_t::write_complete() was executing u_unlock().

buf_flush_wait_flushed(): Correct the logic for registering a wait
around buf_flush_wait() that
commit a091d6ac
recently broke. This should be easily repeatable when using a
non-default startup parameter:

	thread-handling=pool-of-threads

trx_purge_free_segment(): The buffer-fix only prevents a block from
being freed completely from the buffer pool, but it will not prevent
the block from being evicted. Recheck the page identifier after
acquiring an exclusive page latch. If it has changed, backtrack and
invoke buf_page_get_gen() to look up the page normally.

io_callback(): Process the request before releasing the write slot.
Before commit a091d6ac
when we had a duplicated counter for writes, either ordering was fine.
Now, correctness depends on os_aio_wait_until_no_pending_writes().

There is an assumption that when there are are no completed tests,
that means they are still running and then an attempt is made to
identify these tests as stalled.

The other possibility is however there are no tests that where run.

Test this early and then exit quickly and no later misunderstandings
need to be made.

This reverts commit 0e737f78.

As noted by Andrew, this introduces race conditions in the setting
and using of the global $test_name_for_report.

buf_flush_LRU_list_batch(): When evicting clean pages,
release and reacquire the buf_pool.mutex after every 32 pages.
Also, eliminate some conditional branches.

fil_space_t::create(), fil_space_t::add(): Expect the caller to
acquire and release fil_system.mutex. In this way, creating a tablespace
and adding the first (usually only) data file will be atomic.

recv_sys_t::recover_deferred(): Correctly protect some changes by
holding fil_system.mutex.

Tested by: Matthias Leich

trx_assign_rseg_low(): Let us restore the debug variable look_for_rollover
to avoid assertion failures when a server that was created with
multiple undo tablespaces is being started with innodb_undo_tablespaces=0.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

trx_assign_rseg_low(): Correct a debug injection condition.

trx_assign_rseg_low(): Simplify the debug check.

trx_rseg_t::reinit(): Reset the skip_allocation() flag.
This logic was broken in the merge
commit 3e2ad0e9
of commit 0de3be8c
(that is, innodb_undo_log_truncate=ON would never be "completed").

Tested by: Matthias Leich

A GROUP BY query which uses "MIN(pk)" and has "pk<>const" in the
WHERE clause would produce wrong result when handled with "Using index
for group-by".  Here "pk" column is the table's primary key.

The problem was introduced by fix for MDEV-23634. It made the range
optimizer to not produce ranges for conditions in form "pk != const".

However, LooseScan code requires that the optimizer is able to
convert the condition on the MIN/MAX column into an equivalent range.
The range is used to locate the row that has the MIN/MAX value.

LooseScan checks this in check_group_min_max_predicates(). This fix
makes the code in that function to take into account that "pk != const"
does not produce a range.

If SQL_MODE contains ANSI_QUOTES (https://mariadb.com/kb/en/sql-mode/), then
the double-quote character (") is not a legal string delimiter.

In https://github.com/MariaDB/server/commit/13e77930e615f05cc74d408110e887b00e1abcc9#diff-a333d4ebb2d73b6361ef7dfebc86d883f7e19853b4a9eb85984b039058fae47cR2431-R2435,
Daniel Black introduced a case where the double-quote character would be used as
a string delimiter in the SQL queries generated by mariadb-tzinfo-to-sql.

This tool tool generates SQL queries which should be able to run on any
MariaDB server of the matching version.  Therefore, it should be extremely
conservative in the SQL that it outputs, in order to maximize the chance
that it can run regardless of the build or execution environment of the
server.

See MDEV-18778, MDEV-28263, and MDEV-28782 for previous cases where MariaDB
has FAILED TO ENSURE that the generated timezone.sql actually works in
different build and execution environments.  More test coverage is clearly
needed here.

All new code of the whole pull request, including one or several files that are
either new files or modified ones, are contributed under the BSD-new license. I
am contributing on behalf of my employer Amazon Web Services, Inc.

The 2013 error was right to catch the case B of the test unprepared
for an expected simulated crash.

The test gets refined to SELECT a (type of) bool value before the
crash is invoked.

Appending to 'eatmydata' will obviously cause an executable that
doesn't exist. Use an array to create the entire executable.

Also while we are at it, check the fakeroot actually works before
using it.

MDEV-31049 fil_delete_tablespace() returns wrong file handle if tablespace was closed by parallel thread

fil_delete_tablespace() stores file handle in local variable and calls
mtr_t::commit_file()=>fil_system_t::detach(..., detach_handle=true), which
sets space->chain.start->handle = OS_FILE_CLOSED. fil_system_t::detach()
is invoked under fil_system.mutex.

But before the mutex is acquired some parallel thread can change
space->chain.start->handle. fil_delete_tablespace() returns value, stored
in local variable, i.e. wrong value.

File handle can be closed, for example, from buf_flush_space() when the
limit of innodb_open_files exceded and fil_space_t::get() causes
fil_space_t::try_to_close() call.

fil_space_t::try_to_close() is executed under fil_system.mutex. And
mtr_t::commit_file() locks it for fil_system_t::detach() call.
fil_system_t::detach() returns detached file handle if its argument
detach_handle is true. The fix is to let mtr_t::commit_file() to pass
that detached file handle to fil_delete_tablespace().

Post-push fix.

10.5 MDEV-30775 fix inserts just opened tablespace just after the element
which fil_system.space_list_last_opened points to.

In MDEV-25223 fil_system_t::space_list was changed from UT_LIST to
ilist. ilist<...>::insert(iterator pos, reference value) inserts element
to list before pos.

But it was not taken into account during 10.5->10.6 merge in
85cbfaef, and the fix
does not work properly, i.e. it inserted just opened tablespace to the
position preceding fil_system.space_list_last_opened.

Currently autobake-debs.sh does not pass shellcheck
it fails making errors:

 * SC1091 when using shellcheck -x it needs to know where to
   find ./VERSION. As this is not needed we just specify it
   as /dev/null as mentioned in shellcheck documentation:
   https://www.shellcheck.net/wiki/SC1091

 * SC2086 make sure that there is no globbing or word splitting
   in dpkg-buidpackage string. This not big problem or about to happen
   but now extra parameter parsing is more Bash compliant with
   using array.
   Change BUILDPACKAGE_PREPEND to BUILDPACKAGE_DPKGCMD which holds
   'eatmydata' if it's available and needed 'dpkg-buildpackage'
   https://www.shellcheck.net/wiki/SC2086

Fix small script indentation problem.

It is used in the out-of date Ubuntu 21.10 Impish.

Github PR #2424 regressed Ubuntu 18.04 building
other than x86_64 machines. Architecture that are
impacted are PPC64 and ARM64.
This was because of changes in debian/rules file
which caused removing dependency to package 'libpmem-dev'
and CMake which '-DWITH_PMEM' removing not working
correctly. Package libpmem-dev was removed but
it still required to have PMEM with CMake which.
Commit make change that -DWITH_PMEM is correctly removed
if it's not wanted.

tpool::cache::m_mtx: Add PERFORMANCE_SCHEMA instrumentation
(wait/synch/mutex/innodb/tpool_cache_mutex). This covers the
InnoDB read_slots and write_slots for asynchronous data page I/O.

fil_name_process(): Starting with
commit 212994f7
the name is not guaranteed to be NUL terminated.

- This issue caused by race condition between drop thread
and fil_encrypt_thread. fil_encrypt_thread closes
the tablespace if the number of opened files
exceeds innodb_open_files. fil_node_open_file()
closes the tablespace which are open and it doesn't
have pending operations. At that time, InnoDB drop tries
to write the redo log for the file delete operation.
It throws the bad file descriptor error.

- When trying to close the file, InnoDB should check
whether the table is going to be dropped.

os_aio_pending_reads_approx(), os_aio_pending_reads(): Replaces
buf_pool.n_pend_reads.

os_aio_pending_writes(): Replaces buf_dblwr.pending_writes().

buf_dblwr_t::write_cond, buf_dblwr_t::writes_pending: Remove.

buf_flush_try_neighbors(): Tolerate count<2 in case the tablespace
is being dropped.

The problem was earlier fixed by the patch for MDEV-30034.
Adding MTR tests only.

Spider system tables should be created so that wsrep_on=OFF.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Query cache should be invalidated if we are not in applier. For some
reason this condition was incorrect starting from 10.5 but it is
correct in 10.4.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

After MDEV-30830 has added block-nl-join.r_unpack_time_ms, it became
apparent that there is some unaccounted-for time in BNL join operation,
namely the time that is spent after unpacking the join buffer record.

Fix this by adding a Gap_time_tracker to track the time that is spent
after unpacking the join buffer record and before any next time tracking.
The collected time is printed in block-nl-join.r_other_time_ms.

Reviewed by: Monty <monty@mariadb.org>

Also fix it to work with hashed join (MDEV-30830).

Reviewed by: Monty <monty@mariadb.org>

- Adding a new argument "flag" to MY_COLLATION_HANDLER::strnncollsp_nchars()
  and a flag MY_STRNNCOLLSP_NCHARS_EMULATE_TRIMMED_TRAILING_SPACES.
  The flag defines if strnncollsp_nchars() should emulate trailing spaces
  which were possibly trimmed earlier (e.g. in InnoDB CHAR compression).
  This is important for NOPAD collations.

  For example, with this input:
   - str1= 'a '    (Latin letter a followed by one space)
   - str2= 'a  '   (Latin letter a followed by two spaces)
   - nchars= 3
  if the flag is given, strnncollsp_nchars() will virtually restore
  one trailing space to str1 up to nchars (3) characters and compare two
  strings as equal:
  - str1= 'a  '  (one extra trailing space emulated)
  - str2= 'a  '  (as is)

  If the flag is not given, strnncollsp_nchars() does not add trailing
  virtual spaces, so in case of a NOPAD collation, str1 will be compared
  as less than str2 because it is shorter.

- Field_string::cmp_prefix() now passes the new flag.
  Field_varstring::cmp_prefix() and Field_blob::cmp_prefix() do
  not pass the new flag.

- The branch in cmp_whole_field() in storage/innobase/rem/rem0cmp.cc
  (which handles the CHAR data type) now also passed the new flag.

- Fixing UCA collations to respect the new flag.
  Other collations are possibly also affected, however
  I had no success in making an SQL script demonstrating the problem.
  Other collations will be extended to respect this flags in a separate
  patch later.

- Changing the meaning of the last parameter of Field::cmp_prefix()
  from "number of bytes" (internal length)
  to "number of characters" (user visible length).

  The code calling cmp_prefix() from handler.cc was wrong.
  After this change, the call in handler.cc became correct.

  The code calling cmp_prefix() from key_rec_cmp() in key.cc
  was adjusted according to this change.

- Old strnncollsp_nchar() related tests in unittest/strings/strings-t.c
  now pass the new flag.
  A few new tests also were added, without the flag.

The tests innodb.import_tablespace_race, innodn.restart, and innodb.innodb-wl5522 move
the tablespace file between the data directory and the tmp directory specified by
global environment variables. However this is risky because it's not unusual that the
set tmp directory (often under /tmp) is mounted on another disk partition or device,
and 'move_file' command may fail with "Errcode: 18 'Invalid cross-device link.'"

For innodb.import_tablespace_race and innodb.innodb-wl5522, moving files
across directories is not necessary. Modify the tests so they rename
files under the same directory. For innodb.restart, instead of moving
between datadir and MYSQL_TMPDIR, move the files under MYSQLTEST_VARDIR.

All new code of the whole pull request, including one or several files that
are either new files or modified ones, are contributed under the BSD-new license.
I am contributing on behalf of my employer Amazon Web Services, Inc.