This hang was caused by MDEV-23855, and we failed to fix it in
MDEV-24109 (commit 4cbfdeca).

When buf_flush_ahead() is invoked soon before server shutdown
and the non-default setting innodb_flush_sync=OFF is in effect
and the buffer pool contains dirty pages of temporary tables,
the page cleaner thread may remain in an infinite loop
without completing its work, thus causing the shutdown to hang.

buf_flush_page_cleaner(): If the buffer pool contains no
unmodified persistent pages, ensure that buf_flush_sync_lsn= 0
will be assigned, so that shutdown will proceed.

The test case is not deterministic. On my system, it reproduced
the hang with 95% probability when running multiple instances
of the test in parallel, and 4% when running single-threaded.

Thanks to Eugene Kosov for debugging and testing this.

This was noticed wben running "mtr --valgrind main.precedence"

The problem was that Item_func_like::escape could be left unitialized
when used with views combined with UNIONS like in:

create or replace view v1 as select 2 LIKE 1 ESCAPE 3 IN (SELECT 0 UNION SELECT 1), 2 LIKE 1 ESCAPE (3 IN (SELECT 0 UNION SELECT 1)), (2 LIKE 1 ESCAPE 3) IN (SELECT 0 UNION SELECT 1);

The above query causes in fix_escape_item()
escape_item->const_during_execution() to be true
and
escape_item->const_item() to be false

in which case 'escape' is never calculated.

The fix is to make the main logic of fix_escape_item() out to a
separate function and call that function once in Item.

Other things:
- Reorganized fields in Item_func_like class to make it more compact

The clang++ -stdlib=libc++ header file <fstream> depends on
<filesystem> that defines a member function path::root_name(),
which conflicts with the rather unused #define root_name()
that had been introduced in
commit 7c58e97b.

Because an instrumented -stdlib=libc++ (rather than the default
-stdlib=libstdc++) is easier to build for a working -fsanitize=memory
(cmake -DWITH_MSAN=ON), let us remove the conflicting #define for now.

Sorry, only tested commit 4174fc1a
on clang. Other compilers do not define __has_feature().

For some reason, commit 5bb5d4ad
made clang++-11 unhappy about a constexpr declaration.

For some reason, the test was never adjusted for
commit e6a50e41.

The Galera tests were massively failing with debug assertions.

Post-push Windows compilation errors fix.

Change thd->mdl_context.release_transactional_locks() to
thd->mdl_release_transactional_locks()

row_undo_ins_parse_undo_rec(): Do not try to read non-existing
virtual column information for the metadata record.

The replacement is buf_pool.contains_zip().

The new option --log-innodb-page-corruption is introduced.

When this option is set, backup is not interrupted if innodb corrupted
page is detected. Instead it logs all found corrupted pages in
innodb_corrupted_pages file in backup directory and finishes with error.

For incremental backup corrupted pages are also copied to .delta file,
because we can't do LSN check for such pages during backup,
innodb_corrupted_pages will also be created in incremental backup
directory.

During --prepare, corrupted pages list is read from the file just after
redo log is applied, and each page from the list is checked if it is allocated
in it's tablespace or not. If it is not allocated, then it is zeroed out,
flushed to the tablespace and removed from the list. If all pages are removed
from the list, then --prepare is finished successfully and
innodb_corrupted_pages file is removed from backup directory. Otherwise
--prepare is finished with error message and innodb_corrupted_pages contains
the list of the pages, which are detected as corrupted during backup, and are
allocated in their tablespaces, what means backup directory contains corrupted
innodb pages, and backup can not be considered as consistent.

For incremental --prepare corrupted pages from .delta files are applied
to the base backup, innodb_corrupted_pages is read from both base in
incremental directories, and the same action is proceded for corrupted
pages list as for full --prepare. innodb_corrupted_pages file is
modified or removed only in base directory.

If DDL happens during backup, it is also processed at the end of backup
to have correct tablespace names in innodb_corrupted_pages.

The reason for the failure is that
thd->mdl_context.release_transactional_locks()
was called after commit & rollback even in cases where the current
transaction is still active.

For 10.2, 10.3 and 10.4 the fix is simple:
- Replace all calls to thd->mdl_context.release_transactional_locks() with
  thd->release_transactional_locks(). The thd function will only call
  the mdl_context function if there are no active transactional locks.
  In 10.6 we will better fix where we will change the return value for
  some trans_xxx() functions to indicate if transaction did close the
  transaction or not. This will avoid the need of the indirect call.

Other things:
- trans_xa_commit() and trans_xa_rollback() will automatically
  call release_transactional_locks() if the transaction is closed.
- We can't do that for the other functions as the caller of many of these
  are doing additional work (like close_thread_tables) before calling
  release_transactional_locks().
- Added missing abort_result_set() and missing DBUG_RETURN in
  select_create::send_eof()
- Fixed wrong indentation in injector::transaction::commit()

The real fix for MDEV-15532 will be pushed into 10.2 and 10.6
This is an additional fix for 10.4.

In 10.4 trans_xa_detach was introduced.  However THD::cleanup() assumes
that after trans_xa_detach() is done, there is no registered transactions
anymore. In the 10.2 patch there will be an assert to ensure this, which
will cause 10.4 to fail.

The fix used is to reset the transaction flags in trans_xa_detach().

- the intention for my_getevents syscall is now better explained,
why are we using it (to be able to interrupt io_getevents syscall via
io_destroy()).

- Fix comment for MAX_EVENTS in getevent_thread_routine.
MAX_EVENTS is more of less arbitrary constant, chosen such that events array
is big enough to get multiple simultaneous io completions, but small
enough so it does not blow the thread's stack.

If maintenance timer does not do much for prolonged time, it will
wake up less frequently, once every 4 seconds instead of once every 0.4
second.

It will wakeup more often if thread creation is throttled, to avoid stalls.

For some reason, InnoDB debug tests on Windows fail due to rw_lock_t
if the function call overhead for some os_thread_ code is removed.

This change worked fine on Windows in combination with MDEV-24142.

MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison

Allow materialization strategy when collations on the
inner and outer sides of an IN subquery are the same and the
character set of the inner side is a proper subset of the character
set on the outer side.
This allows conversion from utf8mb3 to utf8mb4
as the former is a subset of the later.
This is only allowed when IN predicate is converted to an IN subquery

Backported part of the patch (d6a00d9b) of MDEV-17905.

os_thread_pf(): Remove.

os_thread_eq(), os_thread_yield(), os_thread_get_curr_id():
Define as macros.

ut_print_timestamp(), ut_sprintf_timestamp(): Simplify.

When executing set operations in a pipeline using only one temporary table
additional scans of intermediate results may be needed. The scans are
performed with usage of the rnd_next() handler function that might
leave record buffers used for the temporary table not in a state that
is good for following writes into the table. For example it happens for
aria engine when the last call of rnd_next() encounters only deleted
records. Thus a cleanup of record buffers is needed after each such scan
of the temporary table.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

This change is needed in 10.5 to avoid extra malloc calls in val_str().
In 10.6 it's not needed anymore but the extra +1 byte doesn't harm
that much.

- Fold long comment rows and updated comments
- Moved one private function in class Item_func_rand among other private
  functions

This is for Oracle compatiblity. ENABLED is in Oracle the default case
and just ensures that the NOT NULL constraints will be tested, which is
also default in MariaDB

Reviewed by:serg@mariadb.com

disable thd->count_cuted_fields when populating internal temporary
tables for I_S, because this is how SELECT works standalone.
And if the SELECT is a part of INSERT or UPDATE or RETURN or SET or
anything else that enables thd->count_cuted_fields, this counting should
only apply when storing the result of the SELECT in a field or a
variable, not when populating internal temporary tables for I_S.

This is a fixup patch for MDEV-23991 afc9d00c

We really should read result.n_leaf_pages, which was set previously.

Analysis and fix was provided by Jukka Santala. Thanks!

Reviewed by: Marko Mäkelä

A side effect of MDEV-16264 is that a large number of threads will
be created at server startup, to be destroyed after a minute or two.

One source of such thread creation is srv_start_periodic_timer().
InnoDB is creating 3 periodic tasks: srv_master_callback (1Hz)
srv_error_monitor_task (1Hz), and srv_monitor_task (0.2Hz).

It appears that we can merge srv_error_monitor_task and srv_monitor_task
and have them invoked 4 times per minute (every 15 seconds). This will
affect our ability to enforce innodb_fatal_semaphore_wait_threshold and
some computations around BUF_LRU_STAT_N_INTERVAL.

We could remove srv_master_callback along with the DROP TABLE queue
at some point of time in the future. We must keep it independent
of the innodb_fatal_semaphore_wait_threshold detection, because
the background DROP TABLE queue could get stuck due to dict_sys
being locked by another thread. For now, srv_master_callback
must be invoked once per second, so that
innodb_flush_log_at_timeout=1 can work.

BUF_LRU_STAT_N_INTERVAL: Reduce the precision and extend the time
from 50*1 second to 4*15 seconds.

srv_error_monitor_timer: Remove.

MAX_MUTEX_NOWAIT: Increase from 20*1 second to 2*15 seconds.

srv_refresh_innodb_monitor_stats(): Avoid a repeated call to time(NULL).
Change the interval to less than 60 seconds.

srv_monitor(): Renamed from srv_monitor_task.

srv_monitor_task(): Renamed from srv_error_monitor_task().
Invoked only once in 15 seconds. Invoke also srv_monitor().
Increase the fatal_cnt threshold from 10*1 second to 1*15 seconds.

sync_array_print_long_waits_low(): Invoke time(NULL) only once.
Remove a bogus message about printouts for 30 seconds. Those
printouts were effectively already disabled in MDEV-16264
(commit 5e62b6a5).

The purpose of the InnoDB page cleaner subsystem is to write out
modified pages from the buffer pool to data files. When the
innodb_max_dirty_pages_pct_lwm is not exceeded or
innodb_adaptive_flushing=ON decides not to write out anything,
the page cleaner should keep sleeping indefinitely until the state
of the system changes: a dirty page is added to the buffer pool such
that the page cleaner would no longer be idle.

buf_flush_page_cleaner(): Explicitly note when the page cleaner is idle.
When that happens, use mysql_cond_wait() instead of mysql_cond_timedwait().

buf_flush_insert_into_flush_list(): Wake up the page cleaner if needed.

innodb_max_dirty_pages_pct_update(),
innodb_max_dirty_pages_pct_lwm_update():
Wake up the page cleaner just in case.

Note: buf_flush_ahead(), buf_flush_wait_flushed() and shutdown are
already waking up the page cleaner thread.