btr_attach_half_pages(), btr_lift_page_up(), btr_compress():
Return DB_CORRUPTION if btr_page_get_father_block() returns nullptr

Reviewed by: Thirunarayanan Balathandayuthapani

In debug builds, let us declare dict_sys.latch as index_lock instead of
srw_lock, so that we will benefit from the full tracking of lock ownership.

lock_table_for_trx(): Assert that the current thread is not holding
dict_sys.latch. If the dict_sys.unfreeze() call were moved to the end of
lock_table_children(), this assertion would fail in the test innodb.innodb
and many other tests that use FOREIGN KEY.

MDEV-32899 InnoDB is holding shared dict_sys.latch while waiting for FOREIGN KEY child table lock on DDL

lock_table_children(): A new function to lock all child tables of a table.
We will only hold dict_sys.latch while traversing
dict_table_t::referenced_set. To prevent a race condition with
std::set::erase() we will copy the pointers to the child tables to a
local vector. Once we have acquired references to all child tables,
we can safely release dict_sys.latch, wait for the locks, and finally
release the references.

This fixes up commit 2ca11234 (MDEV-26217)
and commit c3c53926 (MDEV-26554).

MDEV-32879 Server crash in my_decimal::operator= or unexpected ER_DUP_ENTRY upon comparison with INET6 and similar types

This is the 10.6 version of the patch.

Item_bool_rowready_func2, Item_func_between, Item_func_in
did not check if a not-NULL argument of an arbitrary data type
can produce a NULL value on conversion to INET6.

This caused a crash on DBUG_ASSERT() in conversion failures,
because the function returned SQL NULL for something that
has Item::maybe_null() equal to false.

Adding setting NULL-ability in such cases.

Details:

- Removing the code in Item_func::setup_args_and_comparator()
  performing character set aggregation with optional narrowing.
  This aggregation is done inside Arg_comparator::set_cmp_func_string().
  So this code was redundant

- Removing Item_func::setup_args_and_comparator() as it git simplified to
  just to two lines:
    convert_const_compared_to_int_field(thd);
    return cmp->set_cmp_func(thd, this, &args[0], &args[1], true);
  Using these lines directly in:
    - Item_bool_rowready_func2::fix_length_and_dec()
    - Item_func_nullif::fix_length_and_dec()

- Adding a new virtual method:
  - Type_handler::Item_bool_rowready_func2_fix_length_and_dec().

- Adding tests detecting if the data type conversion can return SQL NULL into
  the following methods of Type_handler_fbt:
  - Item_bool_rowready_func2_fix_length_and_dec
  - Item_func_between_fix_length_and_dec
  - Item_func_in_fix_comparator_compatible_types

Let us wait for the completion of purge before testing the KILL of
CREATE INDEX c2d ON t1(c2), so that there will be no table handle
acquisition by a purge task before the operation is rolled back.

Also, let us make the test compatible with ./mtr --repeat,
and convert variable_value from string to integer so that any
comparisons will be performed correctly.

When the constant OS_AIO_N_PENDING_IOS_PER_THREAD is changed from 256 to 1
and the server is run with the minimum parameters
innodb_read_io_threads=1 and innodb_write_io_threads=2, two hangs
were observed.

tpool::cache<T>::put(T*): Ensure that get() in io_slots::acquire()
will be woken up when the cache previously was empty.

buf_pool_t::io_buf_t::reserve(): Schedule a possibly partial doublewrite
batch so that os_aio_wait_until_no_pending_writes() has a chance of
returning. Add a Boolean parameter and pass wait_for_reads=false inside
buf_page_decrypt_after_read(), because those calls will be executed
inside a read completion callback, and therefore
os_aio_wait_until_no_pending_reads() would block indefinitely.

In any test that uses wait_all_purged.inc, ensure that InnoDB tables
will be created without persistent statistics.

This is a follow-up to commit cd04673a
after a similar failure was observed in the innodb_zip.blob test.

trx_purge_free_segment(): If fseg_free_step_not_header() needs to be
called multiple times, acquire an exclusive latch on the
rollback segment header page after restarting the mini-transaction
so that the rest of this function cannot execute concurrently
with trx_undo_create() on the same rollback segment.

This fixes a regression that was introduced in
commit c14a3943 (MDEV-30753).

Note: The buffer-fixes that we are holding across the mini-transaction
restart will prevent the pages from being evicted from the buffer pool.
They may be accessed by other threads or written back to data files
while we are not holding exclusive latches.

Reviewed by: Vladislav Lesin

We must disable persistent statistics, because a transaction commit
from dict_stats_save() would occasionally interfere with this test.

1. fix useless correction of dec_arg *after it was used*
2. use DECIMAL_NOT_SPECIFIED instead of NOT_FIXED_DEC, because the latter
   is defined conditionally and has a wrong value in plugins

This reverts commit 9d19b652.

set the proper charset for a temporary result

dict_find_max_space_id(): Return SELECT MAX(SPACE) FROM SYS_TABLES.

dict_check_tablespaces_and_store_max_id(): In the normal case
(no encryption plugin has been loaded and the change buffer is empty),
invoke dict_find_max_space_id() and do not open any .ibd files.
If a std::set<uint32_t> has been specified, open the files whose
tablespace ID is mentioned. Else, open all data files that are identified
by SYS_TABLES records.

fil_ibd_open(): Remove a call to os_file_get_last_error() that can
report a misleading error, such as EINVAL inside my_realpath() that is
not an actual error. This could be invoked when a data file is found
but the FSP_SPACE_FLAGS are incorrect, such as is the case for
table test.td in
./mtr --mysqld=--innodb-buffer-pool-dump-at-shutdown=0 innodb.table_flags

buf_load(): If any tablespaces could not be found, invoke
dict_check_tablespaces_and_store_max_id() on the missing tablespaces.

dict_load_tablespace(): Try to load the tablespace unless it was found
to be futile. This fixes failures related to FTS_*.ibd files for
FULLTEXT INDEX.

btr_cur_t::search_leaf(): Prevent a crash when the tablespace
does not exist. This was caught by the test innodb_fts.fts_concurrent_insert
when the change to dict_load_tablespaces() was not present.

We modify a few tests to ensure that tables will not be loaded at startup.
For some fault injection tests this means that the corrupted tables
will not be loaded, because dict_load_tablespace() would perform stricter
checks than dict_check_tablespaces_and_store_max_id().

Tested by: Matthias Leich
Reviewed by: Thirunarayanan Balathandayuthapani

This is a 10.5 backport of 10.6
commit d4265fbd.

page_cleaner_flush_pages_recommendation(): If dirty_pct is
between innodb_max_dirty_pages_pct_lwm
and innodb_max_dirty_pages_pct,
scale the effort relative to how close we are to
innodb_max_dirty_pages_pct.

The previous formula was missing a multiplication by 100.

This is a 10.5 backport from 10.6
commit 9593cccf.

Adaptive flushing is enabled by setting innodb_max_dirty_pages_pct_lwm>0
(not default) and innodb_adaptive_flushing=ON (default).
There is also the parameter innodb_adaptive_flushing_lwm
(default: 10 per cent of the log capacity). It should enable some
adaptive flushing even when innodb_max_dirty_pages_pct_lwm=0.
That is not being changed here.

This idea was first presented by Inaam Rana several years ago,
and I discussed it with Jean-François Gagné at FOSDEM 2023.

buf_flush_page_cleaner(): When we are not near the log capacity limit
(neither buf_flush_async_lsn nor buf_flush_sync_lsn are set),
also try to move clean blocks from the buf_pool.LRU list to buf_pool.free
or initiate writes (but not the eviction) of dirty blocks, until
the remaining I/O capacity has been consumed.

buf_flush_LRU_list_batch(): Add the parameter bool evict, to specify
whether dirty least recently used pages (from buf_pool.LRU) should
be evicted immediately after they have been written out. Callers outside
buf_flush_page_cleaner() will pass evict=true, to retain the existing
behaviour.

buf_do_LRU_batch(): Add the parameter bool evict.
Return counts of evicted and flushed pages.

buf_flush_LRU(): Add the parameter bool evict.
Assume that the caller holds buf_pool.mutex and
will invoke buf_dblwr.flush_buffered_writes() afterwards.

buf_flush_list_holding_mutex(): A low-level variant of buf_flush_list()
whose caller must hold buf_pool.mutex and invoke
buf_dblwr.flush_buffered_writes() afterwards.

buf_flush_wait_batch_end_acquiring_mutex(): Remove. It is enough to have
buf_flush_wait_batch_end().

page_cleaner_flush_pages_recommendation(): Avoid some floating-point
arithmetics.

buf_flush_page(), buf_flush_check_neighbor(), buf_flush_check_neighbors(),
buf_flush_try_neighbors(): Rename the parameter "bool lru" to "bool evict".

buf_free_from_unzip_LRU_list_batch(): Remove the parameter.
Only actual page writes will contribute towards the limit.

buf_LRU_free_page(): Evict freed pages of temporary tables.

buf_pool.done_free: Broadcast whenever a block is freed
(and buf_pool.try_LRU_scan is set).

buf_pool_t::io_buf_t::reserve(): Retry indefinitely.
During the test encryption.innochecksum we easily run out of
these buffers for PAGE_COMPRESSED or ENCRYPTED pages.

Tested by Matthias Leich and Axel Schwenke

ha_innobase::extra(): Do not invoke log_buffer_flush_to_disk()
if high_level_read_only holds.

log_buffer_flush_to_disk(): Remove an assertion that duplicates one
at the start of log_write_up_to().

This test occasionally fails with a failure to purge history.
Let us try to purge everything before starting the interesting part,
to make that occasional failure go away.

trx_undo_free_page(): Detect a case of corrupted TRX_UNDO_PAGE_LIST.

trx_undo_truncate_end(): Stop attempts to truncate a corrupted log.

trx_t::commit_empty(): Add an error message of a corrupted log.

Reviewed by: Thirunarayanan Balathandayuthapani

trx_purge_truncate_history(): Do not prematurely mark dirty pages
as clean. This will be done in mtr_t::commit_shrink() as part of
Shrink::operator()(mtr_memo_slot_t*). Also, register each dirty page
only once in the mini-transaction.

fsp_page_create(): Adjust and simplify the page creation during
undo tablespace truncation. We can directly reuse pages that are
already in buf_pool.page_hash.

This fixes a regression that was caused by
commit f5794e1d (MDEV-26445).

Tested by: Matthias Leich
Reviewed by: Thirunarayanan Balathandayuthapani

Commit Removed Ubuntu Bionic from
debian/autobake-debs.sh as it's not used
anymore to build official MariaDB images

REMINDER TO MERGER: This commit should not
be merged up to 10.6 or forward

Commit Removed Ubuntu Bionic from
debian/autobake-debs.sh as it's not used
anymore to build official MariaDB images

innodb_preshutdown(): Only wait for active transactions to be terminated
if InnoDB was started and innodb_force_recovery=3 or larger does not
prevent a rollback.

This fixes the following:

./mtr --parallel=auto --mysqld=--innodb-fast-shutdown=0 \
innodb.log_file_size innodb.innodb_force_recovery \
innodb.read_only_recovery innodb.read_only_recover_committed \
mariabackup.apply-log-only-incr

Fixes up commit 2027c482

MDEV-32737 innodb.log_file_name fails on Assertion `after_apply || !(blocks).end in recv_sys_t::clear

recv_group_scan_log_recs(): Set the debug flag recv_sys.after_apply
after actually completing the log scan.

In the test, suppress some errors that may be reported when
the crash recovery of RENAME TABLE t1 TO t2 is preceded by
copying t2.ibd to t1.ibd.

Checking for kill with thd_kill_level() or check_killed() runs apc
requests, which takes the LOCK_thd_kill mutex. But this is dangerous,
as checking for kill needs to be called while holding many different
mutexes, and can lead to cyclic mutex dependency and deadlock.

But running apc is only "best effort", so skip running the apc if the
LOCK_thd_kill is not available. The apc will then be run on next check
of kill signal.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>