- Fixing mariabackup.full_backup test case

trx_purge_free_segment(): If fseg_free_step_not_header() needs to be
called multiple times, acquire an exclusive latch on the
rollback segment header page after restarting the mini-transaction
so that the rest of this function cannot execute concurrently
with trx_undo_create() on the same rollback segment.

This fixes a regression that was introduced in
commit c14a3943 (MDEV-30753).

Note: The buffer-fixes that we are holding across the mini-transaction
restart will prevent the pages from being evicted from the buffer pool.
They may be accessed by other threads or written back to data files
while we are not holding exclusive latches.

Reviewed by: Vladislav Lesin

Remove ORACLE from the (session) sql_mode in connections made with sql
service to run init queries

The connection is new and the global variable value takes effect
rather than the session value from the caller of spider_db_init.

We must disable persistent statistics, because a transaction commit
from dict_stats_save() would occasionally interfere with this test.

Let us tolerate multiple "Memory pressure event freed"
in case there a real memory pressure event occurred
in addition to the one that this test simulates.
Also, clean up some SET variables.

1. fix useless correction of dec_arg *after it was used*
2. use DECIMAL_NOT_SPECIFIED instead of NOT_FIXED_DEC, because the latter
   is defined conditionally and has a wrong value in plugins

This reverts commit 9d19b652.

set the proper charset for a temporary result

buf_page_t::set_os_unused(): Remove the system call that had been added in
commit 16c97187 and revised in
commit c1fd082e for Microsoft Windows.

buf_pool_t::garbage_collect(): A new function to collect any garbage
from the InnoDB buffer pool that can be removed without writing any
log or data files. This will also invoke madvise() for all of buf_pool.free.

To trigger this the following MDEV is implemented:
MDEV-24670 avoid OOM by linux kernel co-operative memory management

To avoid frequent triggers that caused the MDEV-31953 regression, while
still preserving the 10.11 functionality of non-greedy kernel memory
usage, memory triggers are used.

On the triggering of memory pressure, if supported in the Linux kernel,
trigger the garbage collection of the innodb buffer pool.

The hard coded triggers occur where there is:
* some memory pressure in 5 of the last 10 seconds
* a full stall on memory pressure for 10ms in the last 2 seconds

The kernel will trigger only one in each of these time windows. To avoid
mariadb being in a constant state of memory garbage collection, this has
been limited to once per minute.

For a small set of kernels in 2023 (6.5, 6.6), there was a limit requiring
CAP_SYS_RESOURCE that was lifted[1] to support the use case of user
memory pressure. It not currently possible to set CAP_SYS_RESOURCES in
a systemd service as its setting a capability inside a usernamespace.

Running under systemd v254+ requires the default MemoryPressureWatch=auto
(or alternately "on").

Functionality was tested in a 6.4 kernel Fedora successfully under a
systemd service.

Running in a container requires that (unmask=)/sys/fs/cgroup be writable
by the mariadbd process.

To aid testing, the buf_pool_resize was a convient trigger point on
which to trigger garbage collection.

ref [1]: https://lore.kernel.org/all/CAMw=ZnQ56cm4Txgy5EhGYvR+Jt4s-KVgoA9_65HKWVMOXp7a9A@mail.gmail.com/T/#m3bd2a73c5ee49965cb73a830b1ccaa37ccf4e427

Co-Author: Daniel Black (on memory pressure trigger)

Reviewed by: Marko Mäkelä, Vladislav Vaintroub, Vladislav Lesin,
   Thirunarayanan Balathandayuthapani

Tested by: Matthias Leich

dict_find_max_space_id(): Return SELECT MAX(SPACE) FROM SYS_TABLES.

dict_check_tablespaces_and_store_max_id(): In the normal case
(no encryption plugin has been loaded and the change buffer is empty),
invoke dict_find_max_space_id() and do not open any .ibd files.
If a std::set<uint32_t> has been specified, open the files whose
tablespace ID is mentioned. Else, open all data files that are identified
by SYS_TABLES records.

fil_ibd_open(): Remove a call to os_file_get_last_error() that can
report a misleading error, such as EINVAL inside my_realpath() that is
not an actual error. This could be invoked when a data file is found
but the FSP_SPACE_FLAGS are incorrect, such as is the case for
table test.td in
./mtr --mysqld=--innodb-buffer-pool-dump-at-shutdown=0 innodb.table_flags

buf_load(): If any tablespaces could not be found, invoke
dict_check_tablespaces_and_store_max_id() on the missing tablespaces.

dict_load_tablespace(): Try to load the tablespace unless it was found
to be futile. This fixes failures related to FTS_*.ibd files for
FULLTEXT INDEX.

btr_cur_t::search_leaf(): Prevent a crash when the tablespace
does not exist. This was caught by the test innodb_fts.fts_concurrent_insert
when the change to dict_load_tablespaces() was not present.

We modify a few tests to ensure that tables will not be loaded at startup.
For some fault injection tests this means that the corrupted tables
will not be loaded, because dict_load_tablespace() would perform stricter
checks than dict_check_tablespaces_and_store_max_id().

Tested by: Matthias Leich
Reviewed by: Thirunarayanan Balathandayuthapani

This is a 10.5 backport of 10.6
commit d4265fbd.

page_cleaner_flush_pages_recommendation(): If dirty_pct is
between innodb_max_dirty_pages_pct_lwm
and innodb_max_dirty_pages_pct,
scale the effort relative to how close we are to
innodb_max_dirty_pages_pct.

The previous formula was missing a multiplication by 100.

This is a 10.5 backport from 10.6
commit 9593cccf.

Adaptive flushing is enabled by setting innodb_max_dirty_pages_pct_lwm>0
(not default) and innodb_adaptive_flushing=ON (default).
There is also the parameter innodb_adaptive_flushing_lwm
(default: 10 per cent of the log capacity). It should enable some
adaptive flushing even when innodb_max_dirty_pages_pct_lwm=0.
That is not being changed here.

This idea was first presented by Inaam Rana several years ago,
and I discussed it with Jean-François Gagné at FOSDEM 2023.

buf_flush_page_cleaner(): When we are not near the log capacity limit
(neither buf_flush_async_lsn nor buf_flush_sync_lsn are set),
also try to move clean blocks from the buf_pool.LRU list to buf_pool.free
or initiate writes (but not the eviction) of dirty blocks, until
the remaining I/O capacity has been consumed.

buf_flush_LRU_list_batch(): Add the parameter bool evict, to specify
whether dirty least recently used pages (from buf_pool.LRU) should
be evicted immediately after they have been written out. Callers outside
buf_flush_page_cleaner() will pass evict=true, to retain the existing
behaviour.

buf_do_LRU_batch(): Add the parameter bool evict.
Return counts of evicted and flushed pages.

buf_flush_LRU(): Add the parameter bool evict.
Assume that the caller holds buf_pool.mutex and
will invoke buf_dblwr.flush_buffered_writes() afterwards.

buf_flush_list_holding_mutex(): A low-level variant of buf_flush_list()
whose caller must hold buf_pool.mutex and invoke
buf_dblwr.flush_buffered_writes() afterwards.

buf_flush_wait_batch_end_acquiring_mutex(): Remove. It is enough to have
buf_flush_wait_batch_end().

page_cleaner_flush_pages_recommendation(): Avoid some floating-point
arithmetics.

buf_flush_page(), buf_flush_check_neighbor(), buf_flush_check_neighbors(),
buf_flush_try_neighbors(): Rename the parameter "bool lru" to "bool evict".

buf_free_from_unzip_LRU_list_batch(): Remove the parameter.
Only actual page writes will contribute towards the limit.

buf_LRU_free_page(): Evict freed pages of temporary tables.

buf_pool.done_free: Broadcast whenever a block is freed
(and buf_pool.try_LRU_scan is set).

buf_pool_t::io_buf_t::reserve(): Retry indefinitely.
During the test encryption.innochecksum we easily run out of
these buffers for PAGE_COMPRESSED or ENCRYPTED pages.

Tested by Matthias Leich and Axel Schwenke

ha_innobase::extra(): Do not invoke log_buffer_flush_to_disk()
if high_level_read_only holds.

log_buffer_flush_to_disk(): Remove an assertion that duplicates one
at the start of log_write_up_to().

This test occasionally fails with a failure to purge history.
Let us try to purge everything before starting the interesting part,
to make that occasional failure go away.

MDEV-32811 Potentially broken crash recovery if a mini-transaction frees a page, not modifying previously clean pages

- The 11.2 test innodb.sys_truncate_debug fails while executing insert statement.
Reason for the failure is that same mini-transaction does freeing, allocating
and freeing the same page. Page initialization clears the FIL_PAGE_LSN
on the page, fails to set the FIL_PAGE_LSN after freeing the same page.
This issue is caused by commit f46efb44

mtr_t::commit(): Should set the FIL_PAGE_LSN even though page is freed

trx_undo_free_page(): Detect a case of corrupted TRX_UNDO_PAGE_LIST.

trx_undo_truncate_end(): Stop attempts to truncate a corrupted log.

trx_t::commit_empty(): Add an error message of a corrupted log.

Reviewed by: Thirunarayanan Balathandayuthapani

trx_purge_truncate_history(): Do not prematurely mark dirty pages
as clean. This will be done in mtr_t::commit_shrink() as part of
Shrink::operator()(mtr_memo_slot_t*). Also, register each dirty page
only once in the mini-transaction.

fsp_page_create(): Adjust and simplify the page creation during
undo tablespace truncation. We can directly reuse pages that are
already in buf_pool.page_hash.

This fixes a regression that was caused by
commit f5794e1d (MDEV-26445).

Tested by: Matthias Leich
Reviewed by: Thirunarayanan Balathandayuthapani

Commit Removed Ubuntu Bionic from
debian/autobake-debs.sh as it's not used
anymore to build official MariaDB images

REMINDER TO MERGER: This commit should not
be merged up to 10.6 or forward

Commit Removed Ubuntu Bionic from
debian/autobake-debs.sh as it's not used
anymore to build official MariaDB images

innodb_preshutdown(): Only wait for active transactions to be terminated
if InnoDB was started and innodb_force_recovery=3 or larger does not
prevent a rollback.

This fixes the following:

./mtr --parallel=auto --mysqld=--innodb-fast-shutdown=0 \
innodb.log_file_size innodb.innodb_force_recovery \
innodb.read_only_recovery innodb.read_only_recover_committed \
mariabackup.apply-log-only-incr

recv_ring::copy_if_needed(): If len==0, do not decrypt anything,
to avoid unnecessary work and a possible crash.

This is similar to the condition recv_buf::copy_if_needed() where
a similar condition was present since the beginning
(commit 685d958e).

Tested by: Matthias Leich

Fixes up commit 2027c482

MDEV-32737 innodb.log_file_name fails on Assertion `after_apply || !(blocks).end in recv_sys_t::clear

recv_group_scan_log_recs(): Set the debug flag recv_sys.after_apply
after actually completing the log scan.

In the test, suppress some errors that may be reported when
the crash recovery of RENAME TABLE t1 TO t2 is preceded by
copying t2.ibd to t1.ibd.