Problem:
=======
When we upgrade from "mysql" to "mariadb" if slave is using repositories as
tables their data is completely ignored and no warning is issued in error log.

Fix:
===
"mysql_upgrade" test should check for the presence of data in
"mysql.slave_master_info" and "mysql.slave_relay_log_info" tables. When tables
have some data the upgrade script should report a warning which hints users
that the data in repository tables will be ignored.

This task deals with packing the sort key inside the sort buffer, which  would
lead to efficient usage of the memory allocated for the sort buffer.

The changes brought by this feature are
  1) Sort buffers would have sort keys of variable length
  2) The format for sort keys inside the sort buffer would look like
     |<sort_length><null_byte><key_part1><null_byte><key_part2>.......|
      sort_length is the extra bytes that are required to store the variable
      length of a sort key.
  3) When packing of sort key is done we store the ORIGINAL VALUES inside
     the sort buffer and not the STRXFRM form (mem-comparable sort keys).
  4) Special comparison function packed_keys_comparison() is introduced
     to compare 2 sort keys.

This patch also contains contributions from Sergei Petrunia.

PageBulk::insertPage(): Check the array bounds before comparing.
We used to read one byte beyond the end of the 'rec' payload.
The incorrect logic was originally introduced in
commit 7ae21b18.

buf_flush_freed_page(): Reformat in the common style, and
simplify some code. Prefer to request all information from
smaller data structures (buf_page_t) than from fil_space_t
or the global variable srv_immediate_scrub_data_uncompressed.

SysTablespace::open_or_create(): Assert that the temporary
tablespace will not be created in page_compressed format, so that
buf_flush_freed_page() can avoid checking that on every call.

IORequest: Remove duplicated constructors, and do not explicitly
declare a default constructor.

Related to MDEV-21769.

Related to MDEV-21769.

After the '.so' one is supposed to use the directory name, like we have
correctly in all old man pages:
  mysql_client_test_embedded.1:.so man1/mysql_client_test.1
  mysql_embedded.1:.so man1/mysql.1
  mysqltest_embedded.1:.so man1/mysqltest.1

This change adds the 'man1/' component so the link has the correct format.
Actually using man links is a deprecated practice and using symlinks
would be better, but that can be fixed in a later commit.

From https://www.debian.org/doc/debian-policy/ch-docs.html#manual-pages:
> If one man page needs to be accessible via several names it is better
> to use a symbolic link than the .so feature

Detected via Lintian errors:
  E: mariadb-server-10.5: bad-so-link-within-manual-page usr/share/man/man1/mariadb-show.1.gz
  E: mariadb-client-10.5: bad-so-link-within-manual-page usr/share/man/man1/mariadb-binlog.1.gz
  E: mariadb-client-10.5: bad-so-link-within-manual-page usr/share/man/man1/mariadb-plugin.1.gz

Related to MDEV-21769.

This fixes errors like:

  $ LC_ALL=en_US.UTF-8 MANROFFSEQ='' MANWIDTH=80 \
    man --warnings -E UTF-8 -l -Tutf8 -Z mysql-test-run.pl.1 > /dev/null

  troff: <standard input>:246: warning [p 2, 6.0i, div '3tbd1,1', 0.3i]: can't break line
  troff: <standard input>:275: warning [p 2, 6.0i, div '3tbd6,1', 0.8i]: can't break line

Related to MDEV-21769.

Related to MDEV-21769.

Added CYCLE ... RESTRICT (nonstandard) clause to recursive CTE.

The following parameters are deprecated:

  innodb-background-scrub-data-uncompressed
  innodb-background-scrub-data-compressed
  innodb-background-scrub-data-interval
  innodb-background-scrub-data-check-interval

Removed scrubbing code completely(btr0scrub.h, btr0scrub.cc)
Removed information_schema.innodb_tablespaces_scrubbing tables
Removed the scrubbing logic from fil_crypt_thread()

When a InnoDB data file page is freed, its contents becomes garbage,
and any storage allocated in the data file is wasted. During flushing,
InnoDB initializes the page with zeros if scrubbing is enabled. If the
tablespace is compressed then InnoDB should punch a hole else ignore the
flushing of the freed page.

buf_page_t:
- Replaced the variable file_page_was_freed, init_on_flush in buf_page_t
with status enum variable.
- Changed all debug assert of file_page_was_freed to DBUG_ASSERT
of buf_page_t::status

Removed buf_page_set_file_page_was_freed(),
buf_page_reset_file_page_was_freed().

buf_page_free(): Newly added function which takes X-lock on the page
before marking the status as FREED. So that InnoDB flush handler can
avoid concurrent flush of the freed page. Also while flushing the page,
InnoDB make sure that redo log which does freeing of the page also written
to the disk. Currently, this function only marks the page as FREED if
it is in buffer pool

buf_flush_freed_page(): Newly added function which initializes zeros
asynchorously if innodb_immediate_scrub_data_uncompressed is enabled.
Punch a hole to the file synchorously if page_compressed is enabled.
Reset the io_fix to NORMAL. Release the block from flush list and
associated mutex before writing zeros or punch a hole to the file.

buf_flush_page(): Removed the unnecessary usage of temporary
variable "flush"

fil_io(): Introduce new parameter called punch_hole. It allows fil_io()
to punch the hole to the file for the given offset.

buf_page_create(): Let the callers assign buf_page_t::status.
Every caller should eventually invoke mtr_t::init().

fsp_page_create(): Remove the unused mtr_t parameter.

In all other callers of buf_page_create() except fsp_page_create(),
before invoking mtr_t::init(), invoke
mtr_t::sx_latch_at_savepoint() or mtr_t::x_latch_at_savepoint().

mtr_t::init(): Initialize buf_page_t::status also for the temporary
tablespace (when redo logging is disabled), to avoid assertion failures.

MDEV-21833 Make slave_run_triggers_for_rbr enforce triggers to run on slave, even when there are triggers on the master

A bit changed patch of Anders Karlsson with examples added.

New parameters "ENFORCE" to slave-run-triggers-for-rbr added.

MDEV-21854 xa commit `xid` one phase for already prepared transaction must always error out

Added state and one-phase option checks to XA "external" commit/rollback
branches. While the XA standard does not prohibit it,
Commit and Rollback of an XA external to the current ongoing transaction
is not allowed; after all the current transaction may rollback
to not being able to revert that decision.

Let ulong_num being converted by strtoul() rather than strtol().

Engine (InnoDB) and XA replication MDEV-742 requires the XID member
be of a constant minimum across supported platform ulong size which
is 4 bytes.

recv_log_recover_10_4(): Add a missing bit pattern negation that
was forgotten when commit f8a9f906
(MDEV-12353) removed the support for crash-upgrading.

btr_cur_upd_rec_in_place(): Invoke page_zip_rec_set_deleted()
for ROW_FORMAT=COMPRESSED pages, so that the change will be
written to the redo log.

This part of crash recovery was broken in
commit 08ba3887 (MDEV-12353).

create_log_file(): Delete all old redo log files where they used to be
deleted, after the crash injection point innodb_log_abort_6,
before commit 9ef2d29f
deprecated and ignored the setting innodb_log_files_in_group.

log_crypt_101_read_checkpoint(), log_crypt_101_read_block():
Declare as ATTRIBUTE_COLD. These are only used when
checking that a MariaDB 10.1 encrypted redo log is clean.

log_block_calc_checksum_format_0(): Define in the only
compilation unit where it is needed. This is only used
when reading the checkpoint information from redo logs
before MariaDB 10.2.2.

crypt_info_t: Declare the byte arrays directly with alignas().

log_crypt(): Use memcpy_aligned instead of reinterpret_cast
on integers.

Also, correctly document what recv_sys.mutex is protecting.

It is like show_create_table() but allows the caller to specify the
database_name and table_name which are to be printed.

- Don't try to push down SELECTs that have a side effect

- In case the storage engine did support pushdown of SELECT with an INTO
  clause, write the rows we've got from it into select->join->result,
  and not thd->protocol.  This way, SELECT ... INTO ... FROM
  smart_engine_table will put the result into where instructed, and
  NOT send it to the client.

recv_log_recover_pre_10_2(): Merged from
recv_find_max_checkpoint_0(), recv_log_format_0_recover().

Some fields were protected by log_sys.mutex, which adds quite some
overhead for readers. Some readers were submitting dirty reads.

log_t::lsn: Declare private and atomic. Add wrappers get_lsn()
and set_lsn() that will use relaxed memory access. Many accesses
to log_sys.lsn are still protected by log_sys.mutex; we avoid the
mutex for some readers.

log_t::flushed_to_disk_lsn: Declare private and atomic, and move
to the same cache line with log_t::lsn.

log_t::buf_free: Declare as size_t, and move to the same cache line
with log_t::lsn.

log_t::check_flush_or_checkpoint_: Declare private and atomic,
and move to the same cache line with log_t::lsn.

log_get_lsn(): Define as an alias of log_sys.get_lsn().

log_get_lsn_nowait(), log_peek_lsn(): Remove.

log_get_flush_lsn(): Define as an alias of log_sys.get_flush_lsn().

log_t::initiate_write(): Replaces log_buffer_sync_in_background().

O_DSYNC is faster than O_SYNC because it syncs as little as needed
(e.g. no timestamp changes)

This change is similar to change fsync() -> fdatasync() in MDEV-21382

Since this tool is already shipped in the system, don't provide it from
MariaDB from 10.5 series onwards. It will potentially just create an
installation failure due to conflict if users install rocksdb-tools.

Package rocksdb-tools in Debian since Buster (10):
https://packages.debian.org/search?suite=buster&arch=any&mode=path&searchon=contents&keywords=sst_dump

Package rocksdb-tools in Ubuntu since Disco (19.04):
https://packages.ubuntu.com/search?suite=disco&arch=any&mode=exactfilename&searchon=contents&keywords=sst_dump

This change is adopted from Debian the change has been since Dec 18, 2018.
https://salsa.debian.org/mariadb-team/mariadb-10.3/-/commit/d50e2eaf1fad46c494bd0f090b0e75c19ac3f41a

The command is needed to install/upgrade databases. It should therefore
reside in the server-core package, as that package is supposed to contain
all necessary tools to start and run the database (although manually,
there are no automation or service scripts included in server-core by
design).

Also add Breaks+Replaces in the control so upgrades will not complain about
overwriting files in previously installed packages of different name.

This was adopted from Debian where this change has been since Nov 4, 2018.
https://salsa.debian.org/mariadb-team/mariadb-10.1/-/commit/3689350ad30916aa68fe2c1b35c290fb5095bc9b

In addition, also move all *.sql files referenced mysql_install_db for
the same reasons.

This amends commit 37e7bde1

The ut_crc32() function uses a hard-coded initial CRC-32C value of 0.
Replace it with ut_crc32_low(), which allows to specify the initial
checksum value, and provide an inlined compatibility wrapper ut_crc32().

Also, remove non-inlined wrapper functions on ARMv8 and POWER8,
and remove dead code (the generic implementation) on POWER8.

Note: The original AMD64 instruction set architecture in 2003 only
included SSE2. The CRC-32C instructions are part of the SSE4.2
instruction set extension for IA-32 and AMD64, with first processors
released in November 2007 (using the AMD Barcelona microarchitecture)
and November 2008 (Intel Nehalem microarchiteture). It might be safe
to assume that SSE4.2 is available on all currently used AMD64 based
systems, but we are not taking that step yet.

The configuration parameter innodb_scrub_log never really worked, as
reported in MDEV-13019 and MDEV-18370.

Because MDEV-14425 is changing the redo log format, the innodb_scrub_log
feature would have to be adjusted for it. Due to the known problems,
it is easier to remove the feature for now, and to ignore and deprecate
the parameters.

If old log contents should be kept secret, then enabling innodb_encrypt_log
or setting a smaller innodb_log_file_size could help.

MONITOR_PENDING_CHECKPOINT_WRITE and MONITOR_LOG_IO track
log_sys.n_pending_checkpoint_writes and log_sys.n_log_ios,
respectively. The MONITOR_INC calls are redundant, because
the values will be overwritten in srv_mon_process_existing_counter().

Compute MONITOR_LSN_CHECKPOINT_AGE on demand in
srv_mon_process_existing_counter().
This allows us to remove the overhead of MONITOR_SET
calls for the counter.