We will remove the InnoDB background operation of merging buffered
changes to secondary index leaf pages. Changes will only be merged as a
result of an operation that accesses a secondary index leaf page,
such as a SQL statement that performs a lookup via that index,
or is modifying the index. Also ROLLBACK and some background operations,
such as purging the history of committed transactions, or computing
index cardinality statistics, can cause change buffer merge.
Encryption key rotation will not perform change buffer merge.

The motivation of this change is to simplify the I/O logic and to
allow crash recovery to happen in the background (MDEV-14481).
We also hope that this will reduce the number of "mystery" crashes
due to corrupted data. Because change buffer merge will typically
take place as a result of executing SQL statements, there should be
a clearer connection between the crash and the SQL statements that
were executed when the server crashed.

In many cases, a slight performance improvement was observed.

This is joint work with Thirunarayanan Balathandayuthapani
and was tested by Axel Schwenke and Matthias Leich.

The InnoDB monitor counter innodb_ibuf_merge_usec will be removed.

On slow shutdown (innodb_fast_shutdown=0), we will continue to
merge all buffered changes (and purge all undo log history).

Two InnoDB configuration parameters will be changed as follows:

innodb_disable_background_merge: Removed.
This parameter existed only in debug builds.
All change buffer merges will use synchronous reads.

innodb_force_recovery will be changed as follows:
* innodb_force_recovery=4 will be the same as innodb_force_recovery=3
(the change buffer merge cannot be disabled; it can only happen as
a result of an operation that accesses a secondary index leaf page).
The option used to be capable of corrupting secondary index leaf pages.
Now that capability is removed, and innodb_force_recovery=4 becomes 'safe'.
* innodb_force_recovery=5 (which essentially hard-wires
SET GLOBAL TRANSACTION ISOLATION LEVEL READ UNCOMMITTED)
becomes safe to use. Bogus data can be returned to SQL, but
persistent InnoDB data files will not be corrupted further.
* innodb_force_recovery=6 (ignore the redo log files)
will be the only option that can potentially cause
persistent corruption of InnoDB data files.

Code changes:

buf_page_t::ibuf_exist: New flag, to indicate whether buffered
changes exist for a buffer pool page. Pages with pending changes
can be returned by buf_page_get_gen(). Previously, the changes
were always merged inside buf_page_get_gen() if needed.

ibuf_page_exists(const buf_page_t&): Check if a buffered changes
exist for an X-latched or read-fixed page.

buf_page_get_gen(): Add the parameter allow_ibuf_merge=false.
All callers that know that they may be accessing a secondary index
leaf page must pass this parameter as allow_ibuf_merge=true,
unless it does not matter for that caller whether all buffered
changes have been applied. Assert that whenever allow_ibuf_merge
holds, the page actually is a leaf page. Attempt change buffer
merge only to secondary B-tree index leaf pages.

btr_block_get(): Add parameter 'bool merge'.
All callers of btr_block_get() should know whether the page could be
a secondary index leaf page. If it is not, we should avoid consulting
the change buffer bitmap to even consider a merge. This is the main
interface to requesting index pages from the buffer pool.

ibuf_merge_or_delete_for_page(), recv_recover_page(): Replace
buf_page_get_known_nowait() with much simpler logic, because
it is now guaranteed that that the block is x-latched or read-fixed.

mlog_init_t::mark_ibuf_exist(): Renamed from mlog_init_t::ibuf_merge().
On crash recovery, we will no longer merge any buffered changes
for the pages that we read into the buffer pool during the last batch
of applying log records.

buf_page_get_gen_known_nowait(), BUF_MAKE_YOUNG, BUF_KEEP_OLD: Remove.

btr_search_guess_on_hash(): Merge buf_page_get_gen_known_nowait()
to its only remaining caller.

buf_page_make_young_if_needed(): Define as an inline function.
Add the parameter buf_pool.

buf_page_peek_if_young(), buf_page_peek_if_too_old(): Add the
parameter buf_pool.

fil_space_validate_for_mtr_commit(): Remove a bogus comment
about background merge of the change buffer.

btr_cur_open_at_rnd_pos_func(), btr_cur_search_to_nth_level_func(),
btr_cur_open_at_index_side_func(): Use narrower data types and scopes.

ibuf_read_merge_pages(): Replaces buf_read_ibuf_merge_pages().
Merge the change buffer by invoking buf_page_get_gen().

In commit 0f7732d1
we introduced a innodb_checksum_algorithm=full_crc32 combination
to a number of encryption tests, and also fixed the code accordingly.

The default in MariaDB 10.5 is innodb_checksum_algorithm=full_crc32.
In a test merge to 10.5, the test encryption.innodb-redo-badkey failed
once due to a message that had been added in that commit.

Let us introduce a full_crc32 option to that test.
And let us use strict_crc32 and strict_full_crc32 instead of the
non-strict variants, for the previously augmented tests, to be in
line with the earlier tests encryption.corrupted_during_recovery and
encryption.innodb_encrypt_temporary_tables.

btr_page_get_split_rec_to_left(): Assert that in the leftmost leaf page,
if the metadata record exists, index->is_instant() must hold.
The assertion of commit 01f45bec
could fail during innobase_instant_try().

These two methods:
- Item_result_field::create_tmp_field_ex()
- Item_func_user_var::create_tmp_field_ex()
had duplicate code, except that they used a different type handler.
Adding a protected method Item_result_field::create_tmp_field_ex_from_handler()
with a "const Type_handler*" parameter, and reusing it from the
two mentioned methods.

btr_page_get_split_rec_to_left(): Assert that in the leftmost leaf page,
the metadata record exists if and only if index->is_instant().

page_validate(): Correct the wording of a message.

rec_init_offsets(): Assert that whenever a record is in "instant ALTER"
format, index->is_instant() must hold.

In MDEV-11369 (instant ADD COLUMN) in MariaDB Server 10.3,
we introduced the hidden metadata record that must be the
first record in the clustered index if and only if
index->is_instant() holds.

To catch MDEV-19783, in
commit ed0793e0 and
commit 99dc40d6
we added some assertions to find cases where
the metadata record is missing while it should not be, or a
record exists when it should not. Those assertions were invalid
when traversing the PAGE_FREE list. That list can contain anything;
we must only be able to determine the successor and the size of
each garbage record in it.

page_validate(), page_simple_validate_old(), page_simple_validate_new():
Do not invoke page_rec_get_next_const() for traversing the PAGE_FREE
list, but instead use a lower-level accessor that does not attempt to
validate the REC_INFO_MIN_REC_FLAG.

page_copy_rec_list_end_no_locks(),
page_copy_rec_list_start(), page_delete_rec_list_start():
Add assertions.

btr_page_get_split_rec_to_left(): Remove a redundant return value,
and make the output parameter the return value.

btr_page_get_split_rec_to_right(), btr_page_split_and_insert(): Clean up.

When MDEV-12026 introduced innodb_checksum_algorithm=full_crc32 in
MariaDB 10.4, it accidentally added a dependency on buf_page_t::encrypted.
Now that the flag has been removed, we must adjust the page-read routine.

buf_page_io_complete(): When the full_crc32 page checksum matches but the
tablespace ID in the page does not match after decrypting, we should
declare it a decryption failure and suppress the page dump output and
any attempts to re-read the page.

Fixed warning: -Woverloaded-virtual for GCC 9 (Clang treats it differently)

Caused by c9cba597

Field::marked_for_read() and Field::marked_for_write_or_computed()
are being called from plugin/type_inet/sql_type_inet.cc ever since
commit 6ea5c2b5
and thus they cannot be declared inline any more.

Fixed warnings: -Woverloaded-virtual, -Winconsistent-missing-override

Caused by c9cba597

/mariadb/10.3/wsrep/wsrep_gtid.c:45:26: error: conversion from 'long int' to 'unsigned int' may change value [-Werror=conversion]
   45 |                 offset = endptr - str;

While `handler::next_insert_id` is restored on duplicate key errors
`part_share->next_auto_inc_val` is not restored which causes
discrepancy.

TABLE::mark_columns_needed_for_update(): use_all_columns() assigns
pointer of all_set into read_set and write_set, but this is not good
since all_set is changed later by
TABLE::mark_columns_used_by_index_no_reset().

Do column_bitmaps_signal() whenever we change read_set/write_set.

MDEV-16210 FK constraints on versioned tables use historical rows, which may cause constraint violation

Constraint check is done on secondary index update.
F.ex. DELETE does row_upd_sec_index_entry() and checks constraints in
row_upd_check_references_constraints(). UPDATE is optimized for the
case when order is not changed (node->cmpl_info & UPD_NODE_NO_ORD_CHANGE)
and doesn't do row_upd_sec_index_entry(), so it doesn't check constraints.

Since for versioned DELETE we do UPDATE actually, but expect behaviour
of DELETE in terms of constraints, we should deny this optimization to
get constraints checked.

Fix wrong referenced table check when versioned DELETE inserts history in parent
table. Set check_ref to false in this case.

Removed unused dup_chk_only argument for row_ins_sec_index_entry() and
added check_ref argument.

MDEV-18057 fix was superseded by this fix and reverted.

foreign.test:
All key_type combinations: pk, unique, sec(ondary).

Preparation for MDEV-16210:

replace.test:
key_type combinations: PK and UNIQUE.

foreign.test:
Preparation for key_type combinations.

Other fixes:

* Merged versioning.update2 into versioning.update;
* Removed test2 database and done individual drop instead.

Conflicts:
	sql/sql_yacc.yy

The function was always called with lsn=LSN_MAX.
Remove that redundant parameter.

Spotted by Thirunarayanan Balathandayuthapani.

The field buf_page_t::encrypted was added in MDEV-8588.
It was made mostly redundant in MDEV-12699. Remove the field.

calc_field_event_length(): For type=MYSQL_TYPE_BLOB and meta==0,
return 0 instead of *ptr+1. This was noted by -Wimplicit-fallthrough.

Add auto increment offset save and restore as node(s) are restarted.

Fix wait_condition and use repeatable read with wsrep_sync at the
end.

(fix and explanation came with MDEV-20753 (duplicate of this bug))