MDEV-13807 mariabackup --apply-log-only does generate redo log by performing rollback and possibly other tasks

Skip rollback and other redo-log-generating tasks if
srv_apply_log_only is set.

Instead of assigning the debug variable recv_no_log_write = FALSE,
assign it to srv_apply_log_only, so that any unwanted writes are caught.

The problem was introduced by the patch for MDEV-7661,
which (in addition to the fix itself) included an attempt to make
CONVERT/CAST work in the same way with fields
(i.e. return NULL in strict mode if a non-convertable character found).
It appeared to be a bad idea and some users were affected by this
behavior change. Changing CONVERT/CAST not depend on sql_mode
(restoring pre-10.1.4 behavior).

…porary file

Fixed by removing writing key version to start of every block that
was encrypted. Instead we will use single key version from log_sys
crypt info.

After this MDEV also blocks writen to row log are encrypted and blocks
read from row log aren decrypted if encryption is configured for the
table.

innodb_status_variables[], struct srv_stats_t
	Added status variables for merge block and row log block
	encryption and decryption amounts.

Removed ROW_MERGE_RESERVE_SIZE define.

row_merge_fts_doc_tokenize
	Remove ROW_MERGE_RESERVE_SIZE

row_log_t
	Add index, crypt_tail, crypt_head to be used in case of
	encryption.

row_log_online_op, row_log_table_close_func
	Before writing a block encrypt it if encryption is enabled

row_log_table_apply_ops, row_log_apply_ops
	After reading a block decrypt it if encryption is enabled

row_log_allocate
	Allocate temporary buffers crypt_head and crypt_tail
	if needed.

row_log_free
	Free temporary buffers crypt_head and crypt_tail if they
	exist.

row_merge_encrypt_buf, row_merge_decrypt_buf
	Removed.

row_merge_buf_create, row_merge_buf_write
	Remove ROW_MERGE_RESERVE_SIZE

row_merge_build_indexes
	Allocate temporary buffer used in decryption and encryption
	if needed.

log_tmp_blocks_crypt, log_tmp_block_encrypt, log_temp_block_decrypt
	New functions used in block encryption and decryption

log_tmp_is_encrypted
	New function to check is encryption enabled.

Added test case innodb-rowlog to force creating a row log and
verify that operations are done using introduced status
variables.

recv_reset_logs(): Initialize the redo log buffer, so that no data
from the old redo log can be written to the new redo log.

This bug has very little impact before MariaDB 10.2. The
innodb_log_encrypt option that was introduced in MariaDB 10.1
increases the impact. If the redo log used to be encrypted, and
it is being resized and encryption disabled, then previously
encrypted data could end up being written to the new redo log
in clear text. This resulted in encryption.innodb_encrypt_log
test failures in MariaDB 10.2.

The SEARCH_ABORT logic was not working as intended
(tests were not being aborted).

In XtraDB, buf_block_get_frame() can return NULL, and this value
can be passed to buf_page_print(). Do not declare the parameter
as nonnull.

The bug only affects ROW_FORMAT=DYNAMIC tables.
It is reproducible already with 2 records in the table.

Keep testing with ROW_FORMAT=REDUNDANT just in case.

The bug only affected innodb_page_size=64k. There is no point
running the test with smaller page sizes.

buf_page_print(): Remove the parameter 'flags',
and when a server abort is intended, perform that in the caller.

In this way, page corruption reports due to different reasons
can be distinguished better.

This is non-functional code refactoring that does not fix any
page corruption issues. The change is only made to avoid falsely
grouping together unrelated causes of page corruption.

A new $MYSQLD_LAST_CMD evaluation was too late in case --manual-gdb.
Now it is done before the server restart type branches which is safe
and the args value has been fully computed by the new point of
evaluation.

save thd->select_number between parsing and executions (in case it was not complete executed due to errors (for example epsent table))

This is a backport of the following:

MDEV-13009 10.1.24 does not compile on architectures without 64-bit atomics

Add a missing #include "sync0types.h" that was removed in MDEV-12674.

metadata_lock_info_duration[]: Remove the unused variable.

Add some comments /* fall through */ to silence -Wimplicit-fallthrough

char* parameter is expected by the message ER_KEY_COLUMN_DOES_NOT_EXITS,
thus pass char*, rather than LEX_STRING.

ERROR_FILE_SYSTEM_LIMITATION was seen by support when backing up large
file. However mariabackup error message was not very helpful,
since it mapped the error to generic catch-all EINVAL.

With the patch, ERROR_FILE_SYSTEM_LIMITATION will be mapped to more
appropriate EFBIG. Also add mapping from ERROR_NO_SYSTEM_RESOURCES
to ENOMEM.

There is a race condition in InnoDB startup. A number of
fil_crypt_thread are created by fil_crypt_threads_init(). These threads
may call btr_scrub_complete_space() before btr_scrub_init() was called.
Those too early calls would be accessing an uninitialized scrub_stat_mutex.

innobase_start_or_create_for_mysql(): Invoke btr_scrub_init() before
fil_crypt_threads_init().

fil_crypt_complete_rotate_space(): Only invoke btr_scrub_complete_space()
if scrubbing is enabled. There is no need to update the statistics if
it is not enabled.

After MDEV-13583: Improvements for MTR rebootstrap introduced in
MDEV-12042 bootsrap correctly creates mysql/innodb_table_stats
and mysql/innodb_index_stats InnoDB tables before innodb_encryption
test starts. These tables are also encrypted or decrypted, thus
we need to wait also these tables (if not we could randomly
get different results as system tablespace and these tables
are encrypted or decrypted in parallel).

Problem was incorrect definition of wsrep_recovery,
trx_sys_update_wsrep_checkpoint and
trx_sys_read_wsrep_checkpoint functions causing
innodb_plugin not to load as there was undefined symbols.

wsrep_drop_table_query(): Remove the definition of this ununsed function.

row_upd_sec_index_entry(), row_upd_clust_rec_by_insert():
Evaluate the simplest conditions first. The merge could have slightly
hurt performance by causing extra calls to wsrep_on().

This is not affecting correctness; delete NULL is a valid operation.

recv_find_max_checkpoint(): Refer to MariaDB 10.2.2 instead of
MySQL 5.7.9. Do not hint that a binary downgrade might be possible,
because there are many changes in InnoDB 5.7 that could make
downgrade impossible: a column appended to SYS_INDEXES, added
SYS_* tables, undo log format changes, and so on.

Import the MySQL 5.6 addition from innodb.create-index to a new debug-only
test, innodb.create-index-debug. The existing test innodb.create-index
also runs on a debug server.

FIXME: MDEV-13668 InnoDB unnecessarily rebuilds table

FIXME: MDEV-13671 InnoDB should use case-insensitive column name comparisons
like the rest of the server

FIXME: MDEV-13640 / Properly fix MDEV-9469 'Incorrect key file' on ALTER TABLE

FIXME: investigate result difference in innodb.innodb-alter-autoinc
and ensure that MariaDB does the right thing with auto_increment_increment
and auto_increment_offset, for both ALGORITHM=INPLACE and ALGORITHM=COPY
(Oracle MySQL behaviour differs between those two).

Import some ALTER TABLE test cases from MySQL 5.6 without modification.
The adjustments will be in a separate commit.

Fixes also MDEV-13488: InnoDB writes CRYPT_INFO even though
encryption is not enabled.

Problem was that we created encryption metadata (crypt_data) for
system tablespace even when no encryption was enabled and too early.
System tablespace can be encrypted only using key rotation.

Test innodb-key-rotation-disable, innodb_encryption, innodb_lotoftables
require adjustment because INFORMATION_SCHEMA INNODB_TABLESPACES_ENCRYPTION
contain row only if tablespace really has encryption metadata.

fil_crypt_set_thread_cnt: Send message to background encryption threads
if they exits when they are ready. This is required to find tablespaces
requiring key rotation if no other changes happen.

fil_crypt_find_space_to_rotate: Decrease the amount of time waiting
when nothing happens to better enable key rotation on startup.

fsp_header_init: Write encryption metadata to page 0 only if tablespace is
encrypted or encryption is disabled by table option.

i_s_dict_fill_tablespaces_encryption : Skip tablespaces that do not
contain encryption metadata. This is required to avoid too early
wait condition trigger in encrypted -> unencrypted state transfer.

open_or_create_data_files: Do not create encryption metadata
by default to system tablespace.

Assertions failed due to incorrect handling of the --tc-heuristic-recover
option when InnoDB is in read-only mode either due to innodb_read_only=1
or innodb_force_recovery>3. InnoDB failed to refuse a XA COMMIT or
XA ROLLBACK operation, and there were errors in the error handling in
the upper layer.

This was fixed by making InnoDB XA operations respect the
high_level_read_only flag. The InnoDB part of the fix and
parts of the test main.tc_heuristic_recover were provided
by Marko Mäkelä.

LOCK_log mutex lock/unlock had to be added to fix MDEV-13438.
The measure is confirmed by mysql sources as well.

For testing of the conflicting option combination, mysql-test-run is
made to export a new $MYSQLD_LAST_CMD. It holds the very last value
generated by mtr.mysqld_start().  Even though the options have been
also always stored in $mysqld->{'started_opts'} there were no access
to them beyond the automatic server restart by mtr through the expect
file interface.

Effectively therefore $MYSQLD_LAST_CMD represents a more general
interface to $mysqld->{'started_opts'} which can be used in wider
scopes including server launch with incompatible options.

Notice another existing method to restart the server with incompatible
options relying on $MYSQLD_CMD is is aware of $mysqld->{'started_opts'}
(the actual options that the server is launched by mtr). In order to use
this method they would have to be provided manually.

NOTE: When merging to 10.2, the file search_pattern_in_file++.inc
should be replaced with the pre-existing search_pattern_in_file.inc.

If this variable is set, skip actual AWS calls, and fake/mock
both generation and encryption of the keys.

The advantage of having a mock mode is that more aws_key_management tests
can be enabled on buildbot.

- make re-bootstrap run with all extra options, not only InnoDB ones
- re-use previously created bootstrap.sql
- add --console
- fix debian patch to keep it applicable

cases.

Problem is that page 0 and its possible enrryption information
is not read for undo tablespaces.

fil_crypt_get_latest_key_version(): Do not send event to
encryption threads if event does not yet exists. Seen
on regression testing.

fil_read_first_page: Add new parameter does page belong to
undo tablespace and if it does, we do not read FSP_HEADER.

srv_undo_tablespace_open : Read first page of the tablespace
to get crypt_data if it exists and pass it to fil_space_create.

Tested using innodb_encryption with combinations with
innodb-undo-tablespaces.