Cover innodb.table_flags with the new innodb_page_size.combinations
32k and 64k.

dict_sys_tables_type_validate(): Remove an assertion that made a
check in the function redundant. Remove the excessive output to
the error log, as the invalid SYS_TABLES.TYPE value is already being
output.

Add a missing #include "sync0types.h" that was removed in MDEV-12674.

Add a test case for corrupting SYS_TABLES.TYPE,
and for ROW_FORMAT=REDUNDANT, the unused field SYS_TABLES.MIX_LEN
that must be ignored (InnoDB before MySQL 5.5 wrote uninitialized
garbage to this column).

MariaDB 10.0 appears to validate the SYS_TABLES.TYPE properly.
This is a test-only change.

in innodb_read_only mode.

The reason for the hang is that there was no notification received about
completed read io. File handles are bound to completion_port, and there
were no background "write" threads that would be waiting on completion_port,
only 2 "read" threads waiting on read_completion_port were active.

The fix is to use a single IO completion port for all IOs, if
innodb_read_only is set.

fil_crypt_read_crypt_data(): Remove an unnecessary
acquisition of fil_system->mutex. Remove a duplicated condition
from the callers.

When the server is started in innodb_read_only mode, there cannot be
any writes to persistent InnoDB/XtraDB files. Just like the creation
of buf_flush_page_cleaner_thread is skipped in this case, also
the creation of the XtraDB-specific buf_flush_lru_manager_thread
should be skipped.

When a slow shutdown is performed soon after spawning some work for
background threads that can create or commit transactions, it is possible
that new transactions are started or committed after the purge has finished.
This is violating the specification of innodb_fast_shutdown=0, namely that
the purge must be completed. (None of the history of the recent transactions
would be purged.)

Also, it is possible that the purge threads would exit in slow shutdown
while there exist active transactions, such as recovered incomplete
transactions that are being rolled back. Thus, the slow shutdown could
fail to purge some undo log that becomes purgeable after the transaction
commit or rollback.

srv_undo_sources: A flag that indicates if undo log can be generated
or the persistent, whether by background threads or by user SQL.
Even when this flag is clear, active transactions that already exist
in the system may be committed or rolled back.

innodb_shutdown(): Renamed from innobase_shutdown_for_mysql().
Do not return an error code; the operation never fails.
Clear the srv_undo_sources flag, and also ensure that the background
DROP TABLE queue is empty.

srv_purge_should_exit(): Do not allow the purge to exit if
srv_undo_sources are active or the background DROP TABLE queue is not
empty, or in slow shutdown, if any active transactions exist
(and are being rolled back).

srv_purge_coordinator_thread(): Remove some previous workarounds
for this bug.

innobase_start_or_create_for_mysql(): Set buf_page_cleaner_is_active
and srv_dict_stats_thread_active directly. Set srv_undo_sources before
starting the purge subsystem, to prevent immediate shutdown of the purge.
Create dict_stats_thread and fts_optimize_thread immediately
after setting srv_undo_sources, so that shutdown can use this flag to
determine if these subsystems were started.

dict_stats_shutdown(): Shut down dict_stats_thread. Backported from 10.2.

srv_shutdown_table_bg_threads(): Remove (unused).

Problem appears to be that the function fsp_flags_try_adjust()
is being unconditionally invoked on every .ibd file on startup.
Based on performance investigation also the top function
fsp_header_get_crypt_offset() needs to addressed.

Ported implementation of fsp_header_get_encryption_offset()
function from 10.2 to fsp_header_get_crypt_offset().

Introduced a new function fil_crypt_read_crypt_data()
to read page 0 if it is not yet read.

fil_crypt_find_space_to_rotate(): Now that page 0 for every .ibd
file is not read on startup we need to check has page 0 read
from space that we investigate for key rotation, if it is not read
we read it.

fil_space_crypt_get_status(): Now that page 0 for every .ibd
file is not read on startup here also we need to read page 0
if it is not yet read it. This is needed
as tests use IS query to wait until background encryption
or decryption has finished and this function is used to
produce results.

fil_crypt_thread(): Add is_stopping condition for tablespace
so that we do not rotate pages if usage of tablespace should
be stopped. This was needed for failure seen on regression
testing.

fil_space_create: Remove page_0_crypt_read and extra
unnecessary info output.

fil_open_single_table_tablespace(): We call fsp_flags_try_adjust
only when when no errors has happened and server was not started
on read only mode and tablespace validation was requested or
flags contain other table options except low order bits to
FSP_FLAGS_POS_PAGE_SSIZE position.

fil_space_t::page_0_crypt_read removed.

Added test case innodb-first-page-read to test startup when
encryption is on and when encryption is off to check that not
for all tables page 0 is read on startup.

The doublewrite buffer pages must fit in the first InnoDB system
tablespace data file. The checks that were added in the initial patch
(commit 112b21da)
were at too high level and did not cover all cases.

innodb.log_data_file_size: Test all innodb_page_size combinations.

fsp_header_init(): Never return an error. Move the change buffer creation
to the only caller that needs to do it.

btr_create(): Clean up the logic. Remove the error log messages.

buf_dblwr_create(): Try to return an error on non-fatal failure.
Check that the first data file is big enough for creating the
doublewrite buffers.

buf_dblwr_process(): Check if the doublewrite buffer is available.
Display the message only if it is available.

recv_recovery_from_checkpoint_start_func(): Remove a redundant message
about FIL_PAGE_FILE_FLUSH_LSN mismatch when crash recovery has already
been initiated.

fil_report_invalid_page_access(): Simplify the message.

fseg_create_general(): Do not emit messages to the error log.

innobase_init(): Revert the changes.

trx_rseg_create(): Refactor (no functional change).

Removed this output.

commit 1af8bf39 added unnecessary
calls to fil_write_flushed_lsn() during redo log resizing at
InnoDB server startup.

Because fil_write_flushed_lsn() is neither redo-logged nor doublewrite
buffered, the call is risky and should be avoided, because if the
server killed during the write call, the whole InnoDB instance can
become inaccessible (corrupted page 0 in the system tablespace).

In the best case, this call might prevent a diagnostic message from
being emitted to the error log on the next startup.

MDEV-13032: fix galera_new_cluster to be POSIX

Signed-off-by: Daniel Black <daniel.black@au.ibm.com>

Rewrite the test encryption.innodb-checksum-algorithm not to
require any restarts or re-bootstrapping, and to cover all
innodb_page_size combinations.

Test innodb.101_compatibility with all innodb_page_size combinations.

innodb_page_size_small: A new set of combinations, for
innodb_page_size up to 16k. In MariaDB 10.0, this does not
make a difference, but in 10.1 and later, innodb_page_size
would cover 32k and 64k, for which ROW_FORMAT=COMPRESSED
is not available.

Enable these combinations in a few InnoDB tests.

srv_master_thread(): Pass the correct parameter to srv_shutdown().
This bug was introduced in MDEV-12052, and it affects the MariaDB 10.1.24
release.

Problem:- In the case of multisource replication/named slave
when we run "FLUSH LOGS" , it does not flush logs.

Solution:- A new function Master_info_index->flush_all_relay_logs()
is created which will rotate relay logs for all named slave.
This will be called in reload_acl_and_cache function when
connection_name.length == 0

Problem was that all doublewrite buffer pages must fit to first
system datafile.

Ported commit 27a34df7882b1f8ed283f22bf83e8bfc523cbfde
Author: Shaohua Wang <shaohua.wang@oracle.com>
Date:   Wed Aug 12 15:55:19 2015 +0800

    BUG#21551464 - SEGFAULT WHILE INITIALIZING DATABASE WHEN
    INNODB_DATA_FILE SIZE IS SMALL

To 10.1 (with extended error printout).

btr_create(): If ibuf header page allocation fails report error and
return FIL_NULL. Similarly if root page allocation fails return a error.

dict_build_table_def_step: If fsp_header_init fails return
error code.

fsp_header_init: returns true if header initialization succeeds
and false if not.

fseg_create_general: report error if segment or page allocation fails.

innobase_init: If first datafile is smaller than 3M and could not
contain all doublewrite buffer pages report error and fail to
initialize InnoDB plugin.

row_truncate_table_for_mysql: report error if fsp header init
fails.

srv_init_abort: New function to report database initialization errors.

srv_undo_tablespaces_init, innobase_start_or_create_for_mysql: If
database initialization fails report error and abort.

trx_rseg_create: If segment header creation fails return.

MDEV-12114: install_db shows corruption for rest encryption and innodb_checksum_algorithm=strict_none

Problem was that checksum check resulted false positives that page is
both not encrypted and encryted when checksum_algorithm was
strict_none.

Encrypton checksum will use only crc32 regardless of setting.

buf_zip_decompress: If compression fails report a error message
containing the space name if available (not available during import).
And note if space could be encrypted.

buf_page_get_gen: Do not assert if decompression fails,
instead unfix the page and return NULL to upper layer.

fil_crypt_calculate_checksum: Use only crc32 method.

fil_space_verify_crypt_checksum: Here we need to check
crc32, innodb and none method for old datafiles.

fil_space_release_for_io: Allow null space.

encryption.innodb-compressed-blob is now run with crc32 and none
combinations.

Note that with none and strict_none method there is not really
a way to detect page corruptions and page corruptions after
decrypting the page with incorrect key.

New test innodb-checksum-algorithm to test different checksum
algorithms with encrypted, row compressed and page compressed
tables.

Problem was that FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION field that for
encrypted pages even in system datafiles should contain key_version
except very first page (0:0) is after encryption overwritten with
flush lsn.

Ported WL#7990 Repurpose FIL_PAGE_FLUSH_LSN to 10.1
The field FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION is consulted during
InnoDB startup.

At startup, InnoDB reads the FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION
from the first page of each file in the InnoDB system tablespace.
If there are multiple files, the minimum and maximum LSN can differ.
These numbers are passed to InnoDB startup.

Having the number in other files than the first file of the InnoDB
system tablespace is not providing much additional value. It is
conflicting with other use of the field, such as on InnoDB R-tree
index pages and encryption key_version.

This worklog will stop writing FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION to
other files than the first file of the InnoDB system tablespace
(page number 0:0) when system tablespace is encrypted. If tablespace
is not encrypted we continue writing FIL_PAGE_FLUSH_LSN_OR_KEY_VERSION
to all first pages of system tablespace to avoid unnecessary
warnings on downgrade.

open_or_create_data_files(): pass only one flushed_lsn parameter

xb_load_tablespaces(): pass only one flushed_lsn parameter.

buf_page_create(): Improve comment about where
FIL_PAGE_FIL_FLUSH_LSN_OR_KEY_VERSION is set.

fil_write_flushed_lsn(): A new function, merged from
fil_write_lsn_and_arch_no_to_file() and
fil_write_flushed_lsn_to_data_files().
Only write to the first page of the system tablespace (page 0:0)
if tablespace is encrypted, or write all first pages of system
tablespace and invoke fil_flush_file_spaces(FIL_TYPE_TABLESPACE)
afterwards.

fil_read_first_page(): read flush_lsn and crypt_data only from
first datafile.

fil_open_single_table_tablespace(): Remove output of LSN, because it
was only valid for the system tablespace and the undo tablespaces, not
user tablespaces.

fil_validate_single_table_tablespace(): Remove output of LSN.

checkpoint_now_set(): Use fil_write_flushed_lsn and output
a error if operation fails.

Remove lsn variable from fsp_open_info.

recv_recovery_from_checkpoint_start(): Remove unnecessary second
flush_lsn parameter.

log_empty_and_mark_files_at_shutdown(): Use fil_writte_flushed_lsn
and output error if it fails.

open_or_create_data_files(): Pass only one flushed_lsn variable.

it's unused now.

always use full m_SubStrVec length in pcre_exec, we don't know
how many subexpressions user's regexp will have

buf_page_get_gen(): Remove the error log messages about
page flushing and eviction when
innodb_change_buffering_debug=1 is in effect.

This corrects the patch for mdev-10006.
The current code supports only those semi-join nests that are placed at
the join top level. So such nests cannot depend on other tables or nests.

was corrupted, server crashes in opt_sum_query

Extended keys feature disabled if the length of extended key is longer than MAX_KEY_LEN

performance-schema options in option files should be loose. The tests
will be skipped after have_perfschema.inc check, but if options aren't
loose, the failure happens on server startup, before the check

MDEV-12926 encryption.innodb_onlinealter_encryption, encryption.innodb-bad-key-change failed in buildbot with valgrind

row_merge_write(): Pass the correct (possibly encrypted) buffer
to os_file_write_int_fd().

This bug was introduced in commit 65e1399e
which included a commit to merge changes from MySQL 5.6.36 to
MariaDB Server 10.0.

btr_defragment_thread(): Create the thread in the same place as other
threads. Do not invoke btr_defragment_shutdown(), because
row_drop_tables_for_mysql_in_background() in the master thread can still
keep invoking btr_defragment_remove_table().

logs_empty_and_mark_files_at_shutdown(): Wait for btr_defragment_thread()
to exit.

innobase_start_or_create_for_mysql(), innobase_shutdown_for_mysql():
Skip encryption and scrubbing in innodb_read_only_mode.

srv_export_innodb_status(): Do not export encryption or scrubbing
statistics in innodb_read_only mode, because the threads will not
be running.