main.partition_explicit_prune

Internal transactions may not have trx->mysql_thd.
But at the same time, trx->duplicates should only hold if
REPLACE or INSERT...ON DUPLICATE KEY UPDATE was executed from SQL.

The flag feels misplaced. A more appropriate place for it would
be row_prebuilt_t or similar.

After validating the post-encryption checksum on an encrypted page,
Mariabackup should decrypt the page and validate the pre-encryption
checksum as well. This should reduce the probability of accepting
invalid pages as valid ones.

This is a backport and refactoring of a patch that was
originally written by Thirunarayanan Balathandayuthapani
for the 10.2 branch.

Innochecksum was being built as if WITH_INNODB_BUG_ENDIAN_CRC32:BOOL=OFF
had been specified.

Also, clean up tests:

innodb.innochecksum: Useless; superceded by innodb_zip.innochecksum.
innodb.innodb_zip_innochecksum: Remove; duplicated innodb_zip.innochecksum.
innodb.innodb_zip_innochecksum2: Remove; duplicated innodb_zip.innochecksum_2.
innodb.innodb_zip_innochecksum3: Remove; duplicated innodb_zip.innochecksum_3.

No test case was added. I tested manually by adding debug instrumentation
to both innochecksum and buf_page_is_checksum_valid_crc32() to make
innochecksum write the buggy crc32, and to get warnings for falling back
to the buggy checksum. Automating this would require that tests be
adjusted depending on the build options.

Allow array binding for DELETE, test it.

This is basically re-applying 8fe34dd4.
Assertions about ROW_FORMAT not changing during instant ALTER TABLE can
fail if the MariaDB and InnoDB data dictionaries get out of sync,
for example if innodb_default_row_format is used instead of specifying
ROW_FORMAT in the CREATE TABLE statement. In this case, ALTER_OPTIONS
would not necessarily be set. We would create the ctx->instant_table
with a ROW_FORMAT based on altered_table. When applying it to
the ctx->old_table, we will preserve the original ROW_FORMAT.

The test innodb.instant_alter_crash is exercising crash recovery
for instant ALTER operations. Unfortunately, for some reason the
redo log was not being flushed as expected (to be analyzed in MDEV-18009)
and this error was not detected earlier.

btr_cur_trim_alter_metadata(): New function for the special case of
rolling back an ALTER TABLE operation such that the table will remain
in the MDEV-15562 format (not rolling back to the MDEV-11369 format).
In this case, we must restore the old number of columns from the
old metadata BLOB page.

dict_table_t::init_instant(): Initialize instant->non_pk_col_map[].
Refactored from dict_table_t::instant_column().
Also, assert that the n_nullable for the clustered index is initialized
correctly.

dict_col_t::same_format(): Check if two columns have the same
data type and compatible length.

Allow instant changes of columns in ROW_FORMAT=REDUNDANT
from NOT NULL to NULL.

Later, this may be implemented for ROW_FORMAT=COMPACT or DYNAMIC,
but in that case any indexes on the table must be rebuilt.

dict_table_t::prepare_instant(): Add some debug assertions,
and relax a debug assertion so that the number of fields is
allowed not to change.

dict_index_t::instant_add_field(): Relax a debug assertion,
allowing a column to change from NOT NULL to NULL.

dict_table_t::instant_column(): Add debug assertions.

instant_alter_column_possible(): Allow ALTER_COLUMN_NULLABLE
when applicable.

innodb_insert_sys_columns(): Add the parameter bool update=false
to run UPDATE instead of INSERT.

innobase_instant_add_col(): Remove; let the only caller invoke
innodb_insert_sys_columns() directly.

innobase_instant_try(): Update the SYS_COLUMNS record if the
column is changed. Only convert the table to the instant ALTER TABLE
format if necessary. For ALTER_COLUMN_NULLABLE in ROW_FORMAT=REDUNDANT,
there is no data format change.

Remove the bug-compatible crc32 algorithm variant that was added
to allow an upgrade from data files from big-endian systems where
innodb_checksum_algorithm=crc32 was used on MySQL 5.6
or MariaDB 10.0 or 10.1.

Affected users should be able to recompute page checksums using
innochecksum.

With innodb_default_row_format=redundant, InnoDB would crash when
using table options that are incompatible with ROW_FORMAT=REDUNDANT.

create_table_info_t::m_default_row_format: Cache the value of
innodb_default_row_format.

create_table_info_t::check_table_options(): Validate ROW_TYPE_DEFAULT
with m_default_row_format.

create_table_info_t::innobase_table_flags(): Use the
cached m_default_row_format.

create_table_info_t: Never read m_form->s->row_type.
Use m_create_info->row_type instead.

dict_tf_set(): Never set invalid flags for ROW_FORMAT=REDUNDANT.

ha_innobase::truncate(): Set info.row_type based on the ROW_FORMAT
of the current table.

In MySQL 5.7, it was noticed that files are not portable between
big-endian and little-endian processor architectures
(such as SPARC and x86), because the original implementation of
innodb_checksum_algorithm=crc32 was not byte order agnostic.

A byte order agnostic implementation of innodb_checksum_algorithm=crc32
was only added to MySQL 5.7, not backported to 5.6. Consequently,
MariaDB Server versions 10.0 and 10.1 only contain the CRC-32C
implementation that works incorrectly on big-endian architectures,
and MariaDB Server 10.2.2 got the byte-order agnostic CRC-32C
implementation from MySQL 5.7.

MySQL 5.7 introduced a "legacy crc32" variant that is functionally
equivalent to the big-endian version of the original crc32 implementation.
Thanks to this variant, old data files can be transferred from big-endian
systems to newer versions.

Introducing new variants of checksum algorithms (without introducing
new names for them, or something on the pages themselves to identify
the algorithm) generally is a bad idea, because each checksum algorithm
is like a lottery ticket. The more algorithms you try, the more likely
it will be for the checksum to match on a corrupted page.

So, essentially MySQL 5.7 weakened innodb_checksum_algorithm=crc32,
and MariaDB 10.2.2 inherited this weakening.

We introduce a build option that together with MDEV-17957
makes innodb_checksum_algorithm=strict_crc32 strict again
by only allowing one variant of the checksum to match.

WITH_INNODB_BUG_ENDIAN_CRC32: A new cmake option for enabling the
bug-compatible "legacy crc32" checksum. This is only enabled on
big-endian systems by default, to facilitate an upgrade from
MariaDB 10.0 or 10.1. Checked by #ifdef INNODB_BUG_ENDIAN_CRC32.

ut_crc32_byte_by_byte: Remove (unused function).

legacy_big_endian_checksum: Remove. This variable seems to have
unnecessarily complicated the logic. When the weakening is enabled,
we must always fall back to the buggy checksum.

buf_page_check_crc32(): A helper function to compute one or
two CRC-32C variants.

Also, apply the MDEV-17957 changes to encrypted page checksums,
and remove error message output from the checksum function,
because these messages would be useless noise when mariabackup
is retrying reads of corrupted-looking pages, and not that
useful during normal server operation either.

The error messages in fil_space_verify_crypt_checksum()
should be refactored separately.

This is a backport of a part of
commit 18455ec3
from 10.1.

Problem:

  Innodb_checksum_algorithm checks for all checksum algorithm to
validate the page checksum even though the algorithm is specified as
strict_crc32, strict_innodb, strict_none.

Fix:

   Remove the checks for all checksum algorithm to validate the page
checksum if the algo is specified as strict_* values.

The Microsoft compiler only allows function attributes before the
function signature, not after it.

1. don't run full mysql_upgrade on every server restart,
   use --version-check to do it only once

2. fix syslog tag name in the postinst script, don't pretend
   mysqld_safe generated all these messages. Auto-detect the version
   to simplify maintenance

ha_innobase::truncate(): Because CREATE TEMPORARY TABLE
allows invalid table options when innodb_file_per_table=1,
do allow them also in TRUNCATE for temporary tables.