MDEV-18129 Backup fails for encrypted tables: mariabackup: Database page corruption detected at page 1

If an encrypted table is created during backup, then
mariabackup --backup could wrongly fail.

This caused a failure of the test mariabackup.huge_lsn once on buildbot.

This is due to the way how InnoDB creates .ibd files. It would first
write a dummy page 0 with no encryption information. Due to this,
xb_fil_cur_open() could wrongly interpret that the table is not encrypted.
Subsequently, page_is_corrupted() would compare the computed page
checksum to the wrong checksum. (There are both "before" and "after"
checksums for encrypted pages.)

To work around this problem, we introduce a Boolean option
--backup-encrypted that is enabled by default. With this option,
Mariabackup will assume that a nonzero key_version implies that the
page is encrypted. We need this option in order to be able to copy
encrypted tables from MariaDB 10.1 or 10.2, because unencrypted pages
that were originally created before MySQL 5.1.48 could contain nonzero
garbage in the fields that were repurposed for encryption.

Later, MDEV-18128 would clean up the way how .ibd files are created,
to remove the need for this option.

page_is_corrupted(): Add missing const qualifiers, and do not check
space->crypt_data unless --skip-backup-encrypted has been specified.

xb_fil_cur_read(): After a failed page read, output a page dump.

This is a regression caused by
commit 8c43f963
that was part of the MDEV-12112 fixes.

page_is_corrupted(): Never interpret page_no=0 as encrypted.

This is a regression after MDEV-13671.

The bug is related to key part prefix lengths wich are stored in SYS_FIELDS.
Storage format is not obvious and was handled incorrectly which led to data
dictionary corruption.

SYS_FIELDS.POS actually contains prefix length too in case if any key part
has prefix length.

innobase_rename_column_try(): fixed prefixes handling

Tests for prefixed indexes added too.

Closes #1063

would not hide more interesting information, like invalid memory accesses.


some "leaks" are expected
- partly this is due to weird options parsing, that runs twice, and
does not free memory after the first run.
- also we do not mind to exit()  whenever it makes sense, without full
cleanup.

MDEV-17470 Orphan temporary files after interrupted ALTER cause InnoDB: Operating system error number 17 and eventual fatal error 71

Orphan #sql* tables may remain after ALTER TABLE
was interrupted by timeout or KILL or client disconnect.

This is a regression caused by MDEV-16515.

Similar to temporary tables (MDEV-16647), we had better ignore the
KILL when dropping the original table in the final part of ALTER TABLE.

Closes #1020

This fixes a regression that was introduced in MySQL 5.6.6
in an error handling code path, in the following change:

commit 024f363d6b5f09b20d1bba411af55be95c7398d3
Author: kevin.lewis@oracle.com <>
Date:   Fri Jun 15 09:01:42 2012 -0500

    Bug #14169459 INNODB; DROP TABLE DOES NOT DELETE THE IBD FILE
    FOR A TEMPORARY TABLE.

Docs grammar fixed

update engines/ suites

followup for c4ab352b

it's listed in MYSQL_ADD_PLUGIN() declaration,
so it must always exist, jemalloc or not.

btr_node_ptr_max_size(): Do not reserve extra space for indexed VARCHAR(0)
columns.

os_event::is_set(): protect os_event::m_set with os_event::mutex

When using a wrong key, all encrypted pages will look corrupted,
and occasionally the test may access other pages than
the clustered index root page.

Also fixes:
MDEV-17741 Assertion `thd->Item_change_list::is_empty()' failed in mysql_parse after unsuccessful PS

The problem was introduced by:
  commit f033fbd9
  Changed the test case for MDEV-15571

It was later fixed, but in 10.3 only:

  commit ce2cf855
  MDEV-16043 Assertion thd->Item_change_list::is_empty() failed in mysql_parse
  upon SELECT from a view reading from a versioned table

This patch is a backport of ce2cf855 to 10.2

- Refactor code to isolate page validation in page_is_corrupted() function.

- Introduce --extended-validation parameter(default OFF) for mariabackup
--backup to enable decryption of encrypted uncompressed pages during
backup.

- mariabackup would still always check checksum on encrypted data,
it is needed to detect  partially written pages.

In MDEV-13103, I made a mistake in the error handling of
page_compressed=1 decryption when the default
innodb_compression_algorithm=zlib is used.
Due to this mistake, with certain versions of zlib,
MariaDB would fail to detect a corrupted page.

The problem was uncovered by the following tests:
mariabackup.unencrypted_page_compressed
mariabackup.encrypted_page_compressed