MDEV-17349 Assertion `!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))' failed on concurrent SELECT and DELETE after RENAME from table with index on virtual column

Race condition. field->flags were copied from s->field->flags during
field->clone(), early in open_table_from_share(). But s->field->flags
were getting their PART_INDIRECT_KEY_FLAG bit much later in
TABLE::mark_columns_used_by_virtual_fields() and only once per share.

If two threads were executing the code between field->clone()
and mark_columns_used_by_virtual_fields() at the same time, only
one would get PART_INDIRECT_KEY_FLAG bits in field[].

thd_rpl_stmt_based(): A new predicate to check if statement-based
replication is active. (This can also hold when replication is not
in use, but binlog is.)

que_thr_stop(), row_ins_duplicate_error_in_clust(),
row_ins_sec_index_entry_low(), row_ins(): On a duplicate key error,
only lock all index records when statement-based replication is in use.

Users expect window functions to produce a certain ordering of rows in
the final result set. Although the standard does not require this, we
already have the filesort result done for when we computed the window
function. If there is no ORDER BY attached to the query, just keep it
till the SELECT is completely evaluated and use that to print the
result.

Update test cases as many did not take care to guarantee a stable
result.

set both `password` and `authentication_string` columns in `mysql`.`user`
table for now.

Suppress the "password was ignored" warning if the password is
the same as the authentication string

The parameters bool sync=true, bool read_only=false of mtr_t::start()
were added in
https://github.com/mysql/mysql-server/commit/eca5b0fc17a5bd6d4833d35a0d08c8549dd3b5ec
(MySQL 5.7.3).

The parameter read_only was never used anywhere.
The parameter sync was only copied around, and would be returned
by the unused function mtr_t::is_async().

We do not need this dead code in MariaDB.

At the start of the function, we already dereferenced node,
so the checks for NULL are unnecessary and redundant.

Query INFORMATION_SCHEMA.INNODB_SYS_TABLES only once, and cache results.

As a small cleanup, remove  mdl_lock_con_mutex, the MDL handling
connection is never accessed by multiple threads at the same time.

row_drop_table_for_mysql(): Avoid accessing non-existing dictionary tables.

dict_create_or_check_foreign_constraint_tables(): Add debug instrumentation
for creating and dropping a table before the creation of any non-core
dictionary tables.

trx_purge_add_update_undo_to_history(): Adjust a debug assertion, so that
it will not fail due to the test instrumentation.

- Use the correct range bounds when doing a reverse-ordered range scan
  (this was already done for HA_READ_PREFIX_LAST_OR_PREV but not for
   HA_READ_BEFORE_KEY).

The ONLY_FULL_GROUP_BY mode states that for SELECT ... GROUP BY queries,
disallow SELECTing columns which are not referred to in the GROUP BY clause,
unless they are passed to an aggregate function like COUNT() or MAX().
This holds only for the GROUP BY clause of the query.
The code also checks this for the partition clause of the window function which is
incorrect.

page_id_t: remove m_fold member

various places: pass page_id_t by value instead of by reference

Update the InnoDB version number to 5.7.24.

row_build_index_entry_low(): ext does not contain virtual columns.

row_upd_store_v_row(): Copy virtual column values

This is based on the following fix in MySQL 5.7.24:

commit 4ec2158bec73f1582501c4b3e3de250fed9edc9a
Author: Sachin Agarwal <sachin.z.agarwal@oracle.com>
Date:   Fri Aug 24 14:44:13 2018 +0530

    Bug #27968952 INNODB CRASH/CORRUPTION WITH TEXT PREFIX INDEXES

    Problem:
    There are two problems:
      1. If there is one secondary index on extenally
    stored column and another seconday index on virtual column (whose
    base column is not externally stored). then while updating seconday
    index on vitrual column, virtual column data is replaced by
    externally stoared column.
      2. In row update operation, node->row contains
    shallow copy of virtual data fields. While building an update vector
    containing all the fields to be modified, compute virtual column.
    which may causes change in virtual data fields in node->row.

    In both the above cases, while updating seconday index on virtual
    column, couldn't find the row and hit an explicite assert inside
    ROW_NOT_FOUND.

    Fix:
    1. Added check if column is virtual then its ext flag should be ZERO
    and virtual column data will not be replaced by offset column data.
    2. Deep copy of virtual data fields for node->row.

    RB: #20382
    Reviewed by : Jimmy.Yang@oracle.com

dict_foreign_qualify_index(): Reject both FULLTEXT and SPATIAL
indexes. Remove these checks from the callers.

It is unclear whether this bug affected MariaDB Server.
FOREIGN KEY constraints on geometry column types could have
been rejected due to column type restrictions.

rtr_pcur_getnext_from_path(): Remove a bogus condition.
The predicate lock should be acquired also on no match,
to ensure that the locking read will be repeatable.

This is based on the following fix in MySQL 5.7.24:

commit 365111c590082984dbae42e1d1da28ac3f7fb5bd
Author: Jimmy Yang <jimmy.yang@oracle.com>
Date:   Wed Jun 6 16:23:00 2018 -0700

    Fix Bug 27577612 - CONCURRENT SERIALIZABLE TRANSACTIONS CAN INSERT INTO
    AN AREA SELECTED FOR UPDATE
    Backport fix to mysql-5.7
Reviewed-by: Allen Lai <allen.lai@oracle.com>

No test case is added, because the MySQL 5.7 test case would pass
even when the fix is not present. We would need a test case that
only causes a locking conflict on the spatial index.

row_ins_check_foreign_constraint(): Do not overwrite hard errors
with the soft error DB_LOCK_WAIT. This prevents an infinite
wait loop when DB_INTERRUPTED was returned. For DB_LOCK_WAIT,
row_insert_for_mysql() would keep invoking row_ins_step() and the
transaction would remain active until the server shutdown is initiated.

This reverts commit 2d4075e1
where the debug assertion was added. There seems to be a potential
problem in the purge of indexes that depend on virtual columns.

Ultimately, we should change the InnoDB undo log format so that
all actual secondary index keys are stored there, also for
virtual or spatial indexes. In that way, purge and rollback would
be more straightforward.

On Windows, a mismatch between TIMETPF ("%ld") and time_t would be reported.
Use "%ld" and long, like the code used to be.

The setting innodb_safe_truncate=ON reduces compatibility with older
versions of MariaDB and backup tools in two ways.

First, we will be writing TRX_UNDO_RENAME_TABLE records, which older
versions do not know about. These records could be misinterpreted if
a DDL transaction was recovered and would be rolled back.
Such rollback is only possible if the server was killed while
an incomplete DDL transaction was persisted. On transaction completion,
the insert_undo log pages would only be repurposed for new undo log
allocations, and their contents would not matter. So, older versions
will not have a problem with innodb_safe_truncate=ON if the server was
shut down cleanly.

Second, to prevent such recovery failure, innodb_safe_truncate=ON will
cause a modification of the redo log format identifier, which will
prevent older versions from starting up after a crash. MariaDB Server
versions older than 10.2.13 will refuse to start up altogether, even
after clean shutdown.

A server restart with innodb_safe_truncate=OFF will restore compatibility
with older server and backup versions.

Added only test case because the bug was fixed by the patch for mdev-16992.

Opened up MYSQL_TYPE _DATETIME{,2} <-> _NEWDATE conversions for replication.

row_build_index_entry_low(): Assert that when the value of a
virtual column is not available, this can only happen when
the index creation was completed but not committed yet.

This change is not fixing any bug, making a debug assertion
stricter, so that bugs can be caught in the future.

Ultimately, we should change the InnoDB undo log format so that
all actual secondary index keys are stored there, also for
virtual or spatial indexes. In that way, purge and rollback would
be more straightforward.

When we have a query which has implicit_grouping then we are sure that we would end up with only one
row so there is no point to do DISTINCT computation

The failure caused by MDEV-16387

           upon INSERT .. SELECT

The function Item *Item_direct_view_ref::derived_field_transformer_for_where()
erroneously did not strip off ref wrappers from references to materialized
derived tables / views. As a result the expressions that contained some
references of the type Item_direct_view_ref to columns of a materialized
derived table / view V were pushed into V incorrectly. This could cause
crashes for some INSERT ... SELECT statements.

A prepared backup from Mariabackup does not really need to contain any
redo log file, because all log will have been applied to the data files.

When the user copies a prepared backup to a data directory (overwriting
existing files), it could happen that the data directory already contained
redo log files from the past. mariabackup --copy-back) would delete the
old redo log files, but a user’s own copying script might not do that.
To prevent corruption caused by mixing an old redo log file with data
files from a backup, starting with MDEV-13311, Mariabackup would create
a zero-length ib_logfile0 that would prevent startup.

Actually, there is no need to prevent InnoDB from starting up when a
single zero-length file ib_logfile0 is present. Only if there exist
multiple data files of different lengths, then we should refuse to
start up due to inconsistency. A single zero-length ib_logfile0 should
be treated as if the log files were missing: create new log files
according to the configuration.

open_log_file(): Remove. There is no need to open the log files
at this point, because os_file_get_status() already determined
the size of the file.

innobase_start_or_create_for_mysql(): Move the creation of new
log files a little later, not when finding out that the first log
file does not exist, but after finding out that it does not exist
or it exists as a zero-length file.

Rename the 10.2-specific configuration option innodb_unsafe_truncate
to innodb_safe_truncate, and invert its value.

The default (for now) is innodb_safe_truncate=OFF, to avoid
disrupting users with an undo and redo log format change within
a Generally Available (GA) release series.

While MariaDB Server 10.2 is not really guaranteed to be compatible
with Percona XtraBackup 2.4 (for example, the MySQL 5.7 undo log format
change that could be present in XtraBackup, but was reverted from
MariaDB in MDEV-12289), we do not want to disrupt users who have
deployed xtrabackup and MariaDB Server 10.2 in their environments.

With this change, MariaDB 10.2 will continue to use the backup-unsafe
TRUNCATE TABLE code, so that neither the undo log nor the redo log
formats will change in an incompatible way.

Undo tablespace truncation will keep using the redo log only. Recovery
or backup with old code will fail to shrink the undo tablespace files,
but the contents will be recovered just fine.

In the MariaDB Server 10.2 series only, we introduce the configuration
parameter innodb_unsafe_truncate and make it ON by default. To allow
MariaDB Backup (mariabackup) to work properly with TRUNCATE TABLE
operations, use loose_innodb_unsafe_truncate=OFF.

MariaDB Server 10.3.10 and later releases will always use the
backup-safe TRUNCATE TABLE, and this parameter will not be
added there.

recv_recovery_rollback_active(): Skip row_mysql_drop_garbage_tables()
unless innodb_unsafe_truncate=OFF. It is too unsafe to drop orphan
tables if RENAME operations are not transactional within InnoDB.

LOG_HEADER_FORMAT_10_3: Replaces LOG_HEADER_FORMAT_CURRENT.

log_init(), log_group_file_header_flush(),
srv_prepare_to_delete_redo_log_files(),
innobase_start_or_create_for_mysql(): Choose the redo log format
and subformat based on the value of innodb_unsafe_truncate.

The test needs to be run with rocksdb_flush_log_at_trx_commit=1, otherwise
the changes do not survive a crash.

Add wait on second node.