While the redo log format was changed in MariaDB 10.3.2 and 10.3.3
due to MDEV-12288 and MDEV-11369, it should be technically possible
to upgrade from a crashed MariaDB 10.2 instance.

On a related note, it should be possible for Mariabackup 10.3
to create a backup from a running MariaDB Server 10.2.

mlog_id_t: Put back the 10.2 specific redo log record types
MLOG_UNDO_INSERT, MLOG_UNDO_ERASE_END, MLOG_UNDO_INIT,
MLOG_UNDO_HDR_REUSE.

trx_undo_parse_add_undo_rec(): Parse or apply MLOG_UNDO_INSERT.

trx_undo_erase_page_end(): Apply MLOG_UNDO_ERASE_END.

trx_undo_parse_page_init(): Parse or apply MLOG_UNDO_INIT.

trx_undo_parse_page_header_reuse(): Parse or apply MLOG_UNDO_HDR_REUSE.

recv_log_recover_10_2(): Remove. Always parse the redo log from 10.2.

recv_find_max_checkpoint(), recv_recovery_from_checkpoint_start():
Always parse the redo log from MariaDB 10.2.

recv_parse_or_apply_log_rec_body(): Parse or apply
MLOG_UNDO_INSERT, MLOG_UNDO_ERASE_END, MLOG_UNDO_INIT.

srv_prepare_to_delete_redo_log_files(),
innobase_start_or_create_for_mysql(): Upgrade from a previous (supported)
redo log format.

The bug was that for prepared statments the new TABLE_LIST was
allocated in the wrong arena.

Do not try to apply the splitting optimization to a materialized
derived if it's specified by a select with impossible where
or if all joined tables of this select are constant.

splitting technique for equi-joins of materialized derived tables/views/CTEs.
(see mdev-13369 and mdev-13389).

When running an optimized build, compiler optimizations cause loss of
precision unless we force the expression to use a double type.

Both libraries and includes are required to check for symbols correctly.

In addition, we don't use SHA512_DIGEST_LENGTH symbol so remove the
extraneous check.

Disable tokudb compilation when doing rpl tests. This saves time and
allows us to not timeout on travis.

* fix a data race in debug build

This could crash in mysql_unlock_tables in servers compiled with DBUG

- Fixed innodb.recovery_shutdown.test
- Fixed innodb.instant_alter_rollback

Other changes done to get this to work:
- Added 'internal_tables' to TABLE object to list which sequence tables
  is needed to use the table.
- Mark any expression using DEFAULT() with LEX->default_used.
  This is needed when deciding if we should open internal sequence
  tables when a table is opened (we don't need to open sequence tables
  if the main table is only used with SELECT).
- Create_and_open_temporary_table() can now also open all internal
  sequence tables.
- Added option MYSQL_LOCK_USE_MALLOC to mysql_lock_tables()
  to force memory allocation to be used with malloc instead of
  memroot.
- Added flag to MYSQL_LOCK to remember if allocation was done with
  malloc or memroot (makes code simpler and safer).
- init_one_table_for_prelocking() now takes argument for what lock to
  use instead of it's a routine or something else.
- Renamed prelocking placeholders to make them more understandable as
  they are now used in more code.
- Changed test in check_lock_and_start_stmt() if found table has correct
  locks. The old test didn't work for tables that has lock
  TL_WRITE_ALLOW_WRITE, which is what sequence tables are using.
- Added VCOL_NOT_VIRTUAL option to ensure that sequence functions can't
  be used with virtual columns
- More sequence tests

This is needed for MDEV 13679 Enabled sequences to be used in DEFAULT

Added new option for count_cuted_fields: CHECK_FIELD_EXPRESSION
which is used to check if a DEFAULT expression is correct before
ALTER TABLE starts

Changed also all test:
  if (thd->count_cuted_fields)
  to
  if (thd->count_cuted_fields > CHECK_FIELD_EXPRESSION)

- Fixed also a wrong comment and a wrong argument to printf

This is partial revert of original patch.

Read-only transactions that modified temporary tables are added to
trx_sys_t::rw_trx_ids and trx_sys_t::rw_trx_set. However with patch for
MDEV-14477 they were not removed.

Restore old behaviour in this regard.

* rename in_subquery_conversion_threshold to in_predicate_conversion_threshold
* make it debug-only, hide from users
* change from ulong to uint - same type and range on all architectures

Now that MDEV-14717 made RENAME TABLE crash-safe within InnoDB,
it should be safe to drop the #sql- tables within InnoDB during
crash recovery. These tables can be one of two things:

(1) #sql-ib related to deferred DROP TABLE (follow-up to MDEV-13407)
or to table-rebuilding ALTER TABLE...ALGORITHM=INPLACE
(since MDEV-14378, only related to the intermediate copy of a table),

(2) #sql- related to the intermediate copy of a table during
ALTER TABLE...ALGORITHM=COPY

We will not drop tables whose name starts with #sql2, because
the server can be killed during an ALGORITHM=COPY operation at
a point where the original table was renamed to #sql2 but the
finished intermediate copy was not yet renamed from #sql-
to the original table name.

InnoDB in MariaDB 10.2 appears to only write MLOG_FILE_RENAME2
redo log records during table-rebuilding ALGORITHM=INPLACE operations.
We must write the records for any .ibd file renames, so that the
operations are crash-safe.

If InnoDB is killed during a RENAME TABLE operation, it can happen that
the transaction for updating the data dictionary will be rolled back.
But, nothing will roll back the renaming of the .ibd file
(the MLOG_FILE_RENAME2 only guarantees roll-forward), or for that matter,
the renaming of the dict_table_t::name in the dict_sys cache. We introduce
the undo log record TRX_UNDO_RENAME_TABLE to fix this.

fil_space_for_table_exists_in_mem(): Remove the parameters
adjust_space, table_id and some code that was trying to work around
these deficiencies.

fil_name_write_rename(): Write a MLOG_FILE_RENAME2 record.

dict_table_rename_in_cache(): Invoke fil_name_write_rename().

trx_undo_rec_copy(): Set the first 2 bytes to the length of the
copied undo log record.

trx_undo_page_report_rename(), trx_undo_report_rename():
Write a TRX_UNDO_RENAME_TABLE record with the old table name.

row_rename_table_for_mysql(): Invoke trx_undo_report_rename()
before modifying any data dictionary tables.

row_undo_ins_parse_undo_rec(): Roll back TRX_UNDO_RENAME_TABLE
by invoking dict_table_rename_in_cache(), which will take care
of both renaming the table and the file.

Follow-up to MDEV-13407 innodb.drop_table_background failed in buildbot with "Tablespace for table exists"

The InnoDB background DROP TABLE queue is something that we should
really remove, but are unable to until we remove dict_operation_lock
so that DDL and DML operations can be combined in a single transaction.

Because the queue is not persistent, it is not crash-safe. We should
in some way ensure that the deferred-dropped tables will be dropped
after server restart.

The existence of two separate transactions complicates the error handling
of CREATE TABLE...SELECT. We should really not break locks in DROP TABLE.

Our solution to these problems is to rename the table to a temporary
name, and to drop such-named tables on InnoDB startup. Also, the
queue will use table IDs instead of names from now on.

check-testcase.test: Ignore #sql-ib*.ibd files, because tables may enter
the background DROP TABLE queue shortly before the test finishes.

innodb.drop_table_background: Test CREATE...SELECT and the creation of
tables whose file name starts with #sql-ib.

innodb.alter_crash: Adjust the recovery, now that the #sql-ib tables
will be dropped on InnoDB startup.

row_mysql_drop_garbage_tables(): New function, to drop all #sql-ib tables
on InnoDB startup.

row_drop_table_for_mysql_in_background(): Remove an unnecessary and
misplaced call to log_buffer_flush_to_disk(). (The call should have been
after the transaction commit. We do not care about flushing the redo log
here, because the table would be dropped again at server startup.)

Remove the entry from the list after the table no longer exists.

If server shutdown has been initiated, empty the list without actually
dropping any tables. They will be dropped again on startup.

row_drop_table_for_mysql(): Do not call lock_remove_all_on_table().
Instead, if locks exist, defer the DROP TABLE until they do not exist.
If the table name does not start with #sql-ib, rename it to that prefix
before adding it to the background DROP TABLE queue.

Follow-up fix to MDEV-14008: Let Field_double::val_uint() silently
return 0 on error

WSREP_TO_ISOLATION_BEGIN() call must be after view name is back
on tables list.

During the user-defined variable defined by the recursive CTE handling procedure
check_dependencies_in_with_clauses that checks dependencies between the tables
that are defined in the CTE and find recursive definitions wasn't called.

Signed-off-by: Daniel Black <daniel@linux.vnet.ibm.com>