Problem:
=======
 InnoDB allows virtual index to be referenced index in foreign key
relations. While dropping the virtual column, Inplace alter does
allow the table to be closed and open it using table name to
update dict_table_t::v_cols. While loading the table, it doesn't
allow any error to be ignored. InnoDB can't find the referenced
virtual index and fails to load the table during Inplace alter.

Solution:
=========
  During inplace alter, ignore the foreign key error while loading
the table.

Reviewed-by: Marko Mäkelä

Made cleanup of DROP (udf) FUNCTION procedure and also check of mysql.func (not only loaded udf).

MDEV-16676 Using malloc-lib=jemalloc in MariaDB 10.2 causes non-critical error about missing mysql_config on startup

- This commit is based on patch 84fe9720a4d2483ff67b6a and suggestion of
Jean Weisbuch to use `ldconfig -p`
- Format of ldconfig output:
"libjemalloc.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libjemalloc.so.1"

```
$ ./scripts/mysqld_safe
201013 13:36:50 mysqld_safe Adding '/usr/lib/x86_64-linux-gnu/libjemalloc.so.1' to LD_PRELOAD for mysqld
201013 13:36:50 mysqld_safe Logging to '/home/anel/builds/data-10.5/mysqld_safe_anel.err'.
201013 13:36:50 mysqld_safe Starting mysqld daemon with databases from /home/anel/builds/data-10.5
```
Reviewed by: Daniel Black, Faustin Lammler

Instead of pointlessly waiting for a page flush to occur, take
the matter into our own hands and request an explicit flush.
Also, test with the minimum necessary amount of data (0 or 1 rows)
so that both page encryption and decryption will be exercised.

This fix removes from the list of disabled tests all
tests that were fixed by the patch for MDEV-21770.

This patch fixes several flaws in the SST scripts that cause
failures while running tests that use version 6 IP addresses
for cluster nodes.

First, if the netcat utility is used for streaming (but not socat),
then in accordance with its command line syntax, we need to remove
the square brackets around the IPv6 address. However, for socat,
the address must contain square brackets, as before.

Secondly, if an IPv6 address is used, then from the joiner side for
a number of systems (such as Debian) we need to explicitly specify
the "-6" option, otherwise a listening socket with an IPv6 address
may not be created.

This patch also contains code improvements in the wsrep_sst_common.
Changed the code that pars the connection address - fixed the
shortcomings that sometimes led to incorrect parsing of parameters
when using shells other than the latest versions of bash.

Also, this patch removes the duplicate code that is intended
for parsing the connection address and which was located in the
wsrep_sst_mariabackup file, since all the necessary actions have
already been done in wsrep_sst_common and there they are done in
such a way that any shell is supported, not just bash.

The fix does not require separate tests, since all the
necessary tests are already present in the galera_3nodes suite.
On the contrary, after this fix, tests using IPv6 addresses can
be removed from the disabled list (this will be done in a separate
commit related to MDEV-23659).

As MariaDB tables are expected in a fixed order, the cross upgrade
previously placed roles, default_role and max_statement_time after
now unused mysql-5.7 columns.

Test:

$ cp -a /usr/local/mysql-5.7.31/data/ /tmp/m57data
$ sql/mysqld --no-defaults --skip-networking --datadir=/tmp/m57data  --socket=/tmp/${PWD##*/}.sock   --verbose --lc-messages-dir=$PWD/sql/share
2020-10-02 11:02:05 140135193212864 [Note] sql/mysqld (mysqld 10.2.34-MariaDB) starting as process 1457667 ...
2020-10-02 11:02:05 140135193212864 [Note] InnoDB: Mutexes and rw_locks use GCC atomic built

$ client/mysql_upgrade --no-defaults -u root -pbob  -S /tmp/build-mariadb-server-10.2.sock
MySQL upgrade detected
Phase 1/7: Checking and upgrading mysql database
Processing databases
mysql
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.engine_cost                                  OK
mysql.event                                        OK
mysql.func                                         OK
mysql.gtid_executed                                OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.innodb_index_stats                           OK
mysql.innodb_table_stats                           OK
mysql.ndb_binlog_index                             OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.server_cost                                  OK
mysql.servers                                      OK
mysql.slave_master_info                            OK
mysql.slave_relay_log_info                         OK
mysql.slave_worker_info                            OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
Upgrading from a version before MariaDB-10.1
Phase 2/7: Installing used storage engines

Result:

| user  | CREATE TABLE `user` (
  `Host` char(60) COLLATE utf8_bin NOT NULL DEFAULT '',
  `User` char(80) COLLATE utf8_bin NOT NULL DEFAULT '',
  `Password` char(41) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL DEFAULT '',
  `Select_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Insert_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Update_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Delete_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Create_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Drop_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Reload_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Shutdown_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Process_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `File_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Grant_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `References_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Index_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Alter_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Show_db_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Super_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Create_tmp_table_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Lock_tables_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Execute_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Repl_slave_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Repl_client_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Create_view_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Show_view_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Create_routine_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Alter_routine_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Create_user_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Event_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Trigger_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Create_tablespace_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `ssl_type` enum('','ANY','X509','SPECIFIED') CHARACTER SET utf8 NOT NULL DEFAULT '',
  `ssl_cipher` blob NOT NULL,
  `x509_issuer` blob NOT NULL,
  `x509_subject` blob NOT NULL,
  `max_questions` int(11) unsigned NOT NULL DEFAULT 0,
  `max_updates` int(11) unsigned NOT NULL DEFAULT 0,
  `max_connections` int(11) unsigned NOT NULL DEFAULT 0,
  `max_user_connections` int(11) NOT NULL DEFAULT 0,
  `plugin` char(64) CHARACTER SET latin1 NOT NULL DEFAULT '',
  `authentication_string` text COLLATE utf8_bin NOT NULL,
  `password_expired` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `is_role` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `default_role` char(80) COLLATE utf8_bin NOT NULL DEFAULT '',
  `max_statement_time` decimal(12,6) NOT NULL DEFAULT 0.000000,
  `password_last_changed` timestamp NULL DEFAULT NULL,
  `password_lifetime` smallint(5) unsigned DEFAULT NULL,
  `account_locked` enum('N','Y') COLLATE utf8_bin NOT NULL DEFAULT 'N',
  PRIMARY KEY (`Host`,`User`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='Users and global privileges' |

MariaDB [(none)]>  CREATE ROLE `aRole`;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> SET ROLE `aRole`;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> SET ROLE `aRole`;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> SELECT `User`, `is_role` FROM `mysql`.`user`;
+---------------+---------+
| User          | is_role |
+---------------+---------+
| root          | N       |
| mysql.session | N       |
| mysql.sys     | N       |
| dan           | N       |
| aRole         | Y       |
+---------------+---------+
5 rows in set (0.00 sec)

Reviewer: Anel Husakovic

MDEV-23960 UBSAN ../storage/innobase/buf/buf0buddy.cc:350:6: runtime error: index 4096 out of bounds for type 'byte [38]'

Reviewed by: Marko Mäkelä

Copying generated data from the sequence engine should be faster
than copying from MyISAM to MyISAM.

Reviewed by: Sergei Petrunia

minor fixes of rpl_start_stop_slave and rpl_slave_grp_exec tests, expanding rpl_gtid_delete_domain for easier later analysis

Sometimes blockdev --getss returns 4096.
In that case ROW_FORMAT=COMPRESSED tables might violate
that 4096 bytes alignment.

This patch disables O_DIRECT for COMPRESSED tables.

OS_DATA_FILE_NO_O_DIRECT: new possible value for os_file_create() argument

fil_node_open_file(): do not O_DIRECT
ROW_FORMAT=COMPRESSED tables

AIO::is_linux_native_aio_supported(): minimal alignment in a general case
is 4096 and not 512.

Changes to be committed:
	modified:   mysql-test/suite/galera/disabled.def
	modified:   mysql-test/suite/wsrep/disabled.def

Stabilize test by using correct galera library and restore
original galera cluster at end.

Add necessary wait_conditions to stabilize test.

MDEV-21329 InnoDB: Failing assertion: lock->lock_word.load(std::memory_order_relaxed) == X_LOCK_DECR upon server shutdown

Problem is that dropping of fts table and sync of fts table
happens concurrently during fts optimize thread shutdown.
fts_optimize_remove_table() is executed by a user thread that
performs DDL, and that the fts_optimize_wq may be removed if
fts_optimize_shutdown() has started executing.

fts_optimize_remove_table() doesn't remove the table from the queue
and it leads to above scenario. While removing the table from
fts_optimize_wq, if the table can't be removed then wait till
fts_optimize_thread shuts down.

Reviewed-by: Marko Mäkelä

This bug could manifest itself for a query with WHERE condition containing
top level OR formula such that each conjunct contained a single-range
condition supported by the same index. One of these range conditions must
be fully covered by another range condition that is used later in the OR
formula. Additionally at least one of these condition should be ANDed with
a sargable range condition supported by a different index.

There were several attempts to fix related problems for OR conditions after
the backport of range optimizer code from MySQL (commit
0e19f3e3). Unfortunately the first of these
fixes contained typo remained unnoticed until recently. This typo bug led
to rejection of valid range accesses. This patch fixed this typo bug.
The fix revealed another two bugs: one in a constructor for SEL_ARG,
the other in the function tree_or(). Both are fixed in this patch.

This regression for debug builds was introduced by
MDEV-23101 (commit 224c9504).

Due to MDEV-16664, the parameter
innodb_lock_schedule_algorithm=VATS
is not enabled by default.

The purpose of the added assertions was to enforce the invariant that
Galera replication cannot be enabled together with VATS due to MDEV-12837.
However, upon closer inspection, it is obvious that the variable 'lock'
may be assigned to the null pointer if no match is found in the
previous->hash list.

lock_grant_and_move_on_page(), lock_grant_and_move_on_rec():
Assert !lock->trx->is_wsrep() only after ensuring that lock
is not a null pointer.

1. rr record -h randomizes number of processors. Disable THREAD_POOL_SIZE check.

2. check for kernel.perf_event_paranoid for user-friendly error message.

MDEV-18163 Assertion `table_share->tmp_table != NO_TMP_TABLE || m_lock_type != 2' failed in handler::ha_rnd_next(); / Assertion `table_list->table' failed in find_field_in_table_ref / ERROR 1901 (on optimized builds)

Add the same check for altering DEFAULT used as for CREATE TABLE.

Remove from debian build:
* tokudb
* mroonga
* spider
* ograph
* embedded server

Add ccache to debian build.

Backport 10.3 changes to autobake-deb
that make travis faster.

Merge instructions:
Drop this commit on merge to 10.3

MDEV-23894 UBSAN: several call to function show_binlog_vars(THD*, st_mysql_show_var*, char*) through pointer to incorrect function type 'int (*)(THD *, st_mysql_show_var *, void *, system_status_var *, enum_var_type) errors

MDEV-18593 : galera.galera_gcache_recover_full_gcache: Test failure: galera_gcache_recover_full_gcache.test: assert_grep.inc failed

Grep only the fact that we need to fall back to SST.

In MDEV-21452, SAFE_MUTEX flagged an ordering problem that involved
trx_t::mutex, LOCK_global_system_variables, and LOCK_commit_ordered
when running
./mtr --no-reorder\
 binlog.binlog_checksum,mix binlog.binlog_commit_wait,mix

Because LOCK_commit_ordered is acquired by replication code before
innobase_commit_ordered() is invoked, and because LOCK_commit_ordered
should be below LOCK_global_system_variables in the global latching
order, it turns out that we must avoid acquiring
LOCK_global_system_variables in any low-level code.

It also turns out that lock_rec_lock() acquires lock_sys_t::mutex
and then carries on to call lock_rec_enqueue_waiting(), which may
invoke THDVAR() via thd_lock_wait_timeout(). This call is problematic
if THDVAR() had never been invoked in that thread earlier.

innobase_trx_init(): Let us invoke THDVAR() at the start of an InnoDB
transaction so that future invocations of THDVAR() will avoid
LOCK_global_system_variables acquisition on the same THD. Because
the first call to intern_sys_var_ptr() will initialize all session
variables by not passing the offset to sync_dynamic_session_variables(),
this will indeed make any future THDVAR() invocation mutex-free.

There are some THDVAR() calls in other code (related to indexed virtual
columns, fulltext indexes, and DDL operations). No SAFE_MUTEX warning
was known for those, but there does not appear to be any replication
test coverage for indexed virtual columns or fulltext indexes. DDL should
be covered, and perhaps DDL code paths were already invoking THDVAR()
while not holding any InnoDB mutex.

Side note: MySQL should avoid this type of deadlocks since
mysql/mysql-server@4d275c89954685e2ed1b368812b3b5a29ddf9389.
MariaDB never defined alloc_and_copy_thd_dynamic_variables(),
because we prefer to avoid overhead during connection creation.

An important part of the deadlock could be the current handling of
SET GLOBAL binlog_checksum=NONE; and similar assignments.
In binlog_checksum_update(), we would hold LOCK_global_system_variables
while potentially acquiring LOCK_commit_ordered in MYSQL_BIN_LOG::open().
Even if that code was changed later to release
LOCK_global_system_variables during the write to mysql_bin_log,
it could be a good idea for performance to avoid invoking the
expensive code path of THDVAR() while holding any InnoDB mutexes,
such as lock_sys.mutex in lock_rec_enqueue_waiting().

Thanks to Andrei Elkin for debugging the SAFE_MUTEX issue, and to
Sergei Golubchik for the suggestion to invoke THDVAR() early.

Changes to be committed:
	modified:   mysql-test/suite/sys_vars/r/wsrep_cluster_address_basic.result
	modified:   mysql-test/suite/sys_vars/t/wsrep_cluster_address_basic.test

The setting innodb_lock_schedule_algorithm=VATS that was introduced
in MDEV-11039 (commit 021212b5)
causes conflicting exclusive locks to be incorrectly granted to
two transactions. Specifically, in lock_rec_insert_by_trx_age()
the predicate !lock_rec_has_to_wait_in_queue(in_lock) would hold even
though an active transaction is already holding an exclusive lock.
This was observed between two DELETE of the same clustered index record.
The HASH_DELETE invocation in lock_rec_enqueue_waiting() may be related.

Due to lack of progress in diagnosing the problem, we will deprecate the
option and issue a warning that using it may corrupt data. The unsafe
option was enabled between
commit 0c15d1a6 (MariaDB 10.2.3)
and the parent of
commit 1cc1d042 (MariaDB 10.2.17, 10.3.9).

SYNC_REC_LOCK was never used in the public history of InnoDB,
starting with commit 132e667b.

Amend check for unread client data in threadpool.

THD::NET will have unread data, in case client uses compression, and
wraps multiple commands into  a single compression packet

MariaDB C/C sends COM_STMT_RESET+COM_STMT_EXECUTE, and wraps it into
a single compressed packet, when compression is on, thus trying to use
compression and prepared statements against a threadpool-enabled server
will result into a hang, before this patch.

The orphan declaration was added in MySQL 5.6.2
mysql/mysql-server@2915417e026ef9fdd8e36df17938409ccc157d86
and MariaDB commit 1d0f70c2.

The unused fts_t::bg_threads was added in
mysql/mysql-server@4b1049625c00dbfcd9d7a11ad12a84695ab747e3.

Any usage of fts_t::bg_threads_mutex was removed in
mysql/mysql-server@33c2404b397e1077daaf0ef0ff9edba445430f5f.

In fts_optimize_remove_table(), InnoDB tries to access the
fts_optimize_wq after shutting down the fts optimize thread.
This issue caused by the commit a41d4297.
Fix should check for fts optimize thread shutdown state
before checking fts_optimize_wq.

This has been unused from the very beginning
(mysql/mysql-server@d5e512ae7e37cd1f70c44a3f12205d70b13118ab).