Otherwise test fails:

--- mysql-test/suite/innodb/t/innodb-fk.test
+++ mysql-test/suite/innodb/t/innodb-fk.test
@@ -11,7 +11,7 @@ insert into t1 values (5);
 insert into t1 values (2882);
 insert into t1 values (10);

-let $fk_tables = 120;
+let $fk_tables = 500;

 --disable_query_log
 let $i = $fk_tables;

Do the checks for referenced table like it was done for foreign table.
Upgrade code pushes hints to frm, table is reopened and these hints
kept in the share even if GTS_FK_SHALLOW_HINTS is not set: when
Foreign_key_io::parse() resolves referenced keys to empty list it
falls back to make_shallow_hint.

Then fk_check_and_upgrade() sees these hints and assumes the upgrade
process was already done by setting check_refs to false.

Finally, when foreign table is upgraded it sees the hints in
referenced table and replaces them with real foreign keys.

Tests produce more warnings because of extra open table done by
Share_acquire::upgrade().

FIXME: drop after MDEV-21053

REPAIR TABLE on foreign table updates reference hints in referenced
tables if needed.

Better diagnostic error codes for CHECK TABLE.

The task is to upgrade table foreign key metadata for the old storage
of InnoDB system tables SYS_FOREIGN and SYS_FOREIGN_COLS by reading
the foreign key definitions on table open and storing them into
table's FRM file. The table is upgraded whenever it is opened.

When the system tables SYS_FOREIGN(_COLS) become empty during the
process of foreign keys upgrade they are automatically dropped.

The task is required for the database migration from the previous
versions of MariaDB. Without this functionality any foreign keys
defined in previous versions of MariaDB will not have effect and will
become inaccessible.

== fk_check_legacy_storage(), fk_upgrade_legacy_storage() ==

fk_check_legacy_storage() checks whether upgrade is required for the
given table name by looking SYS_FOREIGN table for corresponding record
existence.

fk_upgrade_legacy_storage() does the upgrade routine which includes
getting the foreign keys from SYS_FOREIGN[_COLS], updating the
foreign/referenced shares as well as their FRM files, deleting the
records from SYS_FOREIGN[_COLS] tables.

Both routines utilize the internal SQL for SYS_FOREIGN[_COLS]
processing.

== Upgrade foreign keys via backoff action ==

When table is opened fk_check_legacy_storage() detects whether upgrade
is required and HA_ERR_FK_UPGRADE is returned to SQL layer which then
handles this error by backoff action from Open_table_context where
table is opened again with HA_OPEN_FOR_REPAIR flag which indicates
that fk_upgrade_legacy_storage() is required. After
fk_upgrade_legacy_storage() is done fk_check_legacy_storage() is
checked again to ensure that SYS_FOREIGN[_COLS] are empty for the
given table.

== DB_LEGACY_FK and HA_ERR_FK_UPGRADE error codes ==

Indicate that table has legacy data in SYS_FOREIGN[_COLS].

== Check foreign/referenced indexes existence ==

fk_upgrade_legacy_storage() via fk_upgrade_push_fk() fails if there
are no indexes in foreign/referenced tables for the given data
acquired from SYS_FOREIGN[_COLS].

== Internal SQL: select into both func and vars extension ==

fk_upgrade_legacy_storage() utilizes syntax extension in internal SQL:

  FETCH c INTO fk_upgrade_create_fk() fk_id, unused;

Thus the data is fetched into both fk_upgrade_create_fk() function and
fk_id variable.

== Rename table, rename column, drop table, drop column handling ==

When foreign table is opened it is automatically upgraded by backoff
action. But if the referenced table is altered or dropped first there
is no chance for the foreign table to get the correct data. So the
SYS_FOREIGN_[COLS] must be kept in sync with the above DDL operations
in respect of referenced names. DROP TABLE for the referenced table is
disabled as usual. DROP TABLE, DROP COLUMN relied in 10.5 on
dict_foreign_t cached data for the referenced tables. Now there is no
such possibility for the legacy data so we have to look at
SYS_FOREIGN_[COLS] directly.

== Reverted some SYS_FOREIGN(_COLS) routines ==

Rename table and rename column handling was done in sync with
SYS_FOREIGN_[COLS] in 10.5. To retain the above DDL consistency for
the referenced tables we still use that old synchronization code.

== ALGORITHM=COPY handling ==

Since we cannot faingrain ALGORITHM=COPY in innobase handler it is
disabled for the referenced tables unless the foreign tables are upgraded.

The check is done in create_table_info_t::create_table() and is
equivalent to DROP TABLE check as we are actually dropping the old
table after the copy routine is done.

== HA_EXTRA_CHECK_LEGACY_FK ==

Extra interface to check whether ALGORITHM=COPY is possible. If there
is legacy data in SYS_FOREIGN[_COLS] about foreign tables referring
altered table, ALGORITHM=COPY is not possible.

== WITH_INNODB_FOREIGN_UPGRADE macro ==

Every SYS_FOREIGN_[COLS] handling is wrapped inside
WITH_INNODB_FOREIGN_UPGRADE compilation macro. When this macro is
disabled the foreign key upgrade is not possible. Future versions will
obsolete the upgrade procedure completely and the wrapped code will be
removed.

== innodb_eval_sql debug interface ==

Test cases must fill SYS_FOREIGN_[COLS] with data. This is done with
setting the new innodb_eval_sql debug variable. The value of that
variable is processed by que_eval_sql().

== Some syntax error-friendly parser handling ==

que_eval_sql() was unfriendly on syntax errors: it just failed with
SIGABRT exception. To keep the server alive some frequent syntax
errors are now returned as DB_ERROR from que_eval_sql().

== Drop empty SYS_FOREIGN[_COLS] after import ==

fk_create_legacy_storage debug flag to create SYS_FOREIGN[_COLS] on
innodb_eval_sql.

fk_release_locks() iterates through all indexes and pages and releases
locks on SUPREMUM records placed there previously by SELECT FOR
UPDATE. Also it releases all table locks.

== pars_info_t::fatal_syntax_err (debug only) ==

When false don't abort server on syntax errors while parsing internal
SQL code.

Refactor dict_load_foreigns() for synchronising TABLE_SHARE foreign
data with dict_table_t cache.

Remove a number of routines working with SYS_FOREIGNS and
SYS_FOREIGN_COLS. innobase_update_foreign_try() is now used solely for
ER_FK_INCORRECT_OPTION check.

Prelock parent tables as well as child tables. This is done for the
case when parent table doesn't know about its children when they
created before parent with foreign_key_checks=0. Opening the parent
table initiates fk_resolve_referenced_keys() which updates its
referenced_keys. Due to CREATE TABLE doesn't not know about "illegal"
children it can not check for foreign consistency. F.ex. this would
succeed:

set foreign_key_checks= 0;
create table child (fk int references parent (id)) engine=innodb;
set foreign_key_checks= 1;

create table parent (id bigint primary key) engine=innodb;

In the above case dict_load_foreigns() deduces which tables are
unknown to opened parent (tables_missing) and reloads their foreign
data via recursion. Infinite recursion is not possible via test case:
a table cannot be "parent after child" and "child before parent"
simultaneously. Though infinite recursion is possible via malicously
crafted FRM file, there is no protection from that at InnoDB level but
there is protection at SQL level: thd->fk_circular_check.

Later though it would not allow DML on child as well as on parent (see
innodb.foreign_key MDEV-10083). So this is pretty acceptable:
foreign_key_checks is unnormal setting, checking parent on CREATE
TABLE would impose all frms scanning which is not acceptable.

ha_innobase::open() then synchronizes these referenced_keys with its
referenced_set cache by calling dict_load_foreigns().

Disable self-references on same column. The Bug 12902967 restricted
them on some condition of "same column/index" (see
innodb_bug12902967.test), though such self-references were not
completely disabled (see other self-ref cases changed in this
patch). It is not clear why they worked if they are "self-refs on same
column/index".

The semantics of dict_foreign_add_to_cache() has been
changed. Foreigns must be synchronized on every table open and cache
must be updated. The old semantics was to keep old foreign while it is
in cache.

Under LOCK TABLES there is no ticket for old_table.

Partial fix. Don't set DA_OK status until all processing that may set
error status is done.

Still fails:

mysqltest: At line 3: query 'rename table t1 to t3, t3 to t1' failed:
<Unknown> (6): Error on delete of './test/t3.frm' (Errcode: 2 "No such
file or directory")

The above depends on MDEV-21053.

1. Access foreign keys via TABLE_SHARE::foreign_keys and
  TABLE_SHARE::referenced_keys;

foreign_keys and referenced_keys are objects in TABLE_SHARE.

2. Remove handler FK interface:

  - get_foreign_key_list()
  - get_parent_foreign_key_list()
  - referenced_by_foreign_key()

3. Invalidate referenced shares on:

  - RENAME TABLE
  - DROP TABLE
  - RENAME COLUMN
  - ADD FOREIGN KEY

When foreign table is created or altered by the above operations all
referenced shares are closed. This blocks the operation while any
referenced shares are used (when at least one its TABLE instance is
locked).

4. Update referenced shares on:

  - CREATE TABLE

On CREATE TABLE add items to referenced_keys of referenced shares.

5. Invalidate foreign shares on:

  - RENAME TABLE
  - RENAME COLUMN

The above-mentioned blocking takes effect.

6. Check foreign/referenced shares consistency on:

  - CHECK TABLE

7. Temporary change until MDEV-21051:

InnoDB fill foreign key info at handler open().

On first TABLE open FK info is loaded from storage engine into
TABLE_SHARE. All referenced shares (if any exist) are closed. This
leads to blocking of first time foreign table open while referenced
tables are used.

Restore states of referenced shares in case of errors.

FOREIGN_KEY_INFO refactored to FK_info holding Lex_cstring

(MDEV-21311) Converge Foreign_key and supplemental generated Key together

mysql_prepare_create_table() does data validation and such utilities
as automatic name generation. But it does that only for indexes and
ignores Foreign_key objects. Now as Foreign_key data needs to be
stored in FRM files as well this processing must be done for them like
for any other Key objects.

Replace Key::FOREIGN_KEY type with Key::foreign flag of type
Key::MULTIPLE and Key::generated set to true. Construct one object
with Key::foreign == true instead of two objects of type
Key::FOREIGN_KEY and Key::MULTIPLE.

(MDEV-21051) datadict refactorings

- Move read_extra2() to datadict.cc
- Refactored extra2_fields to Extra2_info
- build_frm_image() readability

(MDEV-21051) build_table_shadow_filename() refactoring

mysql_prepare_alter_table() leaks fixes

(MDEV-21051) amend system tables locking restriction

Table mysql.help_relation has foreign key to mysql.help_keyword. On
bootstrap when help_relation is opened, it preopens help_keyword for
READ and fails in lock_tables_check().

If system table is opened for write then fk references are opened for
write.

Related to: Bug#25422, WL#3984
Tests: main.lock

(MDEV-21051) Store and read foreign key info into/from FRM files

1. Introduce Foreign_key_io class which creates/parses binary stream
containing foreign key structures. Referenced tables store there only
hints about foreign tables (their db and name), they restore full info
from the corresponding tables.

Foreign_key_io is stored under new EXTRA2_FOREIGN_KEY_INFO field in
extra2 section of FRM file.

2. Modify mysql_prepare_create_table() to generate names for foreign
keys. Until InnoDB storage of foreign keys is removed, FK names must
be unique across the database: the FK name must be based on table
name.

3. Keep stored data in sync on DDL changes. Referenced tables update
their foreign hints after following operations on foreign tables:

  - RENAME TABLE
  - DROP TABLE
  - CREATE TABLE
  - ADD FOREIGN KEY
  - DROP FOREIGN KEY

Foreign tables update their foreign info after following operations on
referenced tables:

  - RENAME TABLE
  - RENAME COLUMN

4. To achieve 3. there must be ability to rewrite extra2 section of
FRM file without full reparse. FRM binary is built from primary
structures like HA_CREATE_INFO and cannot be built from TABLE_SHARE.

Use shadow write and rename like fast_alter_partition_table()
does. Shadow FRM is new FRM file that replaces the old one.

Create table workflow:

1. Foreign_key is constructed in parser, placed into alter_info->key_list;

2. mysql_prepare_create_table() translates them to FK_info, assigns
   foreign_id if needed;

3. build_frm_image() writes two FK_info lists into FRM's extra2
   section, for referenced keys it stores only table names (hints);

4. init_from_binary_frm_image() parses extra2 section and fills
   foreign_keys and referenced_keys of TABLE_SHARE.

   It restores referenced_keys by reading hint list of table names,
   opening corresponding shares and restoring FK_info from their
   foreign_keys. Hints resolution is done only when initializing
   non-temporary shares. Usually temporary share has different
   (temporary) name and it is impossible to resolve foreign keys by that
   name (as we identify them by both foreign and referenced table
   names). Another not unimportant reason is performance: this saves
   spare share acquisitions.

Alter table workflow:

1. Foreign_key is constructed in parser, placed into
   alter_info->key_list;

2. mysql_prepare_alter_table() prepares action lists and share list of
   foreigns/references;

3. mysql_prepare_alter_table() locks list of foreigns/references by
   MDL_INTENTION_EXCLUSIVE, acquires shares;

4. prepare_create_table() converts key_list into FK_list, assigns
   foreign_id;

5. shadow FRM of altered table is created;

6. data is copied;

7. altered table is locked by MDL_EXCLUSIVE;

8. fk_handle_alter() processes action lists, creates FK backups,
   modifies shares, writes shadow FRMs;

9. altered table is closed;

10. shadow FRMs are installed;

11. altered table is renamed, FRM backup deleted;

12. (TBD in MDEV-21053) shadow FRMs installation log closed, backups deleted;

On FK backup system:

In case of failed DDL operation all shares that was modified must be
restored into original state. This is done by FK_ddl_backup (CREATE,
DROP), FK_rename_backup (RENAME), FK_alter_backup (ALTER).

On STL usage:

STL is used for utility not performance-critical algorithms, core
structures hold native List. A wrapper was made to convert STL
exception into bool error status or NULL value.

MDEV-20865 fk_check_consistency() in CHECK TABLE

Self-refs fix

Test table_flags fix: "debug" deviation is now gone.

read_extra2() is now Extra2_info::read()
Additional assertions for checking size consistency.

Extra2_info::write() is used by further MDEV-20865 development.

Current implementation of extra2_write_len() does not guarantee of
writing correct 2-byte values as it skips writing zero at the
beginning.

Refined DBUG_ASSERT() to more truthful limit.

More clear names and constants use.

datadict.h is appropriate place for such types. These data types are
used by Extra2_info, which is added to datadict.h later.

unireg.cc is for older FRM routines. datadict.cc accepts new and
refactored routines.

Three problems:

  1. Premature free of result in execute_inner;

  2. Result is allocated on statement mem_root which is freed after
     statement complete. This is wrong for PS which accesses result in
     more statements;

  3. Bad practice: subclasses should not free members of superclass.

The execution goes like this:

  1. Sql_cmd_delete::execute_inner() as the part of DML command, it
     frees result;

  2. MYSQL_DML_DONE() is called in the same statement, it accesses
     result;

  3. thd->mem_root is freed in the same statement.

The above is valid for PS as well (tested by main.ps): result is
allocated on thd->mem_root no matter in plain or PS, but destructor is
required to free result internal structures.

The patch allocates result on PS arena which outlives end of statement
and allows access in multiple statements (tested by main.ps).

There is no destructor of Sql_cmd_dml as it is constructed on
mem_root. So we free result after MYSQL_DML_DONE().

Caused by 3a9358a4 (MDEV-28883).

When there are GCC-incompatible compiler flags dtrace fails like this:

gcc: error: unrecognized command-line option ‘-fno-limit-debug-info’
gcc: error: unrecognized command-line option ‘-mbranches-within-32B-boundaries’
"gcc .dtrace-temp.3fd6bacf.c" failed
Usage /usr/bin/dtrace [--help] [-h | -G] [-C [-I<Path>]] -s File.d [-o <File>]

Apart from better performance when accessing thread local variables,
we'll get rid of things that depend on initialization/cleanup of
pthread_key_t variables.

Where appropriate, use compiler-dependent pre-C++11 thread-local
equivalents, where it makes sense, to avoid initialization check overhead
that non-static thread_local can suffer from.

Make it clear that refactoring code goes into the latest version.

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Implement automatic creation of temporary accounts for SST and pass
account credentials to SST script via socket as opposed to environment
variables. Delete the user after the SST script returns,

Respect wsrep_sst_auth set by the adminitrator in case some additional
privilege grants are needed for particular SST method.

mysqldump SST requires significant change to make use of the new
automatic user generation facility. For now just make it compatible
by ignoring automatically generated user and rely only on wsrep_sst_auth
setting on the joiner node to keep backward compatibility.

Adapt mysqldump SST to automatic SST user generation changes:
 - disable special treatment for mysqldump SST on donor
 - make mysqldump SST script compatible with the new SST script
   interface.

Differentiate user privileges for different SST methods:
 - grant minimum required privileges for clone and xtrabackup SST
   accounts
 - grant all privileges to custom SST accounts as it is not known what
   is needed.
 - disable SST account generation for rsync SST since it is not needed.

MTR tests:
 - add MTR tests for clone and xtrabackup SSTs without wsrep_sst_auth,
 - add MTR test for testing masking of wsrep_sst_auth.
 - don't attmept to restore original wsrep_sst_auth in MTR tests as it
   is always masked.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Add two-way communication between parent and child in wsp::proc class.
Refactor wsp::thd class to call my_thread_init() conditionally.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

If mariabackup does not get the password on command line or from
the [mariabackup] section of the config file, it initializes the
internal opt_password variable to empty string and considers it
as set in a subsequent check, therefore ignoring the value of
MYSQL_PWD envronment variable. At the same time MariDB server
considers empty string passwords as no password.

Fixing this is necessary to use automatically generated SST users
because mariabackup needs --default-file option to use the same
config as the server and that option does not allow to supply any
extra config files, so using the automatically generated config
with [mariabackup] section is out of question.

Modify check for set password to treat empty string as unset
and fall back to the MYSQL_PWD value if present.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

		tablespace truncation

- InnoDB fails with out of bound write error after temporary
tablespace truncation. This issue caused by
commit c507678b (MDEV-28699).
InnoDB fail to clear freed ranges if shrinking size
is the last offset of the freed range.

Now that we do not pollute systemd's environment but write private
environment files running these as root is not longer required. So
let's drop `PermissionsStartOnly=true`.

Debian adds extra `ExecStartPre=` and `ExecStartPost=`, though.
Use special executable prefix for full privileges there. (See
systemd.service(5) for details.)

Support in service file was dropped in 91f16948
(systemd: multi-instance not for Galera, User/Group flexible).

Now that the systemd unit files use an environment file to pass
_WSREP_START_POSITION we have to update galera_new_cluster as well.

We used to run `systemctl set-environment` to pass
_WSREP_START_POSITION. This is bad because:

* it clutter systemd's environment (yes, pid 1)
* it requires root privileges
* options (like LimitNOFILE=) are not applied

Let's just create an environment file in ExecStartPre=, that is read
before ExecStart= kicks in. We have _WSREP_START_POSITION around for the
main process without any downsides.

mysqldump --dir produces unexpected sql output on stdout.
All sql should go into .sql files (and data to .txt files)

A cleanup for MDEV-25829.

Using /WX (warning as error) during configuration step caused
HAVE_OPENSSL_APPLINK_C to be OFF

This made client utilities fail with "Applink not found" during runtime.

Hardcode -DHAVE_OPENSSL_APPLINK_C=1 for CMake for now, fix connector
later.