======
This is an  adopted version of patch provided in the PR #MariaDB/2680
Original author: Sergey Vanislavskiy
Adopted by: Nikita Maliavin
======

Table opening stage was known to make two list traversals:
1. In find_fk_prelocked_table, all the query_tables list is traversed for each
foreign key of a table to open.
2. MDL_context::find_ticket traverses all mdl tickets, one ticket per table.

Both result in O(tables^2) time total.
This may dramatically increase the query latencty in the following known cases:
* updates/deletes on tables with many children
* DMLs in transactions involving many different tables

Also, it slows down the DROP DATABASE performance, with a big enough amount of
tables.

So to optimize this out the following is done:
* A hash table with all FK-prelocked tables is added to THD. A table is filled
and queried inside find_fk_prelocked_table, and cleaned up at the query end.
* The find_ticket implementation is replaced with two consecutive hash lookups.
* A hash table with all tickets for current context (Query_tables_list) is
added.
* find_fk_prelocked_table now makes a hash lookup as well. We have to calculate
a hash value for this lookup, so MDL_key is created earlier. It's then reused
if a new table list is created.

Given the requirement of no performance degradation for a small table value,
a new data structure is introduced: Open_address_hash.

Open_address_hash is a generic data structure, that is optimized for usage with
a few elements, similarly to the "short string optimization" in c++ standard
libraries: if a number of elements can fit the structure's class body inline,
then they are placed there. One bit is borrowed to indicate whether the inline
mode is used.

This also means that this optimization cannot work in 32-bit environments.

The hash table is implemented from scratch using open address hashing to reduce
memory fragmentation, reduce allocation pressure and memory footprint.

Speaking of the memory footprint, it is expected to be as follows:
* +4x pointer size for each connection (2 pointers per hash, two hashes total)
* +4x pointer size for each prepared statement, or cached/exected stored
procedure.
* If number of tables opened > 2, then +2x pointer size per table, because the
hash load factor is kept 50% at this moment
* If number of FK-prelocked tables opened > 2, then +2x pointer size per
FK-prelocked table.

Under terms of MDEV 27490 we'll add support for non-BMP identifiers
and upgrade casefolding information to Unicode version 14.0.0.
In Unicode-14.0.0 conversion to lower and upper cases can increase octet length
of the string, so conversion won't be possible in-place any more.

This patch removes virtual functions performing in-place casefolding:
  - my_charset_handler_st::casedn_str()
  - my_charset_handler_st::caseup_str()
and fixes the code to use the non-inplace functions instead:
  - my_charset_handler_st::casedn()
  - my_charset_handler_st::caseup()

Only add innodb_async_reads/writes_total_count here.
For all other counters, writing reproducible tests is rather tricky.

Requested by Marko during review

Provide some statistics about asynchronous IO reads and writes:
 - number of pending operations
 - number of completion callbacks that are currently being executed
 - number of completion callbacks that are currently queued
   (due to restriction on number of IO threads)
 - total number of IOs finished
 - total time to wait for free IO slot
 - total number of completions that were queued.

Also revert tpool InnoDB perfschema instrumentation (MDEV-31048)
That instrumentation of cache mutex did not bring any revelation (
the mutex is taken for a couple of instructions), and made it impossible
to use tpool outside of the server (e.g in mariadbimport/dump)

- Add `as <int_type>` to sequence creation options
  - int_type can be signed or unsigned integer types, including
    tinyint, smallint, mediumint, int and bigint
  - Limitation: when alter sequence as <new_int_type>, cannot have any
    other alter options in the same statement
  - Limitation: increment remains signed longlong, and the hidden
    constraint (cache_size x abs(increment) < longlong_max) stays for
    unsigned types. This means for bigint unsigned, neither
    abs(increment) nor (cache_size x abs(increment)) can be between
    longlong_max and ulonglong_max
- Truncating maxvalue and minvalue from user input to the nearest max
  or min value of the type, plus or minus 1. When the truncation
  happens, a warning is emitted
- Information schema table for sequences

After MDEV-31400, plugins are allowed to ask for retries when failing
initialisation. However, such failures also cause plugin system
variables to be deleted (plugin_variables_deinit()) before retrying
and are not re-added during retry.

We fix this by checking that if the plugin has requested a retry the
variables are not deleted. Because plugin_deinitialize() also calls
plugin_variables_deinit(), if the retry fails, the variables will
still be deleted.

Alternatives considered:

- remove the plugin_variables_deinit() from plugin_initialize() error
handling altogether. We decide to take a more conservative approach
here.

- re-add the system variables during retry. It is more complicated
than simply iterating over plugin->system_vars and call
my_hash_insert(). For example we will need to assign values to
the test_load field and extract more code from test_plugin_options(),
if that is possible.

followup for abcd23ad

it was able to enable --ssl-verify-server-cert if explicily requested,
now it can also disable it, if explicitly requested.

In case of failure, the something like the following is now printed:

Slave position:  file: binary.000004  position: 3647
Master position: file: binary.000004  position: 3647

binlog_space_limit is a variable in Percona server used to limit the total
size of all binary logs.

This implementation is based on code from Percona server 5.7.

In MariaDB we decided to call the variable max-binlog-total-size to be
similar to max-binlog-size. This makes it easier to find in the output
from 'mariadbd --help --verbose'). MariaDB will also support
binlog_space_limit for compatibility with Percona.

Some internal notes to explain implementation notes:

- When running MariaDB does not delete binary logs that are either
  used by slaves or have active xid that are not yet committed.

Some implementation notes:

- max-binlog-total-size is by default 0 (no limit).
- max-binlog-total-size can be changed without server restart.
- Binlog file sizes are checked on startup, or if
  max-binlog-total-size is set to a value > 0, not for every log write.
  The total size of all binary logs is cached and dynamically updated
  when updating the binary log on binary log rotation.
- max-binlog-total-size is checked against existing log files during
  serverstart, binlog rotation, FLUSH LOGS, when writing to binary log
  or when max-binlog-total-size changes value.
- Option --slave-connections-needed-for-purge with 1 as default added.
  This allows one to ensure that we do not delete binary logs if there
  is less than 'slave-connections-needed-for-purge' connected.
  Without this option max-binlog-total-size would potentially delete
  binlogs needed by slaves on server startup or when a slave disconnects
  as there are then no connected slaves to protect active binlogs.
- PURGE BINARY LOGS TO ... will be executed as if
  slave-connectitons-needed-for-purge would be zero. In other words
  it will do the purge even if there is no slaves connected. If there
  are connected slaves working on the logs, these will be protected.
- If binary log is on and max-binlog-total_size <> 0 then the status
  variable 'Binlog_disk_use' shows the current size of all old binary
  logs + the state of the current one.
- Removed test of strcmp(log_file_name, log_info.log_file_name) in
  purge_logs_before_date() as this is tested in can_purge_logs()
- To avoid expensive calls of log_in_use() we cache the result for the
  last log that is in use by a slave. Future calls to can_purge_logs()
  for this binary log will be quickly detected and false will be returned
  until a slave starts working on a new log.
- Note that after a binary log rotation caused by max_binlog_size,
  the last log will not be purged directly as it is still in use
  internally. The next binary log write will purge binlogs if needed.

Reviewer:Kristian Nielsen <knielsen@knielsen-hq.org>

libnet-ssleay-perl wasn't auto-detected

Revert "Deb: Stop shipping mariadb-plugin-spider separately, include in server"

This reverts commit 9945d482.

Instead of having a separate plugin, simply include Spider in the main
MariaDB Server package and let users manually enable on server installs
where they want to run it.

Restrict access to KEY_PERIOD_USAGE: show the constraint record iff any
non-select privilege on any table column is granted.

Also drop the unprivileged user in the end of test and add merge anchor.

Set the correct values for idx_field1, idx_field2 (i.e. db and table name
fields) of ST_SCHEMA_TABLE instance.

Move table open result processing to the caller

* st_schema_table::process_table doesn't have to check whether the table
was opened successfully
* It also doesn't have to check for a thd error and convert it to a warning
* This simplifies adding new tables into information_schema
* A callback still can output some info to a user in case of error. In
order to do this, I_S_EXTENDED_ERROR_HANDLING should be specified in
i_s_requested_object.

Now one can call thd->sql_command_flags() instead of
sql_command_flags[thd->lex->sql_command].

Two new information_schema views are added:
* PERIOD table -- columns TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME,
  PERIOD_NAME, START_COLUMN_NAME, END_COLUMN_NAME.
* KEY_PERIOD_USAGE -- works similar to KEY_COLUMN_USAGE, but for periods.
  Columns CONSTRAINT_CATALOG, CONSTRAINT_SCHEMA, CONSTRAINT_NAME,
  TABLE_CATALOG, TABLE_SCHEMA, TABLE_NAME, PERIOD_NAME

Two new columns are added to the COLUMNS view:
 IS_SYSTEM_TIME_PERIOD_START, IS_SYSTEM_TIME_PERIOD_END - contain YES/NO.

Embedded server has an option to opt out grant checks.
sql_show.cc: reduce ifdefs by using embedded acl stub calls
sql_acl.cc: add stub for embedded get_column_grant

Adjust test after fixing the C/C.

On Windows, use --host=127.0.0.2 to fake "insecure" transport
with TCP connection for test purposes. 127.0.0.2 is loopback address,
that can be used instead of usual 127.0.0.1

Unfortunately, this technique does not work on all *nixes the same,
notably neither on BSDs nor Solaris. Thus default --host=localhost
remains "insecure" transport,when TCP is used. but it is not that critical,
the "self-signed" is not nearly as annoying on *nixes as it is on Windows.

Changing the format in error messages:
- ER_PACKAGE_ROUTINE_IN_SPEC_NOT_DEFINED_IN_BODY
- ER_PACKAGE_ROUTINE_FORWARD_DECLARATION_NOT_DEFINED

from
  "Subroutine 'db.pkg.f1' ..."

to a more clear:
  "FUNCTION `db.pkg.f1` ..."
  "PROCEDURE `db.pkg.p1` ..."

This reverts commit c37b2087.

In c37b20887, when re-binlogging a GTID event on a replica,
it will overwrite the thread_id from the primary to be the
value of the slave applier (SQL thread or parallel worker).

This should be the value of the original thread_id on the
master connection though, to both help track temporary
tables, and be consistent with Query_log_event.

Reverting the commit to re-target 11.5, so we can re-test
with the corrected thread_id.

When displaying the ER_SP_DOES_NOT_EXIST error, use
Sp_handler::type_lex_cstring() to the the underlying
object type:
- PROCEDURE
- FUNCTION
- PACKAGE
- PACKAGE BODY
instead of hard-coded "FUNCTION or PROCEDURE".