This patch fixes the issue with missing warnings generated on re-parsing
a failing SP instruction's statement. That is, any warning generated
on re-parsing a statement was discarded after a SP instruction's statement
has been successfully re-parsed.

The reason for discarding warnings after re-parsing is that the method
  THD::set_query_id()
called every time when a failing SP instruction re-parsed.
In result,
  Warning_info::m_warn_id != thd->query_id
and when the method
  Diagnostics_area::opt_clear_warning_info(thd->query_id)
is invoked from sp_head::execute all warnings accumulated during
execution of the current SP instruction is cleared.

So, to fix the issue invokes the method THD::set_query_id()
once per SP instruction, on its first execution. Re-parsing of failing
SP instruction and following run of it don't invoke the method
THD::set_query_id().

This patch is the second part of implementation for cursor's statement
re-parsing. The patch does the following changes:
  - on re-parsing a failed SP instruction that does need to get access
    to LEX a new lex is instantiated for every SP instruction except
    cursor relating SP instructions.
  - items created on re-parsing a cursor relating statement are moved
    to the free_list of sp_lex_cursor.

Added re-parsing of a failing cursor body. Re-parsing of a failing
SP statement is implemented by the method validate_lex_and_exec_core(),
therefore invocation of the method reset_lex_and_exec_core() inside
  sp_lex_keeper::cursor_reset_lex_and_exec_core
was replaced by the method validate_lex_and_exec_core().

Re-parsing of a failed SP statement is relied upon interface provided
by the class sp_lex_instr (the methods used for this goal are
is_invalid(), parse_expr(), invalidate(), get_query(), get_expr_query()).
To provide access to these methods on opening a cursor, the signature of
the method
  sp_lex_keeper::cursor_reset_lex_and_exec_core
was changed to accept a pointer to the class sp_lex_instr instead of
the class sp_instr, and the new method get_push_instr() was added
into the class sp_cursor. This method is to get access to an instance
of the class sp_instr_cpush on opening a cursor (on handling the statement
OPEN cursors_name).

Default implementation of this method just returns NULL pointer of
the type sp_instr_cpush. This method is overridden in the class
sp_instr_cpush with trivial implementation
  { return this; }

On handling the statement DECLARE CURSOR FOR the new instruction of
the type sp_instr_cpush is added into sp_head. The class sp_instr_cpush
holds a text of SELECT query referencing by a cursor declaration.
When a cursor is being opened (on handling the statement 'OPEN cur_name')
a pointer to sp_instr_cpush is returned by the method
  sp_cursor::get_push_instr()
and this pointer is passed to the method
  sp_lex_keeper::cursor_reset_lex_and_exec_core
in order to open a cursor and provide access to an interface required
for SP  statement re-parsing in case metadata changes took place.
Since real access to a lex object is required on handling instruction
sp_instr_cpush (an instance of this class is created during parsing of
cursor declaration statement), calling of the method sp_cursor::open
is moved from the method
  sp_instr_copen::exec_core
into the method
  sp_instr_cpush::exec_core.

Additionally, updated the methods get_query/get_expr_query in the classes
sp_instr_cpush, sp_instr_cursor_copy_struct in order to return correct text of
cursor's body taking into account that lexer treated the clause CURSOR FOR/
CURSOR IS as two different tokens following one after another. So, to return
a correct text of SELECT statement specified in CURSOR declaration statement,
the token FOR/IS should be skipped and text following it should be returned as
a text of cursors's query.

Introduced the new data member new_query_arena_is_set of the class sp_head.
This data member is used as a protection against double invocation of
the method restore_thd_mem_root that is called for restoration of the
current query arena. Previously, the data member sp_head::m_thd
is used for this goal but after support for re-compilation of a failed
stored routine statement has been added the data member sp_head::m_thd
can't be used for this goal. The reason is that on a statement re-compilation
after the method restore_thd_mem_root() is called the method
sp_head::add_instr() invoked to add a new instruction for just re-compiled
statement. The method sp_head::add_instr() de-references m_thd to access
the free_list data member. If m_thd was used as a guard against double
invocation it would result in a crash on dereferencing null pointer.

Fixed memory leakage taken place on execution of the statement
  SHOW CREATE PACKAGE `pkg_name`
The memory leak was caused by implementation of sp_compile()
where a memory root for a stored routine was allocated but
a pointer to the new memory root wasn't passed to sp_package::create
for subsequent forwarding to the constructor of sp_package.
Instead, another one memory root was allocated and the pointer to
the original memory root was missed.

Re-designed a way by that Item_trigger_field objects are arranged in memory.

Item_trigger_field objects created on parsing a trigger's statement
is now stored in a per statement list. All lists of Item_trigger_field
objects created on parsing the whole trigger's body are organized
in the structure "list of lists". So, use binary cycle to iterate every
Item_trigger_field object created on parsing a trigger body.

To organize the data structure 'list of lists' the new data member
  Item_trigger_field::next_trig_field_list
is introduced that links lists in this hierarchy structure.

This re-design is performed in order to avoid refences to already
deleted items on re-compilation of failed trigger's statememt.
Referencing to already deleted items could take place on re-parsing
a trigger's statement since every Item created for a statement
being re-parsed is deleted before the statement be re-parsed,
but deleted items are still referenced from sp_head. So, to avoid
access to dangling references a per statement list of Item_trigger_field
objects are cleared right after the current SP statement be cleaned up
and before re-parsing be started.

Added re-parsing of failed statements inside a stored routine.

General idea of the patch is to install an instance of the class
Reprepare_observer before executing a next SP instruction and
re-parse a statement of this SP instruction in case of
its execution failure.

To implement the described approach the class sp_lex_keeper
has been extended with the method validate_lex_and_exec_core()
that is just a wrapper around the method reset_lex_and_exec_core()
with additional setting/resetting an instance of the class
Reprepare_observer on each iteration of SP instruction
execution.

If reset_lex_and_exec_core() returns error and an instance
of the class Reprepare_observer is installed before running
a SP instruction then a number of attempts to re-run the SP
instruction is checked against a max. limit and in case it doesn't
reach the limit a statement for the failed SP instruction is re-parsed.

Re-parsing of a statement for the failed SP instruction is implemented
by the new method sp_le_inst::parse_expr() that prepends
a SP instruction's statement with the clause 'SELECT' and parse it.
Own SP instruction MEM_ROOT and a separate free_list is used for
parsing of a SP statement. On successful re-parsing of SP instruction's
statement the virtual methods adjust_sql_command() and
on_after_expr_parsing() of the class sp_lex_instr is called
to update the SP instruction state with a new data created
on parsing the statement.

Few words about reason for prepending a SP instruction's statement
with the clause 'SELECT' - this is required step to produce a valid
SQL statement, since for some SP instructions the instructions statement
is not a valid SQL statement. Wrapping such text into 'SELECT ( )'
produces a correct operator from SQL syntax point of view.

For those SP instructions that need to get access to ia LEX object
on execution, added storing of their original sql expressions inside
classes derived from the class sp_lex_instr.

A stored sql expression is returned by the abstract method
  sp_lex_instr::get_expr_query
redefined in derived classes.

Since an expression constituting a SP instruction can be invalid
SQL statement in general case (not parseable statement), the virtual
method sp_lex_instr::get_query() is introduced to return a valid string
for a statement that corresponds to the given instruction.

Additionally, introduced the rule remember_start_opt in the grammar.
The new rule intended to get correct position of a current
token taking into attention the fact whether lookahead was done or not.

This is the prerequisite patch introducing the class sp_lex_instr
that encapsulates access to an instance of the class sp_lex_keeper.
Every SP instruction that need to get access to a LEX object on its
processing should inherit this class and implement two abstract methods:
  is_invalid(),
  invalidate().

These methods will be used in subsequent patches to implement recompilation of
SP instructions on failure.

Currently, the following instructions are derived from the class sp_lex_instr:
  sp_instr_stmt,
  sp_instr_set,
  sp_instr_set_trigger_field,
  sp_instr_jump_if_not,
  sp_instr_freturn,
  sp_instr_cpush,
  sp_instr_cursor_copy_struct,
  sp_instr_set_case_expr

Additionally, this patch converts the class sp_instr_opt_meta
to a base abstract class (that is, not inherited from the class
sp_instr). Since this class originally was designed to provide a way
for opimizer to update a destination address for jump SP-instructions,
the only useful method at the interface of this class is set_destination
and therefore inheritance from the class sp_instr is meaningless.

Every jump SP instruction now must be inherited directly from
the class sp_instr_opt_meta and additionally from either the class
sp_lex_instr or sp_instr depending on whether this SP instruction
need to get access to a LEX object or not.

Moreover, the class sp_cursor doesn't own a data member of
the class sp_lex_keeper any more. Instead, the virtual method
get_lex_keeper() has been added to the class sp_cursor() that
returns nullptr and this method is overridden in the derived class
sp_instr_cpush to provide a pointer to a real instance of
the class sp_lex_keeper. Doing this way we exclude duplication
of a data member of the type sp_lex_keeper at the class sp_instr_cpush
since it is derived both from sp_lex_instr and sp_cursor, and sp_lex_instr
already encapsulates a data member of the class sp_lex_keeper.

This is the prerequisite patch to change a signature of the virtual
method opt_move() in the base class sp_instr and classes derived from it.
The parameterized type of the instuctions list returned in the second
argument is changed from sp_instr to sp_instr_opt_meta since only
jump instructions are placed in this list on returning from
the method call.

This is the prerequisite patch to move the data member
LEX::trg_table_fields to the class sp_head and rename it as
m_trg_table_fields.

This data member is used for handling OLD/NEW pseudo-rows inside
a trigger body and in order to be able to re-parse a trigger body
the data member must be moved from the struct LEX to the class sp_head.

This is the prerequisite patch to remove the data member
sp_head::m_trg_table_fields and the method is_fields_updated_in_trigger
that used it but is not called anywhere in the source code.

The commit 5f1f2fc0 introduced the
data member sp_head::m_trg_table_fields and the method
Table_triggers_list::is_fields_updated_in_trigger() that used this data member.

The method Table_triggers_list::is_fields_updated_in_trigger() was invoked
by the method partition_info::can_prune_insert() also introduced by
the same commit 5f1f2fc0

The method partition_info::can_prune_insert() is not called anywhere
in the code and later these  methods were removed from the source code
but the data member sp_head::m_trg_table_fields wasn't.
So, remove the data member sp_head::m_trg_table_fields and declaration of
the method is_fields_updated_in_trigger() for purpose of code cleaning up.

This is the prerequisite patch to move the sp_instr class and classes derived
from it into the files sp_instr.cc/sp_instr.h. The classes sp_lex_cursor and
sp_lex_keeper are also moved to the files files sp_instr.cc/sp_instr.h.

Additionally,
  * all occurrences of macroses NULL, FALSE, TRUE are replaced
    with the corresponding C++ keywords nullptr, false, true.
  * the keyword 'override' is added in and the keyword 'virtual' is removed
    from signatures of every virtual method implemented in classes derived
    from the base class sp_instr.
  * the keyword 'final' is added into declaration of the class sp_lex_keeper
    since this class shouldn't have a derived class by design.
  * the function cmp_rqp_locations is made static since it is not called
    outside the file sp_instr.cc.
  * the function subst_spvars() is moved into the file sp_instr.cc since this
    function used only by the method sp_instr_stmt::execute

This is the prerequisite patch to make interface of the class
Reprepare_observer more similar to the one used by MySQL.

This patch adds the method can_retry() to the class Reprepare_observer
that returns true in case max. number of attempts to re-run a failing
statement is not yet reached. To control the number of re-run attempts
already done the data member m_attempt has been introduced. Doing this way,
we encapsulate activity with incrementing a counter on every statement
run and checking whether it reaches a limit or not inside implementation
of the class Reprepare_observer instead duplicating such boiler plate code
in every place where controlling for reaching a limit of max. number attempts
for re-running failed statement is required.

Instructions and a script to help speed up language translations
for MariaDB are provided. The README.md provides instructions of
a recommended workflow that involves:

- Extracting the English language entries in the file
errmsg-utf8.txt
- Using online tools to do automatic translation.
- Proof-reading the automatic translations.
- Running the script insert_translations_into_errmsg.py that
will take the original errmsg-utf8.txt file, the extracted
english translations file and the new language translations
file and generate a new file errmsg-utf8-with-new-language.txt
that is a copy of errmsg-utf8.txt but with the new language
entries inserted into it at the correct positions.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.

This patch adds a way to override default collations
(or "character set collations") for desired character sets.

The SQL standard says:
> Each collation known in an SQL-environment is applicable to one
> or more character sets, and for each character set, one or more
> collations are applicable to it, one of which is associated with
> it as its character set collation.

In MariaDB, character set collations has been hard-coded so far,
e.g. utf8mb4_general_ci has been a hard-coded character set collation
for utf8mb4.

This patch allows to override (globally per server, or per session)
character set collations, so for example, uca1400_ai_ci can be set as a
character set collation for Unicode character sets
(instead of compiled xxx_general_ci).

The array of overridden character set collations is stored in a new
(session and global) system variable @@character_set_collations and
can be set as a comma separated list of charset=collation pairs, e.g.:

SET @@character_set_collations='utf8mb3=uca1400_ai_ci,utf8mb4=uca1400_ai_ci';

The variable is empty by default, which mean use the hard-coded
character set collations (e.g. utf8mb4_general_ci for utf8mb4).

The variable can also be set globally by passing to the server startup command
line, and/or in my.cnf.

The `--vertical` option does not appear to behave as documented and
research shows it never appears to have behaved as intended.

As this appears to be a completely unused feature, let's remove it.

Performs an upgrade of mariadb from an earlier version to the rpms built
in CI. Then checks whether log contains evidence of upgrade in the form
of "Needs upgrade" or "Table rebuild required". Designed to check minor
version upgrades which should not trigger rebuilds.

The test is written in bash script so it can be executed from other CI
systems.

$ test_upgrade.sh source_version target_version
$ test_upgrade.sh source_version # defaults to rpm/
$ test_upgrade.sh source_version --rpm-dir <directory>

Binaries must be created with performance schema enabled, or
mysql_upgrade complains about missing tables.

Upgrade testing is parallelized with a "matrix" of source versions.
Others can be introduced later.

This was partially designed to catch issues like that seen in
https://jira.mariadb.org/browse/MDEV-28727 where a minor version upgrade
(e.g. 10.4.8 -> 10.4.26) triggered a system table rebuild.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.

Allow ALTER TABLE ... IMPORT TABLESPACE without creating the table
followed by discarding the tablespace.

That is, assuming we want to import table t1 to t2, instead of

CREATE TABLE t2 LIKE t1;
ALTER TABLE t2 DISCARD TABLESPACE;
FLUSH TABLES t1 FOR EXPORT;
--copy_file $MYSQLD_DATADIR/test/t1.cfg $MYSQLD_DATADIR/test/t2.cfg
--copy_file $MYSQLD_DATADIR/test/t1.ibd $MYSQLD_DATADIR/test/t2.ibd
UNLOCK TABLES;
ALTER TABLE t2 IMPORT TABLESPACE;

We can simply do

FLUSH TABLES t1 FOR EXPORT;
--copy_file $MYSQLD_DATADIR/test/t1.cfg $MYSQLD_DATADIR/test/t2.cfg
--copy_file $MYSQLD_DATADIR/test/t1.frm $MYSQLD_DATADIR/test/t2.frm
--copy_file $MYSQLD_DATADIR/test/t1.ibd $MYSQLD_DATADIR/test/t2.ibd
UNLOCK TABLES;
ALTER TABLE t2 IMPORT TABLESPACE;

We achieve this by creating a "stub" table in the second scenario
while opening the table, where t2 does not exist but needs to import
from t1. The "stub" table is similar to a table that is created but
then instructed to discard its tablespace.

We include tests with various row formats, encryption, with indexes
and auto-increment.

Kenyan Swahili is enabled for error messages as well as datetime.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.

The error code is non-deterministic, presumably due to some race
condition from the SLEEP statement above. The correct error should be
12701 ER_SPIDER_REMOTE_SERVER_GONE_AWAY_NUM as it is the last failure.
Nevertheless, this contrived test is needed to cover the error
reporting when setting lock wait timeout, until we find a better test
case and/or fixing the non-deterministic error reporting (MDEV-31586)

log_t::write_checkpoint(), log_t::resize_start(): Invoke buf_flush_ahead()
with buf_pool.get_oldest_modification(0)+1 so that another checkpoint will
be invoked, to complete the log resizing.

Tested by: Oleg Smirnov (on Microsoft Windows, where this hung most often)

ha_innobase::delete_table(): Also on DROP SEQUENCE, do try to drop any
persistent statistics. They should really not be created for
SEQUENCE objects (which internally are 1-row no-rollback tables),
but that is how happened to always work.

i_s_innodb_buffer_page_get_info(): Correct a condition.
After crash recovery, there may be some buffer pool pages in FREED state,
containing garbage (invalid data page contents). Let us ignore such pages
in the INFORMATION_SCHEMA output.

The test innodb.innodb_defragment_fill_factor will be removed, because
the queries that it is invoking on information_schema.innodb_buffer_page
would start to fail. The defragmentation feature was removed in
commit 7ca89af6 in MariaDB Server 11.1.

Tested by: Matthias Leich

Allocate a temporary buffer instead of using the same buffer in some
cases, and add assertions to verify the buffers do not overlap. See [1]
for reasonsing.

[1] https://github.com/MariaDB/server/pull/2438#discussion_r1137403645Signed-off-by: Trevor Gross <tmgross@umich.edu>

This patch ensures that all direct and indirect calls to
encryption_crypt provide a `dlen` value correctly initialized to the
destination buffer length, allowing encryption plugins to verify
available space. It also adds assertions to verify related invariants.
Signed-off-by: Trevor Gross <tmgross@umich.edu>

* removed universal suppression of everything "Event Schedule" everywhere
* added suppressions in tests as needed
* moved events test to the events suite
* renamed -master.opt -> .opt
* added standard test header
* verified in the test that the error, indeed, was written into the log
* removed useless suppressions
* removed ER_EVENTS_NO_ACL, replaced with ER_OPTION_PREVENTS_STATEMENT
* fixed error message to say exactly what option disabled event scheduler
  instead of "this or that or that, you figure it out"
* also fixed old message for SET event_scheduler=
  (it was also non-translatable)
* changed to use sql_print_error() when an error is not sent to the user
* removed duplicate hard-coded error message

btr_search_hash_table_validate(), btr_search_validate(): Add the
parameter THD for checking if the statement has been killed.
Any non-QUICK CHECK TABLE will validate the entire adaptive hash index
for all InnoDB tables, which may be extremely slow when running
multiple concurrent CHECK TABLE.

During the upgrade procedure on Windows mysqld.exe is started with
the named pipe connection protocol. mysqladmin.exe then pings the
server to check if is up and running. Command line looks like:
   mysqladmin.exe --protocol=pipe --socket=mysql_upgrade_service_xxx ping
But the "socket" parameter resets the "protocol" which was previously
initialized with the "pipe" value, setting it to "socket".
As a result, connection cannot be established and the upgrade
procedure fails.
"socket" in Windows is used to pass the name of the pipe so resetting
the protocol is not valid in this case.

This commit fixes resetting of the "protocol" parameter with "socket"
parameter in the case when protocol has been previously initialized
to "pipe" value

Fake_select_lex->join was prepared at the unit execution stage so
the validation of fake_select_lex before the unit pushdown
was incomplete. That caused pushing down of statements having
an incorrect ORDER BY clause.
This commit moves preparation of the fake_select_lex->join to the unit
prepare() method, before initializing of the pushdown handler,
so incorrect clauses error out before being pushed down

During the upgrade procedure on Windows mysqld.exe is started with
the named pipe connection protocol. mysqladmin.exe then pings the
server to check if is up and running. Command line looks like:
   mysqladmin.exe --protocol=pipe --socket=mysql_upgrade_service_xxx ping
But the "socket" parameter resets the "protocol" which was previously
initialized with the "pipe" value, setting it to "socket".
As a result, connection cannot be established and the upgrade
procedure fails.
"socket" in Windows is used to pass the name of the pipe so resetting
the protocol is not valid in this case.

This commit fixes resetting of the "protocol" parameter with "socket"
parameter in the case when protocol has been previously initialized
to "pipe" value

Set the histogram_type to be DOUBLE_PREC_HB as
it originally was.