Additionally fixes the bugs uncovered in:
  - `MDEV-28332: Alter on temporary table causes ER_TABLE_EXISTS_ERROR note`
    Since there is no `warning` issued by shadowing of base table, this MDEV
    is irrelevant. Keeping for review purposes and for future development
    in case shadowing is going to be implemented
  - `MDEV-28334: SHOW TABLE STATUS shows all temporary tables ignoring database and conditions`
  - `MDEV-28453: SHOW commands are inconsistent for temporary tables`

Reviewed by: <monty@mariadb.org>,
             <vicentiu@mariadb.org>

Reviewer: <vicentiu@mariadb.org>

This bug report was caused by implementation of the task MDEV-5816
(Stored programs: validation of stored program statements).
Changing metadata of a table that has a trigger on AFTER UPDATE or AFTER DELETE
resulted in unexpected output of the error ER_TRG_NO_SUCH_ROW_IN_TR.
It was caused by the fact that characteristics of the trigger dependent on
changed table's metadata wasn't set in a new lex object created on re-parsing
of a failing trigger statement.

To fix the bug the data member lex->trg_chistics.action_time and
lex->trg_chistics.event must be set into real values of the trigger
whose statement being re-parsed.

AES_ENCRYPT(str, key, [, iv [, mode ]])
AES_DECRYPT(str, key, [, iv [, mode ]])

mode is aes-{128,192,256}-{ecb,cbc,ctr} e.g. "aes-128-cbc".

and a @@block_encryption_mode variable for the default value of mode

change in behavior: AES_ENCRYPT(str, key) can no longer
be used in persistent virtual columns (and alike)

apparently it works now just fine, tests don't fail anymore

in particular, this fixes a compiler warning:
sql/sql_lex.h:406:19: warning: ‘sp_data_access_name’ defined but not used [-Wunused-variable]

- Introduce the option :autoshrink attribute to be
added to innodb_data_file_path variable to allow
the shrinking of system tablespace during startup process.

Steps for shrinking the system tablespace:

1) Find the last used extent in system tablespace
by iterating through the BITMAP in extent descriptor pages

2) If the last used extent is lesser than user specified size
then set desired target size to user specified size.

3) Store the page contents of "to be modified" extent
descriptor pages, latches the "to be modified"
extent descriptor pages and check for buffer pool
memory availability

4) Make checkpoint to flush all pages in buffer pool, so
that pages in flush list doesn't have to use doublewrite
buffer and disable doublewrite buffer during shrinking process

5) Update the FSP_SIZE and FSP_FREE_LIMIT in header page

6) Remove the "to be truncated" pages from FSP_FREE and
FSP_FREE_FRAG list

7) Reset the bitmap in the last descriptor pages for the
"to be truncated" pages.

8) In case of multiple files, calculate the truncated last
file size and do the truncation in last file

9) Check whether mini-transaction log size doesn't exceed
the minimum value of innodb_log_buffer_size which is 2MB.
In that case, replace the modified buffer pool pages with
the page old content.

11) Commit the mini-transaction for shrinking the tablespace
and enable/disable the doublewrite buffer depends on user
specified value.

recv_sys_t::apply(): Handle the truncation of system tablespace
only if the recovered tablespace size is lesser than actual
existing size.

The error message for user connections using insecure transport when secured transport is required is very uninformative and doesn't mention the requirement of secure
transport at all.

To make the error message more relevant, introduce a new error
'ER_SECURE_TRANSPORT_REQUIRED', copy of MySQL error message with the
error code 08004 (SQL-server rejected establishment SQL-connection).

Move the code of 'require_secure_transport' to be executed before
authentication verification, as it's not part of authentication but
rather verifying if connection should be allowed in the first place.

All new code of the whole pull request, including one or several files that
are either new files or modified ones, are contributed under the BSD-new license.
I am contributing on behalf of my employer Amazon Web Services, Inc.

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

The main problem is that at ever since
commit aaef2e1d removed the
function buf_wait_for_read(), it is not safe to invoke
buf_page_get_low() with RW_NO_LATCH, that is, only buffer-fixing
the page. If a page read (or decryption or decompression) is in
progress, there would be a race condition when executing consistency
checks, and a page would wrongly be flagged as corrupted.

Furthermore, if the page is actually corrupted and the initial
access to it was with RW_NO_LATCH (only buffer-fixing), the
page read handler would likely end up in an infinite loop in
buf_pool_t::corrupted_evict(). It is not safe to invoke
mtr_t::upgrade_buffer_fix() on a block on which a page latch
was not initially acquired in buf_page_get_low().

btr_block_reget(): Remove the constant parameter rw_latch=RW_X_LATCH.

btr_block_get(): Assert that RW_NO_LATCH is not being used,
and change the parameter type of rw_latch.

btr_pcur_move_to_next_page(), innobase_table_is_empty(): Adjust for the
parameter type change of btr_block_get().

btr_root_block_get(): If mode==RW_NO_LATCH, do not check the integrity of
the page, because it is not safe to do so.

btr_page_alloc_low(), btr_page_free(): If the root page latch is not
previously held by the mini-transaction, invoke btr_root_block_get()
again with the proper latching mode.

btr_latch_prev(): Helper function to safely acquire a latch on a
preceding sibling page while holding a latch on a B-tree page.
To avoid deadlocks, we must not wait for the latch while holding
a latch on the current page, because another thread may be waiting
for our page latch when moving to the next page from our preceding
sibling page. If s_lock_try() or x_lock_try() on the preceding page fails,
we must release the current page latch, and wait for the latch on the
preceding page as well as the current page, in that order.
Page splits or merges will be prevented by the parent page latch
that we are holding.

btr_cur_t::search_leaf(): Make use of btr_latch_prev().

btr_cur_t::open_leaf(): Make use of btr_latch_prev(). Do not invoke
mtr_t::upgrade_buffer_fix() (when latch_mode == BTR_MODIFY_TREE),
because we will already have acquired all page latches upfront.

btr_cur_t::pessimistic_search_leaf(): Do acquire an exclusive index latch
before accessing the page. Make use of btr_latch_prev().

Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

row_ins_sec_index_entry_low(): Correct a condition that was
inadvertently inverted
in commit 89ec4b53 (MDEV-29603).

We are not supposed to buffer INSERT operations into unique indexes,
because duplicate key values would not be checked for. It is only
allowed when using unique_checks=0, and in that case the user is
supposed to guarantee that there are no duplicates.

* invoke parent's cleanup()
* don't reinit memroot, if already inited (causes memory leak)

also move free_root() from destructor to cleanup() to not accumulate
allocations from prepare and multiple executes

The idea is to have simple functions that the user can combine to produce
the exact result one wants, whether the user wants JSON object that has
common keys with another JSON object, or same key/value pair etc. So
making simpler function helps here.

We accomplish this by making three separate functions.
1) JSON_OBJECT_FILTER_KEYS(Obj, Arr_keys):
Put keys ( which are basically strings ) in hash, go over the object and
get key one by one. If the key is present in the hash,
add the key-value pair to result.

2) JSON_OBJECT_TO_ARRAY(Obj) : Create a string variable, Go over the json
object, and add each key value pair as an array into the result.

3) JSON_ARRAY_INTERSECT(arr1, arr2) :
Go over one of the json and add each item of the array
in hash (after normalizing each item). Go over the second array,
search the normalized item one by one in the hash. If item is found,
add it to the result.

Implementation Idea: Holyfoot ( Alexey Botchkov)
Author: tanruixiang and Rucha Deodhar

objects

Idea behind implementation:
We get the json object specified by the json path. Then, transform it into
key-value pairs by going over the json. Get each key-value pair
one-by-one and return the result.

For the sake of readable diffs this change is done separately from the
previous commit and it changes all man page headers to:

- Use correct syntax \fB and \fR to make titles bold instead of previous
  capitalized version that had no effect
- Omit unnecessary \&. syntax, works fine without it
- Bump version to 10.11

Ideally the version would automatically be inherited from the
build/release, but for now at least update it manually to match what is
in the VERSION file of this release branch.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.

Add \FR on lines that start with \FB as it is likely useful, but not
stricly necessary. However if there is just \ the line won't render at
all as happened in example below from 'man mysql-test-test.pl.1':

  AUTHOR
         MariaDB Foundation (http://www.mariadb.org/).
                                                 MYSQL-TEST-RUN()
  ->
  AUTHOR
         MariaDB Foundation (http://www.mariadb.org/).
  MariaDB 10.11        15 May 2020              MYSQL-TEST-RUN(1)

Also use full command name instead of truncated ones.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.

The man pages has several places where \' is intended to escape the single
quote and just print '. However, in reality this will print an ` (acute
accent). Fix this by removing excess backslashes. Also remove some excess
& in front of dots, such escaping is also unnecessary.

Also join several lines to avoid lone words in quotes to be read by groff
as macros when they are just text.

In a table (t{}) the content must be on new lines separately, so groff
will try to read them as macros too, so use \(aq that renders to '
(single quote).

This error was originally detected by Lintian:
https://lintian.debian.org/tags/acute-accent-in-manual-page

Example of man page sections before and after:

  Make a backup of each table´s data file using the name tbl_name.OLD.
  ->
  Make a backup of each table's data file using the name tbl_name.OLD.

  A typical debug_options string is ´d:t:o,file_name´.
  The default is ´d:t:o,/tmp/my_print_defaults.trace´.
  ->
  A typical debug_options string is 'd:t:o,file_name'.
  The default is 'd:t:o,/tmp/my_print_defaults.trace'.

  shell> mysql --xml -uroot -e "SHOW VARIABLES LIKE ´version%´"
  <?xml version="1.0"?>
  <resultset statement="SHOW VARIABLES LIKE ´version%´" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  ->
  shell> mysql --xml -uroot -e "SHOW VARIABLES LIKE 'version%'"
  <?xml version="1.0"?>
  <resultset statement="SHOW VARIABLES LIKE 'version%'" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

This command was used to validate no man pages have syntax errors:

  for x in *.?
  do
    echo "##### $x #####"
    LC_ALL=C.UTF-8 MANROFFSEQ='' MANWIDTH=80 man --warnings -E UTF-8 -l -Tutf8 -Z $x >/dev/null
  done

The man pages are not perfect after this, there are still a lot of
suboptimal syntax, but this helps towards better man pages.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.

50-mariadb_safe.cnf was only ever intended to be a reference file.
With mariadbd-safe falling into disuse in Debian since systemd became
a service it was assumed this files wasn't used except on a few old
legacy init systems. Its assumed these users will have a configuration
already set.

It is however read by the mariabackup galera-sst script. This forks
off the logger process to write the output to /dev/log, which on a
systemd service the journalctl is reading anyway.

The real harm is on containers where there is now a MariaDB operator
that runs Galera based containers. In containers there is no /dev/log,
so information gets lost.

It would be much more direct just to go straight to stdout/error
like what would happen without this configuration.

Rather than risk the galera-sst scripts moving to the [mariadbd-safe]
group name introduced in a1211a4e, we
remove the file for cleanness.

ANALYZE FORMAT=JSON output now includes table.r_engine_stats which
has the engine statistics. Only non-zero members are printed.

Internally: EXPLAIN data structures Explain_table_acccess and
Explain_update now have handler* handler_for_stats pointer.
It is used to read statistics from handler_for_stats->handler_stats.

The following applies only to 10.9+, backport doesn't use it:

Explain data structures exist after the tables are closed. We avoid
walking invalid pointers using this:
- SQL layer calls Explain_query::notify_tables_are_closed() before
  closing tables.
- After that call, printing of JSON output is disabled. Non-JSON output
  can be printed but we don't access handler_for_stats when doing that.

MDEV-31661: Assertion `thd->lex == sp_instr_lex' failed in LEX* sp_lex_instr::parse_expr(THD*, sp_head*, LEX*)

This is the follow-up patch for the task MDEV-5816 that fixes assert failure
that happened after recompilation of a stored routine containing a cursor
on its second execution.

The reason of assertion hit is that a state of the SP instruction sp_instr_cpush
wasn't reset after its SQL statement re-compiled.

To fix the issue the virtual method sp_lex_instr::on_after_expr_parsing
is overridden in the derived class sp_instr_cpush. Implementation of this method
does resetting of the data member sp_instr_cpush::m_metadata_changed

Additionally, implementation of the method
 sp_instr_set_trigger_field::on_after_expr_parsing
has been slightly modified to set the data member
 sp_instr_set_trigger_field::value
just before successful return. This data member is used to check whether this
SP instruction is still valid or should be re-compiled.
Resetting this data member before an instance of the class Item_trigger_field
be successfully allocated theoretically could lead to clearing of instruction's
state despite the fact that memory allocation was failed.

view creation (that mysqltest automatically does in --view protocol)
can cause sp cache invalidation, and that will cause the test to fail
(because the test checks that invalidation did not happen).

disable view protocol in the parts of the test where this is the case

Author: Sergei Golubchik

The follow-up patch to check in mtr tests that recompilation of
a SP's instruction doesn't lead to eviction of SP from sp_cache.

This patch adds the debug keyword 'check_sp_cache_not_invalidated'
checked in sp_cache_flush_obsolete. In case this debug keyword
is set the macros DBUG_SUICIDE() called to cause test failure.

The function sp_cache_flush_obsolete() is called on opening
a stored routine. So setting this keyword before second execution
of some stored routine that supposed to cause recompilation of
SP's statement will guarantee that this stored routine not evicted
from sp_cache.

Suggested approach has one limitation - the statement
 CREATE/ALTER/DROP VIEW
forces invalidation of the whole sp_cache (by invoking the function
 sp_cache_invalidate()).
So, for those tests (actually, there are very small number of such tests)
that create/alter/drop a view before the second execution of some stored
routine, the debug keyword 'check_sp_cache_not_invalidated' isn't set.

The proposal to add some way a check that a stored routine is not force out
from sp_cache on re-parsing a failing statement of a stored routine was
done during reiew, that is the reason the proposed change has been formatted
as a separate patch.

Added mtr tests for MDEV-5816

Fix of existing mtr tests.

This patch fixes the issue with missing warnings generated on re-parsing
a failing SP instruction's statement. That is, any warning generated
on re-parsing a statement was discarded after a SP instruction's statement
has been successfully re-parsed.

The reason for discarding warnings after re-parsing is that the method
  THD::set_query_id()
called every time when a failing SP instruction re-parsed.
In result,
  Warning_info::m_warn_id != thd->query_id
and when the method
  Diagnostics_area::opt_clear_warning_info(thd->query_id)
is invoked from sp_head::execute all warnings accumulated during
execution of the current SP instruction is cleared.

So, to fix the issue invokes the method THD::set_query_id()
once per SP instruction, on its first execution. Re-parsing of failing
SP instruction and following run of it don't invoke the method
THD::set_query_id().

This patch is the second part of implementation for cursor's statement
re-parsing. The patch does the following changes:
  - on re-parsing a failed SP instruction that does need to get access
    to LEX a new lex is instantiated for every SP instruction except
    cursor relating SP instructions.
  - items created on re-parsing a cursor relating statement are moved
    to the free_list of sp_lex_cursor.

Added re-parsing of a failing cursor body. Re-parsing of a failing
SP statement is implemented by the method validate_lex_and_exec_core(),
therefore invocation of the method reset_lex_and_exec_core() inside
  sp_lex_keeper::cursor_reset_lex_and_exec_core
was replaced by the method validate_lex_and_exec_core().

Re-parsing of a failed SP statement is relied upon interface provided
by the class sp_lex_instr (the methods used for this goal are
is_invalid(), parse_expr(), invalidate(), get_query(), get_expr_query()).
To provide access to these methods on opening a cursor, the signature of
the method
  sp_lex_keeper::cursor_reset_lex_and_exec_core
was changed to accept a pointer to the class sp_lex_instr instead of
the class sp_instr, and the new method get_push_instr() was added
into the class sp_cursor. This method is to get access to an instance
of the class sp_instr_cpush on opening a cursor (on handling the statement
OPEN cursors_name).

Default implementation of this method just returns NULL pointer of
the type sp_instr_cpush. This method is overridden in the class
sp_instr_cpush with trivial implementation
  { return this; }

On handling the statement DECLARE CURSOR FOR the new instruction of
the type sp_instr_cpush is added into sp_head. The class sp_instr_cpush
holds a text of SELECT query referencing by a cursor declaration.
When a cursor is being opened (on handling the statement 'OPEN cur_name')
a pointer to sp_instr_cpush is returned by the method
  sp_cursor::get_push_instr()
and this pointer is passed to the method
  sp_lex_keeper::cursor_reset_lex_and_exec_core
in order to open a cursor and provide access to an interface required
for SP  statement re-parsing in case metadata changes took place.
Since real access to a lex object is required on handling instruction
sp_instr_cpush (an instance of this class is created during parsing of
cursor declaration statement), calling of the method sp_cursor::open
is moved from the method
  sp_instr_copen::exec_core
into the method
  sp_instr_cpush::exec_core.

Additionally, updated the methods get_query/get_expr_query in the classes
sp_instr_cpush, sp_instr_cursor_copy_struct in order to return correct text of
cursor's body taking into account that lexer treated the clause CURSOR FOR/
CURSOR IS as two different tokens following one after another. So, to return
a correct text of SELECT statement specified in CURSOR declaration statement,
the token FOR/IS should be skipped and text following it should be returned as
a text of cursors's query.

Introduced the new data member new_query_arena_is_set of the class sp_head.
This data member is used as a protection against double invocation of
the method restore_thd_mem_root that is called for restoration of the
current query arena. Previously, the data member sp_head::m_thd
is used for this goal but after support for re-compilation of a failed
stored routine statement has been added the data member sp_head::m_thd
can't be used for this goal. The reason is that on a statement re-compilation
after the method restore_thd_mem_root() is called the method
sp_head::add_instr() invoked to add a new instruction for just re-compiled
statement. The method sp_head::add_instr() de-references m_thd to access
the free_list data member. If m_thd was used as a guard against double
invocation it would result in a crash on dereferencing null pointer.

Fixed memory leakage taken place on execution of the statement
  SHOW CREATE PACKAGE `pkg_name`
The memory leak was caused by implementation of sp_compile()
where a memory root for a stored routine was allocated but
a pointer to the new memory root wasn't passed to sp_package::create
for subsequent forwarding to the constructor of sp_package.
Instead, another one memory root was allocated and the pointer to
the original memory root was missed.