MDEV-22125 : galera.galera_drop_multi MTR failed: InnoDB: MySQL is trying to drop database `fts`.`` though there are still open handles

MDEV-22140 galera.galera_drop_database MTR failed: InnoDB: MySQL is trying to drop database `fts`.`` though there are still open handles

Add wait conditions to wait that all operations are done in both
nodes.

MDEV-20928 : Galera test failure on galera.galera_var_innodb_disallow_writes: Result length mismatch

Add wait_conditions to force desired execution.

Item_func_json_extract did not implement val_decimal(),
so CAST(JSON_EXTRACT('{"x":true}', '$.x') AS DECIMAL) erroneously
returned 0 with a warning because of convertion from the string "true"
to decimal.

Implementing val_decimal(), so boolean values are correctly handled.

* --rr-arg instead of --rr_option
* Bootstrap saved to rr.bootstrap
* Replication slaves are saved to rr.N dirs
* Perl coding fixes

This feature adds the support for rr in mtr. These 2 options are added
--rr         run   the mysqld in rr record mode
--rr_option= run the rr with custom record option, for multiple
             options use --rr_option= for each option.
             For example
              ./mtr main.view --rr_option=-h --rr_option=-u --rr_option=-c=23
--boot-rr    run the mysqld performing bootstrap in rr record mode

Recording are stored in mysql-test/var/rr folder.
To run recording please run
rr replay var/rr/mysql-X

Limitations
Restart will create a new recording.
Repeat will work on same recording , So might be harder to debug.
If test create the multiple instance of mariadb all will be stored in var/rr

Problem:
========
- InnoDB clears the fts resource when last FTS index is being dropped
if the table has user defined FTS_DOC_ID. While creating the new fts
index, InnoDB expects to have FTS resources.

Fix:
===
fts_drop_index(): Removed the fts resource clear.

fts_clear_all(): Clear the fts resource when there are no new fts
index to be added.

commit_cache_norebuild(), row_merge_drop_indexes():
Tries to call fts resource after removing associated fts index
from table object

copy_data_between_tables() sets to->s->default_fields to 0, as a part
of the code disabling ON UPDATE actions for all old fields
(so ON UPDATE is enable only for new fields during copying).
After the actual copying, copy_data_between_tables() did not restore
to->s->default_fields to the original value. As a result, the
TABLE_SHARE to->s was left in a wrong state after copy_data_between_tables()
and further open_table_from_share() using this TABLE_SHARE did not
populate TABLE::default_field, which further made
TABLE::evaluate_update_default_function() crash on access to NULL
pointer.

Fix:
Changing copy_data_between_tables() to restore to->s->default_fields
to its original value after the copying loop.

For the sake of completeness, call sync_clones in reset_stmt_params, too.

When a prepared statement parameter '?' is used in a CTE that is used
multiple times, the following happens:
- The CTE definition is re-parsed multiple times.
- There are multiple Item_param objects referring to the same "?" in
the original query.
- Prepared_statement::param has a pointer to the first of them, the
  others are "clones".
- When prepared statement parameter gets the value, it should be passed
  over to clones with param->sync_clones() call.

This call is made in insert_params(), etc. It was not made in
insert_params_with_log().

This would cause Item_param to not have any value which would confuse
the query optimizer.

Added the missing call.

The test case that was added for MDEV-21217
(commit b68f1d84)
should have only two possible outcomes for the locking SELECT statement:

(1) The statement is blocked, and the test will eventually fail
with a lock wait timeout. This is what I observed when the
code fix for MDEV-21217 was missing.

(2) The lock conflict will ensure that the statement will execute
after the rollback has completed, and an empty table will be observed.
This is the expected outcome with the recovery fix.

What occasionally happens (in some of our CI environments only, so far)
is that the locking SELECT will return all 1,000 rows of the table that
had been inserted by the transaction that was never supposed to be
committed. One possibility is that the transaction was unexpectedly
committed when the server was killed.

Let us disable the test until the reason of the failure has been
determined and addressed.

trx_roll_must_shutdown(): Correct the condition that detects
the start of shutdown.

Item_cache_datetime::decimals was always copied from example->decimals
without limiting to 6 (maximum possible fractional digits), so
val_str() later crashed on asserts inside my_time_to_str() and
my_datetime_to_str().

MDEV-22755 CREATE USER leads to indirect SIGABRT in __stack_chk_fail () from fill_schema_user_privileges + *** stack smashing detected *** (on optimized builds)

The code erroneously used buff[100] in a fiew places to make
a GRANTEE value in the form:
  'user'@'host'

Fix:
- Fixing the code to use (USER_HOST_BUFF_SIZE + 6) instead of 100.
- Adding a DBUG_ASSERT to make sure the buffer is enough
- Wrapping the code into a class Grantee_str, to reuse it easier in 4 places.

On large hard disks (> 2TB), the plugin won't function correctly, always
showing 2 TB of available space due to integer overflow. Upgrade table
fields to bigint to resolve this problem.

The last traces of the special InnoDB table names were removed in
commit 0af52734
but we forgot to remove the test case.

The real problem was that attempt to roll back cahnes after end of memory in QC was made incorrectly and lead to using uninitialized memory.
(bug has nothing to do with resize operation, it is just lack of resources erro processed incorrectly)

This reverts commit 44339123.

Replacing the slow loop in my_hash_sort_utf8mbX() to the fast
skip_trailing_spaces(), which consumes 8 bytes in one iteration,
and is around 8 times faster on long data.

Also, renaming:
- my_hash_sort_utf8() to my_hash_sort_utf8mb3()
- my_hash_sort_utf8_nopad() to my_hash_sort_utf8mb3_nopad()
to merge to 10.5 easier (automatically?).

innodb: dict_mem_table_add_col - compile warning fix argument 1 null where non-null expected (#1584)

cd /build-mariadb-server-10.5-mysql_release/storage/innobase && /usr/bin/powerpc64le-linux-gnu-g++  -DBTR_CUR_ADAPT -DBTR_CUR_HASH_ADAPT -DCOMPILER_HINTS -DDBUG_TRACE -DEMBEDDED_LIBRARY -DHAVE_BZIP2=1 -DHAVE_C99_INITIALIZERS -DHAVE_CONFIG_H -DHAVE_FALLOC_PUNCH_HOLE_AND_KEEP_SIZE=1 -DHAVE_IB_LINUX_FUTEX=1 -DHAVE_LZ4=1 -DHAVE_LZ4_COMPRESS_DEFAULT=1 -DHAVE_LZMA=1 -DHAVE_NANOSLEEP=1 -DHAVE_OPENSSL -DHAVE_SCHED_GETCPU=1 -DLINUX_NATIVE_AIO=1 -DMUTEX_EVENT -DWITH_INNODB_DISALLOW_WRITES -D_FILE_OFFSET_BITS=64 -Iwsrep-lib/include -Iwsrep-lib/wsrep-API/v26 -I/home/dan/build-mariadb-server-10.5-mysql_release/include -Istorage/innobase/include -Istorage/innobase/handler -Ilibbinlogevents/include -Itpool -Iinclude -Isql  -pie -fPIC -Wl,-z,relro,-z,now -fstack-protector --param=ssp-buffer-size=4 -Wconversion -Wno-sign-conversion -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized -D_FORTIFY_SOURCE=2 -DDBUG_OFF -Wall -Wextra -Wformat-security -Wno-format-truncation -Wno-init-self -Wno-nonnull-compare -Wno-unused-parameter -Woverloaded-virtual -Wnon-virtual-dtor -Wvla -Wwrite-strings   -DUNIV_LINUX -D_GNU_SOURCE=1 -fPIC -fvisibility=hidden -std=gnu++11 -o CMakeFiles/innobase_embedded.dir/dict/dict0load.cc.o -c storage/innobase/dict/dict0load.cc
storage/innobase/dict/dict0load.cc: In function ‘const char* dict_process_sys_columns_rec(mem_heap_t*, const rec_t*, dict_col_t*, table_id_t*, const char**, ulint*)’:
storage/innobase/dict/dict0load.cc:1653:26: warning: argument 1 null where non-null expected [-Wnonnull]
    dict_mem_table_add_col(table, heap, name, mtype,
    ~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~
             prtype, col_len);
             ~~~~~~~~~~~~~~~~
In file included from storage/innobase/include/dict0dict.h:32:0,
                 from storage/innobase/include/btr0pcur.h:30,
                 from storage/innobase/dict/dict0load.cc:31:
storage/innobase/include/dict0mem.h:323:1: note: in a call to function ‘void dict_mem_table_add_col(dict_table_t*, mem_heap_t*, const char*, ulint, ulint, ulint)’ declared here
 dict_mem_table_add_col(
 ^~~~~~~~~~~~~~~~~~~~~~

In case of SELECT without tables which returns either 0 or 1 rows,
JOIN::exec_inner() did not check if the flag representing SQL_CALC_FOUND_ROWS
is set or not and send_records was direclty assigned 0. So SELECT FOUND_ROWS()
was giving 0 in the output. Now it checks if the flag is set, if it is set
send_record=1 else 0. 1 is the number of rows that could have been sent
to the client if the SELECT query had SQL_CALC_FOUND_ROWS.
It is 0 when no rows were sent because the SELECT query did not have
SQL_CALC_FOUND_ROWS.

MDEV-22717: Conditional jump or move depends on uninitialised value(s) in find_uniq_filename(char*, unsigned long)

Fix:
===
Initialize 'number' variable to '0'.

When MDEV-22769 introduced srv_shutdown_state=SRV_SHUTDOWN_INITIATED in
commit efc70da5
we forgot to adjust a few checks for SRV_SHUTDOWN_NONE.

In the initial shutdown step, we are waiting for the background
DROP TABLE queue to be processed or discarded. At that time,
some background tasks (such as buffer pool resizing or dumping
or encryption key rotation) may be terminated, but others must
remain running normally.

srv_purge_coordinator_suspend(), srv_purge_coordinator_thread(),
srv_start_wait_for_purge_to_start(): Treat SRV_SHUTDOWN_NONE
and SRV_SHUTDOWN_INITIATED equally.

MDEV-22728: SIGFPE in Unique::get_cost_calc_buff_size from prepare_search_best_index_intersect on optimized builds

For low sort_buffer_size, in the cost calculation of using the Unique object the elements in the tree were evaluated to 0, make sure to have atleast 1 element in the Unique tree.

Also for the function Unique::get allocate memory for atleast MERGEBUFF2+1 keys.

This bug is the same as the bug MDEV-17024. The crashes caused by these
bugs were due to premature cleanups of the unit specifying recursive CTEs
that happened in some cases when there were several outer references the
same recursive CTE.
The problem of premature cleanups for recursive CTEs could be already
resolved by the correction in TABLE_LIST::set_as_with_table() introduced
in this patch. ALL other changes introduced by the patches for MDEV-17024
and MDEV-22748 guarantee that this clean-ups are performed as soon as
possible: when the select containing the last outer reference to a
recursive CTE is being cleaned up the specification of the recursive CTE
should be cleaned up as well.

page_zip_fields_decode(): Do not dereference index=NULL.
Instead, return NULL early. The only caller does not care
about the values of output parameters in that case.

This bug was introduced in MySQL 5.7.6 by
mysql/mysql-server@9eae0edb7a8e4004328e61157f5f3b39cebe1b2b
and in MariaDB 10.2.2 by
commit 2e814d47.

Thanks to my son for pointing this out after investigating
the output of a static analysis tool.

Compiler tells something about argument-dependent lookup. I do not
understand how that ADL works. But I know that such operators should
be free functions, instead of methods:
http://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#Ro-symmetric

Such syntax defines 'friend' free functions.

When processing a query with a recursive CTE a temporary table is used for
each recursive reference of the CTE. As any temporary table it uses its own
mem-root for table definition structures. Due to specifics of the current
implementation of ANALYZE stmt command this mem-root can be freed only at
the very of query processing. Such deallocation of mem-root memory happens
in close_thread_tables(). The function looks through the list of the tmp
tables rec_tables attached to the THD of the query and frees corresponding
mem-roots. If the query uses a stored function then such list is created
for each query of the function. When a new rec_list has to be created the
old one has to be saved and then restored at the proper moment.
The bug occurred because only one rec_list for the query containing CTE was
created. As a result close_thread_tables() freed tmp mem-roots used for
rec_tables prematurely destroying some data needed for the output produced
by the ANALYZE command.

The background drop table queue in InnoDB is a work-around for
cases where the SQL layer is requesting DDL on tables on which
transactional locks exist.

One such case are XA transactions. Our test case exploits the
fact that the recovery of XA PREPARE transactions will
only resurrect InnoDB table locks, but not MDL that should
block any concurrent DDL.

srv_shutdown_t: Introduce the srv_shutdown_state=SRV_SHUTDOWN_INITIATED
for the initial part of shutdown, to wait for the background drop
table queue to be emptied.

srv_shutdown_bg_undo_sources(): Assign
srv_shutdown_state=SRV_SHUTDOWN_INITIATED
before waiting for the background drop table queue to be emptied.

row_drop_tables_for_mysql_in_background(): On slow shutdown, if
no active transactions exist (excluding ones that are in
XA PREPARE state), skip any tables on which locks exist.

row_drop_table_for_mysql(): Do not unnecessarily attempt to
drop InnoDB persistent statistics for tables that have
already been added to the background drop table queue.

row_mysql_close(): Relax an assertion, and free all memory
even if innodb_force_recovery=2 would prevent the background
drop table queue from being emptied.

This race condition was introduced by
commit ad6171b9 (MDEV-22456).

In the observed case, two threads were executing
btr_search_drop_page_hash_index() on the same block,
to free a stale entry that was attached to a dropped index.
Both threads were holding an S latch on the block.

We must prevent the double-free of block->index by holding
block->lock in exclusive mode.

btr_search_guess_on_hash(): Do not invoke
btr_search_drop_page_hash_index(block) to get rid of
stale entries, because we are not necessarily holding
an exclusive block->lock here.

buf_defer_drop_ahi(): New function, to safely drop stale
entries in buf_page_mtr_lock(). We will skip the call to
btr_search_drop_page_hash_index(block) when only requesting
bufferfixing (no page latch), because in that case, we should
not be accessing the adaptive hash index, and we might get
a deadlock if we acquired the page latch.