MDEV-25830 optimizer_use_condition_selectivity=4 sometimes produces worse plan than optimizer_use_condition_selectivity=1

The issue was that calc_cond_selectivity_for_table prefered ranges with
many parts and when deciding on which selectivity to use.

Fixed by going through ranges according to the number of rows in the range.

This ensures that selectivity from ranges with few rows will be prefered
over ranges with many rows for indexes that uses the same columns.

Remove calls to wsrep_append_fk_parent_table() during REPAIR/OPTIMIZE
TABLE processing. It turns out that REPAIR or OPTIMIZE commands on
table t, do not acquire MDL locks on parent tables of t (as shown in
the included test).  Thus making wsrep_append_fk_parent_table()
unnecessary for OPTIMIZE and REPAIR.
This also fixes MDEV-24446 and reenables test galera.mysql-wsrep#198.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

Spider dereferences a freed select_lex and then results in SIGSEGV.

dict_index_t::reconstruct_fields(): add input validation by replacing some
assertions

handle_instant_metadata(): fix nullptr dereference

Add missing wsrep_sst_auth_free call.

There is no need to open or process temporary tables at
wsrep_append_fk_parent_table.

SET PASSWORD = PASSWORD('foo') would fail for pam plugin with

ERROR HY000: SET PASSWORD is ignored for users authenticating via pam plugin

but SET PASSWORD = 'foo' would not.

Now it will.

Fix wsrep.mysql_tzinfo_to_sql_symlink_skip test result.

This reverts commit b9730226.

Problem:

At some point, we made stored rountines fail at CREATE time
instead of execution time in case of this syntax:

   IF unknown_variable
     ...
   END IF

As a result, a trigger created before this change and contained an unknown
variable worked in a bad way after upgrade:
- It was displayed with an empty trigger name by SHOW CREATE TRIGGER
- It was displayed with an empty trigger name by INFORMATION_SCHEMA.TRIGGERS
- An attempt to DROP this trigger returned errors - nothing happened.
- DROP TABLE did not remove the .TRN file corresponding to this broken trigger.

Underlying code observations:

The old code assumed that the trigger name resides in the current lex:

  if(thd->lex->spname)
    m_trigger_name= &thd->lex->spname->m_name;

This is not always the case. Some SP statements (e.g. IF)
do the following in their beginning:

- create a separate local LEX
- set thd->lex to this new local LEX
- push the new local LEX to the stack in sp_head::m_lex

and the following at the end of the statement:

- pop the previous LEX from the stack sp_head::m_lex
- set thd->lex back to the popped value

So when the parse error happens inside e.g. IF statement, thd->lex->spname
is a NULL pointer, because thd->lex points to the local LEX (without SP name)
rather than the top level LEX (with SP name).

Fix:
- Adding a new method sp_head::find_spname_recursive()
  which walks inside the LEX stack sp_head::m_lex from
  the top (the newest, most local) to the bottom (the oldest),
  and finds the one which contains a non-zero spname pointer.

- Using the new method inside
  Deprecated_trigger_syntax_handler::handle_condition():
  First it still tests thd->lex->spname (like before this change),
  and uses it in case it is not empty.
  Otherwise (if thd->lex->spname is empty), it calls
  sp_head::find_spname_recursive() to find the LEX with a
  non-empty spname inside the LEX stack of the current sphead.

TRT opens statement transaction. Cleanup it in case of error.

The --skip-write-binary-log added to mysql_tzinfo_to_sql in
MDEV-18778 was only effective if galera was enabled on the server.
This is because it tied together three concepts under one option:
1. binary logging
2. wsrep replication, and
3. using innodb as a transitional table type.

Change 1: small change in help option to reflect this.

To solve the performance problem with Aria tables, LOCK TABLES WRITE
is used to eliminate the need to fdatasync until the UNLOCK TABLES.

If galera isn't enabled, then we also want to use the LOCK TABLE WRITE
mechanism.

The START TRANSACTION added in MDEV-23440 needed to be moved to
before LOCK TABLES otherwise it would cancel their effect.

TRUNCATE TABLE statements also need to be before the LOCK TABLES.

When changing back from InnoDB to Aria, include the ORDER BY that
was originally there in 6aaccbcb and matching the final ALTER
TABLE in the timezonedir branch.

Running: mariadb-tzinfo-to-sql --skip-write-binlog /usr/share/zoneinfo
now generates 16 Aria_transaction_log_syncs from 7053.

The Spider storage engine ignored the implicit grouping when
aggregation was converted to constant by the query optimizer.
As a result, the Spider SE returned rows more than expected.

To fix the problem, we notify the Spider SE of the existence of
the implicit grouping via Query::distinct.

MDEV-18848 : Galera: 10.4 node crashed with Assertion `client_state.transaction().active()` after altering SEQUENCE table's engine to myisam and back to innodb

We need to start Galera transaction for select NEXT VALUE FOR
sequence if it is not yet started. Note that ALTER is handled
as TOI and transaction is already started.

Galera replication of new DDL-case was missing

Fix test MDEV-20225.

If the optimizer decides to rewrites a NOT IN predicand of the form
  outer_expr IN (SELECT inner_col FROM ... WHERE subquery_where)
into the EXISTS subquery
  EXISTS (SELECT 1 FROM ... WHERE subquery_where AND
        (outer_expr=inner_col OR inner_col IS NULL))
then the pushed equality predicate outer_expr=inner_col can be used for
ref[or_null] access if inner_col is a reference to an indexed column.
In this case if there is a selective range condition over this column then
a Rowid filter may be employed coupled the with ref[or_null] access. The
filter is 'pushed' into the engine and in InnoDB currently it cannot be
used with index look-ups by primary key. The ref[or_null] access can be
used only when outer_expr is not NULL. Otherwise the original predicand
is evaluated to TRUE only if the result set returned by the query
 SELECT 1 FROM ... WHERE subquery_where
is empty. When performing this evaluation the executor switches to the
table scan by primary key. Before this patch the pushed filter still
remained marked as active and the engine tried to apply the filter. This
was incorrect and in InnoDB this attempt to use the filter led to an
assertion failure.

This patch fixes the problem by disabling usage of the filter when
outer_expr is evaluated to NULL.

Federated and Federatex cannot be used with ROR scans

Federated::position() and Federatex::position() is storing in 'ref' a
pointer into a local result set buffer. This means that one cannot
compare 'ref' from different handler instances to see if they point to the
same physical record.

This bug caused federated.federatedx to return wrong results when the
optimizer tried to use index_merge to resolve some queries.

Fixed by introducing table flag HA_NON_COMPARABLE_ROWID and using this
with the above handlers.

Todo:
- Fix multi_delete(), multi_update and read_records() to use primary key
  instead of 'ref' if case HA_NON_COMPARABLE_ROWID is set. The current
  code only works if we have only one range (like table scan) for the
  tables that will be updated in the second pass.
- Enable DBUG_ASSERT() in ha_federated::cmp_ref() and
  ha_federatedx::cmp_ref().

Fixes main.sp-no-valgrind for valgrind builds not done with BUILD scripts

MDEV-27184 Assertion `(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)' failed, Assertion `str.alloced_length() >= str.length() + data_len' failed

Spider crashes on a query that inserts some rows including float.
This is because Spider allocates a string of insufficient length.

This is the first part of the fixes for MDEV-24097. This commit
contains the fixes for instability when testing Galera and when
restarting nodes quickly:

1) Protection against a "stuck" old SST process during the execution
   of the new SST (after restarting the node) is now implemented for
   mariabackup / xtrabackup, which should help to avoid almost all
   conflicts due to the use of the same ports - both during testing
   with mtr, so and when restarting nodes quickly in a production
   environment.
2) Added more protection to scripts against unexpected return of
   the rc != 0 (in the commands for deleting temporary files, etc).
3) Added protection against unexpected crashes during binlog transfer
   (in SST scripts for rsync).
4) Spaces and some special characters in binlog filenames shouldn't
   be a problem now (at the script level).
5) Daemon process termination tracking has been made more robust
   against crashes due to unexpected termination of the previous SST
   process while new scripts are running.
6) Reading ssl encryption parameters has been moved from specific
   SST scripts to a common wsrep_sst_common.sh script, which allows
   unified error handling, unified diagnostics and simplifies script
   revisions in the future.
7) Improved diagnostics of errors related to the use of openssl.
8) Corrections have been made for xtrabackup-v2 (both in tests and in
   the script code) that restore the work of xtrabackup with updated
   versions of innodb.
9) Fixed some tests for galera_3nodes, although the complete solution
   for the problem of starting three nodes at the same time on fast
   machines will be done in a separate commit.

No additional tests are required as this commit fixes problems with
existing tests.

make_join_select() calls const_cond->val_int(). There are edge cases
where const_cond may have a not-yet optimized subquery.

(The subquery will have used_tables() covered by join->const_tables. It
will still have const_item()==false, so other parts of the optimizer
will not try to evaluate it.  We should probably mark such subqueries
as constant but that is outside the scope of this MDEV)

This could cause out of order wsrep checkpoints due wsrep specific leader
code not being executed in `MYSQL_BIN_LOG::write_transaction_to_binlog_events`.
Move original result assignment to before wsrep logic to prevent that.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

MDEV-22617 Galera node crashes when trying to log to slow_log table in
streaming replication mode

Other things:
- Changed name of wsrep_after_row(two arguments) to
  wsrep_after_row_internal(one argument) to not depended on the
  function signature with unused arguments.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>
	     Added test case

Small postfix to MDEV-23175 to ensure faster option on FreeBSD
and compatibility to Solaris that isn't high resolution.

ftime is left as a backup in case an implementation doesn't
contain any of these clocks.

FreeBSD
    $ ./unittest/mysys/my_rdtsc-t
    1..11
    # ----- Routine ---------------
    # myt.cycles.routine          :             5
    # myt.nanoseconds.routine     :            11
    # myt.microseconds.routine    :            13
    # myt.milliseconds.routine    :            11
    # myt.ticks.routine           :            17
    # ----- Frequency -------------
    # myt.cycles.frequency        :    3610295566
    # myt.nanoseconds.frequency   :    1000000000
    # myt.microseconds.frequency  :       1000000
    # myt.milliseconds.frequency  :           899
    # myt.ticks.frequency         :           136
    # ----- Resolution ------------
    # myt.cycles.resolution       :             1
    # myt.nanoseconds.resolution  :             1
    # myt.microseconds.resolution :             1
    # myt.milliseconds.resolution :             7
    # myt.ticks.resolution        :             1
    # ----- Overhead --------------
    # myt.cycles.overhead         :            26
    # myt.nanoseconds.overhead    :         19140
    # myt.microseconds.overhead   :         19036
    # myt.milliseconds.overhead   :           578
    # myt.ticks.overhead          :         21544
    ok 1 - my_timer_init() did not crash
    ok 2 - The cycle timer is strictly increasing
    ok 3 - The cycle timer is implemented
    ok 4 - The nanosecond timer is increasing
    ok 5 - The nanosecond timer is implemented
    ok 6 - The microsecond timer is increasing
    ok 7 - The microsecond timer is implemented
    ok 8 - The millisecond timer is increasing
    ok 9 - The millisecond timer is implemented
    ok 10 - The tick timer is increasing
    ok 11 - The tick timer is implemented

Small postfix to MDEV-23175 to ensure faster option on FreeBSD
and compatibility to Solaris that isn't high resolution.

ftime is left as a backup in case an implementation doesn't
contain any of these clocks.

FreeBSD
    $ ./unittest/mysys/my_rdtsc-t
    1..11
    # ----- Routine ---------------
    # myt.cycles.routine          :             5
    # myt.nanoseconds.routine     :            11
    # myt.microseconds.routine    :            13
    # myt.milliseconds.routine    :            11
    # myt.ticks.routine           :            17
    # ----- Frequency -------------
    # myt.cycles.frequency        :    3610295566
    # myt.nanoseconds.frequency   :    1000000000
    # myt.microseconds.frequency  :       1000000
    # myt.milliseconds.frequency  :           899
    # myt.ticks.frequency         :           136
    # ----- Resolution ------------
    # myt.cycles.resolution       :             1
    # myt.nanoseconds.resolution  :             1
    # myt.microseconds.resolution :             1
    # myt.milliseconds.resolution :             7
    # myt.ticks.resolution        :             1
    # ----- Overhead --------------
    # myt.cycles.overhead         :            26
    # myt.nanoseconds.overhead    :         19140
    # myt.microseconds.overhead   :         19036
    # myt.milliseconds.overhead   :           578
    # myt.ticks.overhead          :         21544
    ok 1 - my_timer_init() did not crash
    ok 2 - The cycle timer is strictly increasing
    ok 3 - The cycle timer is implemented
    ok 4 - The nanosecond timer is increasing
    ok 5 - The nanosecond timer is implemented
    ok 6 - The microsecond timer is increasing
    ok 7 - The microsecond timer is implemented
    ok 8 - The millisecond timer is increasing
    ok 9 - The millisecond timer is implemented
    ok 10 - The tick timer is increasing
    ok 11 - The tick timer is implemented

The old code erroneously used default_charset_info to compare field names.
default_charset_info can point to any arbitrary collation,
including ucs2*, utf16*, utf32*, including those that do not
support strcasecmp().

my_charset_utf8mb4_unicode_ci, which is used in this scenario:

CREATE TABLE t1 ENGINE=InnoDB WITH SYSTEM VERSIONING AS SELECT 0;

does not support strcasecmp().

Fixing the code to use Lex_ident::streq(), which uses
system_charset_info instead of default_charset_info.

vsnprintf takes the space need for trailing '\0' in consideration, and copies only n-1 characters to destination buffer.
With the old code, only sizeof(buf)-2 characters were copied, this caused that last character of message could be lost.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

Since commit fb335b48 we may have
a null pointer in purge_sys.query when fetch_data_into_cache() is
invoked and innodb_force_recovery>4. This is because the call to
purge_sys.create() would be skipped.

fetch_data_into_cache(): Load the purge_sys pseudo transaction pointer
to a local variable (null pointer if purge_sys is not initialized).

This commit contains a fix, where the replication write set for a CREATE TABLE
will contain, as certification keys, table names for all FK references.
With this, all DML for the FK parent tables will conflict with the CREATE TABLE
statement.

There is also new test galera.MDEV-27276 to verify the fix.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>