MDEV-22370 safe_mutex: Trying to lock uninitialized mutex at /data/src/10.4-bug/sql/rpl_parallel.cc, line 470 upon shutdown during FTWRL

Problem:- When we issue FTWRL with shutdown in parallel, there is race between
FTWRL and shutdown. Shutdown might destroy the mutex (pool->LOCK_rpl_thread_pool)
before FTWRL can lock it. So we can get crash on FTWRL thread

Solution:- mysql_mutex_destroy(pool->LOCK_rpl_thread_pool) should wait for
FTWRL thread to complete its work , and then destroy.
So slave_prepare_for_shutdown will just deactivate the pool, and mutex is destroyed
later in end_slave()

Parallel slave server shutdown found to be hanging in
close_connections() triggered by shutdown due to a slave worker thread
would not be notified to exit in case the worker was sitting idle.

Fixed with destroying the worker pool earlier that is in
slave_prepare_for_shutdown() when all their driver threads have already left.
A test file is added to simulate the bug condition as well as check
multi-sourced and not-idle worker cases.

This commit contains a large set of further bug fixes and
improvements to SST scripts for Galera, continuing the work
that was started in MDEV-24962 to make SST scripts work smoothly
in different network configurations (especially using ipv6) and
with different environment settings:

 1) The ipv6 addresses were incorrectly handled in the SST script
    for rsync (incorrect address substitution for establishing a
    connection, incorrect address substitution for bind, and so on);
 2) Checking the locality of the ip-address in SST scripts did not
    support ipv6 addresses (such as "[::1]"), which were falsely
    identified as non-local ip, which further did not allow running
    two SSTs on different local addresses on the same machine.
    On the other hand, this bug masked some other errors (related
    to handling ipv6 addresses);
 3) The code for checking the locality of the ip address was different
    in the SST scripts for rsync and for mysqldump, with individual
    flaws. This code is now made common and moved to wsrep_sst_common;
 4) Waiting for the start of the transport channel (socat, nc, rsync,
    stunnel) in the wait_for_listen() and check_pid_and_port() functions
    did not process ipv6 addresses correctly in all cases (not for all
    branches);
 5) Waiting for the start of the transport channel (socat, nc, rsync,
    stunnel) in the wait_for_listen() and check_pid_and_port() functions
    for some code branches could give a false positive result due to
    the textual match of prefixes in the port number and/or PID of
    the process;
 6) Waiting for the start of the transport channel (socat, nc, rsync,
    stunnel) was supported through different utilities in SST scripts
    for mariabackup and for rsync, and with various minor flaws in
    the code. Now the code is still different in these scripts, but
    it supports a common set of utilities (lsof, ss, sockstat) and
    is synchronized across patterns that used to check the output
    of  these utilities;
 7) In SST via mariabackup, the signal about readiness to receive data
    is sometimes sent too early - immediately after listen(), and not
    after accept() (which are called by socat or netcat utility).
 8) Checking availability of the some options of some utilities was
    done using the grep pattern, which easily gives false positives;
 9) Common name (CN) for local addresses, if not explicitly specified,
    is now always replaced to "localhost" to avoid the need to generate
    many separate certificates for local addresses of one machine and
    not to depend on which the local address is currently used in test
    (ipv4 or ipv6, etc.);
10) In tests galera_sst_mariabackup_encrypt_with_key_server and
    galera_sst_rsync_encrypt_with_key_server the correct certificate
    is selected to avoid commonname (CN) mismatch problems;
11) Further refactoring to protect against spaces in file names.
12) Further general refactoring to eliminate bash-specific constructs
    or to improve code readability;
13) The code for setting options for the nc (netcat) utility was
    different in different scripts for SST - now it is made identical.
14) Fixed long-time broken encryption via xbcrypt in combination with
    mariabackup and added support for key-based encryption via openssl
    utility, which is now enabled by default for encrypt=1 mode (this
    default mode can be changed using a new configuration file option
    "encypt-format=openssl|xbcrypt", which can be placed in the [mysqld],
    [sst] or in the [xtrabackup] section) - this change will allow us
    to use and to test the encypt=1 encryption without installing
    non-standard third-party utilities.

This is incosistent with other single config generators, where
the default type has always been RelWithDebInfo.

Fixed by moving setting of CMAKE_BUILD_TYPE right before PROJECT.

unless explicitly overruled with -DWITH_JEMALLOC=no

XA transaction only allows to access data in specific states,
in ACTIVE, but not in IDLE or PREPARE.

But even then one should be able to run SHOW STATUS.

if mysqld is restarted during a test, the debugger script
should be reused or overwritten, but not appended to the existing file

mysqltest has no --gdb option

This reverts commit 72fa9dab

but doesn't recover deleted jars - they still exist in
mysql-test/connect/std_data, no need to have them twice.

Also it removes a redundant copy on JavaWrappers.jar

and configuration.

1. Pass joiner's authentication information to donor together with address
   in State Transfer Request. This allows joiner to authenticate donor on
   connection. Previously joiner would accept data from anywhere.

2. Deprecate custom SSL configuration variables tca, tcert and tkey in favor
   of more familiar ssl-ca, ssl-cert and ssl-key. For backward compatibility
   tca, tcert and tkey are still supported.

3. Allow falling back to server-wide SSL configuration in [mysqld] if no SSL
   configuration is found in [sst] section of the config file.

4. Introduce ssl-mode variable in [sst] section that takes standard values
   and has following effects:
    - old-style SSL configuration present in [sst]: no effect
      otherwise:
    - ssl-mode=DISABLED or absent: retains old, backward compatible behavior
      and ignores any other SSL configuration
    - ssl-mode=VERIFY*: verify joiner's certificate and CN on donor,
                        verify donor's secret on joiner
                        (passed to donor via State Transfer Request)
                        BACKWARD INCOMPATIBLE BEHAVIOR
    - anything else enables new SSL configuration convetions but does not
      require verification

    ssl-mode should be set to VERIFY only in a fully upgraded cluster.

    Examples:

    [mysqld]
    ssl-cert=/path/to/cert
    ssl-key=/path/to/key
    ssl-ca=/path/to/ca

    [sst]

     -- server-wide SSL configuration is ignored, SST does not use SSL

    [mysqld]
    ssl-cert=/path/to/cert
    ssl-key=/path/to/key
    ssl-ca=/path/to/ca

    [sst]
    ssl-mode=REQUIRED

     -- use server-wide SSL configuration for SST but don't attempt to
        verify the peer identity

    [sst]
    ssl-cert=/path/to/cert
    ssl-key=/path/to/key
    ssl-ca=/path/to/ca
    ssl-mode=VERIFY_CA

     -- use SST-specific SSL configuration for SST and require verification
        on both sides
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

1. Fix eval command line to correctly pass stunnel option to rsync on donor.
2. Deprecate `tkey`, `tcert` and `tca` options in [sst] section in favor of
   conventional `ssl-key`, `ssl-cert` and `ssl-ca`, but keep their precedence
   for backward compatibility.
3. Default to require SSL encryption if at least SSL key and cert files are
   specified in configuration, either in [sst] or [mysqld] sections.
4. Enable `verify*` option for stunnel on donor only if
   a. CA file is specified somewhere in the configuration
   b. it is explicitly requested in [sst] section by either specifying
   ssl-mode or CA file there. In this case if ssl-mode is not explicitly
   given, it defaults to VERIFY_CA.

   ssl-mode maps to stunnel options as follows:
   VERIFY_CA       -> verifyChain = yes
   VERIFY_IDENTITY -> verifyPeer = yes

   Example to require donor to verify joiner identity:
```
[mysqld]
ssl-cert=/path/to/cert
ssl-key=/path/to/key
ssl-ca=/path/to/ca

[sst]
ssl-mode=VERIFY_IDENTITY
```
5. If SSL verification is requested, joiner verifies donor by checking the
   secret passed to donor via SST request.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

When you only need view structure, don't call handle_derived with
DT_CREATE and rely on its internal hackish check to skip DT_CREATE.
Because handle_derived is called from many different places,
and this internal hackish check is indiscriminative.

Instead, just don't ask handle_derived to do DT_CREATE
if you don't want it to do DT_CREATE.

When you only need view structure, don't call handle_derived with
DT_CREATE and rely on its internal hackish check to skip DT_CREATE.
Because handle_derived is called from many different places,
and this internal hackish check is indiscriminative.

Instead, just don't ask handle_derived to do DT_CREATE
if you don't want it to do DT_CREATE.

After switching to the new mariabackup interface (instead of
the outdated innobackupex interface, which is supported for
compatibility), we need to explicitly pass a path to the datadir
directory as a parameter, since in the new interface the value
of this option is not automatically set in such a way that it
always matches the SST/IST logic. This commit adds passing this
option as an explicit parameter to mariabackup. This commit also
removed unnecessary options that are not used and not supported
by mariabackup.

Also, numerous flaws in the common wsrep_sst_common script have
been fixed:

 1) There are many bash-specific constructs in the script that
    may not be supported by other interpreters, which can lead
    to the most unexpected errors during SST, because failures
    in the interpretation of bash-specific constructs lead to
    incorrect parsing of arguments;
 2) There is parse_cnf() function which is often called by other
    scripts for the "mysqld" or "--mysqld" group, but it does not
    take into account the default group suffix, which leads to
    reading values only from the default group, which then leads
    to errors due to reading the default values instead of the
    values for a specific group;
 3) Some options such as --user, --innodb-data-home-dir or --datadir
    are not removed from the --mysqld-args list, although they are
    processed inside scripts (and passing of these options funther
    may cause problems for mariabackup);
 4) If an argument that the script understands is present in
    the --mysqld-args list twice, then this causes SST to fail,
    instead of reading the most recent value;
 5) The "--host" parameter is technically still supported among
    the arguments of the SST scripts, but in reality scripts do not
    work with it as expected, especially if it has an IPv6 address;
 6) If the port number is absent in the --address parameter value,
    but the port number is explicitly passed through the --port
    argument, then the scripts for mariabackup and xtrabackup-v2
    fail;
 7) If a new address interface is used (with the --address parameter),
    then automatic default port substitution is not performed, although
    it is supported for the legacy --host/--port interface.
 8) If there are spaces in the parameter values after --mysqld_args,
    then their further transfer does not occur correctly, which
    causes mariabackup to fail during SST - the space splits
    the argument in such a way that it breaks the parsing of the
    following parameters;
 9) If most of the parameters that are names or paths to the files
    or directories contain spaces, then SST scripts fail in an
    unpredictable way due to incorrect variable substitutions;
10) If the --log-bin option is passed among the arguments of myqlds
    (--mysqld-args) without a parameter, and the --binlog option
    is not specified, then the script cannot substitute the default
    name for binlog and cannot construct binlog name using the
    --log-basename argument (which is against server specifications);
11) Tail slashes are not removed from the directory names, which,
    upon further substitution, leads to the appearance of a double
    slash in the file paths;
12) The explicit --binlog parameter (which is now always transmitted
    from the server side) and the "hidden" --log-bin parameter in the
    list of arguments after --mysqld-args are perceived as two different
    parameters in different parts of the scripts, and if they are do not
    match for some reason, this will lead to failures during SST;

Also, all new changes from the 10.6 branch have been migrated here,
including the latest pull requests for authentication (only the part
that concerns SST scripts).

It also fixes dozens of other bugs in all SST scripts.

Removed numerous extra blank lines and spaces that interfere with
reading and understanding program code, making it more difficult to
find errors in scripts. I also removed all extra trailing spaces at
the ends of lines, which lead to marking extra lines as changes
(in subsequent changes). The amount of indentation in some parts
of the code has also been normalized.

Fix a race condition in the testcase. The testcase assumed that
State='Sending data' means that the thread is already in an
InnoDB lock wait. This is not case, there is a gap between the
state changing to Sending data and execution reaching the point
where it is waiting for a lock.

Use a more precise check instead, through I_S.INNODB_TRX.

Problem:
========
180511 11:07:58 [ERROR] Slave I/O: Unexpected master's heartbeat data:
heartbeat is not compatible with local info;the event's data: log_file_name
mysql-bin.000009 log_pos 1054262041, Error_code: 1623

Analysis:
=========
In replication setup when master server doesn't have any events to send to
slave server it sends an 'Heartbeat_log_event'. This event carries the
current binary log filename and offset details. The offset values is stored
within 4 bytes of event header. When the size of binary log is higher than
UINT32_MAX the log_pos values will not fit in 4 bytes memory.  It overflows
and hence slave stops with an error.

Fix:
===
Since we cannot extend the common_header of Log_event class, a greater than
4GB value of Log_event::log_pos is made to be transported with a HeartBeat
event's sub-header.  Log_event::log_pos in such case is set to zero to
indicate that the 8 byte sub-header is allocated in the event.

In case of cross version replication following behaviour is expected

OLD - Server without fix
NEW - Server with fix

OLD<->NEW : works bidirectionally as long as the binlog offset is
            (normally) within 4GB.

When log_pos > UINT32_MAX
OLD->NEW  : The 'log_pos' is bound to overflow and NEW slave may report
            an invalid event/incompatible heart beat event error.
NEW->OLD  : Since patched server sets log_pos=0 on overflow, OLD slave will
            report invalid event error.

- Fixing post-push failure of innodb_fts_misc_1 test case.

InnoDB tries to fetch the deleted doc ids for discarded
tablespace. In i_s_fts_deleted_generic_fill(), InnoDB needs
to check whether the table is discarded or not before fetching
deleted doc ids.

fil_ibd_load(): Remove a message that is basically saying that
everything works as expected. The other "Ignoring data file" message
about the presence of an extraneous file will be retained
(and expected by the test innodb.log_file_name).

This reverts commit 8880dff2.

This commits replaces the call of the function setup_tables() with
a call of the function setup_tables_and_check_access() in the method
Multiupdate_prelocking_strategy::handle_end().
There is no known bug that would require this change. However the change
aligns this piece of code with the code existed before the patch for
MDEV-24823.

Attempt to build MariaDB server on MacOS could result in
compilation errors like the following one:

  In file included from server-10.2/storage/perfschema/cursor_by_account.cc:28:
  In file included from server-10.2/include/my_global.h:287:
  In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk/usr/include/c++/v1/math.h:309:
  In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk/usr/include/c++/v1/type_traits:418:
    server-10.2/version:1:1: error: expected unqualified-id

    MYSQL_VERSION_MAJOR=10
    ^
  server-10.2/build.dir/include/my_config.h:529:29: note: expanded from macro 'MYSQL_VERSION_MAJOR'

This kind of compiler errors occur by the reson that compiler's system headers
contain the directive '#include <version>' and a compiler is invoked
with -I${CMAKE_SOURCE_DIR}.

The MariaDB source code root directory contains the file VERSION that is handled
by the compiler during processing the directive #include <version>
since file names on MacOS are case insensetive, so version and VERSION is treated as
the same file name.

To fix the issue the source code root directory should be removed from a list
of directories used by the compiler for include search path.

The problem is that sharing default expression among set instruction
leads to attempt access result field of function created in
other instruction runtime MEM_ROOT and already freed
(a bit different then MySQL problem).

Fix is the same as in MySQL (but no optimisation for constant), turn
DECLARE a, b, c type DEFAULT expr;
to
DECLARE a type DEFAULT expr, b type DEFAULT a, c type DEFAULT a;

Removed explicit InnoDB monitor startup and used just functions
to print current lock information.

Problem was that we should skip strict password validation on
applier nodes similarly as is done for slave nodes.

Replace unnecessary sleeps with real wait_conditions to make
sure correct cluster sizes.