Fix issue was earlier fixed by MDEV-31724. Only adding MTR tests.

Field_varstring::get_copy_func() did not take into account
that functions do_varstring1[_mb], do_varstring2[_mb] do not support
compressed data.

Changing the return value of Field_varstring::get_copy_func()
to `do_field_string` if there is a compresion and truncation
at the same time. This fixes the problem, so now it works as follows:
- val_str() uncompresses the data
- The prefix is then calculated on the uncompressed data

Additionally, introducing two new copying functions
- do_varstring1_no_truncation()
- do_varstring2_no_truncation()

Using new copying functions in cases when:
- a Field_varstring with length_bytes==1 is changing to a longer
    Field_varstring with length_bytes==1
- a Field_varstring with length_bytes==2 is changing to a longer
    Field_varstring with length_bytes==2

In these cases we don't care neither of compression nor
of multi-byte prefixes: the entire data gets fully copied
from the source column to the target column as is.

This is a kind of new optimization, but this also was needed
to preserve existing MTR test results.

The test case accessed slave-relay-bin.000003 without waiting for the IO
thread to write it first. If the IO thread was slow, this could fail.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Revert the old work-around for buggy fdatasync() on Linux ext3. This bug was
fixed in Linux > 10 years ago back to kernel version at least 3.0.
Reviewed-by: Marko Mäkelä <marko.makela@mariadb.com>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

This is also related to
MDEV-31348 Assertion `last_key_entry >= end_pos' failed in virtual bool
           JOIN_CACHE_HASHED::put_record()

Valgrind exposed a problem with the join_cache for hash joins:
=25636== Conditional jump or move depends on uninitialised value(s)
==25636== at 0xA8FF4E: JOIN_CACHE_HASHED::init_hash_table()
          (sql_join_cache.cc:2901)

The reason for this was that avg_record_length contained a random value
if one had used SET optimizer_switch='optimize_join_buffer_size=off'.

This causes either 'random size' memory to be allocated (up to
join_buffer_size) which can increase memory usage or, if avg_record_length
is less than the row size, memory overwrites in thd->mem_root, which is
bad.

Fixed by setting avg_record_length in JOIN_CACHE_HASHED::init()
before it's used.

There is no test case for MDEV-31893 as valgrind of join_cache_notasan
checks that.
I added a test case for MDEV-31348.

This reverts commit 6c405904.

This reverts commit b54e4bf0.

There was two related problems:

(1) Galera node that is defined as a slave to async MariaDB
master at restart might do SST (state stransfer) and
part of that it will copy mysql.gtid_slave_pos table.
Problem is that updates on that table are not replicated
on a cluster. Therefore, table from donor that is not
slave is copied and joiner looses gtid position it was
and start executing events from wrong position of the binlog.
This incorrect position could break replication and
causes node to be dropped and requiring user action.

(2) Slave sql thread might start executing events before
galera is ready (wsrep_ready=ON) and that could also
cause node to be dropped from the cluster.

In this fix we enable replication of mysql.gtid_slave_pos
table on a cluster. In this way all nodes in a cluster
will know gtid slave position and even after SST joiner
knows correct gtid position to start.

Furthermore, we wait galera to be ready before slave
sql thread executes any events to prevent too early
execution.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Now that the systemd unit files use an environment file to pass
_WSREP_START_POSITION we have to update galera_new_cluster as well.

We used to run `systemctl set-environment` to pass
_WSREP_START_POSITION. This is bad because:

* it clutter systemd's environment (yes, pid 1)
* it requires root privileges
* options (like LimitNOFILE=) are not applied

Let's just create an environment file in ExecStartPre=, that is read
before ExecStart= kicks in. We have _WSREP_START_POSITION around for the
main process without any downsides.

failed SHOW CREATE FUNCTION means we don't dump this function,
but should still try to dump all other functions

don't construct open ranges from prefix blob keys for < (less than)
just as it's already done for > (greater than)

because prefix KEY_PART doesn't create prefix Field for blobs
(see open_table_from_share() near "Create a new field for the key part"),
so stored_field_cmp_to_item() will compare the original field to the
value not taking the prefix length into account.

This reverts commit add0c01b

Duplicates must be avoided in FTS_DOC_ID_INDEX

(It's not using include/rpl_init.inc, so it needs to reset the GTID position
explicitly).
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

A simple "SET SESSION gtid_seq_no= DEFAULT" did not work, it would straight
up crash the server! Also, explicitly setting gtid_seq_no to 0 gave an error
in --gtid-strict-mode=1.

Setting to DEFAULT or 0 should disable any prior setting of
gtid_seq_no, so that the next transaction is allocated the next GTID
in sequence, as normal.
Reviewed-by: Monty <monty@mariadb.org>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

This patch adds for "--ps-protocol" second execution
of queries "SELECT".
Also in this patch it is added ability to disable/enable
(--disable_ps2_protocol/--enable_ps2_protocol) second
execution for "--ps-prototocol" in testcases.

MDEV-31407: Add aliases in opt_trace.test for long column name for removing "--disable-view-protocol"

Change tests:
	opt_trace.test
	opt_trace_index_merge.test
	opt_trace_ucs2.test

MDEV-31749 sporadic assert in MDEV-30619 new test

If the workers of a parallel replica are busy (potentially with long
queues), but the SQL thread has no events left to distribute (so it
goes idle), then the next event that comes from the primary will
update mi->last_master_timestamp with its timestamp, even if the
workers have not yet finished.

This patch changes the parallel replica logic which updates
last_master_timestamp after idling from using solely sql_thread_caught_up
(added in MDEV-29639) to using the latter with rli queued/dequeued
event counters.
That is, if  the queued count is equal to the dequeued count, it
means all events have been processed and the replica is considered
idle when the driver thread has also distributed all events.

Low level details of the commit include
- to make a more generalized test for Seconds_Behind_Master on
  the parallel replica, rpl_delayed_parallel_slave_sbm.test
  is renamed to rpl_parallel_sbm.test for this purpose.
- pause_sql_thread_on_next_event usage was removed
  with the MDEV-30619 fixes. Rather than remove it, we adapt it
  to the needs of this test case
- added test case to cover SBM spike of relay log read and LMT
  update that was fixed by MDEV-29639
- rpl_seconds_behind_master_spike.test is made to use
  the negate_clock_diff_with_master debug eval.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>

We introduce simple plugin dependency. A plugin init function may
return HA_ERR_RETRY_INIT. If this happens during server startup when
the server is trying to initialise all plugins, the failed plugins
will be retried, until no more plugins succeed in initialisation or
want to be retried.

This will fix spider init bugs which is caused in part by its
dependency on Aria for initialisation.

The reason we need a new return code, instead of treating every
failure as a request for retry, is that it may be impossible to clean
up after a failed plugin initialisation. Take InnoDB for example, it
has a global variable `buf_page_cleaner_is_active`, which may not
satisfy an assertion during a second initialisation try, probably
because InnoDB does not expect the initialisation to be called
twice.

Since TLS server certificate verification is a client
only option, this flag is removed in both client (C/C)
and MariaDB server capability flags.

This patch reverts commit 89d759b9
(MySQL Bug #21543) and stores the server certificate validation
option in mysql->options.extensions.

Since TLS server certificate verification is a client
only option, this flag is removed in both client (C/C)
and MariaDB server capability flags.

This patch reverts commit 89d759b9
(MySQL Bug #21543) and stores the server certificate validation
option in mysql->options.extensions.

noinline attribute was being ignored by clang-16 and reporting
32 stack size on Gentoo, 16 locally on Fedora 38.

Based on https://stackoverflow.com/questions/54481855/clang-ignoring-attribute-noinline
appended noopt in addition to the gcc recognised attributes.

After that the -pcre_exec(NULL, NULL, NULL, -999, -999, 0, NULL, 0)
returned 1056, simlar to gcc.

From https://bugs.gentoo.org/910188.

Thanks Zhixu Liu for the great bug report.

Restrict vcol_cleanup_expr() in close_thread_tables() to only simple
locked tables mode. Prelocked is cleaned up like normal statement: in
close_thread_table().

First UPDATE under START TRANSACTION does nothing (nstate= nstate),
but anyway generates history. Since update vector is empty we get into
(!uvect->n_fields) branch which only adds history row, but does not do
update. After that we get current row with wrong (old) row_start value
and because of that second UPDATE tries to insert history row again
because it sees trx->id != row_start which is the guard to avoid
inserting multiple trx_id-based history rows under same transaction
(because we have same trx_id and we get duplicate error and this bug
demostrates that). But this try anyway fails because PK is based on
row_end which is constant under same transaction, so PK didn't change.

The fix moves vers_make_update() to an earlier stage of
calc_row_difference(). Therefore it prepares update vector before
(!uvect->n_fields) check and never gets into that branch, hence no
need to handle versioning inside that condition anymore.

Now trx->id and row_start are equal after first UPDATE and we don't
try to insert second history row.

== Cleanups and improvements ==

ha_innobase::update_row():

vers_set_fields and vers_ins_row are cleaned up into direct condition
check. SQLCOM_ALTER_TABLE check now is not used as this is dead code,
assertion is done instead.

upd_node->is_delete is set in calc_row_difference() just to keep
versioning code as much in one place as possible. vers_make_delete()
is still located in row_update_for_mysql() as this is required for
ha_innodbase::delete_row() as well.

row_ins_duplicate_error_in_clust():

Restrict DB_FOREIGN_DUPLICATE_KEY to the better conditions.
VERSIONED_DELETE is used specifically to help lower stack to
understand what caused current insert. Related to MDEV-29813.

On create table tmp as select ... we exited Item_func::fix_fields()
with error. fix_fields_if_needed('foo' or 'bar') failed and we
returned true, but already changed const_item_cache. So the item is in
inconsistent state: fixed == false and const_item_cache == false.

Now we cleanup the item before the return if Item_func::fix_fields()
fails to process.

Use vers_check_update() to avoid inserting history row for ODKU if now
versioned fields specified in update_fields.

Constraints processing row_ins_check_foreign_constraint() was not
called because row_upd_check_references_constraints() didn't see
update as delete: node->is_delete was false.

Since MDEV-30378 we check for TRG_EVENT_DELETE to detect versioned
delete in ha_innobase::update_row().

Now we can use TRG_EVENT_DELETE to set upd_node->is_delete, so
constraints processing is triggered correctly.

1. Exclude merging history rows into fts index.

The check !history_fts && (index->type & DICT_FTS) was just incorrect
attempt to avoid history in fts index.

2. Don't check for duplicates for history rows.

The `safe_strcpy()` function was added in
https://github.com/mariadb/server/commit/567b68129943#diff-23f88d0b52735bf79b7eb76e2ddbbebc96f3b1ca16e784a347525a9c43134d77

Unfortunately, its current implementation triggers many GCC 8+ string
truncation and array bounds warnings, particularly due to the potential
for a false positive `-Warray-bounds`.

For example, the line `safe_strcpy(delimiter, sizeof(delimiter), ";")` in
`client/mysqldump.c` causes the following warning:

    [1669/1914] Building C object client/CMakeFiles/mariadb-dump.dir/mysqldump.c.o
    In file included from /PATH/include/my_sys.h:20,
                     from /PATH/mysqldump.c:51:
    In function ?safe_strcpy?,
        inlined from ?dump_events_for_db.isra? at /PATH/client/mysqldump.c:2595:3:
    /PATH/include/m_string.h:258:39: warning: array subscript 1535 is outside array bounds of ?const char[2]? [-Warray-bounds=]
      258 |   if (dst[dst_size - 2] != '\0' && src[dst_size - 1] != '\0')
          |                                    ~~~^~~~~~~~~~~~~~

GCC is reporting that the `safe_strcpy` function *could* cause an
out-of-bounds read from the constant *source* string `";"`, however this
warning is unhelpful and confusing because it can only happen if the size of
the *destination* buffer is incorrectly specified, which is not the case
here.

In https://github.com/MariaDB/server/pull/2640, Andrew Hutchings proposed
fixing this by disabling the `-Warray-bounds` check in this function
(specifically in
https://github.com/MariaDB/server/pull/2640/commits/be382d01d08739d081f6cf40f350f7414f29b49d#diff-23f88d0b52735bf79b7eb76e2ddbbebc96f3b1ca16e784a347525a9c43134d77R255-R262).

However, this was rejected because it also disables the *helpful*
`-Warray-bounds` check on the destination buffer.

Cherry-picking the commit
https://github.com/MariaDB/server/commit/a7adfd4c52307876d68ad3386cefd3757ee66e92
from 11.2 by Monty Widenius solves the first two problems:

1. It reimplements `safe_strcpy` a bit more efficiently, skipping the
   `memset(dst, 0, dst_size)`. This is unnecessary since `strncpy` already
   pads `dst` with 0 bytes.
2. It will not trigger the `-Warray-bounds` warning, because `src` is
   not read based on an offset determined from `dst_size`.

There is a third problem, however.  Using `strncpy` triggers the
`-Wstringop-truncation` warning
(https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wstringop-truncation),
so we need to disable that.  However, that is a much less broadly and
generally-useful warning so there is no loss of static analysis value caused
by disabling it.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.

Note: We should replace most case of safe_strcpy() with strmake() to avoid
the not needed zerofill.

Do not get value of expensive constants.

Add the end of file marker x1A to the DBF file and handle it
correctly to preserve interoperability with Libreoffice, and others
that have followed the DBF spec.

The file open mode of "a+" was problematic because Linux and the OSX,
the previous main development mode are inconsistent (see man fopen).
The main problem per the bug report was the inability to fseek back to the
beginning to update the records in the header.

As such the "a+" mode is remove and "w+b" is used inserting to a new file
and "r+b" is used for appending to the file.

In DBFFAM::CloseTableFile move PlugCloseFile down to close the file in
all modes.

The year unlike the comments is always since 1900. Use the
YYYY-MM-DD as an unabigious form during tracing.

Thanks for Mr. Zoltan Duna for the descriptive bug report.