Problem:
========
 During upgrade, InnoDB does write the redo log for adjusting
the tablespace size or tablespace flags even before the log
has upgraded to configured format. This could lead to data
inconsistent if any crash happened during upgrade process.

Fix:
===
srv_start(): Write the tablespace flags adjustment, increased
tablespace size redo log only after redo log upgradation.

log_write_low(), log_reserve_and_write_fast(): Check whether
the redo log is in physical format.

Like the fix for MDEV-32753 and MDEV-33242, spider init queries
creates new connections that use the global sql_mode of the existing
connection.

The problem was that SHOW PROCESSLIST was done before the command of
the default connection was cleared.

Reviewer: Sergei Golubchik <serg@mariadb.org>

Previously, the behavior was to error out on the first invalid option
encountered. With this change, a best effort approach is made so that
all invalid options processed will be printed before exiting.

There is a caveat. The options are processed many times at varying
stages of server startup because the server is not aware of all valid
options immediately (e.g. plugins have to be loaded first before the
server knows what are the available plugin options). So, there are some
options that the server can determine are invalid "early" on, and there
are some options that the server cannot determine are invalid until
"later" on. For example, the server can determine an option such as
`--a` is an ambiguous option very early on but an option such as
`--this-does-not-match-any-option` cannot be labelled as invalid until
the server is aware of all available options.

Thus, it is possible that the server will still fail before printing out
all "invalid" options. You can see this by passing `--a
--obvious-invalid-option`.

Test cases were added to `mysqld_option_err.test` to validate that
multiple invalid options will be displayed in the error message.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services.

The root cause is the WAL logging of file operation when the actual
operation fails afterwards. It creates a situation with a log entry for
a operation that would always fail. I could simulate both the backup
scenario error and Innodb recovery failure exploiting the weakness.

We are following WAL for file rename operation and once logged the
operation must eventually complete successfully, or it is a major
catastrophe. Right now, we fail for rename and handle it as normal error
and it is the problem.

I created a patch to address RENAME operation to a non existing schema
where the destination schema directory is missing. The patch checks for
the missing schema before logging in an attempt to avoid the failure
after WAL log is written/flushed. I also checked that the schema cannot
be dropped or there cannot be any race with other rename to the same
file. This is protected by the MDL lock in SQL today.

The patch should this be a good improvement over the current situation
and solves the issue at hand.

If a query with GROUP_CONCAT is executed then the server reports a warning
every time when the length of the result of this function exceeds the set
value of the system variable group_concat_max_len. This bug led to the set
of warnings from the second execution of the prepared statement that did
not coincide with the one from the first execution if the executed query
was a grouping query over a join of tables using GROUP_CONCAT function and
join cache was not allowed to be employed.
The descrepancy of the sets of warnings was due to lack of cleanup for
Item_func_group_concat::row_count after execution of the query.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

Functions extracting non-negative datetime components:

- YEAR(dt),        EXTRACT(YEAR FROM dt)
- QUARTER(td),     EXTRACT(QUARTER FROM dt)
- MONTH(dt),       EXTRACT(MONTH FROM dt)
- WEEK(dt),        EXTRACT(WEEK FROM dt)
- HOUR(dt),
- MINUTE(dt),
- SECOND(dt),
- MICROSECOND(dt),
- DAYOFYEAR(dt)
- EXTRACT(YEAR_MONTH FROM dt)

did not set their max_length properly, so in the DECIMAL
context they created a too small DECIMAL column, which
led to the 'Out of range value' error.

The problem is that most of these functions historically
returned the signed INT data type.

There were two simple ways to fix these functions:
1. Add +1 to max_length.
   But this would also change their size in the string context
   and create too long VARCHAR columns, with +1 excessive size.

2. Preserve max_length, but change the data type from INT to INT UNSIGNED.
   But this would break backward compatibility.
   Also, using UNSIGNED is generally not desirable,
   it's better to stay with signed when possible.

This fix implements another solution, which it makes all these functions
work well in all contexts: int, decimal, string.

Fix details:

- Adding a new special class Type_handler_long_ge0 - the data type
  handler for expressions which:
  * should look like normal signed INT
  * but which known not to return negative values
  Expressions handled by Type_handler_long_ge0 store in Item::max_length
  only the number of digits, without adding +1 for the sign.

- Fixing Item_extract to use Type_handler_long_ge0
  for non-negative datetime components:
   YEAR, YEAR_MONTH, QUARTER, MONTH, WEEK

- Adding a new abstract class Item_long_ge0_func, for functions
  returning non-negative datetime components.
  Item_long_ge0_func uses Type_handler_long_ge0 as the type handler.
  The class hierarchy now looks as follows:

Item_long_ge0_func
  Item_long_func_date_field
    Item_func_to_days
    Item_func_dayofmonth
    Item_func_dayofyear
    Item_func_quarter
    Item_func_year
  Item_long_func_time_field
    Item_func_hour
    Item_func_minute
    Item_func_second
    Item_func_microsecond

- Cleanup: EXTRACT(QUARTER FROM dt) created an excessive VARCHAR column
  in string context. Changing its length from 2 to 1.

As a result of this bug the second execution of the prepared statement
created for select from materialized view could return a wrong result set if
- the specification of the view used a left join
- an inner table the left join was a mergeable derived table
- the derived table contained a constant column.

The problem appeared because the flag 'maybe-null' of the wrapper
Item_direct_view_ref constructed for the constant field of the mergeable
derived table was not set to 'true' on the second execution of the
prepared statement.

The patch always sets this flag properly when calling the function
Item_direct_view_ref::set_null_ref-table(). The latter is invoked in
Item_direct_view_ref constructor if it is created for some reference of
a constant column belonging to a mergeable derived table.

Approved by Oleksandr Byelkin <sanja@mariadb.com>

In an addition to test rpl.rpl_parallel_sbm added by MDEV-32265, the
test uses sleep statements alone to test Seconds_Behind_Master with
delayed replication. On slow running machines, the test can pass the
intended MASTER_DELAY duration and Seconds_Behind_Master can become
0, when the test expects the transaction to still be actively in a
delaying state.

This can be consistently reproduced by adding a sleep statement
before the call to

--let = query_get_value(SHOW SLAVE STATUS, Seconds_Behind_Master, 1)

to sleep past the delay end point.

This patch fixes this by locking the table which the delayed
transaction targets so Second_Behind_Master cannot be updated before
the test reads it for validation.

This removes the error:
"Failed to load slave replication state from table mysql.gtid_slave_pos:
1017: Can't find file: './mysql/' (errno: 2 "No such file or directory")

Should be h2 rather than h1, and GitHub requires an intervening space.

similar to MDEV-30981

- Use "new" math library WOLFSSL_SP_MATH_ALL, which is now  promoted by
  WolfSSL for faster performance. "fastmath" we  used previously is going
  to be deprecated, it was not really always fast.
- Optimize common RSA math operations with WOLFSSL_HAVE_SP_RSA
- Incorporate assembly optimizations, currently for Intel x64 only

This patch significantly reduces execution time for SSL tests like
main.ssl-big and main.ssl_connect, which now run 2 to 3 times faster.

Notably, when this patch is applied to 11.4, server startup in with
ephemeral certificates becomes approximately 10x faster due to optimized
wolfSSL_EVP_PKEY_keygen().

Additionally, refactored WolfSSL by removing old workarounds and
consolidating wolfssl and wolfcrypt into a single library wolfssl, just
like it was done in WolfSSL's own CMake.

CapabilityBoundingSet included CAP_IPC_LOCK in MDEV-9095, however
it requires that the executable has the capability marked in extended
attributes also.

The alternate to this is raising the RLIMIT_MEMLOCK for the service/
process to be able to complete the mlockall system call. This needs to
be adjusted to whatever the MariaDB server was going to allocate.
Rather than leave the non-obvious mapping of settings and tuning,
add the capability so its easier for the user.

We set the capability, if possible, but may never be used depending
on user settings. As such in the Debian postinst script, don't
complain if this fails.

The CAP_IPC_LOCK also facilitates the mmaping of huge memory pages.
(see man mmap), like mariadb uses with --large-pages.

For queries with derived tables populated having some side-effect, we
will fill such a derived table more than once, but without clearing
its rows.  Consequently it will have duplicate rows.
An example query exhibiting the problem is
  SELECT STRAIGHT_JOIN c1 FROM t1 JOIN (SELECT @A := 0) x;
Since mysql_derived_fill will, for UNCACHEABLE_DEPENDENT tables, drop
all rows and repopulate, we relax the condition at line 1204: rather
than assume all uncacheable values prevent early return, we now
allow an early return for uncacheable values other than
UNCACHEABLE_DEPENDENT.  In general, we only populate derived tables
once unless they're dependent tables.

The '<replication_slave>' user is assigned to the slave replication
thread so this name appears in the auditing logs.

buf_read_ahead_linear(): If buf_pool.watch_is_sentinel(*bpage),
do not attempt to read the page frame because the pointer would be null
for the elements of buf_pool.watch[].

Hitting this bug requires the use of a non-default value of
innodb_change_buffering.

The script wsrep_sst_backup was introduced on MariaDB 10.3 in commit
9b2fa2ae. The new script was automatically included in RPM packages but not
in Debian packages (which started to fail on warning about stray file).

Include wsrep_sst_backup in the mariadb-server-10.5+ package, and also
include a stub man page so that packaging of a new script is complete.

Related:
https://galeracluster.com/documentation/html_docs_20210213-1355-master/documentation/backup-cluster.html

This commit was originally submitted in May 2022 in
https://github.com/MariaDB/server/pull/2129 but upstream indicated only
in May 2023 that it might get merged, thus this is for a later release.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.

libxml2 2.12.0 made `xmlGetLastError()` return `const` pointer:

https://gitlab.gnome.org/GNOME/libxml2/-/commit/61034116d0a3c8b295c6137956adc3ae55720711

Clang 16 does not like this:

    error: assigning to 'xmlErrorPtr' (aka '_xmlError *') from 'const xmlError *' (aka 'const _xmlError *') discards qualifiers
    error: cannot initialize a variable of type 'xmlErrorPtr' (aka '_xmlError *') with an rvalue of type 'const xmlError *' (aka 'const _xmlError *')

Let’s update the variables to `const`.
For older versions, it will be automatically converted.

But then `xmlResetError(xmlError*)` will not like the `const` pointer:

    error: no matching function for call to 'xmlResetError'
    note: candidate function not viable: 1st argument ('const xmlError *' (aka 'const _xmlError *')) would lose const qualifier

Let’s replace it with `xmlResetLastError()`.

ALso remove `LIBXMLDOC::Xerr` protected member property.
It was introduced in 65b0e545
along with the `xmlResetError` calls.
It does not appear to be used for anything.

These changes are submitted under the BSD 3-clause License.

The original ticket describes a server crash when using a UDF in the WHERE clause of a view.  The crash also happens when using a UDF in the WHERE clause of a SELECT that uses a sub-query in the FROM clause.

When the UDF does not have a _deinit function the server crashes in udf_handler::cleanup (sql/item_func.cc:3467).
When the UDF has both an _init and a _deinit function but _init does not allocate memory for initid->ptr the server crashes in udf_handler::cleanup (sql/item_func.cc:3467).
When the UDF has both an _init and a _deinit function and allocates/deallocates  memory for initid->ptr the server crashes in the memory deallocation of the _deinit function.

The sequence of events seen are:
  1. A UDF, U, is created for the query.
  2. The UDF _init function is called using U->initid.
  3. U is cloned for the sub-query using the [default|implicit] copy constructor, resulting in V.
  4. The UDF _init function is called using V->initid.  U->initid and V->initid are the same value.
  5. The UDF function is called.
  6. The UDF _deinit function is called using U->initid.  If any memory was allocated for initid->ptr it is deallocated here.
  7. udf_handler::cleanup deletes the U->buffers String array.
  8. The UDF _deinit function is called using V->initid.  If any memory was allocated for initid->ptr it was previously deallocated and _deinit crashes the server.
  9. udf_handler::cleanup deletes the V->buffers String array. V->buffers was the same values as U->buffers which was already deallocated.  The server crashes.

The solution is to create a[n explicit] copy constructor for udf_handler which sets not_original to true.  Later, not_original is set back to false (0) after udf_handler::fix_fields has set up a new value for initid->ptr.

The `unused-but-set-variable` warning is raised on MacOS from the
`posix_fadvise` standin macro, since offset is often otherwise unused. Add a
cast to absorb this warning.
Signed-off-by: Trevor Gross <tmgross@umich.edu>

Also fix the usage of timeout(1) on NetBSD, OpenBSD and DragonFly.

ha_innobase::check_if_supported_inplace_alter(): Require ALGORITHM=COPY
when creating a FULLTEXT INDEX on a versioned table.

row_merge_buf_add(), row_merge_read_clustered_index(): Remove the parameter
or local variable history_fts that had been added in the attempt to fix
MDEV-25004.

Reviewed by: Thirunarayanan Balathandayuthapani
Tested by: Matthias Leich

After MDEV-31400, plugins are allowed to ask for retries when failing
initialisation. However, such failures also cause plugin system
variables to be deleted (plugin_variables_deinit()) before retrying
and are not re-added during retry.

We fix this by checking that if the plugin has requested a retry the
variables are not deleted. Because plugin_deinitialize() also calls
plugin_variables_deinit(), if the retry fails, the variables will
still be deleted.

Alternatives considered:

- remove the plugin_variables_deinit() from plugin_initialize() error
handling altogether. We decide to take a more conservative approach
here.

- re-add the system variables during retry. It is more complicated
than simply iterating over plugin->system_vars and call
my_hash_insert(). For example we will need to assign values to
the test_load field and extract more code from test_plugin_options(),
if that is possible.

fts_doc_id_cmp(): Replaces several duplicated functions for
comparing two doc_id_t*. On IA-32, AMD64, ARMv7, ARMv8, RISC-V
this should make use of some conditional ALU instructions.
On POWER there will be conditional jumps. Unlike the original
functions, these will return the correct result even if the
difference of the two doc_id does not fit in the int data type.
We use static_assert() and offsetof() to check at compilation time
that this function is compatible with the rbt_create() calls.

fts_query_compare_rank(): As documented, return -1 and not 1
when the rank are equal and r1->doc_id < r2->doc_id. This will
affect the result of ha_innobase::ft_read().

fts_ptr2_cmp(), fts_ptr1_ptr2_cmp(): These replace
fts_trx_table_cmp(), fts_trx_table_id_cmp().
The fts_savepoint_t::tables will be sorted by dict_table_t*
rather than dict_table_t::id. There was no correctness bug in
the previous comparison predicates. We can avoid one level of
unnecessary pointer dereferencing in this way.
Actually, fts_savepoint_t is duplicating trx_t::mod_tables.
MDEV-33401 was filed about removing it.

The added unit test innodb_rbt-t covers both the previous buggy comparison
predicate and the revised fts_doc_id_cmp(), using keys which led to
finding the bug. Thanks to Shaohua Wang from Alibaba for providing the
example and the revised comparison predicate.

Reviewed by: Thirunarayanan Balathandayuthapani

The innodb_changed_pages plugin only was part of XtraDB, never InnoDB.
It would be useful for incremental backups.
We will remove the code from mariadb-backup for now, because it cannot
serve any useful purpose until the server part has been implemented.

fts_doc_ids_sort(): Sort an array of doc_id_t by C++11 std::sort().

fts_doc_id_cmp(), ib_vector_sort(): Remove. The comparison was
returning an incorrect result when the difference exceeded the int range.

Reviewed by: Thirunarayanan Balathandayuthapani

This test was prone to failures for a few reasons, summarized below:

 1) MDEV-32168 introduced “only_running_threads=1” to
slave_stop.inc, which allowed the stop logic to bypass an
attempting-to-reconnect IO thread. That is, the IO thread could
realize the master shutdown in `read_event()`, and thereby call into
`try_to_reconnect()`. This would leave the IO thread up when the
test expected it to be stopped. Fixed by explicitly stopping the
IO thread and allowing an error state, as the above case would
lead to errno 2003.

 2) On slow systems (or those running profiling tools, e.g. MSAN),
the waiting-for-ack transaction can complete before the system
processes the `SHUTDOWN WAIT FOR ALL SLAVES`. There was shutdown
preparation logic in-between the transaction and shutdown itself,
which contributes to this problem. This patch also moves this
preparation logic before the transaction, so there is less to do
in-between the calls.

 3) Changed work-around for MDEV-28141 to use debug_sync instead
of sleep delay, as it was still possible to hit the bug on very
slow systems.

 4) Masked MTR variable reset with disable/enable query log

Reviewed By:
============
Kristian Nielsen <knielsen@knielsen-hq.org>

A race condition with the SQL thread, where depending on if it was
killed before or after it had executed the fake/generated IGN_GTIDS
Gtid_list_log_event, may or may not update gtid_slave_pos with the
position of the ignored events. Then, the slave would be restarted
while resetting IGNORE_DOMAIN_IDS to be empty, which would result in
the slave requesting different starting locations, depending on
whether or not gtid_slave_pos was updated. And, because previously
ignored events could now be requested and executed (no longer
ignored), their presence would fail the test.

This patch fixes this in two ways. First, to use GTID positions for
synchronization rather than binlog file positions. Then second, to
synchronize the SQL thread’s gtid_slave_pos with the ignored events
before killing the SQL thread.

To consistently reproduce the test failure, the following patch can
be applied:

diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc
index f51f5b7deec..de62233acff 100644
--- a/sql/log_event_server.cc
+++ b/sql/log_event_server.cc
@@ -3686,6 +3686,12 @@ Gtid_list_log_event::do_apply_event(rpl_group_info *rgi)
     void *hton= NULL;
     uint32 i;

+    sleep(1);
+    if (rli->sql_driver_thd->killed || rli->abort_slave)
+    {
+      return 0;
+    }
+

Reviewed By:
============
Kristian Nielsen <knielsen@knielsen-hq.org>

	JOIN::optimize_inner(), Condition pushdown from HAVING into WHERE
	  not shown in optimizer trace.

When executing a statement of the form
  SELECT AGGR_FN(DISTINCT c1, c2,..,cn) FROM t1,
where AGGR_FN is an aggregate function such as COUNT(), AVG() or SUM(),
and a unique index exists on table t1 covering some or all of the
columns (c1, c2,..,cn), the retrieved values are inherently unique.
Consequently, the need for de-duplication imposed by the DISTINCT
clause can be eliminated, leading to optimization of aggregation
operations.
This optimization applies under the following conditions:
  - only one table involved in the join (not counting const tables)
  - some arguments of the aggregate function are fields
        (not functions/subqueries)

This optimization extends to queries of the form
  SELECT AGGR_FN(c1, c2,..,cn) GROUP BY cx,..cy
when a unique index covers some or all of the columns
(c1, c2,..cn, cx,..cy)