The log_sys.lsn_lock is a very contended resource with a small
critical section in log_sys.append_prepare(). On many processor
microarchitectures, replacing the system call based log_sys.lsn_lock
with a pure spin lock would fare worse during high concurrency workloads,
wasting a significant amount of CPU cycles in the spin loop.

On other microarchitectures, we would see a significant amount of time
being spent in native_queued_spin_lock_slowpath() in the Linux kernel,
plus context switching between user and kernel address space. This was
pointed out by Steve Shaw from Intel Corporation.

Depending on the workload and the hardware implementation, it may be
useful to use a pure spin lock in log_sys.append_prepare().
We will introduce a parameter. The statement

	SET GLOBAL INNODB_LOG_SPIN_WAIT_DELAY=50;

would enable a spin lock that will execute that many MY_RELAX_CPU()
operations (such as the x86 PAUSE instruction) between successive
attempts of acquiring the spin lock. The use of a system call based
log_sys.lsn_lock (which is the default setting) can be enabled by

	SET GLOBAL INNODB_LOG_SPIN_WAIT_DELAY=0;

This patch will also introduce #ifdef LOG_LATCH_DEBUG
(part of cmake -DWITH_INNODB_EXTRA_DEBUG=ON) for more accurate
tracking of log_sys.latch ownership and reorganize the fields of
log_sys to improve the locality of reference and to reduce the
chances of false sharing.

When a spin lock is being used, it will be maintained in the
most significant bit of log_sys.buf_free. This is useful, because that is
one of the fields that is covered by the lock. For IA-32 or AMD64, we
implement the spin lock specially via log_t::lsn_lock_bts(), employing the
i386 LOCK BTS instruction. A straightforward std::atomic::fetch_or() would
translate into an inefficient loop around LOCK CMPXCHG.

mtr_t::spin_wait_delay: The value of innodb_log_spin_wait_delay.

mtr_t::finisher: Pointer to the currently used mtr_t::finish_write()
implementation. This allows to avoid introducing conditional branches.
We no longer invoke log_sys.is_pmem() at the mini-transaction level,
but we would do that in log_write_up_to().

mtr_t::finisher_update(): Update finisher when spin_wait_delay is
changed from or to 0 (the spin lock is changed to log_sys.lsn_lock or
vice versa).

remove bunch of  *.rej and *.orig files, checked in by mistake.

Remove incorrect deprecation.
Reviewed-by: Brandon Nesterenko <brandon.nesterenko@mariadb.com>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

The patch for MDEV-15530 incorrectly added a column in the middle of SHOW
SLAVE STATUS output. This is wrong, as it breaks backwards compatibility
with existing applications and scripts. In this case, it even broke
mariadb-dump, which is included in the server source tree!

Revert the incorrect change, putting the new Replicate_Rewrite_DB at the end
of SHOW SLAVE STATUS output.

Add a testcase for the mariadb-dump --dump-slave wrong output problem. Also
add a testcase rpl.rpl_show_slave_status to hopefully prevent any future
incorrect additions to SHOW SLAVE STATUS.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Code taken from MDEV-33423: show_analyze sporadically fails

Thanks to Sergei Petrunia for pointing this out.

This was confusing when mtr failed as it made it confusing where the
failure could be.

This commit fixes the following issues:
- memory leak checking enabled for mysqltest. This cover all cases except
  calls to 'die()' that only happens in case of internal failures in
  mysqltest. die() is not called anymore in the result files differs.
- One can now run mtr --embedded without failures (this crashed or hang
  before)
- cleanup_and_exit() has a new parameter that indicates that it is called
  from die(), in which case we should not do memory leak checks. We now
  always call cleanup_and_exit() instead of exit() to be able to free up
  memory and discover memory leaks.
- Lots of new assert to catch error conditions
- More DBUG statements.
- Fixed that all results are freed in mysqltest (Fixed a memory leak in
  mysqltest when using prepared statements).
- Fixed race condition in do_stmt_close() that caused embedded server
  to not free memory. (Memory leak in mysqltest with embedded server).
- Fixed two memory leaks in embedded server when using prepared statements.
  These memory leaks caused timeout hangs in mtr when server was compiled
  with safemalloc. This issue was not noticed (except as timeouts) as
  memory report checking was done but output of it was disabled.

If mariabackup with backup locks is used on SST we do not
pause and desync galera provider at all. If WSREP_MODE_BF_MARIABACKUP
case provider is paused and desync at BLOCK_COMMIT phase. In
other cases provider is paused and desync at BLOCK_DDL phase.

Problem is that not all conflicting transactions have THD object.
Therefore, it must be checked that victim has THD
before it's identification is added to victim list as victim's
thread identification is later requested using thd_get_thread_id
function that requires that we have valid pointer to THD object
in trx->mysql_thd.

Victim might not have trx->mysql_thd in two cases:

(1) An incomplete transaction that was recovered from undo logs
on server startup (and not yet rolled back).

(2) Transaction that is in XA PREPARE state and whose client
connection was disconnected.

Neither of these can complete before lock_wait_wsrep()
releases lock_sys.latch.

(1) trx_t::commit_in_memory() is clearing both
trx_t::state and trx_t::is_recovered before it invokes
lock_release(trx_t*) (which would be blocked by the exclusive
lock_sys.latch that we are holding here). Hence, it is not
possible to write a debug assertion to document this scenario.

(2) If is in XA PREPARE state, it would eventually be rolled
back and the lock conflict would be resolved when an XA COMMIT
or XA ROLLBACK statement is executed in some other connection.

Other things:
- Disabled not supported options
- Updated copyright message

Added support to BACKUP STAGE to maria-backup

This is a port of the code from ES 10.6
See MDEV-5336 for backup stages description.

The following old options are not supported by the new code:
--rsync             ; This is because rsync will not work on tables
                      that are in used.
--no-backup-locks   ; This is disabled as mariadb-backup will always
                      use backup locks for better performance.

- Updated prototype for is_binary_frm_header().
- Added extra argument to ma_control_file_open().
- Added ma_control_file_open_or_create() for usage by tests.
  (to make test a bit simpler).

Problem:
REPAIR TABLE executed for a pre-MDEV-29959 table (with the old UUID format)
updated the server version in the FRM file without rewriting the data,
so it created a new FRM for old UUIDs. After that MariaDB could not
read UUIDs correctly.

Fix:

- Adding a new virtual method in class Type_handler:

      virtual bool type_handler_for_implicit_upgrade() const;

  * For the up-to-date data types it returns "this".
  * For the data types which need to be implicitly upgraded
    during REPAIR TABLE or ALTER TABLE, it returns a pointer
    to a new replacement data type handler.

    Old VARCHAR and old UUID type handlers override this method.
    See more comments below.

- Changing the semantics of the method

    Type_handler::Column_definition_implicit_upgrade(Column_definition *c)

  to the opposite, so now:
    * c->type_handler() references the old data type (to upgrade from)
    * "this" references the new data type (to upgrade to).

  Before this change Column_definition_implicit_upgrade() was supposed
  to be called with the old data type handler (to upgrade from).

  Renaming the method to Column_definition_implicit_upgrade_to_this(),
  to avoid automatic merges in this method.

  Reflecting this change in Create_field::upgrade_data_types().

- Replacing the hard-coded data type tests inside handler::check_old_types()
  to a call for the new virtual method
  Type_handler::type_handler_for_implicit_upgrade()

- Overriding Type_handler_fbt::type_handler_for_implicit_upgrade()
  to call a new method FbtImpl::type_handler_for_implicit_upgrade().

  Reasoning:

  Type_handler_fbt is a template, so it has access only to "this".
  So in case of UUID data types, the type handler for old UUID
  knows nothing about the type handler of new UUID inside sql_type_fixedbin.h.
  So let's have Type_handler_fbt delegate type_handler_for_implicit_upgrade()
  to its Type_collection, which knows both new UUID and old UUID.

- Adding Type_collection_uuid::type_handler_for_implicit_upgrade().
  It returns a pointer to the new UUID type handler.

- Overriding Type_handler_var_string::type_handler_for_implicit_upgrade()
  to return a pointer to type_handler_varchar (true VARCHAR).

- Cleanup: these two methods:
    handler::check_old_types()
    handler::ha_check_for_upgrade()
  were always called consequently.
  So moving the call for check_old_types() inside ha_check_for_upgrade(),
  and making check_old_types() private.

- Cleanup: removing the "bool varchar" parameter from fill_alter_inplace_info(),
  as its not used any more.

srw_lock_debug::have_rd(), srw_lock_debug::have_wr():
For SUX_LOCK_GENERIC (no futex based synchronization primitives),
we cannot check if the underlying srw_lock is held by us.

Thanks to Dmitry Shulga for pointing out this build failure.

Problem:
========
Currently import operation fails with schema mismatch when
cfg file has hidden fts document id and hidden fts document index.

Fix:
====
To fix this issue, simply add the fts doc id column,
indexes in table definition and try to import the table.
In case of success:
1) update the fts document id in sys columns.
2) update the number of columns in sys tables.
3) insert the new fts index entry in sys indexes table
and sys fields.
4) Reload the table with new table definition

Ever since commit 412ee033
or commit a440d6ed
InnoDB should generally not abort when failing to open or create files.
In Datafile::open_or_create() we had failed to set the flag
to avoid abort() on failure, but everywhere else we were setting it.

We may still call abort() via os_file_handle_error().

Reviewed by: Vladislav Vaintroub

Apparently, invoking fcntl(fd, F_SETFL, O_DIRECT) will lead to
unexpected behaviour on Linux bcachefs and possibly other file systems,
depending on the operating system version. So, let us avoid doing that,
and instead just attempt to pass the O_DIRECT flag to open(). This should
make us compatible with NetBSD, IBM AIX, as well as Solaris and its
derivatives.

This fix does not change the fact that we had only implemented
innodb_log_file_buffering=OFF on systems where we can determine the
physical block size (typically 512 or 4096 bytes).
Currently, those operating systems are Linux and Microsoft Windows.

HAVE_FCNTL_DIRECT, os_file_set_nocache(): Remove.

OS_FILE_OVERWRITE, OS_FILE_CREATE_PATH: Remove (never used parameters).

os_file_log_buffered(), os_file_log_maybe_unbuffered(): Helper functions.

os_file_create_simple_func(): When applicable, initially attempt to
open files in O_DIRECT mode.

os_file_create_func(): When applicable, initially attempt to
open files in O_DIRECT mode.
For type==OS_LOG_FILE && create_mode != OS_FILE_CREATE
we will first invoke stat(2) on the file name to find out if the size
is compatible with O_DIRECT. If create_mode == OS_FILE_CREATE, we will
invoke fstat(2) on the created log file afterwards, and may close and
reopen the file in O_DIRECT mode if applicable.

create_temp_file(): Support O_DIRECT. This is only used if O_TMPFILE is
available and innodb_disable_sort_file_cache=ON (non-default value).
Notably, that setting never worked on Microsoft Windows.

row_merge_file_create_mode(): Split from row_merge_file_create_low().
Create a temporary file in the specified mode.

Reviewed by: Vladislav Vaintroub

The effect is reduced performance, at least on hybrid processor systems.

To fix, turn off throttling power explicitly at server startup.

MDEV-33363 CI failure: innodb.import_corrupted: Assertion failed: oldest_lsn > log_sys.last_checkpoint_lsn

This regression is introduced in MDEV-28708 where the MTR_LOG_NO_REDO
mtrs are assigned last_checkpoint_lsn as the LSN. It causes a race with
checkpoint in pending state. The concurrent checkpoint writes a
checkpoint LSN of larger value after pages with older checkpoint LSN is
inserted into the flush list. The next checkpoint sees the reversal in
checkpoint sequence and asserts if the pages are not yet flushed.

There could be several ways to solve this issue. Ideally the unlogged
mtr should take the latest LSN as opposed to going behind and use the
previous checkpoint LSN. It has been the older design and seems good.
Also, other than the critical race, using the old checkpoint LSN adds
the pages to other end of flush list overriding all existing dirty
pages and looks counter intuitive.

purge_sys_t::truncating_tablespace(): Clamp the
innodb_max_undo_log_size to the maximum number of pages
before converting the result into a 32-bit unsigned integer.

This fixes up commit f8c88d90 (MDEV-33213).

In later major versions, we would use 32-bit unsigned integer here
due to commit ca501ffb
and the code would crash also on 64-bit processors.

Reviewed by: Debarun Banerjee

buf_read_ahead_linear(): If buf_pool.watch_is_sentinel(*bpage),
do not attempt to read the page frame because the pointer would be null
for the elements of buf_pool.watch[].

Hitting this bug requires the use of a non-default value of
innodb_change_buffering.

The script wsrep_sst_backup was introduced on MariaDB 10.3 in commit
9b2fa2ae. The new script was automatically included in RPM packages but not
in Debian packages (which started to fail on warning about stray file).

Include wsrep_sst_backup in the mariadb-server-10.5+ package, and also
include a stub man page so that packaging of a new script is complete.

Related:
https://galeracluster.com/documentation/html_docs_20210213-1355-master/documentation/backup-cluster.html

This commit was originally submitted in May 2022 in
https://github.com/MariaDB/server/pull/2129 but upstream indicated only
in May 2023 that it might get merged, thus this is for a later release.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.

libxml2 2.12.0 made `xmlGetLastError()` return `const` pointer:

https://gitlab.gnome.org/GNOME/libxml2/-/commit/61034116d0a3c8b295c6137956adc3ae55720711

Clang 16 does not like this:

    error: assigning to 'xmlErrorPtr' (aka '_xmlError *') from 'const xmlError *' (aka 'const _xmlError *') discards qualifiers
    error: cannot initialize a variable of type 'xmlErrorPtr' (aka '_xmlError *') with an rvalue of type 'const xmlError *' (aka 'const _xmlError *')

Let’s update the variables to `const`.
For older versions, it will be automatically converted.

But then `xmlResetError(xmlError*)` will not like the `const` pointer:

    error: no matching function for call to 'xmlResetError'
    note: candidate function not viable: 1st argument ('const xmlError *' (aka 'const _xmlError *')) would lose const qualifier

Let’s replace it with `xmlResetLastError()`.

ALso remove `LIBXMLDOC::Xerr` protected member property.
It was introduced in 65b0e545
along with the `xmlResetError` calls.
It does not appear to be used for anything.

These changes are submitted under the BSD 3-clause License.

The original ticket describes a server crash when using a UDF in the WHERE clause of a view.  The crash also happens when using a UDF in the WHERE clause of a SELECT that uses a sub-query in the FROM clause.

When the UDF does not have a _deinit function the server crashes in udf_handler::cleanup (sql/item_func.cc:3467).
When the UDF has both an _init and a _deinit function but _init does not allocate memory for initid->ptr the server crashes in udf_handler::cleanup (sql/item_func.cc:3467).
When the UDF has both an _init and a _deinit function and allocates/deallocates  memory for initid->ptr the server crashes in the memory deallocation of the _deinit function.

The sequence of events seen are:
  1. A UDF, U, is created for the query.
  2. The UDF _init function is called using U->initid.
  3. U is cloned for the sub-query using the [default|implicit] copy constructor, resulting in V.
  4. The UDF _init function is called using V->initid.  U->initid and V->initid are the same value.
  5. The UDF function is called.
  6. The UDF _deinit function is called using U->initid.  If any memory was allocated for initid->ptr it is deallocated here.
  7. udf_handler::cleanup deletes the U->buffers String array.
  8. The UDF _deinit function is called using V->initid.  If any memory was allocated for initid->ptr it was previously deallocated and _deinit crashes the server.
  9. udf_handler::cleanup deletes the V->buffers String array. V->buffers was the same values as U->buffers which was already deallocated.  The server crashes.

The solution is to create a[n explicit] copy constructor for udf_handler which sets not_original to true.  Later, not_original is set back to false (0) after udf_handler::fix_fields has set up a new value for initid->ptr.

While a -f debian/mariadb-plugin-columnstore.install idempotent check
existed, the tying of the install file to the control file has some
weaknesses.

Used sed as an alternative to replace the debian/control
mariadb-plugin-columnstore package defination and replace it with the
one from the columnstore submodule.

Per MDEV-20601, REPLICA should be an alias for SLAVE in SQL
statements.

Reviewed By:
============
Kristian Nielsen <knielsen@knielsen-hq.org>
Andrei Elkin <andrei.elkin@mariadb.com>

The `unused-but-set-variable` warning is raised on MacOS from the
`posix_fadvise` standin macro, since offset is often otherwise unused. Add a
cast to absorb this warning.
Signed-off-by: Trevor Gross <tmgross@umich.edu>

The `--mariadbd` option was added in 10.11.3, this commit adds the option
to the man page as well.

Also fix the usage of timeout(1) on NetBSD, OpenBSD and DragonFly.

ha_innobase::check_if_supported_inplace_alter(): Require ALGORITHM=COPY
when creating a FULLTEXT INDEX on a versioned table.

row_merge_buf_add(), row_merge_read_clustered_index(): Remove the parameter
or local variable history_fts that had been added in the attempt to fix
MDEV-25004.

Reviewed by: Thirunarayanan Balathandayuthapani
Tested by: Matthias Leich