Before doing mark_start_commit(), check that there is no pending deadlock
kill. If there is a pending kill, we won't commit (we will abort, roll back,
and retry). Then we should not mark the commit as started, since that could
potentially make the following GCO start too early, before we completed the
commit after the retry.

This condition could trigger in some corner cases, where InnoDB would take
temporarily table/row locks that are released again immediately, not held
until the transaction commits. This happens with dict_stats updates and
possibly auto-increment locks.

Such locks can be passed to thd_rpl_deadlock_check() and cause a deadlock
kill to be scheduled in the background. But since the blocking locks are
held only temporarily, then can be released before the background kill
happens. This way, the kill can be delayed until after mark_start_commit()
has been called. Thus we need to check the synchronous indication
rgi->killed_for_retry, not just the asynchroneous thd->killed.
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

Some cleanup, and try to diagnose ERROR_SHARING_VIOLATION
in the innodb.log_file_size_online test

trx_flush_log_if_needed(): Do not pass a callback to non-durable
memory-mapped write.

When using the default innodb_log_buffer_size=2m, mariadb-backup --backup
would spend a lot of time re-reading and re-parsing the log. For reads,
it would be beneficial to memory-map the entire ib_logfile0 to the
address space (typically 48 bits or 256 TiB) and read it from there,
both during --backup and --prepare.

We will introduce the Boolean SET GLOBAL innodb_log_file_mmap
that will be OFF by default, but ON by default in mariadb-backup
when not running on Microsoft Windows. On Microsoft Windows,
setting innodb_log_file_mmap=ON may cause aggressive read-ahead of
the entire ib_logfile0 in when only a tiny portion would be accessed.
On Linux, the read-ahead would have to be explicitly enabled by
specifying MAP_POPULATE to mmap(2). Because we avoid MAP_POPULATE,
the file will be read on demand.

The setting innodb_log_file_mmap=ON is available also in the server.
This could speed up I/O and allow the log data to be shared between
mariadbd and mariadb-backup --backup in the RAM buffer.
Note: It might not be advisable to enable memory-mapped log writes
while backup is not running. It could make sense with a small
innodb_log_file_size that fits in RAM.

Memory-mapped regular files differ from log_sys.is_pmem() in the way
that an open file handle to ib_logfile0 will be retained. That allows
log_t::set_mmap() to enable or disable the interface with fewer
operations.

On log checkpoint we will invoke madvise() with MADV_DONTNEED in order
to reduce the memory pressure. This could lead to reads of old
garbage contents of the circular log file when a page fault occurs
while writing a record. There does not seem to be any way around this;
on Linux, invoking fallocate() with FALLOC_FL_ZERO_RANGE would make
things even worse by triggering additional metadata writes.

Most references to HAVE_PMEM or log_sys.is_pmem() are replaced with
HAVE_INNODB_MMAP or log_sys.is_mmap(). The main difference is that
PMEM skips the use of write_lock and flush_lock and uses pmem_persist(),
while the memory-mapped interface will use a combination of msync()
and fdatasync().

Starting with Linux 2.6.19, msync(MS_ASYNC) is a no-op, so we will not
invoke it on Linux. For durable writes, we will invoke msync(MS_SYNC).

In commit bf0b82d2 (MDEV-33515)
the function log_t::init_lsn_lock() was removed. This was fine on
those platforms where InnoDB uses futex-based mutexes (Linux, FreeBSD,
OpenBSD, NetBSD, DragonflyBSD).

Dave Gosselin debugged this on Apple macOS and submitted a fix where
pthread_mutex_wrapper::pthread_mutex_wrapper() would invoke init().
We do not really need that; we only need to invoke lsn_lock.init()
like we used to do before commit bf0b82d2.
This should be a no-op for the futex based mutexes, which intentionally
rely on zero initialization.

The missing pthread_mutex_init() call would cause race conditions
and corruption of log_sys.buf because multiple threads could
apparently hold log_sys.lsn_lock concurrently in
log_t::append_prepare().  The error would be caught by a debug
assertion in log_t::write_buf(), or in non-debug builds by the
fact that the server cannot be restarted due to an apparently
missing FILE_CHECKPOINT record (because it had been written
to wrong offset in log_sys.buf).

The failure in log_t::append_prepare() was caught on Microsoft Windows
after enabling SUX_LOCK_GENERIC and therefore forcing the use of
pthread_mutex_wrapper for the log_sys.lsn_lock.  It appears to be fine
to omit the pthread_mutex_init() call on GNU/Linux.

log_t::create(): Invoke lsn_lock.init().

log_t::close(): Invoke lsn_lock.destroy().

To better catch this kind of issues in the future by simply defining
SUX_LOCK_GENERIC on any platform, a separate debug instrumentation patch
will be applied to the 10.6 branch later.

Reviewed by: Debarun Banerjee

- During copy algorithm, InnoDB should use bulk insert operation
for row by row insert operation. By doing this, copy algorithm
can effectively build indexes. This optimization is disabled
for temporary table, versioning table and table which has
foreign key relation.

Introduced the variable innodb_alter_copy_bulk to allow
the bulk insert operation for copy alter operation
inside InnoDB. This is enabled by default

ha_innobase::extra(): HA_EXTRA_END_ALTER_COPY mode tries to apply
the buffered bulk insert operation, updates the non-persistent
table stats.

row_merge_bulk_t::write_to_index(): Update stat_n_rows after
applying the bulk insert operation

row_ins_clust_index_entry_low(): In case of copy algorithm,
switch to bulk insert operation.

copy_data_error_ignore(): Handles the error while copying
the data from source to target file.

Added a missed
  --source include/save_master_gtid.inc
by the previous commit.

When mysqldump is run to dump the `mysql` system database, it generates
INSERT statements into the table `mysql.gtid_slave_pos`.
After running the backup script
those inserts did not produce the expected gtid state on slave. In
particular the maximum of mysql.gtid_slave_pos.sub_id did not make
into
   rpl_global_gtid_slave_state.last_sub_id

an in-memory object that is supposed to match the current state of the
table. And that was regardless of whether --gtid option was specified
or not. Later when the backup recipient server starts as slave
in *non-gtid* mode this desychronization may lead to a duplicate key
error.

This effect is corrected for --gtid mode mysqldump/mariadb-dump only
as the following.  The fixes ensure the insert block of the dump
script is followed with a "summing-up" SET @global.gtid_slave_pos
assignment.

For the implemenation part, note a deferred print-out of
SET-gtid_slave_pos and associated comments is prefered over relocating
of the entire blocks if (opt_master,slave_data &&
do_show_master,slave_status) ...  because of compatiblity
concern. Namely an error inside do_show_*() is handled in the new code
the same way, as early as, as before.

A regression test can be run in how-to-reproduce mode as well.
One affected mtr test observed.
rpl_mysqldump_slave.result "mismatch" shows now the new deferring print
of SET-gtid_slave_pos policy in action.

Additional tests for MDEV-28345 ASAN: use-after-poison or unknown-crash in my_strtod_int from charset_info_st::strntod or test_if_number

The test was missing a save_master_gtid.inc on the master,
leading to the slave thinking it was in sync after executing
sync_with_master_gtid.inc, despite not having executed the
latest transaction. This skipped transaction, XA COMMIT,
was supposed to error-to-be-ignored because its XID could not
be found, but be thrown out because the replication filters
would filter out the target database. However, if the slave
was able to stop before executing the transaction, then
the replication filer is reset (to empty), and when the
slave is later restarted, that transactions error would
no longer be ignored.

Additionally, as the test cases added in MDEV-33921 rely
on GTID synchronization, the test cases now force
master_use_gtid=slave_pos for consistency

This reverts commit 938b9293. Not needed after 90d376e0.

quote a database name in the slow log

remove it

refine mariadb-backup password zapping check

protect MariaDB conditional comments from a bug
in Percona MySQL comment parser

for ALTER_PARTITION_ADMIN (CHECK/REPAIR/LOAD INDEX/CACHE INDEX/etc)
partitioning marks affected partitions with PART_ADMIN state.

The assumption is that the server will call a corresponding
method of ha_partition which will reset the state back to PART_NORMAL.

This assumption is invalid, the server is not required to do so,
indeed, in CHECK ... FOR UPGRADE the server might decide early that
the table is fine and won't call ha_partition::check(), leaving
partitions in the wrong state. It will thus leak into the next
statement confusing the engine about what it is doing (see
ha_partition::create_handler_file()), causing a crash later.

Let's force all partitions into PART_NORMAL state after the admin
operation succeeded, in case it did so without consulting the engine.

There are 3 diff in result:
1) NULL value from SELECT
Due to incorrect truncating of the hex value, incorrect value is
written instead of original value to the view frm. This results in reading
incorrect value from frm, so eventual result is NULL.
2) 'Name_exp1' in column name (in gis.test)
This was because the identifier in SELECT is longer than 64 characters,
so 'Name_exp1' alias is also written to the view frm.
3)diff in explain extended
This was because the query plan for view protocol doesn't
contain database name. As a fix, disable view protocol for that particular
query.

Import only the required functions instead of all the functions from the
module to reduce the unnecessary functions in the namespace and prevent
shadowing. Note: All code changes are non-functional.

All new code of the whole pull request, including one or several
files that are either new files or modified ones, are contributed
under the BSD-new license. I am contributing on behalf of my
employer Amazon Web Services, Inc.

> After May 31, 2024, CentOS Stream 8 will be archived and no further
updates will be provided. [1]

CentOS Stream 8 is now EOL and should be updated to using CentOS Stream
9 for compatibility testing in GitLab CI.

[1] https://blog.centos.org/2023/04/end-dates-are-coming-for-centos-stream-8-and-centos-linux-7/
https://www.centos.org/centos-linux-eol/

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services.

MDEV-28345 ASAN: use-after-poison or unknown-crash in my_strtod_int from charset_info_st::strntod or test_if_number

This patch fixes two problems:

- The code inside my_strtod_int() in strings/dtoa.c could test the byte
  behind the end of the string when processing the mantissa.
  Rewriting the code to avoid this.

- The code in test_if_number() in sql/sql_analyse.cc called my_atof()
  which is unsafe and makes the called my_strtod_int() look behind
  the end of the string if the input string is not 0-terminated.
  Fixing test_if_number() to use my_strtod() instead, passing the correct
  end pointer.

Mroonga UDFs can't be used without loading Mroonga.

As is everywhere in mariadbd is used it more than
convienient to use mariadbd-safe than mysql_safe
in init script also in upstream test use output
mariadb-test-run-junit.xml than mysql-test-run-junit.xml

Conflict with Debian package libmariadbd19t64 as it marks
that package is compliant with 64-bit time and does not
suffer from year 2038 effect

Update changes from Debian Salsa that they are
in sync

autopkgtests from package are not passing currently and make them pass with
upgrading Salsa-CI YAML file.

For making smoke test work sync current Debian Salsa-CI version
for making needed changes

Make all init.d script for loops to use new
{1..5} syntax and rework one not to use seq as
all the rest use new Bash syntax.

Make small adjustment to MyISAM recovery function
SQL statement and how to handle it.