Issue: During mtr_t:commit, if there is not enough space available in
redo log buffer, we flush the buffer. During flush, the LSN lock is
released allowing other concurrent mtr to commit. After flush we
reacquire the lock but use the old LSN obtained before check. It could
lead to redo log corruption. As the LSN moves backwards with the
possibility of data loss and unrecoverable server if the server aborts
for any reason or if server is shutdown with innodb_fast_shutdown=2.
With normal shutdown, recovery fails to map the checkpoint LSN to
correct offset.

In debug mode it hits log0log.cc:863: lsn_t log_t::write_buf()
Assertion `new_buf_free == ((lsn - first_lsn) & write_size_1)' failed.

In release mode, after normal shutdown, restart fails.
[ERROR] InnoDB: Missing FILE_CHECKPOINT(8416546) at 8416546
[ERROR] InnoDB: Log scan aborted at LSN 8416546

Backup fails reading the corrupt redo log.
[00] 2024-07-31 20:59:10 Retrying read of log at LSN=7334851
[00] FATAL ERROR: 2024-07-31 20:59:11 Was only able to copy log from
7334851 to 7334851, not 8416446; try increasing innodb_log_file_size

Unless a backup is tried or the server is shutdown or killed
immediately, the corrupt redo part is eventually truncated and there
may not be any visible issues seen in release mode.

This issue was introduced by the following commit.

commit a635c406
    MDEV-27774 Reduce scalability bottlenecks in mtr_t::commit()

Fix: If we need to release latch and flush redo before writing mtr
logs, make sure to get the latest system LSN after reacquiring the
redo system latch.

Fixes error:

/usr/share/mysql/debian-start.inc.sh: line 39: /usr/bin/mariadb --defaults-file=/etc/mysql/debian.cnf: No such file or directory

In commit bf0b82d2 (MDEV-33515)
the function log_t::init_lsn_lock() was removed. This was fine on
those platforms where InnoDB uses futex-based mutexes (Linux, FreeBSD,
OpenBSD, NetBSD, DragonflyBSD).

Dave Gosselin debugged this on Apple macOS and submitted a fix where
pthread_mutex_wrapper::pthread_mutex_wrapper() would invoke init().
We do not really need that; we only need to invoke lsn_lock.init()
like we used to do before commit bf0b82d2.
This should be a no-op for the futex based mutexes, which intentionally
rely on zero initialization.

The missing pthread_mutex_init() call would cause race conditions
and corruption of log_sys.buf because multiple threads could
apparently hold log_sys.lsn_lock concurrently in
log_t::append_prepare().  The error would be caught by a debug
assertion in log_t::write_buf(), or in non-debug builds by the
fact that the server cannot be restarted due to an apparently
missing FILE_CHECKPOINT record (because it had been written
to wrong offset in log_sys.buf).

The failure in log_t::append_prepare() was caught on Microsoft Windows
after enabling SUX_LOCK_GENERIC and therefore forcing the use of
pthread_mutex_wrapper for the log_sys.lsn_lock.  It appears to be fine
to omit the pthread_mutex_init() call on GNU/Linux.

log_t::create(): Invoke lsn_lock.init().

log_t::close(): Invoke lsn_lock.destroy().

To better catch this kind of issues in the future by simply defining
SUX_LOCK_GENERIC on any platform, a separate debug instrumentation patch
will be applied to the 10.6 branch later.

Reviewed by: Debarun Banerjee

- During copy algorithm, InnoDB should use bulk insert operation
for row by row insert operation. By doing this, copy algorithm
can effectively build indexes. This optimization is disabled
for temporary table, versioning table and table which has
foreign key relation.

Introduced the variable innodb_alter_copy_bulk to allow
the bulk insert operation for copy alter operation
inside InnoDB. This is enabled by default

ha_innobase::extra(): HA_EXTRA_END_ALTER_COPY mode tries to apply
the buffered bulk insert operation, updates the non-persistent
table stats.

row_merge_bulk_t::write_to_index(): Update stat_n_rows after
applying the bulk insert operation

row_ins_clust_index_entry_low(): In case of copy algorithm,
switch to bulk insert operation.

copy_data_error_ignore(): Handles the error while copying
the data from source to target file.

Added a missed
  --source include/save_master_gtid.inc
by the previous commit.

When mysqldump is run to dump the `mysql` system database, it generates
INSERT statements into the table `mysql.gtid_slave_pos`.
After running the backup script
those inserts did not produce the expected gtid state on slave. In
particular the maximum of mysql.gtid_slave_pos.sub_id did not make
into
   rpl_global_gtid_slave_state.last_sub_id

an in-memory object that is supposed to match the current state of the
table. And that was regardless of whether --gtid option was specified
or not. Later when the backup recipient server starts as slave
in *non-gtid* mode this desychronization may lead to a duplicate key
error.

This effect is corrected for --gtid mode mysqldump/mariadb-dump only
as the following.  The fixes ensure the insert block of the dump
script is followed with a "summing-up" SET @global.gtid_slave_pos
assignment.

For the implemenation part, note a deferred print-out of
SET-gtid_slave_pos and associated comments is prefered over relocating
of the entire blocks if (opt_master,slave_data &&
do_show_master,slave_status) ...  because of compatiblity
concern. Namely an error inside do_show_*() is handled in the new code
the same way, as early as, as before.

A regression test can be run in how-to-reproduce mode as well.
One affected mtr test observed.
rpl_mysqldump_slave.result "mismatch" shows now the new deferring print
of SET-gtid_slave_pos policy in action.

Additional tests for MDEV-28345 ASAN: use-after-poison or unknown-crash in my_strtod_int from charset_info_st::strntod or test_if_number

The test was missing a save_master_gtid.inc on the master,
leading to the slave thinking it was in sync after executing
sync_with_master_gtid.inc, despite not having executed the
latest transaction. This skipped transaction, XA COMMIT,
was supposed to error-to-be-ignored because its XID could not
be found, but be thrown out because the replication filters
would filter out the target database. However, if the slave
was able to stop before executing the transaction, then
the replication filer is reset (to empty), and when the
slave is later restarted, that transactions error would
no longer be ignored.

Additionally, as the test cases added in MDEV-33921 rely
on GTID synchronization, the test cases now force
master_use_gtid=slave_pos for consistency

This reverts commit 938b9293. Not needed after 90d376e0.

quote a database name in the slow log

remove it

refine mariadb-backup password zapping check

protect MariaDB conditional comments from a bug
in Percona MySQL comment parser

for ALTER_PARTITION_ADMIN (CHECK/REPAIR/LOAD INDEX/CACHE INDEX/etc)
partitioning marks affected partitions with PART_ADMIN state.

The assumption is that the server will call a corresponding
method of ha_partition which will reset the state back to PART_NORMAL.

This assumption is invalid, the server is not required to do so,
indeed, in CHECK ... FOR UPGRADE the server might decide early that
the table is fine and won't call ha_partition::check(), leaving
partitions in the wrong state. It will thus leak into the next
statement confusing the engine about what it is doing (see
ha_partition::create_handler_file()), causing a crash later.

Let's force all partitions into PART_NORMAL state after the admin
operation succeeded, in case it did so without consulting the engine.

There are 3 diff in result:
1) NULL value from SELECT
Due to incorrect truncating of the hex value, incorrect value is
written instead of original value to the view frm. This results in reading
incorrect value from frm, so eventual result is NULL.
2) 'Name_exp1' in column name (in gis.test)
This was because the identifier in SELECT is longer than 64 characters,
so 'Name_exp1' alias is also written to the view frm.
3)diff in explain extended
This was because the query plan for view protocol doesn't
contain database name. As a fix, disable view protocol for that particular
query.

Import only the required functions instead of all the functions from the
module to reduce the unnecessary functions in the namespace and prevent
shadowing. Note: All code changes are non-functional.

All new code of the whole pull request, including one or several
files that are either new files or modified ones, are contributed
under the BSD-new license. I am contributing on behalf of my
employer Amazon Web Services, Inc.

> After May 31, 2024, CentOS Stream 8 will be archived and no further
updates will be provided. [1]

CentOS Stream 8 is now EOL and should be updated to using CentOS Stream
9 for compatibility testing in GitLab CI.

[1] https://blog.centos.org/2023/04/end-dates-are-coming-for-centos-stream-8-and-centos-linux-7/
https://www.centos.org/centos-linux-eol/

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services.

MDEV-28345 ASAN: use-after-poison or unknown-crash in my_strtod_int from charset_info_st::strntod or test_if_number

This patch fixes two problems:

- The code inside my_strtod_int() in strings/dtoa.c could test the byte
  behind the end of the string when processing the mantissa.
  Rewriting the code to avoid this.

- The code in test_if_number() in sql/sql_analyse.cc called my_atof()
  which is unsafe and makes the called my_strtod_int() look behind
  the end of the string if the input string is not 0-terminated.
  Fixing test_if_number() to use my_strtod() instead, passing the correct
  end pointer.

Mroonga UDFs can't be used without loading Mroonga.

As is everywhere in mariadbd is used it more than
convienient to use mariadbd-safe than mysql_safe
in init script also in upstream test use output
mariadb-test-run-junit.xml than mysql-test-run-junit.xml

Conflict with Debian package libmariadbd19t64 as it marks
that package is compliant with 64-bit time and does not
suffer from year 2038 effect

Update changes from Debian Salsa that they are
in sync

autopkgtests from package are not passing currently and make them pass with
upgrading Salsa-CI YAML file.

For making smoke test work sync current Debian Salsa-CI version
for making needed changes

Make all init.d script for loops to use new
{1..5} syntax and rework one not to use seq as
all the rest use new Bash syntax.

Make small adjustment to MyISAM recovery function
SQL statement and how to handle it.

MariaDB installs/upgrades in Docker containers (and elsewhere where
systemd is not used) occasionally fail with output like:

  Starting MariaDB database server: mariadbd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . failed!
  invoke-rc.d: initscript mariadb, action "start" failed.
  dpkg: error processing package mariadb-server-10.5 (--configure):
   installed mariadb-server-10.5 package post-installation script subprocess returned error exit status 1

This is not very helpful. Thus extend the init script to try the server
start/restart one more time but with error log defined separately,
and then print out the error log contents of this single start attempt.

  ...
  Starting MariaDB database server: mariadbd . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
  230103 01:06:48 mysqld_safe Can't log to error log and syslog at the same time.  Remove all --log-error configuration options for --syslog to take effect.
  230103 01:06:48 mysqld_safe Logging to '/tmp/tmp.JlE4sdUMZz.err'.
  230103 01:06:49 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql
  Running '/etc/init.d/mariadb start' failed with error log:
  230103 01:06:49 mysqld_safe Starting mariadbd daemon with databases from /var/lib/mysql
  2023-01-03  1:06:49 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.18-MariaDB-0+deb11u1) starting as process 10417 ...
  2023-01-03  1:06:49 0 [Note] InnoDB: Uses event mutexes
  2023-01-03  1:06:49 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
  2023-01-03  1:06:49 0 [Note] InnoDB: Number of pools: 1
  2023-01-03  1:06:49 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
  2023-01-03  1:06:49 0 [Note] InnoDB: Using Linux native AIO
  2023-01-03  1:06:49 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
  2023-01-03  1:06:49 0 [Note] InnoDB: Completed initialization of buffer pool
  2023-01-03  1:06:49 0 [ERROR] InnoDB: Invalid flags 0x4800 in ./ibdata1
  ...

Adapted from upstream commit 8171f9da but separated only the datadir
section from the commit and wrote it in a way that does not trigger
Shellcheck or English grammar nags.

This check is intentionally not added to the preinst script as was done
upstream in 30fb72ca as the preinst script will always create the
data directory if missing, and thus checking for it right after the
creation is moot.

Fix a large amount of minor fixes to maintainer scripts and other done
downstream in the official Debian packaging.

Changes include:

https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/38198d0b9e1c7821ddd074e308b25034bdcdce5b
> Limit check of running mysqld/mariadbd to system users (Closes: #1032047)
>
> If a random user has their own copy of mysqld/mariadbd running, the
> dpkg maintainer script should not care about it.

https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/8116354d22e0b8eade6d8f0594c57300d5d5cff5
> Make error more helpful in case server restart fails (Related: #1033234)
>
> Bugs such as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1033234
> and https://bugs.launchpad.net/ubuntu/+source/mariadb-10.6/+bug/2011293
> show that currently dpkg stopping on service stop/start does not have
> a very helpful error message.

https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/8675e97202171812a1afdb438a17cb29a99836fb
> Complement upstream commits with more complete mysql->mariadb conversion
>
> The upstream commit 952af4a1 missed some places where 'mysql' or
> 'MySQL' can and should be converted to use 'mariadb' or 'MariaDB'.

https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/c98361330063e7dccbf8d21aa20e48179ba5c1e4
> Fix indentation in Debian post and pre scripts
>
> There is several misindentation inside Debian post and pre
> installation scripts. False indentation with space as indent space
> should be 2 and indentation with tabs.
>
> Adopt upstream commit 7cbb45d1 in Debian by conserving customizations
> in:
> - debian/mariadb-server.postinst
> - debian/mariadb-server.postrm
> - debian/mariadb-server.preinst

https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/d0bcab443fa6d44084dc674ba29b79516c6239ba
> Ensure spaces are used everywhere instead of tabs for indentation

https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/0300a9157cc69f75e01ac9c0d6e033d8be661492
> Complement previous upstream commits to fix Shellcheck issues
>
> - Unify if/then and while/do on separate lines
> - Fix indentation to be consistent
> - Use "$()" instead of backticks for subshells
> - Exit code cannot be -1, must be 0-255
> - Remove unused variables MYCHECK and MYCHECK_PARAMS
> - Rewrite messy command-line database calls to an easier to read form
>   that does exactly the same
> - Use 'command -v' test instead of 'which'
>
> With this commit, all of debian/* is Shellcheck clean.

Also
* Update mariadb.conf.d template to tell users where to create logdir
  if they are not using journald
* Remove use of work 'slave'
* Add minor workaround for Debian Bug #1022994 if TMPDIR is empty
* Make start/stop in maintainer scripts correctly check mariadbd
  ownership and only start/stop processes owned by root or 'mysql'
* Remove obsolete 'NO_UPDATE_BUILD_VERSION=1' as it did not affect the
  RocksDB build reproducibility as previously assumed
* Run 'wrap-and-sort -av'