Fix a large amount of minor fixes to maintainer scripts and other done
downstream in the official Debian packaging.

Changes include:

https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/38198d0b9e1c7821ddd074e308b25034bdcdce5b
> Limit check of running mysqld/mariadbd to system users (Closes: #1032047)
>
> If a random user has their own copy of mysqld/mariadbd running, the
> dpkg maintainer script should not care about it.

https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/8116354d22e0b8eade6d8f0594c57300d5d5cff5
> Make error more helpful in case server restart fails (Related: #1033234)
>
> Bugs such as https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1033234
> and https://bugs.launchpad.net/ubuntu/+source/mariadb-10.6/+bug/2011293
> show that currently dpkg stopping on service stop/start does not have
> a very helpful error message.

https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/8675e97202171812a1afdb438a17cb29a99836fb
> Complement upstream commits with more complete mysql->mariadb conversion
>
> The upstream commit 952af4a1 missed some places where 'mysql' or
> 'MySQL' can and should be converted to use 'mariadb' or 'MariaDB'.

https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/c98361330063e7dccbf8d21aa20e48179ba5c1e4
> Fix indentation in Debian post and pre scripts
>
> There is several misindentation inside Debian post and pre
> installation scripts. False indentation with space as indent space
> should be 2 and indentation with tabs.
>
> Adopt upstream commit 7cbb45d1 in Debian by conserving customizations
> in:
> - debian/mariadb-server.postinst
> - debian/mariadb-server.postrm
> - debian/mariadb-server.preinst

https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/d0bcab443fa6d44084dc674ba29b79516c6239ba
> Ensure spaces are used everywhere instead of tabs for indentation

https://salsa.debian.org/mariadb-team/mariadb-server/-/commit/0300a9157cc69f75e01ac9c0d6e033d8be661492
> Complement previous upstream commits to fix Shellcheck issues
>
> - Unify if/then and while/do on separate lines
> - Fix indentation to be consistent
> - Use "$()" instead of backticks for subshells
> - Exit code cannot be -1, must be 0-255
> - Remove unused variables MYCHECK and MYCHECK_PARAMS
> - Rewrite messy command-line database calls to an easier to read form
>   that does exactly the same
> - Use 'command -v' test instead of 'which'
>
> With this commit, all of debian/* is Shellcheck clean.

Also
* Update mariadb.conf.d template to tell users where to create logdir
  if they are not using journald
* Remove use of work 'slave'
* Add minor workaround for Debian Bug #1022994 if TMPDIR is empty
* Make start/stop in maintainer scripts correctly check mariadbd
  ownership and only start/stop processes owned by root or 'mysql'
* Remove obsolete 'NO_UPDATE_BUILD_VERSION=1' as it did not affect the
  RocksDB build reproducibility as previously assumed
* Run 'wrap-and-sort -av'

- Unify on MTR_SKIP_TEST_LIST in both d/rules and autopkgtests
- Unify MTR command in both d/rules and autopkgtests
- Make d/rules section more verbose to help debugging why tests
  sometimes ran and sometimes not
- If MTR fails, make the log a bit more verbose
  (inspired by https://github.com/MariaDB/buildbot/pull/76/files)

The way DPKG_GENSYMBOLS_CHECK_LEVEL was exported did not actually
have any effect on the build. Fix the syntax so that build will
indeed fail if there there are new symbols in new upstream version.

- mariadb-server: recursive-privilege-change "chown -R"
- use correct URL https://jira.mariadb.org everywhere
- dependency-is-not-multi-archified libmariadb-dev-compat depends on libmariadb-dev (multi-arch: no)
- dependency-is-not-multi-archified mariadb-plugin-gssapi-client depends on mariadb-client (multi-arch: no)

Based on riscv64 build logs the RocksDB plugin currently builds fine on
it, and the riscv64 platform is 64-bit and has correct endianness for
RocksDB, so all the pre-requisites for it working exist, so it should
work.

Make sure that Debian respects systemd disabled by bumping
to deb compat 11 which is available from Debian 10 and Ubuntu 20.04
and it provides better integration with systemd.

Start using dh_installsystemd which is new recommended way
in compat 11

As mariadb.pc contains mostly the same than
libmariadb.pc and it mainly only creates distortion
for client developers. They use libmariadb.pc not mariadb.pc
(which is for embbeded use mainly).

Move mariadb.pc to not-installed from libmariadbd-dev
to clear out this situation

There is some package-contains-documentation-outside-usr-share-doc that
are better to be there than move somewhere else. They are:

 * mariadb-server: package-contains-documentation-outside-usr-share-doc [usr/share/mysql/errmsg-utf8.txt]
 * mariadb-server-core: package-contains-documentation-outside-usr-share-doc [usr/share/mysql/charsets/README]
 * mariadb-test: package-contains-documentation-outside-usr-share-doc [usr/share/mysql/mysql-test/README]

Also fix Mroonga

 * mariadb-plugin-mroonga: extra-license-file [usr/share/mysql/mroonga/COPYING]

There is couple spare-manual-page problems which means that
there is man page but no binary for that.

wsrep_sst_backup is in not-installed and man page is in
mariadb-server package. Move man page also to not-installed

mysql-test-run.pl is in unusual location which makes
lintian think that it not available.

There is unused override which is not needed an polluting
output

 * unused-override arch-dependent-file-not-in-arch-specific-directory usr/bin/mariadb_config [usr/share/lintian/overrides/libmariadb-dev:2]

Lintian warn that mariadb-common and mysql-common descriptions
are sypnosis as they contain dot but dots are used in other
purposes so they are just false-positives:

 * mysql-common: synopsis-is-a-sentence "MariaDB client common configuration files package (e.g. /etc/mysql/my.cnf)"
 * mariadb-server: unused-debconf-template mariadb-server/old_data_directory_saved [templates:2]

Lot's of binaries have 'spelling errors' which are there in
purpose and they are simply false positives some list of them
are:

 * I: libmariadbd19: spelling-error-in-binary noone no one [usr/lib/x86_64-linux-gnu/libmariadbd.so.19]
 * I: libmariadbd19: spelling-error-in-binary thats that's [usr/lib/x86_64-linux-gnu/libmariadbd.so.19]
 * I: libmariadbd19: spelling-error-in-binary theres there's [usr/lib/x86_64-linux-gnu/libmariadbd.so.19]
 * I: libmariadbd19: spelling-error-in-binary yuR your [usr/lib/x86_64-linux-gnu/libmariadbd.so.19]
 * I: mariadb-backup: spelling-error-in-binary exising existing [usr/bin/mariadb-backup]
 * I: mariadb-backup: spelling-error-in-binary noone no one [usr/bin/mariadb-backup]
 ...

 which are being handled

When new man pages are generated for libmariadb13 then manpage
typos are fixed and now they just pollute Lintian output

 * typo-in-manual-page handshak handshake [usr/share/man/man3/mariadb_get_infov.3.gz:173]
 * typo-in-manual-page occured occurred [usr/share/man/man3/mysql_ping.3.gz:39]
 * typo-in-manual-page occured occurred [usr/share/man/man3/mysql_set_character_set.3.gz:37]
 * typo-in-manual-page occured occurred [usr/share/man/man3/mysql_stmt_error.3.gz:30]
 * typo-in-manual-page occured occurred [usr/share/man/man3/mysql_stmt_reset.3.gz:27]
 * typo-in-manual-page occured occurred [usr/share/man/man3/mysql_store_result.3.gz:34]
 * typo-in-manual-page releated related [usr/share/man/man3/mariadb_get_infov.3.gz:8]
 * typo-in-manual-page reponse response [usr/share/man/man3/mysql_send_query.3.gz:8]
 * typo-in-manual-page seperated separated [usr/share/man/man3/mysql_real_connect.3.gz:144]
 * typo-in-manual-page seperated separated [usr/share/man/man3/mysql_real_connect.3.gz:148]
 * typo-in-manual-page sucess success [usr/share/man/man3/mariadb_reconnect.3.gz:25]

There is lot of warnings like this:

 * source-contains-autogenerated-visual-c++-file [extra/wolfssl/wolfssl/IDE/WIN10/resource.h]
 * source-contains-autogenerated-visual-c++-file [extra/wolfssl/wolfssl/IDE/WIN10/wolfssl-fips.rc]
 * source-contains-autogenerated-visual-c++-file [extra/wolfssl/wolfssl/resource.h]
 * source-contains-autogenerated-visual-c++-file [storage/columnstore/columnstore/dbcon/ddlpackageproc/libddlpackageproc.rc]
 * source-contains-autogenerated-visual-c++-file [storage/columnstore/columnstore/dbcon/ddlpackageproc/resource.h]
 * source-contains-autogenerated-visual-c++-file [storage/columnstore/columnstore/dbcon/dmlpackageproc/libdmlpackageproc.rc]

Which are needed and should be there. They are just false-positives

Also MariaDB rebuilds these Java arhives but they are available for testing purposes still
so they are also false-positives

 * source-contains-prebuilt-java-object [storage/connect/mysql-test/connect/std_data/JavaWrappers.jar]
 * source-contains-prebuilt-java-object [storage/connect/mysql-test/connect/std_data/JdbcMariaDB.jar]
 * source-contains-prebuilt-java-object [storage/connect/mysql-test/connect/std_data/Mongo2.jar]
 * source-contains-prebuilt-java-object [storage/connect/mysql-test/connect/std_data/Mongo3.jar]

Some packages are needed to conflict packages that are not in
this package and they emerge some warnings. Remove them from
lintian output as they are not errors or something that should
be handled.

I: libmariadb-dev: conflicts-with-version libmariadb-dev-compat (<< 3.0.0)
I: libmariadb-dev-compat: conflicts-with-version libmariadbclient-dev (<< 1:10.11.8+maria~deb11)
I: libmariadb3: conflicts-with-version libmariadbclient18 (<< 10.2.0)
I: mariadb-client: conflicts-with-version mysql-client (<< 5.0.51)
I: mariadb-client-core: conflicts-with-version mysql-client (<< 5.0.51)

 lintian output

Lintian have many warnings that there is shebang '/bin/sh' which can
be populary /bin/bash or /bin/zsh but also many others like Ksh
or even Fish.

Warned lintian problems are testing that are run under Bash or some other
shell so they are or they are words in comments like let

 * I: mariadb-client: bash-term-in-posix-shell '`echo "testing\c"' [usr/bin/mariadb-secure-installation:191]
 * I: mariadb-server: bash-term-in-posix-shell '${BASH_VERSION' [usr/share/mysql/wsrep_notify:86]
 * I: mariadb-server: bash-term-in-posix-shell '[ "$url" ==' [usr/bin/mariadbd-safe:216]
 * I: mariadb-server: bash-term-in-posix-shell 'let' [usr/bin/mariadbd-safe:41]
 * I: mariadb-server: bash-term-in-posix-shell 'ulimit' [usr/bin/mariadbd-safe:712]
 * I: mariadb-server: bash-term-in-posix-shell 'ulimit' [usr/bin/mariadbd-safe:832]
 * I: mariadb-server-core: bash-term-in-posix-shell 'source tree,' [usr/bin/mariadb-install-db:93]
 * I: mariadb-test-data: bash-term-in-posix-shell '${BASH_VERSION' [usr/share/mysql/mysql-test/std_data/wsrep_notify.sh:87]
 * I: mariadb-test-data: bash-term-in-posix-shell '${BASH_VERSION' [usr/share/mysql/mysql-test/std_data/wsrep_notify_ssl.sh:87]

These are not big ones.

These are needed in *BSD family

 * incorrect-path-for-interpreter /usr/bin/env perl != /usr/bin/perl [usr/share/mysql/mysql-test/mariadb-stress-test.pl]
 * incorrect-path-for-interpreter /usr/bin/env perl != /usr/bin/perl [usr/share/mysql/mysql-test/mariadb-test-run.pl]

and these are needed to make sure that these packages get removed if they exist:

 * version-substvar-for-external-package Replaces ${source:Version} libmariadb-dev -> libmysqlclient-dev [debian/control:*]
 * version-substvar-for-external-package Replaces ${source:Version} libmariadb-dev -> libmysqld-dev [debian/control:*]
 * version-substvar-for-external-package Replaces ${source:Version} libmariadbd-dev -> libmariadbclient-dev [debian/control:*]

There is not-installed static libraries which only
have x86-64 path so they will be installed in other
architectures like ARM. Fix them replacing with '*'
char

 * W: mariadb source: unwanted-path-too-specific

Lintian overdrives tends to rot and with Debian 12
Lintian got overhaul with syntax changes which made
most of the old overrides obsoleted. Change old ones
to new ones and remove unneeded onews also add some
needed overrides.

 * W: mariadb-test-data: mismatched-override *
   - There was serveral old syntax overrides which should be just removed

 * W: mariadb-test-data: national-encoding  *
   - There is lot's test files which are in ISO/IEC 8859-1 encoding
     and not UTF-8 for purpose. Remove then from polluting lintian

 * W: mariadb-plugin-provider-bzip2: mismatched-override
 * W: mariadb-plugin-provider-lz4: mismatched-override
 * W: mariadb-plugin-provider-lzma: mismatched-override
 * W: mariadb-plugin-provider-lzo: mismatched-override
 * W: mariadb-plugin-provider-snappy: mismatched-override
   - Remove old style overrides from packages

 * W: mariadb-test: shared-library-lacks-prerequisites [usr/lib/mysql/plugin/auth_0x0100.so]
 * W: mariadb-test: shared-library-lacks-prerequisites [usr/lib/mysql/plugin/debug_key_management.so]
 * W: mariadb-test: shared-library-lacks-prerequisites [usr/lib/mysql/plugin/test_sql_service.so]
   - These libraries are like that for a purpose

Debian control file has few lintian warning level notifications

 * mariadb-test: breaks-without-version mariadb-server-5.5
   - Several packages conflicts with mariadb-server-5.5 but
     on mariadb-test breaks without it so it's controversial
     and can be resolved only with conflicting in mariadb-test

 * W: libmariadbd-dev: extended-description-line-too-long line 6
 * W: mariadb-plugin-cracklib-password-check: extended-description-line-too-long line 4
   - Lines are too long (over 80 chars) and they are just separated
     to new line.

MDEV-34274 did not fix the test failure. The test has a START SLAVE
UNTIL condition, where we can't use sync_with_master_gtid.inc,
wait_for_slave_to_start.inc, or wait_for_slave_to_stop.inc because
our MTR connection thread races with the start/stop of the SQL/IO
threads. So instead, for slave start, we prove the threads started
by waiting for the connection count to increase by 2; and for slave
stop, we wait for the processlist count to return to its pre start
slave number.

MDEV-34274: Test rpl.rpl_change_master_demote frequently fails on buildbot with "IO thread should not be running..."

Note this is a backport of 8c8b3ab7
from 11.1.

The test rpl.rpl_change_master_demote used a `sleep 1` command
to give time for a START SLAVE UNTIL to start the slave threads
and wait for them to automatically die by UNTIL.  On machines
with heavy load (especially MSAN bb builders), one second was
not enough, and the test would fail due to the IO thread
still being up.

This patch fixes the test by replacing the sleep with specific
conditions to wait for. The test cannot wait for the IO or SQL
threads to start, as it would be possible that they would be
started and stopped by the time the MTR executor would check
the slave status. So instead, we test for proof that they
existed via the Connections status variable being incremented
by at least 2 (Connections just shows the global thread id).
At this point, we still can't use the wait_for_slave_to_stop
helper, as the SQL/IO_Running fields of SHOW SLAVE STATUS
may not be updated yet. So instead, we use
information_schema.processlist, which would show the presence
of the Slave_SQL/IO threads. So to "wait for the slave to stop",
we wait for the Slave_SQL/IO threads to be gone from the
processlist.

Numerous fixes included from upstream.

/utf8 needed for Windows, but may as well
enable globally.

my_like_range*() can create longer keys than Field::char_length().
This caused warnings during print_range().

Fix:

Suppressing warnings in print_range().

Line numbers had to be removed from the ignorelists in order to be
diffed against since locations of the same findings can differ
across runs. Therefore preprocessing has to be done on the CI findings
so that it can be compared to the ignorelist and new findings can be
outputted. However, since line numbers have to be removed, a situation
occurs where it is difficult to reference the location of findings
in code given the output of the CI job.

To lessen this pain, change the cppcheck template to include
code snippets which make it easier to reference where in the code
the finding is referring to, even in the absence of line numbers.
Ignorelisting works as before since locations of the finding may
change but not the code it is referring to.

Furthermore, due to the innate difficulty in maintaining ignorelists
across branches and triaging new findings, allow failure as to not
have constantly failing pipelines as a result of a new findings that
have not been addressed yet.

Lastly, update SAST ignorelists to match the newly refactored cppcheck
job and the current state of the codebase.

All new code of the whole pull request, including one or several
files that are either new files or modified ones, are contributed
under the BSD-new license. I am contributing on behalf of my
employer Amazon Web Services, Inc.

Rectify cases of mismatched brackets and address
possible cases of division by zero by checking if
the denominator is zero before dividing.

No functional changes were made.

All new code of the whole pull request, including one or several
files that are either new files or modified ones, are contributed
under the BSD-new license. I am contributing on behalf of my
employer Amazon Web Services, Inc.

crc32_avx512(): Explicitly cast ssize_t(size) to make it clear that
we are indeed applying a negative offset to a pointer.

The issue was that when repairing an Aria table of row format PAGE and
the data file was bigger the 4G, the data file length was cut short
because of wrong parameters to MY_ALIGN().

The effect was that ALTER TABLE, OPTIMIZE TABLE or REPAIR TABLE would fail
on these tables, possibly corrupting them.
The MDEV also exposed a bug where error state was not propagated properly
to the upper level if the number of rows in the table changed.

The current semi-sync binlog fail-over recovery process uses
rpl_semi_sync_slave_enabled==TRUE as its condition to truncate a
primary server’s binlog, as it is anticipating the server to re-join
a replication topology as a replica. However, for servers configured
with both rpl_semi_sync_master_enabled=1 and
rpl_semi_sync_slave_enabled=1, if a primary is just re-started (i.e.
retaining its role as master), it can truncate its binlog to drop
transactions which its replica(s) has already received and executed.
If this happens, when the replica reconnects, its gtid_slave_pos can
be ahead of the recovered primary’s gtid_binlog_pos, resulting in an
error state where the replica’s state is ahead of the primary’s.

This patch changes the condition for semi-sync recovery to truncate
the binlog to instead use the configuration variable
--init-rpl-role, when set to SLAVE. This allows for both
rpl_semi_sync_master_enabled and rpl_semi_sync_slave_enabled to be
set for a primary that is restarted, and no transactions will be
lost, so long as --init-rpl-role is not set to SLAVE.

Reviewed By:
============
Sergei Golubchik <serg@mariadb.com>

The special logic used by the memory storage engine
to keep slaves in sync with the master on a restart can
break replication. In particular, after a restart, the
master writes DELETE statements in the binlog for
each MEMORY-based table so the slave can empty its
data. If the DELETE is not executable, e.g. due to
invalid triggers, the slave will error and fail, whereas
the master will never see the problem.

Instead of DELETE statements, use TRUNCATE to
keep slaves in-sync with the master, thereby bypassing
triggers.

Reviewed By:
===========
Kristian Nielsen <knielsen@knielsen-hq.org>
Andrei Elkin <andrei.elkin@mariadb.com>

During read only mode, InnoDB doesn't allow checkpoint to happen.
So InnoDB should throw the warning when InnoDB tries to
force the checkpoint when innodb_read_only = 1 or
innodb_force_recovery = 6.

MariaDB supports a "wait-free concurrent allocator based on pinning addresses".
In `lf_pinbox_real_free()` it tries to sort the pinned addresses for better
performance to use binary search during "real free". `alloca()` was used to
allocate stack memory and copy addresses.

To prevent a stack overflow when allocating the stack memory the function checks
if there's enough stack space. However, the available stack size was calculated
inaccurately which eventually caused database crash due to stack overflow.

The crash was seen on MariaDB 10.6.11 but the same code defect exists on all
MariaDB versions.

A similar issue happened previously and the fix in fc2c1e43 was to add a
`ALLOCA_SAFETY_MARGIN` which is 8192 bytes. However, that safety margin is not
enough during high connection workloads.

MySQL also had a similar issue and the fix
https://github.com/mysql/mysql-server/commit/b086fda was to remove the use of
`alloca` and replace qsort approach by a linear scan through all pointers (pins)
owned by each thread.

This commit is mostly the same as it is the only way to solve this issue as:
1. Frame sizes in different architecture can be different.
2. Number of active (non-null) pinned addresses varies, so the frame
   size for the recursive sorting function `msort_with_tmp` is also hard
   to predict.
3. Allocating big memory blocks in stack doesn't seem to be a very good
   practice.

For further details see the mentioned commit in MySQL and the inline comments.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.

The symptoms were: take a server with no activity and a table that's
not in the buffer pool. Run a query that reads the whole table and
observe that r_engine_stats.pages_read_count shows about 2% of the table
was read. Who reads the rest?

The cause was that page prefetching done inside InnoDB was not counted.

This counts page prefetch requests made in buf_read_ahead_random() and
buf_read_ahead_linear() and makes them visible in:

- ANALYZE: r_engine_stats.pages_prefetch_read_count
- Slow Query Log: Pages_prefetched:

This patch intentionally doesn't attempt to count the time to read the
prefetched pages:
* there's no obvious place where one can do it
* prefetch reads may be done in parallel (right?), it is not clear how
  to count the time in this case.

The crash is caused by the attempt to refix the constant subquery during
pushdown from HAVING into WHERE optimization.

Every condition that is going to be pushed into WHERE clause is first
cleaned up, then refixed. Constant subqueries are not cleaned or refixed
because they will remain the same after refixing, so this complicated
procedure can be omitted for them (introduced in MDEV-21184).
Constant subqueries are marked with flag IMMUTABLE_FL, that helps to miss
the cleanup stage for them. Also they are marked as fixed, so refixing is
also not done for them.
Because of the multiple equality propagation several references to the same
constant subquery can exist in the condition that is going to be pushed
into WHERE. Before this patch, the problem appeared in the following way.
After the first reference to the constant subquery is processed, the flag
IMMUTABLE_FL for the constant subquery is disabled.
So, when the second reference to this constant subquery is processed, the
flag is already disabled and the subquery goes through the procedure of
cleaning and refixing. That causes a crash.

To solve this problem, IMMUTABLE_FL should be disabled only after all
references to the constant subquery are processed, so after the whole
condition that is going to be pushed is cleaned up and refixed.

Approved by Igor Babaev <igor@maridb.com>

Fixing applying the COLLATE clause to a parameter caused an error error:
  COLLATION '...' is not valid for CHARACTER SET 'binary'

Fix:

- Changing the collation derivation for a non-prepared Item_param
  to DERIVATION_IGNORABLE.

- Allowing to apply any COLLATE clause to expressions with DERIVATION_IGNORABLE.
  This includes:
    1. A non-prepared Item_param
    2. An explicit NULL
    3. Expressions derived from #1 and #2

  For example:
    SELECT ? COLLATE utf8mb_unicode_ci;
    SELECT NULL COLLATE utf8mb_unicode_ci;
    SELECT CONCAT(?) COLLATE utf8mb_unicode_ci;
    SELECT CONCAT(NULL) COLLATE utf8mb_unicode_ci

- Additional change: preserving the collation of an expression when
  the expression gets assigned to a PS parameter and evaluates to SQL NULL.
  Before this change, the collation of the parameter was erroneously set
  to &my_charset_binary.

- Additional change: removing the multiplication to mbmaxlen from the
  fix_char_length_ulonglong() argument, because the multiplication already
  happens inside fix_char_length_ulonglong().
  This fixes a too large column size created for a COLLATE clause.