MDEV-28769 Assertion `(m_ci->state & 32) || m_with_collate' failed in Lex_exact_charset_opt_extended_collate::Lex_exact_charset_opt_extended_collate on SET NAMES

These system variables:
  @@character_set_client
  @@character_set_connection
  @@character_set_database
  @@character_set_filesystem
  @@character_set_results
  @@character_set_server

can now be set in numeric format only to IDs of default collations, e.g.:

SET @@character_set_xxx=9;  -- OK    (latin2_general_ci  is default)
SET @@character_set_xxx=2;  -- ERROR (latin2_czech_cs is not default)
SET @@character_set_xxx=21; -- ERROR (latin2_hungarian_ci is not default)

Before this change the server accepted IDs of non-default collations
so all three examples above worked without errors,
but this could lead to unexpected behavior in later statements.

In commit c4c88307 (MDEV-28111) we disabled
the file system cache on the InnoDB write-ahead log file (ib_logfile0)
by default on Linux.

It turns out that especially with innodb_flush_trx_log_at_commit=2,
writing to the log via the file system cache typically improves throughput,
especially on slow storage or at a small number of concurrent transactions.
For other values of innodb_flush_log_at_trx_commit, direct writes were
observed to be mostly but not always faster. Whether it pays off to
disable the file system cache on the log may depend on the type of storage,
the workload, and the operating system kernel version.

On Linux and Microsoft Windows, we will introduce the settable Boolean
global variable innodb_log_file_buffering that indicates whether the
file system cache on the redo log file is enabled. The default value is
innodb_log_file_buffering=OFF. If the server is started up with
innodb_flush_log_at_trx_commit=2, the value will be changed to
innodb_log_file_buffering=ON.

When a persistent memory interface is being used for the log,
the value cannot be changed from innodb_log_file_buffering=OFF.
On Linux, when the physical block size cannot be determined
to be a power of 2 between 64 and 4096 bytes, the file system cache
cannot be disabled, and innodb_log_file_buffering=ON cannot be changed.

Server log messages will indicate whether the file system cache is
enabled for the redo log:

[Note] InnoDB: Buffered log writes (block size=512 bytes)
[Note] InnoDB: File system buffers for log disabled (block size=512 bytes)

After this change, the startup parameter innodb_flush_method will no
longer control whether O_DIRECT will be set on the redo log on Linux.

On other operating systems that support O_DIRECT, no interface has been
implemented for controlling the file system cache for the redo log.
The innodb_flush_method values O_DIRECT, O_DIRECT_NO_FSYNC, O_DSYNC
will enable O_DIRECT for data files, not the log.

Tested by: Matthias Leich, Axel Schwenke

trx_purge_free_segment(): Do mark that the block will be modified.

It seems possible that this regression was introduced by the
changes to the page-freeing logic
in commit 4179f93d (MDEV-18976).

Tested by: Matthias Leich

- innodb_fts.sync_block doesn't make sense after MDEV-25581's patch
because fts cache syncing is done as a part of insert operation
and it leads to completion of select over insert sometimes.
This test case is not relevant any more

In commit 73fee39e (MDEV-27985)
a regression was introduced that would cause bpage=nullptr to
be referenced.

buf_flush_LRU_list_batch(): Always terminate the loop upon
encountering a null pointer.

MDEV-26127 Assertion `err != DB_DUPLICATE_KEY' failed or InnoDB: Failing assertion: id != 0 on ALTER ... REBUILD PARTITION

During rebuild of partition, the partitioning engine calls
alter_close_table(), which does not unlock and close some table
instances of the target table.
Then, the engine fails to rename partitions because there are table
instances that are still locked.

Closing all the table instance of the target table fixes the bug.

innodb_drop_database(): Use explicit TO_BINARY casts on
SYS_TABLES.NAME, which for historical reasons uses the wrong collation
latin1_swedish_ci instead of BINARY.

These had been damaged during some merge which caused --embedded
test to fail

Compiling hashicorp plugin statically got many InnoDB
encryption test to fail with errors like:
Error: InnoDB: The page [page id: space=5, page number=3] in file './test/t1.ibd' cannot be decrypted.

MariaDB codebase is huge and Lintian has lots of test than
can fire false-positive warnings which leads to situation
where real problems can't be spotted.

Suspend obvious false-positive Lintian warnings and
let Lintian problems that needs some love shine
out.

Suspends in package mariadb-test-data

Supporting BSD family needs to use '/usr/bin/env perl' and not '/usr/bin/perl'
Perl script are for testing and not for production in mariadb-test-data
package:
 * incorrect-path-for-interpreter

There is several files with national-encoding which are test file so they
can't be in unicode charset
 * national-encoding

Serveral test paths are intentionally repeated:
 * repeated-path-segment

Suspends in package mariadb-test

Supporting BSD family needs to use '/usr/bin/env perl' and not '/usr/bin/perl'
Perl script are for testing and not for production in mariadb-test-data
package:
 * incorrect-path-for-interpreter

Suspends in package source package

Remade some 'version-substvar-for-external-package' to use
regex.

MGroonga is missing source file 'jquery-ui-1.8.18.custom.js' correct
lintian suspend with regex:
 * source-is-missing

There is several files with very long line lenghts. Add suspends
for those that can't be corrected in several places. Most
of them are test result files, SQL test files or intentional
long lines that can't be splitted.
  * very-long-line-length-in-source-file

There is several autogenerated C++ files which probably should not
be there but they should not do any harm:
 * source-contains-autogenerated-visual-c++-file

Problem:
========
  InnoDB FTS requesting the fts sync of the table once the fts
cache size reaches 1/10 of innodb_ft_cache_size. But fts_sync()
releases cache lock when writing the word. By doing this, InnoDB
insert thread increases the innodb fts cache memory and
SYNC operation will take more time to complete.

Solution:
=========
  Remove the fts sync operation(FTS_MSG_SYNC_TABLE) from
the fts optimize background thread. Instead of that,
allow user thread to sync the InnoDB fts cache when
the cache size exceeds 512 kb. User thread holds
cache lock while doing cache syncing, it make sure that
other threads doesn't add the docs into the cache.

Removed FTS_MSG_SYNC_TABLE and its related function
because we do remove the FTS_MSG_SYNC_TABLE message
itself.

Removed fts_sync_index_check() and all related
function because other threads doesn't add while
cache operation going on.

This fixes up commit 3d241eb9

In commit f80deb95 (MDEV-27868)
a fix for a correctness regression caused a performance regression
by increasing the amount of work that is executed while holding
buf_pool.flush_list_mutex.

buf_page_t::set_temp_modified(): Relax an assertion, to allow an
already dirty block to be marked as dirty.

buf_page_t::flush_list_requests: Note that the variable is not
always protected by buf_pool.flush_list_mutex. Already dirty blocks
that are being written to will increment the counter without
holding buf_pool.flush_list_mutex.

mtr_t::process_freed_pages(): Handle pages that were freed during
the execution of the mini-transaction.

ReleaseUnlogged, mtr_t::release_unlogged(): Release modified pages when
no log was written. This is for pages of the temporary tablespace, or for
IMPORT TABLESPACE.

ReleaseModified: Renamed from ReleaseBlocks.
Assume that buf_pool.flush_list_mutex was acquired by the caller.

ReleaseSimple: A combination of ReleaseLatches and ReleaseModified,
for the case that for any modified pages, some earlier modifications
are already waiting to be written.

mtr_t::commit(): Invoke one of release_unlogged(), ReleaseModified,
ReleaseSimple, ReleaseAll. Acquire and release buf_pool.flush_list_mutex
at most once.

memo_slot_release(): Simplify the code.

mtr_t::sx_latch_at_savepoint(), mtr_t::x_latch_at_savepoint():
Reduce the size of the critical section.

fil_space_t::update_last_freed_lsn(), fil_space_t::clear_freed_ranges(),
fil_space_t::add_free_range(): Assume that freed_range_mutex is held
by the caller.

buf_pool_t::prepare_insert_into_flush_list(): Determine the insert
position for buf_pool_t::insert_into_flush_list(). Remove any clean
blocks from buf_pool.flush_list that were encountered while searching.

buf_pool_t::insert_into_flush_list(): Insert the block at the
predetermined position.

PageConverter::update_header(): Remove an unnecessary write.
The field that was originally called FIL_PAGE_FILE_FLUSH_LSN only
made sense for the first page of the system tablespace
(initially, for the first page of each file of the system tablespace).
It never had any meaning for .ibd files, and it lost its original
meaning in MariaDB Server 10.8.1 when
commit b07920b6 (MDEV-27199)
removed the ability to start without ib_logfile0.

If the most significant 32 bits of the LSN are nonzero, this
unnecessary write would write the wrong encryption key identifier
to the page. The first page of any file is never encrypted,
so normally those bytes should be 0 for any .ibd file.

The zoneinfo directory is littered with non-timezone information files.

These frequently contain extensions, not present in real timezone files.

Alo leapseconds is frequently there and is not a timezone file.

The else condition is meant to be here to define the functions
if the Red Hat include file isn't there.

Fixes: commit 467011bc / MDEV-26614

RedHat -> Red Hat by Daniel Black

Continue the effort of a previous commit (PR#2114) which changed the man
pages titles from MariaDB to MySQL, to further update the man pages.

Update the man page NAME sections to use mariadb-* instead of mysql* for
MariaDB binaries that are drop-in replacements for MySQL equivalents,
indicating that the commands are actually of the MariaDB version.

Before:
    NAME
        mysql_upgrade - check tables for MariaDB upgrade
    ...

After:
    NAME
        mariadb-upgrade - check tables for MariaDB upgrade (mysql_upgrade
        is now a symlink to mariadb-upgrade)
    ...

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.

ha_innobase::build_template may initialize m_prebuilt->idx_cond
even if there is no valid pushed_idx_cond_keyno.
This potentially problematic piece of code was found while
working on MDEV-27366

A prominent remaining source of crashes on corrupted index pages
is page directory corruption.

A frequent caller of page_dir_find_owner_slot() is page_rec_get_prev().
Some of those calls can be replaced with simpler logic that is less
prone to fail.

page_dir_find_owner_slot(),
page_rec_get_prev(), page_rec_get_prev_const(),
btr_pcur_move_to_prev(), btr_pcur_move_to_prev_on_page(),
btr_cur_upd_rec_sys(),
page_delete_rec_list_end(),
rtr_page_copy_rec_list_end_no_locks(),
rtr_page_copy_rec_list_start_no_locks(): Return an error code on failure.

fil_space_t::io(), buf_page_get_low(): Use DB_CORRUPTION for
out-of-bounds page reads.

PageBulk::getSplitRec(), PageBulk::copyOut(): Simplify the code.

btr_validate_level(): Prevent some more CHECK TABLE crashes on
corrupted pages.

btr_block_get(), btr_pcur_move_to_next_page(): Implement some checks that
were previously only part of IndexPurge::next().

IndexPurge::next(): Use btr_pcur_move_to_next_page().

MariaDB never supported this form of preemption via high-priority
transactions. This error code shold not have been added in the
first place, in commit 2e814d47.

and failing spider partition test.

With some small datatype changes to the Linux/Solaris my_gethwaddr implementation
the hardware address of AIX can be returned. This is an important aspect
in Spider (and UUID).

Spider test change reviewed by Nayuta Yanagisawa.

my_gethwaddr review by Monty in #2081

fil_page_type_validate(): Remove. This debug check was mostly redundant
and added little value to the code paths that deal with page_compressed
or encrypted pages.

fil_get_page_type_name(): Remove; unused function.

fil_space_decrypt(): Return an error if the page is not
supposed to be encrypted. It is possible that an unencrypted page
contains a nonzero key_version field even though it is not supposed
to be encrypted. Previously we would crash in such a situation.

buf_page_decrypt_after_read(): Simplify the code. Remove some
unnecessary error message about temporary tablespace corruption.
This is where we would usually invoke fil_space_decrypt().

Even after commit 0b47c126
there are a few ib::fatal() calls in non-debug code
that can be replaced easily.

btr_page_reorganize_low(): On size invariant violation, return
an error code instead of crashing.

btr_check_blob_fil_page_type(): On an invalid page type, report
an error but do not crash.

btr_copy_blob_prefix(): Truncate the output if a page type is invalid.

dict_load_foreign_cols(): On an error, return DB_CORRUPTION instead
of crashing.

fil_space_decrypt_full_crc32(), fil_space_decrypt_for_non_full_checksum():
On error, return DB_DECRYPTION_FAILED instead of crashing.

fil_set_max_space_id_if_bigger(): Replace ib::fatal() with an
equivalent ut_a() assertion.

Spider is now a part of the server. So, it doesn't make sense for Spider
to have its own version number apart from the server's one.

This patch fixes the following issues in Aria error reporting in case
of read errors & crashed tables:
- Added the table name to the most error messages, including in case of
  read errors or when encrypting/decrypting a table. The format for
  error messages was changed sligtly to accomodate logging of errors
  from lower level routines.
- If we got an read error from storage (hard disk, ssd, S3 etc) we only
  reported 'table is crashed'. Now the error number from the storage
  is reported.
- Added checking of read failure from records_in_range()
- Calls to ma_set_fatal_error() did not inform the SQL level of
  errors (to not spam the user with multiple error messages).
  Now the first error message and any fatal error messages are reported
  to the user.