Signed-off-by: Michael Okoko <okokomichaels@outlook.com>

Two Problems
1. Upgrade wizard failed to retrieve path to service executable,
if it contained non-ASCII.
Fixed by setlocale(LC_ALL, "en_US.UTF8"), which was missing in upgrade wizard

2.mysql_upgrade_service only updated (converted to UTF8) the server's sections
leaving client's as-is

Corrected typo.

3. Fixed assertion in my_getopt, turns out to be too strict.

Log messages like total size = 17179869184, chunk size = 134217728
get hard to read. If we normalize it down to IEC units is easier.

Idea thanks to Axel Schwenke.

Review thanks to Eugene Kosov and Marko Mäkelä

$ mariadblocal --innodb-buffer-pool-size=30G --innodb-log-file-size=128M
Installing MariaDB/MySQL system tables in '/tmp/build-mariadb-server-10.7-datadir' ...
2021-12-09  9:54:04 0 [Note] /home/dan/repos/build-mariadb-server-10.7/sql/mysqld (server 10.7.2-MariaDB) starting as process 250473 ...
2021-12-09  9:54:04 0 [Note] InnoDB: The first data file './ibdata1' did not exist. A new tablespace will be created!
2021-12-09  9:54:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-12-09  9:54:04 0 [Note] InnoDB: Number of transaction pools: 1
2021-12-09  9:54:04 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-12-09  9:54:04 0 [Note] InnoDB: Using liburing
2021-12-09  9:54:04 0 [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 128.000MiB
2021-12-09  9:54:04 0 [Note] InnoDB: Completed initialization of buffer pool
2021-12-09  9:54:04 0 [Note] InnoDB: Setting O_DIRECT on file ./ibdata1 failed
2021-12-09  9:54:04 0 [Note] InnoDB: Setting file './ibdata1' size to 12.000MiB. Physically writing the file full; Please wait ...
2021-12-09  9:54:04 0 [Note] InnoDB: File './ibdata1' size is now 12.000MiB.
2021-12-09  9:54:04 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 96.000MiB
2021-12-09  9:54:04 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2021-12-09  9:54:04 0 [Note] InnoDB: New log file created, LSN=10317
2021-12-09  9:54:04 0 [Note] InnoDB: Doublewrite buffer not found: creating new
2021-12-09  9:54:04 0 [Note] InnoDB: Doublewrite buffer created
2021-12-09  9:54:04 0 [Note] InnoDB: 128 rollback segments are active.
2021-12-09  9:54:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-12-09  9:54:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2021-12-09  9:54:04 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2021-12-09  9:54:04 0 [Note] InnoDB: 10.7.2 started; log sequence number 0; transaction id 3
OK
2021-12-09  9:54:04 0 [Note] sql/mysqld (server 10.7.2-MariaDB) starting as process 250501 ...
2021-12-09  9:54:04 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-12-09  9:54:04 0 [Note] InnoDB: Number of transaction pools: 1
2021-12-09  9:54:04 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2021-12-09  9:54:04 0 [Note] InnoDB: Using liburing
2021-12-09  9:54:04 0 [Note] InnoDB: Initializing buffer pool, total size = 30.000GiB, chunk size = 128.000MiB
2021-12-09  9:54:04 0 [Note] InnoDB: Completed initialization of buffer pool
2021-12-09  9:54:04 0 [Note] InnoDB: Setting O_DIRECT on file ./ibdata1 failed
2021-12-09  9:54:04 0 [Note] InnoDB: Resizing redo log from 96.000MiB to 128.000MiB; LSN=41361
2021-12-09  9:54:04 0 [Note] InnoDB: Starting to delete and rewrite log file.
2021-12-09  9:54:04 0 [Note] InnoDB: Setting log file ./ib_logfile101 size to 128.000MiB
2021-12-09  9:54:04 0 [Note] InnoDB: Renaming log file ./ib_logfile101 to ./ib_logfile0
2021-12-09  9:54:04 0 [Note] InnoDB: New log file created, LSN=41361
2021-12-09  9:54:04 0 [Note] InnoDB: 128 rollback segments are active.
2021-12-09  9:54:04 0 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-12-09  9:54:04 0 [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ...
2021-12-09  9:54:04 0 [Note] InnoDB: File './ibtmp1' size is now 12.000MiB.
2021-12-09  9:54:04 0 [Note] InnoDB: 10.7.2 started; log sequence number 41349; transaction id 14
2021-12-09  9:54:04 0 [Note] InnoDB: Loading buffer pool(s) from /tmp/build-mariadb-server-10.7-datadir/ib_buffer_pool
2021-12-09  9:54:04 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-12-09  9:54:04 0 [Note] InnoDB: Buffer pool(s) load completed at 211209  9:54:04
2021-12-09  9:54:04 0 [Note] sql/mysqld: ready for connections.
Version: '10.7.2-MariaDB'  socket: '/tmp/build-mariadb-server-10.7.sock'  port: 0  Source distribution
2021-12-09  9:56:57 0 [Note] sql/mysqld (initiated by: unknown): Normal shutdown
2021-12-09  9:56:57 0 [Note] InnoDB: FTS optimize thread exiting.
2021-12-09  9:56:57 0 [Note] InnoDB: Starting shutdown...
2021-12-09  9:56:57 0 [Note] InnoDB: Dumping buffer pool(s) to /tmp/build-mariadb-server-10.7-datadir/ib_buffer_pool
2021-12-09  9:56:57 0 [Note] InnoDB: Buffer pool(s) dump completed at 211209  9:56:57
2021-12-09  9:56:57 0 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1"
2021-12-09  9:56:57 0 [Note] InnoDB: Shutdown completed; log sequence number 42602; transaction id 15
2021-12-09  9:56:57 0 [Note] sql/mysqld: Shutdown complete

The previous default innodb_buffer_pool_chunk_size of 128M
made sense when the innodb buffer pool size was a few GB.

When the pool size is 128GB this means the chunk size is 0.1%
of this. Fine tuning the buffer pool size on such a fine
increment doesn't make practical sense. Also on extremely
large buffer pool systems, initializing on the default 128M can
also take a considerable amount of time.

When large pages are enabled, the chunk size has to be a multiple
of an available large page size or memory allocation without
use can occur.

Previously the default 0 was documented as disabling resizing.
With srv_buf_pool_chunk_unit > 0 assertions in the code and the
minimium value set, I doubt this was ever the case.

As such the autosizing (based on default 0) takes place as follows:
* a 64th of the innodb_buffer_pool_size
* if large pages, this is rounded down the the nearest multiple
  of the large page size.
* If less than 1MB, set to 1MB.

This does mean the new default innodb_buffer_pool_chunk size is
2MB, derived form the above formular with 128MB as the buffer pool
size.

The innodb_buffer_pool_chunk_size is changed to a size_t for
better compatiblity with the memory allocations which use size_t.
The previous upper limit is changed to the maxium of a size_t. The
maximium value used is the buffer pool size anyway.

Getting this default value of the chunk size to a more practical
size facilitates further development of more automated resizing
without significant overhead or memory fragmentation.

innodb_buffer_pool_resize test adjusted based on 1M default
chunk size thanks Wlad.

In commit 4c3ad244 (MDEV-27416)
an unnecessarily strict wait condition was introduced in the
function buf_flush_wait(). Most callers actually only care that
the pages have been flushed, not that a checkpoint has completed.

Only in the buf_flush_sync() call for log resizing, we might care
about the log checkpoint. But, in fact,
srv_prepare_to_delete_redo_log_file() is explicitly disabling
checkpoints. So, we can simply remove the unnecessary wait loop.

Thanks to Krunal Bauskar for reporting this performance regression
that we failed to repeat in our testing.

bump maturity to beta

MDEV-25373 DROP TABLE doesn't raise error while dropping non-existing table in MariaDB 10.5.9 when OQGraph SE is loaded to the server

don't auto-succeed every DROP TABLE

buf_pool_t::realloc(): Invoke page_cleaner_wakeup()
if buf_LRU_get_free_only() returns a null pointer.

Ever since commit 7b1252c0 (MDEV-24278)
the page cleaner would remain in untimed sleep, expecting explicit
calls to buf_pool_t::page_cleaner_wakeup() when the ratio of dirty pages
could change.

Failure to wake up the page cleaner will cause all page writes to be
initiated by buf_flush_LRU_list_batch(). That might work too,
provided that the buffer pool size is at least BUF_LRU_MIN_LEN (256)
pages, but it would not advance the log checkpoint.

In commit c5fd9aa5 (MDEV-25919)
we prevented the function dict_stats_save_index_stat()
from being called in read-only mode in dict_stats_save(),
but not elsewhere.

dict_stats_save_defrag_summary(), dict_stats_save_defrag_stats():
If the transaction is in read-only mode, return DB_READ_ONLY
and do not attempt to lock or modify anything.

When this file exists, Debian builds will automatically compare the built
ABI and symbols to the ones defined in the list. If there is a mismatch,
developers need to update the libmariadb3.symbols file, otherwise the build
fails.

This ensures there are no accidental symbol changes. This also helps track
what symbols changed and in what versions.

Also add a README embedded in the sources to facilitate correct use of this
new file.

This change is made for branch 10.2, as it was the first server version to
introduce an embedded libmariadb3 client library, and from this version
the file with the same libmariadb3 specific content will be merged to
10.3, 10.4 and 10.5.

Related: MDEV-21732

Windows builds failed due to the following error:
'#': invalid character: possibly the result of a macro expansion

The commit e954d9de gave different lifetime to wide_share and
partition_handler_share. This introduced the possibility that
partition_handler_share could be accessed even after it was freed.

We stop sharing partitoiin_handler_share and make it belong to
a single wide_handler to fix the problem.

In MDEV-14425, an early plan was to introduce a separate log file
for file-level records and checkpoint information. The reasoning was
that fil_system.mutex contention would be reduced by not having to
maintain fil_system.named_spaces. The mutex contention was actually
fixed in MDEV-23855 by making some data fields in fil_space_t and
fil_node_t use std::atomic.

Using a single circular log file simplifies recovery and backup.

The function buf_page_free() that was introduced
in commit a35b4ae8 (MDEV-15528)
failed to remove any adaptive hash index entries for the page
before freeing the page.

This caused an assertion failure on shutdown of 10.6 server of
in the function buf_pool_t::clear_hash_index() with the expression:
(s >= buf_page_t::UNFIXED || s == buf_page_t::REMOVE_HASH).
The assertion would fail for a block that is in the freed state.

The failing assertion was added in
commit aaef2e1d
in the 10.6 branch.

Thanks to Matthias Leich for finding the bug and testing the fix.

buf_page_get_gen(): After recv_sys_t::recover_low() returned,
the page must not be read-fixed, but it may be write-fixed,
because the io-fix state is protected by block->page.lock,
which we are not holding yet.

Also, let us copy the block descriptor state to a local variable
for examination, so that in case an assertion would fail again,
we will have the sampled state in the core dump. In a core dump of
the assertion failure, we had block->page.fix() == buf_page_t::UNFIXED,
that is, the assertion expression was holding again.

create_table_info_t::create_foreign_keys() expects equal number of
iterations through fk->columns and fk->ref_columns. If fk->ref_columns
is empty copy it from fk->columns.

  - InnoDB fails to apply the bulk insert operation for the stats
table during DDL. During create table, InnoDB does dict_stats_save()
and freshly insert into innodb_table_stats and innodb_index_stats
table.

 InnoDB fails to add the tuple size which is greater than
innodb_sort_buffer_size. InnoDB should write the field
which are greater than 2000 bytes into the temporary file
and place the offset, length and make it as a new tuple.
InnoDB should buffer the newly created tuple without any
problem during bulk index

This follows up commit 017d1b86.
In commit aaef2e1d (MDEV-27058)
some more problematic debug assertions were added.

btr_search_update_block_hash_info(), trx_purge_truncate_history():
Use simpler assertions to check that an uncompressed page is present.

mtr_t::modify(): Remove a debug assertion that had been added
in commit 05fa4558 (MDEV-22110).
The function buf_pool_t::is_uncompressed() is only safe to invoke
while holding a buf_pool.page_hash latch so that buf_pool_t::resize()
cannot concurrently invoke free() on any chunks.

The problem was introduced by the removal of buf_pool.flush_rbt
in commit 46b1f500 (MDEV-23399)

recv_sys_t::apply(): don't write to disc and fsync() the last batch.
Insead, sort it by oldest_modification for MariaDB server and some
mariabackup operations.

log_sort_flush_list(): a thread-safe function which sorts buf_pool::flush_list

trx_has_lock_x() fails to find whether the trx has X-lock on the table
when other transactions are waiting for an X or S lock on the table.

There was an intention to add a CommandQueue in
mysql/mysql-server@eca5b0fc17a5bd6d4833d35a0d08c8549dd3b5ec
but it never appeared in any release (not even MySQL 5.7.3
where that commit appeared).

Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK)

Analysis: KILL_QUERY is not ignored when local memory used exceeds maximum
session memory. Hence the query proceeds, OK is sent and we end up
reopening tables that are marked for reopen. During this, kill status is
eventually checked and assertion failure happens during trying to send error
message because OK has already been sent.
Fix: Ok is already sent so statement has already executed. It is too
late to give error. So ignore kill.

The warning reads:

CMake Deprecation Warning at CMakeLists.txt:101 (CMAKE_MINIMUM_REQUIRED):
Compatibility with CMake < 2.8.12 will be removed from a future version of
CMake.

In commit 18535a40 (MDEV-24811)
the implementation of innodb_evict_tables_on_commit_debug
depended on dict_sys.mutex and SAFE_MUTEX.
That is no longer the case.

SAFE_MUTEX is not available on Microsoft Windows.

The mysql_install_db script is used to initialize the data directory.
However, if the user needs to apply some customized init commands (for
example to change user or password) they have to start the database
with `--skip-grant-tables` after the install script, and then restart
the database with normal mode.

To make it easier to include customization in the mysql_install_db
script, in this commit, a new parameter `extra-file` is added in
mysql_install_db script to support some extra customized init commands.

With this option we can support applying extra file containing sql for
mysql_install_db command line at runtime, which reduces the complexity
of customized initialization process.

This script is only used for install and is not included in the mtr
bootstrap file.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.