MDEV-30133 MariaDB startup does not validate plugin-wsrep-provider when wsrep_mode=off or wsrep_provider is not set

Refuse to start if WSREP_ON=OFF or wsrep_provider is not set or
it is set to 'none' if plugin-wsrep-provider is used.

When wsrep-provider-options plugin is initialized we need to update
wsrep-provider-options variable as READ_ONLY.

- Provider options are read from the provider during
  startup, before plugins are initialized.
- New wsrep_provider plugin for which sysvars are generated
  dynamically from options read from the provider.
- The plugin is enabled by option plugin-wsrep-provider=ON.
  If enabled, wsrep_provider_options can no longer be used,
  (an error is raised on attempts to do so).
- Each option is either string, integer, double or bool
- Options can be dynamic / readonly
- Options can be deprecated

Limitations:

- We do not check that the value of a provider option falls
  within a certain range. This type of validation is still
  done in Galera side.
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

For both Deb and RPM, create mariadb-client-compat and
mariadb-server-compat containing the mysql links to the mariadb
named executables/scripts.

The mariadb-client-core mysqlcheck was moved to mariadb-client-compat.

The symlinks in MYSQL_ADD_EXECUTABLE is tagged as a
{Client,Server}Symlinks component and placed in
the symlinks packages.

Man pages are restructured be installed into compat package
if that matches the executable.

Columnstore has a workaround as it doesn't use the cmake/plugin.cmake.

Scripts likewise have compatibility symlinks are in
the {server,client}-compat packages.

Co-author: Andrew Hutchings <andrew@linuxjedi.co.uk>

Closes #2390

like where the man page and Debian package put it.

* move them from ManPagesX component to X (works better for plugins),
  but keep ManPagesDevelopment as C/C is using it
* move backup manpages to Backup
* move plugin manpages (s3, rocksdb) to plugins

for consistency

and remove unused function INSTALL_MANPAGE

Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

don't include my_progname in the error message, my_error starts from it
automatically, resulting in, like

/usr/bin/mysqladmin: Notice: /usr/bin/mysqladmin is deprecated and will be removed in a future release, use command 'mariadb-admin'

and remove "Notice" so that the problem description would directly
follow the executable name.

make the check to work when the executable is in the PATH
(so, invoked simply like 'mysql' and thus readlink cannot find it)

fix the check in mysql_install_db and mysql_secure_installation to not
print the warning if the intermediate path contains "mysql" substring

add this message also to
* mysql_waitpid
* mysql_convert_table_format
* mysql_find_rows
* mysql_setpermissions
* mysqlaccess
* mysqld_multi
* mysqld_safe
* mysqldumpslow
* mysqlhotcopy
* mysql_ldb

Closes #2273

Eventually mysql symlinks will go away, as MariaDB and MySQL keep
diverging and we do not want to make it impossible to install
MariaDB and MySQL side-by-side when users want it.

It also useful if people start using MariaDB tools with MariaDB.

If the exe doesn't begine with "mariadb" or is a symlink,
print a warning to use the resolved name.

In my_readlink, add check on my_thread_var as its used by comp_err
and other build utils that also use my_init.

Renames the upgrade state file, and ensures the old
file is properly removed when `mariadb-upgrade` tool is executed.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.

SUPER privilege used to allow various actions that were alternatively
allowed by one of BINLOG ADMIN, BINLOG MONITOR, BINLOG REPLAY,
CONNECTION ADMIN, FEDERATED ADMIN, REPL MASTER ADMIN, REPL SLAVE ADMIN,
SET USER, SLAVE MONITOR.

Now SUPER no longer does that, one has to grant one of the fine-grained
privileges above to be to perform corresponding actions.

On upgrade from MariaDB versions 10.11 and below all the privileges
above are granted automatically if the user has SUPER.

As a side-effect, such an upgrade will allow SUPER-user to run SHOW
BINLOG EVENTS, SHOW RELAYLOG EVENTS, SHOW SLAVE HOSTS, even if he wasn't
able to do it before the upgrade.

including patches from Andrei Elkin

This is intended to be the start of a (not complete) coding standards
document we can refer contributors to. This can be modified to add more
nuances and become stricter over time. It can also have additional
content for other file types (CMake, YACC, etc).

It does not cover plugins which should each individually have their own coding
standards.

no longer needed, MySQL replication was fixed meanwhile.

client code still can recognize and strip the prefix though.

The Boolean flag mlog_init_t::init::created was only needed by
mark_ibuf_exist(), which commit f27e9c89
removed. We only need to store the page initialization LSN in the map.

buf_LRU_get_free_block(): Replace the Boolean parameter with a
ternary parameter, so that have_no_mutex_soft can be specified
reduce the chances of initiating page eviction flushing in read-ahead.

buf_read_acquire(): Invoke buf_LRU_get_free_block(have_no_mutex_soft)
and check in each caller for a nullptr return value.

buf_pool_t::page_hash_contains(): Check if a page is cached.

buf_read_ahead_random(), buf_read_page_background(),
buf_read_ahead_linear(): Before invoking buf_read_page_low(),
preallocate a buffer page for the read request.

buf_read_page(), buf_page_init_for_read(), buf_read_page_low():
Add a parameter for the buf_pool.page_hash chain, to avoid duplicated
computations.

buf_page_t::read_complete(): Only attempt recovery if an uncompressed
page frame has been allocated.

buf_page_init_for_read(): Before trying to acquire buf_pool.mutex, acquire
an exclusive buf_pool.page_hash latch and check if the page is already
located in the buffer pool. If the buf_pool.mutex is not immediately
available, release both latches and acquire them in the correct order,
and then recheck if the page is already in the buffer pool. This should
hopefully reduce some contention on buf_pool.mutex.

buf_page_init_for_read(), buf_read_page_low(): Input the "recovery needed"
flag in the least significant bit of zip_size.

buf_read_acquire(), buf_read_release(): Interface for allocating and
freeing buffer pages for reading.

buf_read_recv_pages(): Set the flag that recovery is needed.
Other ROW_FORMAT=COMPRESSED reads during recovery
will not need any recovery.

This also fixes part of MDEV-29835 Partial server freeze
which is caused by violations of the latching order that was
defined in https://dev.mysql.com/worklog/task/?id=6326
(WL#6326: InnoDB: fix index->lock contention). Unless the
current thread is holding an exclusive dict_index_t::lock,
it must acquire page latches in a strict parent-to-child,
left-to-right order. Not all cases of MDEV-29835 are fixed yet.
Failure to follow the correct latching order will cause deadlocks
of threads due to lock order inversion.

As part of these changes, the BTR_MODIFY_TREE mode is modified
so that an Update latch (U a.k.a. SX) will be acquired on the
root page, and eXclusive latches (X) will be acquired on all pages
leading to the leaf page, as well as any left and right siblings
of the pages along the path. The DEBUG_SYNC test innodb.innodb_wl6326
will be removed, because at the time the DEBUG_SYNC point is hit,
the thread is actually holding several page latches that will be
blocking a concurrent SELECT statement.

We also remove double bookkeeping that was caused due to excessive
information hiding in mtr_t::m_memo. We simply let mtr_t::m_memo
store information of latched pages, and ensure that
mtr_memo_slot_t::object is never a null pointer.
The tree_blocks[] and tree_savepoints[] were redundant.

buf_page_get_low(): If innodb_change_buffering_debug=1, to avoid
a hang, do not try to evict blocks if we are holding a latch on
a modified page. The test innodb.innodb-change-buffer-recovery
will be removed, because change buffering may no longer be forced
by debug injection when the change buffer comprises multiple pages.
Remove a debug assertion that could fail when
innodb_change_buffering_debug=1 fails to evict a page.
For other cases, the assertion is redundant, because we already
checked that right after the got_block: label. The test
innodb.innodb-change-buffering-recovery will be removed, because
due to this change, we will be unable to evict the desired page.

mtr_t::lock_register(): Register a change of a page latch
on an unmodified buffer-fixed block.

mtr_t::x_latch_at_savepoint(), mtr_t::sx_latch_at_savepoint():
Replaced by the use of mtr_t::upgrade_buffer_fix(), which now
also handles RW_S_LATCH.

mtr_t::set_modified(): For temporary tables, invoke
buf_page_t::set_modified() here and not in mtr_t::commit().
We will never set the MTR_MEMO_MODIFY flag on other than
persistent data pages, nor set mtr_t::m_modifications when
temporary data pages are modified.

mtr_t::commit(): Only invoke the buf_flush_note_modification() loop
if persistent data pages were modified.

mtr_t::get_already_latched(): Look up a latched page in mtr_t::m_memo.
This avoids many redundant entries in mtr_t::m_memo, as well as
redundant calls to buf_page_get_gen() for blocks that had already
been looked up in a mini-transaction.

btr_get_latched_root(): Return a pointer to an already latched root page.
This replaces btr_root_block_get() in cases where the mini-transaction
has already latched the root page.

btr_page_get_parent(): Fetch a parent page that was already latched
in BTR_MODIFY_TREE, by invoking mtr_t::get_already_latched().
If needed, upgrade the root page U latch to X.
This avoids bloating mtr_t::m_memo as well as performing redundant
buf_pool.page_hash lookups. For non-QUICK CHECK TABLE as well as for
B-tree defragmentation, we will invoke btr_cur_search_to_nth_level().

btr_cur_search_to_nth_level(): This will only be used for non-leaf
(level>0) B-tree searches that were formerly named BTR_CONT_SEARCH_TREE
or BTR_CONT_MODIFY_TREE. In MDEV-29835, this function could be
removed altogether, or retained for the case of
CHECK TABLE without QUICK.

btr_cur_t::left_block: Remove. btr_pcur_move_backward_from_page()
can retrieve the left sibling from the end of mtr_t::m_memo.

btr_cur_t::open_leaf(): Some clean-up.

btr_cur_t::search_leaf(): Replaces btr_cur_search_to_nth_level()
for searches to level=0 (the leaf level). We will never release
parent page latches before acquiring leaf page latches. If we need to
temporarily release the level=1 page latch in the BTR_SEARCH_PREV or
BTR_MODIFY_PREV latch_mode, we will reposition the cursor on the
child node pointer so that we will land on the correct leaf page.

btr_cur_t::pessimistic_search_leaf(): Implement new BTR_MODIFY_TREE
latching logic in the case that page splits or merges will be needed.
The parent pages (and their siblings) should already be latched on
the first dive to the leaf and be present in mtr_t::m_memo; there
should be no need for BTR_CONT_MODIFY_TREE. This pre-latching almost
suffices; it must be revised in MDEV-29835 and work-arounds removed
for cases where mtr_t::get_already_latched() fails to find a block.

rtr_search_to_nth_level(): A SPATIAL INDEX version of
btr_search_to_nth_level() that can search to any level
(including the leaf level).

rtr_search_leaf(), rtr_insert_leaf(): Wrappers for
rtr_search_to_nth_level().

rtr_search(): Replaces rtr_pcur_open().

rtr_latch_leaves(): Replaces btr_cur_latch_leaves(). Note that unlike
in the B-tree code, there is no error handling in case the sibling
pages are corrupted.

rtr_cur_restore_position(): Remove an unused constant parameter.

btr_pcur_open_on_user_rec(): Remove the constant parameter
mode=PAGE_CUR_GE.

row_ins_clust_index_entry_low(): Use a new
mode=BTR_MODIFY_ROOT_AND_LEAF to gain access to the root page
when mode!=BTR_MODIFY_TREE, to write the PAGE_ROOT_AUTO_INC.

BTR_SEARCH_TREE, BTR_CONT_SEARCH_TREE: Remove.

BTR_CONT_MODIFY_TREE: Note that this is only used by
rtr_search_to_nth_level().

btr_pcur_optimistic_latch_leaves(): Replaces
btr_cur_optimistic_latch_leaves().

ibuf_delete_rec(): Acquire exclusive ibuf.index->lock in order
to avoid a deadlock with ibuf_insert_low(BTR_MODIFY_PREV).

btr_blob_log_check_t(): Acquire a U latch on the root page,
so that btr_page_alloc() in btr_store_big_rec_extern_fields()
will avoid a deadlock.

btr_store_big_rec_extern_fields(): Assert that the root page latch
is being held.

Tested by: Matthias Leich
Reviewed by: Vladislav Lesin

- introduce table key construction function in wsrep service interface
- don't add row keys when replicating bulk insert
- don't start bulk insert on applier or when transaction is not active
- don't start bulk insert on system versioned tables
- implement actual bulk insert table-level key replication
Reviewed-by: Jan Lindström <jan.lindstrom@mariadb.com>

During testing of RPM packages in MDEV-30203:
  file /usr/share/man/man3 from install of
  MariaDB-devel-11.0.1-1.el7_9.x86_64 conflicts with file from
  package filesystem-3.2-25.el7.x86_64

MariaDB is the first libmariadb to include man3 man pages
so make the changes here like what is done for man1 and man8.

Fall back to using .mysql_history if .mariadb_history isn't
present and .mysql_history is present.

Also replace the use of MYSQL_HISTFILE as an environment variable
with MARIADB_HISTFILE and fall back to using MYSQL_HISTFILE if
MARIADB_HISTFILE isn't present.

Problem:
========
- InnoDB fails to remove the newly created table or index from
data dictionary and table cache if the alter fails in commit phase

Solution:
========
- InnoDB should restart the transaction to remove the newly
created table and index when it fails in commit phase of an alter
operation. innodb_fts.misc_debug tests the scenario with the
help of debug point "stats_lock_fail"

In commit 24648768, some use of
O_DIRECT was added without proper #ifdef guard. That broke the
compilation in environments that do not define O_DIRECT, such as
OpenBSD.