Part1: Functional changes

Backporting upstream changes.
commit a7e1ef858ee82493dd8ad9a76bc9c22fe3b8c05b
Author: Neha Kumari <neha.n.kumari@oracle.com>
Note:
From the upstream patch only the new option binlog_expire_logs_seconds
specific changes are taken.

* Unlike in the upstream patch 'binlog_expire_logs_seconds' does not
  replace the "old" 'expire_logs_days', to preserve backward-compatibility.

* Datatype of 'expire_logs_days' variable is changed to double.

* Default value of 'binlog_expire_logs_seconds=0' similar to
  'expire_logs_days'.

* The purge_time can be specified in days with the micro-day precision.
  Eg:
  expire_logs_days=1 is the same as expire_logs_days=1.000000 to make
    binlog_expire_logs_seconds=86400.
  binlog_expire_logs_seconds=1 is the same as expire_logs_days=0.000012.

* If binary log is disabled and option 'expire_logs_days' or
  'binlog_expire_logs_seconds' used with purge_time > 0 a warning will be
  issued.

If IPO is enabled, MSVC compiles with /GL  which is not
compatible with cmake's WINDOWS_EXPORT_ALL_SYMBOLS.

Since server.dll can't autoexport symbols from /GL compiled objects,
IPO must be disabled for dbug, string,mysys, and sql targets (in case we
build "plugin" DLLs that need server.dll's symbols)

The warning brought by code analysis, unsafe arithmetic in timer related code

Avoid calling constructor for Protocol_text, if not necessary.

Remove operations on fil_system.n_open from mariabackup, as they are not
protected by the mutex, and serve no higher purpose anyway.

dict_drop_index_tree(): Even if SYS_INDEXES.PAGE contains the
special value FIL_NULL, the tablespace identified by SYS_INDEXES.SPACE
may exist and may need to be dropped. This would definitely be the case
if the server had been killed right after a FILE_CREATE record was
persistently written during CREATE TABLE, but before the transaction
was committed.

btr_free_if_exists(): Simplify the interface, to avoid repeated
tablespace lookup.

One more scenario is known to be broken: If the server is killed
during DROP TABLE (or table-rebuilding ALTER TABLE) right after a
FILE_DELETE record has been persistently written but before the
file was deleted, then we could end up recovering no tablespace
at all, and failing to delete the file, in either of fil_name_process()
or dict_drop_index_tree().

Thanks to Elena Stepanova for providing "rr replay" and data directories
of these scenarios.

Make DDL operations that involve FULLTEXT INDEX atomic.
In particular, we must drop the internal FTS_ tables in the same
DDL transaction with ALTER TABLE.

Remove all references to fts_drop_orphaned_tables().

row_merge_drop_temp_indexes(): Drop also the internal FTS_ tables
that are associated with index stubs that were created in
prepare_inplace_alter_table_dict() for
CREATE FULLTEXT INDEX before the server was killed.

fts_clear_all(): Remove the fts_drop_tables() call. It has to be
executed before the transaction is committed!

dict_load_indexes(): Do not load any metadata for index stubs
that had been created by prepare_inplace_alter_table_dict()

fts_create_one_common_table(), fts_create_common_tables(),
fts_create_one_index_table(), fts_create_index_tables():
Remove redundant error handling. The tables will be dropped
just fine by dict_drop_index_tree().

commit_try_norebuild(): Also drop the FTS_ tables when dropping
FULLTEXT INDEX.

The changes to the test case innodb_fts.crash_recovery has been
extensively tested. The non-debug server will be killed while
the 3 ALTER TABLE are in any phase of execution. With the debug
server, DEBUG_SYNC should make the test deterministic.

058d93d4 introduced a switch in autobake
script to append dh-systemd package when it is necessary. However, the
change was not complete as it didn't actually remove the dh-systemd
dependency in debian/control.

after b9d64989 the test for MDEV-16962 is not suitable anymore
(and probably the bug is not reproducible).

1) Checked presence of mysys_var.
2) Removed unneeded thd_clear_errors,
  its content added to THD::change_user.

The test innodb.doublewrite could occasionally fail with 64KiB page size
because the page 0 would no longer be in the doublewrite buffer.
Let us stop purge before the server is killed, and ensure that
the entire buffer pool will be flushed before we initiate an extra
write of page 0.

galera_var_wsrep_on_off : Add wait conditions to make sure DDL is
replicated before continuing.

wsrep.[variables|variables_debug] :  Remove unnecessary parts
and add check to correct number of variables or skip

galera_ssl_reload: Add version check and SSL checks.

Fixed NULL pointer reference to db.str

Keep the readline installation step in Salsa-CI stages that install
MariaDB.org packages in Debian Sid (or releases after is, such as
Debian 11 "Bullseye" or Ubuntu 21.04 "Hirsute") as those old packages
still depend on readline for the MariaDB Server itself.

When you only need view structure, don't call handle_derived with
DT_CREATE and rely on its internal hackish check to skip DT_CREATE.
Because handle_derived is called from many different places,
and this internal hackish check is indiscriminative.

Instead, just don't ask handle_derived to do DT_CREATE
if you don't want it to do DT_CREATE.

Problem:
=======
Test assertion fails on slave.

Assertion text: 'Last_Seen_Transaction should show .'
Assertion condition: '"0-1-1" = ""'
Assertion condition, interpolated: '"0-1-1" = ""'
Assertion result: '0'

Analysis:
========
Test case creates a table on master and it waits for it to be replicated to
slave and applied by slave applier. On completion 'Last_Seen_Transaction'
value from 'performance_schema.replication_applier_status_by_worker' table
is compared with '@@gtid_slave_pos' to ensure its correctness. The test
should ensure that user table and 'gtid_slave_pos' table are of same engine
type 'InnoDB' to get consistent test results. This guarantees that
'gtid_slave_pos' table is updated as part of transaction commit. In the of
such an engine consistency, user table gets created in default MyISAM
storage engine and 'mysql.gtid_slave_pos' table gets created in Aria
storage engine. When the test code reaches above assert there might be a
delay in an update to 'gtid_slave_pos' table, which leads to test assertion
failure.

Fix:
===
Use InnoDB engine for both user table and 'mysql.gtid_slave_pos' table.

Before we create an InnoDB data file, we must have persistently
started a DDL transaction and written a record in SYS_INDEXES
as well as a FILE_CREATE record for creating the file.
In that way, if InnoDB is killed before the DDL transaction is
committed, the rollback will be able to delete the file in
dict_drop_index_tree().

dict_build_table_def_step(): Do not create the tablespace.
At this point, we have not written any log, not even for
inserting the SYS_TABLES record.

dict_create_sys_indexes_tuple(): Relax an assertion to tolerate
a missing tablespace before the first index has been created in
dict_create_index_step().

dict_build_index_def_step(): Relax the dict_table_open_on_name()
parameter, because no tablespace may be available yet.

tab_create_graph_create(), row_create_table_for_mysql(), tab_node_t:
Remove key_id, mode.

ind_create_graph_create(), row_create_index_for_mysql(), ind_node_t:
Add key_id, mode.

dict_create_index_space(): New function, to create the tablespace
during clustered index creation.

dict_create_index_step(): After the SYS_INDEXES record has been
written, invoke dict_create_index_space() to create the tablespace
if needed.

fil_ibd_create(): Before creating the file, persistently write a
FILE_CREATE record. This will also ensure that an incomplete DDL
transaction will be recovered. After creating the file, invoke
fsp_header_init().

InnoDB used to support at most one CREATE TABLE or DROP TABLE
per transaction. This caused complications for DDL operations on
partitioned tables (where each partition is treated as a separate
table by InnoDB) and FULLTEXT INDEX (where each index is maintained
in a number of internal InnoDB tables).

dict_drop_index_tree(): Extend the MDEV-24589 logic and treat
the purge or rollback of SYS_INDEXES records of clustered indexes
specially: by dropping the tablespace if it exists. This is the only
form of recovery that we will need.

trx_undo_ddl_type: Document the DDL undo log record types better.

trx_t::dict_operation: Change the type to bool.

trx_t::ddl: Remove.

trx_t::table_id, trx_undo_t::table_id: Remove.

dict_build_table_def_step(): Remove trx_t::table_id logging.

dict_table_close_and_drop(), row_merge_drop_table(): Remove.

row_merge_lock_table(): Merged to the only callers, which can
call lock_table_for_trx() directly.

fts_aux_table_t, fts_aux_id, fts_space_set_t: Remove.

fts_drop_orphaned_tables(): Remove.

row_merge_rename_index_to_drop(): Remove. Thanks to MDEV-24589,
we can simply delete the to-be-dropped indexes from SYS_INDEXES,
while still being able to roll back the operation.

ha_innobase_inplace_ctx: Make a few data members const.
Preallocate trx.

prepare_inplace_alter_table_dict(): Simplify the logic. Let the
normal rollback take care of some cleanup.

row_undo_ins_remove_clust_rec(): Simplify the parsing of SYS_COLUMNS.

trx_rollback_active(): Remove the special DROP TABLE logic.

trx_undo_mem_create_at_db_start(), trx_undo_reuse_cached():
Always write TRX_UNDO_TABLE_ID as 0.

galera_var_wsrep_on_off : Add wait conditions to make sure DDL is
replicated before continuing.

wsrep.[variables|variables_debug] :  Remove unnecessary parts
and add check to correct number of variables or skip

galera_ssl_reload: Add version check and SSL checks.

Fix a race condition in the testcase. The testcase assumed that
State='Sending data' means that the thread is already in an
InnoDB lock wait. This is not case, there is a gap between the
state changing to Sending data and execution reaching the point
where it is waiting for a lock.

Use a more precise check instead, through I_S.INNODB_TRX.