InnoDB tables that lack a primary key (and any UNIQUE INDEX whose
all columns are NOT NULL) will use an internally generated index,
called GEN_CLUST_INDEX(DB_ROW_ID) in the InnoDB data dictionary,
and hidden from the SQL layer.

The 48-bit (6-byte) DB_ROW_ID is being assigned from a
global sequence that is persisted in the DICT_HDR page.

There is absolutely no reason for the DB_ROW_ID to be globally
unique across all InnoDB tables.

A downgrade to earlier versions will be prevented by the file format
change related to removing the InnoDB change buffer (MDEV-29694).

DICT_HDR_ROW_ID, dict_sys_t::row_id: Remove.

dict_table_t::row_id: The per-table sequence of DB_ROW_ID.

commit_try_rebuild(): Copy dict_table_t::row_id from the old table.

btr_cur_instant_init(), row_import_cleanup(): If needed, perform
the equivalent of SELECT MAX(DB_ROW_ID) to initialize
dict_table_t::row_id.

row_ins(): If needed, obtain DB_ROW_ID from dict_table_t::row_id.
Should it exceed the maximum 48-bit value, return DB_OUT_OF_FILE_SPACE
to prevent further inserts into the table.

dict_load_table_one(): Move a condition to btr_cur_instant_init_low()
so that dict_table_t::row_id will be restored also for
ROW_FORMAT=COMPRESSED tables.

Tested by: Matthias Leich

The purpose of the change buffer was to reduce random disk access,
which could be useful on rotational storage, but maybe less so on
solid-state storage.
When we wished to
(1) insert a record into a non-unique secondary index,
(2) delete-mark a secondary index record,
(3) delete a secondary index record as part of purge (but not ROLLBACK),
and the B-tree leaf page where the record belongs to is not in the buffer
pool, we inserted a record into the change buffer B-tree, indexed by
the page identifier. When the page was eventually read into the buffer
pool, we looked up the change buffer B-tree for any modifications to the
page, applied these upon the completion of the read operation. This
was called the insert buffer merge.

We remove the change buffer, because it has been the source of
various hard-to-reproduce corruption bugs, including those fixed in
commit 5b9ee8d8 and
commit 165564d3 but not limited to them.

A downgrade will fail with a clear message starting with
commit db14eb16 (MDEV-30106).

buf_page_t::state: Merge IBUF_EXIST to UNFIXED and
WRITE_FIX_IBUF to WRITE_FIX.

buf_pool_t::watch[]: Remove.

trx_t: Move isolation_level, check_foreigns, check_unique_secondary,
bulk_insert into the same bit-field. The only purpose of
trx_t::check_unique_secondary is to enable bulk insert into an
empty table. It no longer enables insert buffering for UNIQUE INDEX.

btr_cur_t::thr: Remove. This field was originally needed for change
buffering. Later, its use was extended to cover SPATIAL INDEX.
Much of the time, rtr_info::thr holds this field. When it does not,
we will add parameters to SPATIAL INDEX specific functions.

ibuf_upgrade_needed(): Check if the change buffer needs to be updated.

ibuf_upgrade(): Merge and upgrade the change buffer after all redo log
has been applied. Free any pages consumed by the change buffer, and
zero out the change buffer root page to mark the upgrade completed,
and to prevent a downgrade to an earlier version.

dict_load_tablespaces(): Renamed from
dict_check_tablespaces_and_store_max_id(). This needs to be invoked
before ibuf_upgrade().

btr_cur_open_at_rnd_pos(): Specialize for use in persistent statistics.
The change buffer merge does not need this function anymore.

btr_page_alloc(): Renamed from btr_page_alloc_low(). We no longer
allocate any change buffer pages.

btr_cur_open_at_rnd_pos(): Specialize for use in persistent statistics.
The change buffer merge does not need this function anymore.

row_search_index_entry(), btr_lift_page_up(): Add a parameter thr
for the SPATIAL INDEX case.

rtr_page_split_and_insert(): Specialized from btr_page_split_and_insert().

rtr_root_raise_and_insert(): Specialized from btr_root_raise_and_insert().

Note: The support for upgrading from the MySQL 3.23 or MySQL 4.0
change buffer format that predates the MySQL 4.1 introduction of
the option innodb_file_per_table was removed in MySQL 5.6.5
as part of mysql/mysql-server@69b6241a79876ae98bb0c9dce7c8d8799d6ad273
and MariaDB 10.0.11 as part of 1d0f70c2.

In the tests innodb.log_upgrade and innodb.log_corruption, we create
valid (upgraded) change buffer pages.

Tested by: Matthias Leich

We introduce the following settable Boolean global variables:

innodb_log_file_write_through: Whether writes to ib_logfile0 are
write-through (disabling any caching, as in O_SYNC or O_DSYNC).

innodb_data_file_write_through: Whether writes to any InnoDB data files
(including the temporary tablespace) are write-through.

innodb_data_file_buffering: Whether the file system cache is enabled
for InnoDB data files.

All these parameters are OFF by default, that is, the file system cache
will be disabled, but any hardware caching is enabled, that is,
explicit calls to fsync(), fdatasync() or similar functions are needed.

On systems that support FUA it may make sense to enable write-through,
to avoid extra system calls.

If the deprecated read-only start-up parameter is set to one of the
following values, then the values of the 4 Boolean flags (the above 3
plus innodb_log_file_buffering) will be set as follows:

O_DSYNC:
innodb_log_file_write_through=ON, innodb_data_file_write_through=ON,
innodb_data_file_buffering=OFF, and
(if supported) innodb_log_file_buffering=OFF.

fsync, littlesync, nosync, or (Microsoft Windows specific) normal:
innodb_log_file_write_through=OFF, innodb_data_file_write_through=OFF,
and innodb_data_file_buffering=ON.

Note: fsync() or fdatasync() will only be disabled if the separate
parameter debug_no_sync (in the code, my_disable_sync) is set.

In mariadb-backup, the parameter innodb_flush_method will be ignored.

The Boolean parameters can be modified by SET GLOBAL while the
server is running. This will require reopening the ib_logfile0
or all currently open InnoDB data files.

We will open files straight in O_DSYNC or O_SYNC mode when applicable.
Data files we will try to open straight in O_DIRECT mode when the
page size is at least 4096 bytes. For atomically creating data files,
we will invoke os_file_set_nocache() to enable O_DIRECT afterwards,
because O_DIRECT is not supported on some file systems. We will also
continue to invoke os_file_set_nocache() on ib_logfile0 when
innodb_log_file_buffering=OFF can be fulfilled.

For reopening the ib_logfile0, we use the same logic that was developed
for online log resizing and reused for updates of
innodb_log_file_buffering.

Reopening all data files is implemented in the new function
fil_space_t::reopen_all().

Reviewed by: Vladislav Vaintroub
Tested by: Matthias Leich

Before commit 6112853c in MySQL 4.1.1
introduced the parameter innodb_file_per_table, all InnoDB data was
written to the InnoDB system tablespace (often named ibdata1).
A serious design problem is that once the system tablespace has grown to
some size, it cannot shrink even if the data inside it has been deleted.

There are also other design problems, such as the server hang MDEV-29930
that should only be possible when using innodb_file_per_table=0 and
innodb_undo_tablespaces=0 (storing both tables and undo logs in the
InnoDB system tablespace).

The parameter innodb_change_buffering was deprecated
in commit b5852ffb.
Starting with commit baf276e6
(MDEV-19229) the number of innodb_undo_tablespaces can be increased,
so that the undo logs can be moved out of the system tablespace
of an existing installation.

If all these things (tables, undo logs, and the change buffer) are
removed from the InnoDB system tablespace, the only variable-size
data structure inside it is the InnoDB data dictionary.

DDL operations on .ibd files was optimized in
commit 86dc7b4d (MDEV-24626).
That should have removed any thinkable performance advantage of
using innodb_file_per_table=0.

Since there should be no benefit of setting innodb_file_per_table=0,
the parameter should be deprecated. Starting with MySQL 5.6 and
MariaDB Server 10.0, the default value is innodb_file_per_table=1.

The MDEV-25004 test innodb_fts.versioning is omitted because ever since
commit 685d958e InnoDB would not allow
writes to a database where the redo log file ib_logfile0 is missing.

			to copy datafile

- Mariabackup fails to copy the undo log tablespace when it undergoes
truncation. So Mariabackup should detect the redo log which does
undo tablespace truncation and also backup should read the minimum
file size of the tablespace and ignore the error while reading.

- Throw error when innodb undo tablespace read failed, but backup
doesn't find the redo log for undo tablespace truncation

As a consequence of this it doesn't need to manually set CPACK_PACKAGE_NAME
or CPACK_RPM_PACKAGE_NAME.

- Prefer Breaks+Replaces over Conflicts+Replaces in debian/control.

- Clean away conflict/break/replace with self as it either has no
  effect or harmfully prevents upgrades to self (packages from this
  source version).

- Remove references to mariadb-server-10.11 as the package names no
  longer have versions and thus such references are unnecessary cruft.

- Apply 'wrap-and-sort -av' and minor typo fixing.

- Extend Salsa-CI to test for upgrades from 10.9 and 10.10 to 10.11.

- Add minimal required new Lintian overrides so Salsa-CI would not
  fail on Lintian.

        fix up version detection in debian/additions/innotop
        after 10->11

procedures

MDEV-22224 caused the parsing of keys with hyphens to break by setting
the state transitions for parsing keys to JE_SYN (syntax error) when
they encounter a hyphen. However json key names may contain hyphens and
still be considered valid json.

This patch changes the state transition table so that key names with
hyphens remain valid. Note that unquoted key names in paths like
$.key-name are also valid again. This restores the previous behaviour
when hyphens were considered part of the P_ETC character class.

In the Develop package because of their use from sql_class.h
which is the main file for THD needed by server plugins.

Since 7c58e97b the PSI_memory_key was added to some routines in the
mysys/. This commit fixes synopses of functions that were updated with
the PSI_memory_key parameter.

MDEV-30325 Wrong result upon range query using index condition wrong result upon range query using index condition

This was caused by a bug in key_or() when SEL_ARG* key1 has been cloned
and is overlapping with SEL_ARG *key2

Cloning of SEL_ARG's happens only in very special cases, which is why this
bug has remained undetected for years.

It happend in the following query:

SELECT COUNT(*) FROM lineitem force index
(i_l_orderkey_quantity,i_l_shipdate) WHERE
l_shipdate < '1994-01-01' AND l_orderkey < 800 OR
l_quantity > 3 AND l_orderkey NOT IN ( 157, 1444 );

Because there are two different indexes that can be used and the code for
IN causes a 'tree_or', which causes all SEL_ARG's to be cloned.

Other things:
- While checking the code, I found a bug in SEL_ARG::SEL_ARG(SEL_ARG &arg)
 - This was incrementing next_key_part->use_count as part of creating a
   copy of an existing SEL_ARG.
   This is however not enough as the 'reverse operation' when the copy is
   not needed is 'key2_cpy.increment_use_count(-1)', which does something
   completely different.
   Fixed by calling increment_use_count(1) in SEL_ARG::SEL_ARG.

- Returns DB_LOCK_WAIT_TIMEOUT for the stats_lock_fail
debug sync point

When building with -DWITH_WSREP=OFF, files required for MTR tests are
excluded and several tests fail. This is cause by a recent commit
7b44d0ba which attempted to resolve MDEV-23230.

Even when building without WSREP/Galera support some of the MTR include
files named *wsrep* are required by other tests.

Removing the following from the CMake install macros will avoid
excluding the MTR test .inc files:
`|include/((w.*)?wsrep.*|.*galera.*)\\.inc`

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer
Amazon Web Services, Inc.

Using clang generates the unused-function warning on embedded like:

FAILED: libmysqld/CMakeFiles/sql_embedded.dir/__/sql/sql_acl.cc.o
/usr/lib64/ccache/clang++ .... libmysqld/CMakeFiles/sql_embedded.dir/__/sql/sql_acl.cc.o -MF libmysqld/CMakeFiles/sql_embedded.dir/__/sql/sql_acl.cc.o.d -o libmysqld/CMakeFiles/sql_embedded.dir/__/sql/sql_acl.cc.o -c /home/dan/repos/mariadb-server-10.11/sql/sql_acl.cc

sql_acl.cc:113:20: error: unused function 'is_public' [-Werror,-Wunused-function]
static inline bool is_public(const LEX_USER *l) { return is_public(&l->user); }

The problem was that when storing rows into a temporary table,
MIN/MAX items that where marked as constants (as theire value had
been computed at start of query) would be reset.

Fixed by not reseting MIN/MAX items that are marked as const in
Item_sum_min_max::clear().

MDEV-30154: Assertion `strcasecmp(rolename, public_name.str) || acl_public == role' failed in acl_update_role on GRANT ... TO PUBLIC

Reset of acl_public was made too early (before saving it to restore in case
of error).

--header applied an ORDER BY to ensure that the header
row was the first row in the output given UNION ALL
doesn't in the standard enforce the order.

We change that now only add the ORDER BY if --order-by-primary
is used.

An assumption that if UNION ALL change to a different behaviour
the resulting mysqldump-header test may also change.

* mariadb-service-convert to use mariadbd-safe
* galera_recovery to use mariadbd
* mtr - wsrep use mariadb executables
* debian/mariadb-server.mariadb.init use mariadbd-safe
* debian/tests/smoke uses mariadb instead of mysql as client.

Co-Author: Daniel Black <daniel@mariadb.org>

move mariadb_capi_rename.h out of private server headers,
because it's included by mysql.h which is not private.

custom code in `case OPT_SKIP_RESOLVE`
was overriding the correct value from handle_options().

covers `git commit -a` and `git citool` (`git add`, `git status`, etc).
they can still be added explicitly

the parser couldn't parse `1=2 not between 3 and 5`
after `2` it expected only NOT2_SYM, but not NOT_SYM
(visible from the sql_yacc.output file), which resulted in
Syntax error ... near 'not between 3 and 4'

The parser was confused by a rather low NOT_SYM precedence and
%prec BETWEEN_SYM didn't resolve this confusion.

As a fix, let's remove any %precedence from NOT_SYM and
specify %prec explicitly in the only place where it matters for NOT_SYM.

In other places, such as for NOT BETWEEN, NOT_SYM won't have a
precedence, so bison won't be confused about it.

Without Galera, mariabackup should ignore the --galera-info option
and not fail with rc != 0 like it does now. This commit fixes this flaw.