InnoDB was writing unnecessary information to the
update undo log records. Most notably, if an indexed column is updated,
the old value of the column would be logged twice: first as part of
the update vector, and then another time because it is an indexed column.

Because the InnoDB undo log record must fit in a single page,
this would cause unnecessary failure of certain updates.

Even after this fix, InnoDB still seems to be unnecessarily logging
indexed column values for non-updated columns. It seems that non-updated
secondary index columns only need to be logged when a PRIMARY KEY
column is updated. To reduce risk, we are not fixing this remaining flaw
in GA versions.

trx_undo_page_report_modify(): Log updated indexed columns only once.

The method Field_iterator_table::create_item() must take into
account that it can be called when processing DELETE RETURNING.

An overflow of the double variable storing the estimate of the
number of rows in a partial join could trigger an assertion
failure during the optimization stage.

MDEV-13918 Race condition between INFORMATION_SCHEMA.INNODB_SYS_TABLESTATS and ALTER/DROP/TRUNCATE TABLE

i_s_sys_tables_fill_table_stats(): Acquire dict_operation_lock
S-latch before acquiring dict_sys->mutex, to prevent the table
from being removed from the data dictionary cache and from
being freed while i_s_dict_fill_sys_tablestats() is accessing
the table handle.

The ownership of the field query->intersection usually transfers
to query->doc_ids. In some error scenario, it could be possible
that fts_query_free() would be invoked with query->intersection!=NULL.
Let us handle that case, instead of intentionally crashing the server.

In MariaDB, InnoDB fulltext search does not currently
truncate results based on LIMIT. In MySQL 5.7, it does.

MDEV-14086 Setting innodb_buffer_pool_load_now or innodb_buffer_load_abort will crash if innodb_read_only

When MySQL 5.6.10 introduced innodb_read_only mode, it skipped the
creation of the InnoDB buffer pool dump/restore subsystem in that mode.
Attempts to set the variable innodb_buf_pool_dump_now would have
no effect in innodb_read_only mode, but the corresponding condition
was forgotten in from the other two update functions.

MySQL 5.7.20 would fix the innodb_buffer_pool_load_now,
but not innodb_buffer_pool_load_abort. Let us fix both in MariaDB.

based on:

commit f7316aa0
Author: Ajo Robert <ajo.robert@oracle.com>
Date:   Thu Aug 24 17:03:21 2017 +0530

    Bug#26361149  MYSQL SERVER CRASHES AT: COL IN(IFNULL(CONST,
                           COL), NAME_CONST('NAME', NULL))

    Backport of Bug#19143243 fix.

    NAME_CONST item can return NULL_ITEM type in case of incorrect arguments.
    NULL_ITEM has special processing in Item_func_in function.
    In Item_func_in::fix_length_and_dec an array of possible comparators is
    created. Since NAME_CONST function has NULL_ITEM type, corresponding
    array element is empty. Then NAME_CONST is wrapped to ITEM_CACHE.
    ITEM_CACHE can not return proper type(NULL_ITEM) in Item_func_in::val_int(),
    so the NULL_ITEM is attempted compared with an empty comparator.
    The fix is to disable the caching of Item_name_const item.

Partition wasn't setting HA_OPTION_PACK_RECORD on ALTER TABLE
if the row format was PAGE.

(so one bit in the null bitmap was reserved for a deleted bit -
see make_empty_rec - and all actual null bits were one off)

if it's a DROP TABLE, we cannot detect whether a table is
temporary by looking in thd->temporary_tables - because the
table might simply not exist at all.

backport ce6c0e58

MDEV-8960: Can't refer the same column twice in one ALTER TABLE

Problem was that if column was created in alter table when
it was refered again it was not tried to find from list
of current columns.

mysql_prepare_alter_table:
  There is two cases
    (1) If alter table adds a new column and then later alter
        changes the field definition, there was no check from
	list of new columns, instead an incorrect error was given.
    (2) If alter table adds a new column and then later alter
        changes the default, there was no check from list of
	new columns, instead an incorrect error was given.

always search in compile-time specified paths
INSTALL_BINDIR, INSTALL_SBINDIR, INSTALL_MYSQLSHAREDIR. User
can set them to arbitrary values, it's not enough to search only
in their usual values of bin, sbin and libexec, share and share/mysql.

1. detect resolveip location, don' assume it's in $basedir/bin
2. don't guess $scriptdir to (incorrectly) construct the $0 path
3. rename find_in_basedir -> find_in_dirs, don't prepend $basedir
   automatically. This allows to use identical path lists in
   find_in_dirs and in cannot_find_file.
4. move search path lists to CMakeLists.txt to avoid specifying the
   same path list twice (in find_in_dirs and in cannot_find_file).

mostly caused by -Wimplicit-fallthrough

fts_get_next_doc_id(): Assign the first and subsequent FTS_DOC_ID
in the same way: by post-incrementing the cached value.
If there is a user-specified FTS_DOC_ID, do not touch the internal
sequence.

There are two bugs related to failed ADD INDEX and
the InnoDB table cache eviction.

dict_table_close(): Try dropping failed ADD INDEX when releasing
the last table handle, not when releasing the last-but-one.

dict_table_remove_from_cache_low(): Do not invoke
row_merge_drop_indexes() after freeing all index metadata.
Instead, directly invoke row_merge_drop_indexes_dict() to
remove the metadata from the persistent data dictionary
and to free the index pages.

MDEV-9619: Assertion `null_ref_table' failed in virtual table_map Item_direct_view_ref::used_tables() const on 2nd execution of PS

Refer left expression indirectly in case it changes from execution to execution.

For each SELECT the list sj_nests is built by the
function simplify_joins() when scanning different
join nests. This function may be called several
times for the same join nest. That's why before
adding a new member to sj_nests it is necessary
to check if it's already in the list.
The code of simplify_joins() lacked this check and
as a result it could cause memory overwright for
some queries.

             in buildbot

Removed plugin_array_version: it is being checked without mutex protection and
thus is prone to data race and race conditions. In effect plugins are not
protected from concurrent destruction.

Removed state_mask inversion: doesn't seem to make any sense.

When collecting local plugins list, only add plugins that match state_mask.

Use plugin ref counting to protect against concurrent plugin destruction.

MDEV-9208: Function->Function->View = Mysqld segfault (Server crashes in Dependency_marker::visit_field on 2nd execution with merged subquery)

Prevent crossing name resolution border in finding item tables.

MDEV-13676: Field "create Procedure" is NULL, even if the the user has role which is the definer. (SHOW CREATE PROCEDURE)

During show create procedure we ommited to check the current role, if it
is the actual definer of the procedure. In addition, we should support
indirectly granted roles to the current role. Implemented a recursive
lookup to search the tree of grants if the rolename is present.

SQL Standard 2016, Part 5 Section 53 View I_S.ROUTINES selects
ROUTINE_BODY and its WHERE clause says that the GRANTEE must be
either PUBLIC, or CURRENT_USER or in the ENABLED_ROLES.

Reverted incorrect changes done on MDEV-7367 and MDEV-9469. Fixes properly
also related bugs:

MDEV-13668: InnoDB unnecessarily rebuilds table when renaming a column and adding index
MDEV-9469: 'Incorrect key file' on ALTER TABLE
MDEV-9548: Alter table (renaming and adding index) fails with "Incorrect key file for table"
MDEV-10535: ALTER TABLE causes standalone/wsrep cluster crash
MDEV-13640: ALTER TABLE CHANGE and ADD INDEX on auto_increment column fails with "Incorrect key file for table..."

Root cause for all these bugs is the fact that MariaDB .frm file
can contain virtual columns but InnoDB dictionary does not and
previous fixes were incorrect or unnecessarily forced table
rebuilt. In index creation key_part->fieldnr can be bigger than
number of columns in InnoDB data dictionary. We need to skip not
stored fields when calculating correct column number for InnoDB
data dictionary.

dict_table_get_col_name_for_mysql
        Remove

innobase_match_index_columns
        Revert incorrect change done on MDEV-7367

innobase_need_rebuild
        Remove unnecessary rebuild force when column is renamed.

innobase_create_index_field_def
        Calculate InnoDB column number correctly and remove
        unnecessary column name set.

innobase_create_index_def, innobase_create_key_defs
        Remove unneeded fields parameter. Revert unneeded memset.

prepare_inplace_alter_table_dict
        Remove unneeded col_names parameter

index_field_t
        Remove unneeded col_name member.

row_merge_create_index
        Remove unneeded col_names parameter and resolution.

Effected tests:
         innodb-alter-table : Add test case for MDEV-13668
         innodb-alter : Remove MDEV-13668, MDEV-9469 FIXMEs
                        and restore original tests
         innodb-wl5980-alter : Remove MDEV-13668,  MDEV-9469 FIXMEs
                        and restore original tests

fixes failures of func_regexp_pcre

Caused by 2fcd8c12. It used the documented pcre API

 -pcre_exec(NULL, NULL, NULL, -999, -999, 0, NULL, 0)

to calculate the pcre stack frame size. Unfortunately, modern compilers
broke it by cloning and inlining pcre match() function. 2fcd8c12
tried to workaround it by setting the stack frame size to at least 500.
It didn't work, 500 is not a universal constant.

Now we fix our copy of pcre to not inline or clone match() - so that
stack frame detection would work again - and detect at cmake time
whether system pcre is broken or usable.

Also use stack, not (much slower) malloc in bundled pcre, unless on Windows

fts_create_doc_id(): Remove.

row_mysql_convert_row_to_innobase(): Implement the logic of
fts_create_doc_id(). Reuse a buffer for the hidden FTS_DOC_ID.

row_get_prebuilt_insert_row(): Allocate a buffer for the hidden
FTS_DOC_ID at the end of prebuilt->ins_upd_rec_buff.

Post-fix: RedHat does not use rh-skipped-tests.list anymore

A note about unstable-tests lists in mysql-test/README, RedHat
version of it, and also various changes to bring the file
up-to-date