MDEV-23492 performance_schema_digests_size changing from default to 5000 when enabling performance_schema

max allowed value limit should be larger than any auto-sized value

define symbols as C/C does to avoid "macro redefined" warnings

This patch removes unnecessary #ifdefs in cmake macros CHECK_C_SOURCE_COMPILES.

This patch fixes incorrect argument type passed
to the last parameter of getgrouplist() in cmake
macros CHECK_C_SOURCE_COMPILES()

Some GSS-API functions like gss_import_name(), gss_release_buffer()
used in plugin/auth_gssapi and libmariadb/plugins/auth are marked
as deprecated in MacOS starting from  version 10.14+. It results in
extra warnings output on server building.

To eliminate extra warnings the flag '-Wno-deprecated-declarations'
has been added to compiler invocation string for those source
files that invoke deprecated GSS-API functions.

This patch moves definitions of macros variables
   HAVE_PAM_SYSLOG, HAVE_PAM_EXT_H, HAVE_PAM_APPL_H, HAVE_STRNDUP
from command line (in the form -Dmacros) to the auto-generated
header file config_auth_pam.h

MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS

Compiler warnings like one listed below are generated during server build on MacOS:

[88%] Building C object plugin/auth_pam/CMakeFiles/pam_user_map.dir/mapper/pam_user_map.c.o
mariadb/server-10.2/plugin/auth_pam/mapper/pam_user_map.c:87:41: error: passing
'gid_t *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types
with different sign [-Werror,-Wpointer-sign]
  if (getgrouplist(user, user_group_id, loc_groups, &ng) < 0)
     ^~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/unistd.h:650:43: note:
  passing argument to parameter here
  int getgrouplist(const char *, int, int *, int *);
      ^

In case MariaDB server is build with -DCMAKE_BUILD_TYPE=Debug it results in
build error.

The reason of compiler warnings is that declaration of the Posix C API function
getgrouplist() on MacOS differs from declaration of getgrouplist() proposed
by Posix.

To suppress this compiler warning cmake configure was adapted to detect what
kind of getgrouplist() function is declared on the build platform and
set the macros HAVE_POSIX_GETGROUPLIST in case the building platform supports
Posix compatible interface for the getgrouplist() function. Depending on
whether this macros is set the compatible type of arguments is used to pass
parameter values to the function.

MDEV-20755 InnoDB: Database page corruption on disk or a failed file read of tablespace upon prepare of mariabackup incremental backup

The problem:

When incremental backup is taken, delta files are created for innodb tables
which are marked as new tables during innodb ddl tracking. When such
tablespace is tried to be opened during prepare in
xb_delta_open_matching_space(), it is "created", i.e.
xb_space_create_file() is invoked, instead of opening, even if
a tablespace with the same name exists in the base backup directory.

xb_space_create_file() writes page 0 header the tablespace.
This header does not contain crypt data, as mariabackup does not have
any information about crypt data in delta file metadata for
tablespaces.

After delta file is applied, recovery process is started. As the
sequence of recovery for different pages is not defined, there can be
the situation when crypt data redo log event is executed after some
other page is read for recovery. When some page is read for recovery, it's
decrypted using crypt data stored in tablespace header in page 0, if
there is no crypt data, the page is not decryped and does not pass corruption
test.

This causes error for incremental backup --prepare for encrypted
tablespaces.

The error is not stable because crypt data redo log event updates crypt
data on page 0, and recovery for different pages can be executed in
undefined order.

The fix:

When delta file is created, the corresponding write filter copies only
the pages which LSN is greater then some incremental LSN. When new file
is created during incremental backup, the LSN of all it's pages must be
greater then incremental LSN, so there is no need to create delta for
such table, we can just copy it completely.

The fix is to copy the whole file which was tracked during incremental backup
with innodb ddl tracker, and copy it to base directory during --prepare
instead of delta applying.

There is also DBUG_EXECUTE_IF() in innodb code to avoid writing redo log
record for crypt data updating on page 0 to make the test case stable.

Note:

The issue is not reproducible in 10.5 as optimized DDL's are deprecated
in 10.5. But the fix is still useful because it allows to decrease
data copy size during backup, as delta file contains some extra info.
The test case should be removed for 10.5 as it will always pass.

PROXY_USER event added.

When first argument to the JSON_MERGE_PATCH was NULL and second - the
invalid JSON line, the error code was garbage. So it should be set to 0
initially.

Compiler warnings like one listed below are generated during server build on MacOS:
In file included from server-10.2-MDEV-23564/mysys_ssl/openssl.c:33:
In file included from /usr/local/include/openssl/evp.h:16:
In file included from /usr/local/include/openssl/bio.h:20:
/usr/local/include/openssl/crypto.h:206:10: warning: 'CRYPTO_cleanup_all_ex_data' macro redefined [-Wmacro-redefined]
           ^
  /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:46:9: note: previous definition is here
          ^

In case MariaDB serer is build with -DCMAKE_BUILD_TYPE=Debug it results in
build error.

The reason of compiler warnings is that header file <ssl_compat.h>
included before the openssl system header files. File ssl_compat.h
contains some macros with the same names as SSL API functions declared
in the openssl system header files. It resulted in duplicate
symbols that produces compiler warnings.

To fix the issue the header file ssl_compat.h should be included
after a line where openssl system header is included.

It was only from CMake-3.14.0 that CMAKE_REQUIRED_LINK_OPTIONS
was used in CHECK_CXX_SOURCE_COMPILES. Without this, it could be
the case (as was on OSX) that a flag was never checked in
CHECK_CXX_SOURCE_COMPILES, the CHECK successfully passed, but
failed at link time.

As such we use CMAKE_REQUIRED_LIBRARIES to include the flags to check
as its compatible enough with the cmake versions for non-Windows
compilers/linkers.

Tested on x86_64 with:
* 3.11.4
* 3.17.4

Corrects: 7473e184

In the future:
* cmake >=3.14.0 can use CMAKE_REQUIRED_LINK_OPTIONS
* cmake >=3.18.0 can use CHECK_LINKER_FLAG (with policy CMP0057 NEW)
(e.g: commit c7ac2deff9a2c965887dcc67cbf2a3a7c3e0123d)

CMAKE_REQUIRED_LIBRARIES suggested by serg@mariadb.com

Reviewed-by: anel@mariadb.org

The only applicable InnoDB change to MariaDB that was made
between MySQL 5.6.49 and MySQL 5.6.50 is MDEV-23999.

fts_query_t::nested_sub_exp: Keep track of nested
fts_ast_visit_sub_exp() calls.

fts_ast_visit_sub_exp(): Return DB_OUT_OF_MEMORY if the
maximum recursion depth is exceeded.

This is motivated by a change in MySQL 5.6.50:
mysql/mysql-server@e2a46b4834a0030e5380d45789772cb6beab6e92
Bug #29929684 USING MANY NESTED ARGUMENTS WITH BOOLEAN FTS CAN LEAD
TO TERMINATE SERVER

In row_undo_ins_remove_clust_rec() and similar places,
an assertion !node->trx->dict_operation_lock_mode could fail,
because an online ALTER is not allowed to run at the same time
while DDL operations on the table are being rolled back.

This race condition would be fixed by always acquiring an InnoDB
table lock in ha_innobase::prepare_inplace_alter_table() or
prepare_inplace_alter_table_dict(), or by ensuring that recovered
transactions are protected by MDL that would block concurrent DDL
until the rollback has been completed.

This reverts commit 15093639
and commit 22c4a751.

The function innodb_show_mutex_status() is the only ultimate caller of
LatchCounter::iterate() via MutexMonitor::iterate(). Because the call
is not protected by LatchCounter::m_mutex, any mutex_create() or
mutex_free() that is invoked concurrently during the execution, bad
things such as a crash could happen.

The most likely way for this to happen is buffer pool resizing,
which could cause buf_block_t::mutex (which existed before MDEV-15053)
to be created or freed. We could also register InnoDB mutexes in
TrxFactory::init() if trx_pools needs to grow.

The view INFORMATION_SCHEMA.INNODB_MUTEXES is not affected, because it
only displays information about rw-locks, not mutexes.

This commit intentionally touches also MutexMonitor::iterate()
and the only code that interfaces with LatchCounter::iterate()
to make it clearer for future readers that the scattered code
that is obfuscated by templates belongs together.

This is based on
mysql/mysql-server@273a93396f49c7e0a8b07b260128d9a990c2b154

- Based on patch: d6a983351c5a454bd0cb113852f
- Update combination example for 10.2 (commit 2a3fe45d added change
for 10.3+)
```
==============================================================================

TEST                                      RESULT   TIME (ms) or COMMENT
--------------------------------------------------------------------------

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
rpl.rpl_invoked_features 'innodb,mix'    [ pass ]   1677
rpl.rpl_invoked_features 'innodb,row'    [ pass ]   3516
rpl.rpl_invoked_features 'innodb,stmt'   [ pass ]   1609
--------------------------------------------------------------------------
```
- `gdb` option will be added during the merge

- Updated combination example to use `innodb_plugin`
```
$ ./mysql-test/mtr rpl.rpl_invoked_features # no rpl.rpl_invoked_features,mix,xtradb_plugin

worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019
rpl.rpl_invoked_features 'innodb_plugin,mix' [ pass ]    904
rpl.rpl_invoked_features 'mix,xtradb'    [ pass ]   1707
rpl.rpl_invoked_features 'innodb_plugin,row' [ pass ]    927
rpl.rpl_invoked_features 'row,xtradb'    [ pass ]    828
rpl.rpl_invoked_features 'innodb_plugin,stmt' [ pass ]    855
rpl.rpl_invoked_features 'stmt,xtradb'   [ pass ]    952
--------------------------------------------------------------------------
```
- `gdb` option can take optional gdb arguments (good documentation in
KB)

To fix this, it is necessary to add an option to exclude the
database with the name "lost+found" from processing (the database
name will be checked by the check_if_skip_database_by_path() or
by the check_if_skip_database() function, and as a result
"lost+found" will be skipped).

In addition, it is necessary to slightly modify the verification
logic in the check_if_skip_database() function.

Also added a new test galera_sst_mariabackup_lost_found.test

fix an error with locked taböes

is_bulk_op())' fails on UPDATE on a partitioned table with subquery
(MySQL:71630)

Analysis and fix: Error is not checked. So correct error state is not returned.
Fix: Check for error and return the error state.

Diagnostics_area::set_error_status

Analysis: When strict mode is enabled, all warnings are converted to errors
including those which do not occur because of bad data.
Fix: Query should not be aborted when we have warning because limit to
examine rows was reached because it doesn't happen due to bad data.
So thd->abort_on_warning should be false.

MDEV-13318 introduced a condition to Mariabackup that can cause it to
hang if the server goes idle after writing a log block that has no
payload after the 12-byte header. Normal recovery in log0recv.cc would
allow blocks with exactly 12 bytes of length, and only reject blocks
where the length field is shorter than that.

This issue is caused by the commit af40a2b4.
In btr_search_update_hash_on_insert(), btr_search_sys->hash_tables
is being accessed without taking proper ahi latch. During buffer pool
resizing, btr_get_search_table() is being accessed and it leads to
segmentation fault.

Reviewed-by: Marko Mäkelä

Problem:
=======
 InnoDB allows virtual index to be referenced index in foreign key
relations. While dropping the virtual column, Inplace alter does
allow the table to be closed and open it using table name to
update dict_table_t::v_cols. While loading the table, it doesn't
allow any error to be ignored. InnoDB can't find the referenced
virtual index and fails to load the table during Inplace alter.

Solution:
=========
  During inplace alter, ignore the foreign key error while loading
the table.

Reviewed-by: Marko Mäkelä

Made cleanup of DROP (udf) FUNCTION procedure and also check of mysql.func (not only loaded udf).

MDEV-16676 Using malloc-lib=jemalloc in MariaDB 10.2 causes non-critical error about missing mysql_config on startup

- This commit is based on patch 84fe9720a4d2483ff67b6a and suggestion of
Jean Weisbuch to use `ldconfig -p`
- Format of ldconfig output:
"libjemalloc.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libjemalloc.so.1"

```
$ ./scripts/mysqld_safe
201013 13:36:50 mysqld_safe Adding '/usr/lib/x86_64-linux-gnu/libjemalloc.so.1' to LD_PRELOAD for mysqld
201013 13:36:50 mysqld_safe Logging to '/home/anel/builds/data-10.5/mysqld_safe_anel.err'.
201013 13:36:50 mysqld_safe Starting mysqld daemon with databases from /home/anel/builds/data-10.5
```
Reviewed by: Daniel Black, Faustin Lammler

Instead of pointlessly waiting for a page flush to occur, take
the matter into our own hands and request an explicit flush.
Also, test with the minimum necessary amount of data (0 or 1 rows)
so that both page encryption and decryption will be exercised.

- row_search_mvcc() should return DB_INTERRUPTED when it got killed.
- Add a syncpoint for the ICP check.
- Add test coverage for killed-during-ICP-check scenario

Backport of MDEV-22761 fixes for ICP from 10.4 commits:
* a6f95648
* c03885cd

XtraDB was fixed in deb3b9a1

Reviewer: Daniel Black

This fix removes from the list of disabled tests all
tests that were fixed by the patch for MDEV-21770.

This patch fixes several flaws in the SST scripts that cause
failures while running tests that use version 6 IP addresses
for cluster nodes.

First, if the netcat utility is used for streaming (but not socat),
then in accordance with its command line syntax, we need to remove
the square brackets around the IPv6 address. However, for socat,
the address must contain square brackets, as before.

Secondly, if an IPv6 address is used, then from the joiner side for
a number of systems (such as Debian) we need to explicitly specify
the "-6" option, otherwise a listening socket with an IPv6 address
may not be created.

This patch also contains code improvements in the wsrep_sst_common.
Changed the code that pars the connection address - fixed the
shortcomings that sometimes led to incorrect parsing of parameters
when using shells other than the latest versions of bash.

Also, this patch removes the duplicate code that is intended
for parsing the connection address and which was located in the
wsrep_sst_mariabackup file, since all the necessary actions have
already been done in wsrep_sst_common and there they are done in
such a way that any shell is supported, not just bash.

The fix does not require separate tests, since all the
necessary tests are already present in the galera_3nodes suite.
On the contrary, after this fix, tests using IPv6 addresses can
be removed from the disabled list (this will be done in a separate
commit related to MDEV-23659).

As MariaDB tables are expected in a fixed order, the cross upgrade
previously placed roles, default_role and max_statement_time after
now unused mysql-5.7 columns.

Test:

$ cp -a /usr/local/mysql-5.7.31/data/ /tmp/m57data
$ sql/mysqld --no-defaults --skip-networking --datadir=/tmp/m57data  --socket=/tmp/${PWD##*/}.sock   --verbose --lc-messages-dir=$PWD/sql/share
2020-10-02 11:02:05 140135193212864 [Note] sql/mysqld (mysqld 10.2.34-MariaDB) starting as process 1457667 ...
2020-10-02 11:02:05 140135193212864 [Note] InnoDB: Mutexes and rw_locks use GCC atomic built

$ client/mysql_upgrade --no-defaults -u root -pbob  -S /tmp/build-mariadb-server-10.2.sock
MySQL upgrade detected
Phase 1/7: Checking and upgrading mysql database
Processing databases
mysql
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.engine_cost                                  OK
mysql.event                                        OK
mysql.func                                         OK
mysql.gtid_executed                                OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.innodb_index_stats                           OK
mysql.innodb_table_stats                           OK
mysql.ndb_binlog_index                             OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.server_cost                                  OK
mysql.servers                                      OK
mysql.slave_master_info                            OK
mysql.slave_relay_log_info                         OK
mysql.slave_worker_info                            OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
Upgrading from a version before MariaDB-10.1
Phase 2/7: Installing used storage engines

Result:

| user  | CREATE TABLE `user` (
  `Host` char(60) COLLATE utf8_bin NOT NULL DEFAULT '',
  `User` char(80) COLLATE utf8_bin NOT NULL DEFAULT '',
  `Password` char(41) CHARACTER SET latin1 COLLATE latin1_bin NOT NULL DEFAULT '',
  `Select_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Insert_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Update_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Delete_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Create_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Drop_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Reload_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Shutdown_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Process_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `File_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Grant_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `References_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Index_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Alter_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Show_db_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Super_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Create_tmp_table_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Lock_tables_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Execute_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Repl_slave_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Repl_client_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Create_view_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Show_view_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Create_routine_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Alter_routine_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Create_user_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Event_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Trigger_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `Create_tablespace_priv` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `ssl_type` enum('','ANY','X509','SPECIFIED') CHARACTER SET utf8 NOT NULL DEFAULT '',
  `ssl_cipher` blob NOT NULL,
  `x509_issuer` blob NOT NULL,
  `x509_subject` blob NOT NULL,
  `max_questions` int(11) unsigned NOT NULL DEFAULT 0,
  `max_updates` int(11) unsigned NOT NULL DEFAULT 0,
  `max_connections` int(11) unsigned NOT NULL DEFAULT 0,
  `max_user_connections` int(11) NOT NULL DEFAULT 0,
  `plugin` char(64) CHARACTER SET latin1 NOT NULL DEFAULT '',
  `authentication_string` text COLLATE utf8_bin NOT NULL,
  `password_expired` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `is_role` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N',
  `default_role` char(80) COLLATE utf8_bin NOT NULL DEFAULT '',
  `max_statement_time` decimal(12,6) NOT NULL DEFAULT 0.000000,
  `password_last_changed` timestamp NULL DEFAULT NULL,
  `password_lifetime` smallint(5) unsigned DEFAULT NULL,
  `account_locked` enum('N','Y') COLLATE utf8_bin NOT NULL DEFAULT 'N',
  PRIMARY KEY (`Host`,`User`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='Users and global privileges' |

MariaDB [(none)]>  CREATE ROLE `aRole`;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> SET ROLE `aRole`;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> SET ROLE `aRole`;
Query OK, 0 rows affected (0.00 sec)

MariaDB [(none)]> SELECT `User`, `is_role` FROM `mysql`.`user`;
+---------------+---------+
| User          | is_role |
+---------------+---------+
| root          | N       |
| mysql.session | N       |
| mysql.sys     | N       |
| dan           | N       |
| aRole         | Y       |
+---------------+---------+
5 rows in set (0.00 sec)

Reviewer: Anel Husakovic

MDEV-23960 UBSAN ../storage/innobase/buf/buf0buddy.cc:350:6: runtime error: index 4096 out of bounds for type 'byte [38]'

Reviewed by: Marko Mäkelä

Copying generated data from the sequence engine should be faster
than copying from MyISAM to MyISAM.

Reviewed by: Sergei Petrunia

minor fixes of rpl_start_stop_slave and rpl_slave_grp_exec tests, expanding rpl_gtid_delete_domain for easier later analysis

Sometimes blockdev --getss returns 4096.
In that case ROW_FORMAT=COMPRESSED tables might violate
that 4096 bytes alignment.

This patch disables O_DIRECT for COMPRESSED tables.

OS_DATA_FILE_NO_O_DIRECT: new possible value for os_file_create() argument

fil_node_open_file(): do not O_DIRECT
ROW_FORMAT=COMPRESSED tables

AIO::is_linux_native_aio_supported(): minimal alignment in a general case
is 4096 and not 512.