Merge pull request https://github.com/MariaDB/server/pull/343

contributed by Eric Herman.

xtradb: fil_crypt_rotate_page, space_id should be compared to TRX_SYS_SPACE

like 9a218f4f fil_crypt_rotate_page
 - space_id should be compared to TRX_SYS_SPACE not space
Signed-off-by: Daniel Black <daniel.black@au.ibm.com>

fil_crypt_rotate_page - space_id should be compared to TRX_SYS_SPACE not space

Fixes compile error that highlights problem:

/source/storage/innobase/fil/fil0crypt.cc: In function 'void fil_crypt_rotate_page(const key_state_t*, rotate_thread_t*)':
/source/storage/innobase/fil/fil0crypt.cc:1770:15: error: ISO C++ forbids comparison between pointer and integer [-fpermissive]
  if (space == TRX_SYS_SPACE && offset == TRX_SYS_PAGE_NO) {
Signed-off-by: Daniel Black <daniel.black@au.ibm.com>

Also, some MDEV-11738/MDEV-11581 post-push fixes.

In MariaDB 10.1, there is no fil_space_t::is_being_truncated field,
and the predicates fil_space_t::stop_new_ops and fil_space_t::is_stopping()
are interchangeable. I requested the fil_space_t::is_stopping() to be added
in the review, but some added checks for fil_space_t::stop_new_ops were
not replaced with calls to fil_space_t::is_stopping().

buf_page_decrypt_after_read(): In this low-level I/O operation, we must
look up the tablespace if it exists, even though future I/O operations
have been blocked on it due to a pending DDL operation, such as DROP TABLE
or TRUNCATE TABLE or other table-rebuilding operations (ALTER, OPTIMIZE).
Pass a parameter to fil_space_acquire_low() telling that we are performing
a low-level I/O operation and the fil_space_t::is_stopping() status should
be ignored.

specify BUILD_BYPRODUCTS, ninja needs it

that is, download AWS SDK there, not into the builddir root.
and .gitignore them all.

into non-standard compiler/linker path (e.g vcpkg on Windows).

Also fix linking with static preinstalled aws c++ sdk libraries

* add a clarifying comment
* apply /wd4244 /wd4146 to ref10 source files, not to the library

The test was unnecessarily depending on InnoDB purge, which can
sometimes fail to proceed.

Let us rewrite the test to use BEGIN;INSERT;ROLLBACK to cause the
immediate removal of the desired records.

- silence warnings in ed25519 reference implementation
- fix signed/unsigned warning in popular header item_func.h

verify that tests don't leave mysql_upgrade_info in datadir

shouldn't have been added in the first place, MariaDB Server is GPLv2

The test is not expected to crash. With a non-debug server,
Valgrind completes in reasonable time without any failure.

Also, it does not make sense to store and restore parameters
when the parameters are already being restored by a server restart.

- Before this patch during startup all slave threads was started without
  any check that they had started properly.
- If one did a START SLAVE, STOP SLAVE or CHANGE MASTER as first command to the server
  there was a chance that server could access structures that where not
  properly  initialized which could lead to crashes in
  Log_event::read_log_event
- Fixed by waiting for slave threads to start up properly also during
  server startup, like we do with START SLAVE.

The following is an updated commit message for the following commit
that was pushed before I had a chance to update the commit message:
c5e25c8b

Fixed dead locks when doing stop slave while slave was starting.

- Added a separate lock for protecting start/stop/reset of a specific slave.
  This solves some possible dead locks when one calls stop slave while
  the slave is starting as the old run_locks was over used for other things.
- Set hash->records to 0 before calling free of all hash elements.
  This was set to stop concurrent threads to loop over hash elements and
  access members that was already freed.
  This was a problem especially in start_all_slaves/stop_all_slaves
  as the mutex protecting the hash was temporarily released while a slave
  was started/stopped.
- Because of change to hash->records during hash_reset(),
  any_slave_sql_running() will return 1 during shutdown as one can't
  loop over master_info_index->master_info_hash while hash_reset() of it
  is in progress.
  This also fixes a potential old bug in any_slave_sql_running() where
  during shutdown and ~Master_info_index(), my_hash_free() we could
  potentially try to access elements that was already freed.

restore mysql_file_delete_with_symlink() but let it use
new my_handler_delete_with_symlink() mysys helper.

Do not include base64.h, it was recently removed.

MDEV-11581: Mariadb starts InnoDB encryption threads
when key has not changed or data scrubbing turned off

Background: Key rotation is based on background threads
(innodb-encryption-threads) periodically going through
all tablespaces on fil_system. For each tablespace
current used key version is compared to max key age
(innodb-encryption-rotate-key-age). This process
naturally takes CPU. Similarly, in same time need for
scrubbing is investigated. Currently, key rotation
is fully supported on Amazon AWS key management plugin
only but InnoDB does not have knowledge what key
management plugin is used.

This patch re-purposes innodb-encryption-rotate-key-age=0
to disable key rotation and background data scrubbing.
All new tables are added to special list for key rotation
and key rotation is based on sending a event to
background encryption threads instead of using periodic
checking (i.e. timeout).

fil0fil.cc: Added functions fil_space_acquire_low()
to acquire a tablespace when it could be dropped concurrently.
This function is used from fil_space_acquire() or
fil_space_acquire_silent() that will not print
any messages if we try to acquire space that does not exist.
fil_space_release() to release a acquired tablespace.
fil_space_next() to iterate tablespaces in fil_system
using fil_space_acquire() and fil_space_release().
Similarly, fil_space_keyrotation_next() to iterate new
list fil_system->rotation_list where new tables.
are added if key rotation is disabled.
Removed unnecessary functions fil_get_first_space_safe()
fil_get_next_space_safe()

fil_node_open_file(): After page 0 is read read also
crypt_info if it is not yet read.

btr_scrub_lock_dict_func()
buf_page_check_corrupt()
buf_page_encrypt_before_write()
buf_merge_or_delete_for_page()
lock_print_info_all_transactions()
row_fts_psort_info_init()
row_truncate_table_for_mysql()
row_drop_table_for_mysql()
    Use fil_space_acquire()/release() to access fil_space_t.

buf_page_decrypt_after_read():
    Use fil_space_get_crypt_data() because at this point
    we might not yet have read page 0.

fil0crypt.cc/fil0fil.h: Lot of changes. Pass fil_space_t* directly
to functions needing it and store fil_space_t* to rotation state.
Use fil_space_acquire()/release() when iterating tablespaces
and removed unnecessary is_closing from fil_crypt_t. Use
fil_space_t::is_stopping() to detect when access to
tablespace should be stopped. Removed unnecessary
fil_space_get_crypt_data().

fil_space_create(): Inform key rotation that there could
be something to do if key rotation is disabled and new
table with encryption enabled is created.
Remove unnecessary functions fil_get_first_space_safe()
and fil_get_next_space_safe(). fil_space_acquire()
and fil_space_release() are used instead. Moved
fil_space_get_crypt_data() and fil_space_set_crypt_data()
to fil0crypt.cc.

fsp_header_init(): Acquire fil_space_t*, write crypt_data
and release space.

check_table_options()
	Renamed FIL_SPACE_ENCRYPTION_* TO FIL_ENCRYPTION_*

i_s.cc: Added ROTATING_OR_FLUSHING field to
information_schema.innodb_tablespace_encryption
to show current status of key rotation.

failed with SELECT SQ, TEXT field

The functon find_all_keys does call Item_subselect::walk, which calls walk() for the subquery
The issue is that when a field is represented by Item_outer_ref(Item_direct_ref(Item_copy_string( ...))).
Item_copy_string does have a pointer to an Item_field in Item_copy::item but does not implement Item::walk method, so we are not
able to set the bitmap for that field. This is the reason why the assert fails.

Fixed by adding the walk method to Item_copy class.

Fix symlink-aria && symlink-myisam to account for this possibility.

force SELinux policies to be built in mysql_release RPM packages