Issue
=====
The original issue was that the size of a fil_per_table tablespace was calculated
incorrectly during truncate in the presence of an fts index. This incorrect calculation
was fixed as part of BUG#25053705 along with a testcase to reproduce the bug. The
assert that was added as part of it to reproduce the bug was wrong and resulted in
this bug.

Fix
===
Although the assert was removed earlier in a seperate commit as it was blocking the
ntest, this patch replaces the other parts of the code that were added to reproduce
the bug and replaces it with code that tries to reproduce the bug in a different way.

The new code basically tries to tweak conditions so as to simulate the random read
where a page that doesn't exist is tried to be read.

RB: 15890
Reviewed-by: Jimmy Yang <Jimmy.Yang@oracle.com>
Reviewed-by: Satya Bodapati <satya.bodapati@oracle.com>

MySQL 5.7 added code to push down the LIMIT to fulltext search
in InnoDB:

commit 2cd0ebf97e1b265e2282d7ceb5d8dfb663ffc48f
Author: Thirunarayanan Balathandayuthapani <thirunarayanan.balathandayuth@oracle.com>
Date:   Fri May 27 13:49:28 2016 +0530

    Bug #22709692   FTS QUERY EXCEEDS RESULT CACHE LIMIT

The code was disabled when MySQL 5.7.9 was merged to MariaDB 10.2.2.

We shall remove the disabled code and unnecessary variables.

fil_mutex_enter_and_prepare_for_io(): Reacquire fil_system->mutex after
failing to close a file and before retrying.

This was a false alarm in a debug check that was introduced in
commit 48192f96 which was a
10.2 code refactoring in preparation for
MDEV-11369 (instant ADD COLUMN) in 10.3.2. The code refactoring
only affected debug builds.

InnoDB B-tree record locks are only supposed to exist on leaf page
records. An assertion failed, because the debug function lock_validate()
was invoking lock_rec_block_validate() on a page for which there were
no locks set in the record lock bitmap. This could happen on a page split.
Especially when the index size grows from a single page to multiple pages,
the root page would transform from a leaf node into an internal node,
and its record lock bitmap would be emptied.

lock_validate(): Skip empty lock bitmaps.

There is no call to os_thread_join(), so we should detach the thread
handles in order to avoid any resource leaks.

Problem was that we could take page latches on different
order than wat is entitled with SX-lock. To follow the
latching order defined in WL#6326, acquire index->lock X-latch.
This entitles us to acquire page latches in any order for the index.

btr0btr.cc
	Document latch rules before and after MariaDB 10.2.2

sync0rw.cc
	Document latch compatibility rules better.

btr_defragment_merge_pages
	Fix parameter value.

btr_defragment_thread
	Acquire X-lock to dict_index_t::lock before restoring
	cursor position and continuing defragmentation.

ha_innobase::optimize
	Restore defragment feature.

Testing
	Add GIS-index and FT-index to table being defragmented.

	Defragmentation is not done to GIS-indexes and FT auxiliary
	tables.

A reference to a CTE may occur not in the master of the CTE
specification. In this case if the reference to the CTE is
the first one the specification should be detached from its
master and attached to the referencing select.

Also fixed the TYPE column in the lines of the EXPLAIN output
created for CTE tables.

rdb_sst_info.cc must be compiled with RTTI

Changing Field::set_default from void to int.
It now uses the same return value notation with Field::store*()
and Item::save_in_field().

in 10.1 innodb was basically ignoring virtual columns. In particular,
information about them was not stored in system tables. To make 10.1
table usable in 10.2 it needs to be rebuilt to have virtual colunm
metadata properly recreated.

See also a followup:
MDEV-14046 Allow ALGORITHM=INPLACE for 10.1 tables that contain virtual columns

fil_ibd_create(): commit b731a5bc
introduced a variable that was unused outside Windows. Use it
on all platforms.

btr_store_big_rec_extern_fields(): Remove the unused parameter 'upd'
that was added in https://github.com/mysql/mysql-server/commit/ce0a1e85e24e48b8171f767b44330d
and merged to MariaDB 10.2.2
in commit 2e814d47.

    is_bulk_op())' failed upon altering table with geometry field.

        Tests added.

                    is_bulk_op())' failed upon altering table with geometry field.

        Check for the validity of the DEFAULT value for the
        geometry field.

This fixes some truncation warnings on Win64

Reapply a MySQL 5.6.23/5.7.10 Oracle Bug#20029625 fix that was inadvertently reverted in MariaDB 10.2.2

The fix https://github.com/mysql/mysql-server/commit/716f97e2714e70f35d2dc01f0125ed833c62b408
was inadvertently reverted in
commit 2e814d47
"Merge InnoDB 5.7 from mysql-5.7.9".

Reapply the fix, because the test of the bug would fail
after merging MDEV-13838, which replaced an earlier incorrect
bug fix with a correct one.

Added cstdlib librabry to rdb_datadic.h

of WSREP_TO_ISOLATION_BEGIN.

Mariabackup 10.2.7 would delete the redo log files after a successful
--prepare operation. If the user is manually copying the prepared files
instead of using the --copy-back option, it could happen that some old
redo log file would be preserved in the restored location. These old
redo log files could cause corruption of the restored data files when
the server is started up.

We prevent this scenario by creating a "poisoned" redo log file
ib_logfile0 at the end of the --prepare step. The poisoning consists
of simply truncating the file to an empty file. InnoDB will refuse
to start up on an empty redo log file.

copy_back(): Delete all redo log files in the target if the source
file ib_logfile0 is empty. (Previously we did this if the source
file is missing.)

SRV_OPERATION_RESTORE_EXPORT: A new variant of SRV_OPERATION_RESTORE
when the --export option is specified. In this mode, we will keep
deleting all redo log files, instead of truncating the first one.

delete_log_files(): Add a parameter for the first file to delete,
to be passed as 0 or 1.

innobase_start_or_create_for_mysql(): In mariabackup --prepare,
tolerate an empty ib_logfile0 file. Otherwise, require the first
redo log file to be longer than 4 blocks (2048 bytes). Unless
--export was specified, truncate the first log file at the
end of --prepare.

Use GetFileInformationByHandleEx with FileAttributeTagInfo to query whether
the file is sparse. This saves 1 syscall, as GetFileInformationByHandle()
would additionally query volume info.

Try to fix fragmentation (unsparse files), for pre-existing
installations.

Unsparse the innodb file, when it needs to be extended, unless compression
is used. For Win7/2008R2 unsparse  does not work (as documented in MSDN),
therefore for sparse files in older Windows, file extension will be done
via writing zeroes at the end of file.

MDEV-13310 Preparing an incremental backup twice can corrupt data

The last parameter to this function is now,"bool is_sparse", like in 10.1
rather than the  unused/useless "bool is_readonly", merged from MySQL 5.7

Like in 10.1, this function now supports sparse files, and efficient
platform specific mechanisms for file extension

os_file_set_size() is now consistenly used in all places where
innodb files are extended.

galera_events test shows a regression with the original fix for MW-416
Reason was that Events::drop_event() can be called also from inside event
execution, and there we have a speacial treatment for event, which executes
"DROP EVENT" statement, and runs TOI replication inside the event processing body.
This resulted in executing WSREP_TO_ISOLATION two times for such DROP EVENT statement.
Fix is to call WSREP_TO_ISOLATION_BEGIN only in Events::drop_event()

Changed return code for replicatio error to TRUE.
This is aligned with native mysql convention to return TRUE (defined to 1) or FALSE (defined to 0) from a bool function.
This is wrong, but follows the mysql conventiosn, at least...

Moved TOI replication to happen after ACL checking for commands:
SQLCOM_CREATE_EVENT
SQLCOM_ALTER_EVENT
SQLCOM_DROP_EVENT
SQLCOM_CREATE_VIEW
SQLCOM_CREATE_TRIGGER
SQLCOM_DROP_TRIGGER
SQLCOM_INSTALL_PLUGIN
SQLCOM_UNINSTALL_PLUGIN

MariaDB 10.1 introduced non-indexed virtual columns for InnoDB tables.
When MySQL 5.7 introduced virtual columns in InnoDB tables, it also
introduced the table SYS_VIRTUAL that stores metadata on virtual
columns. This table does not initially exist in data files that were
imported from 10.1. So, we do not always have virtual column metadata
inside InnoDB.

dict_index_contains_col_or_prefix(): In the clustered index records,
all non-virtual columns are present and no virtual columns are present.

ha_innobase::build_template(): In the clustered index, do not
include virtual columns in the query template. The SQL layer is
supposed to compute the virtual column values when needed.

When btr_create() invokes btr_free_root() after running out of space,
fseg_create() would have acquired an SX-latch on the root page, not
an X-latch. Relax the debug assertion in btr_free_root() accordingly.

(In this case, SX-latch and X-latch are equivalent. During the CREATE
operation there should be MDL_EXCLUSIVE and dict_operation_lock X-latch
preventing concurrent access to the index. Normally the purpose of the
SX-latch is to allow concurrent reads of the root page while certain
fields in the root page are updated in place.)

The part of the test that counts IO should be run with default
rocksdb_flush_log_at_trx_commt.

Some innobase/xtrabackup changes around  from 10.1 are null merged
, in partucular using os_set_file_size to extend tablespaces in server
or mariabackup.
They require non-trivial amount of additional  work in 10.2, due to
innobase differences between 10.1 and 10.2

        Space for the next operation wasn't reserved in
        Item_func_spatial_relate::val_str()