MDEV-31743 Server crash in store_length, assertion failure in Type_handler_string_result::sort_length

After MDEV-21580 the truncation of SORT_FIELD::length
  set_if_smaller(sortorder->length, thd->variables.max_sort_length)

became conditional:
  if (is_variable_sized())
    set_if_smaller(length, thd->variables.max_sort_length)

To provide correct functioning of is_variable_sized() SORT_FIELD::type
must be set properly. This commit adds the necessary initialization
of SORT_FIELD::type to JOIN_TAB::remove_duplicates() as it is done
in filesort's sortlength() function.

DBUG_ASSERT is added to sortlength() just in case to prevent
a possible uint32 overflow

recv_scan_log(): On recv_sys_t::PREMATURE_EOF, keep reading more log
if recv_sys.lsn < recv_sys.scanned_lsn.

recv_recovery_from_checkpoint_start(): Add a safety check to abort
crash recovery if recv_sys.lsn is not recv_sys.scanned_lsn.

This fixes a serious database corruption bug that was introduced by
commit 2f9e2647 (MDEV-29911).

MDEV-31822 ALTER TABLE ENGINE=x started failing instead of producing warning on unsupported TRANSACTIONAL=1

make TRANSACTIONAL table option behave similar to other engine-defined
table options. If the engine doesn't suport it:
* if specified expicitly in CREATE or ALTER - it's ER_UNKNOWN_OPTION
* an error or a warning depending on sql_mode IGNORE_BAD_TABLE_OPTIONS
* in ALTER TABLE from the engine that suppors it to the engine that
  doesn't - silently preserved (no warning)
* it is commented out in SHOW CREATE unless IGNORE_BAD_TABLE_OPTIONS

* invoke check_expression() for all vcol_info's in
  mysql_prepare_create_table() to check for FK CASCADE
* also check for SET NULL and SET DEFAULT
* to check against existing FKs when a vcol is added in ALTER TABLE,
  old FKs must be added to the new_key_list just like other indexes are
* check columns recursively, if vcol1 references vcol2,
  flags of vcol2 must be taken into account
* remove check_table_name_processor(), put that logic under
  check_vcol_func_processor() to avoid walking the tree twice

recv_ring::copy_if_needed(): If the record wraps around the
memory-mapped ib_logfile0, do copy it also if len==0
(the record consists only of a header, like FREE_PAGE and INIT_PAGE
records do).

recv_sys_t::parse(): Invoke recv_ring::copy_if_needed() for INIT_PAGE
and FREE_PAGE records, so that if these records wrap around the
memory-mapped ib_logfile0, they will be correctly copied to
recv_sys.pages.

Together with commit 0d175968 (MDEV-31354)
this fixes occasional failures of the test innodb.recovery_memory.

mark old keys in the ALTER TABLE with the `old` flag, not with
the `key_create_info.check_for_duplicate_indexes`.

This allows to mark old foreign keys too.

to declutter Item_field::check_vcol_func_processor(), move alter_info
specific part of it into Alter_info::check_vcol_field()

also, prefer Lex_table_name and Lex_ident over LEX_CSTRING

HA_UNIQUE_CHECK was
* only used internally by MyISAM/Aria
* only used for internal temporary tables (for DISTINCT)
* never saved in frm
* saved in MYI/MAD but only for temporary tables
* only set, never checked

it's safe to remove it and free the bit (there are only 16 of them)

Do not attempt to produce "r_engine_stats" on the temporary (=work) tables.
These tables may be
- re-created during the query execution
- freed during the query execution (This is done e.g. in JOIN::cleanup(),
  before we produce ANALYZE FORMAT=JSON output).

- (Also, make save_explain_data() functions not set handler_for_stats
  to point to handler objects that do not have handler->handler_stats set.
  If the storage engine is not collecting handler_stats, it will not have
  them when we're producing ANALYZE FORMAT=JSON output, either).

buf_page_t::write_complete(), buf_page_write_complete(),
IORequest::write_complete(): Add a parameter for passing
an error code. If an error occurred, we will release the
io-fix, buffer-fix and page latch but not reset the
oldest_modification field. The block would remain in
buf_pool.LRU and possibly buf_pool.flush_list, to be written
again later, by buf_flush_page_cleaner(). If all page writes
start consistently failing, all write threads should eventually
hang in log_free_check() because the log checkpoint cannot
be advanced to make room in the circular write-ahead-log ib_logfile0.

IORequest::read_complete(): Add a parameter for passing
an error code. If a read operation fails, we report the error
and discard the page, just like we would do if the page checksum
was not validated or the page could not be decrypted.
This only affects asynchronous reads, due to linear or random read-ahead
or crash recovery. When buf_page_get_low() invokes buf_read_page(),
that will be a synchronous read, not involving this code.

This was tested by randomly injecting errors in
write_io_callback() and read_io_callback(), like this:

  if (!ut_rnd_interval(100))
    cb->m_err= 42;

buf_LRU_free_page(): The block may also be in the IBUF_EXIST state
when executing the test innodb.innodb_bulk_create_index_debug.

buf_LRU_free_page(): When we are discarding the uncompressed copy of a
ROW_FORMAT=COMPRESSED page, buf_page_t::can_relocate() must have ensured
that the block descriptor state is one of FREED, UNFIXED, REINIT.
Do not overwrite the state with UNFIXED. We do not want to write back
pages that were actually freed, and we want to avoid doublewrite for
pages that were (re)initialized by log records written since the latest
checkpoint. Last but not least, we do not want crashes like those that
commit dc1bd180 (MDEV-31386)
was supposed to fix.

The test innodb_zip.wl5522_zip should typically cover all 3 states.

This bug is a regression due to
commit aaef2e1d (MDEV-27058).

failed SHOW CREATE FUNCTION means we don't dump this function,
but should still try to dump all other functions

don't construct open ranges from prefix blob keys for < (less than)
just as it's already done for > (greater than)

because prefix KEY_PART doesn't create prefix Field for blobs
(see open_table_from_share() near "Create a new field for the key part"),
so stored_field_cmp_to_item() will compare the original field to the
value not taking the prefix length into account.

This reverts commit add0c01b

Duplicates must be avoided in FTS_DOC_ID_INDEX

Problem:
========
 - InnoDB scans the complete redo log to ensure that there is
no corruption and to find the end of the log. During this scan,
InnoDB saves all the freed ranges, but it doesn't save
recovered size. Later, InnoDB recovery applies partial
redo logs and IO thread tries to flush the all freed
ranges which was noted during previous complete scan of redo logs.

Fix:
====
 InnoDB should store the freed pages only when InnoDB stores
 the redo log records.

(It's not using include/rpl_init.inc, so it needs to reset the GTID position
explicitly).
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

A simple "SET SESSION gtid_seq_no= DEFAULT" did not work, it would straight
up crash the server! Also, explicitly setting gtid_seq_no to 0 gave an error
in --gtid-strict-mode=1.

Setting to DEFAULT or 0 should disable any prior setting of
gtid_seq_no, so that the next transaction is allocated the next GTID
in sequence, as normal.
Reviewed-by: Monty <monty@mariadb.org>
Signed-off-by: Kristian Nielsen <knielsen@knielsen-hq.org>

log_sort_flush_list(): Wait for any pending page writes to cease
before sorting the buf_pool.flush_list. Starting with
commit 22b62eda (MDEV-25113),
it is possible that some buf_page_t::oldest_modification_
that we will be comparing in std::sort() will be updated from
some value >2 to 1 while we are holding buf_pool.flush_list_mutex.

To catch this type of trouble better in the future, we will clean
garbage (pages that have been written out) from buf_pool.flush_list
while constructing the array for sorting, and check with debug
assertions that all blocks that we are copying from the array to the
list will be dirty (requiring a writeback) while we sort and copy
the array back to buf_pool.flush_list.

This failure was observed by chance exactly once when running the test
innodb.recovery_memory. It was never reproduced in the same form
afterwards. Unrelated to this change, the test does occasionally
reproduce a failure to start up InnoDB due to a corrupted page
being read in recovery. The ticket MDEV-31791 was filed for that.

Tested by: Matthias Leich