MDEV-27107 prevent two mariadb-upgrade running in parallel
MDEV-27279 mariadb_upgrade add --check-if-upgrade-is-needed /
           restrict tests to major version

Code is based of pull request from Daniel Black, but with a several
extensions.

- mysql_upgrade now locks the mysql_upgrade file with my_lock()
  (Advisory record locking). This ensures that two mysql_upgrades
   cannot be run in parallel.
- Added --check-if-upgrade-is-needed to mysql_upgrade. This will return
  0 if one has to run mysql_upgrade.

Other changes:
- mysql_upgrade will now immediately exit if the major version and minor
  version  (two first numbers in the version string) is same as last run.
  Before this change mysql_upgrade was run if the version string was different
  from last run.
- Better messages when there is no need to run mysql_upgrade.
- mysql_upgrade --verbose now prints out a lot more information about
  the version checking.
- mysql_upgrade --debug now uses default debug arguments if there is no
  option to --debug
- "MySQL" is renamed to MariaDB in the messages
- mysql_upgrade version increased to 2.0

Notes
Verifying "prevent two mariadb-upgrade running in parallel" was
done in a debugger as it would be a bit complex to do that in mtr.

Reviewer: Danial Black <daniel@mariadb.org>

 - InnoDB change buffer doesn't support spatial index. Spatial
index should avoid change the buffer bitmap page when the page
split happens.

Mixed declarations and code is not allowed for C90 so
fix it to avoid compilation break on some platforms.

Database names are 64 utf8 characters per the system tables
that refer to them.

The current database() function is returning 34 characters.

The result of limiting this function results to max length of 34
became apparent when used in a UNION ALL where the results are
truncated to 34 characters.

For (uninvestigated) reasons, SELECT DATABASE() on its own
would always return the right number of characters.

Thanks Alexander Barkov for the review.

Thanks dave for noticing the bug in the stackexchange post
https://dba.stackexchange.com/questions/306183/why-is-my-database-name-truncated

Running a query using cursor could lead to a server crash on
building a temporary table used for handling the query.

For example, the following cursor

DECLARE cur1 CURSOR FOR
  SELECT t2.c1 AS c1 FROM t1 LEFT JOIN t2 ON t1.c1 = t2.c1
  WHERE EXISTS (SELECT 1 FROM t1 WHERE c2 = -1) ORDER BY c1;

declared and executed inside a stored routine could result in server
crash on creating a temporary table used for handling the ORDER BY clause.

Crash occurred on attempt to create the temporary table's fields based
on fields whose data located in a memory root that already freed.

It happens inside the function return_zero_rows() where the method
Select_materialize::send_result_set_metadata() is invoked for cursor case.
This method calls the st_select_lex_unit::get_column_types() in order to
get a list of items with types of columns for the temporary table being created.
The method st_select_lex_unit::get_column_types() returns
  first_select()->join->fields
in case it is invoked for a cursor. Unfortunately, this memory has been already
deallocated bit earlier by calling
  join->join_free();
inside the function return_zero_rows().

In case the query listed in the example is run in conventional way (without
using cursor) the method st_select_lex_unit::get_column_types()
returns first_select()->item_list that is not touched by invocation
of the method join->join_free() so everything is fine for that.

So, to fix the issue the resources allocated for the JOIN class should be
released after any activities with the JOIN class has been completed,
that is as the last statement before returning from the function
return_zero_rows().

This patch includes tests both for the case when a cursor is run explicitly
from within a stored routine and for the case when a cursor is opened
implicitly as prescribed by the STMT_ATTR_CURSOR_TYPE attribute of
binary protocol (the case of prepared statement).

- Cherry-pick 2fcff310 (MDEV-21902)
- Closed PR #1145
Reviewed by: holyfoot@mariadb.com

.. to be the same as startup.

In resolving MDEV-27461, BUF_LRU_MIN_LEN (256) is the minimum number of
pages for the innodb buffer pool size. Obviously we need more than just
flushing pages. Taking the 16k page size and its default minimum, an
extra 25% is needed on top of the flushing pages to make a workable buffer
pool.

The minimum innodb_buffer_pool_chunk_size (1M) restricts the minimum
otherwise we'd have a pool made up of different chunk sizes.

The resulting minimum innodb buffer pool sizes are:

Page Size, Previously minimum (startup), with change.
        4k                            5M           2M
        8k                            5M           3M
       16k                            5M           5M
       32k                           24M          10M
       64k                           24M          20M

With this patch, SET GLOBAL innodb_buffer_pool_size minimums are
enforced.

The evident minimum system variable size for innodb_buffer_pool_size
is 2M, however this is only setable if using 4k page size. As
the order of the page_size and buffer_pool_size aren't fixed, we can't
hide this change.

Subsequent changes:
* innodb_buffer_pool_resize_with_chunks.test - raised of pool resize due to new
  minimums. Chunk size also needed increase as the test was for
  pool_size < chunk_size to generate a warning.
* Removed srv_buf_pool_min_size and replaced use with MYSQL_SYSVAR_NAME(buffer_pool_size).min_val
* Removed srv_buf_pool_def_size and replaced constant defination in
  MYSQL_SYSVAR_LONGLONG(buffer_pool_size)
* Reordered ha_innodb to allow for direct use of MYSQL_SYSVAR_NAME(buffer_pool_size).min_val
* Moved buf_pool_size_align into ha_innodb to access to MYSQL_SYSVAR_NAME(buffer_pool_size).min_val
* loose-innodb_disable_resize_buffer_pool_debug is needed in the
  innodb.restart.opt test so that under debug mode, resizing of the
  innodb buffer pool can occur.

There were no InnoDB changes in the MySQL 5.7.37 release that would be
relevant to MariaDB Server. We will merely update the reported
InnoDB version number.

MDEV-26129 Bad results with join comparing case insensitive VARCHAR/ENUM/SET expression to a _bin ENUM column

Range optimizer incorrectly was used for ENUM columns
when the operation collation did not match the column collation.

Adding a virtual implementation of Field_enum::can_optimize_range()
which tests if the column and the operation collation match.

added dependency devel->shared and conflict with previous versions
update C/C 3.1 with the corresponding C/C part of the fix

information_schema.statistics.collation column needs OPEN_FULL_TABLE,
because it checks index_flags() for HA_READ_ORDER capability.

mtr detects a forced combination if the command line for a test already
includes all options from this combination. options are stored in a perl
hash as (key,value) pairs.

this breaks if the command line has two options with the same name,
like --plugin-load-add=foo --plugin-load-add=bar, and the combination
forces plugin foo.

In particular, this resulted in warnings when running
federated.federatedx_versioning test

The column INFORMATION_SCHEMA.INNODB_LOCKS.LOCK_DATA
would report NULL when the page that contains the locked
record does not reside in the buffer pool.

Pages may be evicted from the buffer pool due to some background
activity, such as the purge of transaction history loading
undo log pages to the buffer pool. The regression tests intentionally
run with a small buffer pool size setting.

To prevent the intermittent test failures, we will filter out the
contents of the LOCK_DATA column from the output.

Fix regression introduced in MDEV-19893

Some errors must be sent with seqno = 0, e.g those that are detected
before server sends its first "welcome" packet (e.g too many connections)
This was not taken into account originally in MDEV-19893 fix.

We need to check sql_errno, before fixing sequence number, to see
if the error we send is really an out-of-bound, e.g a KILL.

MDEV-25201 : Assertion `thd->wsrep_trx_meta.gtid.seqno == (-1)' failed in int wsrep_to_isolation_begin(THD*, const char*, const char*, const TABLE_LIST*, Alter_info*)

Test case does not assert anymore but works incorrectly. We should
not replicate PREPARE using TOI.

is connected so moved test here.

MDEV-25549 : Assertion `*new_engine' failed in bool check_engine(THD*, const char*, const char*, HA_CREATE_INFO*)

In Galera case we call check_engine that could set create_info->db_type
to NULL e.g. if TEMPORARY is not supported by storage engine. Thus,
we need to restore it after that call because it is needed later
on mysql_create_table that will also call check_engine.

For MERGE-tables we need to init children list before calling
show_create_table and then detach children before we continue
normal mysql_create_like_table execution.

MDEV-25472 : Server crashes when wsrep_cluster_address set to unkown address and wsrep_slave_threads to 0

Return failure if we are not connected when slave threads are set

MDEV-20325: Assertion `outer_context || !*from_field || *from_field == not_found_field' failed in Item_field::fix_outer_field | `!derived->is_excluded()' failed in TABLE_LIST::set_check_materialized | SIGEGV in st_select_lex::mark_as_dependent (optimized builds)

Re-execution of a query containing subquery in the FROM clause results
in assert failure in case the query is run as part of a stored routine or
as a prepared statement AND derived table merge optimization is off.
As an example, the following test case
  CREATE TABLE t1 (a INT) ;
  CREATE PROCEDURE sp() SELECT * FROM (SELECT a FROM t1) tb;
  CALL sp();
  SET optimizer_switch='derived_merge=off';
  CALL sp();
results in assert failure on the second invocation of the 'sp' stored routine.

The reason for assertion failure is that the expression
  derived->is_excluded()
returns the value true where the value false expected.

The method is_excluded() returns the value true for a derived table
that has been merged to a parent select. Such transformation happens as part
of Derived Table Merge Optimization that is performed on first invocation of
a stored routine or a prepared statement containing a query with subquery
in the FROM clause of the main SELECT.

When the same routine or prepared statement is run the second time and
Derived Table Merge Optimization is OFF the MariaDB server tries to materialize
a derived table specified by the subquery that fails since this subquery
has already been merged to the top-most SELECT. This transformation is permanent
and can't be reverted. That is the reason why the assert
  DBUG_ASSERT(!derived->is_excluded());
fails inside the function TABLE_LIST::set_check_materialized().

Similar behaviour can be observed in case a stored routine or prepared statement
containing a SELECT statement with subquery in the FROM clause, first is run
with the optimizer_switch option set to derived_merge=off and re-run after this
option has been switched to derived_merge=on. In this case a derived table for
subquery is materialized on the first execution and marked as merged derived
table on the second execution that results in error with misleading error
message:

MariaDB [test]> CALL sp1();
ERROR 1030 (HY000): Got error 1 "Operation not permitted" from storage engine MEMORY

To fix the issue, a derived table that has been already optimized shouldn't be
re-marked for one more round of optimization.

One significant consequence following from suggested change is that the data
member TABLE_LIST::derived_type is not updated once the table optimization
has been done. This fact should be taken into account when Prepared Statement
being handled since once a table listed in a query has been optimized on
execution of the statement PREPARE FROM it won't be touched anymore on handling
the statement EXECUTE.

One side effect caused by this change could be observed for the following
test case:
  CREATE TABLE t1 (s1 INT);
  CREATE VIEW v1 AS
    SELECT s1,s2 FROM (SELECT s1 as s2 FROM t1 WHERE s1 <100) x, t1 WHERE t1.s1=x.s2;
  INSERT INTO v1 (s1) VALUES (-300);

  PREPARE stmt FROM "INSERT INTO v1 (s1) VALUES (-300)";
  EXECUTE stmt;

Execution of the above EXECUTE statement results in issuing the error
ER_COLUMNACCESS_DENIED_ERROR since table_ref->is_merged_derived() is false
and check_column_grant_in_table_ref() called for a temporary table that
shouldn't be. To fix this issue the function find_field_in_tables has been
modified in such a way that the function check_column_grant_in_table_ref()
is not called for a temporary table.

This bug could cause a crash of the server for queries with a derived table
whose specification contained the set function using a subquery over a view
as its only argument. The crash could happen if the specification of the
view contained an outer reference. In this case the aggregation select
could be determined incorrectly.
The crash also could be observed if a CTE is used instead of the view, but
only for queries having at least two references to the CTE.

The cause of this bug is the same as of the bug MDEV-24454.
This bug manifested itself at the second execution of the queries that
contained a set function whose only argument was outer reference to
a column of a mergeable view or derived table or CTE. The first execution
of such query worked fine, but the second execution of the query caused
a crash of the server because the aggregation select for the used set
function was determined incorrectly at the name resolution phase of the
second execution.

This patch reverts the fixes of the bugs MDEV-24454 and MDEV-25631 from
the commit 3690c549.
It leaves the changes in plugin/feedback/feedback.cc and corresponding
test files introduced in this commit intact.

Proper fixes for the bug MDEV-24454 and MDEV-25631 will follow immediately.

The bug that made build stuck in bison call seems to have disappeared
with image update

allow injecting extra files into the bintar

Problem:
========
A slave’s relay log format description event is used when
calculating Seconds_Behind_Master (SBM). This forces the SBM
value to spike when processing these events, as their creation
date is set to the timestamp that the IO thread begins.

Solution:
========
When the slave generates a format description event, mark the
event as a relay log event so it does not update the
rli->last_master_timestamp variable.

Reviewed By:
============
Andrei Elkin <andrei.elkin@mariadb.com>

table: rows are counted twice

Analysis: When the table we are trying to insert into and the SELECT table
are same for INSERT ... SELECT, rows from the SELECT table are copied into
internal temporary table and then to the INSERT table. We only want to
count the rows when we start inserting into the table.
Fix: Reset the counter to 1 before starting to copy from internal temporary
table to select table and then increment the counter.

... when two packages are installed.

(fc35 with i686 and x86_64 packages of libsepol installed).
$ rpm -q --qf "%{VERSION}" libsepol
3.33.3

Restricting the version to the current achitecture generates
a much more obtainable version dependency.

$ rpm -q --qf "%{VERSION}" libsepol.x86_64
3.3

This make dependency resolution easier preventing:
$ sudo dnf localinstall  MariaDB-server-10.8.0-1.fc35.x86_64.rpm ...
Last metadata expiration check: 2:06:49 ago on Thu 30 Dec 2021 14:02:32.
Error:
 Problem 1: conflicting requests
  - nothing provides libsepol >= 3.33.3 needed by MariaDB-server-10.8.0-1.fc35.x86_64

The CMAKE_SYSTEM_PROCESSOR is used in the generation of architecture
filenames so its preduent to just use the same version.

Adding 10.2 specific tests only. No code changes.

The problem was earlier fixed by MDEV-21580 and MDEV-22715.

The problem happened because Item_default_value did not overload
properly the val_xxx_result() family methods.

This change backports the patch for:
MDEV-24958 Server crashes in my_strtod / Value_source::Converter_strntod::Converter_strntod with DEFAULT(blob)
which earlier fixed the problem in 10.3.

failed in Diagnostics_area::set_ok_status in my_ok from
mysql_sql_stmt_prepare

Analysis: Before PREPARE is executed, binlog_format is STATEMENT.
This PREPARE had SET STATEMENT which sets binlog_format to ROW. Now after
PREPARE is done we reset the binlog_format (back to STATEMENT). But we have
temporary table, it doesn't let changing binlog_format=ROW to
binlog_format=STATEMENT and gives error which goes unreported. This
unreported error eventually causes assertion failure.
Fix: Change return type for LEX::restore_set_statement_var() to bool and
make it return error state.

MDEV-27364 Build from 10.2-10.4 srpm fails on RHEL8 family due to discrepancy in jemalloc requirements

pass jemalloc requirement down to srpm build

If when extracting a range condition for an index from the WHERE condition
Range Optimizer sees that the range condition covers the whole index then
such condition should be discarded because it cannot be used in any range
scan. In some cases Range Optimizer really does it, but there remained some
conditions for which it was not done. As a result the optimizer could
produce index merge plans with the full index scan for one of the indexes
participating in the index merge.
This could be observed in one of the test cases from index_merge1.inc
where a plan with index_merge_sort_union was produced and in the test case
reported for this bug where a plan with index_merge_sort_intersect was
produced. In both cases one of two index scans participating in index merge
ran over the whole index.
The patch slightly changes the original above mentioned test case from
index_merge1.inc to be able to produce an intended plan employing
index_merge_sort_union. The original query was left to show that index
merge is not used for it anymore.
It should be noted that for the plan with index_merge_sort_intersect could
be chosen for execution only due to a defect in the InnoDB code that
returns wrong estimates for the cardinality of big ranges.

This bug led to serious problems in 10.4+ where the optimization using
Rowid filters is employed (see mdev-26446).

Approved by Sergey Petrunia <sergey@mariadb.com>