bulk_insert_apply_for_table(dict_table_t*)

This issue is caused by
commit 188c5da7 (MDEV-32453).

trx_t::bulk_insert_apply_for_table(): Remove the assert
check_unique_secondary and check_foreigns. InnoDB can
apply the bulk insert operation even after disabling
the check_foreigns and check_unique_secondary variable.

- When `libcurl` is installed in path out of default path, like on
Windows, `include_directories` failed to find `curl/curl.h`.
- Fix `cmake` by using modern syntax with imported target and
`find_package`
- Fix warnings treated as the errors
  - Remove `HASHICORP_HAVE_EXCEPTIONS` macro and related code
- Add package to `Server` component in Windows
- Tested with `$ ./mysql-test/mtr --suite=vault`
- Closes PR #3068
- Reviewer: <wlad@mariadb.com>
            <julius.goryavsky@mariadb.com>

SyncFileIO should do multiple read/write call
when length is greater than os_file_request_size_max value.

- This issue is caused by commit 188c5da7 (MDEV-32453).
InnoDB fails to end the bulk insert for the table after
applying the bulk insert operation. This leads to assertion
during commit process.

Before MDEV-15158, wsrep xid information was stored in only one place:
in the TRX_SYS page. Starting with 10.3, it is not stored there but
in the rollback segment header pages, and the latest one is what
matters. MDEV-19229 allows the undo tablespaces to be rebuilt when
innodb_undo_tablespaces is changed on startup. Previously it was not
possible to change that parameter.

These changes caused the fact that rollback segment header pages could
contain several wsrep xid's stored and when undo tablespaces were
rebuilt there was a effort to restore wsrep xid back to rollback
segment header page but because there was several of them the latest
wsrep xid was overwritten with older one.

trx_rseg_read_wsrep_checkpoint
trx_rseg_init_wsrep_xid
	Return true if read xid is wsrep xid, false if not

trx_rseg_mem_restore
	Try to read wsrep xid and if it is found copy it to
	trx_sys.recovered_wsrep_xid if read xid has larger
	seqno.

log_t: Define buf_size, max_buf_free as 32-bit and next_checkpoint_no
as byte (we only need a bit) and rearrange some data members,
so that on AMD64 we can fit log_sys.latch and log_sys.log in
the same 64-byte cache line.

mtr_t::commit_log(), mtr_t::commit_logger: A part of mtr_t::commit()
split into a separate function, so that we will not unnecessarily invoke
log_sys.get_write_target() when running on a memory-mapped log file,
or log_sys.is_pmem().

Reviewed by: Vladislav Vaintroub
Tested by: Matthias Leich

On Microsoft Windows, ReadFile() as well as WriteFile() limit the size
of the request to DWORD, which is 32 bits (at most 4 GiB - 1) also on
64-bit systems.

On FreeBSD, sysctl debug.iosize_max_clamp could limit the size of a
write request to INT_MAX. The size of a read request is always limited
to INT_MAX. This would allow the request size to be 4095 bytes more than
the Linux limit (0x7ffff000 according to "man 2 read" and "man 2 write").

On OpenBSD, Solaris and possibly NetBSD, the read request size is limited
to SSIZE_T_MAX, which would be half the current maximum
innodb_log_buffer_size. This should be not much of an issue anyway,
because on contemporary 64-bit platforms, the virtual addresses are
limited to 48 bits.

IBM AIX documentation mentions OFF_MAX which would apply when
a 64-bit application is running on a 32-bit kernel.

Let us declare innodb_log_buffer_size as 32-bit unsigned and make the
maximum 0x7ffff000, to be compatible with the least common
denominator (Linux).

The maximum innodb_sort_buffer_size already was 64 MiB,
which is not a problem.

SyncFileIO::execute(): Assert that the size of a synchronous read or
write request is limited to the maximum.

Reviewed by: Vladislav Vaintroub

Json test about max statement time fails with freebsd because on some
architectures the test might execute faster and the statement may not fail.

To simulate failure regardless of architecture, introduce a wait of seconds
longer than the max_statement_time.

Reason:
=======
- InnoDB fails to apply the buffered insert operation if the
after insert trigger does change the same table. This behaviour
leads to empty table for the subsequent insert operation
and server abort.

Solution:
========
- InnoDB should apply buffered insert operation if
"after insert" trigger changes the same table.

In commit f5ca4077, an the "--log" option was added but it was implemented by
invoking a sub-shell. Any errors in the sub-shell would not be propagated and
thus cause mini-benchmark to suppress errors.

The fix here is to propagate the exit status of the sub-shell.

All new code of the whole pull request, including one or several files that are
either new files or modified ones, are contributed under the BSD-new license. I
am contributing on behalf of my employer Amazon Web Services, Inc.

As performance improves, the permitted number of CPU cycles to be used
during the benchmark should be lowered to catch performance regressions. An
option is added to better support this.

All new code of the whole pull request, including one or several files that are
either new files or modified ones, are contributed under the BSD-new license. I
am contributing on behalf of my employer Amazon Web Services, Inc.

Original problem was error when configuring without initialized
submodule Columnstore:

  The source directory

    /home/midenok/src/mariadb/10.6c/src/storage/columnstore/columnstore

  does not contain a CMakeLists.txt file.

The original fix disabled Columnstore build when PLUGIN_COLUMNSTORE
was not defined, but this seems to be wrong and any plugin should be
built if it is not explicitly disabled. This is expected by buildbots.

Thanks to Vladislav Vaintroub <vvaintroub@gmail.com> for the fix

In mysql_install_db_win_admin test, dump bootstrap output in case of
bootstrap failure.

CMake doesn't set ${PLUGIN_COLUMNSTORE} to anything.

Delayed_insert has its own THD (initialized at mysql_insert()) and
hence its own LEX. Delayed_insert initalizes a very few parameters for
LEX and 'duplicates' is not in this list. Now we copy this missing
parameter from parser LEX (as well as sql_command).

MDEV-31903 Server crashes in _ma_reset_history upon UNLOCK table with auto-create history partitions

When INSERT does auto-create for t1 all its handler instances are
closed by alter_close_table(). At this time down the stack
maria_close() clears share->state_history. Later when we unlock the
tables Aria transaction manager accesses old share instance (the one
before t1 was closed) and tries to reset its state_history.

The problem is maria_close() didn't remove table from transaction's
list (used_tables). The fix does _ma_remove_table_from_trnman() which
is triggered by HA_EXTRA_PREPARE_FOR_RENAME.

Some fixes related to commit f838b2d7 and
Rows_log_event::do_apply_event() and Update_rows_log_event::do_exec_row()
for system-versioned tables were provided by Nikita Malyavin.
This was required by test versioning.rpl,trx_id,row.

The mini-benchmark.sh script failed to run in the latest Fedora
distributions in GitLab CI. Executing the benchmark inside a Docker
container had failed because the check for `perf` was done in a way that
caused the benchmark to exit because of the `set -e` option. Test and
skip `perf` to allowing the remaining benchmark activities to proceed.

This check was added in acb6684 but inadvertantly reverted in 42a1f94.

Logic was corrected to only run perf when the flag is enabled, and to
prevent perf stat and perf record from being simultaneously enabled.

Set -ex is also added to enable easier identification of mini-benchmark
issues in the future.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.

Item_func_group_concat::print() did not take into account
that Item_func_group_concat::separator can be of a different character set
than the "String *str" (when the printing is being done to).
Therefore, printing did not work correctly for:
- non-ASCII separators when GROUP_CONCAT is done on 8bit data
  or multi-byte data with mbminlen==1.
- all separators (even including simple ones like comma)
  when GROUP_CONCAT is done on ucs2/utf16/utf32 data (mbminlen>1).

Because of this problem, VIEW definitions did not print correctly to
their FRM files. This later led to a wrong SELECT and SHOW CREATE output.

Fix:

- Adding new String methods:

  bool append_for_single_quote_using_mb_wc(const char *str, size_t length,
                                           CHARSET_INFO *cs);

  bool append_for_single_quote_opt_convert(const char *str,
                                           size_t length,
                                           CHARSET_INFO *cs)

  which perform both escaping and character set conversion at the same time.

- Adding a new String method escaped_wc_for_single_quote(),
  to reuse the code between the old and the new methods.

- Fixing Item_func_group_concat::print() to use the new
  method append_for_single_quote_opt_convert().

innodb_log_spin_wait_delay_update(): Always acquire log_sys.latch
to protect the change of mtr_t::spin_wait_delay.

log_t::lock_lsn(): In the general case, actually use
mtr_t::spin_wait_delay as it was intended. In the x86 specific
log_t::lock_lsn_bts() we used mtr_t::spin_wait_delay.

Queries that select concatenated constant strings now have
colname and value that match.  For example,
  SELECT '123' 'x';
will return a result where the column name and value both
are '123x'.

Review: Daniel Black

.. even with MDEV-9095 fix

CapabilityBounding sets require filesystem setcap attributes
for the executable to gain privileges during execution.

A side effect of this however is the getauxvec(AT_SECURE) gets
set, and the secure_getenv from OpenSSL internals on
OPENSSL_CONF environment variable will get ignored (openssl gh issue
21770).

According to capabilities(7), Ambient capabilities don't trigger
ld.so triggering the secure execution mode.

Include SELinux and Apparmor capabilities for ipc_lock

This was the orginal implementation that reverted with a bunch of
commits.

This reverts commit a13e521b.

Revert "cmake: append to the array correctly"
This reverts commit 51e3f1da.

Revert "build failure with cmake < 3.10"
This reverts commit 49cf702e.

Revert "MDEV-33301 memlock with systemd still not working"
This reverts commit 8a1904d7.

We should not set debug sync point when holding a mutex
to avoid mutex ordering failure.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

User transactions may acquire explicit MDL locks from InnoDB level
when persistent statistics is re-read for a table.
If such a transaction would be subject to BF-abort, it was improperly
detected as a system transaction and wouldn't get aborted.

The fix: Check if a transaction holding explicit MDL locks is a user
transaction in the MDL conflict handling code.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

Add "real ip:<ip_or_localhost>" part to the aborted message
Only for proxy-protocoled connection, so it does not  not to cause
confusion to normal users.

Problem is that not all conflicting transactions have THD object.
Therefore, it must be checked that victim has THD
before it's identification is added to victim list as victim's
thread identification is later requested using thd_get_thread_id
function that requires that we have valid pointer to THD object
in trx->mysql_thd.

Victim might not have trx->mysql_thd in two cases:

(1) An incomplete transaction that was recovered from undo logs
on server startup (and not yet rolled back).

(2) Transaction that is in XA PREPARE state and whose client
connection was disconnected.

Neither of these can complete before lock_wait_wsrep()
releases lock_sys.latch.

(1) trx_t::commit_in_memory() is clearing both
trx_t::state and trx_t::is_recovered before it invokes
lock_release(trx_t*) (which would be blocked by the exclusive
lock_sys.latch that we are holding here). Hence, it is not
possible to write a debug assertion to document this scenario.

(2) If is in XA PREPARE state, it would eventually be rolled
back and the lock conflict would be resolved when an XA COMMIT
or XA ROLLBACK statement is executed in some other connection.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

.. is not updating some system tables

Some schema changes from MDEV-24312 master_host has 60 character limit, increase to 255 bytes
failed to happen in the upgrade for tables in the mysql schema:
* mysql.global_priv
* mysql.procs_priv
* mysql.proxies_priv
* mysql.roles_mapping

MDEV-32787 : Assertion `!wsrep_has_changes(thd) || (thd->lex->sql_command == SQLCOM_CREATE_TABLE && !thd->is_current_stmt_binlog_format_row()) || thd->wsrep_cs().transaction().state() == wsrep::transaction::s_aborted' failed in void wsrep_commit_empty(THD*, bool)

When we commit empty transaction we should allow wsrep
transaction to be on s_must_replay state for DDL that
was killed during certification.

Fix is tested with RQG because deterministic mtr-testcase
was not found.
Signed-off-by: Julius Goryavsky <julius.goryavsky@mariadb.com>

MONITOR_INC_VALUE_CUMULATIVE is a multiline macro, so the second statement
will be executed always, regardless of "if" condition.

These problems first started with
commit b1ab211d (MDEV-15053).

Thanks to Yury Chaikou from ServiceNow for the report.

From the correctness point of view, it should be safe to release
all locks on index records that were not modified by the transaction.
Doing so should make the locks after XA PREPARE fully compatible
with what would happen if the server were restarted: InnoDB table
IX locks and exclusive record locks would be resurrected based on
undo log records.

Concurrently running transactions that are waiting for a lock may invoke
lock_rec_convert_impl_to_expl() to create an explicit record lock object
on behalf of the lock-owning transaction so that they can attaching
their waiting lock request on the explicit record lock object. Explicit
locks would be released by trx_t::release_locks() during commit or
rollback.

Any clustered index record whose DB_TRX_ID belongs to a transaction that
is in active or XA PREPARE state will be implicitly locked by that
transaction. On XA PREPARE, we can release explicit exclusive locks on
records whose DB_TRX_ID does not match the current transaction identifier.

lock_rec_unlock_unmodified(): Release record locks that are not implicitly
held by the current transaction.

lock_release_on_prepare_try(), lock_release_on_prepare():
Invoke lock_rec_unlock_unmodified().

row_trx_id_offset(): Declare non-static.

lock_rec_unlock(): Replaces lock_rec_unlock_supremum().

Reviewed by: Vladislav Lesin

By design, InnoDB has always hung when permanently running out of
buffer pool, for example when several threads are waiting to allocate
a block, and all of the buffer pool is buffer-fixed by the active threads.

The hang that we are fixing here occurs when the buffer pool is only
temporarily running out and the situation could be rescued by writing out
some dirty pages or evicting some clean pages.

buf_LRU_get_free_block(): Simplify the way how we wait for
the buf_flush_page_cleaner thread. This fixes occasional hangs
of the test encryption.innochecksum that were introduced by
commit a55b951e (MDEV-26827).
To play it safe, we use a timed wait when waiting for the
buf_flush_page_cleaner() thread to perform its job. Should that
thread get stuck, we will invoke buf_pool.LRU_warn() in order to
display a message that pages could not be freed, and keep trying
to wake up the buf_flush_page_cleaner() thread.

The INFORMATION_SCHEMA.INNODB_METRICS counters
buffer_LRU_single_flush_failure_count and
buffer_LRU_get_free_waits will be removed.
The latter is represented by buffer_pool_wait_free.

Also removed will be the message
"InnoDB: Difficult to find free blocks in the buffer pool"
because in d34479dc we
introduced a more precise message
"InnoDB: Could not free any blocks in the buffer pool"
in the buf_flush_page_cleaner thread.

buf_pool_t::LRU_warn(): Issue the warning message that we could
not free any blocks in the buffer pool. This may also be invoked
by buf_LRU_get_free_block() if buf_flush_page_cleaner() appears
to be stuck.

buf_pool_t::n_flush_dec(): Remove.

buf_pool_t::n_flush_dec_holding_mutex(): Rename to n_flush_dec().

buf_flush_LRU_list_batch(): Increment the eviction counter for blocks
of temporary, discarded or dropped tablespaces.

buf_flush_LRU(): Make static, and remove the constant parameter
evict=false. The only caller will be the buf_flush_page_cleaner()
thread.

IORequest::is_LRU(): Remove. The only case of evicting pages on
write completion will be when we are writing out pages of the
temporary tablespace. Those pages are not in buf_pool.flush_list,
only in buf_pool.LRU.

buf_page_t::flush(): Remove the parameter evict.

buf_page_t::write_complete(): Change the parameter "bool temporary"
to "bool persistent" and add a parameter for an already read state().

Reviewed by: Debarun Banerjee