1. 11 Apr, 2023 1 commit
  2. 04 Apr, 2023 2 commits
  3. 02 Apr, 2023 3 commits
  4. 31 Mar, 2023 1 commit
    • Sergei Petrunia's avatar
      MDEV-30806: ANALYZE FORMAT=JSON: better support for BNL and BNL-H joins · dc1d6213
      Sergei Petrunia authored
      In block-nl-join, add:
      
      - r_loops - this shows how many incoming record combinations this
        query plan node had.
      
      - r_effective_rows - this shows the average number of matching rows
        that this table had for each incoming record combination. This is
        comparable with r_rows in non-blocked access methods.
        For BNL-joins, it is always equal to
         $.table.r_rows * $.table.r_filtered
        For BNL-H joins the value cannot be computed from other values
      
      Reviewed by: Monty <monty@mariadb.org>
      dc1d6213
  5. 30 Mar, 2023 1 commit
    • Jan Lindström's avatar
      MDEV-30413 : run sequence nextval got [Note] WSREP: MDL BF-BF conflict and [ERROR] Aborting · 169def14
      Jan Lindström authored
      Sequence objects are implemented using special tables.
      These tables do not have primary key and only one row.
      NEXTVAL is basically update from existing value to new
      value. In Galera this could mean that two write-sets
      from different nodes do not conflict and this could
      lead situation where write-sets are executed
      concurrently and possibly in wrong seqno order.
      
      This is fixed by using table-level exclusive key for
      SEQUENCE updates. Note that this naturally works
      correctly only if InnoDB storage engine is used for
      sequence.
      
      This fix does not contain a test case because while
      it is possible to syncronize appliers using dbug_sync
      it was too hard to syncronize MDL-lock requests to
      exact objects. Testing done for this fix is documented
      on MDEV.
      Signed-off-by: default avatarJulius Goryavsky <julius.goryavsky@mariadb.com>
      169def14
  6. 28 Mar, 2023 3 commits
    • Marko Mäkelä's avatar
      Merge 10.5 into 10.6 · 0760ad33
      Marko Mäkelä authored
      0760ad33
    • Marko Mäkelä's avatar
      MDEV-30936 fixup · 402f36dd
      Marko Mäkelä authored
      fil_space_t::~fil_space_t(): Invoke ut_free(name) because
      doing so in the callers would trip MSAN_OPTIONS=poison_in_dtor=1
      402f36dd
    • Marko Mäkelä's avatar
      MDEV-30936 clang 15.0.7 -fsanitize=memory fails massively · dfa90257
      Marko Mäkelä authored
      handle_slave_io(), handle_slave_sql(), os_thread_exit():
      Remove a redundant pthread_exit(nullptr) call, because it
      would cause SIGSEGV.
      
      mysql_print_status(): Add MEM_MAKE_DEFINED() to work around
      some missing instrumentation around mallinfo2().
      
      que_graph_free_stat_list(): Invoke que_node_get_next(node) before
      que_graph_free_recursive(node). That is the logical and
      MSAN_OPTIONS=poison_in_dtor=1 compatible way of freeing memory.
      
      ins_node_t::~ins_node_t(): Invoke mem_heap_free(entry_sys_heap).
      
      que_graph_free_recursive(): Rely on ins_node_t::~ins_node_t().
      
      fts_t::~fts_t(): Invoke mem_heap_free(fts_heap).
      
      fts_free(): Replace with direct calls to fts_t::~fts_t().
      
      The failures in free_root() due to MSAN_OPTIONS=poison_in_dtor=1
      will be covered in MDEV-30942.
      dfa90257
  7. 24 Mar, 2023 4 commits
    • Andrei's avatar
      MDEV-26071: rpl.rpl_perfschema_applier_status_by_worker failed in bb … · 216d99bb
      Andrei authored
      …with: Test assertion failed
      
      Problem:
      =======
      Assertion text: 'Value returned by SSS and PS table for Last_Error_Number
      should be same.'
      Assertion condition: '"1146" = "0"'
      Assertion condition, interpolated: '"1146" = "0"'
      Assertion result: '0'
      
      Analysis:
      ========
      In parallel replication when slave is started the worker pool gets
      activated and it gets cleared when slave stops. Each time the worker pool
      gets activated a backup worker pool also gets created to store worker
      specific perforance schema information in case of errors. On error, all
      relevant information is copied from rpl_parallel_thread to rli and it gets
      cleared from thread.  Then server waits for all workers to complete their
      work, during this stage performance schema table specific worker info is
      stored into the backup pool and finally the actual pool gets cleared. If
      users query the performance schema table to know the status of workers the
      information from backup pool will be used. The test simulates
      ER_NO_SUCH_TABLE error and verifies the worker information in pfs table.
      Test works fine if execution occurs in following order.
      
      Step 1. Error occurred 'worker information is copied to backup pool'.
      Step 2. handle_slave_sql invokes 'rpl_parallel_resize_pool_if_no_slaves' to
      deactivate worker pool, it marks the pool->count=0
      Step 3. PFS table is queried, since actual pool is deactivated backup pool
      information is read.
      
      If the Step 3 happens prior to Step2 the pool is yet to be deactivated and
      the actual pool is read, which doesn't have any error details as they were
      cleared. Hence test ocasionally fails.
      
      Fix:
      ===
      Upon error mark the back pool as being active so that if PFS table is
      quried since the backup pool is flagged as valid its information will be
      read, in case it is not flagged regular pool will be read.
      
      This work is one of the last pieces created by the late Sujatha Sivakumar.
      216d99bb
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-29545 InnoDB: Can't find record during replace stmt · e06c6046
      Thirunarayanan Balathandayuthapani authored
      Problem:
      ========
      - InnoDB replace statement returns can't find record as result during
      bulk insert operation. InnoDB returns DB_END_OF_INDEX blindly when
      bulk transaction is visible to current transaction even though
      the search tuple is inserted as a part of current replace statement.
      
      Solution:
      =========
      row_search_mvcc(): InnoDB should allow the transaction to read
      all the rows when innodb intends to do any locking on the
      record even though bulk insert transaction changes are
      visible to the current transaction
      e06c6046
    • Marko Mäkelä's avatar
      MDEV-30900 Crash on macOS due to zero-initialized buf_dblwr.write_cond · 07460c31
      Marko Mäkelä authored
      buf_dblwr_t::init(), buf_dblwr_t::close(): Cover also write_cond,
      which was added in commit a55b951e
      without explicit initialization. On GNU/Linux, PTHREAD_COND_INITIALIZER
      is a zero-initializer. That is why the default zero initialization
      happened to work on that platform.
      07460c31
    • Daniel Black's avatar
      rpl_reporting: sprintf -> snprintf · 15ca6c5a
      Daniel Black authored
      This was failing to compile with AppleClang 14.0.0.14000029.
      
      Thanks to Arunesh Choudhary for noticing.
      15ca6c5a
  8. 23 Mar, 2023 1 commit
  9. 22 Mar, 2023 2 commits
    • Marko Mäkelä's avatar
      Merge 10.5 into 10.6 · 1efdf67e
      Marko Mäkelä authored
      1efdf67e
    • Marko Mäkelä's avatar
      MDEV-30882 Crash on ROLLBACK in a ROW_FORMAT=COMPRESSED table · 701399ad
      Marko Mäkelä authored
      btr_cur_upd_rec_in_place(): Avoid calling page_zip_write_rec() if we
      are not modifying any fields that are stored in compressed format.
      
      btr_cur_update_in_place_zip_check(): New function to check if a
      ROW_FORMAT=COMPRESSED record can actually be updated in place.
      
      btr_cur_pessimistic_update(): If the BTR_KEEP_POS_FLAG is not set
      (we are in a ROLLBACK and cannot write any BLOBs), ignore the potential
      overflow and let page_zip_reorganize() or page_zip_compress() handle it.
      This avoids a failure when an attempted UPDATE of an NULL column to 0 is
      rolled back. During the ROLLBACK, we would try to move a non-updated
      long column to off-page storage in order to avoid a compression failure
      of the ROW_FORMAT=COMPRESSED page.
      
      page_zip_write_trx_id_and_roll_ptr(): Remove an assertion that would fail
      in row_upd_rec_in_place() because the uncompressed page would already
      have been modified there.
      
      This is a 10.5 version of commit ff3d4395
      (different because of commit 08ba3887).
      701399ad
  10. 21 Mar, 2023 1 commit
    • Tingyao Nian's avatar
      [MDEV-30824] Fix binlog to use 'String' for setting 'character_set_client' · dccbb5a6
      Tingyao Nian authored
      Commit a923d6f4 disabled numeric setting
      of character_set_* variables with non-default values:
      
        MariaDB [(none)]> set character_set_client=224;
        ERROR 1115 (42000): Unknown character set: '224'
      
      However the corresponding binlog functionality still write numeric
      values for log event, and this will break binlog replay if the value is
      not default. Now make the server use 'String' type for
      'character_set_client' when generating binlog events
      
      Before:
      
        /*!\C utf8mb4 *//*!*/;
        SET @@session.character_set_client=224,@@session.collation_connection=224,@@session.collation_server=33/*!*/;
      
      After:
      
        /*!\C utf8mb4 *//*!*/;
        SET @@session.character_set_client=utf8mb4,@@session.collation_connection=33,@@session.collation_server=8/*!*/;
      
      Note: prior to the previous commit, setting with '224' or '45' or
      'utf8mb4' have the same effect, as they all set the parameter to
      'utf8mb4'.
      
      All new code of the whole pull request, including one or several files
      that are either new files or modified ones, are contributed under the
      BSD-new license. I am contributing on behalf of my employer Amazon Web
      Services, Inc.
      dccbb5a6
  11. 20 Mar, 2023 2 commits
  12. 17 Mar, 2023 2 commits
  13. 16 Mar, 2023 6 commits
    • Marko Mäkelä's avatar
      MDEV-26827 Make page flushing even faster · a55b951e
      Marko Mäkelä authored
      For more convenient monitoring of something that could greatly affect
      the volume of page writes, we add the status variable
      Innodb_buffer_pool_pages_split that was previously only available
      via information_schema.innodb_metrics as "innodb_page_splits".
      This was suggested by Axel Schwenke.
      
      buf_flush_page_count: Replaced with buf_pool.stat.n_pages_written.
      We protect buf_pool.stat (except n_page_gets) with buf_pool.mutex
      and remove unnecessary export_vars indirection.
      
      buf_pool.flush_list_bytes: Moved from buf_pool.stat.flush_list_bytes.
      Protected by buf_pool.flush_list_mutex.
      
      buf_pool_t::page_cleaner_status: Replaces buf_pool_t::n_flush_LRU_,
      buf_pool_t::n_flush_list_, and buf_pool_t::page_cleaner_is_idle.
      Protected by buf_pool.flush_list_mutex. We will exclusively broadcast
      buf_pool.done_flush_list by the buf_flush_page_cleaner thread,
      and only wait for it when communicating with buf_flush_page_cleaner.
      There is no need to keep a count of pending writes by the
      buf_pool.flush_list processing. A single flag suffices for that.
      
      Waits for page write completion can be performed by
      simply waiting on block->page.lock, or by invoking
      buf_dblwr.wait_for_page_writes().
      
      buf_LRU_block_free_non_file_page(): Broadcast buf_pool.done_free and
      set buf_pool.try_LRU_scan when freeing a page. This would be
      executed also as part of buf_page_write_complete().
      
      buf_page_write_complete(): Do not broadcast buf_pool.done_flush_list,
      and do not acquire buf_pool.mutex unless buf_pool.LRU eviction is needed.
      Let buf_dblwr count all writes to persistent pages and broadcast a
      condition variable when no outstanding writes remain.
      
      buf_flush_page_cleaner(): Prioritize LRU flushing and eviction right after
      "furious flushing" (lsn_limit). Simplify the conditions and reduce the
      hold time of buf_pool.flush_list_mutex. Refuse to shut down
      or sleep if buf_pool.ran_out(), that is, LRU eviction is needed.
      
      buf_pool_t::page_cleaner_wakeup(): Add the optional parameter for_LRU.
      
      buf_LRU_get_free_block(): Protect buf_lru_free_blocks_error_printed
      with buf_pool.mutex. Invoke buf_pool.page_cleaner_wakeup(true) to
      to ensure that buf_flush_page_cleaner() will process the LRU flush
      request.
      
      buf_do_LRU_batch(), buf_flush_list(), buf_flush_list_space():
      Update buf_pool.stat.n_pages_written when submitting writes
      (while holding buf_pool.mutex), not when completing them.
      
      buf_page_t::flush(), buf_flush_discard_page(): Require that
      the page U-latch be acquired upfront, and remove
      buf_page_t::ready_for_flush().
      
      buf_pool_t::delete_from_flush_list(): Remove the parameter "bool clear".
      
      buf_flush_page(): Count pending page writes via buf_dblwr.
      
      buf_flush_try_neighbors(): Take the block of page_id as a parameter.
      If the tablespace is dropped before our page has been written out,
      release the page U-latch.
      
      buf_pool_invalidate(): Let the caller ensure that there are no
      outstanding writes.
      
      buf_flush_wait_batch_end(false),
      buf_flush_wait_batch_end_acquiring_mutex(false):
      Replaced with buf_dblwr.wait_for_page_writes().
      
      buf_flush_wait_LRU_batch_end(): Replaces buf_flush_wait_batch_end(true).
      
      buf_flush_list(): Remove some broadcast of buf_pool.done_flush_list.
      
      buf_flush_buffer_pool(): Invoke also buf_dblwr.wait_for_page_writes().
      
      buf_pool_t::io_pending(), buf_pool_t::n_flush_list(): Remove.
      Outstanding writes are reflected by buf_dblwr.pending_writes().
      
      buf_dblwr_t::init(): New function, to initialize the mutex and
      the condition variables, but not the backing store.
      
      buf_dblwr_t::is_created(): Replaces buf_dblwr_t::is_initialised().
      
      buf_dblwr_t::pending_writes(), buf_dblwr_t::writes_pending:
      Keeps track of writes of persistent data pages.
      
      buf_flush_LRU(): Allow calls while LRU flushing may be in progress
      in another thread.
      
      Tested by Matthias Leich (correctness) and Axel Schwenke (performance)
      a55b951e
    • Marko Mäkelä's avatar
      MDEV-26055: Improve adaptive flushing · 9593cccf
      Marko Mäkelä authored
      Adaptive flushing is enabled by setting innodb_max_dirty_pages_pct_lwm>0
      (not default) and innodb_adaptive_flushing=ON (default).
      There is also the parameter innodb_adaptive_flushing_lwm
      (default: 10 per cent of the log capacity). It should enable some
      adaptive flushing even when innodb_max_dirty_pages_pct_lwm=0.
      That is not being changed here.
      
      This idea was first presented by Inaam Rana several years ago,
      and I discussed it with Jean-François Gagné at FOSDEM 2023.
      
      buf_flush_page_cleaner(): When we are not near the log capacity limit
      (neither buf_flush_async_lsn nor buf_flush_sync_lsn are set),
      also try to move clean blocks from the buf_pool.LRU list to buf_pool.free
      or initiate writes (but not the eviction) of dirty blocks, until
      the remaining I/O capacity has been consumed.
      
      buf_flush_LRU_list_batch(): Add the parameter bool evict, to specify
      whether dirty least recently used pages (from buf_pool.LRU) should
      be evicted immediately after they have been written out. Callers outside
      buf_flush_page_cleaner() will pass evict=true, to retain the existing
      behaviour.
      
      buf_do_LRU_batch(): Add the parameter bool evict.
      Return counts of evicted and flushed pages.
      
      buf_flush_LRU(): Add the parameter bool evict.
      Assume that the caller holds buf_pool.mutex and
      will invoke buf_dblwr.flush_buffered_writes() afterwards.
      
      buf_flush_list_holding_mutex(): A low-level variant of buf_flush_list()
      whose caller must hold buf_pool.mutex and invoke
      buf_dblwr.flush_buffered_writes() afterwards.
      
      buf_flush_wait_batch_end_acquiring_mutex(): Remove. It is enough to have
      buf_flush_wait_batch_end().
      
      page_cleaner_flush_pages_recommendation(): Avoid some floating-point
      arithmetics.
      
      buf_flush_page(), buf_flush_check_neighbor(), buf_flush_check_neighbors(),
      buf_flush_try_neighbors(): Rename the parameter "bool lru" to "bool evict".
      
      buf_free_from_unzip_LRU_list_batch(): Remove the parameter.
      Only actual page writes will contribute towards the limit.
      
      buf_LRU_free_page(): Evict freed pages of temporary tables.
      
      buf_pool.done_free: Broadcast whenever a block is freed
      (and buf_pool.try_LRU_scan is set).
      
      buf_pool_t::io_buf_t::reserve(): Retry indefinitely.
      During the test encryption.innochecksum we easily run out of
      these buffers for PAGE_COMPRESSED or ENCRYPTED pages.
      
      Tested by Matthias Leich and Axel Schwenke
      9593cccf
    • Marko Mäkelä's avatar
      MDEV-30357 Performance regression in locking reads from secondary indexes · 4105017a
      Marko Mäkelä authored
      lock_sec_rec_some_has_impl(): Remove a harmful condition that caused the
      performance regression and should not have been added in
      commit b6e41e38 in the first place.
      Locking transactions that have not modified any persistent tables
      can carry the transaction identifier 0.
      
      trx_t::max_inactive_id: A cache for trx_sys_t::find_same_or_older().
      The value is not reset on transaction commit so that previous results
      can be reused for subsequent transactions. The smallest active
      transaction ID can only increase over time, not decrease.
      
      trx_sys_t::find_same_or_older(): Remember the maximum previous id for which
      rw_trx_hash.iterate() returned false, to avoid redundant iterations.
      
      lock_sec_rec_read_check_and_lock(): Add an early return in case we are
      already holding a covering table lock.
      
      lock_rec_convert_impl_to_expl(): Add a template parameter to avoid
      a redundant run-time check on whether the index is secondary.
      
      lock_rec_convert_impl_to_expl_for_trx(): Move some code from
      lock_rec_convert_impl_to_expl(), to reduce code duplication due
      to the added template parameter.
      
      Reviewed by: Vladislav Lesin
      Tested by: Matthias Leich
      4105017a
    • Marko Mäkelä's avatar
      MDEV-29835 InnoDB hang on B-tree split or merge · f2096478
      Marko Mäkelä authored
      This is a follow-up to
      commit de4030e4 (MDEV-30400),
      which fixed some hangs related to B-tree split or merge.
      
      btr_root_block_get(): Use and update the root page guess. This is just
      a minor performance optimization, not affecting correctness.
      
      btr_validate_level(): Remove the parameter "lockout", and always
      acquire an exclusive dict_index_t::lock in CHECK TABLE without QUICK.
      This is needed in order to avoid latching order violation in
      btr_page_get_father_node_ptr_for_validate().
      
      btr_cur_need_opposite_intention(): Return true in case
      btr_cur_compress_recommendation() would hold later during the
      mini-transaction, or if a page underflow or overflow is possible.
      If we return true, our caller will escalate to aqcuiring an exclusive
      dict_index_t::lock, to prevent a latching order violation and deadlock
      during btr_compress() or btr_page_split_and_insert().
      
      btr_cur_t::search_leaf(), btr_cur_t::open_leaf():
      Also invoke btr_cur_need_opposite_intention() on the leaf page.
      
      btr_cur_t::open_leaf(): When escalating to exclusive index locking,
      acquire exclusive latches on all pages as well.
      
      innobase_instant_try(): Return an error code if the root page cannot
      be retrieved.
      
      In addition to the normal stress testing with Random Query Generator (RQG)
      this has been tested with
      ./mtr --mysqld=--loose-innodb-limit-optimistic-insert-debug=2
      but with the injection in btr_cur_optimistic_insert() for non-leaf pages
      adjusted so that it would use the value 3. (Otherwise, infinite page
      splits could occur in some mtr tests.)
      
      Tested by: Matthias Leich
      f2096478
    • Marko Mäkelä's avatar
      Merge 10.5 into 10.6 · 85cbfaef
      Marko Mäkelä authored
      85cbfaef
    • Marko Mäkelä's avatar
      MDEV-30860 Race condition between buffer pool flush and log file deletion in... · 1495f057
      Marko Mäkelä authored
      MDEV-30860 Race condition between buffer pool flush and log file deletion in mariadb-backup --prepare
      
      srv_start(): If we are going to close the log file in
      mariadb-backup --prepare, call buf_flush_sync() before
      calling recv_sys.debug_free() to ensure that the log file
      will not be accessed.
      
      This fixes a rather rare failure in the test
      mariabackup.innodb_force_recovery where buf_flush_page_cleaner()
      would invoke log_checkpoint_low() because !recv_recovery_is_on()
      would hold due to the fact that recv_sys.debug_free() had
      already been called. Then, the log write for the checkpoint
      would fail because srv_start() had invoked log_sys.log.close_file().
      1495f057
  14. 10 Mar, 2023 2 commits
    • Vlad Lesin's avatar
      MDEV-30775 Performance regression in fil_space_t::try_to_close() introduced in MDEV-23855 · 7d6b3d40
      Vlad Lesin authored
      fil_node_open_file_low() tries to close files from the top of
      fil_system.space_list if the number of opened files is exceeded.
      
      It invokes fil_space_t::try_to_close(), which iterates the list searching
      for the first opened space. Then it just closes the space, leaving it in
      the same position in fil_system.space_list.
      
      On heavy files opening, like during 'SHOW TABLE STATUS ...' execution,
      if the number of opened files limit is reached,
      fil_space_t::try_to_close() iterates more and more closed spaces before
      reaching any opened space for each fil_node_open_file_low() call. What
      causes performance regression if the number of spaces is big enough.
      
      The fix is to keep opened spaces at the top of fil_system.space_list,
      and move closed files at the end of the list.
      
      For this purpose fil_space_t::space_list_last_opened pointer is
      introduced. It points to the last inserted opened space in
      fil_space_t::space_list. When space is opened, it's inserted to the
      position just after the pointer points to in fil_space_t::space_list to
      preserve the logic, inroduced in MDEV-23855. Any closed space is added
      to the end of fil_space_t::space_list.
      
      As opened spaces are located at the top of fil_space_t::space_list,
      fil_space_t::try_to_close() finds opened space faster.
      
      There can be the case when opened and closed spaces are mixed in
      fil_space_t::space_list if fil_system.freeze_space_list was set during
      fil_node_open_file_low() execution. But this should not cause any error,
      as fil_space_t::try_to_close() still iterates spaces in the list.
      
      There is no need in any test case for the fix, as it does not change any
      functionality, but just fixes performance regression.
      7d6b3d40
    • Marko Mäkelä's avatar
      Merge 10.5 into 10.6 · f169dfb4
      Marko Mäkelä authored
      f169dfb4
  15. 09 Mar, 2023 1 commit
    • Marko Mäkelä's avatar
      MDEV-30819 InnoDB fails to start up after downgrading from MariaDB 11.0 · 08267ba0
      Marko Mäkelä authored
      While downgrades are not supported and misguided attempts at it could
      cause serious corruption especially after
      commit b07920b6
      it might be useful if InnoDB would start up even after an upgrade to
      MariaDB Server 11.0 or later had removed the change buffer.
      
      innodb_change_buffering_update(): Disallow anything else than
      innodb_change_buffering=none when the change buffer is corrupted.
      
      ibuf_init_at_db_start(): Mention a possible downgrade in the corruption
      error message. If innodb_change_buffering=none, ignore the error but do
      not initialize ibuf.index.
      
      ibuf_free_excess_pages(), ibuf_contract(), ibuf_merge_space(),
      ibuf_update_max_tablespace_id(), ibuf_delete_for_discarded_space(),
      ibuf_print(): Check for !ibuf.index.
      
      ibuf_check_bitmap_on_import(): Remove some unnecessary code.
      This function is only accessing change buffer bitmap pages in a
      data file that is not attached to the rest of the database.
      It is not accessing the change buffer tree itself, hence it does
      not need any additional mutex protection.
      
      This has been tested both by starting up MariaDB Server 10.8 on
      a 11.0 data directory, and by running ./mtr --big-test while
      ibuf_init_at_db_start() was tweaked to always fail.
      08267ba0
  16. 08 Mar, 2023 1 commit
  17. 07 Mar, 2023 1 commit
  18. 06 Mar, 2023 3 commits
  19. 02 Mar, 2023 1 commit
  20. 28 Feb, 2023 2 commits