1. 16 Mar, 2023 1 commit
    • Marko Mäkelä's avatar
      MDEV-30860 Race condition between buffer pool flush and log file deletion in... · 1495f057
      Marko Mäkelä authored
      MDEV-30860 Race condition between buffer pool flush and log file deletion in mariadb-backup --prepare
      
      srv_start(): If we are going to close the log file in
      mariadb-backup --prepare, call buf_flush_sync() before
      calling recv_sys.debug_free() to ensure that the log file
      will not be accessed.
      
      This fixes a rather rare failure in the test
      mariabackup.innodb_force_recovery where buf_flush_page_cleaner()
      would invoke log_checkpoint_low() because !recv_recovery_is_on()
      would hold due to the fact that recv_sys.debug_free() had
      already been called. Then, the log write for the checkpoint
      would fail because srv_start() had invoked log_sys.log.close_file().
      1495f057
  2. 10 Mar, 2023 1 commit
    • Vlad Lesin's avatar
      MDEV-30775 Performance regression in fil_space_t::try_to_close() introduced in MDEV-23855 · 7d6b3d40
      Vlad Lesin authored
      fil_node_open_file_low() tries to close files from the top of
      fil_system.space_list if the number of opened files is exceeded.
      
      It invokes fil_space_t::try_to_close(), which iterates the list searching
      for the first opened space. Then it just closes the space, leaving it in
      the same position in fil_system.space_list.
      
      On heavy files opening, like during 'SHOW TABLE STATUS ...' execution,
      if the number of opened files limit is reached,
      fil_space_t::try_to_close() iterates more and more closed spaces before
      reaching any opened space for each fil_node_open_file_low() call. What
      causes performance regression if the number of spaces is big enough.
      
      The fix is to keep opened spaces at the top of fil_system.space_list,
      and move closed files at the end of the list.
      
      For this purpose fil_space_t::space_list_last_opened pointer is
      introduced. It points to the last inserted opened space in
      fil_space_t::space_list. When space is opened, it's inserted to the
      position just after the pointer points to in fil_space_t::space_list to
      preserve the logic, inroduced in MDEV-23855. Any closed space is added
      to the end of fil_space_t::space_list.
      
      As opened spaces are located at the top of fil_space_t::space_list,
      fil_space_t::try_to_close() finds opened space faster.
      
      There can be the case when opened and closed spaces are mixed in
      fil_space_t::space_list if fil_system.freeze_space_list was set during
      fil_node_open_file_low() execution. But this should not cause any error,
      as fil_space_t::try_to_close() still iterates spaces in the list.
      
      There is no need in any test case for the fix, as it does not change any
      functionality, but just fixes performance regression.
      7d6b3d40
  3. 09 Mar, 2023 1 commit
    • Marko Mäkelä's avatar
      MDEV-30819 InnoDB fails to start up after downgrading from MariaDB 11.0 · 08267ba0
      Marko Mäkelä authored
      While downgrades are not supported and misguided attempts at it could
      cause serious corruption especially after
      commit b07920b6
      it might be useful if InnoDB would start up even after an upgrade to
      MariaDB Server 11.0 or later had removed the change buffer.
      
      innodb_change_buffering_update(): Disallow anything else than
      innodb_change_buffering=none when the change buffer is corrupted.
      
      ibuf_init_at_db_start(): Mention a possible downgrade in the corruption
      error message. If innodb_change_buffering=none, ignore the error but do
      not initialize ibuf.index.
      
      ibuf_free_excess_pages(), ibuf_contract(), ibuf_merge_space(),
      ibuf_update_max_tablespace_id(), ibuf_delete_for_discarded_space(),
      ibuf_print(): Check for !ibuf.index.
      
      ibuf_check_bitmap_on_import(): Remove some unnecessary code.
      This function is only accessing change buffer bitmap pages in a
      data file that is not attached to the rest of the database.
      It is not accessing the change buffer tree itself, hence it does
      not need any additional mutex protection.
      
      This has been tested both by starting up MariaDB Server 10.8 on
      a 11.0 data directory, and by running ./mtr --big-test while
      ibuf_init_at_db_start() was tweaked to always fail.
      08267ba0
  4. 08 Mar, 2023 1 commit
  5. 07 Mar, 2023 1 commit
  6. 06 Mar, 2023 2 commits
  7. 28 Feb, 2023 2 commits
    • Sergei Golubchik's avatar
    • Marko Mäkelä's avatar
      MDEV-30753 Possible corruption due to trx_purge_free_segment() · c14a3943
      Marko Mäkelä authored
      Starting with commit 0de3be8c (MDEV-30671),
      the field TRX_UNDO_NEEDS_PURGE lost its previous meaning.
      The following scenario is possible:
      
      (1) InnoDB is killed at a point of time corresponding to the durable
      execution of some fseg_free_step_not_header() but not
      trx_purge_remove_log_hdr().
      (2) After restart, the affected pages are allocated for something else.
      (3) Purge will attempt to access the newly reallocated pages when looking
      for some old undo log records.
      
      trx_purge_free_segment(): Invoke trx_purge_remove_log_hdr() as the first
      thing, to be safe. If the server is killed, some pages will never be
      freed. That is the lesser evil. Also, before each mtr.start(), invoke
      log_free_check() to prevent ib_logfile0 overrun.
      c14a3943
  8. 27 Feb, 2023 1 commit
    • Monty's avatar
      Added detection of memory overwrite with multi_malloc · 57c526ff
      Monty authored
      This patch also fixes some bugs detected by valgrind after this
      patch:
      
      - Not enough copy_func elements was allocated by Create_tmp_table() which
        causes an memory overwrite in Create_tmp_table::add_fields()
        I added an ASSERT() to be able to detect this also without valgrind.
        The bug was that TMP_TABLE_PARAM::copy_fields was not correctly set
        when calling create_tmp_table().
      - Aria::empty_bits is not allocated if there is no varchar/char/blob
        fields in the table.  Fixed code to take this into account.
        This cannot cause any issues as this is just a memory access
        into other Aria memory and the content of the memory would not be used.
      - Aria::last_key_buff was not allocated big enough. This may have caused
        issues with rtrees and ma_extra(HA_EXTRA_REMEMBER_POS) as they
        would use the same memory area.
      - Aria and MyISAM didn't take extended key parts into account, which
        caused problems when copying rec_per_key from engine to sql level.
      - Mark asan builds with 'asan' in version strihng to detect these in
        not_valgrind_build.inc.
        This is needed to not have main.sp-no-valgrind fail with asan.
      57c526ff
  9. 24 Feb, 2023 1 commit
    • Marko Mäkelä's avatar
      MDEV-30671 InnoDB undo log truncation fails to wait for purge of history · 0de3be8c
      Marko Mäkelä authored
      It is not safe to invoke trx_purge_free_segment() or execute
      innodb_undo_log_truncate=ON before all undo log records in
      the rollback segment has been processed.
      
      A prominent failure that would occur due to premature freeing of
      undo log pages is that trx_undo_get_undo_rec() would crash when
      trying to copy an undo log record to fetch the previous version
      of a record.
      
      If trx_undo_get_undo_rec() was not invoked in the unlucky time frame,
      then the symptom would be that some committed transaction history is
      never removed. This would be detected by CHECK TABLE...EXTENDED that
      was impleented in commit ab019010.
      Such a garbage collection leak should be possible even when using
      innodb_undo_log_truncate=OFF, just involving trx_purge_free_segment().
      
      trx_rseg_t::needs_purge: Change the type from Boolean to a transaction
      identifier, noting the most recent non-purged transaction, or 0 if
      everything has been purged. On transaction start, we initialize this
      to 1 more than the transaction start ID. On recovery, the field may be
      adjusted to the transaction end ID (TRX_UNDO_TRX_NO) if it is larger.
      
      The field TRX_UNDO_NEEDS_PURGE becomes write-only; only some debug
      assertions that would validate the value. The field reflects the old
      inaccurate Boolean field trx_rseg_t::needs_purge.
      
      trx_undo_mem_create_at_db_start(), trx_undo_lists_init(),
      trx_rseg_mem_restore(): Remove the parameter max_trx_id.
      Instead, store the maximum in trx_rseg_t::needs_purge,
      where trx_rseg_array_init() will find it.
      
      trx_purge_free_segment(): Contiguously hold a lock on
      trx_rseg_t to prevent any concurrent allocation of undo log.
      
      trx_purge_truncate_rseg_history(): Only invoke trx_purge_free_segment()
      if the rollback segment is empty and there are no pending transactions
      associated with it.
      
      trx_purge_truncate_history(): Only proceed with innodb_undo_log_truncate=ON
      if trx_rseg_t::needs_purge indicates that all history has been purged.
      
      Tested by: Matthias Leich
      0de3be8c
  10. 16 Feb, 2023 2 commits
  11. 15 Feb, 2023 2 commits
    • Marko Mäkelä's avatar
      MDEV-30657 InnoDB: Not applying UNDO_APPEND due to corruption · 5300c0fb
      Marko Mäkelä authored
      This almost completely reverts
      commit acd23da4 and
      retains a safe optimization:
      
      recv_sys_t::parse(): Remove any old redo log records for the
      truncated tablespace, to free up memory earlier.
      If recovery consists of multiple batches, then recv_sys_t::apply()
      will must invoke recv_sys_t::trim() again to avoid wrongly
      applying old log records to an already truncated undo tablespace.
      5300c0fb
    • Monty's avatar
      MDEV-30333 Wrong result with not_null_range_scan and LEFT JOIN with empty table · 192427e3
      Monty authored
      There was a bug in JOIN::make_notnull_conds_for_range_scans() when
      clearing TABLE->tmp_set, which was used to mark fields that could not be
      null.
      
      This function was only used if 'not_null_range_scan=on' is set.
      
      The effect was that tmp_set contained a 'random value' and this caused
      the optimizer to think that some fields could not be null.
      FLUSH TABLES clears tmp_set and because of this things worked temporarily.
      
      Fixed by clearing tmp_set properly.
      192427e3
  12. 14 Feb, 2023 4 commits
  13. 10 Feb, 2023 2 commits
  14. 09 Feb, 2023 5 commits
    • Brandon Nesterenko's avatar
      MDEV-30608: rpl.rpl_delayed_parallel_slave_sbm sometimes fails with... · eecd4f14
      Brandon Nesterenko authored
      MDEV-30608: rpl.rpl_delayed_parallel_slave_sbm sometimes fails with Seconds_Behind_Master should not have used second transaction timestamp
      
      One of the constraints added in the MDEV-29639 patch, is that only
      the first event after idling should update last_master_timestamp;
      and as long as the replica has more events to execute, the variable
      should not be updated. The corresponding test,
      rpl_delayed_parallel_slave_sbm.test, aims to verify this; however,
      if the IO thread takes too long to queue events, the SQL thread can
      appear to catch up too fast.
      
      This fix ensures that the relay log has been fully written before
      executing the events.
      
      Note that the underlying cause of this test failure needs to be
      addressed as a bug-fix, this is a temporary fix to stop test
      failures. To track work on the bug-fix for the underlying issue,
      please see MDEV-30619.
      eecd4f14
    • Igor Babaev's avatar
      MDEV-30586 DELETE with aggregation in subquery of WHERE returns bogus error · c6376842
      Igor Babaev authored
      The parser code for single-table DELETE missed the call of the function
      LEX::check_main_unit_semantics(). As a result the the field nested level
      of SELECT_LEX structures remained set 0 for all non-top level selects.
      This could lead to different kind of problems. In particular this did not
      allow to determine properly the selects where set functions had to be
      aggregated when they were used in inner subqueries.
      
      Approved by Oleksandr Byelkin <sanja@mariadb.com>
      c6376842
    • Vicențiu Ciorbaru's avatar
      Apply clang-tidy to remove empty constructors / destructors · 08c85202
      Vicențiu Ciorbaru authored
      This patch is the result of running
      run-clang-tidy -fix -header-filter=.* -checks='-*,modernize-use-equals-default' .
      
      Code style changes have been done on top. The result of this change
      leads to the following improvements:
      
      1. Binary size reduction.
      * For a -DBUILD_CONFIG=mysql_release build, the binary size is reduced by
        ~400kb.
      * A raw -DCMAKE_BUILD_TYPE=Release reduces the binary size by ~1.4kb.
      
      2. Compiler can better understand the intent of the code, thus it leads
         to more optimization possibilities. Additionally it enabled detecting
         unused variables that had an empty default constructor but not marked
         so explicitly.
      
         Particular change required following this patch in sql/opt_range.cc
      
         result_keys, an unused template class Bitmap now correctly issues
         unused variable warnings.
      
         Setting Bitmap template class constructor to default allows the compiler
         to identify that there are no side-effects when instantiating the class.
         Previously the compiler could not issue the warning as it assumed Bitmap
         class (being a template) would not be performing a NO-OP for its default
         constructor. This prevented the "unused variable warning".
      08c85202
    • Vladislav Vaintroub's avatar
      MDEV-30624 HeidiSQL 12.3 · 8dab6614
      Vladislav Vaintroub authored
      8dab6614
    • Vladislav Vaintroub's avatar
  15. 08 Feb, 2023 3 commits
  16. 07 Feb, 2023 4 commits
    • Daniel Black's avatar
      MDEV-30572: my_large_malloc will only retry on ENOMEM · 2b494ccc
      Daniel Black authored
      Correct error in to only say "continuing to smaller size" if it really
      is.
      2b494ccc
    • Daniel Black's avatar
      MDEV-30554 RockDB libatomic linking on riscv64 · 17423c6c
      Daniel Black authored
      The existing storage/rocksdb/CMakeCache.txt defined
      ATOMIC_EXTRA_LIBS when atomics where required. This was
      determined by the toplevel configure.cmake test
      (HAVE_GCC_C11_ATOMICS_WITH_LIBATOMIC).
      
      As build_rocksdb.cmake is included after ATOMIC_EXTRA_LIBS
      was set, we just need to use it. As such no riscv64
      specific macro is needed in build_rocksdb.cmake.
      
      As highlighted by Gianfranco Costamagna (@LocutusOfBorg)
      in #2472 overwriting SYSTEM_LIBS was problematic.
      This is corrected in case in future SYSTEM_LIBS is changed
      elsewhere.
      
      Closes #2472.
      17423c6c
    • Daniel Black's avatar
      MDEV-30492 Crash when use mariabackup.exe with config 'innodb_flush_method=async_unbuffered' · ecc93c98
      Daniel Black authored
      Normalize innodb_flush_method, the same as the service, before
      attempting to print it.
      ecc93c98
    • Daniel Black's avatar
      MDEV-30558: ER_KILL_{,QUERY_}DENIED_ERROR - normalize id type · 762fe015
      Daniel Black authored
      The error string from ER_KILL_QUERY_DENIED_ERROR took a different
      type to ER_KILL_DENIED_ERROR for the thread id. This shows
      up in differences on 32 big endian arches like powerpc (Deb notation).
      
      Normalize the passing of the THD->id to its real type of my_thread_id,
      and cast to (long long) on output. As such normalize the
      ER_KILL_QUERY_DENIED_ERROR to that convention too.
      
      Note for upwards merge, convert the type to %lld on new translations
      of ER_KILL_QUERY_DENIED_ERROR.
      762fe015
  17. 06 Feb, 2023 7 commits