An error occurred fetching the project authors.
  1. 10 Sep, 2018 1 commit
    • Marko Mäkelä's avatar
      MDEV-17158 TRUNCATE is not atomic after MDEV-13564 · 75f8e86f
      Marko Mäkelä authored
      It turned out that ha_innobase::truncate() would prematurely
      commit the transaction already before the completion of the
      ha_innobase::create(). All of this must be atomic.
      
      innodb.truncate_crash: Use the correct DEBUG_SYNC point, and
      tolerate non-truncation of the table, because the redo log
      for the TRUNCATE transaction commit might be flushed due to
      some InnoDB background activity.
      
      dict_build_tablespace_for_table(): Merge to the function
      dict_build_table_def_step().
      
      dict_build_table_def_step(): If a table is being created during
      an already started data dictionary transaction (such as TRUNCATE),
      persistently write the table_id to the undo log header before
      creating any file. In this way, the recovery of TRUNCATE will be
      able to delete the new file before rolling back the rename of
      the original table.
      
      dict_table_rename_in_cache(): Add the parameter replace_new_file,
      used as part of rolling back a TRUNCATE operation.
      
      fil_rename_tablespace_check(): Add the parameter replace_new.
      If the parameter is set and a file identified by new_path exists,
      remove a possible tablespace and also the file.
      
      create_table_info_t::create_table_def(): Remove some debug assertions
      that no longer hold. During TRUNCATE, the transaction will already
      have been started (and performed a rename operation) before the
      table is created. Also, remove a call to dict_build_tablespace_for_table().
      
      create_table_info_t::create_table(): Add the parameter create_fk=true.
      During TRUNCATE TABLE, do not add FOREIGN KEY constraints to the
      InnoDB data dictionary, because they will also not be removed.
      
      row_table_add_foreign_constraints(): If trx=NULL, do not modify
      the InnoDB data dictionary, but only load the FOREIGN KEY constraints
      from the data dictionary.
      
      ha_innobase::create(): Lock the InnoDB data dictionary cache only
      if no transaction was passed by the caller. Unlock it in any case.
      
      innobase_rename_table(): Add the parameter commit = true.
      If !commit, do not lock or unlock the data dictionary cache.
      
      ha_innobase::truncate(): Lock the data dictionary before invoking
      rename or create, and let ha_innobase::create() unlock it and
      also commit or roll back the transaction.
      
      trx_undo_mark_as_dict(): Renamed from trx_undo_mark_as_dict_operation()
      and declared global instead of static.
      
      row_undo_ins_parse_undo_rec(): If table_id is set, this must
      be rolling back the rename operation in TRUNCATE TABLE, and
      therefore replace_new_file=true.
      75f8e86f
  2. 07 Sep, 2018 2 commits
    • Marko Mäkelä's avatar
      MDEV-14717 RENAME TABLE in InnoDB is not crash-safe · cf2a4426
      Marko Mäkelä authored
      This is a backport of commit 0bc36758
      and commit 9eb3fcc9.
      
      InnoDB in MariaDB 10.2 appears to only write MLOG_FILE_RENAME2
      redo log records during table-rebuilding ALGORITHM=INPLACE operations.
      We must write the records for any .ibd file renames, so that the
      operations are crash-safe.
      
      If InnoDB is killed during a RENAME TABLE operation, it can happen that
      the transaction for updating the data dictionary will be rolled back.
      But, nothing will roll back the renaming of the .ibd file
      (the MLOG_FILE_RENAME2 only guarantees roll-forward), or for that matter,
      the renaming of the dict_table_t::name in the dict_sys cache. We introduce
      the undo log record TRX_UNDO_RENAME_TABLE to fix this.
      
      fil_space_for_table_exists_in_mem(): Remove the parameters
      adjust_space, table_id and some code that was trying to work around
      these deficiencies.
      
      fil_name_write_rename(): Write a MLOG_FILE_RENAME2 record.
      
      dict_table_rename_in_cache(): Invoke fil_name_write_rename().
      
      trx_undo_rec_copy(): Set the first 2 bytes to the length of the
      copied undo log record.
      
      trx_undo_page_report_rename(), trx_undo_report_rename():
      Write a TRX_UNDO_RENAME_TABLE record with the old table name.
      
      row_rename_table_for_mysql(): Invoke trx_undo_report_rename()
      before modifying any data dictionary tables.
      
      row_undo_ins_parse_undo_rec(): Roll back TRX_UNDO_RENAME_TABLE
      by invoking dict_table_rename_in_cache(), which will take care
      of both renaming the table and the file.
      
      ha_innobase::truncate(): Remove a work-around.
      cf2a4426
    • Marko Mäkelä's avatar
      MDEV-13564 Mariabackup does not work with TRUNCATE · 055a3334
      Marko Mäkelä authored
      Implement undo tablespace truncation via normal redo logging.
      
      Implement TRUNCATE TABLE as a combination of RENAME to #sql-ib name,
      CREATE, and DROP.
      
      Note: Orphan #sql-ib*.ibd may be left behind if MariaDB Server 10.2
      is killed before the DROP operation is committed. If MariaDB Server 10.2
      is killed during TRUNCATE, it is also possible that the old table
      was renamed to #sql-ib*.ibd but the data dictionary will refer to the
      table using the original name.
      
      In MariaDB Server 10.3, RENAME inside InnoDB is transactional,
      and #sql-* tables will be dropped on startup. So, this new TRUNCATE
      will be fully crash-safe in 10.3.
      
      ha_mroonga::wrapper_truncate(): Pass table options to the underlying
      storage engine, now that ha_innobase::truncate() will need them.
      
      rpl_slave_state::truncate_state_table(): Before truncating
      mysql.gtid_slave_pos, evict any cached table handles from
      the table definition cache, so that there will be no stale
      references to the old table after truncating.
      
      == TRUNCATE TABLE ==
      
      WL#6501 in MySQL 5.7 introduced separate log files for implementing
      atomic and crash-safe TRUNCATE TABLE, instead of using the InnoDB
      undo and redo log. Some convoluted logic was added to the InnoDB
      crash recovery, and some extra synchronization (including a redo log
      checkpoint) was introduced to make this work. This synchronization
      has caused performance problems and race conditions, and the extra
      log files cannot be copied or applied by external backup programs.
      
      In order to support crash-upgrade from MariaDB 10.2, we will keep
      the logic for parsing and applying the extra log files, but we will
      no longer generate those files in TRUNCATE TABLE.
      
      A prerequisite for crash-safe TRUNCATE is a crash-safe RENAME TABLE
      (with full redo and undo logging and proper rollback). This will
      be implemented in MDEV-14717.
      
      ha_innobase::truncate(): Invoke RENAME, create(), delete_table().
      Because RENAME cannot be fully rolled back before MariaDB 10.3
      due to missing undo logging, add some explicit rename-back in
      case the operation fails.
      
      ha_innobase::delete(): Introduce a variant that takes sqlcom as
      a parameter. In TRUNCATE TABLE, we do not want to touch any
      FOREIGN KEY constraints.
      
      ha_innobase::create(): Add the parameters file_per_table, trx.
      In TRUNCATE, the new table must be created in the same transaction
      that renames the old table.
      
      create_table_info_t::create_table_info_t(): Add the parameters
      file_per_table, trx.
      
      row_drop_table_for_mysql(): Replace a bool parameter with sqlcom.
      
      row_drop_table_after_create_fail(): New function, wrapping
      row_drop_table_for_mysql().
      
      dict_truncate_index_tree_in_mem(), fil_truncate_tablespace(),
      fil_prepare_for_truncate(), fil_reinit_space_header_for_table(),
      row_truncate_table_for_mysql(), TruncateLogger,
      row_truncate_prepare(), row_truncate_rollback(),
      row_truncate_complete(), row_truncate_fts(),
      row_truncate_update_system_tables(),
      row_truncate_foreign_key_checks(), row_truncate_sanity_checks():
      Remove.
      
      row_upd_check_references_constraints(): Remove a check for
      TRUNCATE, now that the table is no longer truncated in place.
      
      The new test innodb.truncate_foreign uses DEBUG_SYNC to cover some
      race-condition like scenarios. The test innodb-innodb.truncate does
      not use any synchronization.
      
      We add a redo log subformat to indicate backup-friendly format.
      MariaDB 10.4 will remove support for the old TRUNCATE logging,
      so crash-upgrade from old 10.2 or 10.3 to 10.4 will involve
      limitations.
      
      == Undo tablespace truncation ==
      
      MySQL 5.7 implements undo tablespace truncation. It is only
      possible when innodb_undo_tablespaces is set to at least 2.
      The logging is implemented similar to the WL#6501 TRUNCATE,
      that is, using separate log files and a redo log checkpoint.
      
      We can simply implement undo tablespace truncation within
      a single mini-transaction that reinitializes the undo log
      tablespace file. Unfortunately, due to the redo log format
      of some operations, currently, the total redo log written by
      undo tablespace truncation will be more than the combined size
      of the truncated undo tablespace. It should be acceptable
      to have a little more than 1 megabyte of log in a single
      mini-transaction. This will be fixed in MDEV-17138 in
      MariaDB Server 10.4.
      
      recv_sys_t: Add truncated_undo_spaces[] to remember for which undo
      tablespaces a MLOG_FILE_CREATE2 record was seen.
      
      namespace undo: Remove some unnecessary declarations.
      
      fil_space_t::is_being_truncated: Document that this flag now
      only applies to undo tablespaces. Remove some references.
      
      fil_space_t::is_stopping(): Do not refer to is_being_truncated.
      This check is for tablespaces of tables. Potentially used
      tablespaces are never truncated any more.
      
      buf_dblwr_process(): Suppress the out-of-bounds warning
      for undo tablespaces.
      
      fil_truncate_log(): Write a MLOG_FILE_CREATE2 with a nonzero
      page number (new size of the tablespace in pages) to inform
      crash recovery that the undo tablespace size has been reduced.
      
      fil_op_write_log(): Relax assertions, so that MLOG_FILE_CREATE2
      can be written for undo tablespaces (without .ibd file suffix)
      for a nonzero page number.
      
      os_file_truncate(): Add the parameter allow_shrink=false
      so that undo tablespaces can actually be shrunk using this function.
      
      fil_name_parse(): For undo tablespace truncation,
      buffer MLOG_FILE_CREATE2 in truncated_undo_spaces[].
      
      recv_read_in_area(): Avoid reading pages for which no redo log
      records remain buffered, after recv_addr_trim() removed them.
      
      trx_rseg_header_create(): Add a FIXME comment that we could write
      much less redo log.
      
      trx_undo_truncate_tablespace(): Reinitialize the undo tablespace
      in a single mini-transaction, which will be flushed to the redo log
      before the file size is trimmed.
      
      recv_addr_trim(): Discard any redo logs for pages that were
      logged after the new end of a file, before the truncation LSN.
      If the rec_list becomes empty, reduce n_addrs. After removing
      any affected records, actually truncate the file.
      
      recv_apply_hashed_log_recs(): Invoke recv_addr_trim() right before
      applying any log records. The undo tablespace files must be open
      at this point.
      
      buf_flush_or_remove_pages(), buf_flush_dirty_pages(),
      buf_LRU_flush_or_remove_pages(): Add a parameter for specifying
      the number of the first page to flush or remove (default 0).
      
      trx_purge_initiate_truncate(): Remove the log checkpoints, the
      extra logging, and some unnecessary crash points. Merge the code
      from trx_undo_truncate_tablespace(). First, flush all to-be-discarded
      pages (beyond the new end of the file), then trim the space->size
      to make the page allocation deterministic. At the only remaining
      crash injection point, flush the redo log, so that the recovery
      can be tested.
      055a3334
  3. 03 Aug, 2018 1 commit
  4. 30 Jan, 2018 1 commit
    • Marko Mäkelä's avatar
      MDEV-11415 Remove excessive undo logging during ALTER TABLE…ALGORITHM=COPY · 0ba6aaf0
      Marko Mäkelä authored
      If a crash occurs during ALTER TABLE…ALGORITHM=COPY, InnoDB would spend
      a lot of time rolling back writes to the intermediate copy of the table.
      To reduce the amount of busy work done, a work-around was introduced in
      commit fd069e2b in MySQL 4.1.8 and 5.0.2,
      to commit the transaction after every 10,000 inserted rows.
      
      A proper fix would have been to disable the undo logging altogether and
      to simply drop the intermediate copy of the table on subsequent server
      startup. This is what happens in MariaDB 10.3 with MDEV-14717,MDEV-14585.
      In MariaDB 10.2, the intermediate copy of the table would be left behind
      with a name starting with the string #sql.
      
      This is a backport of a bug fix from MySQL 8.0.0 to MariaDB,
      contributed by jixianliang <271365745@qq.com>.
      
      Unlike recent MySQL, MariaDB supports ALTER IGNORE. For that operation
      InnoDB must for now keep the undo logging enabled, so that the latest
      row can be rolled back in case of an error.
      
      In Galera cluster, the LOAD DATA statement will retain the existing
      behaviour and commit the transaction after every 10,000 rows if
      the parameter wsrep_load_data_splitting=ON is set. The logic to do
      so (the wsrep_load_data_split() function and the call
      handler::extra(HA_EXTRA_FAKE_START_STMT)) are joint work
      by Ji Xianliang and Marko Mäkelä.
      
      The original fix:
      
      Author: Thirunarayanan Balathandayuthapani <thirunarayanan.balathandayuth@oracle.com>
      Date:   Wed Dec 2 16:09:15 2015 +0530
      
      Bug#17479594 AVOID INTERMEDIATE COMMIT WHILE DOING ALTER TABLE ALGORITHM=COPY
      
      Problem:
      
      During ALTER TABLE, we commit and restart the transaction for every
      10,000 rows, so that the rollback after recovery would not take so long.
      
      Fix:
      
      Suppress the undo logging during copy alter operation. If fts_index is
      present then insert directly into fts auxiliary table rather
      than doing at commit time.
      
      ha_innobase::num_write_row: Remove the variable.
      
      ha_innobase::write_row(): Remove the hack for committing every 10000 rows.
      
      row_lock_table_for_mysql(): Remove the extra 2 parameters.
      
      lock_get_src_table(), lock_is_table_exclusive(): Remove.
      Reviewed-by: default avatarMarko Mäkelä <marko.makela@oracle.com>
      Reviewed-by: default avatarShaohua Wang <shaohua.wang@oracle.com>
      Reviewed-by: default avatarJon Olav Hauglid <jon.hauglid@oracle.com>
      0ba6aaf0
  5. 09 Nov, 2017 1 commit
    • Marko Mäkelä's avatar
      MDEV-13795/MDEV-14332 Corruption during online table-rebuilding ALTER when VIRTUAL columns exist · 5d142f99
      Marko Mäkelä authored
      When MySQL 5.7 introduced indexed virtual columns, it introduced
      several bugs into the online table-rebuilding ALTER, that is,
      the row_log_table_apply() family of functions.
      
      The online_log format that was introduced for online table-rebuilding
      ALTER in MySQL 5.6 should be sufficient. Ideally, any indexed virtual
      column values would be evaluated based on the log records in the temporary
      file. There is no need to log virtual column values.
      
      (For ADD INDEX, that is row_log_apply(), we always must log the values of
      the keys, no matter if the columns are virtual.)
      
      Because omitting the virtual column values removes any chance of
      row_log_table_apply() working with indexed virtual columns, we
      will for now refuse LOCK=NONE in table-rebuilding ALTER operations
      when indexes on virtual columns exist. This restriction would be
      lifted in MDEV-14341.
      
      innobase_indexed_virtual_exist(): New predicate, to determine if
      indexed virtual columns exist in a table definition.
      
      ha_innobase::check_if_supported_inplace_alter(): Refuse online rebuild
      if indexed virtual columns exist.
      
      rec_get_converted_size_temp_v(), rec_convert_dtuple_to_temp_v(): Remove.
      
      row_log_table_delete(), row_log_table_update(, row_log_table_insert():
      Remove parameters for virtual columns.
      
      trx_undo_read_v_rows(): Remove the col_map parameter.
      
      row_log_table_apply(): Do not deal with virtual columns.
      5d142f99
  6. 03 Oct, 2017 1 commit
    • Marko Mäkelä's avatar
      Remove dict_disable_redo_if_temporary() · 770231f3
      Marko Mäkelä authored
      The function dict_disable_redo_if_temporary() was supposed to
      disable redo logging for temporary tables. It was invoked
      unnecessarily for two read-only operations:
      row_undo_search_clust_to_pcur() and
      dict_stats_update_transient_for_index().
      
      When a table is not temporary and not in the system tablespace,
      the tablespace should be flagged for MLOG_FILE_NAME logging.
      We do not need this overhead for temporary tables. Therefore,
      either mtr_t::set_log_mode() or mtr_t::set_named_space() should
      be invoked.
      
      dict_table_t::is_temporary(): Determine if a table is temporary.
      
      dict_table_is_temporary(): Redefined as a macro wrapper for
      dict_table_t::is_temporary().
      
      dict_disable_redo_if_temporary(): Remove.
      770231f3
  7. 20 Sep, 2017 1 commit
    • Marko Mäkelä's avatar
      Add the parameter bool leaf to rec_get_offsets() · 48192f96
      Marko Mäkelä authored
      This should affect debug builds only. Debug builds will check that
      the status bits of ROW_FORMAT!=REDUNDANT records match the is_leaf
      parameter.
      
      The only observable change to non-debug should be the addition of
      the is_leaf parameter to the function rec_copy_prefix_to_dtuple(),
      and the removal of some calls to update the adaptive hash index
      (it is only built for the leaf pages).
      
      This change should have been made in MySQL 5.0.3, instead of
      introducing the status flags in the ROW_FORMAT=COMPACT record header.
      48192f96
  8. 28 Aug, 2017 1 commit
    • Marko Mäkelä's avatar
      MDEV-13637 InnoDB change buffer housekeeping can cause redo log overrun and possibly deadlocks · f87cb652
      Marko Mäkelä authored
      The function ibuf_remove_free_page() may be called while the caller
      is holding several mutexes or rw-locks. Because of this, this
      housekeeping loop may cause performance glitches for operations that
      involve tables that are stored in the InnoDB system tablespace.
      Also deadlocks might be possible.
      
      The worst impact of all is that due to the mutexes being held, calls to
      log_free_check() had to be skipped during this housekeeping.
      This means that the cyclic InnoDB redo log may be overwritten.
      If the system crashes during this, it would be unable to recover.
      
      The entry point to the problematic code is ibuf_free_excess_pages().
      It would make sense to call it before acquiring any mutexes or rw-locks,
      in any 'pessimistic' operation that involves the system tablespace.
      
      fseg_create_general(), fseg_alloc_free_page_general(): Do not call
      ibuf_free_excess_pages() while potentially holding some latches.
      
      ibuf_remove_free_page(): Do call log_free_check(), like every operation
      that is about to generate redo log should do.
      
      ibuf_free_excess_pages(): Remove some assertions that are replaced
      by stricter assertions in the log_free_check() that is now called by
      ibuf_remove_free_page().
      
      row_mtr_start(): New function, to perform necessary preparations when
      starting a mini-transaction for row operations. For pessimistic operations
      on secondary indexes that are located in the system tablespace,
      this includes calling ibuf_free_excess_pages().
      
      row_undo_ins_remove_sec_low(), row_undo_mod_del_mark_or_remove_sec_low(),
      row_undo_mod_del_unmark_sec_and_undo_update(): Call row_mtr_start().
      
      row_ins_sec_index_entry(): Call ibuf_free_excess_pages() if the operation
      may involve allocating pages and change buffering in the system tablespace.
      
      row_upd_sec_index_entry(): Slightly refactor the code. The
      delete-marking of the old entry is done in-place. It could be
      change-buffered, but the old code should be unlikely to have
      invoked ibuf_free_excess_pages() in this case.
      f87cb652
  9. 25 Aug, 2017 1 commit
    • Marko Mäkelä's avatar
      MDEV-13637 InnoDB change buffer housekeeping can cause redo log overrun and possibly deadlocks · 582545a3
      Marko Mäkelä authored
      The function ibuf_remove_free_page() may be called while the caller
      is holding several mutexes or rw-locks. Because of this, this
      housekeeping loop may cause performance glitches for operations that
      involve tables that are stored in the InnoDB system tablespace.
      Also deadlocks might be possible.
      
      The worst impact of all is that due to the mutexes being held, calls to
      log_free_check() had to be skipped during this housekeeping.
      This means that the cyclic InnoDB redo log may be overwritten.
      If the system crashes during this, it would be unable to recover.
      
      The entry point to the problematic code is ibuf_free_excess_pages().
      It would make sense to call it before acquiring any mutexes or rw-locks,
      in any 'pessimistic' operation that involves the system tablespace.
      
      fseg_create_general(), fseg_alloc_free_page_general(): Do not call
      ibuf_free_excess_pages() while potentially holding some latches.
      
      ibuf_remove_free_page(): Do call log_free_check(), like every operation
      that is about to generate redo log should do.
      
      ibuf_free_excess_pages(): Remove some assertions that are replaced
      by stricter assertions in the log_free_check() that is now called by
      ibuf_remove_free_page().
      
      row_ins_sec_index_entry(), row_undo_ins_remove_sec_low(),
      row_undo_mod_del_mark_or_remove_sec_low(),
      row_undo_mod_del_unmark_sec_and_undo_update(): Call
      ibuf_free_excess_pages() if the operation may involve allocating pages
      and change buffering in the system tablespace.
      582545a3
  10. 01 Jul, 2017 1 commit
    • Marko Mäkelä's avatar
      Assert that DB_TRX_ID must be set on delete-marked records · c436338d
      Marko Mäkelä authored
      This is preparation for MDEV-12288, which would set DB_TRX_ID=0
      when purging history. Also with that change in place, delete-marked
      records must always refer to an undo log record via a nonzero
      DB_TRX_ID column. (The DB_TRX_ID is only present in clustered index
      leaf page records.)
      
      btr_cur_parse_del_mark_set_clust_rec(), rec_get_trx_id():
      Statically allocate the offsets
      (should never use the heap). Add some debug assertions.
      
      Replace some use of rec_get_trx_id() with row_get_rec_trx_id().
      
      trx_undo_report_row_operation(): Add some sanity checks that are
      common for all operations that produce undo log.
      c436338d
  11. 19 May, 2017 1 commit
    • Marko Mäkelä's avatar
      MDEV-12698 innodb.innodb_stats_del_mark test failure · 0bfa3dff
      Marko Mäkelä authored
      In my merge of the MySQL fix for Oracle Bug#23333990 / WL#9513
      I overlooked some subsequent revisions to the test, and I also
      failed to notice that the test is actually always failing.
      
      Oracle introduced the parameter innodb_stats_include_delete_marked
      but failed to consistently take it into account in FOREIGN KEY
      constraints that involve CASCADE or SET NULL.
      
      When innodb_stats_include_delete_marked=ON, obviously the purge of
      delete-marked records should update the statistics as well.
      
      One more omission was that statistics were never updated on ROLLBACK.
      We are fixing that as well, properly taking into account the
      parameter innodb_stats_include_delete_marked.
      
      dict_stats_analyze_index_level(): Simplify an expression.
      (Using the ternary operator with a constant operand is unnecessary
      obfuscation.)
      
      page_scan_method_t: Revert the change done by Oracle. Instead,
      examine srv_stats_include_delete_marked directly where it is needed.
      
      dict_stats_update_if_needed(): Renamed from
      row_update_statistics_if_needed().
      
      row_update_for_mysql_using_upd_graph(): Assert that the table statistics
      are initialized, as guaranteed by ha_innobase::open(). Update the
      statistics in a consistent way, both for FOREIGN KEY triggers and
      for the main table. If FOREIGN KEY constraints exist, do not dereference
      a freed pointer, but cache the proper value of node->is_delete so that
      it matches prebuilt->table.
      
      row_purge_record_func(): Update statistics if
      innodb_stats_include_delete_marked=ON.
      
      row_undo_ins(): Update statistics (on ROLLBACK of a fresh INSERT).
      This is independent of the parameter; the record is not delete-marked.
      
      row_undo_mod(): Update statistics on the ROLLBACK of updating key columns,
      or (if innodb_stats_include_delete_marked=OFF) updating delete-marks.
      
      innodb.innodb_stats_persistent: Renamed and extended from
      innodb.innodb_stats_del_mark. Reduced the unnecessarily large dataset
      from 262,144 to 32 rows. Test both values of the configuration
      parameter innodb_stats_include_delete_marked.
      Test that purge is updating the statistics.
      
      innodb_fts.innodb_fts_multiple_index: Adjust the result. The test
      is performing a ROLLBACK of an INSERT, which now affects the statistics.
      
      include/wait_all_purged.inc: Moved from innodb.innodb_truncate_debug
      to its own file.
      0bfa3dff
  12. 26 Apr, 2017 3 commits
    • Marko Mäkelä's avatar
      MariaDB adjustments for Oracle Bug#23070734 fix · 849af74a
      Marko Mäkelä authored
      Split the test case so that a server restart is not needed.
      Reduce the test cases and use a simpler mechanism for triggering
      and waiting for purge.
      
      fil_table_accessible(): Check if a table can be accessed without
      enjoying MDL protection.
      849af74a
    • Aditya A's avatar
      Bug #23070734 CONCURRENT TRUNCATE TABLES CAUSE STALLS · 62dca454
      Aditya A authored
      PROBLEM
      
      When truncating single tablespace tables, we need to scan the entire
      buffer pool to remove the pages of the table from the buffer pool.
      During this scan and removal dict_sys->mutex is being held ,causing
      stalls in other DDL operations.
      
      FIX
      
      Release the dict_sys->mutex during the scan and reacquire it after the
      scan. Make sure that purge thread doesn't purge the records of the table
      being truncated and background stats collection thread skips the updation
      of stats for the table being truncated.
      
      [#rb 14564 Approved by Jimmy and satya ]
      62dca454
    • Jan Lindström's avatar
      MDEV-12253: Buffer pool blocks are accessed after they have been freed · 765a4360
      Jan Lindström authored
      Problem was that bpage was referenced after it was already freed
      from LRU. Fixed by adding a new variable encrypted that is
      passed down to buf_page_check_corrupt() and used in
      buf_page_get_gen() to stop processing page read.
      
      This patch should also address following test failures and
      bugs:
      
      MDEV-12419: IMPORT should not look up tablespace in
      PageConverter::validate(). This is now removed.
      
      MDEV-10099: encryption.innodb_onlinealter_encryption fails
      sporadically in buildbot
      
      MDEV-11420: encryption.innodb_encryption-page-compression
      failed in buildbot
      
      MDEV-11222: encryption.encrypt_and_grep failed in buildbot on P8
      
      Removed dict_table_t::is_encrypted and dict_table_t::ibd_file_missing
      and replaced these with dict_table_t::file_unreadable. Table
      ibd file is missing if fil_get_space(space_id) returns NULL
      and encrypted if not. Removed dict_table_t::is_corrupted field.
      
      Ported FilSpace class from 10.2 and using that on buf_page_check_corrupt(),
      buf_page_decrypt_after_read(), buf_page_encrypt_before_write(),
      buf_dblwr_process(), buf_read_page(), dict_stats_save_defrag_stats().
      
      Added test cases when enrypted page could be read while doing
      redo log crash recovery. Also added test case for row compressed
      blobs.
      
      btr_cur_open_at_index_side_func(),
      btr_cur_open_at_rnd_pos_func(): Avoid referencing block that is
      NULL.
      
      buf_page_get_zip(): Issue error if page read fails.
      
      buf_page_get_gen(): Use dberr_t for error detection and
      do not reference bpage after we hare freed it.
      
      buf_mark_space_corrupt(): remove bpage from LRU also when
      it is encrypted.
      
      buf_page_check_corrupt(): @return DB_SUCCESS if page has
      been read and is not corrupted,
      DB_PAGE_CORRUPTED if page based on checksum check is corrupted,
      DB_DECRYPTION_FAILED if page post encryption checksum matches but
      after decryption normal page checksum does not match. In read
      case only DB_SUCCESS is possible.
      
      buf_page_io_complete(): use dberr_t for error handling.
      
      buf_flush_write_block_low(),
      buf_read_ahead_random(),
      buf_read_page_async(),
      buf_read_ahead_linear(),
      buf_read_ibuf_merge_pages(),
      buf_read_recv_pages(),
      fil_aio_wait():
              Issue error if page read fails.
      
      btr_pcur_move_to_next_page(): Do not reference page if it is
      NULL.
      
      Introduced dict_table_t::is_readable() and dict_index_t::is_readable()
      that will return true if tablespace exists and pages read from
      tablespace are not corrupted or page decryption failed.
      Removed buf_page_t::key_version. After page decryption the
      key version is not removed from page frame. For unencrypted
      pages, old key_version is removed at buf_page_encrypt_before_write()
      
      dict_stats_update_transient_for_index(),
      dict_stats_update_transient()
              Do not continue if table decryption failed or table
              is corrupted.
      
      dict0stats.cc: Introduced a dict_stats_report_error function
      to avoid code duplication.
      
      fil_parse_write_crypt_data():
              Check that key read from redo log entry is found from
              encryption plugin and if it is not, refuse to start.
      
      PageConverter::validate(): Removed access to fil_space_t as
      tablespace is not available during import.
      
      Fixed error code on innodb.innodb test.
      
      Merged test cased innodb-bad-key-change5 and innodb-bad-key-shutdown
      to innodb-bad-key-change2.  Removed innodb-bad-key-change5 test.
      Decreased unnecessary complexity on some long lasting tests.
      
      Removed fil_inc_pending_ops(), fil_decr_pending_ops(),
      fil_get_first_space(), fil_get_next_space(),
      fil_get_first_space_safe(), fil_get_next_space_safe()
      functions.
      
      fil_space_verify_crypt_checksum(): Fixed bug found using ASAN
      where FIL_PAGE_END_LSN_OLD_CHECKSUM field was incorrectly
      accessed from row compressed tables. Fixed out of page frame
      bug for row compressed tables in
      fil_space_verify_crypt_checksum() found using ASAN. Incorrect
      function was called for compressed table.
      
      Added new tests for discard, rename table and drop (we should allow them
      even when page decryption fails). Alter table rename is not allowed.
      Added test for restart with innodb-force-recovery=1 when page read on
      redo-recovery cant be decrypted. Added test for corrupted table where
      both page data and FIL_PAGE_FILE_FLUSH_LSN_OR_KEY_VERSION is corrupted.
      
      Adjusted the test case innodb_bug14147491 so that it does not anymore
      expect crash. Instead table is just mostly not usable.
      
      fil0fil.h: fil_space_acquire_low is not visible function
      and fil_space_acquire and fil_space_acquire_silent are
      inline functions. FilSpace class uses fil_space_acquire_low
      directly.
      
      recv_apply_hashed_log_recs() does not return anything.
      765a4360
  13. 17 Mar, 2017 1 commit
    • Marko Mäkelä's avatar
      MDEV-12271 Port MySQL 8.0 Bug#23150562 REMOVE UNIV_MUST_NOT_INLINE AND UNIV_NONINL · 4e1116b2
      Marko Mäkelä authored
      Also, remove empty .ic files that were not removed by my MySQL commit.
      
      Problem:
      InnoDB used to support a compilation mode that allowed to choose
      whether the function definitions in .ic files are to be inlined or not.
      This stopped making sense when InnoDB moved to C++ in MySQL 5.6
      (and ha_innodb.cc started to #include .ic files), and more so in
      MySQL 5.7 when inline methods and functions were introduced
      in .h files.
      
      Solution:
      Remove all references to UNIV_NONINL and UNIV_MUST_NOT_INLINE from
      all files, assuming that the symbols are never defined.
      Remove the files fut0fut.cc and ut0byte.cc which only mattered when
      UNIV_NONINL was defined.
      4e1116b2
  14. 08 Sep, 2016 1 commit
    • Jan Lindström's avatar
      Merge InnoDB 5.7 from mysql-5.7.14. · fec844ac
      Jan Lindström authored
      Contains also:
             MDEV-10549 mysqld: sql/handler.cc:2692: int handler::ha_index_first(uchar*): Assertion `table_share->tmp_table != NO_TMP_TABLE || m_lock_type != 2' failed. (branch bb-10.2-jan)
             Unlike MySQL, InnoDB still uses THR_LOCK in MariaDB
      
             MDEV-10548 Some of the debug sync waits do not work with InnoDB 5.7 (branch bb-10.2-jan)
             enable tests that were fixed in MDEV-10549
      
             MDEV-10548 Some of the debug sync waits do not work with InnoDB 5.7 (branch bb-10.2-jan)
             fix main.innodb_mysql_sync - re-enable online alter for partitioned innodb tables
      fec844ac
  15. 02 Sep, 2016 1 commit
    • Jan Lindström's avatar
      Merge InnoDB 5.7 from mysql-5.7.9. · 2e814d47
      Jan Lindström authored
      Contains also
      
      MDEV-10547: Test multi_update_innodb fails with InnoDB 5.7
      
      	The failure happened because 5.7 has changed the signature of
      	the bool handler::primary_key_is_clustered() const
      	virtual function ("const" was added). InnoDB was using the old
      	signature which caused the function not to be used.
      
      MDEV-10550: Parallel replication lock waits/deadlock handling does not work with InnoDB 5.7
      
      	Fixed mutexing problem on lock_trx_handle_wait. Note that
      	rpl_parallel and rpl_optimistic_parallel tests still
      	fail.
      
      MDEV-10156 : Group commit tests fail on 10.2 InnoDB (branch bb-10.2-jan)
        Reason: incorrect merge
      
      MDEV-10550: Parallel replication can't sync with master in InnoDB 5.7 (branch bb-10.2-jan)
        Reason: incorrect merge
      2e814d47
  16. 21 Jun, 2016 1 commit
  17. 04 May, 2015 2 commits
  18. 06 May, 2014 2 commits
  19. 26 Feb, 2014 1 commit
  20. 16 Dec, 2013 2 commits
  21. 07 Nov, 2013 1 commit
  22. 07 Sep, 2013 1 commit
  23. 14 Dec, 2011 1 commit
  24. 14 Jul, 2011 1 commit
  25. 03 Sep, 2010 1 commit
  26. 12 Apr, 2010 2 commits
  27. 07 Apr, 2010 2 commits
  28. 30 Mar, 2010 2 commits
  29. 26 Mar, 2010 2 commits
    • vdimov's avatar
      Non-functional change: update copyright year to 2010 of the files · a5b61a20
      vdimov authored
      that have been modified after 2010-01-01 according to svn.
      
      for f in $(svn log -v -r{2010-01-01}:HEAD |grep "^   M " |cut -b 16- |sort -u) ; do sed -i "" -E 's/(Copyright \(c\) [0-9]{4},) [0-9]{4}, (.*Innobase Oy.+All Rights Reserved)/\1 2010, \2/' $f ; done
      a5b61a20
    • vdimov's avatar
      Non-functional change: update copyright year to 2010 of the files · 9d214d6f
      vdimov authored
      that have been modified after 2010-01-01 according to svn.
      
      for f in $(svn log -v -r{2010-01-01}:HEAD |grep "^   M " |cut -b 16- |sort -u) ; do sed -i "" -E 's/(Copyright \(c\) [0-9]{4},) [0-9]{4}, (.*Innobase Oy.+All Rights Reserved)/\1 2010, \2/' $f ; done
      9d214d6f
  30. 17 Mar, 2010 1 commit
    • marko's avatar
      branches/innodb+: btr_cur_search_to_nth_level(): Treat BTR_DELETE in · c158c47a
      marko authored
      the same way as BTR_INSERT and BTR_DELETE_MARK: only perform (buffer)
      the operation when the page is not in the buffer pool.
      
      BTR_INSERT, BTR_DELETE_MARK, BTR_DELETE: Make the documentation say that
      these flags are ignored when the page is in the buffer pool.
      
      enum row_search_result: Remove ROW_NOT_DELETED, which
      BTR_CUR_DELETE_FAILED was mapped to.
      
      enum btr_cur_method: Remove BTR_CUR_DELETE_FAILED.  The
      btr_cur_search_to_nth_level() will no longer attempt to execute the
      BTR_DELETE when the page is in the buffer pool.
      
      row_search_index_entry(): Remove the mapping from
      BTR_CUR_DELETE_FAILED to ROW_NOT_DELETED.  The caller will have to
      attempt purge when the record is in the buffer pool.
      
      row_purge_remove_sec_if_poss_leaf(): Attempt to purge the record if it
      was found in the buffer pool.
      
      This addresses Issue #466.
      rb://268
      c158c47a