1. 03 Jan, 2017 1 commit
    • Marko Mäkelä's avatar
      Post-fix for MDEV-11195 NUMA does not get enabled even when checks are passed · 4c610d10
      Marko Mäkelä authored
      The C preprocessor symbol WITH_NUMA is never defined. Instead, the symbol
      HAVE_LIBNUMA is used for checking if the feature is to be used.
      
      If cmake -DWITH_NUMA=OFF is specified, HAVE_LIBNUMA will not be defined
      at compilation time even if the library is available.
      
      If cmake -DWITH_NUMA=ON is specified but the library is not available
      at configuration time, the compilation will be aborted.
      4c610d10
  2. 01 Jan, 2017 1 commit
  3. 30 Dec, 2016 10 commits
    • Marko Mäkelä's avatar
      MDEV-11687 innodb_use_fallocate has no effect · b727213d
      Marko Mäkelä authored
      Deprecate the variable in MariaDB 10.2, saying
      that it will be removed in 10.3.
      b727213d
    • Marko Mäkelä's avatar
      MDEV-11690 Remove UNIV_HOTBACKUP · 63574f12
      Marko Mäkelä authored
      The InnoDB source code contains quite a few references to a closed-source
      hot backup tool which was originally called InnoDB Hot Backup (ibbackup)
      and later incorporated in MySQL Enterprise Backup.
      
      The open source backup tool XtraBackup uses the full database for recovery.
      So, the references to UNIV_HOTBACKUP are only cluttering the source code.
      63574f12
    • Marko Mäkelä's avatar
      Merge 10.1 into 10.2 · 9ebd7673
      Marko Mäkelä authored
      9ebd7673
    • Marko Mäkelä's avatar
      MDEV-11687 innodb_use_fallocate has no effect · 1ab3866d
      Marko Mäkelä authored
      The configuration parameter innodb_use_fallocate, which is mapped to
      the variable srv_use_posix_fallocate, has no effect in MariaDB 10.2.2
      or MariaDB 10.2.3.
      Thus the configuration parameter and the variable should be removed.
      1ab3866d
    • Marko Mäkelä's avatar
      Remove dead references to NO_FALLOCATE. · d4342702
      Marko Mäkelä authored
      d4342702
    • Marko Mäkelä's avatar
      cbf80b0d
    • Marko Mäkelä's avatar
      MDEV-11556 InnoDB redo log apply fails to adjust data file sizes · 8451e090
      Marko Mäkelä authored
      fil_space_t::recv_size: New member: recovered tablespace size in pages;
      0 if no size change was read from the redo log,
      or if the size change was implemented.
      
      fil_space_set_recv_size(): New function for setting space->recv_size.
      
      innodb_data_file_size_debug: A debug parameter for setting the system
      tablespace size in recovery even when the redo log does not contain
      any size changes. It is hard to write a small test case that would
      cause the system tablespace to be extended at the critical moment.
      
      recv_parse_log_rec(): Note those tablespaces whose size is being changed
      by the redo log, by invoking fil_space_set_recv_size().
      
      innobase_init(): Correct an error message, and do not require a larger
      innodb_buffer_pool_size when starting up with a smaller innodb_page_size.
      
      innobase_start_or_create_for_mysql(): Allow startup with any initial
      size of the ibdata1 file if the autoextend attribute is set. Require
      the minimum size of fixed-size system tablespaces to be 640 pages,
      not 10 megabytes. Implement innodb_data_file_size_debug.
      
      open_or_create_data_files(): Round the system tablespace size down
      to pages, not to full megabytes, (Our test truncates the system
      tablespace to more than 800 pages with innodb_page_size=4k.
      InnoDB should not imagine that it was truncated to 768 pages
      and then overwrite good pages in the tablespace.)
      
      fil_flush_low(): Refactored from fil_flush().
      
      fil_space_extend_must_retry(): Refactored from
      fil_extend_space_to_desired_size().
      
      fil_mutex_enter_and_prepare_for_io(): Extend the tablespace if
      fil_space_set_recv_size() was called.
      
      The test case has been successfully run with all the
      innodb_page_size values 4k, 8k, 16k, 32k, 64k.
      8451e090
    • Marko Mäkelä's avatar
      f493e395
    • Marko Mäkelä's avatar
      Merge 10.1 into 10.2 · 970f17cb
      Marko Mäkelä authored
      970f17cb
    • Marko Mäkelä's avatar
      Merge 10.1 into 10.2 · 341c375d
      Marko Mäkelä authored
      341c375d
  4. 29 Dec, 2016 7 commits
    • Marko Mäkelä's avatar
      MDEV-11679 Remove redundant function fsp_header_get_crypt_offset() · f2fe6510
      Marko Mäkelä authored
      fsp_header_get_crypt_offset(): Remove.
      
      xdes_arr_size(): Remove.
      
      fsp_header_get_encryption_offset(): Make this an inline function.
      
      The correctness of this change was ensured with the following patch
      that ensures that the two functions returned the same value, only
      differing by FSP_HEADER_OFFSET (38 bytes):
      
      diff --git a/storage/innobase/fsp/fsp0fsp.cc b/storage/innobase/fsp/fsp0fsp.cc
      index f2a4c6bf218..e96c788b7df 100644
      --- a/storage/innobase/fsp/fsp0fsp.cc
      +++ b/storage/innobase/fsp/fsp0fsp.cc
      @@ -850,6 +850,7 @@ fsp_parse_init_file_page(
       	return(ptr);
       }
      
      +static ulint fsp_header_get_encryption_offset(const page_size_t&);
       /**********************************************************************//**
       Initializes the fsp system. */
       void
      @@ -868,6 +869,31 @@ fsp_init(void)
       #endif
      
       	/* Does nothing at the moment */
      +
      +	for (ulint sz = 4096; sz <= 65536; sz *= 2) {
      +		ulint m;
      +		if (sz <= 16384) {
      +			for (ulint ph = 1024; ph <= sz; ph *= 2) {
      +				const page_size_t ps(ph, sz, true);
      +				ulint maria = fsp_header_get_crypt_offset(ps, &m),
      +					oracle = fsp_header_get_encryption_offset(ps);
      +				if (maria != oracle + 38) {
      +					ib::error() << "zip size mismatch: "
      +						    << maria << "!=" << oracle
      +						    << "(" << ph <<","<<sz<<")"
      +						    << m;
      +				}
      +			}
      +		}
      +		const page_size_t p(sz, sz, false);
      +		ulint maria = fsp_header_get_crypt_offset(p, &m),
      +			oracle = fsp_header_get_encryption_offset(p);
      +		if (maria != oracle + 38) {
      +			ib::error() << "size mismatch: "
      +				    << maria << "!=" << oracle
      +				    << "(" <<sz<<")" << m;
      +		}
      +	}
       }
      
       /**********************************************************************//**
      f2fe6510
    • Marko Mäkelä's avatar
      7bcae22b
    • Sergei Golubchik's avatar
      enable tests that were skipped because of have_xtradb · ab89359d
      Sergei Golubchik authored
      * some of these tests run just fine with InnoDB:
         -> s/have_xtradb/have_innodb/
      * sys_var tests did basic tests for xtradb only variables
         -> remove them, they're useless anyway (sysvar_innodb does it better)
      * multi_update had innodb specific tests
         -> move to multi_update_innodb.test
      ab89359d
    • Sergei Golubchik's avatar
      cleanup: binlog.binlog_killed_simulate · b3d6cbc2
      Sergei Golubchik authored
      b3d6cbc2
    • Sergei Golubchik's avatar
      cleanup: binlog.binlog_row_annotate · 0d897c2c
      Sergei Golubchik authored
      0d897c2c
    • Sergei Golubchik's avatar
      Merge branch '10.1' into 10.2 · 4a5d25c3
      Sergei Golubchik authored
      4a5d25c3
    • Sergei Golubchik's avatar
      cleanup: redundant memcmp() · 48dc7cc6
      Sergei Golubchik authored
      48dc7cc6
  5. 28 Dec, 2016 4 commits
    • Oleksandr Byelkin's avatar
      MDEV-11584: GRANT inside an SP does not work well on 2nd execution · 23cc1be2
      Oleksandr Byelkin authored
      Allocate password hash in statment memory
      23cc1be2
    • Oleksandr Byelkin's avatar
      MDEV-11584: GRANT inside an SP does not work well on 2nd execution · 100f721c
      Oleksandr Byelkin authored
      Allocate password hash in statment memory
      100f721c
    • Jan Lindström's avatar
      MDEV-11656: 'Data structure corruption' IMPORT TABLESPACE doesn't work for... · 283e9cf4
      Jan Lindström authored
      MDEV-11656: 'Data structure corruption' IMPORT TABLESPACE doesn't work for encrypted InnoDB tables if space_id changed
      
      Problem was that for encryption we use temporary scratch area for
      reading and writing tablespace pages. But if page was not really
      decrypted the correct updated page was not moved to scratch area
      that was then written. This can happen e.g. for page 0 as it is
      newer encrypted even if encryption is enabled and as we write
      the contents of old page 0 to tablespace it contained naturally
      incorrect space_id that is then later noted and error message
      was written. Updated page with correct space_id was lost.
      
      If tablespace is encrypted we use additional
      temporary scratch area where pages are read
      for decrypting readptr == crypt_io_buffer != io_buffer.
      
      Destination for decryption is a buffer pool block
      block->frame == dst == io_buffer that is updated.
      Pages that did not require decryption even when
      tablespace is marked as encrypted are not copied
      instead block->frame is set to src == readptr.
      
      If tablespace was encrypted we copy updated page to
      writeptr != io_buffer. This fixes above bug.
      
      For encryption we again use temporary scratch area
      writeptr != io_buffer == dst
      that is then written to the tablespace
      
      (1) For normal tables src == dst ==  writeptr
      ut_ad(!encrypted && !page_compressed ?
      	src == dst && dst == writeptr + (i * size):1);
      (2) For page compressed tables src == dst == writeptr
      ut_ad(page_compressed && !encrypted ?
      	src == dst && dst == writeptr + (i * size):1);
      (3) For encrypted tables src != dst != writeptr
      ut_ad(encrypted ?
      	src != dst && dst != writeptr + (i * size):1);
      283e9cf4
    • Marko Mäkelä's avatar
      MDEV-9282 Debian: the Lintian complains about "shlib-calls-exit" in ha_innodb.so · d50cf42b
      Marko Mäkelä authored
      Replace all exit() calls in InnoDB with abort() [possibly via ut_a()].
      Calling exit() in a multi-threaded program is problematic also for
      the reason that other threads could see corrupted data structures
      while some data structures are being cleaned up by atexit() handlers
      or similar.
      
      In the long term, all these calls should be replaced with something
      that returns an error all the way up the call stack.
      d50cf42b
  6. 27 Dec, 2016 1 commit
  7. 24 Dec, 2016 4 commits
  8. 23 Dec, 2016 1 commit
    • Marko Mäkelä's avatar
      Follow-up for MDEV-11630 Call mutex_free() before freeing the mutex list · bbb3fb31
      Marko Mäkelä authored
      fil_tablespace_iterate(): Call fil_space_destroy_crypt_data() to
      invoke mutex_free() for the mutex_create() that was done in
      fil_space_read_crypt_data(). Also, remember to free
      iter.crypt_io_buffer.
      
      The failure to call mutex_free() would cause sync_latch_meta_destroy()
      to access freed memory on shutdown. This affected the IMPORT of
      encrypted tablespaces.
      bbb3fb31
  9. 22 Dec, 2016 7 commits
    • Marko Mäkelä's avatar
      MDEV-11635 innodb.innodb_mysql test hangs · 08f79bde
      Marko Mäkelä authored
      Copy and adapt the test from MySQL 5.7.17.
      08f79bde
    • Marko Mäkelä's avatar
      MDEV-11630 Call mutex_free() before freeing the mutex list · d6a1f9f1
      Marko Mäkelä authored
      fil_space_crypt_cleanup(): Call mutex_free() to pair with
      fil_space_crypt_init().
      
      fil_space_destroy_crypt_data(): Call mutex_free() to pair with
      fil_space_create_crypt_data() and fil_space_read_crypt_data().
      
      fil_crypt_threads_cleanup(): Call mutex_free() to pair with
      fil_crypt_threads_init().
      
      fil_space_free_low(): Invoke fil_space_destroy_crypt_data().
      
      fil_close(): Invoke fil_space_crypt_cleanup(), just like
      fil_init() invoked fil_space_crypt_init().
      
      Datafile::shutdown(): Set m_crypt_info=NULL without dereferencing
      the pointer. The object will be freed along with the fil_space_t
      in fil_space_free_low().
      Remove some unnecessary conditions (ut_free(NULL) is OK).
      
      srv_shutdown_all_bg_threads(): Shut down the encryption threads
      by calling fil_crypt_threads_end().
      
      srv_shutdown_bg_undo_sources(): Do not prematurely call
      fil_crypt_threads_end(). Many pages can still be written by
      change buffer merge, rollback of incomplete transactions, and
      purge, especially in slow shutdown (innodb_fast_shutdown=0).
      
      innobase_shutdown_for_mysql(): Call fil_crypt_threads_cleanup()
      also when innodb_read_only=1, because the threads will have been
      created also in that case.
      
      sync_check_close(): Re-enable the invocation of sync_latch_meta_destroy()
      to free the mutex list.
      d6a1f9f1
    • Marko Mäkelä's avatar
      Remove an unnecessary comparison. · 545c9126
      Marko Mäkelä authored
      545c9126
    • Marko Mäkelä's avatar
      MDEV-11630 Call mutex_free() before freeing the mutex list · 7e02fd1f
      Marko Mäkelä authored
      Make some global fil_crypt_ variables static.
      
      fil_close(): Call mutex_free(&fil_system->mutex) also in InnoDB, not
      only in XtraDB. In InnoDB, sync_close() was called before fil_close().
      
      innobase_shutdown_for_mysql(): Call fil_close() before sync_close(),
      similar to XtraDB shutdown.
      
      fil_space_crypt_cleanup(): Call mutex_free() to pair with
      fil_space_crypt_init().
      
      fil_crypt_threads_cleanup(): Call mutex_free() to pair with
      fil_crypt_threads_init().
      7e02fd1f
    • Jan Lindström's avatar
      MDEV-11218: encryption.innodb_encryption_discard_import failed in buildbot · 55eb7120
      Jan Lindström authored
      Try to stabilize test cases. These test behave badly when run in certain order.
      55eb7120
    • Sergey Vojtovich's avatar
      MDEV-11227 - mysqlimport -l doesn't issue UNLOCK TABLES · 0c3791ca
      Sergey Vojtovich authored
      Merged fix for innodb_mysql from 5.7.
      0c3791ca
    • Vladislav Vaintroub's avatar
  10. 21 Dec, 2016 4 commits
    • Monty's avatar
      Fixed compiler warning · c51c885d
      Monty authored
      c51c885d
    • Monty's avatar
      MDEV-7558 analyze_stmt_slow_query_log fails sporadically in buildbot · c33c638f
      Monty authored
      The reason was that the test was reusing the same log file without deleting it between tests.
      Fixed by creating a new log file as part of the test
      c33c638f
    • Sergey Vojtovich's avatar
      Updated test results · 27f20d19
      Sergey Vojtovich authored
      27f20d19
    • Sergey Vojtovich's avatar
      MDEV-11227 - mysqlimport -l doesn't issue UNLOCK TABLES · 8774a023
      Sergey Vojtovich authored
      Implementation of MDEV-7660 introduced unwanted incompatible change:
      modifications under LOCK TABLES with autocommit enabled are rolled back on
      disconnect. Previously everything was committed, because LOCK TABLES didn't
      adjust autocommit setting.
      
      This patch restores original behavior by reverting some changes done in
      MDEV-7660:
      - sql/sql_parse.cc: do not reset autocommit on LOCK TABLES
      - sql/sql_base.cc: do not set autocommit on UNLOCK TABLES
      - test cases: main.lock_tables_lost_commit, main.partition_explicit_prune,
        rpl.rpl_switch_stm_row_mixed, tokudb.nested_txn_implicit_commit,
        tokudb_bugs.db806
      
      But it makes InnoDB tables under LOCK TABLES ... READ [LOCAL] not protected
      against DML. To restore protection some changes from WL#6671 were merged,
      specifically MDL_SHARED_READ_ONLY and test cases.
      
      WL#6671 merge highlights:
      - Not all tests merged.
      - In MySQL LOCK TABLES ... READ acquires MDL_SHARED_READ_ONLY for all engines,
        in MariaDB MDL_SHARED_READ is always acquired first and then upgraded to
        MDL_SHARED_READ_ONLY for InnoDB only.
      - The above allows us to omit MDL_SHARED_WRITE_LOW_PRIO implementation in
        MariaDB, which is rather useless with InnoDB. In MySQL it is needed to
        preserve locking behavior between low priority writes and LOCK TABLES ... READ
        for non-InnoDB engines (covered by sys_vars.sql_low_priority_updates_func).
      - Omitted HA_NO_READ_LOCAL_LOCK, we rely on lock_count() instead.
      - Omitted "piglets": in MariaDB stream of DML against InnoDB table may lead to
        concurrent LOCK TABLES ... READ starvation.
      - HANDLER ... OPEN acquires MDL_SHARED_READ instead of MDL_SHARED in MariaDB.
      - Omitted SNRW->X MDL lock upgrade for IMPORT/DISCARD TABLESPAECE under LOCK
        TABLES.
      - Omitted strong locks for views, triggers and SP under LOCK TABLES.
      - Omitted IX schema lock for LOCK TABLES READ.
      - Omitted deadlock weight juggling for LOCK TABLES.
      
      Full WL#6671 merge status:
      - innodb.innodb-lock: fully merged
      - main.alter_table: not merged due to different HANDLER solution
      - main.debug_sync: fully merged
      - main.handler_innodb: not merged due to different HANDLER solution
      - main.handler_myisam: not merged due to different HANDLER solution
      - main.innodb_mysql_lock: fully merged
      - main.insert_notembedded: fully merged
      - main.lock: not merged (due to no strong locks for views)
      - main.lock_multi: not merged
      - main.lock_sync: fully merged (partially in MDEV-7660)
      - main.mdl_sync: not merged
      - main.partition_debug_sync: not merged due to different HANDLER solution
      - main.status: fully merged
      - main.view: fully merged
      - perfschema.mdl_func: not merged (no such test in MariaDB)
      - perfschema.table_aggregate_global_2u_2t: not merged (didn't fail in MariaDB)
      - perfschema.table_aggregate_global_2u_3t: not merged (didn't fail in MariaDB)
      - perfschema.table_aggregate_global_4u_2t: not merged (didn't fail in MariaDB)
      - perfschema.table_aggregate_global_4u_3t: not merged (didn't fail in MariaDB)
      - perfschema.table_aggregate_hist_2u_2t: not merged (didn't fail in MariaDB)
      - perfschema.table_aggregate_hist_2u_3t: not merged (didn't fail in MariaDB)
      - perfschema.table_aggregate_hist_4u_2t: not merged (didn't fail in MariaDB)
      - perfschema.table_aggregate_hist_4u_3t: not merged (didn't fail in MariaDB)
      - perfschema.table_aggregate_thread_2u_2t: not merged (didn't fail in MariaDB)
      - perfschema.table_aggregate_thread_2u_3t: not merged (didn't fail in MariaDB)
      - perfschema.table_aggregate_thread_4u_2t: not merged (didn't fail in MariaDB)
      - perfschema.table_aggregate_thread_4u_3t: not merged (didn't fail in MariaDB)
      - perfschema.table_lock_aggregate_global_2u_2t: not merged (didn't fail in MariaDB)
      - perfschema.table_lock_aggregate_global_2u_3t: not merged (didn't fail in MariaDB)
      - perfschema.table_lock_aggregate_global_4u_2t: not merged (didn't fail in MariaDB)
      - perfschema.table_lock_aggregate_global_4u_3t: not merged (didn't fail in MariaDB)
      - perfschema.table_lock_aggregate_hist_2u_2t: not merged (didn't fail in MariaDB)
      - perfschema.table_lock_aggregate_hist_2u_3t: not merged (didn't fail in MariaDB)
      - perfschema.table_lock_aggregate_hist_4u_2t: not merged (didn't fail in MariaDB)
      - perfschema.table_lock_aggregate_hist_4u_3t: not merged (didn't fail in MariaDB)
      - perfschema.table_lock_aggregate_thread_2u_2t: not merged (didn't fail in MariaDB)
      - perfschema.table_lock_aggregate_thread_2u_3t: not merged (didn't fail in MariaDB)
      - perfschema.table_lock_aggregate_thread_4u_2t: not merged (didn't fail in MariaDB)
      - perfschema.table_lock_aggregate_thread_4u_3t: not merged (didn't fail in MariaDB)
      - sys_vars.sql_low_priority_updates_func: not merged
      - include/thr_rwlock.h: not merged, rw_pr_lock_assert_write_owner and
        rw_pr_lock_assert_not_write_owner are macros in MariaDB
      - sql/handler.h: not merged (HA_NO_READ_LOCAL_LOCK)
      - sql/mdl.cc: partially merged (MDL_SHARED_READ_ONLY only)
      - sql/mdl.h: partially merged (MDL_SHARED_READ_ONLY only)
      - sql/lock.cc: fully merged
      - sql/sp_head.cc: not merged
      - sql/sp_head.h: not merged
      - sql/sql_base.cc: partially merged (MDL_SHARED_READ_ONLY only)
      - sql/sql_base.h: not merged
      - sql/sql_class.cc: fully merged
      - sql/sql_class.h: fully merged
      - sql/sql_handler.cc: merged partially (different solution in MariaDB)
      - sql/sql_parse.cc: partially merged, mostly omitted low priority write part
      - sql/sql_reload.cc: not merged comment change
      - sql/sql_table.cc: not merged SNRW->X upgrade for IMPORT/DISCARD TABLESPACE
      - sql/sql_view.cc: not merged
      - sql/sql_yacc.yy: not merged (MDL_SHARED_WRITE_LOW_PRIO, MDL_SHARED_READ_ONLY)
      - sql/table.cc: not merged (MDL_SHARED_WRITE_LOW_PRIO)
      - sql/table.h: not merged (MDL_SHARED_WRITE_LOW_PRIO)
      - sql/trigger.cc: not merged
      - storage/innobase/handler/ha_innodb.cc: merged store_lock()/lock_count()
        changes (in MDEV-7660), didn't merge HA_NO_READ_LOCAL_LOCK
      - storage/innobase/handler/ha_innodb.h: fully merged in MDEV-7660
      - storage/myisammrg/ha_myisammrg.cc: not merged comment change
      - storage/perfschema/table_helper.cc: not merged (no MDL support in MariaDB PFS)
      - unittest/gunit/mdl-t.cc: not merged
      - unittest/gunit/mdl_sync-t.cc: not merged
      
      MariaDB specific changes:
      - handler.heap: different HANDLER solution, MDEV-7660
      - handler.innodb: different HANDLER solution, MDEV-7660
      - handler.interface: different HANDLER solution, MDEV-7660
      - handler.myisam: different HANDLER solution, MDEV-7660
      - main.mdl_sync: MDEV-7660 specific changes
      - main.partition_debug_sync: removed test due to different HANDLER solution,
        MDEV-7660
      - main.truncate_coverage: removed test due to different HANDLER solution,
        MDEV-7660
      - mysql-test/include/mtr_warnings.sql: additional cleanup, MDEV-7660
      - mysql-test/lib/v1/mtr_report.pl: additional cleanup, MDEV-7660
      - plugin/metadata_lock_info/metadata_lock_info.cc: not in MySQL
      - sql/sql_handler.cc: MariaDB specific fix for mysql_ha_read(), MDEV-7660
      8774a023