1. 13 Oct, 2013 2 commits
    • Michael Widenius's avatar
      Fixes for parallel slave: · 2e100cc5
      Michael Widenius authored
      - Made slaves temporary table multi-thread slave safe by adding mutex around save_temporary_table usage.
        - rli->save_temporary_tables is the active list of all used temporary tables
        - This is copied to THD->temporary_tables when temporary tables are opened and updated when temporary tables are closed
        - Added THD->lock_temporary_tables() and THD->unlock_temporary_tables() to simplify this.
      - Relay_log_info->sql_thd renamed to Relay_log_info->sql_driver_thd to avoid wrong usage for merged code.
      - Added is_part_of_group() to mark functions that are part of the next function. This replaces setting IN_STMT when events are executed.
      - Added is_begin(), is_commit() and is_rollback() functions to Query_log_event to simplify code.
      - If slave_skip_counter is set run things in single threaded mode. This simplifies code for skipping events.
      - Updating state of relay log (IN_STMT and IN_TRANSACTION) is moved to one single function: update_state_of_relay_log()
        We can't use OPTION_BEGIN to check for the state anymore as the sql_driver and sql execution threads may be different.
        Clear IN_STMT and IN_TRANSACTION in init_relay_log_pos() and Relay_log_info::cleanup_context() to ensure the flags doesn't survive slave restarts
        is_in_group() is now independent of state of executed transaction.
      - Reset thd->transaction.all.modified_non_trans_table() if we did set it for single table row events.
        This was mainly for keeping the flag as documented.
      - Changed slave_open_temp_tables to uint32 to be able to use atomic operators on it.
      - Relay_log_info::sleep_lock -> rpl_group_info::sleep_lock
      - Relay_log_info::sleep_cond -> rpl_group_info::sleep_cond
      - Changed some functions to take rpl_group_info instead of Relay_log_info to make them multi-slave safe and to simplify usage
        - do_shall_skip()
        - continue_group()
        - sql_slave_killed()
        - next_event()
      - Simplifed arguments to io_salve_killed(), check_io_slave_killed() and sql_slave_killed(); No reason to supply THD as this is part of the given structure.
      - set_thd_in_use_temporary_tables() removed as in_use is set on usage
      - Added information to thd_proc_info() which thread is waiting for slave mutex to exit.
      - In open_table() reuse code from find_temporary_table()
      
      Other things:
      - More DBUG statements
      - Fixed the rpl_incident.test can be run with --debug
      - More comments
      - Disabled not used function rpl_connect_master()
      
      mysql-test/suite/perfschema/r/all_instances.result:
        Moved sleep_lock and sleep_cond to rpl_group_info
      mysql-test/suite/rpl/r/rpl_incident.result:
        Updated result
      mysql-test/suite/rpl/t/rpl_incident-master.opt:
        Not needed anymore
      mysql-test/suite/rpl/t/rpl_incident.test:
        Fixed that test can be run with --debug
      sql/handler.cc:
        More DBUG_PRINT
      sql/log.cc:
        More comments
      sql/log_event.cc:
        Added DBUG statements
        do_shall_skip(), continue_group() now takes rpl_group_info param
        Use is_begin(), is_commit() and is_rollback() functions instead of inspecting query string
        We don't have set slaves temporary tables 'in_use' as this is now done when tables are opened.
        Removed IN_STMT flag setting. This is now done in update_state_of_relay_log()
        Use IN_TRANSACTION flag to test state of relay log.
        In rows_event_stmt_cleanup() reset thd->transaction.all.modified_non_trans_table if we had set this before.
      sql/log_event.h:
        do_shall_skip(), continue_group() now takes rpl_group_info param
        Added is_part_of_group() to mark events that are part of the next event. This replaces setting IN_STMT when events are executed.
        Added is_begin(), is_commit() and is_rollback() functions to Query_log_event to simplify code.
      sql/log_event_old.cc:
        Removed IN_STMT flag setting. This is now done in update_state_of_relay_log()
        do_shall_skip(), continue_group() now takes rpl_group_info param
      sql/log_event_old.h:
        Added is_part_of_group() to mark events that are part of the next event.
        do_shall_skip(), continue_group() now takes rpl_group_info param
      sql/mysqld.cc:
        Changed slave_open_temp_tables to uint32 to be able to use atomic operators on it.
        Relay_log_info::sleep_lock -> Rpl_group_info::sleep_lock
        Relay_log_info::sleep_cond -> Rpl_group_info::sleep_cond
      sql/mysqld.h:
        Updated types and names
      sql/rpl_gtid.cc:
        More DBUG
      sql/rpl_parallel.cc:
        Updated TODO section
        Set thd for event that is execution
        Use new  is_begin(), is_commit() and is_rollback() functions.
        More comments
      sql/rpl_rli.cc:
        sql_thd -> sql_driver_thd
        Relay_log_info::sleep_lock -> rpl_group_info::sleep_lock
        Relay_log_info::sleep_cond -> rpl_group_info::sleep_cond
        Clear IN_STMT and IN_TRANSACTION in init_relay_log_pos() and Relay_log_info::cleanup_context() to ensure the flags doesn't survive slave restarts.
        Reset table->in_use for temporary tables as the table may have been used by another THD.
        Use IN_TRANSACTION instead of OPTION_BEGIN to check state of relay log.
        Removed IN_STMT flag setting. This is now done in update_state_of_relay_log()
      sql/rpl_rli.h:
        Changed relay log state flags to bit masks instead of bit positions (most other code we have uses bit masks)
        Added IN_TRANSACTION to mark if we are in a BEGIN ... COMMIT section.
        save_temporary_tables is now thread safe
        Relay_log_info::sleep_lock -> rpl_group_info::sleep_lock
        Relay_log_info::sleep_cond -> rpl_group_info::sleep_cond
        Relay_log_info->sql_thd renamed to Relay_log_info->sql_driver_thd to avoid wrong usage for merged code
        is_in_group() is now independent of state of executed transaction.
      sql/slave.cc:
        Simplifed arguments to io_salve_killed(), sql_slave_killed() and check_io_slave_killed(); No reason to supply THD as this is part of the given structure.
        set_thd_in_use_temporary_tables() removed as in_use is set on usage in sql_base.cc
        sql_thd -> sql_driver_thd
        More DBUG
        Added update_state_of_relay_log() which will calculate the IN_STMT and IN_TRANSACTION state of the relay log after the current element is executed.
        If slave_skip_counter is set run things in single threaded mode.
        Simplifed arguments to io_salve_killed(), check_io_slave_killed() and sql_slave_killed(); No reason to supply THD as this is part of the given structure.
        Added information to thd_proc_info() which thread is waiting for slave mutex to exit.
        Disabled not used function rpl_connect_master()
        Updated argument to next_event()
      sql/sql_base.cc:
        Added mutex around usage of slave's temporary tables. The active list is always kept up to date in sql->rgi_slave->save_temporary_tables.
        Clear thd->temporary_tables after query (safety)
        More DBUG
        When using temporary table, set table->in_use to current thd as the THD may be different for slave threads.
        Some code is ifdef:ed with REMOVE_AFTER_MERGE_WITH_10 as the given code in 10.0 is not yet in this tree.
        In open_table() reuse code from find_temporary_table()
      sql/sql_binlog.cc:
        rli->sql_thd -> rli->sql_driver_thd
        Remove duplicate setting of rgi->rli
      sql/sql_class.cc:
        Added helper functions rgi_lock_temporary_tables() and rgi_unlock_temporary_tables()
        Would have been nicer to have these inline, but there was no easy way to do that
      sql/sql_class.h:
        Added functions to protect slaves temporary tables
      sql/sql_parse.cc:
        Added DBUG_PRINT
      sql/transaction.cc:
        Added comment
      2e100cc5
    • Michael Widenius's avatar
      Give a warning, not an error, if the log file size in innodb doesn't match what is on disk · 37844322
      Michael Widenius authored
      This helps when moving from MariaDB 5.5 to MariaDB 10.0 as sometimes the log file size is rounded differently.
      
      storage/innobase/srv/srv0start.c:
        Give a warning, not an error, if the log file size in innodb doesn't match what is on disk
      storage/xtradb/srv/srv0start.c:
        Give a warning, not an error, if the log file size in innodb doesn't match what is on disk
      37844322
  2. 08 Oct, 2013 1 commit
    • unknown's avatar
      MDEV-4506: Parallel replication. · 12c760ef
      unknown authored
      Improve STOP SLAVE in parallel mode.
      
      Now, the parallel part will queue the current event group to the
      end, and then stop queing any more events. Each worker will
      complete the current event group, and then just skip any further
      queued events.
      12c760ef
  3. 30 Sep, 2013 1 commit
  4. 23 Sep, 2013 2 commits
  5. 19 Sep, 2013 2 commits
  6. 17 Sep, 2013 2 commits
  7. 16 Sep, 2013 1 commit
  8. 13 Sep, 2013 1 commit
  9. 12 Jul, 2013 3 commits
  10. 09 Jul, 2013 1 commit
  11. 08 Jul, 2013 1 commit
    • unknown's avatar
      MDEV-4506: Parallel replication: intermediate commit. · a99356fb
      unknown authored
      Fix a bunch of issues found with locking, ordering, and non-thread-safe stuff
      in Relay_log_info.
      
      Now able to do a simple benchmark, showing 4.5 times speedup for applying a
      binlog with 10000 REPLACE statements.
      a99356fb
  12. 04 Jul, 2013 3 commits
    • unknown's avatar
      MDEV-4506: Parallel replication: Intermediate commit. · e654be38
      unknown authored
      Impement options --binlog-commit-wait-count and
      --binlog-commit-wait-usec.
      
      These options permit the DBA to deliberately increase latency
      of an individual commit to get more transactions in each
      binlog group commit. This increases the opportunity for
      parallel replication on the slave, and can also decrease I/O
      load on the master.
      
      The options also make it easier to test the parallel
      replication with mysql-test-run.
      e654be38
    • unknown's avatar
      MDEV-4506: Parallel replication: Intermediate commit. · b5a496a7
      unknown authored
      Fix some bugs around waiting for worker threads to end during SQL slave stop.
      
      Free Log_event after parallel execution (still needs to be made thread-safe by
      using rpl_group_info rather than rli).
      b5a496a7
    • unknown's avatar
      MDEV-4506: Parallel replication: Intermediate commit. · a1cfd473
      unknown authored
      Wait for all worker threads to finish when stopping the SQL thread.
      (Only a basic wait; this still needs to be fixed to include timeout
      logic as in sql_slave_killed()).
      a1cfd473
  13. 03 Jul, 2013 2 commits
  14. 28 Jun, 2013 1 commit
  15. 26 Jun, 2013 1 commit
    • unknown's avatar
      MDEV-4506: Parallel replication. Intermediate commit. · 7e5dc4f0
      unknown authored
      Implement facility for the commit in one thread to wait for the commit of
      another to complete first. The wait is done in a way that does not hinder
      that a waiter and a waitee can group commit together with a single fsync()
      in both binlog and InnoDB. The wait is done efficiently with respect to
      locking.
      
      The patch was originally made to support TaoBao parallel replication with
      in-order commit; now it will be adapted to also be used for parallel
      replication of group-committed transactions.
      
      A waiter THD registers itself with a prior waitee THD. The waiter will then
      complete its commit at the earliest in the same group commit of the waitee
      (when using binlog). The wait can also be done explicitly by the waitee.
      7e5dc4f0
  16. 25 Jun, 2013 2 commits
  17. 24 Jun, 2013 1 commit
  18. 08 Jun, 2013 1 commit
  19. 07 Jun, 2013 4 commits
    • unknown's avatar
      MDEV-4490: Old-style master position points at the last GTID event after slave restart · b5fcf33d
      unknown authored
      Now whenever we reach the GTID point requested from the slave (when using GTID
      position to connect), we send a fake Gtid_list event. This event is used by
      the slave to know the current old-style position for MASTER_POS_WAIT(), and
      later the similar binlog position for MASTER_GTID_WAIT().
      
      Without this fake event, if the slave is already fully up-to-date with the
      master, there may be no events sent at the given position for an indeterminate
      time.
      b5fcf33d
    • unknown's avatar
      MDEV-4486: Allow to start old-style replication even if mysql.rpl_slave_state is unavailable · 03aa4876
      unknown authored
      If the mysql.gtid_slave_pos table is not available, we cannot load nor update
      the current GTID position persistently. This can happen eg. after an upgrade,
      before mysql_upgrade_db is run, or if the table is InnoDB and the server is
      restarted without the InnoDB storage engine enabled.
      
      Before, replication always failed to start if the table was unavailable. With
      this patch, we try to continue with old-style replication, after suitable
      complaints in the error log. In strict mode, or if slave is configured to use
      GTID, slave still refuses to start.
      03aa4876
    • unknown's avatar
      MDEV-4591:Setting gtid* values from inside a transaction might cause unexpected results · dbe2c506
      unknown authored
      Now we give an error on attempts to set @@SESSION.gtid_domain_id or
      @@SESSION.gtid_seq_no when a transaction is active.
      dbe2c506
    • unknown's avatar
      MDEV-4483: CHANGE MASTER TO master_use_gtid=xxx looses old-style coordinates. · 7b6ab563
      unknown authored
      There was some old code that cleared the position in CHANGE MASTER,
      it was forgotten to be removed.
      
      In addition, add code that saves/restores the old-style position
      when we nuke the old relay logs as part of GTID slave start.
      Normally we will not use these, but it could be useful in case
      the GTID connect fails and user wants to go back to the old-style
      coordinates.
      7b6ab563
  20. 06 Jun, 2013 3 commits
    • Sergei Golubchik's avatar
      5.5 merge · 4749d40c
      Sergei Golubchik authored
      4749d40c
    • Vladislav Vaintroub's avatar
      fix compile error · 1ff1cb10
      Vladislav Vaintroub authored
      1ff1cb10
    • Michael Widenius's avatar
      Fixed timing failure in myisam-metadata.test · 5cf5a9a1
      Michael Widenius authored
      mysql-test/include/wait_show_condition.inc:
        Print failing statement if timeout
      mysql-test/r/myisam-metadata.result:
        Updated DBUG_SYNC
      mysql-test/t/myisam-metadata.test:
        Updated DBUG_SYNC.
        Removed wait_show_condtion, as this is not needed when we use DBUG_SYNC
        This should fix timing issues with the test
      mysys/thr_mutex.c:
        Added comments
      sql/sql_acl.cc:
        atoi -> atoll()  (Safety)
      storage/myisam/ha_myisam.cc:
        Send signal before mi_repair_by_sort.
      5cf5a9a1
  21. 05 Jun, 2013 2 commits
    • unknown's avatar
      Fix two small problems in previous push. · 64e53a0f
      unknown authored
      64e53a0f
    • unknown's avatar
      MDEV-26: Global transaction ID. · 5cb486d1
      unknown authored
      Fix problems related to reconnect. When we need to reconnect (ie. explict
      stop/start of just the IO thread by user, or automatic reconnect due to
      loosing network connection with the master), it is a bit complex to correctly
      resume at the right point without causing duplicate or missing events in the
      relay log. The previous code had multiple problems in this regard.
      
      With this patch, the problem is solved as follows. The IO thread keeps track
      (in memory) of which GTID was last queued to the relay log. If it needs to
      reconnect, it resumes at that GTID position. It also counts number of events
      received within the last, possibly partial, event group, and skips the same
      number of events after a reconnect, so that events already enqueued before the
      reconnect are not duplicated.
      
      (There is no need to keep any persistent state; whenever we restart slave
      threads after both of them being stopped (such as after server restart), we
      erase the relay logs and start over from the last GTID applied by SQL thread.
      But while the SQL thread is running, this patch is needed to get correct relay
      log).
      5cb486d1
  22. 03 Jun, 2013 1 commit
    • unknown's avatar
      MDEV-4605: Failing to load GTID slave position from rpl.gtid_slave_pos · 7ad47ab0
      unknown authored
      There were several cases where the slave GTID position was not loaded
      correctly before being used. This caused various failures such as
      corrupting the position at slave start and empty values of
      @@gtid_slave_pos and @@gtid_current_pos.
      
      Fixed by adding more checks for loaded position, and by always loading
      the position at server startup.
      7ad47ab0
  23. 01 Jun, 2013 2 commits