1. 22 May, 2021 1 commit
  2. 21 May, 2021 2 commits
    • Julius Goryavsky's avatar
      MDEV-25719: stunnel uses "verifyChain" without subject checks · 8e280f30
      Julius Goryavsky authored
      Another batch of changes that should make the SST process
      more reliable in all scenarios:
      
       1) Added hostname or CN verification when stunnel is used
          with certificate chain verification (verifyChain = yes);
       2) Added check for the absence of the stunnel utility for
          mtr tests;
       3) Deletion of working files before and after SST is done
          more accurately;
       4) rsync on joiner can be run even if the path to its
          configuration file contains spaces;
       5) More accurate directory creation (for data files and
          for logs);
       6) IST with mysqldump no longer turns off statement logging;
       7) Reset password for mysqldump when password is empty but
          username is specified;
       8) More reliable quoting when generating statements in
          wsrep_sst_mysqldump;
       9) Added explicit generation of 2048-bit Diffie-Hellman
          parameters for sockat < 1.7.3, by analogy with xtrabackup;
      10) Compression parameters for qpress are read from all
          suitable server groups in configuration file, as well as
          from the [sst] and [xtrabackup] groups;
      11) Added a test that checks compression using qpress;
      12) Checking for optional utilities is modified to work even
          if they implemented as built-in shell commands (unlikely
          on real systems, but more reliable).
      8e280f30
    • mkaruza's avatar
      MDEV-25562 Assertion `pause_seqno_.is_undefined() == false' failed in void... · 5667baad
      mkaruza authored
      MDEV-25562 Assertion `pause_seqno_.is_undefined() == false' failed in void wsrep::server_state::resume()
      
      If pause() is not executed in galera and returns seqno = -1 we should
      skip resume().
      5667baad
  3. 18 May, 2021 4 commits
  4. 17 May, 2021 12 commits
    • Julius Goryavsky's avatar
      MDEV-25693: SST failed due to incorrect connection address · e861e057
      Julius Goryavsky authored
      Fixed bugs caused by inaccuracies in automatic merging
      from other branches:
      
      1) Authentication information is not removed from the connection
         address, which causes some tests to fail;
      2) wsrep_debug=on should be replaced with wsrep_debug=1;
      3) Added missing "connection" lines to test result file;
      4) Some tests have been corrected for Galera 4.x (10.4+).
      e861e057
    • Julius Goryavsky's avatar
      cf4dd3cc
    • Julius Goryavsky's avatar
      9f03a394
    • Julius Goryavsky's avatar
      23cad4d8
    • Julius Goryavsky's avatar
      MDEV-25669: SST scripts should check all server groups in config files · f92cd0c5
      Julius Goryavsky authored
      1) This commit implements reading all sections from configuration
      files while looking for the current value of any server variable,
      which were previously only read from the [mysqld.suffix] group and
      from [mysqld], but not from other groups such as [mariadb.suffix],
      [mariadb] or, for example, [server].
      
      2) This commit also fixes misrecognition of some parameters when
      parsing a command line containing a special marker for the end
      of the list of options ("--") or when short option names (such
      as "-s", "-a" and "-h arg") chained together (like a "-sah arg").
      Such parameters can be passed to the SST script in the list of
      arguments after "--mysqld-args" if the server is started with a
      complex set of options - this was revealed during manual testing
      of changes to read configuration files.
      
      3) The server-side preparation code for the "--mysqld-args"
      option list has also been simplified to make it easier to change
      in the future (if needed), and has been improved to properly
      handle the special backquote ("`") character in the argument
      values.
      f92cd0c5
    • Julius Goryavsky's avatar
      16437e5e
    • Julius Goryavsky's avatar
      MDEV-25669: SST scripts should check all server groups in config files · f9f8e33f
      Julius Goryavsky authored
      1) This commit implements reading all sections from configuration
      files while looking for the current value of any server variable,
      which were previously only read from the [mysqld.suffix] group and
      from [mysqld], but not from other groups such as [mariadb.suffix],
      [mariadb] or, for example, [server].
      
      2) This commit also fixes misrecognition of some parameters when
      parsing a command line containing a special marker for the end
      of the list of options ("--") or when short option names (such
      as "-s", "-a" and "-h arg") chained together (like a "-sah arg").
      Such parameters can be passed to the SST script in the list of
      arguments after "--mysqld-args" if the server is started with a
      complex set of options - this was revealed during manual testing
      of changes to read configuration files.
      
      3) The server-side preparation code for the "--mysqld-args"
      option list has also been simplified to make it easier to change
      in the future (if needed), and has been improved to properly
      handle the special backquote ("`") character in the argument
      values.
      f9f8e33f
    • Julius Goryavsky's avatar
      16898e7f
    • Julius Goryavsky's avatar
    • Julius Goryavsky's avatar
      MDEV-23580: WSREP_SST: [ERROR] rsync daemon port has been taken · 27ae7f2a
      Julius Goryavsky authored
      This commit contains a large set of further bug fixes and
      improvements to SST scripts for Galera, continuing the work
      that was started in MDEV-24962 to make SST scripts work smoothly
      in different network configurations (especially using ipv6) and
      with different environment settings:
      
       1) The ipv6 addresses were incorrectly handled in the SST script
          for rsync (incorrect address substitution for establishing a
          connection, incorrect address substitution for bind, and so on);
       2) Checking the locality of the ip-address in SST scripts did not
          support ipv6 addresses (such as "[::1]"), which were falsely
          identified as non-local ip, which further did not allow running
          two SSTs on different local addresses on the same machine.
          On the other hand, this bug masked some other errors (related
          to handling ipv6 addresses);
       3) The code for checking the locality of the ip address was different
          in the SST scripts for rsync and for mysqldump, with individual
          flaws. This code is now made common and moved to wsrep_sst_common;
       4) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          did not process ipv6 addresses correctly in all cases (not for all
          branches);
       5) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          for some code branches could give a false positive result due to
          the textual match of prefixes in the port number and/or PID of
          the process;
       6) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) was supported through different utilities in SST scripts
          for mariabackup and for rsync, and with various minor flaws in
          the code. Now the code is still different in these scripts, but
          it supports a common set of utilities (lsof, ss, sockstat) and
          is synchronized across patterns that used to check the output
          of  these utilities;
       7) In SST via mariabackup, the signal about readiness to receive data
          is sometimes sent too early - immediately after listen(), and not
          after accept() (which are called by socat or netcat utility).
       8) Checking availability of the some options of some utilities was
          done using the grep pattern, which easily gives false positives;
       9) Common name (CN) for local addresses, if not explicitly specified,
          is now always replaced to "localhost" to avoid the need to generate
          many separate certificates for local addresses of one machine and
          not to depend on which the local address is currently used in test
          (ipv4 or ipv6, etc.);
      10) In tests galera_sst_mariabackup_encrypt_with_key_server and
          galera_sst_rsync_encrypt_with_key_server the correct certificate
          is selected to avoid commonname (CN) mismatch problems;
      11) Further refactoring to protect against spaces in file names.
      12) Further general refactoring to eliminate bash-specific constructs
          or to improve code readability;
      13) The code for setting options for the nc (netcat) utility was
          different in different scripts for SST - now it is made identical.
      14) Fixed long-time broken encryption via xbcrypt in combination with
          mariabackup and added support for key-based encryption via openssl
          utility, which is now enabled by default for encrypt=1 mode (this
          default mode can be changed using a new configuration file option
          "encypt-format=openssl|xbcrypt", which can be placed in the [mysqld],
          [sst] or in the [xtrabackup] section) - this change will allow us
          to use and to test the encypt=1 encryption without installing
          non-standard third-party utilities.
      27ae7f2a
    • Sujatha's avatar
      MDEV-22530: Aborting OPTIMIZE TABLE still logs in binary log and replicates to the Slave server. · 88c7a58e
      Sujatha authored
      Problem:
      ========
      Aborting OPTIMIZE TABLE still logs in binary logs and replicates to the
      Slave server. "Optimize table" command under execution, is killed by using
      "Ctrl-C" as shown below.
      
      MariaDB [test]> optimize table t2;
      ^CCtrl-C -- query killed. Continuing normally.
      
      In spite of query execution being interrupted the query gets written to
      binary log.
      
      Analysis:
      ========
      Admin command execution logic is not handling KILL command, hence it
      ignores the KILL command and completes its execution.
      
      Fix:
      ===
      Check for thread killed notification, during admin command execution and
      handle it. If thread kill occurs prior to any table modification the query
      will not be written to binary log. If kill happens after at least one table
      is modified then the query will be written to binary log. Ex: command in
      execution is 'OPTIMIZE TABLE t1,t2' and the thread kill happens after t1
      table is modified then 'OPTIMIZE TABLE t1,t2' will be written to binary log
      as admin commands will not make the slave to diverge from master.
      88c7a58e
    • Sujatha's avatar
      MDEV-17515: GTID Replication in optimistic mode deadlock · 410e3c1a
      Sujatha authored
      Problem:
      =======
      In slave_parallel_mode=optimistic configuration, when admin commands and
      DML operation on the same table are scheduled simultaneously for execution,
      it results in lock conflict and slave server either hangs due to
      deadlock or goes down with an assert.
      
      Analysis:
      ========
      Admin commands OPTIMIZE, REPAIR and ANALYZE are written to binary log as
      ordinary transactions. When 'slave_parallel_mode' is 'optimistic' DMLs are
      allowed to run in parallel. But these locks are not detected by parallel
      replication deadlock detection-and-handling mechanism. At times they result
      in deadlock or assertion.
      
      Fix:
      ===
      Flag admin commands as DDL in Gtid_log_event at the time of writing to
      binary log. Add a new bit EXECUTED_TABLE_ADMIN_CMD to
      'm_unsafe_rollback_flags'. During 'mysql_admin_table' command execution it
      accepts a list of tables to be processed and executes them in a loop. Upon
      successful execution enable 'EXECUTED_TABLE_ADMIN_CMD' bit in
      thd->transaction.stmt_unsafe_rollback_flags. Gtid_log_event constructor
      will notice this flag and mark the current transaction with 'FL_DDL' flag.
      Gtid_log_events marked as FL_DDL will not be scheduled parallel execution,
      on the slave. They will execute in isolation to prevent deadlocks.
      
      Note: Removed the call to 'trans_commit_implicit' from 'mysql_admin_table'
      function as 'mysql_execute_command' will take care of invoking
      'trans_commit_implicit'.
      410e3c1a
  5. 16 May, 2021 1 commit
  6. 15 May, 2021 2 commits
    • Julius Goryavsky's avatar
      MDEV-25669: SST scripts should check all server groups in config files · 6811ed3e
      Julius Goryavsky authored
      1) This commit implements reading all sections from configuration
      files while looking for the current value of any server variable,
      which were previously only read from the [mysqld.suffix] group and
      from [mysqld], but not from other groups such as [mariadb.suffix],
      [mariadb] or, for example, [server].
      
      2) This commit also fixes misrecognition of some parameters when
      parsing a command line containing a special marker for the end
      of the list of options ("--") or when short option names (such
      as "-s", "-a" and "-h arg") chained together (like a "-sah arg").
      Such parameters can be passed to the SST script in the list of
      arguments after "--mysqld-args" if the server is started with a
      complex set of options - this was revealed during manual testing
      of changes to read configuration files.
      
      3) The server-side preparation code for the "--mysqld-args"
      option list has also been simplified to make it easier to change
      in the future (if needed), and has been improved to properly
      handle the special backquote ("`") character in the argument
      values.
      6811ed3e
    • Julius Goryavsky's avatar
      4675febb
  7. 14 May, 2021 4 commits
    • Igor Babaev's avatar
      MDEV-25682 Explain shows an execution plan different from actually executed · 677f1ef6
      Igor Babaev authored
      If a select query contained an ORDER BY clause that followed a LIMIT clause
      or an ORDER BY clause or ORDER BY with LIMIT the EXPLAIN output for the
      query showed an execution plan different from that was actually executed.
      
      Approved by Roman Nozdrin <roman.nozdrin@mariadb.com>
      677f1ef6
    • Sachin Kumar's avatar
      MDEV-25336 Parallel replication causes failed assert while restarting · e607f339
      Sachin Kumar authored
      Problem:- When slave is shutdown, we will get this assertion failure
      sql/sql_list.h:642: void ilink::assert_linked(): Assertion `prev != 0
      && next != 0' failed.
      
      Solution:- In close_connections when we call threads.get() it resets to
      prev and next to NULL. And in parallel worker thread(handle_rpl_parallel_thread)
      calls unlink_not_visible_thd() which assert on prev and next being not NULL.
      .unlink_not_visible_thd() should be always called first before threads.get()
      is called. To make sure worker calls unlink_not_visible_thd() in
      slave_prepare_for_shutdown() we are deactivating the  worker thread pool
      which in turn will close all worker threads. Since this is already done in 10.4
      and 10.5 I am backPorting MDEV-20821 and MDEV-22370 to 10.2. Mdev-22370
      is improving the MDEV-20821 patch.
      e607f339
    • Sachin Kumar's avatar
      MDEV-22370 safe_mutex: Trying to lock uninitialized mutex at... · 355dc74b
      Sachin Kumar authored
      MDEV-22370 safe_mutex: Trying to lock uninitialized mutex at /data/src/10.4-bug/sql/rpl_parallel.cc, line 470 upon shutdown during FTWRL
      
      Problem:- When we issue FTWRL with shutdown in parallel, there is race between
      FTWRL and shutdown. Shutdown might destroy the mutex (pool->LOCK_rpl_thread_pool)
      before FTWRL can lock it. So we can get crash on FTWRL thread
      
      Solution:- mysql_mutex_destroy(pool->LOCK_rpl_thread_pool) should wait for
      FTWRL thread to complete its work , and then destroy.
      So slave_prepare_for_shutdown will just deactivate the pool, and mutex is destroyed
      later in end_slave()
      355dc74b
    • Andrei Elkin's avatar
      MDEV-20821 parallel slave server shutdown hang · 3616640a
      Andrei Elkin authored
      Parallel slave server shutdown found to be hanging in
      close_connections() triggered by shutdown due to a slave worker thread
      would not be notified to exit in case the worker was sitting idle.
      
      Fixed with destroying the worker pool earlier that is in
      slave_prepare_for_shutdown() when all their driver threads have already left.
      A test file is added to simulate the bug condition as well as check
      multi-sourced and not-idle worker cases.
      3616640a
  8. 11 May, 2021 5 commits
    • Sergei Golubchik's avatar
      ec348f55
    • Ramesh Sivaraman's avatar
      Updated galera_3nodes disabled.def file · 1447b394
      Ramesh Sivaraman authored
      1447b394
    • Julius Goryavsky's avatar
    • Julius Goryavsky's avatar
    • Julius Goryavsky's avatar
      MDEV-23580: WSREP_SST: [ERROR] rsync daemon port has been taken · 7e8a8938
      Julius Goryavsky authored
      This commit contains a large set of further bug fixes and
      improvements to SST scripts for Galera, continuing the work
      that was started in MDEV-24962 to make SST scripts work smoothly
      in different network configurations (especially using ipv6) and
      with different environment settings:
      
       1) The ipv6 addresses were incorrectly handled in the SST script
          for rsync (incorrect address substitution for establishing a
          connection, incorrect address substitution for bind, and so on);
       2) Checking the locality of the ip-address in SST scripts did not
          support ipv6 addresses (such as "[::1]"), which were falsely
          identified as non-local ip, which further did not allow running
          two SSTs on different local addresses on the same machine.
          On the other hand, this bug masked some other errors (related
          to handling ipv6 addresses);
       3) The code for checking the locality of the ip address was different
          in the SST scripts for rsync and for mysqldump, with individual
          flaws. This code is now made common and moved to wsrep_sst_common;
       4) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          did not process ipv6 addresses correctly in all cases (not for all
          branches);
       5) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          for some code branches could give a false positive result due to
          the textual match of prefixes in the port number and/or PID of
          the process;
       6) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) was supported through different utilities in SST scripts
          for mariabackup and for rsync, and with various minor flaws in
          the code. Now the code is still different in these scripts, but
          it supports a common set of utilities (lsof, ss, sockstat) and
          is synchronized across patterns that used to check the output
          of  these utilities;
       7) In SST via mariabackup, the signal about readiness to receive data
          is sometimes sent too early - immediately after listen(), and not
          after accept() (which are called by socat or netcat utility).
       8) Checking availability of the some options of some utilities was
          done using the grep pattern, which easily gives false positives;
       9) Common name (CN) for local addresses, if not explicitly specified,
          is now always replaced to "localhost" to avoid the need to generate
          many separate certificates for local addresses of one machine and
          not to depend on which the local address is currently used in test
          (ipv4 or ipv6, etc.);
      10) In tests galera_sst_mariabackup_encrypt_with_key_server and
          galera_sst_rsync_encrypt_with_key_server the correct certificate
          is selected to avoid commonname (CN) mismatch problems;
      11) Further refactoring to protect against spaces in file names.
      12) Further general refactoring to eliminate bash-specific constructs
          or to improve code readability;
      13) The code for setting options for the nc (netcat) utility was
          different in different scripts for SST - now it is made identical.
      14) Fixed long-time broken encryption via xbcrypt in combination with
          mariabackup and added support for key-based encryption via openssl
          utility, which is now enabled by default for encrypt=1 mode (this
          default mode can be changed using a new configuration file option
          "encypt-format=openssl|xbcrypt", which can be placed in the [mysqld],
          [sst] or in the [xtrabackup] section) - this change will allow us
          to use and to test the encypt=1 encryption without installing
          non-standard third-party utilities.
      7e8a8938
  9. 10 May, 2021 6 commits
    • Daniel Bartholomew's avatar
      bump the VERSION · b2bb747f
      Daniel Bartholomew authored
      b2bb747f
    • Daniel Bartholomew's avatar
      bump the VERSION · bc4d995c
      Daniel Bartholomew authored
      bc4d995c
    • Daniel Bartholomew's avatar
      bump the VERSION · 089d82a7
      Daniel Bartholomew authored
      089d82a7
    • Julius Goryavsky's avatar
      MDEV-23580: WSREP_SST: [ERROR] rsync daemon port has been taken · 8fef2b86
      Julius Goryavsky authored
      This commit contains a large set of further bug fixes and
      improvements to SST scripts for Galera, continuing the work
      that was started in MDEV-24962 to make SST scripts work smoothly
      in different network configurations (especially using ipv6) and
      with different environment settings:
      
       1) The ipv6 addresses were incorrectly handled in the SST script
          for rsync (incorrect address substitution for establishing a
          connection, incorrect address substitution for bind, and so on);
       2) Checking the locality of the ip-address in SST scripts did not
          support ipv6 addresses (such as "[::1]"), which were falsely
          identified as non-local ip, which further did not allow running
          two SSTs on different local addresses on the same machine.
          On the other hand, this bug masked some other errors (related
          to handling ipv6 addresses);
       3) The code for checking the locality of the ip address was different
          in the SST scripts for rsync and for mysqldump, with individual
          flaws. This code is now made common and moved to wsrep_sst_common;
       4) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          did not process ipv6 addresses correctly in all cases (not for all
          branches);
       5) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          for some code branches could give a false positive result due to
          the textual match of prefixes in the port number and/or PID of
          the process;
       6) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) was supported through different utilities in SST scripts
          for mariabackup and for rsync, and with various minor flaws in
          the code. Now the code is still different in these scripts, but
          it supports a common set of utilities (lsof, ss, sockstat) and
          is synchronized across patterns that used to check the output
          of  these utilities;
       7) In SST via mariabackup, the signal about readiness to receive data
          is sometimes sent too early - immediately after listen(), and not
          after accept() (which are called by socat or netcat utility).
       8) Checking availability of the some options of some utilities was
          done using the grep pattern, which easily gives false positives;
       9) Common name (CN) for local addresses, if not explicitly specified,
          is now always replaced to "localhost" to avoid the need to generate
          many separate certificates for local addresses of one machine and
          not to depend on which the local address is currently used in test
          (ipv4 or ipv6, etc.);
      10) In tests galera_sst_mariabackup_encrypt_with_key_server and
          galera_sst_rsync_encrypt_with_key_server the correct certificate
          is selected to avoid commonname (CN) mismatch problems;
      11) Further refactoring to protect against spaces in file names.
      12) Further general refactoring to eliminate bash-specific constructs
          or to improve code readability;
      13) The code for setting options for the nc (netcat) utility was
          different in different scripts for SST - now it is made identical.
      14) Fixed long-time broken encryption via xbcrypt in combination with
          mariabackup and added support for key-based encryption via openssl
          utility, which is now enabled by default for encrypt=1 mode (this
          default mode can be changed using a new configuration file option
          "encypt-format=openssl|xbcrypt", which can be placed in the [mysqld],
          [sst] or in the [xtrabackup] section) - this change will allow us
          to use and to test the encypt=1 encryption without installing
          non-standard third-party utilities.
      8fef2b86
    • Marko Mäkelä's avatar
      Merge 10.3 into 10.4 · 8c73fab7
      Marko Mäkelä authored
      8c73fab7
    • Marko Mäkelä's avatar
      Merge 10.2 into 10.3 · 98e61598
      Marko Mäkelä authored
      98e61598
  10. 09 May, 2021 1 commit
  11. 08 May, 2021 2 commits