1. 22 May, 2021 3 commits
  2. 21 May, 2021 4 commits
    • Igor Babaev's avatar
      MDEV-23886 Reusing CTE inside a function fails with table doesn't exist · 43c9fcef
      Igor Babaev authored
      In the code existed just before this patch binding of a table reference to
      the specification of the corresponding CTE happens in the function
      open_and_process_table(). If the table reference is not the first in the
      query the specification is cloned in the same way as the specification of
      a view is cloned for any reference of the view. This works fine for
      standalone queries, but does not work for stored procedures / functions
      for the following reason.
      When the first call of a stored procedure/ function SP is processed the
      body of SP is parsed. When a query of SP is parsed the info on each
      encountered table reference is put into a TABLE_LIST object linked into
      a global chain associated with the query. When parsing of the query is
      finished the basic info on the table references from this chain except
      table references to derived tables and information schema tables is put
      in one hash table associated with SP. When parsing of the body of SP is
      finished this hash table is used to construct TABLE_LIST objects for all
      table references mentioned in SP and link them into the list of such
      objects passed to a pre-locking process that calls open_and_process_table()
      for each table from the list.
      When a TABLE_LIST for a view is encountered the view is opened and its
      specification is parsed. For any table reference occurred in
      the specification a new TABLE_LIST object is created to be included into
      the list for pre-locking. After all objects in the pre-locking have been
      looked through the tables mentioned in the list are locked. Note that the
      objects referenced CTEs are just skipped here as it is impossible to
      resolve these references without any info on the context where they occur.
      Now the statements from the body of SP are executed one by one that.
      At the very beginning of the execution of a query the tables used in the
      query are opened and open_and_process_table() now is called for each table
      reference mentioned in the list of TABLE_LIST objects associated with the
      query that was built when the query was parsed.
      For each table reference first the reference is checked against CTEs
      definitions in whose scope it occurred. If such definition is found the
      reference is considered resolved and if this is not the first reference
      to the found CTE the the specification of the CTE is re-parsed and the
      result of the parsing is added to the parsing tree of the query as a
      sub-tree. If this sub-tree contains table references to other tables they
      are added to the list of TABLE_LIST objects associated with the query in
      order the referenced tables to be opened. When the procedure that opens
      the tables comes to the TABLE_LIST object created for a non-first
      reference to a CTE it discovers that the referenced table instance is not
      locked and reports an error.
      Thus processing non-first table references to a CTE similar to how
      references to view are processed does not work for queries used in stored
      procedures / functions. And the main problem is that the current
      pre-locking mechanism employed for stored procedures / functions does not
      allow to save the context in which a CTE reference occur. It's not trivial
      to save the info about the context where a CTE reference occurs while the
      resolution of the table reference cannot be done without this context and
      consequentially the specification for the table reference cannot be
      determined.
      
      This patch solves the above problem by moving resolution of all CTE
      references at the parsing stage. More exactly references to CTEs occurred in
      a query are resolved right after parsing of the query has finished. After
      resolution any CTE reference it is marked as a reference to to derived
      table. So it is excluded from the hash table created for pre-locking used
      base tables and view when the first call of a stored procedure / function
      is processed.
      This solution required recursive calls of the parser. The function
      THD::sql_parser() has been added specifically for recursive invocations of
      the parser.
      43c9fcef
    • Marko Mäkelä's avatar
      MDEV-25664 Potential hang in purge for virtual columns · 9739cf18
      Marko Mäkelä authored
      ha_innobase::open(): If the table is only being opened by purge
      for evaluating virtual column values, avoid invoking
      initialize_auto_increment(), because the purge thread may already
      be holding an shared latch on the clustered index root page.
      Shared latches are not recursive. The additional request would lead
      to a hang if another thread has started waiting for an exclusive latch.
      9739cf18
    • Sergei Petrunia's avatar
      MDEV-22462: Item_in_subselect::create_single_in_to_exists_cond(JOIN *, Item... · 2087d47a
      Sergei Petrunia authored
      MDEV-22462: Item_in_subselect::create_single_in_to_exists_cond(JOIN *, Item **, Item **): Assertion `false' failed.
      
      Item_in_subselect::create_single_in_to_exists_cond() should handle the
      case where the subquery is a table-less select but it is not a result
      of a UNION.
      
      (Table-less subqueries like "(SELECT 1)" are "substituted" with their select
      list, but table-less subqueries with WHERE or HAVING clause, like
      "(SELECT 1 WHERE ...)" are not substituted. They are handled with regular
      execution path)
      2087d47a
    • Julius Goryavsky's avatar
      MDEV-25719: stunnel uses "verifyChain" without subject checks · 8c8a6ed3
      Julius Goryavsky authored
      Another batch of changes that should make the SST process
      more reliable in all scenarios:
      
       1) Added hostname or CN verification when stunnel is used
          with certificate chain verification (verifyChain = yes);
       2) Added check for the absence of the stunnel utility for
          mtr tests;
       3) Deletion of working files before and after SST is done
          more accurately;
       4) rsync on joiner can be run even if the path to its
          configuration file contains spaces;
       5) More accurate directory creation (for data files and
          for logs);
       6) IST with mysqldump no longer turns off statement logging;
       7) Reset password for mysqldump when password is empty but
          username is specified;
       8) More reliable quoting when generating statements in
          wsrep_sst_mysqldump;
       9) Added explicit generation of 2048-bit Diffie-Hellman
          parameters for sockat < 1.7.3, by analogy with xtrabackup;
      10) Compression parameters for qpress are read from all
          suitable server groups in configuration file, as well as
          from the [sst] and [xtrabackup] groups;
      11) Added a test that checks compression using qpress;
      12) Checking for optional utilities is modified to work even
          if they implemented as built-in shell commands (unlikely
          on real systems, but more reliable).
      8c8a6ed3
  3. 20 May, 2021 1 commit
    • Rucha Deodhar's avatar
      MDEV-25462: Assertion `m_status == DA_ERROR || m_status == DA_OK || · 62944917
      Rucha Deodhar authored
      m_status == DA_OK_BULK' failed in Diagnostics_area::message from
      get_schema_tables_record
      
      Analysis: SET NAMES changes character set for character_set_client,
      character_set_connection, character_set_results to 'filename'. The .frm file of view
      has @xx sequences in the SELECT query, which give parsing error because 'filename'
      character set is not parser friendly. When we get parsing error (ER_PARSE_ERROR), we
      directly return true without setting error status. This is caught later in assertion.
      Fix: Disallow 'filename' character set in SET NAMES because it is not parser
      friendly.
      62944917
  4. 19 May, 2021 3 commits
  5. 18 May, 2021 2 commits
  6. 17 May, 2021 3 commits
    • Julius Goryavsky's avatar
      23cad4d8
    • Sujatha's avatar
      MDEV-22530: Aborting OPTIMIZE TABLE still logs in binary log and replicates to the Slave server. · 88c7a58e
      Sujatha authored
      Problem:
      ========
      Aborting OPTIMIZE TABLE still logs in binary logs and replicates to the
      Slave server. "Optimize table" command under execution, is killed by using
      "Ctrl-C" as shown below.
      
      MariaDB [test]> optimize table t2;
      ^CCtrl-C -- query killed. Continuing normally.
      
      In spite of query execution being interrupted the query gets written to
      binary log.
      
      Analysis:
      ========
      Admin command execution logic is not handling KILL command, hence it
      ignores the KILL command and completes its execution.
      
      Fix:
      ===
      Check for thread killed notification, during admin command execution and
      handle it. If thread kill occurs prior to any table modification the query
      will not be written to binary log. If kill happens after at least one table
      is modified then the query will be written to binary log. Ex: command in
      execution is 'OPTIMIZE TABLE t1,t2' and the thread kill happens after t1
      table is modified then 'OPTIMIZE TABLE t1,t2' will be written to binary log
      as admin commands will not make the slave to diverge from master.
      88c7a58e
    • Sujatha's avatar
      MDEV-17515: GTID Replication in optimistic mode deadlock · 410e3c1a
      Sujatha authored
      Problem:
      =======
      In slave_parallel_mode=optimistic configuration, when admin commands and
      DML operation on the same table are scheduled simultaneously for execution,
      it results in lock conflict and slave server either hangs due to
      deadlock or goes down with an assert.
      
      Analysis:
      ========
      Admin commands OPTIMIZE, REPAIR and ANALYZE are written to binary log as
      ordinary transactions. When 'slave_parallel_mode' is 'optimistic' DMLs are
      allowed to run in parallel. But these locks are not detected by parallel
      replication deadlock detection-and-handling mechanism. At times they result
      in deadlock or assertion.
      
      Fix:
      ===
      Flag admin commands as DDL in Gtid_log_event at the time of writing to
      binary log. Add a new bit EXECUTED_TABLE_ADMIN_CMD to
      'm_unsafe_rollback_flags'. During 'mysql_admin_table' command execution it
      accepts a list of tables to be processed and executes them in a loop. Upon
      successful execution enable 'EXECUTED_TABLE_ADMIN_CMD' bit in
      thd->transaction.stmt_unsafe_rollback_flags. Gtid_log_event constructor
      will notice this flag and mark the current transaction with 'FL_DDL' flag.
      Gtid_log_events marked as FL_DDL will not be scheduled parallel execution,
      on the slave. They will execute in isolation to prevent deadlocks.
      
      Note: Removed the call to 'trans_commit_implicit' from 'mysql_admin_table'
      function as 'mysql_execute_command' will take care of invoking
      'trans_commit_implicit'.
      410e3c1a
  7. 16 May, 2021 1 commit
  8. 15 May, 2021 2 commits
    • Julius Goryavsky's avatar
      MDEV-25669: SST scripts should check all server groups in config files · 6811ed3e
      Julius Goryavsky authored
      1) This commit implements reading all sections from configuration
      files while looking for the current value of any server variable,
      which were previously only read from the [mysqld.suffix] group and
      from [mysqld], but not from other groups such as [mariadb.suffix],
      [mariadb] or, for example, [server].
      
      2) This commit also fixes misrecognition of some parameters when
      parsing a command line containing a special marker for the end
      of the list of options ("--") or when short option names (such
      as "-s", "-a" and "-h arg") chained together (like a "-sah arg").
      Such parameters can be passed to the SST script in the list of
      arguments after "--mysqld-args" if the server is started with a
      complex set of options - this was revealed during manual testing
      of changes to read configuration files.
      
      3) The server-side preparation code for the "--mysqld-args"
      option list has also been simplified to make it easier to change
      in the future (if needed), and has been improved to properly
      handle the special backquote ("`") character in the argument
      values.
      6811ed3e
    • Julius Goryavsky's avatar
      4675febb
  9. 14 May, 2021 4 commits
    • Igor Babaev's avatar
      MDEV-25682 Explain shows an execution plan different from actually executed · 677f1ef6
      Igor Babaev authored
      If a select query contained an ORDER BY clause that followed a LIMIT clause
      or an ORDER BY clause or ORDER BY with LIMIT the EXPLAIN output for the
      query showed an execution plan different from that was actually executed.
      
      Approved by Roman Nozdrin <roman.nozdrin@mariadb.com>
      677f1ef6
    • Sachin Kumar's avatar
      MDEV-25336 Parallel replication causes failed assert while restarting · e607f339
      Sachin Kumar authored
      Problem:- When slave is shutdown, we will get this assertion failure
      sql/sql_list.h:642: void ilink::assert_linked(): Assertion `prev != 0
      && next != 0' failed.
      
      Solution:- In close_connections when we call threads.get() it resets to
      prev and next to NULL. And in parallel worker thread(handle_rpl_parallel_thread)
      calls unlink_not_visible_thd() which assert on prev and next being not NULL.
      .unlink_not_visible_thd() should be always called first before threads.get()
      is called. To make sure worker calls unlink_not_visible_thd() in
      slave_prepare_for_shutdown() we are deactivating the  worker thread pool
      which in turn will close all worker threads. Since this is already done in 10.4
      and 10.5 I am backPorting MDEV-20821 and MDEV-22370 to 10.2. Mdev-22370
      is improving the MDEV-20821 patch.
      e607f339
    • Sachin Kumar's avatar
      MDEV-22370 safe_mutex: Trying to lock uninitialized mutex at... · 355dc74b
      Sachin Kumar authored
      MDEV-22370 safe_mutex: Trying to lock uninitialized mutex at /data/src/10.4-bug/sql/rpl_parallel.cc, line 470 upon shutdown during FTWRL
      
      Problem:- When we issue FTWRL with shutdown in parallel, there is race between
      FTWRL and shutdown. Shutdown might destroy the mutex (pool->LOCK_rpl_thread_pool)
      before FTWRL can lock it. So we can get crash on FTWRL thread
      
      Solution:- mysql_mutex_destroy(pool->LOCK_rpl_thread_pool) should wait for
      FTWRL thread to complete its work , and then destroy.
      So slave_prepare_for_shutdown will just deactivate the pool, and mutex is destroyed
      later in end_slave()
      355dc74b
    • Andrei Elkin's avatar
      MDEV-20821 parallel slave server shutdown hang · 3616640a
      Andrei Elkin authored
      Parallel slave server shutdown found to be hanging in
      close_connections() triggered by shutdown due to a slave worker thread
      would not be notified to exit in case the worker was sitting idle.
      
      Fixed with destroying the worker pool earlier that is in
      slave_prepare_for_shutdown() when all their driver threads have already left.
      A test file is added to simulate the bug condition as well as check
      multi-sourced and not-idle worker cases.
      3616640a
  10. 11 May, 2021 2 commits
  11. 10 May, 2021 2 commits
    • Daniel Bartholomew's avatar
      bump the VERSION · 089d82a7
      Daniel Bartholomew authored
      089d82a7
    • Julius Goryavsky's avatar
      MDEV-23580: WSREP_SST: [ERROR] rsync daemon port has been taken · 8fef2b86
      Julius Goryavsky authored
      This commit contains a large set of further bug fixes and
      improvements to SST scripts for Galera, continuing the work
      that was started in MDEV-24962 to make SST scripts work smoothly
      in different network configurations (especially using ipv6) and
      with different environment settings:
      
       1) The ipv6 addresses were incorrectly handled in the SST script
          for rsync (incorrect address substitution for establishing a
          connection, incorrect address substitution for bind, and so on);
       2) Checking the locality of the ip-address in SST scripts did not
          support ipv6 addresses (such as "[::1]"), which were falsely
          identified as non-local ip, which further did not allow running
          two SSTs on different local addresses on the same machine.
          On the other hand, this bug masked some other errors (related
          to handling ipv6 addresses);
       3) The code for checking the locality of the ip address was different
          in the SST scripts for rsync and for mysqldump, with individual
          flaws. This code is now made common and moved to wsrep_sst_common;
       4) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          did not process ipv6 addresses correctly in all cases (not for all
          branches);
       5) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          for some code branches could give a false positive result due to
          the textual match of prefixes in the port number and/or PID of
          the process;
       6) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) was supported through different utilities in SST scripts
          for mariabackup and for rsync, and with various minor flaws in
          the code. Now the code is still different in these scripts, but
          it supports a common set of utilities (lsof, ss, sockstat) and
          is synchronized across patterns that used to check the output
          of  these utilities;
       7) In SST via mariabackup, the signal about readiness to receive data
          is sometimes sent too early - immediately after listen(), and not
          after accept() (which are called by socat or netcat utility).
       8) Checking availability of the some options of some utilities was
          done using the grep pattern, which easily gives false positives;
       9) Common name (CN) for local addresses, if not explicitly specified,
          is now always replaced to "localhost" to avoid the need to generate
          many separate certificates for local addresses of one machine and
          not to depend on which the local address is currently used in test
          (ipv4 or ipv6, etc.);
      10) In tests galera_sst_mariabackup_encrypt_with_key_server and
          galera_sst_rsync_encrypt_with_key_server the correct certificate
          is selected to avoid commonname (CN) mismatch problems;
      11) Further refactoring to protect against spaces in file names.
      12) Further general refactoring to eliminate bash-specific constructs
          or to improve code readability;
      13) The code for setting options for the nc (netcat) utility was
          different in different scripts for SST - now it is made identical.
      14) Fixed long-time broken encryption via xbcrypt in combination with
          mariabackup and added support for key-based encryption via openssl
          utility, which is now enabled by default for encrypt=1 mode (this
          default mode can be changed using a new configuration file option
          "encypt-format=openssl|xbcrypt", which can be placed in the [mysqld],
          [sst] or in the [xtrabackup] section) - this change will allow us
          to use and to test the encypt=1 encryption without installing
          non-standard third-party utilities.
      8fef2b86
  12. 09 May, 2021 1 commit
  13. 08 May, 2021 5 commits
  14. 07 May, 2021 3 commits
  15. 06 May, 2021 1 commit
    • Alexey Yurchenko's avatar
      MDEV-25418: Improve mariabackup SST script compliance with native MariaDB SSL practices · 54d7ba96
      Alexey Yurchenko authored
      and configuration.
      
      1. Pass joiner's authentication information to donor together with address
         in State Transfer Request. This allows joiner to authenticate donor on
         connection. Previously joiner would accept data from anywhere.
      
      2. Deprecate custom SSL configuration variables tca, tcert and tkey in favor
         of more familiar ssl-ca, ssl-cert and ssl-key. For backward compatibility
         tca, tcert and tkey are still supported.
      
      3. Allow falling back to server-wide SSL configuration in [mysqld] if no SSL
         configuration is found in [sst] section of the config file.
      
      4. Introduce ssl-mode variable in [sst] section that takes standard values
         and has following effects:
          - old-style SSL configuration present in [sst]: no effect
            otherwise:
          - ssl-mode=DISABLED or absent: retains old, backward compatible behavior
            and ignores any other SSL configuration
          - ssl-mode=VERIFY*: verify joiner's certificate and CN on donor,
                              verify donor's secret on joiner
                              (passed to donor via State Transfer Request)
                              BACKWARD INCOMPATIBLE BEHAVIOR
          - anything else enables new SSL configuration convetions but does not
            require verification
      
          ssl-mode should be set to VERIFY only in a fully upgraded cluster.
      
          Examples:
      
          [mysqld]
          ssl-cert=/path/to/cert
          ssl-key=/path/to/key
          ssl-ca=/path/to/ca
      
          [sst]
      
           -- server-wide SSL configuration is ignored, SST does not use SSL
      
          [mysqld]
          ssl-cert=/path/to/cert
          ssl-key=/path/to/key
          ssl-ca=/path/to/ca
      
          [sst]
          ssl-mode=REQUIRED
      
           -- use server-wide SSL configuration for SST but don't attempt to
              verify the peer identity
      
          [sst]
          ssl-cert=/path/to/cert
          ssl-key=/path/to/key
          ssl-ca=/path/to/ca
          ssl-mode=VERIFY_CA
      
           -- use SST-specific SSL configuration for SST and require verification
              on both sides
      Signed-off-by: default avatarJulius Goryavsky <julius.goryavsky@mariadb.com>
      54d7ba96
  16. 05 May, 2021 3 commits
    • Alexey Yurchenko's avatar
      MDEV-25418 rsync SST does not work with stunnel encryption · cf67ca48
      Alexey Yurchenko authored
      1. Fix eval command line to correctly pass stunnel option to rsync on donor.
      2. Deprecate `tkey`, `tcert` and `tca` options in [sst] section in favor of
         conventional `ssl-key`, `ssl-cert` and `ssl-ca`, but keep their precedence
         for backward compatibility.
      3. Default to require SSL encryption if at least SSL key and cert files are
         specified in configuration, either in [sst] or [mysqld] sections.
      4. Enable `verify*` option for stunnel on donor only if
         a. CA file is specified somewhere in the configuration
         b. it is explicitly requested in [sst] section by either specifying
         ssl-mode or CA file there. In this case if ssl-mode is not explicitly
         given, it defaults to VERIFY_CA.
      
         ssl-mode maps to stunnel options as follows:
         VERIFY_CA       -> verifyChain = yes
         VERIFY_IDENTITY -> verifyPeer = yes
      
         Example to require donor to verify joiner identity:
      ```
      [mysqld]
      ssl-cert=/path/to/cert
      ssl-key=/path/to/key
      ssl-ca=/path/to/ca
      
      [sst]
      ssl-mode=VERIFY_IDENTITY
      ```
      5. If SSL verification is requested, joiner verifies donor by checking the
         secret passed to donor via SST request.
      Signed-off-by: default avatarJulius Goryavsky <julius.goryavsky@mariadb.com>
      cf67ca48
    • Julius Goryavsky's avatar
    • Julius Goryavsky's avatar