1. 17 May, 2021 12 commits
    • Julius Goryavsky's avatar
      MDEV-25693: SST failed due to incorrect connection address · 34340fb5
      Julius Goryavsky authored
      Fixed bugs caused by inaccuracies in automatic merging
      from other branches:
      
      1) Authentication information is not removed from the connection
         address, which causes some tests to fail;
      2) wsrep_debug=on should be replaced with wsrep_debug=1;
      3) Added missing "connection" lines to test result file;
      4) Some tests have been corrected for Galera 4.x (10.4+).
      34340fb5
    • Julius Goryavsky's avatar
      d45f61c3
    • Julius Goryavsky's avatar
      MDEV-25669: SST scripts should check all server groups in config files · 9e8e82cc
      Julius Goryavsky authored
      1) This commit implements reading all sections from configuration
      files while looking for the current value of any server variable,
      which were previously only read from the [mysqld.suffix] group and
      from [mysqld], but not from other groups such as [mariadb.suffix],
      [mariadb] or, for example, [server].
      
      2) This commit also fixes misrecognition of some parameters when
      parsing a command line containing a special marker for the end
      of the list of options ("--") or when short option names (such
      as "-s", "-a" and "-h arg") chained together (like a "-sah arg").
      Such parameters can be passed to the SST script in the list of
      arguments after "--mysqld-args" if the server is started with a
      complex set of options - this was revealed during manual testing
      of changes to read configuration files.
      
      3) The server-side preparation code for the "--mysqld-args"
      option list has also been simplified to make it easier to change
      in the future (if needed), and has been improved to properly
      handle the special backquote ("`") character in the argument
      values.
      9e8e82cc
    • Julius Goryavsky's avatar
    • Julius Goryavsky's avatar
      MDEV-23580: WSREP_SST: [ERROR] rsync daemon port has been taken · bcd6af93
      Julius Goryavsky authored
      This commit contains a large set of further bug fixes and
      improvements to SST scripts for Galera, continuing the work
      that was started in MDEV-24962 to make SST scripts work smoothly
      in different network configurations (especially using ipv6) and
      with different environment settings:
      
       1) The ipv6 addresses were incorrectly handled in the SST script
          for rsync (incorrect address substitution for establishing a
          connection, incorrect address substitution for bind, and so on);
       2) Checking the locality of the ip-address in SST scripts did not
          support ipv6 addresses (such as "[::1]"), which were falsely
          identified as non-local ip, which further did not allow running
          two SSTs on different local addresses on the same machine.
          On the other hand, this bug masked some other errors (related
          to handling ipv6 addresses);
       3) The code for checking the locality of the ip address was different
          in the SST scripts for rsync and for mysqldump, with individual
          flaws. This code is now made common and moved to wsrep_sst_common;
       4) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          did not process ipv6 addresses correctly in all cases (not for all
          branches);
       5) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) in the wait_for_listen() and check_pid_and_port() functions
          for some code branches could give a false positive result due to
          the textual match of prefixes in the port number and/or PID of
          the process;
       6) Waiting for the start of the transport channel (socat, nc, rsync,
          stunnel) was supported through different utilities in SST scripts
          for mariabackup and for rsync, and with various minor flaws in
          the code. Now the code is still different in these scripts, but
          it supports a common set of utilities (lsof, ss, sockstat) and
          is synchronized across patterns that used to check the output
          of  these utilities;
       7) In SST via mariabackup, the signal about readiness to receive data
          is sometimes sent too early - immediately after listen(), and not
          after accept() (which are called by socat or netcat utility).
       8) Checking availability of the some options of some utilities was
          done using the grep pattern, which easily gives false positives;
       9) Common name (CN) for local addresses, if not explicitly specified,
          is now always replaced to "localhost" to avoid the need to generate
          many separate certificates for local addresses of one machine and
          not to depend on which the local address is currently used in test
          (ipv4 or ipv6, etc.);
      10) In tests galera_sst_mariabackup_encrypt_with_key_server and
          galera_sst_rsync_encrypt_with_key_server the correct certificate
          is selected to avoid commonname (CN) mismatch problems;
      11) Further refactoring to protect against spaces in file names.
      12) Further general refactoring to eliminate bash-specific constructs
          or to improve code readability;
      13) The code for setting options for the nc (netcat) utility was
          different in different scripts for SST - now it is made identical.
      14) Fixed long-time broken encryption via xbcrypt in combination with
          mariabackup and added support for key-based encryption via openssl
          utility, which is now enabled by default for encrypt=1 mode (this
          default mode can be changed using a new configuration file option
          "encypt-format=openssl|xbcrypt", which can be placed in the [mysqld],
          [sst] or in the [xtrabackup] section) - this change will allow us
          to use and to test the encypt=1 encryption without installing
          non-standard third-party utilities.
      bcd6af93
    • Marko Mäkelä's avatar
      MDEV-24626 Remove synchronous write of page0 file during file creation · 86dc7b4d
      Marko Mäkelä authored
      During data file creation, InnoDB holds dict_sys mutex, tries to
      write page 0 of the file and flushes the file. This not only causing
      unnecessary contention but also a deviation from the write-ahead
      logging protocol.
      
      The clean sequence of operations is that we first start a dictionary
      transaction and write SYS_TABLES and SYS_INDEXES records that identify
      the tablespace. Then, we durably write a FILE_CREATE record to the
      write-ahead log and create the file.
      
      Recovery should not unnecessarily insist that the first page of each
      data file that is referred to by the redo log is valid. It must be
      enough that page 0 of the tablespace can be initialized based on the
      redo log contents.
      
      We introduce a new data structure deferred_spaces that keeps track
      of corrupted-looking files during recovery. The data structure holds
      the last LSN of a FILE_ record referring to the data file, the
      tablespace identifier, and the last known file name.
      
      There are two scenarios can happen during recovery:
      i) Sufficient memory: InnoDB can reconstruct the
      tablespace after parsing all redo log records.
      
      ii) Insufficient memory(multiple apply phase): InnoDB should
      store the deferred tablespace redo logs even though
      tablespace is not present. InnoDB should start constructing
      the tablespace when it first encounters deferred tablespace
      id.
      
      Mariabackup copies the zero filled ibd file in backup_fix_ddl() as
      the extension of .new file. Mariabackup test case does page flushing
      when it deals with DDL operation during backup operation.
      
      fil_ibd_create(): Remove the write of page0 and flushing of file
      
      fil_ibd_load(): Return FIL_LOAD_DEFER if the tablespace has
      zero filled page0
      
      Datafile: Clean up the error handling, and do not report errors
      if we are in the middle of recovery. The caller will check
      Datafile::m_defer.
      
      fil_node_t::deferred: Indicates whether the tablespace loading was
      deferred during recovery
      
      FIL_LOAD_DEFER: Returned by fil_ibd_load() to indicate that tablespace
      file was cannot be loaded.
      
      recv_sys_t::recover_deferred(): Invoke deferred_spaces.create() to
      initialize fil_space_t based on buffered metadata and records to
      initialize page 0. Ignore the flags in fil_name_t, because they are
      intentionally invalid.
      
      fil_name_process(): Update deferred_spaces.
      
      recv_sys_t::parse(): Store the redo log if the tablespace id
      is present in deferred spaces
      
      recv_sys_t::recover_low(): Should recover the first page of
      the tablespace even though the tablespace instance is not
      present
      
      recv_sys_t::apply(): Initialize the deferred tablespace
      before applying the deferred tablespace records
      
      recv_validate_tablespace(): Skip the validation for deferred_spaces.
      
      recv_rename_files(): Moved and revised from recv_sys_t::apply().
      For deferred-recovery tablespaces, do not attempt to rename the
      file if a deferred-recovery tablespace is associated with the name.
      
      recv_recovery_from_checkpoint_start(): Invoke recv_rename_files()
      and initialize all deferred tablespaces before applying redo log.
      
      fil_node_t::read_page0(): Skip page0 validation if the tablespace
      is deferred
      
      buf_page_create_deferred(): A variant of buf_page_create() when
      the fil_space_t is not available yet
      
      This is joint work with Thirunarayanan Balathandayuthapani,
      who implemented an initial prototype.
      86dc7b4d
    • Marko Mäkelä's avatar
      Simplify a preprocessor condition · c290c0d7
      Marko Mäkelä authored
      Compilers older than GCC 4.8 are not supported anyway.
      c290c0d7
    • Sujatha's avatar
      MDEV-19371: Implement binlog_expire_logs_seconds for purging of binary logs · 7e1ec155
      Sujatha authored
      Post push fixes.
      
      * Set binlog_expire_logs_seconds max valid value to 8553600
        i.e 99 * 24 * 60 * 60 (99 days)
      7e1ec155
    • Marko Mäkelä's avatar
      MDEV-25689: Remove MONITOR_TABLE_REFERENCE, MONITOR_TABLE_CLOSE · 1b1882c6
      Marko Mäkelä authored
      The counter metadata_table_reference_count was not updated consistently
      ever since mysql-server@65c0af9a1dedae43b63797134aff6b32304ced52
      or commit 2e814d47
      introduced dict_table_t::release().
      
      The counter metadata_table_handles_closed was being incremented
      unconditionally in ha_innobase::close(), while the corresponding
      counter metadata_table_handles_opened would be incremented in
      ha_innobase::open() if the function returned early due to an error.
      1b1882c6
    • Marko Mäkelä's avatar
      Cleanup: Remove unused DB_TABLE_IS_BEING_USED · adb0fdb2
      Marko Mäkelä authored
      The error code DB_TABLE_IS_BEING_USED was orphaned in
      commit ddc663ef (MySQL 3.23.37).
      Let us finally remove it.
      adb0fdb2
    • Marko Mäkelä's avatar
      MDEV-25687: Remove trx_active_transactions · a6cff02a
      Marko Mäkelä authored
      MONITOR_TRX_ACTIVE: Remove. The count is not being updated consistently,
      and it would also include read-only transactions that are otherwise
      fully invisible to any other threads.
      
      If it later turns out that a reliable count of active transactions
      is needed, it can be exposed via a different interface.
      
      trx_commit_for_mysql(): If the transaction was not started, return
      immediately.
      a6cff02a
    • Marko Mäkelä's avatar
      MDEV-18518 fixup: Remove row_drop_table_after_create_fail() · cdbd04dd
      Marko Mäkelä authored
      row_drop_table_after_create_fail(): Remove. This function was only used
      during InnoDB initialization (or upgrade) when creating a system table
      failed. Rollback will clean up a failed CREATE just fine, by invoking
      dict_drop_index_tree().
      cdbd04dd
  2. 15 May, 2021 1 commit
    • Jan Lindström's avatar
      MDEV-9609 : wsrep_debug only logs DDL information on originating node · bee1bb05
      Jan Lindström authored
      Added DDL logging to applier and replaying also so that
      DDL is logged on other than originating node.
      
      wsrep.h
      	Removed wsrep_thd_is_local conditions and cleaned up
      	the macros. Removed WSREP_TO_ISOLATION_END.
      
      Event_job_data::execute
      change_password
      acl_set_default_role
      mysql_execute_command
      	Replaced macro by function call
      
      wsrep_to_isolation_begin
      wsrep_to_isolation_end
      	If execution is not local log DDL-information when
      	wsrep_debug is enabled
      
      No new tests required as current regression setting is
      already testing these code paths.
      bee1bb05
  3. 14 May, 2021 4 commits
  4. 13 May, 2021 3 commits
    • Sergei Petrunia's avatar
      MDEV-25154: JSON_TABLE: Queries involving ordinality columns are unsafe ... · 4547c6f2
      Sergei Petrunia authored
      Mark the JSON_TABLE function as SBR-unsafe.
      It is not unsafe for the current implementation. But we still mark it as such
      in order to be future-proof and keep it possible to change JSON data
      representation in the future.
      4547c6f2
    • Marko Mäkelä's avatar
      fixup e3b33842: wsrep test results · 916c28c9
      Marko Mäkelä authored
      916c28c9
    • Sujatha's avatar
      MDEV-25502: rpl.rpl_perfschema_applier_status_by_worker failed in bb with: Test assertion failed · fe945067
      Sujatha authored
      Problem:
      =======
      Test fails with 3 different symptoms
      connection slave;
      Assertion text: 'Last_Seen_Transaction should show .'
      Assertion condition: '"0-1-1" = ""'
      Assertion condition, interpolated: '"0-1-1" = ""'
      Assertion result: '0'
      
      connection slave;
      Assertion text: 'Value returned by SSS and PS table for Last_Error_Number
                       should be same.'
      Assertion condition: '"1146" = "0"'
      Assertion condition, interpolated: '"1146" = "0"'
      Assertion result: '0'
      
      connection slave;
      Assertion text: 'Value returned by PS table for worker_idle_time should be
                      >= 1'
      Assertion condition: '"0" >= "1"'
      Assertion condition, interpolated: '"0" >= "1"'
      Assertion result: '0'
      
      Fix1:
      ====
      Performance schema table's Last_Seen_Transaction is compared with 'SELECT
      gtid_slave_pos'. Since DDLs are not transactional changes to user table and
      gtid_slave_pos table are not guaranteed to be synchronous. To fix the
      issue Gtid_IO_Pos value from SHOW SLAVE STATUS command will be used to
      verify the correctness of Performance schema specific
      Last_Seen_Transaction.
      
      Fix2:
      ====
      On error worker thread information is stored as part of backup pool. Access
      to this backup pool should be protected by 'LOCK_rpl_thread_pool' mutex so
      that simultaneous START SLAVE cannot destroy the backup pool, while it is
      being queried by performance schema.
      
      Fix3:
      ====
      When a worker is waiting for events if performance schema table is queried,
      at present it just returns the difference between current_time and
      start_time.  This is incorrect. It should be worker_idle_time +
      (current_time - start_time).
      
      For example a worker thread was idle for 10 seconds and then it got events
      to process. Upon completion it goes to idle state, now if the pfs table is
      queried it should return current_idle time  + worker_idle_time.
      fe945067
  5. 12 May, 2021 3 commits
    • Nikita Malyavin's avatar
      tztime_to_sql: quote `Offset` field · e3b33842
      Nikita Malyavin authored
      e3b33842
    • Kentoku SHIBA's avatar
      Fix the following valgrind error on Spider · 982290fe
      Kentoku SHIBA authored
      ==22532== Conditional jump or move depends on uninitialised value(s)
      ==22532==    at 0x9EEDFA: String::append(char const*, unsigned long, charset_info_st const*) (sql_string.cc:587)
      ==22532==    by 0x17C12BA7: spider_string::append(char const*, unsigned int, charset_info_st const*) (spd_malloc.cc:913)
      ==22532==    by 0x17C68BD0: spider_db_mysql_util::append_column_value(ha_spider*, spider_string*, Field*, unsigned char const*, charset_info_st const*) (spd_db_mysql.cc:4573)
      ==22532==    by 0x17B7D298: spider_db_append_key_where_internal(spider_string*, spider_string*, spider_string*, st_key_range const*, st_key_range const*, ha_spider*, bool, unsigned long, unsigned int) (spd_db_conn.cc:1978)
      ==22532==    by 0x17C81A7C: spider_mbase_handler::append_key_where(spider_string*, spider_string*, spider_string*, st_key_range const*, st_key_range const*, unsigned long, bool) (spd_db_mysql.cc:11146)
      ==22532==    by 0x17C819B7: spider_mbase_handler::append_key_where_part(st_key_range const*, st_key_range const*, unsigned long) (spd_db_mysql.cc:11130)
      ==22532==    by 0x17C4B0F0: ha_spider::append_key_where_sql_part(st_key_range const*, st_key_range const*, unsigned long) (ha_spider.cc:15012)
      ==22532==    by 0x17B7FC8F: spider_db_append_key_where(st_key_range const*, st_key_range const*, ha_spider*) (spd_db_conn.cc:2707)
      ==22532==    by 0x17C28DF6: ha_spider::multi_range_read_next_first(void**) (ha_spider.cc:4559)
      ==22532==    by 0x17C2E0F5: ha_spider::multi_range_read_next(void**) (ha_spider.cc:5879)
      ==22532==    by 0xE27019: QUICK_RANGE_SELECT::get_next() (opt_range.cc:12647)
      
      Conflicts:
      	storage/spider/spd_malloc.cc
      982290fe
    • Marko Mäkelä's avatar
      Merge 10.5 into 10.6 · 370b310b
      Marko Mäkelä authored
      370b310b
  6. 11 May, 2021 4 commits
  7. 10 May, 2021 8 commits
  8. 09 May, 2021 5 commits