1. 27 May, 2024 32 commits
    • Anson Chung's avatar
      Alter thr_timer to allow server startup past 2038 · dcbc526a
      Anson Chung authored
      In previous commits, changes were made to use the entire 32 bit
      unsigned range in order to support timestamps past 2038 on
      64 bit systems. Without changing set_max_time on thr_timer to also
      match this new range, the server will crash when attempting to
      startup past 2038-01-19. Like the previous commits, this only
      applies to 64 bit systems.
      
      All new code of the whole pull request, including one or several
      files that are either new files or modified ones, are contributed
      under the BSD-new license. I am contributing on behalf of my
      employer Amazon Web Services, Inc.
      dcbc526a
    • Sergei Golubchik's avatar
      84331c5b
    • Sergei Golubchik's avatar
      ER_VARIABLE_DELETED fix typos, adjust wording, fix plugins. · cc758332
      Sergei Golubchik authored
      plugins can have unused variables too. If they use a literal "Unused"
      string a compiler might or might not merge two identical strings into
      one (-fmerge-constants) and depending on that the server will or will
      not issue a "variable is ignored" warning.
      cc758332
    • Monty's avatar
      ae9a4799
    • Monty's avatar
      Added 'crash_error' to Aria for recording of reason for crash) · 9e7e1f62
      Monty authored
      This will allow us to in the future add better error messages why
      an Aria table is crashed.
      9e7e1f62
    • Monty's avatar
      ALTER TABLE and replication should convert old row_end timestamps to new timestamp range · 24c57165
      Monty authored
      MDEV-32188 make TIMESTAMP use whole 32-bit unsigned range
      
      - Added --update-history option to mariadb-dump to change 2038
        row_end timestamp to 2106.
      - Updated ALTER TABLE ... to convert old row_end timestamps to
        2106 timestamp for tables created before MariaDB 11.4.0.
      - Fixed bug in CHECK TABLE where we wrongly suggested to USE REPAIR
        TABLE when ALTER TABLE...FORCE is needed.
      - mariadb-check printed table names that where used with REPAIR TABLE but
        did not print table names used with ALTER TABLE or with name repair.
        Fixed by always printing a table that is fixed if --silent is not
        used.
      - Added TABLE::vers_fix_old_timestamp() that will change max-timestamp
        for versioned tables when replication from a pre-11.4.0 server.
      
      A few test cases changed. This is caused by:
      - CHECK TABLE now prints 'Please do ALTER TABLE... instead of
        'Please do REPAIR TABLE' when there is a problem with the information
        in the .frm file (for example a very old frm file).
      - mariadb-check now prints repaired table names.
      - mariadb-check also now prints nicer error message in case ALTER TABLE
        is needed to repair a table.
      24c57165
    • Monty's avatar
      MDEV-33449 improving repair of tables · c4cad8d5
      Monty authored
      This task is to ensure we have a clear definition and rules of how to
      repair or optimize a table.
      
      The rules are:
      
      - REPAIR should be used with tables that are crashed and are
        unreadable (hardware issues with not readable blocks, blocks with
        'unexpected data' etc)
      - OPTIMIZE table should be used to optimize the storage layout for the
        table (recover space for delete rows and optimize the index
        structure.
      - ALTER TABLE table_name FORCE should be used to rebuild the .frm file
        (the table definition) and the table (with the original table row
        format). If the table is from and older MariaDB/MySQL release with a
        different storage format, it will convert the data to the new
        format. ALTER TABLE ... FORCE is used as part of mariadb-upgrade
      
      Here follows some more background:
      
      The 3 ways to repair a table are:
      1) ALTER TABLE table_name FORCE" (not other options).
         As an alias we allow: "ALTER TABLE table_name ENGINE=original_engine"
      2) "REPAIR TABLE" (without FORCE)
      3) "OPTIMIZE TABLE"
      
      All of the above commands will optimize row space usage (which means that
      space will be needed to hold a temporary copy of the table) and
      re-generate all indexes. They will also try to replicate the original
      table definition as exact as possible.
      
      For ALTER TABLE and "REPAIR TABLE without FORCE", the following will hold:
      If the table is from an older MariaDB version and data conversion is
      needed (for example for old type HASH columns, MySQL JSON type or new
      TIMESTAMP format) "ALTER TABLE table_name FORCE, algorithm=COPY" will be
      used.
      
      The differences between the algorithms are
      1) Will use the fastest algorithm the engine supports to do a full repair
         of the table (except if data conversions are is needed).
      2) Will use the storage engine internal REPAIR facility (MyISAM, Aria).
         If the engine does not support REPAIR then
         "ALTER TABLE FORCE, ALGORITHM=COPY" will be used.
         If there was data incompatibilities (which means that FORCE was used)
         then there will be a warning after REPAIR that ALTER TABLE FORCE is
         still needed.
         The reason for this is that REPAIR may be able to go around data
         errors (wrong incompatible data, crashed or unreadable sectors) that
         ALTER TABLE cannot do.
      3) Will use the storage engine internal OPTIMIZE. If engine does not
         support optimize, then "ALTER TABLE FORCE" is used.
      
      The above will ensure that ALTER TABLE FORCE is able to
      correct almost any errors in the row or index data.  In case of
      corrupted blocks then REPAIR possible followed by ALTER TABLE is needed.
      This is important as mariadb-upgrade executes ALTER TABLE table_name
      FORCE for any table that must be re-created.
      
      Bugs fixed with InnoDB tables when using ALTER TABLE FORCE:
      - No error for INNODB_DEFAULT_ROW_FORMAT=COMPACT even if row length
        would be too wide. (Independent of innodb_strict_mode).
      - Tables using symlinks will be symlinked after any of the above commands
        (Independent of the setting of --symbolic-links)
      
      If one specifies an algorithm together with ALTER TABLE FORCE, things
      will work as before (except if data conversion is required as then
      the COPY algorithm is enforced).
      
      ALTER TABLE .. OPTIMIZE ALL PARTITIONS will work as before.
      
      Other things:
      - FORCE argument added to REPAIR to allow one to first run internal
        repair to fix damaged blocks and then follow it with ALTER TABLE.
      - REPAIR will not update frm_version if ha_check_for_upgrade() finds
        that table is still incompatible with current version. In this case the
        REPAIR will end with an error.
      - REPAIR for storage engines that does not have native repair, like InnoDB,
        is now using ALTER TABLE FORCE.
      - REPAIR csv-table USE_FRM now works.
        - It did not work before as CSV tables had extension list in wrong
          order.
      - Default error messages length for %M increased from 128 to 256 to not
        cut information from REPAIR.
      - Documented HA_ADMIN_XX variables related to repair.
      - Added HA_ADMIN_NEEDS_DATA_CONVERSION to signal that we have to
        do data conversions when converting the table (and thus ALTER TABLE
        copy algorithm is needed).
      - Fixed typo in error message (caused test changes).
      c4cad8d5
    • Monty's avatar
      MDEV-33655 Remove alter_algorithm · 2464ee75
      Monty authored
      Remove alter_algorithm but keep the variable as no-op (with a warning).
      
      The reasons for removing alter_algorithm are:
      - alter_algorithm was introduced as a replacement for the
        old_alter_table that was used to force the usage of the original
        alter table algorithm (copy) in the cases where the new alter
        algorithm did not work. The new option was added as a way to force
        the usage of a specific algorithm when it should instead have made
        it possible to disable algorithms that would not work for some
        reason.
      - alter_algorithm introduced some cases where ALTER TABLE would not
        work without specifying the ALGORITHM=XXX option together with
        ALTER TABLE.
      - Having different values of alter_algorithm on master and slave could
        cause slave to stop unexpectedly.
      - ALTER TABLE FORCE, as used by mariadb-upgrade, would not always work
        if alter_algorithm was set for the server.
      - As part of the MDEV-33449 "improving repair of tables" it become
        clear that alter- algorithm made it harder to provide a better and
        more consistent ALTER TABLE FORCE and REPAIR TABLE and it would be
        better to remove it.
      2464ee75
    • Monty's avatar
      Fix that one cannot change value for deleted variables · 6254fcf8
      Monty authored
      This enasures that the 'variables that only exists for compatibility'
      are always kept as its default value.
      
      Based on an idea from Sergei Golubchik
      6254fcf8
    • Monty's avatar
      Fixed warnings when using deprecated variables · 8af7a994
      Monty authored
      Also fixed that all unused variables are using the same variable comment.
      The warning will be tested with the next commit that deprecates the
      variable alter_algorithm.
      8af7a994
    • Monty's avatar
      Updated CONNECT engine to support date and timestamp for the full range. · c8624163
      Monty authored
      MDEV-32188 make TIMESTAMP use whole 32-bit unsigned range
      
      This was done by changing DTVAL to use longlong to internally store
      the date/timestamp instead of an int.
      
      I had to preserve the old binary representation used for dates in
      TABLE_TYPE=BIN. One consequence is that TABLE_TYPE=BIN cannot support
      the full date bit range.
      c8624163
    • Monty's avatar
      More windows changes for 32 bit unsigned timestamp: · b879b8a5
      Monty authored
      MDEV-32188 make TIMESTAMP use whole 32-bit unsigned range
      
      - Changed usage of timeval to my_timeval as the timeval parts on windows
        are 32-bit long, which causes some compiler issues on windows.
      b879b8a5
    • Monty's avatar
      Extends 64 bit windows to support timestamps up to year 2106. · b8ffd99c
      Monty authored
      MDEV-32188 make TIMESTAMP use whole 32-bit unsigned range
      
      This is done by changing my_time_t from long to unsigned long.
      The effect of this is that on windows compling old clients may
      get warnings of if they compare my_time_t with as signed variable.
      
      Other things
      - Removed my_time_t from include/*.pp files as it is different on windows
        and linux.
      - Changed do_abi_check.cmake to first print abi_check and then the
        conflicting file (this makes it easier to find the cause of the error).
      b8ffd99c
    • Monty's avatar
      MDEV-32188 make TIMESTAMP use whole 32-bit unsigned range · dfdedd46
      Monty authored
      This patch extends the timestamp from
      2038-01-19 03:14:07.999999 to 2106-02-07 06:28:15.999999
      for 64 bit hardware and OS where 'long' is 64 bits.
      This is true for 64 bit Linux but not for Windows.
      
      This is done by treating the 32 bit stored int as unsigned instead of
      signed.  This is safe as MariaDB has never accepted dates before the epoch
      (1970).
      The benefit of this approach that for normal timestamp the storage is
      compatible with earlier version.
      
      However for tables using system versioning we before stored a
      timestamp with the year 2038 as the 'max timestamp', which is used to
      detect current values.  This patch stores the new 2106 year max value
      as the max timestamp. This means that old tables using system
      versioning needs to be updated with mariadb-upgrade when moving them
      to 11.4. That will be done in a separate commit.
      dfdedd46
    • Monty's avatar
      MDEV-33620 Improve times and states in show processlist for replication · ce6cce85
      Monty authored
      - Slave_IO thread time is now reset between reading events. Before
        this commit Slave_IO always showed "Waiting for master to send
        event" and the time was from SLAVE START. Now it shows time since
        reading last event.
      ce6cce85
    • Monty's avatar
      Generate a warning(note) and write to error log if master_pos_wait() fails. · d2e6fe02
      Monty authored
      This is to make it easier to understand why master_pos_wait() fails in mtr.
      d2e6fe02
    • Monty's avatar
    • Sergei Golubchik's avatar
      cleanup: redundant condition · 201e28fa
      Sergei Golubchik authored
      201e28fa
    • Sergei Golubchik's avatar
    • Monty's avatar
      MDEV-33881 Userstat skips system tables inconsistently · 0f414f63
      Monty authored
      Fixed that no tables from 'mysql' schema are included in userstat.
      A beneif of this is that the server is not reading statistics tables
      if mysql.proc or other tables in mysql is accessed.
      0f414f63
    • Monty's avatar
      Optimize checking if a table is a statistics table · ab513b00
      Monty authored
      ab513b00
    • Monty's avatar
      MDEV-33152 Add QUERIES to INDEX_STATISTICS · a00e99ac
      Monty authored
      Other changes:
      - Do not collect index statistics for system tables like index_stats
        table_stats, performance_schema, information_schema etc as the user
        has no control of these and the generate noise in the statistics.
      a00e99ac
    • Sergei Golubchik's avatar
      cleanup: remove thd->stmt_changes_data · 869e67c9
      Sergei Golubchik authored
      what is done in the plugin - stays in the plugin
      869e67c9
    • Sergei Golubchik's avatar
      mark the deprecated sysvar deprecated · 3781848b
      Sergei Golubchik authored
      and adjust the copyright year
      3781848b
    • Elena Stepanova's avatar
    • Monty's avatar
      MDEV-33501 Extend query_response_time plugin to be compatible with Percona server · 243b9f3c
      Monty authored
      This is to update the plugin to be compatible with Percona's
      query_response_time plugin, with some additions.
      Some of the tests are taken from Percona server.
      
      - Added plugins QUERY_RESPONSE_TIME_READ, QUERY_RESPONSE_TIME_WRITE and
        QUERY_RESPONSE_TIME_READ_WRITE.
      - Added option query_response_time_session_stats, with possible values
        GLOBAL, ON or OFF, to the query_response_time plugin.
      
      Notes:
      - All modules are dependent on QUERY_RESPONSE_READ_TIME. This must always
        be enabled if any of the other modules are used.
        This will be auto-enabled in the near future.
      - Accounting are done per statement. Stored functions are regarded
        as part of the original statement.
      - For stored procedures the accounting are done per statement executed
        in the stored procedure. CALL will not be accounted because of this.
      - FLUSH commands will not be accounted for. This is to ensure that
        FLUSH QUERY_RESPONSE_TIME is not part of the statistics.
        (This helps when testing with mtr and otherwise).
      - FLUSH QUERY_RESPONSE_TIME_READ and FLUSH QUERY_RESPONSE_TIME_READ
        only resets the corresponding status.
      - FLUSH QUERY_RESPONSE_TIME and FLUSH QUERY_RESPONSE_TIME_READ_WRITE or
        changing the value of query_response_time_range_base followed by
        any FLUSH of QUERY_RESPOSNSE_TIME resets all status.
      243b9f3c
    • Sergei Golubchik's avatar
      MDEV-28671 post-testing fixes · 5296f908
      Sergei Golubchik authored
      Various help message improvements:
      * MySQL->MariaDB, mysqld->mariadbd, "mysqld daemon" -> "mariadbd process"
      * typos
      * don't specify defaults directly in the help message
      * don't say that an option is deprecated, mark is as such
      * missing spaces in the middle of the text
      etc
      5296f908
    • Sergei Golubchik's avatar
      cleanup: remove convert_dash_to_underscore · 53582238
      Sergei Golubchik authored
      it was a no-op, plugin variables don't have dashes
      53582238
    • Sergei Golubchik's avatar
      MDEV-28671 post-merge fixes · df10a945
      Sergei Golubchik authored
      * use new deprecated printer for all deprecated server options
      * restore alphabetic option sorting order
      * move deprecated printer from mysqld.cc to my_getopt.c
      * in --help print deprecation message at the end of the option help
      * move 'ALL' help text where it belongs - to other SET options, and
        with a correct indentation.
      * consistently end all or none command-line option help strings
        with a dot - my_print_help() needs that.
        It's about 50/50 now, so let's do none, less line wraps in --help
      * remove trailing spaces from command-line option help strings
      df10a945
    • Christian Gonzalez's avatar
      MDEV-28671 Enable var deprecation for mysqld help output · 4186fa72
      Christian Gonzalez authored
      Currently there are mechanism to mark a system variable as
      deprecated, but they are only used to print warning messages
      when a deprecated variable is set.
      
      Leverage the existing mechanisms in order to make the
      deprecation information available at the --help output of mysqld by:
      
      * Moving the deprecation information (i.e `deprecation_substitute`
        attribute) from the `sys_var` class into the `my_option` struct.
        As every `sys_var` contains its own `my_option` struct, the access
        to the deprecation information remains available to `sys_var`
        objects. `my_getotp` functions, which works directly with
        `my_option` structs, gain access to this information while building
        the --help output.
      
      * For plugin variables, leverages the `PLUGIN_VAR_DEPRECATED` flag
        and set the `deprecation_substitute` attribute  accordingly when
        building the `my_option` objects.
      
      * Change the `option_cmp` function to use the `deprecation_substitute`
        attribute instead of the name when sorting the options. This way
        deprecated options and the substitutes will be grouped together.
      
      All new code of the whole pull request, including one or several files
      that are either new files or modified ones, are contributed under the
      BSD-new license. I am contributing on behalf of my employer
      Amazon Web Services, Inc.
      4186fa72
    • Sergei Golubchik's avatar
      after merge fix · 25094f33
      Sergei Golubchik authored
      25094f33
    • Alexander Barkov's avatar
      MDEV-33696 main.dyncol and ctype_unicode_casefold_bmp.inc in --view · 78325a25
      Alexander Barkov authored
      The problem seems to be fixed earlier in the MDEV-25829 stage branch.
      Removing the --disable_view_protocol command.
      78325a25
  2. 26 May, 2024 2 commits
  3. 25 May, 2024 1 commit
  4. 24 May, 2024 5 commits
    • Alexander Barkov's avatar
      MDEV-33701 upgrades from 11.4 to 11.5 don't work · 283b9285
      Alexander Barkov authored
      A user variable and a literal with different collation
      produce an illegal mix of collation error.
      Rewriting the script to avoid such arguments.
      283b9285
    • Alexander Barkov's avatar
      MDEV-33698 tests in the jp suite fail · a0cde364
      Alexander Barkov authored
      a0cde364
    • Alexander Barkov's avatar
      MDEV-25829 Change default Unicode collation to uca1400_ai_ci · 903b5d6a
      Alexander Barkov authored
      Step#3 The main patch
      903b5d6a
    • Alexander Barkov's avatar
      MDEV-25829 Change default Unicode collation to uca1400_ai_ci · a3117c79
      Alexander Barkov authored
      Step#2 - Adding a new collation derivation level for CAST and CONVERT.
      
      Now character string cast functions:
        - CAST(string_expr AS CHAR)
        - CONVERT(expr USING charset_name)
      
      have a new collation derivation level between:
      
        - string literals
        - utf8 metadata functions, e.g. user() and database()
      
      Before the change these cast functions had collation derivation equal
      to table columns, which caused more illegal mix of collation conflicts.
      
      Note, binary string cast functions:
        - BINARY(expr)
        - CAST(string_expr AS BINARY)
        - CONVERT(expr USING binary)
      did not change their collation derivation, to preserve the behaviour of
      queries like these:
      SELECT database()=BINARY'test';
      SELECT user()=CAST('root' AS BINARY);
      SELECT current_role()=CONVERT('role' USING binary);
      
      Derivation levels after the change look as follows:
      
        DERIVATION_IGNORABLE= 7, // Explicit NULL
      
        DERIVATION_NUMERIC= 6,   // Numbers in string context,
                                 // Numeric user variables
                                 // CAST(numeric_expr AS CHAR)
      
        DERIVATION_COERCIBLE= 5, // Literals, string user variables
      
        DERIVATION_CAST= 4,      // CAST(string_expr AS CHAR),
                                 // CONVERT(string_expr USING cs)
      
        DERIVATION_SYSCONST= 3,  // utf8 metadata functions, e.g. user(), database()
        DERIVATION_IMPLICIT= 2,  // Table columns, SP variables, BINARY(expr)
        DERIVATION_NONE= 1,      // A mix (e.g. CONCAT) of two differrent collations
        DERIVATION_EXPLICIT= 0   // An explicit COLLATE clause
      a3117c79
    • Alexander Barkov's avatar
      MDEV-25829 Change default Unicode collation to uca1400_ai_ci · 1b65cc9d
      Alexander Barkov authored
      Step#1 - Changing collation derivation for string user variables
      from IMPLICIT to COERCIBLE.
      
      Retionale:
      
      Without this preparatory change, switching the default collation for
      Unicode character sets from xxx_general_ci to uca1400_ai_ci would cause
      "Illegal mix of collations" errors in scenarios comparing a column with
      a non-default collation to a string user variable
      
      This is especially important for queries to INFORMATION_SCHEMA tables,
      whose columns use utf8mb3_general_ci.
      
      See the description of MDEV-25829 for more details and SQL script examples.
      1b65cc9d