1. 11 Oct, 2018 1 commit
    • Marko Mäkelä's avatar
      Revert "WIP: Add row_log_t::instant" · 008815f5
      Marko Mäkelä authored
      This reverts commit 26897301
      and adds some changes.
      
      Only a table on which the 10.3 ADD COLUMN...LAST was performed
      can "lose its instantness" when it becomes empty. After
      instant DROP COLUMN or reordering columns, we cannot simply
      convert the table to the canonical format, because the data
      dictionary cache and all possibly existing references to it
      from other client connection threads would have to be adjusted.
      
      rec_init_offsets_comp_ordinary(): Remove a FIXME comment.
      
      row_log_allocate(): Replace dict_index_t::is_instant()
      with a more appropriate condition that ignores dict_table_t::instant.
      008815f5
  2. 10 Oct, 2018 15 commits
    • Marko Mäkelä's avatar
      5912f133
    • Marko Mäkelä's avatar
      Avoid losing precision on 64-bit Windows · d03446c9
      Marko Mäkelä authored
      d03446c9
    • Marko Mäkelä's avatar
      MDEV-15562: Remove dict_table_t::rollback_instant(unsigned n) · f545e3cf
      Marko Mäkelä authored
      On the rollback of changes to SYS_COLUMNS, MDEV-15562 will
      break the assumption that the only instantaneous changes to columns
      are the addition to the end of the column list.
      
      The function dict_table_t::rollback_instant(unsigned n)
      is inherently incompatible with instantly dropping or reordering
      columns.
      
      When a change to SYS_COLUMNS is rolled back, we must simply evict
      the affected table definition, at the end of the rollback. We cannot
      free the table object immediately, because the current transaction
      that is being rolled back may be holding a lock on the table and
      its metadata record.
      
      dict_table_remove_from_cache_low(): Replaced
      by dict_table_remove_from_cache().
      
      dict_table_remove_from_cache(): Add a third parameter keep=false,
      so that the table can be freed by the caller.
      
      trx_lock_t::evicted_tables: List of tables on which trx_t::evict_table()
      was invoked.
      
      trx_t::evict_table(): Evict a table definition during rollback.
      
      trx_commit_in_memory(): Empty the trx->lock.evicted_tables list
      after the locks were released, by freeing the table objects.
      
      row_undo_ins_remove_clust_rec(), row_undo_mod_clust_low():
      Invoke trx_t::evict_table() on the affected table if a change to
      SYS_COLUMNS is being rolled back.
      f545e3cf
    • Marko Mäkelä's avatar
      MDEV-15562: Simplify FOREIGN KEY error handling on DDL · f58a0b3a
      Marko Mäkelä authored
      The error handling for ALTER TABLE…ALGORITHM=COPY as well as
      CREATE TABLE used to commit the CREATE TABLE transaction and then
      issue DROP TABLE in a separate transaction. This is unnecessarily
      breaking atomicity during DDL operations. Let us revise it so
      that the DROP TABLE will be executed within the same transaction,
      which will finally be rolled back.
      
      FIXME: Introduce an undo log record so that the data file would be
      deleted on rollback and no DROP TABLE would be needed at all.
      
      FIXME: Avoid unnecessary access to per-table tablespace during DROP TABLE.
      If the .ibd file is going to be deleted anyway, we should not bother
      to mark the pages free.
      
      dict_create_add_foreigns_to_dictionary(): Do not commit the transaction.
      We want simple rollback in case dict_load_foreigns() would fail.
      
      create_table_info_t::create_table(), row_create_index_for_mysql(),
      row_table_add_foreign_constraints(): Before invoking rollback, drop
      the table. Rollback would invoke trx_t::evict_table(), and after that
      dropping the table would be a no-op.
      
      ha_innobase::create(): Before rollback, drop the table. If the SQL
      layer invoked ha_innobase::delete_table() later, it would be a no-op
      because the rollback would have invoked trx_t::evict_table().
      f58a0b3a
    • Marko Mäkelä's avatar
      MDEV-15562: Add dict_index_t::first_user_field() · cd081734
      Marko Mäkelä authored
      dict_index_t::first_user_field(): Return the first data field in
      a clustered index, that is, the field after the PRIMARY KEY and
      the two system columns DB_TRX_ID, DB_ROLL_PTR.
      
      dtuple_convert_big_rec(): Remove some local variables.
      cd081734
    • Marko Mäkelä's avatar
      Merge 10.3 into 10.4 · 2a955c7a
      Marko Mäkelä authored
      2a955c7a
    • Marko Mäkelä's avatar
      Merge 10.4 into bb-10.4-mdev-15562 · f9807c7c
      Marko Mäkelä authored
      f9807c7c
    • Marko Mäkelä's avatar
      Enable an assertion · 46d768eb
      Marko Mäkelä authored
      46d768eb
    • Marko Mäkelä's avatar
      Fix bogus pointer arithmetics · 2f5ac0a4
      Marko Mäkelä authored
      2f5ac0a4
    • Marko Mäkelä's avatar
      Remove a goto and bogus assertions · d6e501f1
      Marko Mäkelä authored
      d6e501f1
    • Marko Mäkelä's avatar
      Merge 10.2 into 10.3 · 61b32df9
      Marko Mäkelä authored
      61b32df9
    • Marko Mäkelä's avatar
      MDEV-16946 innodb.alter_kill failed in buildbot with wrong result · 00b6c7d8
      Marko Mäkelä authored
      Ensure that no redo log checkpoint occurs in a critical section
      of a recovery test.
      00b6c7d8
    • Marko Mäkelä's avatar
      MDEV-16273 innodb.alter_kill fails Unknown storage engine 'InnoDB' · 2610c26a
      Marko Mäkelä authored
      The test is shutting down InnoDB, corrupting a file, and finally
      restarting InnoDB. Before the shutdown, the test created the table
      and inserted some records. Before MDEV-12288, there would be no access
      to the table after server restart, but after MDEV-12288 purge would
      reset the transaction identifier after the INSERT, and this would
      sometimes happen after the restart.
      
      To make the test deterministic, wait for purge to complete before the
      shutdown.
      2610c26a
    • Marko Mäkelä's avatar
    • Marko Mäkelä's avatar
      Fix the non-debug build · db2edfaf
      Marko Mäkelä authored
      db2edfaf
  3. 09 Oct, 2018 10 commits
    • Marko Mäkelä's avatar
      Work around a test failure · 9c458f54
      Marko Mäkelä authored
      9c458f54
    • Marko Mäkelä's avatar
    • Marko Mäkelä's avatar
      Adjust a test. · 78e7a16b
      Marko Mäkelä authored
      FIXME: DB_TRX_ID,DB_ROLL_PTR are not reset for the metadata record
      78e7a16b
    • Marko Mäkelä's avatar
      Try to support ADD/DROP/reorder with virtual columns · 6e4c840b
      Marko Mäkelä authored
      FIXME: The virtual column code does not really support any reordering
      of virtual columns. The function check_v_col_in_order() should be
      removed and all the code fixed.
      
      ha_innobase::check_if_supported_inplace_alter(): For now,
      forbid ADD COLUMN of a stored column together with ADD/DROP of
      virtual columns. This works around an assertion failure in
      gcol.innodb_virtual_basic line 541.
      
      dict_table_t::instant_column(): Copy and adjust v_cols[].
      
      dict_table_t::rollback_instant(): Roll back v_cols[].
      
      innobase_build_col_map(): Skip added virtual columns.
      
      prepare_inplace_add_virtual(): Correctly compute num_to_add_vcol.
      Remove some unnecessary code. Note that the call to
      innodb_base_col_setup() should be executed later.
      
      innobase_add_virtual_try(), innobase_drop_virtual_try(): Let
      the caller invoke innodb_update_cols().
      
      ha_innobase::commit_inplace_alter_table(): Add a FIXME comment
      that we should not reload the table when virtual columns are added
      or dropped. Currently, reloading is the only way to add the base
      columns of virtual columns into the InnoDB data dictionary cache.
      
      innobase_instant_try(): If any virtual column is dropped, rewrite
      all virtual column metadata. Use a shortcut only for adding
      virtual columns. This is because innobase_drop_virtual_try()
      assumes that the dropped virtual columns still exist in the cache.
      6e4c840b
    • Sergei Petrunia's avatar
      MDEV-16577: rocksdb.issue255 fails in buildbot · 8b371e4b
      Sergei Petrunia authored
      Make the testcase stable
      8b371e4b
    • Marko Mäkelä's avatar
      Fix FOREIGN KEY error handling on DDL · ae73b7d8
      Marko Mäkelä authored
      dict_create_add_foreigns_to_dictionary(): Do not commit the transaction.
      We want simple rollback in case dict_load_foreigns() would fail.
      
      create_table_info_t::create_table(), row_create_index_for_mysql(),
      row_table_add_foreign_constraints(): Before invoking rollback, drop
      the table. Rollback would invoke trx_t::evict_table(), and after that
      dropping the table would be a no-op.
      
      ha_innobase::create(): Before rollback, drop the table. If the SQL
      layer invoked ha_innobase::delete_table() later, it would be a no-op
      because the rollback would have invoked trx_t::evict_table().
      
      FIXME: Introduce an undo log record so that the data file would be
      deleted on rollback. Also, avoid unnecessary access to the tablespace
      during DROP TABLE or the rollback of CREATE TABLE. If the file is going
      to be deleted anyway, we should not bother to mark the pages free.
      ae73b7d8
    • Alexander Barkov's avatar
      MDEV-17216 Assertion `!dt->fraction_remainder(decimals())' failed in... · 5646c431
      Alexander Barkov authored
      MDEV-17216 Assertion `!dt->fraction_remainder(decimals())' failed in Field_temporal_with_date::store_TIME_with_warning
      
      The problem happened because {{Field_xxx::store(longlong nr, bool unsigned_val)}} erroneously passed {{unsigned_flag}} to the {{usec}} parameter of this constructor:
      {code:cpp}
      Datetime(int *warn, longlong sec, ulong usec, date_conv_mode_t flags)
      {code}
      
      1. Changing Time and Datetime constructors to accept data as Sec6 rather than as
      longlong/double/my_decimal, so it's not possible to do such mistakes
      in the future. Additional good effect of these changes:
      - This reduced some amount of similar code (minus ~35 lines).
      - The code now does not rely on the fact that "unsigned_flag" is
         not important inside Datetime().
        The constructor always gets all three parts: sign, integer part,
        fractional part. The simple the better.
      
      2. Fixing Field_xxx::store() to use the new Datetime constructor format.
         This change actually fixes the problem.
      
      3. Adding "explicit" keyword to all Sec6 constructors,
      to avoid automatic hidden conversion from double/my_decimal to Sec6,
      as well as from longlong/ulonglong through double to Sec6.
      
      4. Change#1 caused (as a dependency) changes in a few places
         with code like this:
      
        bool neg= nr < 0 && !unsigned_val;
        ulonglong value= m_neg ? (ulonglong) -nr : (ulonglong) nr;
      
      These fragments relied on a non-standard behavior with
      the operator "minus" applied to the lowest possible negative
      signed long long value. This can lead to different results
      depending on the platform and compilation flags.
      We have fixed such bugs a few times already.
      So instead of modifying the old wrong code to a new wrong code,
      replacing all such fragments to use Longlong_hybrid,
      which correctly handles this special case with -LONGLONG_MIN
      in its method abs().
      This also reduced the amount of similar code
      (1 or 2 new lines instead 3 old lines in all 6 such fragments).
      
      5. Removing ErrConvInteger(longlong nr, bool unsigned_flag= false)
         and adding ErrConvInteger(Longlong_hybrid) instead, to encourage
         use of safe Longlong_hybrid instead of unsafe pairs nr+neg.
      
      6. Removing unused ErrConvInteger from Item_cache_temporal::get_date()
      5646c431
    • Vladislav Vaintroub's avatar
      MDEV-17279 Windows : link C runtime dynamically · f4cdf90d
      Vladislav Vaintroub authored
      Changed the build to use /MD flag so that DDL version of C runtime is used.
      
      To make sure MariaDB is always runnable on target system, include
      redistributable CRT libraries into installer.
      
      For MSI package, use Microsoft's merge modules.
      For ZIP  use "applocal" approach,i.e place redistributable dlls
      into the bin directory of the package(via InstallRequiredSystemLibraries
      cmake module) The space overhead of libraries in negligible, ~ 3MB unpacked.
      
      There are 2 cases, where we still link C runtime statically
      
      - Upgrade wizard, it uses MFC, and we link statically to avoid
      redistribute also whole MFC (for this single application, does not
      make much sense).
      
      - MSI installer's custom action dll wixca.dll.Here, we need static link
      so that MSI won't fail on a target system that does not have VC++2015
      runtime already installed.
      f4cdf90d
    • Marko Mäkelä's avatar
      Merge 10.2 into 10.3 · 43ee6915
      Marko Mäkelä authored
      43ee6915
    • Alexander Barkov's avatar
      MDEV-17400 The result of TIME('42949672965959-01') depends on architecture · c57bbb25
      Alexander Barkov authored
      - Fixing portabibily problems in sql-common/my_time.c
        (and additionally in sql/sql_time.cc)
      
      - Re-enabling func_time.test
        Now all new chunks added in MDEV-17351 work fine on all platforms.
      c57bbb25
  4. 08 Oct, 2018 13 commits
    • Marko Mäkelä's avatar
      Fix instant ADD after instant DROP · 51bd08c8
      Marko Mäkelä authored
      row_metadata_to_tuple(), row_rec_to_index_entry_impl():
      Add the parameter "pad" to determine whether
      the tuple should be padded to the index fields (on ALTER TABLE it should),
      or whether it should remain at its original size (on rollback).
      
      trx_undo_page_report_modify(): Avoid out-of-bounds access to record fields.
      
      row_quiesce_write_index_fields(): Do not crash when the table contains
      an instantly dropped column.
      51bd08c8
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-16980 Wrongly set tablename len while opening the · e9d9ca8c
      Thirunarayanan Balathandayuthapani authored
      			table for purge thread
      
      Problem:
      =======
      	Purge tries to fetch mdl lock for the whole table even though it tries
      to open one of the partition. But table name length was wrongly set to indicate
      the partition name too.
      
      Solution:
      ========
      - Table name length should identify the table name only not the partition name.
      e9d9ca8c
    • Thirunarayanan Balathandayuthapani's avatar
      MDEV-16980 Wrongly set tablename len while opening the · 7d4beb72
      Thirunarayanan Balathandayuthapani authored
      		table for purge thread
      
      Problem:
      =======
      	Purge tries to fetch mdl lock for the whole table even though it tries
      to open one of the partition. But table name length was wrongly set to indicate
      the partition name too.
      
      Solution:
      ========
      - Table name length should identify the table name only not the partition name.
      7d4beb72
    • Igor Babaev's avatar
      MDEV-17381 Wrong query result with LATERAL DERIVED optimization · 85953617
      Igor Babaev authored
                and join_cache_level=6
      
      This bug was fixed by the patch for mdev-17382 applied to 5.5.
      85953617
    • Marko Mäkelä's avatar
    • Alexander Barkov's avatar
    • Igor Babaev's avatar
      MDEV-17382 Hash join algorithm should not be used to join materialized · e2535dcc
      Igor Babaev authored
                 derived table / view by equality
      
      Now rows of a materialized derived table are always put into a
      temporary table before join operation. If BNLH is used to join this
      table with the result of a partial join then both operands of the
      join are actually put into main memory. In most cases this is not
      efficient.
      We could avoid this by sending the rows of the derived table directly
      to the join operation. However this kind of data flow is not supported
      yet.
      Fixed by not allowing usage of hash join algorithm to join a materialized
      derived table if it's joined by an equality predicate of the form
      f=e where f is a field of the derived table.
      
      Change for the test case in 10.3: splitting must be turned off to preserve
      the explain.
      e2535dcc
    • Marko Mäkelä's avatar
    • Marko Mäkelä's avatar
      Fix a rollback bug · edcd3f08
      Marko Mäkelä authored
      row_rec_to_index_entry_impl<true, 2>(): Correctly convert a record
      from instant ALTER TABLE format into instant ADD COLUMN format.
      
      row_undo_search_clust_to_pcur(): Relax too tight debug assertions.
      edcd3f08
    • Marko Mäkelä's avatar
      3905ba7c
    • Marko Mäkelä's avatar
      Remove dict_table_t::rollback_instant(unsigned n) · a140cd1f
      Marko Mäkelä authored
      On rollback of changes to SYS_COLUMNS, we can no longer assume that
      the change was the instant addition of a column at the end of the
      column list. The function dict_table_t::rollback_instant(unsigned n)
      was inherently incompatible with instantly dropping or reordering
      columns.
      
      When a change to SYS_COLUMNS is rolled back, we must simply evict
      the affected table definition, at the end of the rollback. We cannot
      free the table object immediately, because the current transaction
      that is being rolled back may be holding locks on the table.
      
      dict_table_remove_from_cache_low(): Replaced
      by dict_table_remove_from_cache().
      
      dict_table_remove_from_cache(): Add a third parameter keep=false,
      so that the table can be freed by the caller.
      
      dict_table_t::rollback_instant(): Add the parameter old_n_fields,
      and simplify some logic.
      
      trx_lock_t::evicted_tables: List of tables on which trx_t::evict_table()
      was invoked.
      
      trx_t::evict_table(): Evict a table definition during rollback.
      
      trx_commit_in_memory(): Empty the trx->lock.evicted_tables list
      after the locks were released, by freeing the table objects.
      
      row_undo_ins_remove_clust_rec: Invoke trx_t::evict_table() on the
      affected table if a change to SYS_COLUMNS is being rolled back.
      
      row_undo_mod_clust_low(): FIXME: do the same
      a140cd1f
    • Alexander Barkov's avatar
      MDEV-17351 Wrong results for GREATEST,TIMESTAMP,ADDTIME with an out-of-range TIME-alike argument · b639fe2b
      Alexander Barkov authored
      Problems:
      
      Functions LEAST() and GREATEST() in TIME context, as well as functions
      TIMESTAMP(a,b) and ADDTIME(a,b), returned confusing results when the
      input TIME-alike value in a number or in a string was out of the TIME
      supported range.
      
      In case of TIMESTAMP(a,b) and ADDTIME(a,b), the second argument
      value could get extra unexpected digits. For example, in:
          ADDTIME('2001-01-01 00:00:00', 10000000)  or
          ADDTIME('2001-01-01 00:00:00', '1000:00:00')
      the second argument was converted to '838:59:59.999999'
      with six fractional digits, which contradicted "decimals"
      previously set to 0 in fix_length_and_dec().
      These unexpected fractional digits led to confusing function results.
      
      Changes:
      1. GREATEST(), LEAST()
      
         - fixing Item_func_min_max::get_time_native()
         to respect "decimals" set by fix_length_and_dec().
         If a value of some numeric or string time-alike argument
         goes outside of the TIME range and gets limited to '838:59:59.999999',
         it's now right-truncated to the correct fractional precision.
      
         - fixing, Type_handler_temporal_result::Item_func_min_max_fix_attributes()
         to take into account arguments' time_precision() or datetime_precision(),
         rather than rely on "decimals" calculated by the generic implementation
         in Type_handler::Item_func_min_max_fix_attributes(). This makes
         GREATEST() and LEAST() return better data types, with the same
         fractional precision with what TIMESTAMP(a,b) and ADDTIME(a,b) return
         for the same arguments, and with DATE(a) and TIMESTAMP(a).
      
      2. Item_func_add_time and Item_func_timestamp
      
         It was semantically wrong to apply the limit of the TIME data type
         to the argument "b", which plays the role of "INTERVAL DAY TO SECOND" here.
         Changing the code to fetch the argument "b" as INTERVAL rather than as TIME.
      
         The low level routine calc_time_diff() now gets the interval
         value without limiting to '838:59:59.999999', so in these examples:
           ADDTIME('2001-01-01 00:00:00', 10000000)
           ADDTIME('2001-01-01 00:00:00', '1000:00:00')
         calc_time_diff() gets '1000:00:00' as is.  The SQL function result
         now gets limited to the supported result data type range
         (datetime or time) inside calc_time_diff(), which now calculates
         the return value using the real fractional digits that
         came directly from the arguments (without the effect of limiting
         to the TIME range), so the result does not have any unexpected
         fractional digits any more.
      
         Detailed changes in TIMESTAMP() and ADDTIME():
      
         - Adding a new class Interval_DDhhmmssff. It's similar to Time, but:
           * does not try to parse datetime format, as it's not needed for
             functions TIMESTAMP() and ADDTIME().
           * does not cut values to '838:59:59.999999'
      
           The maximum supported Interval_DDhhmmssff's hard limit is
           'UINT_MAX32:59:59.999999'. The maximum used soft limit is:
           - '87649415:59:59.999999'   (in 'hh:mm:ss.ff' format)
           - '3652058 23:59:59.999999' (in 'DD hh:mm:ss.ff' format)
           which is a difference between:
           - TIMESTAMP'0001-01-01 00:00:00' and
           - TIMESTAMP'9999-12-31 23:59:59.999999'
           (the minimum datetime that supports arithmetic, and the
           maximum possible datetime value).
      
         - Fixing get_date() methods in the classes related to functions
           ADDTIME(a,b) and TIMESTAMP(a,b) to use the new class Interval_DDhhmmssff
           for fetching data from the second argument, instead of get_date().
      
         - Fixing fix_length_and_dec() methods in the classes related
           to functions ADDTIME(a,b) and TIMESTAMP(a,b) to use
           Interval_DDhhmmssff::fsp(item) instead of item->time_precision()
           to get the fractional precision of the second argument correctly.
      
         - Splitting the low level function str_to_time() into smaller pieces
           to reuse the code. Adding a new function str_to_DDhhmmssff(), to
           parse "INTERVAL DAY TO SECOND" values.
      
         After these changes, functions TIMESTAMP() and ADDTIME()
         return much more predictable results, in terms of fractional
         digits, and in terms of the overall result.
      
         The full ranges of DATETIME and TIME values are now covered by TIMESTAMP()
         and ADDTIME(), so the following can now be calculated:
      
          SELECT ADDTIME(TIMESTAMP'0001-01-01 00:00:00', '87649415:59:59.999999');
          -> '9999-12-31 23:59:59.999999'
      
          SELECT TIMESTAMP(DATE'0001-01-01', '87649415:59:59.999999')
          -> '9999-12-31 23:59:59.999999'
      
          SELECT ADDTIME(TIME'-838:59:59.999999', '1677:59:59.999998');
          -> '838:59:59.999999'
      b639fe2b
    • Igor Babaev's avatar
      MDEV-17382 Hash join algorithm should not be used to join materialized · 1ebe841f
      Igor Babaev authored
                 derived table / view by equality
      
      Now rows of a materialized derived table are always put into a
      temporary table before join operation. If BNLH is used to join this
      table with the result of a partial join then both operands of the
      join are actually put into main memory. In most cases this is not
      efficient.
      We could avoid this by sending the rows of the derived table directly
      to the join operation. However this kind of data flow is not supported
      yet.
      Fixed by not allowing usage of hash join algorithm to join a materialized
      derived table if it's joined by an equality predicate of the form
      f=e where f is a field of the derived table.
      1ebe841f
  5. 07 Oct, 2018 1 commit
    • Igor Babaev's avatar
      MDEV-17360 Server crashes in optimize_keyuse · d03581bf
      Igor Babaev authored
      This was a bug in the code of MDEV-12387 "Push conditions into materialized
      subqueries". The bug manifested itself in rather rare situations. An
      affected query must contain IN subquery predicate whose left operand
      was an outer field of a mergeable derived table or view and right operand
      was a materialized subquery.
      The erroneous code in fact stripped off the Item_direct_ref wrapper from
      the left operand of the IN subquery predicate when building equalities
      produced by the conversion of the predicate into a semi-join. As a result
      the left operand was not considered as an outer reference anymore and
      used_tables() was calculated incorrectly. This caused a crash in the
      function optimize_keyuse().
      d03581bf