revno: 3545
revision-id: marko.makela@oracle.com-20100818110110-zfs0i1vfrccfb4yw
parent: vasil.dimov@oracle.com-20100817193934-1yl7zz2odikxauf8
committer: Marko Mäkelä <marko.makela@oracle.com>
branch nick: 5.1-innodb
timestamp: Wed 2010-08-18 14:01:10 +0300
message:
  Bug#55626: MIN and MAX reading a delete-marked record from secondary index

  Remove a bogus debug assertion that triggered the bug.
  Add assertions precisely where records must not be delete-marked.
  And a comment to clarify when the record is allowed to be delete-marked.

dict_load_index_low(): Rename the parameter "cached" to "allocated"
and clarify the comments.

Followup to vasil.dimov@oracle.com-20100817063430-inglmzgdtj95t29d
which didn't fully fix the test because the order of the returned
rows was different in embedded and non-embedded version. So the only
way to fix this is to add an ORDER BY clause.

lock wait time. Including the InnoDB lock time in the exiting "Lock_time"
output.

This is a followup to vasil.dimov@oracle.com-20100816142329-yimenbuktd416z1a
which improved the sampling algorithm. The endspace test is non-deterministic
because it does not include ORDER BY clause in its queries.

This is a followup to vasil.dimov@oracle.com-20100816142329-yimenbuktd416z1a
which improved the sampling algorithm.

This is a followup to vasil.dimov@oracle.com-20100816142329-yimenbuktd416z1a
which improved the sampling algorithm.

  
This is a followup to vasil.dimov@oracle.com-20100816142329-yimenbuktd416z1a
which improved the sampling algorithm.

This is a followup to vasil.dimov@oracle.com-20100816142329-yimenbuktd416z1a
which improved the sampling algorithm. I have manually checked that the new
values are actually the correct ones, for example:
-rows	16
+rows	32
the number of rows returned by the query is 32.

Improve the range estimation algorithm.

Previously:
For a given level the algo knows the number of pages in the requested range and the n

With this change:
Same idea, but peek a few (10) of the intermediate pages to get a better estimate of 

In the bug report one of the examples has a btree with a snippet of the leaf level li
page1(899 records), page2(1 record), page3(1 record), page4(1 record)
so when trying to estimate, the previous algo, assumed there are average (899+1)/2=45
Fix Bug#53761 RANGE estimation for matched rows may be 200 times different

Improve the range estimation algorithm.

Previously:
For a given level the algo knows the number of pages in the requested range
and the number of records on the leftmost and the rightmost page. Then it
assumes all pages in between contain the average between the two border pages
and multiplies this average number by the number of intermediate pages.

With this change:
Same idea, but peek a few (10) of the intermediate pages to get a better
estimate of the average number of records per page. If there are less than 10
intermediate pages then all of them will be scanned and the result will be
precise, not an estimation.

In the bug report one of the examples has a btree with a snippet of the leaf
level like this:
page1(899 records), page2(1 record), page3(1 record), page4(1 record)
so when trying to estimate, the previous algo, assumed there are average
(899+1)/2=450 records per page which went terribly wrong. With this change
page2 and page3 will be read and the exact number of records will be returned.

Approved by:	Sunny (rb://401)

           locks on the table

Fixing the partitioning specifics after TRUNCATE TABLE in
bug-42643 was fixed.

Reorganize of code to decrease the size of the giant switch
in mysql_execute_command, and to prepare for future parser
reengineering. Moved code into Sql_statement objects.

Updated patch according to davi's review comments.

libmysqld/CMakeLists.txt:
  Added new files.
libmysqld/Makefile.am:
  Added new files.
mysql-test/r/not_partition.result:
  now returning error on partitioning commands
  if partitioning is not enabled.
mysql-test/r/partition_disabled.result:
  There is no partition handlerton, so it cannot
  find the specified engine in the .frm file.
mysql-test/r/partition_truncate.result:
  Updated test results.
mysql-test/suite/parts/inc/partition_mgm.inc:
  Added check that TRUNCATE PARTITION does not delete on failure.
mysql-test/suite/parts/r/partition_debug_sync_innodb.result:
  updated results.
mysql-test/suite/parts/r/partition_mgm_lc0_archive.result:
  updated results.
mysql-test/suite/parts/r/partition_mgm_lc1_archive.result:
  updated results.
mysql-test/suite/parts/r/partition_mgm_lc2_archive.result:
  updated results.
mysql-test/suite/parts/t/partition_debug_sync_innodb.test:
  Test case for this bug.
mysql-test/t/not_partition.test:
  Added check for TRUNCATE PARTITION without partitioning.
mysql-test/t/partition_truncate.test:
  Added test of TRUNCATE PARTITION on non partitioned table.
sql/CMakeLists.txt:
  Added new files.
sql/Makefile.am:
  Added new files.
sql/datadict.cc:
  Moved out the storage engine check into an own
  function, including assert for lock.
sql/datadict.h:
  added dd_frm_storage_engine.
sql/sql_alter_table.cc:
  moved the code for SQLCOM_ALTER_TABLE in mysql_execute_command
  into its own file, and using the Sql_statement object to
  prepare for future parser reengineering.
sql/sql_alter_table.h:
  Created Sql_statement object for ALTER TABLE.
sql/sql_lex.cc:
  resetting m_stmt.
sql/sql_lex.h:
  Temporary hack for forward declaration of enum_alter_table_change_level.
sql/sql_parse.cc:
  Moved out ALTER/ANALYZE/CHECK/OPTIMIZE/REPAIR TABLE
  from the giant switch into their own Sql_statement
  objects.
sql/sql_parse.h:
  Exporting check_merge_table_access.
sql/sql_partition_admin.cc:
  created Sql_statement for
  ALTER TABLE t ANALYZE/CHECK/OPTIMIZE/REPAIR/TRUNCATE
  PARTITION. To be able to reuse the TABLE equivalents.
sql/sql_partition_admin.h:
  Added Sql_statement of partition admin statements.
sql/sql_table.cc:
  Moved table maintenance code into sql_table_maintenance.cc
sql/sql_table.h:
  Moved table maintenance code into sql_table_maintenance.h
  exporting functions used by sql_table_maintenance.
sql/sql_table_maintenance.cc:
  Moved table maintenance code from sql_table.cc
sql/sql_table_maintenance.h:
  Sql_statement objects for ANALYZE/CHECK/OPTIMIZE/REPAIR TABLE.
  Also declaring the keycache functions.
sql/sql_truncate.cc:
  Moved code from SQLCOM_TRUNCATE in mysql_execute_command into
  Truncate_statement::execute.
  Added check for partitioned table on TRUNCATE PARTITION.
  Moved locking fix for partitioned table into
  Alter_table_truncate_partition::execute.
sql/sql_truncate.h:
  Truncate_statement declaration (sub class of Sql_statement).
sql/sql_yacc.yy:
  Using the new Sql_statment objects.

pushdown.
      
NDB supports only a limited set of item nodes for use in engine condition
pushdown. Because of this adding cache for const expression effectively
disabled this optimization.
      
The ndb_serialize_cond function is extended to support Item_cache and treat
it as a constant values.
A helper function called ndb_serialize_const is added. It is used to create
Ndb_cond value node from given const item.


mysql-test/suite/ndb/t/disabled.def:
  Bug#49746: Const expression caching led to NDB not using engine condition
  pushdown.
  Enabled ndb_condition_pushdown test after fixing appropriate bug.
sql/ha_ndbcluster_cond.cc:
  Bug#49746: Const expression caching led to NDB not using engine condition
  pushdown.
  The ndb_serialize_cond function is extended to support Item_cache and treat
  it as a constant values.
  A helper function called ndb_serialize_const is added. It is used to create
  Ndb_cond value node from given const item.
sql/item.cc:
  Bug#49746: Const expression caching led to NDB not using engine condition
  pushdown.
  The Item::cache_const_expr_analyzer function is adjusted to not create
  cache for Item_int_with_ref objects.
sql/item.h:
  Bug#49746: Const expression caching led to NDB not using engine condition
  pushdown.
  The result_type() method is added to Item_cache class.
  The Item_cache_str now initializes its collation.

Note that this was originally pushed by Calvin but the was later
reverted by mistake.

bug#54702

           corruption on ADD PARTITION and LOCK TABLE
Bug#53770: Server crash at handler.cc:2076 on
           LOAD DATA after timed out COALESCE PARTITION

5.5 fix for:
Bug#51042: REORGANIZE PARTITION can leave table in an
           inconsistent state in case of crash
Needs to be back-ported to 5.1

5.5 fix for:
Bug#50418: DROP PARTITION does not interact with
           transactions

Main problem was non-persistent operations done
before meta-data lock was taken (53770+53676).
And 53676 needed to keep the table/partitions opened and locked
while copying the data to the new partitions.

Also added thorough tests to spot some additional bugs
in the ddl_log code, which could result in bad state
between the .frm and partitions.

Collapsed patch, includes all fixes required from the reviewers.

mysql-test/r/partition_innodb.result:
  updated result with new test
mysql-test/suite/parts/inc/partition_crash.inc:
  crash test include file
mysql-test/suite/parts/inc/partition_crash_add.inc:
  test all states in fast_alter_partition_table
  ADD PARTITION branch
mysql-test/suite/parts/inc/partition_crash_change.inc:
  test all states in fast_alter_partition_table
  CHANGE PARTITION branch
mysql-test/suite/parts/inc/partition_crash_drop.inc:
  test all states in fast_alter_partition_table
  DROP PARTITION branch
mysql-test/suite/parts/inc/partition_fail.inc:
  recovery test including injecting errors
mysql-test/suite/parts/inc/partition_fail_add.inc:
  test all states in fast_alter_partition_table
  ADD PARTITION branch
mysql-test/suite/parts/inc/partition_fail_change.inc:
  test all states in fast_alter_partition_table
  CHANGE PARTITION branch
mysql-test/suite/parts/inc/partition_fail_drop.inc:
  test all states in fast_alter_partition_table
  DROP PARTITION branch
mysql-test/suite/parts/inc/partition_mgm_crash.inc:
  include file that runs all crash and failure injection tests.
mysql-test/suite/parts/r/partition_debug_innodb.result:
  new test result file
mysql-test/suite/parts/r/partition_debug_myisam.result:
  new test result file
mysql-test/suite/parts/r/partition_special_innodb.result:
  updated result
mysql-test/suite/parts/r/partition_special_myisam.result:
  updated result
mysql-test/suite/parts/t/partition_debug_innodb-master.opt:
  opt file for using with crashing tests of partitioned innodb
mysql-test/suite/parts/t/partition_debug_innodb.test:
  partitioned innodb test that require debug builds
mysql-test/suite/parts/t/partition_debug_myisam-master.opt:
  opt file for using with crashing tests of partitioned myisam
mysql-test/suite/parts/t/partition_debug_myisam.test:
  partitioned myisam test that require debug builds
mysql-test/suite/parts/t/partition_special_innodb-master.opt:
  added innodb-file-per-table to easier verify partition status on disk
mysql-test/suite/parts/t/partition_special_innodb.test:
  added test case
mysql-test/suite/parts/t/partition_special_myisam.test:
  added test case
mysql-test/t/partition_innodb.test:
  added test case
sql/sql_base.cc:
  Moved alter_close_tables to sql_partition.cc
sql/sql_base.h:
  removed some non existing and duplicated functions.
sql/sql_partition.cc:
  fast_alter_partition_table:
  Spletted abort_and_upgrad_lock_and_close_table
  to its parts (wait_while_table_is_used and
  alter_close_tables) and always have
  wait_while_table_is_used before any persistent
  operations (including logs, which will be executed
  on failure) and alter_close_tables after
  create/read/write operations and before
  drop operations.
  
  moved alter_close_tables here from sql_base.cc
  
  Added error injections for better test coverage.
  
  write_log_final_change_partition:
  fixed a log_entry linking bug (delete_frm was not
  linked to change/drop partition)
  and drop partition must be executed before
  change partition (change partition can rename a
  partition to an old name, like REORG p1 INTO (p1,p2).
  
  write_log_add_change_partition:
  need to use drop_frm first, and relinking that entry
  and reusing its execute entry.
sql/sql_table.cc:
  added initialization of next_active_log_entry.
sql/table.h:
  removed a duplicate declaration.

Fixing copyright text.

      
Removed table SETUP_OBJECTS.
      
Renamed table PROCESSLIST to THREADS.
      
Renamed table EVENTS_WAITS_SUMMARY_BY_EVENT_NAME
to EVENTS_WAITS_SUMMARY_GLOBAL_BY_EVENT_NAME.
      
Adjusted Makefiles, code and tests accordingly.

Disable the GCC visibility attribute on all sun platforms.

Approved by Marko on IRC.

  "MySQL server is not restarted properly after RPM upgrade"

The problem is that with the general spec file cleanup and
alignment we also did a name change, dropping the "-community"
part from the package file name.

As a result of this, RPM (some versions of it) will report
file conflicts, because it considers this name difference
to imply different packages.
To avoid this, the spec file explicitly "obsoletes" the old
packages (with "-community" in the file name).

Now, RPM will first install these packages and the remove the
old ones, and part of that removal is running the "%preun"
section which stops the server and uninstalls the service
(removes the symlinks to "/etc/init.d/mysql" from the run
level directories).
This stop/uninstall will affect the new server!

The fix is to define a "%triggerpostun" in this spec file
which will watch for removal of the "-community" server.
If this is done (as part of this install/upgrade), the
trigger code will re-install the service and restart the
server process.

In addition, the "sleep" calls after starting the server
have been cleaned up: Rather than doing 2* "sleep 2",
it is now 1 "sleep 5".

it couldn't parse the --ssl option.

client/mysql_upgrade.c:
  mysql_upgrade parses its options and passes some of them to the underlying
  tools (mysqlcheck etc). To do this passdown, it reconstructs a
  command-line-suitable text from the my_option object (which
  contains the option's name and option's value). For options
  which expect no parameter, it just had to use the option's name;
  for other options, it had to concatenate the option's name,
  a "=" symbol, and the option's value; it had code to handle
  this latter case, but only for GET_STR options (options taking a
  string as value). But since the work on WL 4738, the --ssl
  option, a GET_BOOL, which used to have no parameter (NO_ARG), can
  now have one (OPT_ARG), so with --ssl we came to the "default"
  label, error. Fixed by constructing the command-line-suitable
  representation for GET_BOOL too. For --ssl it will produce
  --ssl=1 ; for --ssl=0, it will produce --ssl=0.
mysql-test/include/mysql_upgrade_preparation.inc:
  handles requirements of tests which use mysql_upgrade
mysql-test/r/mysql_upgrade_ssl.result:
  result; without the code fix we would get "internal error".
mysql-test/t/mysql_upgrade.test:
  This test has requirements before running; moved them
  to an include file in order to share with mysql_upgrade_ssl.
mysql-test/t/mysql_upgrade_ssl.test:
  test for bug. Couldn't go into mysql_upgrade.test as this new test requires
  SSL support. --force is needed, in case mysql_upgrade.test run before
  (in which case mysql_upgrade_ssl would say that upgrade has already been
  done); --force forces the upgrade in all cases.

This is a performance improvement fix.

Removed the "volatile" property of PFS_events_waits::m_wait_class.
Simplified the code accordingly.

                        
After BUG#36649, warnings for sub-statements are cleared when a 
new sub-statement is started. This is problematic since it suppresses
warnings for unsafe statements in some cases. It is important that we
always give a warning to the client, because the user needs to know
when there is a risk that the slave goes out of sync.
                        
We fixed the problem by generating warning messages for unsafe statements
while returning from a stored procedure, function, trigger or while
executing a top level statement.
                        
We also started checking unsafeness when both performance and log tables are
used. This is necessary after the performance schema which does a distinction
between performance and log tables.

mysql-test/extra/rpl_tests/create_recursive_construct.inc:
  Changed the order of the calls in the procedure because the code
  that checks if a warning message is printed out expects that the
  first statement gives an warning what is not the case for INSERT
  INTO ta$CRC_ARG_level VALUES (47);
mysql-test/suite/binlog/r/binlog_stm_unsafe_warning.result:
  Updated the result file.
mysql-test/suite/binlog/r/binlog_unsafe.result:
  There are several changes here:
          
  (1) - Changed the CREATE PROCEDURE $CRC.
                                  
  (2) - The procedure $CRC was failing and the content of the binlog
        was being printed out, after fix (1) the failure disappeared.
                                  
  (3) - The warning message for unsafeness due to auto-increment collumns was
        changed.
                                  
  (4) - The warning message for unsafeness due to VERSION(), RAND() was changed.
mysql-test/suite/binlog/t/binlog_stm_unsafe_warning.test:
  Tested filters.
mysql-test/suite/binlog/t/binlog_unsafe.test:
  Reenabled the test case binlog_unsafe.
mysql-test/suite/binlog/t/disabled.def:
  Reenabled the test case binlog_unsafe.
mysql-test/suite/rpl/r/rpl_begin_commit_rollback.result:
  Updated the result file.
mysql-test/suite/rpl/r/rpl_non_direct_stm_mixing_engines.result:
  Updated the result file.
mysql-test/suite/rpl/r/rpl_stm_auto_increment_bug33029.result:
  Updated the result file.
sql/sql_class.cc:
  Moved the stmt_accessed_table_flag variable and related information to the
  LEX as we need the variable reset after each statement even inside a stored
  procedure, what did not happen if the information was in the THD.
          
  Changed the routine in the THD::binlog_query that prints the warning
  messages to avoid trying to print them when inside a stored procedure,
  function or trigger.
                          
  Checked for unsafeness when both performance and log tables where used.
  After the introduction of the performance schema, we need to check both.

Reduce ibuf_mutex and ibuf_pessimistic_insert_mutex contention further.

Protect ibuf->empty by the insert buffer root page latch, not ibuf_mutex.

ibuf_tree_root_get(): Assert that ibuf_mutex is owned by the
caller. Assert that the stamped page number is correct. Assert that
ibuf->empty agrees with the root page.

ibuf_size_update(): Do not update ibuf->empty.

ibuf_init_at_db_start(): Update ibuf->empty while holding the root page latch.

ibuf_add_free_page(): Return TRUE/FALSE instead of DB_SUCCESS/DB_STRONG_FAIL.

ibuf_remove_free_page(): Release ibuf_pessimistic_insert_mutex as
early as possible.

ibuf_contract_ext(): Rely on a dirty read of ibuf->empty, unless the
server is being shut down. Never acquire ibuf_mutex. Eliminate n_stored.

ibuf_contract_after_insert(): Never acquire ibuf_mutex. Perform dirty
reads of ibuf->size and ibuf->max_size.

ibuf_insert_low(): Only acquire ibuf_mutex for mode==BTR_MODIFY_TREE.
Perform dirty reads of ibuf->size and ibuf->max_size. Update
ibuf->empty while holding the root page latch.

ibuf_delete_rec(): Update ibuf->empty while holding the root page latch.

ibuf_is_empty(): Release ibuf_mutex earlier.

Fix by adding "DEPENDS gen_lex_hash" to ADD_CUSTOM_COMMAND(),
so it regenerates lex_hash.h whenever gen_lex_lash changes.

Also, make sql dependent on GenServerSource to avoid concurrent 
generation of lex_hash.h (once for sql , and in parallel for embedded)