switching binlog format to ROW

BUG 52616 fixed the case which the user would switch from STMT to
ROW binlog format, but the server would silently ignore it. After
that fix thd->is_current_stmt_binlog_format_row() reports correct
value at logging time and events are logged in ROW (as expected)
instead of STMT as they were previously and wrongly logged.

However, the fix was only partially complete, because on
disconnect, at THD cleanup, the implicit logging of temporary
tables is conditionally performed. If the binlog_format==ROW and
thd->is_current_stmt_binlog_format_row() is true then DROPs are
not logged. Given that the user can switch from STMT to ROW, this
is wrong because the server cannot tell, just by relying on the
ROW binlog format, that the tables have been dropped before. This
is effectively similar to the MIXED scenario when a switch from
STMT to ROW is triggered.

We fix this by removing this condition from
close_temporary_tables.

mysql-test/extra/binlog_tests/drop_temp_table.test:
  Added binlog test case.
mysql-test/suite/binlog/r/binlog_row_drop_tmp_tbl.result:
  Result changes because:
  - there is a missing drop on three temporary tables
  - it now contains results for the test added
mysql-test/suite/binlog/r/binlog_row_mix_innodb_myisam.result:
  Result now contains the implicit drop for the temporary table.
mysql-test/suite/binlog/r/binlog_stm_drop_tmp_tbl.result:
  Result file changed because it now contains results for added
  test case.
mysql-test/suite/rpl/r/rpl_drop_temp.result:
  Result file changed because it now contains results for added
  test case.
mysql-test/suite/rpl/t/rpl_drop_temp.test:
  Added replication test case.
sql/sql_base.cc:
  Removed the condition that would make the server to skip
  logging implicit drops when ROW binary log format mode was 
  in use.
  Additionally, deployed DBUG_ENTER/RETURN macros.

   Bug#47633 - assert in ha_myisammrg::info during OPTIMIZE
 
   The server crashed on an attempt to optimize a MERGE table with
   non-existent child table.
 
   mysql_admin_table() relied on the table to be successfully open
   if a table object had been allocated.
 
   Changed code to check return value of the open function before
   calling a handler:: function on it.

mysql-test/r/merge.result:
  Backport of revid:ingo.struewing@sun.com-20091223200354-r2uzbdkj2v6yv111
      Bug#47633 - assert in ha_myisammrg::info during OPTIMIZE
      Updated result file.
mysql-test/t/merge.test:
  Backport of revid:ingo.struewing@sun.com-20091223200354-r2uzbdkj2v6yv111
      Bug#47633 - assert in ha_myisammrg::info during OPTIMIZE
      Changed tests to respect changed TEMPORARY MERGE locking (unrelated).
      Changed tests to respect changed CREATE TABLE ... LIKE (unrelated).
      Changed tests to respect that no new tables can be created
      under LOCK TABLE (unrelated).
      Added test for Bug#47633.
  Changed error numbers to symbolic names.
  Added test for child locking for ALTER under LOCK TABLE.
  
  Since Bug 36171 is not pushed yet, not the whole patch has been backported.
mysys/my_delete.c:
  Backport of revid:ingo.struewing@sun.com-20091223200354-r2uzbdkj2v6yv111
      Bug#47633 - assert in ha_myisammrg::info during OPTIMIZE
      Fixed error reporting.
      Fixed indentation.
mysys/my_mmap.c:
  Backport of revid:ingo.struewing@sun.com-20091223200354-r2uzbdkj2v6yv111
      Bug#47633 - assert in ha_myisammrg::info during OPTIMIZE
      Added DBUG.
sql/item_func.cc:
  Backport of revid:ingo.struewing@sun.com-20091223200354-r2uzbdkj2v6yv111
  Added Debug Sync point, required by merge_sync.test.
sql/sql_table.cc:
  Backport of revid:ingo.struewing@sun.com-20091223200354-r2uzbdkj2v6yv111
      Bug#47633 - assert in ha_myisammrg::info during OPTIMIZE
      Do not call handler:: functions if the table was not opened
      successfully.
  Added Debug Sync point, required by merge_sync.test.
storage/myisam/mi_check.c:
  Backport of revid:ingo.struewing@sun.com-20091223200354-r2uzbdkj2v6yv111
      Bug#47633 - assert in ha_myisammrg::info during OPTIMIZE
      Unmap memory before exchanging data files. Needed on Windows.
storage/myisammrg/ha_myisammrg.cc:
  Backport of revid:ingo.struewing@sun.com-20091223200354-r2uzbdkj2v6yv111
  Added Debug Sync point, required by merge_sync.test.
  
  merge_sync.test will be introduced by a patch for Bug 36171,
  which is not pushed yet.

           DML flow and SAVEPOINT

The problem was that replication could break if a transaction involving
both transactional and non-transactional tables was rolled back to a
savepoint. It broke if a concurrent connection tried to drop a
transactional table which was locked after the savepoint was set.
This DROP TABLE completed when ROLLBACK TO SAVEPOINT was executed as the
lock on the table was dropped by the transaction. When the slave later
tried to apply the binlog, it would fail as the table would already
have been dropped.

The reason for the problem is that transactions involving both
transactional and non-transactional tables are written fully to the
binlog during ROLLBACK TO SAVEPOINT. At the same time, metadata locks
acquired after a savepoint, were released during ROLLBACK TO SAVEPOINT.
This allowed a second connection to drop a table only used between
SAVEPOINT and ROLLBACK TO SAVEPOINT. Which caused the transaction binlog
to refer to a non-existing table when it was written during ROLLBACK
TO SAVEPOINT.

This patch fixes the problem by not releasing metadata locks when
ROLLBACK TO SAVEPOINT is executed if binlogging is enabled.

Accidental change in compile-time definitions for FreeBSD


Revert the accidental setting of "HAVE_BROKEN_REALPATH"
on current versions of FreeBSD,

do it for both autotools ("configure.in")
and cmake ("cmake/os/FreeBSD.cmake").

collections. Originally, they had only been added to default.push, so
trees named after mysql-[trunk|next-mr] would not skip those tests.

Don't call member functions for a NIL pointer.


mysql-test/r/subselect4.result:
  Add test case.
mysql-test/t/subselect4.test:
  Add test case.
sql/sql_select.cc:
  If the (virtual) member function clone_item() returns NULL,
  there is no substitution to be made, and we don't need to set the collation.
  The test was invoking Item_cache::clone_item()

            use limit efficiently
Bug #36569: UPDATE ... WHERE ... ORDER BY... always does a
            filesort even if not required

Also two bugs reported after QA review (before the commit
of bugs above to public trees, no documentation needed):

Bug #53737: Performance regressions after applying patch
            for bug 36569
Bug #53742: UPDATEs have no effect after applying patch
            for bug 36569


Execution of single-table UPDATE and DELETE statements did not use the 
same optimizer as was used in the compilation of SELECT statements. 
Instead, it had an optimizer of its own that did not take into account 
that you can omit sorting by retrieving rows using an index.

Extra optimization has been added: when applicable, single-table 
UPDATE/DELETE statements use an existing index instead of filesort. A 
corresponding SELECT query would do the former.

Also handling of the DESC ordering expression has been added when
reverse index scan is applicable.

From now on most single table UPDATE and DELETE statements show the 
same disk access patterns as the corresponding SELECT query. We verify 
this by comparing the result of SHOW STATUS LIKE 'Sort%

Currently the get_index_for_order function 
a) checks quick select index (if any) for compatibility with the
   ORDER expression list or
b) chooses the cheapest available compatible index, but only if 
   the index scan is cheaper than filesort.
Second way is implemented by the new test_if_cheaper_ordering
function (extracted part the test_if_skip_sort_order()).



mysql-test/r/log_state.result:
  Updated result for optimized query, bug #36569.
mysql-test/r/single_delete_update.result:
  Test case for bug #30584, bug #36569 and bug #53742.
mysql-test/r/update.result:
  Updated result for optimized query, bug #30584.
  Note:
  "Handler_read_last 1" omitted, see bug 52312:
  lost Handler_read_last status variable.
mysql-test/t/single_delete_update.test:
  Test case for bug #30584, bug #36569 and bug #53742.
sql/opt_range.cc:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
  
  * get_index_for_order() has been rewritten entirely and moved
    to sql_select.cc
  
  New QUICK_RANGE_SELECT::make_reverse method has been added.
sql/opt_range.h:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
  
  * get_index_for_order() has been rewritten entirely and moved
    to sql_select.cc
  
  New functions:
  * QUICK_SELECT_I::make_reverse()
  * SQL_SELECT::set_quick()
sql/records.cc:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
  
  * init_read_record_idx() has been modified to allow reverse index scan
  
  New functions:
  * rr_index_last()
  * rr_index_desc()
sql/records.h:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
  
  init_read_record_idx() has been modified to allow reverse index scan
sql/sql_delete.cc:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
      
  mysql_delete: an optimization has been added to skip
  unnecessary sorting with ORDER BY clause where select
  result ordering is acceptable.
sql/sql_select.cc:
  Bug #30584, bug #36569, bug #53737, bug #53742:
    UPDATE/DELETE ... WHERE ... ORDER BY...  always does a filesort
    even if not required
      
  The const_expression_in_where function has been modified
  to accept both Item and Field pointers.
  
  New functions:
  * get_index_for_order()
  * test_if_cheaper_ordering() has been extracted from
    test_if_skip_sort_order() to share with get_index_for_order()
  * simple_remove_const()
sql/sql_select.h:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
      
  New functions:
  * test_if_cheaper_ordering()
  * simple_remove_const()
  * get_index_for_order()
sql/sql_update.cc:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
      
  mysql_update: an optimization has been added to skip
  unnecessary sorting with ORDER BY clause where a select
  result ordering is acceptable.
sql/table.cc:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
  
  New functions:
  * TABLE::update_const_key_parts()
  * is_simple_order()
sql/table.h:
  Bug #30584, bug #36569: UPDATE/DELETE ... WHERE ... ORDER BY...
                          always does a filesort even if not required
  
  New functions:
  * TABLE::update_const_key_parts()
  * is_simple_order()

Fix syntax error (missing space in SET command),  that effectively prevents mysqld from being build with SSL.

Handle this warning in the future as error, this will prevent pushing to main trees.

mysql-test/suite/perfschema/t/no_threads-master.opt:
  Innodb will start multiple threads, which is
  not compatible with --one-thread.   Disable
  innodb to be able to run the test case.

InnoDB came in a new version "simultanously" with the
commit.  This patch is a minor change to the new innodb
regression test suite. 

The default storage engine is changed from MyISAM to
InnoDB, in all builds except for the embedded server.

In addition, the following system variables are 
changed:

  * innodb_file_per_table is enabled
  * innodb_strict_mode is enabled
  * innodb_file_format_name_update is changed
    to 'Barracuda'

The test suite is changed so that tests that do not
explicitly include the have_innodb.inc are run with
--default-storage-engine=MyISAM.  This is to ease the
transition, so that most regression tests are run
with the same engine as before.

Some tests are disabled for the embedded server
regression test, as the output of certain statements
will be different that for the regular server
(i.e SELECT @@default_storage_engine).  This is to
ease transition.



mysql-test/mysql-test-run.pl:
  The regression test suite now adds a
  --default-storage-engine=MyISAM for all non-innodb 
  tests.  This behaviour can be controlled by the
  default-myisam switch in mysql-test-run
mysql-test/t/bootstrap-master.opt:
  The bootstrap test can only be run without InnoDB
  as it starts several mysqld instances on the same
  datadir. This is possible with MyISAM, but not
  with InnoDB.
storage/innobase/CMakeLists.txt:
  Build InnoDB per default
storage/innobase/handler/ha_innodb.cc:
  Change default values for system variables
  
    Enable file_per_table
    Enable strict_mode
    Upgrade default file format to Barracuda

line exceeds the limit

Merge the fix into "trunk-bugfixing".

line exceeds the limit

Upmerge the fix from 5.1 to 5.5 ("trunk").

line exceeds the limit

Upmerge the fix from 5.0 to 5.1

           line exceeds the limit

The number and/or names of our files for the main test suite
(contents of "mysql-test/t/") now exceeds the command line
length limit on AIX.
Solve the problem by using separate "cp" commands for the
various file name extensions.

to trunk-bugfixing, no contents changes (already done separate).