LOCK TABLE+INSERT DELAYED

The problem was that the server could crash if the insert delayed
handler thread was killed due to a conflicting shared metadata
lock. This could happen because the metadata lock ticket was
added to the handler thread before it was properly initialized.

This patch moves the cloning of the acquired metadata lock ticket
until after the handler thread has been properly initialized.

The problem was that deadlocks involving INSERT DELAYED were not detected.

The reason for this is that two threads are involved in INSERT DELAYED:
the connection thread and the handler thread. The connection thread would
wait while the handler thread acquired locks and opened the table.
In essence, this adds an edge to the wait-for-graph between the 
connection thread and the handler thread that the deadlock detector is
unaware of. Therefore many deadlocks involving INSERT DELAYED were not 
detected.

This patch fixes the problem by having the connection thread acquire the
metadata lock the table before starting the handler thread. This allows the
deadlock detector to detect any possible deadlocks resulting from trying to
acquire a metadata lock the table. If a metadata lock is successfully acquired,
the handler thread is started and given a copy of the ticket representing the
metadata lock. When the handler thread then tries to lock and open the table,
it will find that it already has the metadata lock and therefore not acquire
any new metadata locks.

Test cases added to delayed.test.

      
Problem: ENUM columns are sorted and distributed according to their
numeric value, but Field::hash() incorrectly passed string character set
(utf32) in combination with numeric value to the hash function,
which made assertion fail.

Fix: pass "binary" character set in combination with numeric value
to the hash function.

  mysql-test/suite/parts/r/part_ctype_utf32.result
  Adding tests

  mysql-test/suite/parts/t/part_ctype_utf32.test
  Adding test

  sql/field.cc
  Pass correct character set pointer to the hash function.

The include/mysqlhotcopy.inc had an error in the 'if' condition, so it failed
if the mysqlhotcopy tool was found.

per-file comments:
  mysql-test/include/mysqlhotcopy.inc
        test should proceed exactly if the mysqlhotcopy was set.
  mysql-test/mysql-test-run.pl
        don't set the MYSQL_HOTCOPY variable if no mysqlhotcopy was found.

Recording fixed correct results results.

my_thread_global_end() now sets mysql_thread_basic_global_init_done= 0
to ensure destroyed mutexes are not reused.

I verified that clearing this flag will not result in the redundant allocation
of other resources allocated by my_thread_global_init() and
my_thread_basic_global_init().

mysys/my_thr_init.c:
  Clear mysql_thread_basic_global_init_done flag at the end of my_thread_global_end()

The ALTER PARTITION and SELECT seemed to be deadlocked
when having innodb_thread_concurrency = 1.

Problem was that there was unreleased latches
in the ALTER PARTITION thread which was needed
by the SELECT thread to be able to continue.

Solution was to release the latches by commit 
before requesting upgrade to exclusive MDL lock.

Updated according to reviewers comments (3).

mysql-test/r/partition_innodb.result:
  updated test result
mysql-test/t/partition_innodb.test:
  added test
sql/sql_partition.cc:
  Moved implicit commit into mysql_change_partition
  so that if latches are taken, they are always released
  before waiting on exclusive lock.
sql/sql_table.cc:
  refactored the code to prepare and commit
  around copy_data_between_tables, to be able
  to reuse it in mysql_change_partitions
sql/sql_table.h:
  exporting mysql_trans_prepare/commit_alter_copy_data

            attempt to install a plugin twice

Server crashes when [UN]INSTALL PLUGIN fails (returns an
error) and general log is disabled and there are audit
plugins interested in MYSQL_AUDIT_GENERAL_CLASS. 

When audit event is triggered, audit subsystem acquires interested
plugins by walking through plugin list. Evidently plugin list
iterator protects plugin list by acquiring LOCK_plugin, see
plugin_foreach_with_mask().

On the other hand [UN]INSTALL PLUGIN is acquiring LOCK_plugin
rather for a long time.

When audit event is triggered during [UN]INSTALL PLUGIN, plugin
list iterator acquires the same lock (within the same thread)
second time.

Repeatable only with general_log disabled, because general_log
triggers MYSQL_AUDIT_GENERAL_LOG event, which acquires audit
plugins before [UN]INSTALL PLUGIN acquired LOCK_plugin.

With this fix we pre-acquire audit plugins for events that
may potentially occur during [UN]INSTALL PLUGIN.

This hack should be removed when LOCK_plugin is fixed so it
protects only what it supposed to protect.

No test case for this fix - we do not have facility to test
audit plugins yet.

sql/sql_audit.cc:
  Move "acquire audit plugin" logics to a separate
  function.
sql/sql_audit.h:
  Move "acquire audit plugin" logics to a separate
  function.
sql/sql_plugin.cc:
  Pre-acquire audit plugins for events that may potentially occur
  during [UN]INSTALL PLUGIN.

end_*_wait() functions now honor TIMER_STATE_UNTIMED

storage/perfschema/pfs.cc:
  Don't aggregate stats if TIMER_STATE_UNTIMED

    files for cmake had some minor bugs causing this.
      
per-file comments:
  mysql-test/CMakeLists.txt
Bug#54129      Missing the execute bit for scripts
    use same permissions as in the source folder

  mysql-test/t/disabled.def
Bug#54129      Missing the execute bit for scripts
      mysqlhotcopy tests enabled

  scripts/CMakeLists.txt
Bug#54129      Missing the execute bit for scripts
    chmod +x for the script files

Problems:
- dot character was always printed as decimal point
  instead of localized decimal point for short
  numbers without thousands
- Item_func_format::val_str always returned values in ASCII
format,
  regargless of @@character_set_connection, which in case of utf32
  led to crash in debug build, or to incorrect values in release build.

Fix:
- Adding a piece of code to replace dot character to
  localized decimal point in short numbers.
- Changing parent class for Item_func_format to
  Item_str_ascii_func, because its val_str() implementation is heavily ASCII oriented.

was caused by change of thread state name from "Waiting for
table" to "Waiting for table metadata lock" (which has 
happened as part of fix for bug 52044 "FLUSH TABLES WITH READ
LOCK and FLUSH TABLES <list> WITH READ LOCK are incompati").

           on CREATE TABLE .. SELECT I_S.PART

This assert was triggered if an InnoDB table was created using
CREATE TABLE ... AS SELECT where the query used an I_S table, and
a view existed in the database. It would also be triggered for
any statement changing an InnoDB table (e.g. INSERT, UPDATE, DELETE)
which had a subquery referencing an I_S table.

The assert was triggered if open_normal_and_derived_tables() failed
and a statement transaction had been started. This will usually not
happen as tables are opened before a statement transaction is started.
However, e.g. CREATE TABLE ... AS SELECT starts a transaction in order
to insert tuples into the new table. And if the subquery references
an I_S table, all current tables and views can be opened in order to
fill the I_S table on the fly. If a view is discovered, open will fail
as it is instructed to open tables only (OPEN_TABLE_ONLY). This would
cause the assert to be triggered.

The assert was added in the patch for Bug#52044 and was therefore
not in any released versions of the server.

This patch fixes the problem by adjusting the assert to take into
consideration the possibility of tables being opened as part of
an I_S query. This is similar to what is already done for 
close_tables_for_reopen().

Test case added to information_schema_inno.test.

temp table
            
This patch introduces two key changes in the replication's behavior.
            
Firstly, it reverts part of BUG#51894 which puts any update to temporary tables
into the trx-cache. Now, updates to temporary tables are handled according to
the type of their engines as a regular table.
            
Secondly, an unsafe mixed statement, (i.e. a statement that access transactional
table as well non-transactional or temporary table, and writes to any of them),
are written into the trx-cache in order to minimize errors in the execution when
the statement logging format is in use.
            
Such changes has a direct impact on which statements are classified as unsafe
statements and thus part of BUG#53259 is reverted.

Problem: a few functions did not calculate their max_length correctly.
This is an after-fix for WL#2649 Number-to-string conversions".

Fix: changing the buggy functions to calculate max_length
using fix_char_length() introduced in WL#2649,
instead of setting max_length directly

  mysql-test/include/ctype_numconv.inc
     Adding new tests

  mysql-test/r/ctype_binary.result
     Adding new tests

  mysql-test/r/ctype_cp1251.result
     Adding new tests

  mysql-test/r/ctype_latin1.result
     Adding new tests

  mysql-test/r/ctype_ucs.result
     Adding new tests

  mysql-test/r/ctype_utf8.result
     Adding new tests

  mysql-test/t/ctype_utf8.test
    Including ctype_numconv

  sql/item.h
    - Introducing new method fix_char_length_ulonglong(),
    for the cases when length is potentially greater
    than UINT_MAX32. This method removes a few
    instances of duplicate code, e.g. in item_strfunc.cc.
    - Setting collation in Item_copy properly. This change
    fixes wrong metadata on client side in some cases, when
    "binary" instead of the real character set was reported.

  sql/item_cmpfunc.cc
    - Using fix_char_length() and max_char_length() methods,
    instead of direct access to max_length, to calculate
    item length properly.
    - Moving count_only_length() in COALESCE after
    agg_arg_charsets_for_string_result(). The old
    order was incorrect and led to wrong length
    calucation in case of multi-byte character sets.
    
  sql/item_func.cc
    Fixing that count_only_length() didn't work
    properly for multi-byte character sets.
    Using fix_char_length() and max_char_length()
    instead of direct access to max_length.

  sql/item_strfunc.cc
    - Using fix_char_length(), fix_char_length_ulonglong(),
    max_char_length() instead of direct access to max_length.
    - Removing wierd condition: "if (collation.collation->mbmaxlen > 0)",
    which is never FALSE.

              #ifdef THREAD removed from mysql.cc.
               No reason was found for this limitation to persist.

per-file comments:
  client/mysql.cc
Bug#54466      client 5.5 built from source lacks "pager" support
        now we have USE_POPEN always if not __WIN__
  mysql-test/r/mysql.result
Bug#54466      client 5.5 built from source lacks "pager" support
        result updated.
  mysql-test/t/mysql.test
Bug#54466      client 5.5 built from source lacks "pager" support
        test case added.

           check_if_table_exists()

This assert was triggered when the server tried to load plugins
while running in embedded server mode. In embedded server mode,
check_if_table_exists() was used to check if mysql.plugin existed
so that ER_NO_SUCH_TABLE could be silently ignored.
The problem was that this check was done without acquiring a metadata
lock on mysql.plugin first. This triggered the assert.

This patch fixes the problem by removing the call to
check_if_table_exists() from plugin_load(). Instead an error handler
which traps ER_NO_SUCH_TABLE is installed before trying to open
mysql.plugin when running in embedded server mode.

No test coverage added since this assert was triggered by 
existing tests running in embedded server mode.


sql/sql_base.cc:
  Renamed Prelock_error_handler to No_such_table_error_handler
  and moved the declaration to sql_base.h to make it usable
  in plugin_load().
sql/sql_base.h:
  Renamed Prelock_error_handler to No_such_table_error_handler
  and moved the declaration to sql_base.h to make it usable
  in plugin_load().
sql/sql_plugin.cc:
  Removed call to check_if_table_exists() used to check for mysql.plugin 
  in plugin_load() for embedded server. Instead install error handler
  which traps ER_NO_SUCH_TABLE during open_and_lock_tables().

from mysql-trunk-bugfixing (5.6.1-m5) from mysql-5.5-bugfixing (5.5.6-m3).

Problem: Item_func_hex::val_str() returned data in ASCII format,
which did not match collation.collation pointing to my_charset_utf32_general_ci.
Fix: changing parent class of Item_func_hex to Item_str_ascii_func,
as val_str() implementation is heavily ASCII-oriented.

  mysql-test/r/ctype_utf32.result
  mysql-test/t/ctype_utf32.test
  Adding test case


  sql/item_strfunc.cc
  sql/item_strfunc.h
  - Changing parent class to Item_str_ascii_func
  - Clean-up in Item_func_hex::fix_length_and_dec()
    Using fix_char_length() instead of setting max_length directly.

        'CREATE TABLE IF NOT EXISTS ... SELECT' behaviour
BUG#47132, BUG#47442, BUG49494, BUG#23992 and BUG#48814 will disappear
automatically after the this patch.
BUG#55617 is fixed by this patch too.
            
This is the 5.5 part.
It implements:
- 'CREATE TABLE IF NOT EXISTS ... SELECT' statement will not insert
  anything and binlog anything if the table already exists.
  It only generate a warning that table already exists.
- A couple of test cases for the behavior changing.

      
Added InnoDB to the 'default' plugin group, and modified 
the autoconf script so the 'default' group is actually 
built by default.
      
(i.e  ./configure.am == ./configure.am --with-plugins=default ,
instead of being ./configure.am --with-plugins=none )

dict_load_index_low(): Rename the parameter "cached" to "allocated"
and clarify the comments.

Followup to vasil.dimov@oracle.com-20100817063430-inglmzgdtj95t29d
which didn't fully fix the test because the order of the returned
rows was different in embedded and non-embedded version. So the only
way to fix this is to add an ORDER BY clause.

           rpl_ndb.rpl_ndb_2other fails

The two regressions tests failed after WL#5349 was
pushed, since they were writted with the implicit
requirement that MyISAM is the default storage engine.

Adding --default-storage-engine=MyISAM as startup
parameter, to mimic the pre-wl#5349 situation.


mysql-test/suite/rpl_ndb/t/disabled.def:
  Re-enabling rpl_ndb_stm_innodb and rpl_ndb_2other
mysql-test/suite/rpl_ndb/t/rpl_ndb_2other-slave.opt:
  Setting MyISAM as default storage engine
  explicitly
mysql-test/suite/rpl_ndb/t/rpl_ndb_stm_innodb-slave.opt:
  The slave also needs to be started with
  MyISAM as default engine.