MDL deadlock detector".

Deadlock could have occurred when workload containing mix
of DML, DDL and FLUSH TABLES statements affecting same
set of tables was executed in heavily concurrent environment.

This deadlock occurred when several connections tried to
perform deadlock detection in metadata locking subsystem.
The first connection started traversing wait-for graph,
encountered sub-graph representing wait for flush, acquired
LOCK_open and dived into sub-graph inspection. When it has
encounterd sub-graph corresponding to wait for metadata lock
and blocked while trying to acquire rd-lock on
MDL_lock::m_rwlock (*) protecting this subgraph, since some
other thread had wr-lock on it. When this wr-lock was released
it could have happened (if there was other pending wr-lock
against this rwlock) that rd-lock from the first connection
was left unsatisfied but at the same time new rd-lock request
from the second connection sneaked in and was satisfied (for
this to be possible second rd- request should come exactly
after wr-lock is released but before pending wr-lock manages
to grab rwlock, which is possible both on Linux and in our
own rwlock implementation). If this second connection
continued traversing wait-for graph and encountered sub-graph
representing wait for flush it tried to acquire LOCK_open
and thus deadlock was created.

This patch tries to workaround this problem but not allowing
deadlock detector to lock LOCK_open mutex if some other thread
doing deadlock detection already owns it and current search
depth is greater than 0. Instead deadlock is reported.

Other possible solutions are either known to have negative
effects on performance or require much more time for proper
implementation and testing.

No test case is provided as this bug is very hard to repeat
in MTR environment but is repeatable with the help of RQG
tests.

sql/mdl.cc:
  Moved Deadlock_detection_visitor::m_current_search_depth to
  parent class to make it available in
  TABLE_SHARE::visit_subgraph().
  Added MDL_wait_for_graph_visitor::abort_traversal() method
  which allows to abort traversal of a wait-for graph and
  report a deadlock.
sql/mdl.h:
  Moved Deadlock_detection_visitor::m_current_search_depth to
  parent class to make it available in
  TABLE_SHARE::visit_subgraph().
  Added MDL_wait_for_graph_visitor::abort_traversal() method
  which allows to abort traversal of a wait-for graph and
  report a deadlock.
sql/sql_base.cc:
  Added dd_owns_lock_open counter and mutex protecting it to
  track number of connections which do deadlock detection and
  own or try to acquire LOCK_open.
sql/sql_base.h:
  Added dd_owns_lock_open counter and mutex protecting it to
  track number of connections which do deadlock detection and
  own or try to acquire LOCK_open.
sql/table.cc:
  Workaround bug #56405 but not allowing MDL deadlock detector
  to lock LOCK_open mutex if some other thread doing deadlock
  detection already owns it and current search depth is greater
  than 0. Instead report deadlock.

The problem was not caused by a change in the client,
rather by the tests using the Windows built-in "echo"
and not the one built by MySQL.
This again happened because the binary was missing in the package,
caused by the wrong macro being used to build it in "cmake".

Original changeset:
------------------------------------------------------------
revno: 3186
revision-id: davi.arnaut@oracle.com-20100831142822-2qhufn3hho4xqr4p
parent: gshchepa@mysql.com-20100830222201-ie7n43mjy4irlllk
committer: Davi Arnaut <davi.arnaut@oracle.com>
branch nick: mysql-5.5-merge
timestamp: Tue 2010-08-31 11:28:22 -0300
message:
  Bug#55846: Link tests fail on Windows - my_compiler.h missing
  
  Make the my_compiler.h header, like my_attribute.h, part of
  the distribution. This is required due to the dependency of
  the former on the latter (which can undefine __attribute__).
------------------------------------------------------------

Original changeset:
------------------------------------------------------------
revno: 3197
revision-id: alik@sun.com-20100831135426-h5a4s2w6ih1d8q2x
parent: magnus.blaudd@sun.com-20100830120632-u3xzy002mdwueli8
committer: Alexander Nozdrin <alik@sun.com>
branch nick: mysql-5.5-bugfixing
timestamp: Tue 2010-08-31 17:54:26 +0400
message:
  Bug#55980 Character sets: supplementary character _bin ordering is wrong
  
  Problem:
  - ORDER BY for utf8mb4_bin, utf16_bin and utf32_bin returned
    results in a wrong order, because old functions
    (supporting only BMP range) were used to handle these collations.
  - Additionally, utf16_bin did not sort supplementary characters
    between U+D700 and U+E000, as WL#1213 specification specified.
------------------------------------------------------------

```---------------------------------------------------------
revno: 3124
revision-id: dlenev@mysql.com-20100831090419-rzr5ktekby2gspm1
parent: alik@sun.com-20100827083901-x4wvtc10u9p7gcs9
committer: Dmitry Lenev <dlenev@mysql.com>
branch nick: mysql-5.5-rt-56137
timestamp: Tue 2010-08-31 13:04:19 +0400
message:
  Bug #56137 "Assertion `thd->lock == 0' failed on upgrading 
  from 5.1.50 to 5.5.6".
  
  Debug builds of the server aborted due to an assertion
  failure when DROP DATABASE statement was run on an
  installation which had outdated or corrupt mysql.proc table.
  Particularly this affected the mysql_upgrade tool which is
  run as part of 5.1 to 5.5 upgrade.
  
  The problem was that sp_drop_db_routines(), which was invoked
  during dropping of the database, could have returned without
  closing and unlocking mysql.proc table in cases when this
  table was not up-to-date with the current server. As a result
  further attempt to open and lock the mysql.event table, which
  was necessary to complete dropping of the database, ended up
  with an assert.
  
  This patch solves this problem by ensuring that
  sp_drop_db_routines() always closes mysql.proc table and
  releases metadata locks on it. This is achieved by changing
  open_proc_table_for_update() function to close tables and
  release metadata locks acquired by it in case of failure.
  This step also makes behavior of the latter function
  consistent with behavior of open_proc_table_for_read()/
  open_and_lock_tables().
  
  
  Test case for this bug was added to sp-destruct.test.
```

---------------------------------------------------------

           file .\dtoa.c

The assertion failure was correct because the 'width' argument
of my_gcvt() has the signed integer type, whereas the unsigned
value UINT_MAX32 was being passed by the caller
(Field_double::val_str()) leading to a negative width in
my_gcvt().

The following chain of problems was found by further analysis:

1. The display width for a floating point number is calculated
in Field_double::val_str() as either field_length or the
maximum possible length of string representation of a floating
point number, whichever is greater. Since in the bug's test
case field_length is UINT_MAX32, we get the same value as the
display width. This does not make any sense because for numeric
values field_length only matters for ZEROFILL columns,
otherwise it does not make sense to allocate that much memory
just to print a number. Field_float::val_str() has a similar
problem.

2. Even if the above wasn't the case, we would still get a
crash on a slightly different test case when trying to allocate
UINT_MAX32 bytes with String::alloc() because the latter does
not handle such large input values correctly due to alignment
overflows.

3. Even when String::alloc() is fixed to return an error when
an alignment overflow occurs, there is still a problem because
almost no callers check its return value, and
Field_double::val_str() is not an exception (same for
Field_float::val_str()).

4. Even if all of the above wasn't the case, creating a
Field_double object with UINT_MAX32 as its field_length does
not make much sense either, since the .frm code limits it to
MAX_FIELD_CHARLENGTH (255) bytes. Such a beast can only be
created by create_tmp_field_from_item() from an Item with
REAL_RESULT as its result_type() and UINT_MAX32 as its
max_length.

5. For the bug's test case, the above condition (REAL_RESULT
Item with max_length = UINT_MAX32) was a result of
Item_func_if::fix_length_and_dec() "shortcutting" aggregation
of argument types when one of the arguments was a constant
NULL. In this case, the attributes of the aggregated type were
simply copied from the other, non-NULL argument, but max_length
was still calculated as per the general, non-shortcut case, by
choosing the greatest of argument's max_length, which is
obviously not correct.

The patch addresses all of the above problems, even though
fixing the assertion failure for the particular test case would
require only a subset of the above problems to be solved.


client/sql_string.cc:
  Return an error in case of uint32 overflow in alignment.
  Also assert there was no overflow to help find such conditions
  in debug builds, since almost no callers check the return value
  of String::alloc().
mysql-test/r/func_if.result:
  Add a test case for bug #55077.
mysql-test/t/func_if.test:
  Add a test case for bug #55077.
sql/field.cc:
  - Assert we don't operate with fields wider than 255 
  (MAX_FIELD_CHARLENGTH) bytes in both Field_float and  
  Field_double. 
  - Don't take field_length into account when calculating the 
  output buffer length.
  - Check the return value of String::alloc()
sql/item_cmpfunc.cc:
  When shortcutting type aggregation, don't take the NULL 
  argument's max_length into account.
sql/sql_string.cc:
  Return an error in case of uint32 overflow in alignment.
  Also assert there was no overflow to help find such conditions
  in debug builds, since almost no callers check the return value
  of String::alloc().

with two connections doing LOCK TABLE+INSERT DELAYED".

Disabled --ps-protocol for this part of the test as INSERT
DELAYED simply doesn't work with it under LOCK TABLES.

             LOCK TABLE+INSERT DELAYED

The problem was that the server could crash if the insert delayed
handler thread was killed due to a conflicting shared metadata
lock. This could happen because the metadata lock ticket was
added to the handler thread before it was properly initialized.

This patch moves the cloning of the acquired metadata lock ticket
until after the handler thread has been properly initialized.

The Item_cache_datetime::val_str function wasn't taking into account that time
could be negative. This led to failed assertion.
Now Item_cache_datetime::val_str correctly converts negative time values
from integer to string representation.

mysql-test/r/func_group.result:
  Added a test case for the bug#56120.
mysql-test/t/func_group.test:
  Added a test case for the bug#56120.
sql/item.cc:
  Bug#56120: Failed assertion on MIX/MAX on negative time value
  Now Item_cache_datetime::val_str correctly converts negative time values
  from integer to string representation.

The problem was that deadlocks involving INSERT DELAYED were not detected.

The reason for this is that two threads are involved in INSERT DELAYED:
the connection thread and the handler thread. The connection thread would
wait while the handler thread acquired locks and opened the table.
In essence, this adds an edge to the wait-for-graph between the 
connection thread and the handler thread that the deadlock detector is
unaware of. Therefore many deadlocks involving INSERT DELAYED were not 
detected.

This patch fixes the problem by having the connection thread acquire the
metadata lock the table before starting the handler thread. This allows the
deadlock detector to detect any possible deadlocks resulting from trying to
acquire a metadata lock the table. If a metadata lock is successfully acquired,
the handler thread is started and given a copy of the ticket representing the
metadata lock. When the handler thread then tries to lock and open the table,
it will find that it already has the metadata lock and therefore not acquire
any new metadata locks.

Test cases added to delayed.test.

      
Problem: ENUM columns are sorted and distributed according to their
numeric value, but Field::hash() incorrectly passed string character set
(utf32) in combination with numeric value to the hash function,
which made assertion fail.

Fix: pass "binary" character set in combination with numeric value
to the hash function.

  mysql-test/suite/parts/r/part_ctype_utf32.result
  Adding tests

  mysql-test/suite/parts/t/part_ctype_utf32.test
  Adding test

  sql/field.cc
  Pass correct character set pointer to the hash function.

The include/mysqlhotcopy.inc had an error in the 'if' condition, so it failed
if the mysqlhotcopy tool was found.

per-file comments:
  mysql-test/include/mysqlhotcopy.inc
        test should proceed exactly if the mysqlhotcopy was set.
  mysql-test/mysql-test-run.pl
        don't set the MYSQL_HOTCOPY variable if no mysqlhotcopy was found.

Recording fixed correct results results.

my_thread_global_end() now sets mysql_thread_basic_global_init_done= 0
to ensure destroyed mutexes are not reused.

I verified that clearing this flag will not result in the redundant allocation
of other resources allocated by my_thread_global_init() and
my_thread_basic_global_init().

mysys/my_thr_init.c:
  Clear mysql_thread_basic_global_init_done flag at the end of my_thread_global_end()

The ALTER PARTITION and SELECT seemed to be deadlocked
when having innodb_thread_concurrency = 1.

Problem was that there was unreleased latches
in the ALTER PARTITION thread which was needed
by the SELECT thread to be able to continue.

Solution was to release the latches by commit 
before requesting upgrade to exclusive MDL lock.

Updated according to reviewers comments (3).

mysql-test/r/partition_innodb.result:
  updated test result
mysql-test/t/partition_innodb.test:
  added test
sql/sql_partition.cc:
  Moved implicit commit into mysql_change_partition
  so that if latches are taken, they are always released
  before waiting on exclusive lock.
sql/sql_table.cc:
  refactored the code to prepare and commit
  around copy_data_between_tables, to be able
  to reuse it in mysql_change_partitions
sql/sql_table.h:
  exporting mysql_trans_prepare/commit_alter_copy_data

            attempt to install a plugin twice

Server crashes when [UN]INSTALL PLUGIN fails (returns an
error) and general log is disabled and there are audit
plugins interested in MYSQL_AUDIT_GENERAL_CLASS. 

When audit event is triggered, audit subsystem acquires interested
plugins by walking through plugin list. Evidently plugin list
iterator protects plugin list by acquiring LOCK_plugin, see
plugin_foreach_with_mask().

On the other hand [UN]INSTALL PLUGIN is acquiring LOCK_plugin
rather for a long time.

When audit event is triggered during [UN]INSTALL PLUGIN, plugin
list iterator acquires the same lock (within the same thread)
second time.

Repeatable only with general_log disabled, because general_log
triggers MYSQL_AUDIT_GENERAL_LOG event, which acquires audit
plugins before [UN]INSTALL PLUGIN acquired LOCK_plugin.

With this fix we pre-acquire audit plugins for events that
may potentially occur during [UN]INSTALL PLUGIN.

This hack should be removed when LOCK_plugin is fixed so it
protects only what it supposed to protect.

No test case for this fix - we do not have facility to test
audit plugins yet.

sql/sql_audit.cc:
  Move "acquire audit plugin" logics to a separate
  function.
sql/sql_audit.h:
  Move "acquire audit plugin" logics to a separate
  function.
sql/sql_plugin.cc:
  Pre-acquire audit plugins for events that may potentially occur
  during [UN]INSTALL PLUGIN.

end_*_wait() functions now honor TIMER_STATE_UNTIMED

storage/perfschema/pfs.cc:
  Don't aggregate stats if TIMER_STATE_UNTIMED

    files for cmake had some minor bugs causing this.
      
per-file comments:
  mysql-test/CMakeLists.txt
Bug#54129      Missing the execute bit for scripts
    use same permissions as in the source folder

  mysql-test/t/disabled.def
Bug#54129      Missing the execute bit for scripts
      mysqlhotcopy tests enabled

  scripts/CMakeLists.txt
Bug#54129      Missing the execute bit for scripts
    chmod +x for the script files