MDL_lock::find_deadlock".

On some platforms deadlock detector in metadata locking 
subsystem under certain conditions might have exhausted
stack space causing server crashes.

Particularly this caused failures of rqg_mdl_stability
test on Solaris in PushBuild.

During search for deadlock MDL deadlock detector could 
sometimes encounter loop in the waiters graph in which 
MDL_context which has started search for a deadlock 
does not participate. In such case our algorithm will 
continue looping assuming that either this deadlock will 
be resolved by MDL_context which has created it (i.e.
by one of loop participants) or maximum search depth
will be reached. 
Since max search depth was set to 1000 in the latter case 
on platforms where each iteration of deadlock search 
algorithm needs more than DEFAULT_STACK_SIZE/1000 bytes 
of stack (around 192 bytes for 32-bit and around 256 bytes 
for 64-bit platforms) we might have exhausted stack space.

This patch solves this problem by reducing maximum search
depth for MDL deadlock detector to 32. This should be safe
at the moment as it is unlikely that each iteration of the 
current deadlock detector algorithm will consume more than 
1K of stack (thus total amount of stack required can't be
more than 32K) and we require at least 80K of stack in order
to open any table. Also this value should be (hopefully) big
enough to not cause too much false deadlock errors (there
is an anecdotal evidence that real-life deadlocks are
typically shorter than that).

Additional reasearch should be conducted in future in order
to determine the more optimal value of maximum search depth.

This patch does not include test case as existing
rqg_mdl_stability test can serve as one.

This patch removes the unused server variable
"table_lock_wait_timeout".


mysql-test/r/variables.result:
  Updated the test for Bug#28580 to reflect that 
  "table_lock_wait_timeout" no longer exists.
mysql-test/t/variables.test:
  Updated the test for Bug#28580 to reflect that 
  "table_lock_wait_timeout" no longer exists.

TEMPORARY + HANDLER + LOCK + SP".

Server crashed when one: 
1) Opened HANDLER or acquired global read lock
2) Then locked one or several temporary tables with
   LOCK TABLES statement (but no base tables).
3) Then issued any statement causing commit (explicit 
   or implicit).
4) Issued statement which should have closed HANDLER
   or released global read lock.
   
The problem was that when entering LOCK TABLES mode in the
scenario described above we incorrectly set transactional
MDL sentinel to zero. As result during commit all metadata 
locks were released (including lock for open HANDLER or
global metadata shared lock). Indeed, attempt to release
metadata lock for the second time which happened during
HANLDER CLOSE or during release of GLR caused crash.

This patch fixes problem by changing MDL_context's
set_trans_sentinel() method to set sentinel to correct 
value (it should point to the most recent ticket).

mysql-test/include/handler.inc:
  Added test for bug #51136 "Crash in pthread_rwlock_rdlock on 
  TEMPORARY + HANDLER + LOCK + SP".
mysql-test/r/flush.result:
  Updated test results (see flush.test).
mysql-test/r/handler_innodb.result:
  Updated test results (see include/handler.inc).
mysql-test/r/handler_myisam.result:
  Updated test results (see include/handler.inc).
mysql-test/t/flush.test:
  Added additional coverage for bug #51136 "Crash in
  pthread_rwlock_rdlock on TEMPORARY + HANDLER + LOCK +
  SP".
sql/mdl.h:
  When setting new value of transactional sentinel use 
  pointer to the most recent ticket instead of value 
  returned by MDL_context::mdl_savepoint(). 
  This allows to handle correctly situation when the new 
  value of sentinel should be the same as its current value 
  (MDL_context::mdl_savepoint() returns NULL in this case).

DDL workload".

When a RENAME TABLE or LOCK TABLE ... WRITE statement which
mentioned the same table several times were aborted during 
the process of acquring metadata locks (due to deadlock 
which was discovered or because of KILL statement) server 
might have crashed.

When attempt to acquire all locks requested had failed we
went through the list of requests and released locks which
we have managed to acquire by that moment one by one. Since 
in the scenario described above list of requests contained 
duplicates this led to releasing the same ticket twice and 
a crash as result.

This patch solves the problem by employing different approach
to releasing locks in case of failure to acquire all locks
requested. 
Now we take a MDL savepoint before starting acquiring locks 
and simply rollback to it if things go bad.

mysql-test/r/lock_multi.result:
  Updated test results (see lock_multi.test).
mysql-test/t/lock_multi.test:
  Added test case for bug #51134 "Crash in MDL_lock::destroy
  on a concurrent DDL workload".
sql/mdl.cc:
  MDL_context::acquire_locks():
    When attempt to acquire all locks requested has failed do
    not go through the list of requests and release locks which
    we have managed to acquire one by one. 
    Since list of requests can contain duplicates such approach
    may lead to releasing the same ticket twice and a crash as
    result.
    Instead use the following approach - take a MDL savepoint
    before starting acquiring locks and simply rollback to it
    if things go bad.

failed in enter_locked_tables_mode".

Server was aborted due to assertion failure when one tried to 
execute statement requiring prelocking (i.e. firing triggers
or using stored functions) while having open HANDLERs.

The problem was that THD::enter_locked_tables_mode() method
which was called at the beginning of execution of prelocked 
statement assumed there are no open HANDLERs. It had to do 
so because corresponding THD::leave_locked_tables_mode()
method was unable to properly restore MDL sentinel when
leaving LOCK TABLES/prelocked mode in the presence of open 
HANDLERs.

This patch solves this problem by changing the latter method
to properly restore MDL sentinel and thus removing need for 
this assumption. As a side-effect, it lifts unjustified
limitation by allowing to keep HANDLERs open when entering 
LOCK TABLES mode.

mysql-test/include/handler.inc:
  Adjusted tests after making LOCK TABLES not to close
  open HANDLERs. Added coverage for bug #50908
  "Assertion `handler_tables_hash.records == 0' failed
  in enter_locked_tables_mode".
mysql-test/r/handler_innodb.result:
  Updated test results (see include/handler.inc).
mysql-test/r/handler_myisam.result:
  Updated test results (see include/handler.inc).
sql/mysql_priv.h:
  Introduced mysql_ha_move_tickets_after_trans_sentinel()
  routine which allows to move tickets for metadata locks
  corresponding to open HANDLERs after transaction sentinel.
sql/sql_class.cc:
  Changed THD::leave_locked_tables_mode() to correctly restore 
  MDL sentinel value in the presence of open HANDLERs.
sql/sql_class.h:
  Removed assert from THD::enter_locked_tables_mode() as we
  no longer have to close HANDLERs when entering LOCK TABLES 
  or prelocked modes. Instead we keep them open and correctly
  restore MDL sentinel value after leaving them.
  Removal of assert also fixes problem from the bug report.
sql/sql_handler.cc:
  Introduced mysql_ha_move_tickets_after_trans_sentinel()
  routine which allows to move tickets for metadata locks
  corresponding to open HANDLERs after transaction sentinel.
sql/sql_parse.cc:
  We no longer have to close HANDLERs when entering LOCK TABLES 
  mode. Instead we keep them open and simply correctly restore 
  MDL sentinel value after leaving this mode.

fix lock_sync.test failure in row based replication mode.

SELECT * FROM t1 on slave, first make sure that the slave has received
the CREATE TABLE from the master.

update the condition to wait for in wait_condition
to reflect type-of-operation aware metadata locks.

The test case for this bug relies on getting a ER_LOCK_WAIT_TIMEOUT
error. However with the introduction of MDL, the test would hang
forever since the metadata locks would not timeout.

MDL timeouts are now introduced in the scope of Bug#45225. This
patch changes the testcase for Bug#34604 to set the new server
variable "lock_wait_timeout" to one second which makes the test
generate the necessary timeout again.

This patch introduces timeouts for metadata locks. 

The timeout is specified in seconds using the new dynamic system 
variable  "lock_wait_timeout" which has both GLOBAL and SESSION
scopes. Allowed values range from 1 to 31536000 seconds (= 1 year). 
The default value is 1 year.

The new server parameter "lock-wait-timeout" can be used to set
the default value parameter upon server startup.

"lock_wait_timeout" applies to all statements that use metadata locks.
These include DML and DDL operations on tables, views, stored procedures
and stored functions. They also include LOCK TABLES, FLUSH TABLES WITH
READ LOCK and HANDLER statements.

The patch also changes thr_lock.c code (table data locks used by MyISAM
and other simplistic engines) to use the same system variable.
InnoDB row locks are unaffected.

One exception to the handling of the "lock_wait_timeout" variable
is delayed inserts. All delayed inserts are executed with a timeout
of 1 year regardless of the setting for the global variable. As the
connection issuing the delayed insert gets no notification of 
delayed insert timeouts, we want to avoid unnecessary timeouts.

It's important to note that the timeout value is used for each lock
acquired and that one statement can take more than one lock.
A statement can therefore block for longer than the lock_wait_timeout 
value before reporting a timeout error. When lock timeout occurs, 
ER_LOCK_WAIT_TIMEOUT is reported.

Test case added to lock_multi.test.


include/my_pthread.h:
  Added macros for comparing two timespec structs.
include/thr_lock.h:
  Introduced timeouts for thr_lock.c locks.
mysql-test/r/mysqld--help-notwin.result:
  Updated result file with the new server variable.
mysql-test/r/mysqld--help-win.result:
  Updated result file with the new server variable.
mysql-test/suite/sys_vars/r/lock_wait_timeout_basic.result:
  Added basic test for the new server variable.
mysql-test/suite/sys_vars/t/lock_wait_timeout_basic.test:
  Added basic test for the new server variable.
mysys/thr_lock.c:
  Introduced timeouts for thr_lock.c locks.
sql/mdl.cc:
  Introduced timeouts for metadata locks.
sql/mdl.h:
  Introduced timeouts for metadata locks.
sql/sql_base.cc:
  Introduced timeouts in tdc_wait_for_old_versions().
sql/sql_class.h:
  Added new server variable lock_wait_timeout.
sql/sys_vars.cc:
  Added new server variable lock_wait_timeout.

rqg_mdl_stability".

When start of statement's waiting on a metadata lock 
created more than one loop in waiters graph server might 
have entered deadlock condition.

The problem was that in the case described above MDL deadlock 
detector had to perform several searches for deadlock but
forgot to reset Deadlock_detection_context before performing 
new search. 
Failure to do so has broken assumption in code resposible for 
choosing victim that if Deadlock_detection_context::victim
is set we also have read lock on m_waiting_for_lock for this
context. As result this lock could have been unlocked more
times than it was acquired which corrupted rwlock's state
which led to server deadlock.

This fix ensures that such reset is done before each attempt
to find a deadlock.

mysql-test/r/mdl_sync.result:
  Added test for bug #50998 "Deadlock in MDL code during test
  rqg_mdl_stability" as well as coverage for the case when
  addition of statement waiting for metadata lock adds several
  loops in the waiters graph and therefore several searches
  for deadlock should be performed by MDL deadlock detector.
mysql-test/t/mdl_sync.test:
  Added test for bug #50998 "Deadlock in MDL code during test
  rqg_mdl_stability" as well as coverage for the case when
  addition of statement waiting for metadata lock adds several
  loops in the waiters graph and therefore several searches
  for deadlock should be performed by MDL deadlock detector.
sql/mdl.cc:
  Ensure that in cases when MDL deadlock detector had to
  perform several searches for deadlock because several loops
  in waiters graph are possible we reset
  Deadlock_detection_context before performing each search.
  Failure to do so has broken assumption in code resposible
  for choosing victim that if Deadlock_detection_context::victim
  is set we also have read lock on m_waiting_for_lock for this
  context. As result this lock could have been unlocked more
  times than it was acquired which corrupted rwlock's state
  (no one was able to acquire write lock on it anymore).

2. Update test result files.

Conflicts:
  - mysql-test/suite/ndb/r/ndb_dd_ddl.result

fixing bug#50950.

fixing bug#50950.

and MDL".

Concurrent execution of a multi-DELETE statement and ALTER
TABLE statement which affected one of the tables used in
the multi-DELETE sometimes led to deadlock.
Similar deadlocks might have occured when one performed
INSERT/UPDATE/DELETE on a view and concurrently executed
ALTER TABLE for the view's underlying table, or when one
concurrently executed TRUNCATE TABLE for InnoDB table and
ALTER TABLE for the same table.

These deadlocks were caused by a discrepancy between types of
metadata and thr_lock.cc locks acquired by those statements.

What happened was that multi-DELETE/TRUNCATE/DML-through-the-
view statement in the first connection acquired SR lock on a
table, then ALTER TABLE would come in in the second connection
and acquire SNW metadata lock and TL_WRITE_ALLOW_READ
thr_lock.c lock and then would start waiting for the first
connection during lock upgrade. After that the statement in
the first connection would try to acquire TL_WRITE lock on
table and would start waiting for the second connection,
creating a deadlock.

This patch solves this problem by ensuring that we acquire
SW metadata lock in all cases in which we acquiring write
thr_lock.c lock. This guarantees that deadlocks like the
one described above won't occur since all lock conflicts
in such situation are resolved within MDL subsystem.

This patch also adds assert which should guarantee that
such situations won't arise in future.

mysql-test/r/lock_multi.result:
  Added main test for bug #50913 "Deadlock between
  open_and_lock_tables_derived and MDL".
mysql-test/r/mdl_sync.result:
  Added additional coverage for bug #50913 "Deadlock
  between open_and_lock_tables_derived and MDL".
mysql-test/t/lock_multi.test:
  Added main test for bug #50913 "Deadlock between
  open_and_lock_tables_derived and MDL".
mysql-test/t/mdl_sync.test:
  Added additional coverage for bug #50913 "Deadlock
  between open_and_lock_tables_derived and MDL".
sql/lock.cc:
  Added assert that enforces that when we are locking
  a non-temporary table we have an appropriate type of
  metadata lock on this table.
sql/mysql_priv.h:
  Added separate flag for open_tables() to be able specify that
  SH metadata locks on table to be open should be acquired.
  We can no longer use MYSQL_LOCK_IGNORE_FLUSH flag for this
  as in addition to use in I_S implementation it is also used
  for opening system tables. Since in the latter case we also
  acquire thr_lock.c locks using SH metadata lock in it instead
  of SR or SW locks may lead to deadlock.
sql/sql_base.cc:
  When opening tables don't interpret MYSQL_LOCK_IGNORE_FLUSH
  flag as request to acquire SH metadata locks. This flag is
  also used for opening system tables for which we also take
  thr_lock.c locks and thus proper metadata lock to take in
  this case is SR or SW lock (otherwise deadlocks can occur).
  In cases when SH lock is really required (e.g. when tables
  are open by I_S implementation) we rely on that newly
  introduced MYSQL_OPEN_FORCE_SHARED_HIGH_PRIO_MDL flag is
  used.
sql/sql_delete.cc:
  mysql_truncate_by_delete():
    Adjust type of metadata lock to be requested after changing
    type of thr_lock.c lock for table list element from one
    which was set in parser to TL_WRITE.
    This removes discrepancy between types of these locks which
    allowed deadlocks to creep in.
sql/sql_handler.cc:
  When closing table which was open by HANDLER statement clear
  TABLE::open_by_handler flag. This allows to use this flag as
  a reliable indication that TABLE instance was open by HANDLER
  statement in assert which was added to mysql_lock_tables().
sql/sql_parse.cc:
  multi_delete_set_locks_and_link_aux_tables():
    Adjust type of metadata lock to be requested after changing
    type of thr_lock.c lock for table list element from one
    which was set in parser to TL_WRITE.
    This removes discrepancy between types of these locks which
    allowed deadlocks to creep in.
sql/sql_show.cc:
  Use newly introduced MYSQL_OPEN_FORCE_SHARED_HIGH_PRIO_MDL
  flag in order to acquire SH metadata locks when opening tables
  in I_S implementation.
sql/sql_update.cc:
  Added comment explaining why in multi-update after deciding
  that we need weaker thr_lock.c lock on a table we don't
  downgrade metadata lock on it.
sql/sql_view.cc:
  When merging view into main statement adjust type of metadata
  lock to be requested after changing type of thr_lock.c lock
  for table. This removes discrepancy between types of these
  locks which allowed deadlocks to creep in.

                in message printed at end of configure

New text for the success message of "configure".

configure.in:
  The message must be changed to drop the "www.mysql.com" URL.

mysql-test/t/disabled.def:
  Restore disabled ssl tests: SSL certificates were updated.
  Disable sp_sync.test, the test case can't work in next-4284.
mysql-test/t/partition_innodb.test:
  Disable parsing of the test case for Bug#47343, 
  the test can not work in next-4284.
mysql-test/t/ps_ddl.test:
  Update results (CREATE TABLE IF NOT EXISTS takes
  into account existence of the temporary table).

           failed on HANDLER + I_S

This assert was triggered when an I_S query tried to acquire a
metadata lock on a table which was already locked by a HANDLER
statement in the same connection.

First the HANDLER took a MDL_SHARED lock. Afterwards, the I_S query
requested a MDL_SHARED_HIGH_PRIO lock. The existing MDL_SHARED ticket
is found in find_ticket() since it satisfies 
ticket->has_stronger_or_equal_type(mdl_request->type) as MDL_SHARED
and MDL_SHARED_HIGH_PRIO have equal strengths, just different priority.

However, two asserts later check lock type strengths using relational
operators (>= and <=) rather than MDL_ticket::has_stronger_or_equal_type().
These asserts are triggered since MDL_SHARED >= MDL_SHARED_HIGH_PRIORITY
is false (mapped to 1 and 2 respectively).

This patch updates the asserts to use MDL_ticket::has_stronger_or_equal_type()
rather than relational operators to check lock type strength.

Test case added to include/handler.inc.

After merge fixes.
Adjust replication test cases.

mysql-test/suite/rpl/r/rpl_mixed_row_innodb.result:
  Update results with a new test.
mysql-test/suite/rpl/r/rpl_stm_loadfile.result:
  Add a warning, which I believe is an expected one.
mysql-test/suite/rpl/t/rpl_killed_ddl.test:
  Sort results to avoid test failurs under load.
mysql-test/suite/rpl_ndb/r/rpl_ndb_binlog_format_errors.result:
  Update results (next-4284 merge).
mysql-test/suite/rpl_ndb/t/rpl_ndb_binlog_format_errors.test:
  Adjust test output to the new table opening scheme: decide_logging_format() is now called in CREATE/DROP trigger.

Original revision:
------------------------------------------------------------
revision-id: kent.boortz@sun.com-20100204182709-dw1dwpglkd5qrehb
committer: Kent Boortz <kent.boortz@sun.com>
branch nick: mysql-5.1-bugteam
timestamp: Thu 2010-02-04 19:27:09 +0100
message:
  LT_INIT and LT_PREREQ was added in libtool 2.2 2008, a bit too
  recent, switched back to the older AC_PROG_LIBTOOL
------------------------------------------------------------

format.

mysql-test/suite/rpl/t/rpl_view_multi.test:
  Add a suppression (fix a failing test).

Change the error code for ER_WARN_I_S_SKIPPED_TABLE, to not
upset the tests that rely on ER_SLAVE_CONVERSION_ERROR error
code = 1667.
Fix a merge bug with binlogging of CREATE TABLE (temporary tables).

           HANDLER OPEN

The problem was a too restrictive assert in the code for 
HANDLER ... OPEN and HANDLER ... READ that checked table->next
to verify that we didn't open views or merge tables.

This pointer is also used to link temporary tables together
(see thd->temporary_tables). In this case TABLE::next can be
set even if we're trying to open a single table.

This patch adjust the two asserts to also check for the presence
of temporary tables.

Test case added to handler_myisam.test.

fix a merge bug when write_bin_log called from mysql_routine_grant()
would chew up the error.
rpl_do_grant test would fail on assert that the diagnostics area is
empty.

Make all mutexes and conditions of type mysql_mutex_t, mysql_cond_t,
since it's now the expectation of THD::awake().

Cherry-pick a fix Bug#37148 from next-mr, to preserve
file ids of the added files, and ensure that all the necessary
changes have been pulled.

Since initially Bug#37148 was null-merged into 6.0,
the changeset that is now being cherry-picked was likewise
null merged into next-4284.

Now that Bug#37148 has been reapplied to 6.0, try to make
it work with next-4284. This is also necessary to be able
to pull other changes from 5.1-rep into next-4284.

To resolve the merge issues use this changeset applied
to 6.0:
revid:jperkin@sun.com-20091216103628-ylhqf7s6yegui2t9
revno: 3776.1.1
committer: He Zhenxing <zhenxing.he@sun.com>
branch nick: 6.0-codebase-bugfixing
timestamp: Thu 2009-12-17 17:02:50 +0800
message:
  Fix merge problem with Bug#37148