Analysis:
The problem lies in filesort.cc:find_all_keys().

When find_all_keys() is called for the outer query, it resets all
of the tree sets of fields - [read,write,vcol]_set and recomputes
them with respect to sorting.

However, in the loop for each current record the procedure calls
select->skip_record(thd), which evaluates the where clause, which
in turns evaluates the subquery. The JOIN evaluation of the
subquery eventually calls Field_long::val_int to evaluate the field
alias1.f1. The assertion condition
  "bitmap_is_set(table->read_set, field_index)"
fails, because the outer query changed the read_set of table "alias1".

Solution:
Restore the original read_set of the table before calling
SQL_SELECT::skip_record, then revert back to the read_set used in
find_all_keys.

Changed rows_read and rows_sent status variables to be longlong (to avoid compiler warnings)


sql/item_func.cc:
  Fixed wrong usage of alias
sql/item_subselect.cc:
  Changed buffer size to ulonglong to be able detect buffers bigger than size_t
sql/item_subselect.h:
  Changed buffer size to ulonglong to be able detect buffers bigger than size_t
sql/multi_range_read.cc:
  Fixed compiler warning by using correct type for function argument
sql/mysqld.cc:
  Changed rows_read and rows_sent status variables to be longlong
sql/opt_subselect.h:
  Fixed compiler warning by using correct type for function argument
sql/sql_class.cc:
  Changed rows_read and rows_sent status variables to be longlong
sql/sql_class.h:
  Changed rows_read and rows_sent status variables to be longlong
  Changed max_nulls_in_row to uint as this is number of columns in row.
  This fixed some compiler warnings.
sql/sql_select.cc:
  Added casts to avoid compiler warnings
storage/heap/ha_heap.cc:
  Initilize different types separate
storage/oqgraph/ha_oqgraph.cc:
  Fixed argument to store(longlong) to avoid compiler warnings

- Changed Cached_item_field not copy data for fields with NULL value
- In key_copy() and key_restore() don't copy data for fields with NULL value

Fixed code to avoid valgrind warnings
- Use c_ptr_safe instead of c_ptr()

Removed "QQ" from comments (QQ was ment to be used for internal comments that should be removed before pushing)
Fixed wrong alias used (from previous patch)


sql/event_db_repository.cc:
  Update testing if event table is valid (to avoid valgrind errors)
sql/ha_partition.cc:
  m_ordered_scan_ongoing was not initialized
  Reset null bits in record to avoid valgrind errors
sql/handler.h:
  Added flag if storage engine will write row verbatim and the row contains varchar or null fields
  (in which case we must clear the row to avoid valgrind warnings)
sql/item_buff.cc:
  Changed Cached_item_field not copy data for fields with NULL value
  (Optimization and avoids valgrind warnings)
sql/item_func.cc:
  c_ptr() -> c_ptr_safe()
sql/key.cc:
  In key_copy() and key_restore() don't copy data for fields with NULL value
sql/opt_range.cc:
  c_ptr() -> c_ptr_safe()
sql/sql_base.cc:
  Added TRASH() to table->record[0] to find out if we access not initialzed data.
  Initialize null_bytes to:
  - Get consistent tests
  - Ensure we don't get valgrind warnings for null fields (as we may only update a couple of bits in a byte)
sql/sql_class.cc:
  Removed "QQ" from comments
sql/sql_insert.cc:
  Initialize row to default values if we are using valgrind and row will be copied verbatim to disk in storage engine.
sql/sql_load.cc:
  QQ -> TODO
sql/sql_parse.cc:
  Removed old not used code marked QQ and withing "#ifdef REMOVED"
sql/sql_select.cc:
  QQ -> TODO
  Initialize some variables that was used uninitialized
  Added DBUG_ASSERT() to find out if thd was not properly initialized for sub queries
sql/sql_test.cc:
  Fixed format for printing to DBUG file
  Fixed wrong alias used (from previous patch)
sql/sql_trigger.h:
  QQ -> TODO
sql/table.cc:
  QQ -> TODO
storage/maria/ha_maria.cc:
  Mark table with HA_RECORD_MUST_BE_CLEAN_ON_WRITE, if row is written verbatim to disk and contains varchar or null fields.
storage/maria/ma_open.c:
  Added flags if table has varchar or null fields
storage/maria/maria_def.h:
  Added flags if table has varchar or null fields
storage/myisam/ha_myisam.cc:
  Mark table with HA_RECORD_MUST_BE_CLEAN_ON_WRITE, if row is written verbatim to disk and contains varchar or null fields.
storage/myisam/mi_open.c:
  Fixed memory overrun bug when using fulltext keys
storage/xtradb/row/row0sel.c:
  Removed initialization of null bits. (not needed anymore)

Analysis:
JOIN::optimize performs constant optimization of GROUP by clauses
by calling remove_const():
    group_list= remove_const(this, (old_group_list= group_list), conds,
                             rollup.state == ROLLUP::STATE_NONE,
			     &simple_group);
If it turns out that a GROUP clause references a field that is
computed by a single-row subquery, then the said optimization
performs premature execution of the subquery referenced by the
group clause.

Solution:
Block the evaluation of subqueries similarly to the approach
for the WHERE and JOIN..ON clauses.

Fix MySQL BUG#52344 - Subquery materialization: Assertion if subquery in on-clause of outer join

Original fix and comments from Oysten, adjusted for the different
subquery optimization in MariaDB.
"
Problem: If tables of an outer join are constant tables,
the associated on-clause will be evaluated in the optimization
phase. If the on-clause contains a query that is to be
executed with subquery materialization, this will not work
since the infrastructure for such execution is not yet set up.
      
Solution: Do not evaluate on-clause in optimization phase if
is_expensive() returns true for this clause.  This is how the
problem is currently avoided for where-clauses.  This works
because, Item_in_subselect::is_expensive_processor returns true
if query is to be executed with subquery materialization.
"
In addition, after MWL#89, in MariaDB if the IN-EXISTS strategy
is chosen, the in-to-exists predicates are insterted after
join_read_const_table() is called, resulting in evaluation of
the subquery without the in-to-exists predicates.

  
Analysis:
The send_data method of the result sink class used to collect
data statistics about materialized subqueries incorrectly assumed
that duplicate rows are removed prior to calling send_data. As
a result the collected statistics was wrong, which resulted in
an incorrect maximal number of keys in the Ordered_key buffer.
  
Solution:
Try to insert each row into the materialized temp table before
collecting statistics, and if the insertion results in a duplicate
row, do not count the current row.

- Changed TABLE->alias to String to get fewer reallocs when alias are used.
- Preallocate some buffers

Changed some String->c_ptr() -> String->ptr() when \0 is not needed.
Fixed wrong usage of String->ptr() when we need a \0 terminated string.
Use my_strtod() instead of my_atof() to avoid having to add \0 to string.
c_ptr() -> c_ptr_safe() to avoid warnings from valgrind.
zr 

sql/event_db_repository.cc:
  Update usage of TABLE->alias
sql/event_scheduler.cc:
  c_ptr() -> c_ptr_safe()
sql/events.cc:
  c_ptr() -> ptr() as \0 was not needed
sql/field.cc:
  Update usage of TABLE->alias
sql/field.h:
  Update usage of TABLE->alias
sql/ha_partition.cc:
  Update usage of TABLE->alias
sql/handler.cc:
  Update usage of TABLE->alias
  Fixed wrong usage of str.ptr()
sql/item.cc:
  Fixed error where code wrongly assumed string was \0 terminated.
sql/item_func.cc:
  c_ptr() -> c_ptr_safe()
  Update usage of TABLE->alias
sql/item_sum.h:
  Use my_strtod() instead of my_atof() to avoid having to add \0 to string
sql/lock.cc:
  Update usage of TABLE->alias
sql/log.cc:
  c_ptr() -> ptr() as \0 was not needed
sql/log_event.cc:
  c_ptr_quick() -> ptr() as \0 was not needed
sql/opt_range.cc:
  ptr() -> c_ptr() as \0 is needed
sql/opt_subselect.cc:
  Update usage of TABLE->alias
sql/opt_table_elimination.cc:
  Update usage of TABLE->alias
sql/set_var.cc:
  ptr() -> c_ptr() as \0 is needed
  c_ptr() -> c_ptr_safe()
sql/sp.cc:
  c_ptr() -> ptr() as \0 was not needed
sql/sp_rcontext.cc:
  
  Update usage of TABLE->alias
sql/sql_base.cc:
  Preallocate buffers
  Update usage of TABLE->alias
sql/sql_class.cc:
  Fix arguments to sprintf() to work even if string is not \0 terminated
sql/sql_insert.cc:
  Update usage of TABLE->alias
  c_ptr() -> ptr() as \0 was not needed
sql/sql_load.cc:
  Preallocate buffers
  Trivial optimizations
sql/sql_parse.cc:
  Trivial optimization
sql/sql_plugin.cc:
  c_ptr() -> ptr() as \0 was not needed
sql/sql_select.cc:
  Update usage of TABLE->alias
sql/sql_show.cc:
  Update usage of TABLE->alias
sql/sql_string.h:
  Added move() function to move allocated memory from one object to another.
sql/sql_table.cc:
  Update usage of TABLE->alias
  c_ptr() -> c_ptr_safe()
sql/sql_test.cc:
  ptr() -> c_ptr_safe()
sql/sql_trigger.cc:
  Update usage of TABLE->alias
  c_ptr() -> c_ptr_safe()
sql/sql_update.cc:
  Update usage of TABLE->alias
sql/sql_view.cc:
  ptr() -> c_ptr_safe()
sql/sql_yacc.yy:
  ptr() -> c_ptr()
sql/table.cc:
  
  Update usage of TABLE->alias
sql/table.h:
  Changed TABLE->alias to String to get fewer reallocs when alias are used.
storage/federatedx/ha_federatedx.cc:
  Use c_ptr_safe() to ensure strings are \0 terminated.
storage/maria/ha_maria.cc:
  Update usage of TABLE->alias
storage/myisam/ha_myisam.cc:
  Update usage of TABLE->alias
storage/xtradb/row/row0sel.c:
  Ensure that null bits in record are properly reset.
  (Old code didn't work as row_search_for_mysql() can be called twice while reading fields from one row.

mysql-test/r/heap_btree.result:
  Test of index over bit firld in hash table.
mysql-test/r/heap_hash.result:
  Test of index over bit firld in hash table.
mysql-test/t/heap_btree.test:
  Test of index over bit firld in hash table.
mysql-test/t/heap_hash.test:
  Test of index over bit firld in hash table.
storage/heap/ha_heap.cc:
  Adding bit field support for heap tables.
storage/heap/hp_create.c:
  Adding bit field support for heap tables.
storage/heap/hp_hash.c:
  Adding bit field support for heap tables.

Analysis:
Single-row subqueries are not considered expensive and are
evaluated both during EXPLAIN in to detect errors like
"Subquery returns more than 1 row", and during optimization to
perform constant optimization.

The cause for the failed ASSERT is in JOIN::join_free, where we set
  bool full= (!select_lex->uncacheable && !thd->lex->describe);
Thus for EXPLAIN statements full == FALSE, and as a result the call to
JOIN::cleanup doesn't call JOIN_TAB::cleanup which should have
called table->disable_keyread().

Solution:
Consider all kinds of subquery predicates as expensive.

Analysis:
This another instance of the problem fixed in LP BUG#675981 -
evaluation of subqueries during EXPLAIN when the subquery plan
is incomplete because JOIN::optimize() generally doesn't create
complete execution plans for EXPLAIN statements.

In this case the call path is:
mysql_explain_union -> outer_join.exec -> outer_join.init_execution ->
create_sort_index -> filesort -> find_all_keys ->
SQL_SELECT::skip_record -> outer_where_clause.val_int -> ...
-> subselect_join.exec -> ... -> sub_select_cache

When calling sub_select_cache JOIN_TAB::cache is NULL because the cache
objects are not created for EXPLAIN statements.

Solution:
Delay the call to init_execution() after all EXPLAIN related processing
is completed. Thus init_execution() is not called at all during EXPLAIN.

Cause:
The optimize() phase for the subquery selected to use join buffering via setting
JOIN_TAB::next_select= sub_select_cache in make_join_readinfo, however, the call
to check_join_cache_usage() from make_join_readinfo didn't create the corresponding
JOIN_CACHE_BNL object because of the condition:
    if ((options & SELECT_DESCRIBE) ||
        (((tab->cache= new JOIN_CACHE_BNL(join, tab, prev_cache))) &&
         !tab->cache->init()))
Since EXPLAIN for subqueries runs regular execution, the constant predicates that
were delayed to be evaluated at the exec() phase, were evaluated during EXPLAIN.
As a result the outer JOIN::exec called JOIN::exec for the subquery, while the
subquery execution plan was no properly created, which resulted in an failed ASSERT.

Fix:
The patch blocks evaluation of constant expensive conditions during EXPLAIN. Notice
that these conditions are "constant" with respect to the outer query, thus in
general they could be arbitrarily expensive, which may result in very slow EXPLAINs.

This is a backport of the fix for
MySQL BUG#52317: Assertion failing in Field_varstring::store () at field.cc:6833

The orginal comment by Oystein is:

In order for EXPLAIN to print const-refs, a Store_key_const_item object
is created. This is different for normal execution of subqueries where
a temporary store_key_item object is used instead. The problem is that
EXPLAIN will execute subqueries.  This leads to a scenario where a
store_key_const_item object it told to write to its underlying field.
This results in a failing assert since the write set of the underlying
table does not reflect this.  

The resolution is to do the same trick as for store_key_item::copy_inner().
That is, temporarily change the write set to allow writes to all columns.
This is only necessary in debug version since non-debug version does not
contain asserts on write_set.

sql/sql_select.h:
  Temporarily change write_set in store_key_const_item::copy_inner() to
  allow initialization of underlying field.  This is necessary since 
  subqueries are executed for EXPLAIN.  (For normal execution, 
  store_key_item::copy_inner is used.)

  anticipate different execution paths resulting in different
  thd->proc_info.
- Fixed subselect_cache to contain correct results. The results
  are currently wrong in 5.3, but are correct in 5.2, and 5.3-mwl89.

The cause for the bug was two-fold:
1. Incorrect detection of whether a table is the first one in a query plan -
  "used_table & 1" actually checks if used_table is table with number "1".
2. Missing logic to delay the evaluation of (expensive) constant conditions
  during the execution phase.

The fix adds/changes:
The patch:
- removes incorrect treatment of expensive predicates from make_cond_for_table,
  and lets the caller decide when to evaluate expensive predicates.
- saves expensive constant conditions in JOIN::exec_const_cond,
  which is evaluated once in the beginning of JOIN::exec.

mysql-test/suite/maria/t/compat_aliases.test:
  Removed column with version number
sql/sql_show.cc:
  Fixed compiler warning

The pushdown condition for the sorted table in a query can be complemented
by the conditions from HAVING. This transformation is done in JOIN::exec
pretty late after the original pushdown condition have been saved in the
field pre_idx_push_select_cond for the sorted table. So this field must
be updated after the inclusion of the condition from HAVING.

- Make mi_open() use less stack space

Avoid doing reallocs
Prealloc some strings / provide extension allocation size to some strings
This gave a 25 % speedup in some mysql-test-run tests.



mysys/safemalloc.c:
  More DBUG_PRINT
sql/net_serv.cc:
  Make all mallocs() look the similar. (just-for-safety fix)
sql/protocol.cc:
  Ensure that communication packet buffer is allocated.
  (It's freed by stored precedures and some DLL statements)
sql/sp.cc:
  Fixed valgrind warning
sql/sql_select.cc:
  Set extent allocation for buffer that has a lot of append() calls.
sql/sql_show.cc:
  Fixed wrong usage of string buffer. Old code worked in test suite 'just-by-chance'
sql/sql_string.cc:
  Call realloc_with_extra_if_needed() in append() functions.
sql/sql_string.h:
  Added 'extra_alloc' member, to specify chunck size for realloc().
  extra_alloc is addaptive to catch cases where preallocation of buffers is not done properly.
  Simplified free() to allow compiler to optimize things better (and to keep things consistent).
  Fixed shrink() to take into account the extra memory added to the Alloced_length in realloc(). This saves us a realloc() per query.
sql/sql_test.cc:
  Set extent allocation for buffer that has a lot of append() calls.
sql/table.cc:
  Set extent allocation for buffer that has a lot of append() calls.

The bug itself got fixed after merging with the main 5.3.

Merge 5.3-mwl89 into 5.3 main.

There is one remaining test failure in this merge:
innodb_mysql_lock2. All other tests have been checked to
deliver the same results/explains as 5.3-mwl89, including
the few remaining wrong results.