Problem was that ha_partition::records_in_range called
records_in_range for all non pruned partitions, even if
an estimate should be given.

Solution is to only use 1/3 of the partitions (up to 10) for
records_in_range and estimate the total from this subset.
(And continue until a non zero return value from the called
partitions records_in_range is given, since 0 means no rows
will match.)

sql/ha_partition.cc:
  Bug#48846: Too much time spent in ha_partition::records_in_range if not able to prune
  
  estimate_rows_upper_bound and records_in_range are very similar
  (the only difference is the function and its parameters to use)
  so I created a common function for this.
  
  Since these calls from the optimizer are only estimates, it is
  not neccesary to call them for every partition, it can use
  a much smaller subset of the used partitions instead,
  which improves performance for selects.
sql/ha_partition.h:
  Bug#48846: Too much time spent in ha_partition::records_in_range if not able to prune
  
  Added two private functions to help some
  optimizer calls.

           deadlock was encountered

The bug is caused by an inconsistent handling of the IGNORE
clause.  A read from a const table caused a lock timeout
(ER_LOCK_TIMEOUT) in innodb.  Since the IGNORE clause was
given, the timeout was converted into a warning instead of
an error, thus not populating the diagnostics area.  When
innodb subsequently marked the transaction for rollback,
mysql asserted since the diag.area was empty.

This patch consists of only a test case, as the bug itself
was fixed by the patch for Bug #46539

       on any access

Archive engine for 5.1 (and latter) version uses a modified 
version of zlib (azlib). These two version are incompatible
so a proper upgrade is needed before tables created in 5.0 
can be used reliable.

This upgrade can be performed using repair. But due to lack 
of test its risky to allow upgrade for now. This patch addresses
only the crashing issue. Any attempt to repair will be blocked.

Eventually repair can be allowed to run through (which will also
cause an upgrade from older version to newer) but only after a 
thorough testing.

mysql-test/r/archive.result:
  Updated result file for test case for bug#47012
mysql-test/std_data/bug47012.ARM:
  part of archive table (t1) created in mysql 5.0
mysql-test/std_data/bug47012.ARZ:
  part of archive table (t1) created in mysql 5.0
mysql-test/std_data/bug47012.frm:
  part of archive table (t1) created in mysql 5.0
mysql-test/t/archive.test:
  Added test case for bug#47012.
storage/archive/azio.c:
  Fixed a minor issues (minor version overwriting version in 
  stream structure)
  Removed assertion when an older version is found. Instead
  setting the correct version (2) in s->version
  If an unknown version is found marked it as corrupt.
storage/archive/ha_archive.cc:
  Detecting the archive version in getShare and marking
  it as need to upgrade.
  Blocking open if the archive needs an upgrade. This
  can be allowed in case of open for repair to upgrade
  the archive but needs to tested.

The crash occurs because SAFEMALLOC is defined for the MySQL server
but not for the Archive or Federated engines, resulting in a 
parameter mismatch between the function prototype and definition
for functions using the CALLER_INFO macro.

storage/archive/CMakeLists.txt:
  Set SAFEMALLOC by default to be consistent with the server.
storage/federated/CMakeLists.txt:
  Set SAFEMALLOC by default to be consistent with the server.

  inside subquery

Re-setting a fulltext index was a no-operation if not all
the matches of a search were consumed by reading them.
This was preventing a joined table using a fulltext index
in a subquery that requires only 1 row of output (e.g. EXISTS) 
from working correctly because the second execution of the 
sub-query has the fulltext index cursor in a wrong state and
was not finding results.
Fixed by making the re-init code _ftb_init_index_search() 
to re-set open cursors in addition to depleted ones.

 values
 
 We should re-set the access method functions when changing the access
 method when switching to another index to avoid sorting.
 
 Fixed by doing a little re-engineering : encapsulating all the function
 assignment into a special function and calling it when flipping the 
 indexes.

  
The prefix --skip- didn't work on 64 bit big endian machines
because of how the value pointer was casted.

mysys/my_getopt.c:
  * Use the interface! The value pointer must correspond to the type mask or it will break on big endian platforms.

  memory

The server was doing a bad class typecast causing setting of 
wrong value for the maximum number of items in an internal
structure used in equality propagation.
Fixed by not doing the wrong typecast and asserting the type
of the Item where it should be done.

When values of different types are compared they're converted to a type that
allows correct comparison. This conversion is done for each comparison and
takes some time. When a constant is being compared it's possible to cache the
value after conversion to speedup comparison. In some cases (large dataset,
complex WHERE condition with many type conversions) query might be executed
7% faster.

A test case isn't provided because all changes are internal and isn't visible
outside.

The behavior of the Item_cache is changed to cache values on the first request
of cached value rather than at the moment of storing item to be cached.
A flag named value_cached is added to the Item_cache class. It's set to TRUE
when cache holds the value of the last stored item.
Function named cache_value() is added to the Item_cache class and derived classes.
This function actually caches the value of the saved item.
Item_cache_xxx::store functions now only store item to be cached and set
value_cached flag to FALSE.
Item_cache_xxx::val_xxx functions are changed to call cache_value function
prior to returning cached value if value_cached is FALSE.
The Arg_comparator::set_cmp_func function now calls cache_converted_constant
to cache constants if they need a type conversion.
The Item_cache::get_cache function is overloaded to allow setting of the
cache type.
The cache_converted_constant function is added to the Arg_comparator class.
It checks whether a value can and should be cached and if so caches it.

sql/item.cc:
  Bug#34384: Slow down on constant conversion.
  Function named cache_value() is added to the Item_cache class and derived classes.
  This function actually caches the value of the saved item.
  Item_cache_xxx::store functions now only store item to be cached and set
  value_cached flag to FALSE.
  Item_cache_xxx::val_xxx functions are changed to call cache_value function
  prior to returning cached value if value_cached is FALSE.
  The Item_cache::get_cache function is overloaded to allow setting of the
  cache type.
sql/item.h:
  Bug#34384: Slow down on constant conversion.
  A flag named value_cached is added to the Item_cache class. It's set to TRUE
  when we need to start caching values when the store method is called.
  Function named cache_value() is added to the Item_cache class and derived classes.
sql/item_cmpfunc.cc:
  Bug#34384: Slow down on constant conversion.
  A helper function cache_converted_constant is added to the Arg_comparator class.
  It checks whether a given item can and should be cached and caches it if so.
  The Arg_comparator::set_cmp_func function now calls cache_converted_constant
  to cache constants if they need a type conversion.
sql/item_cmpfunc.h:
  Bug#34384: Slow down on constant conversion.
  The cache_converted_constant function is added to the Arg_comparator class.
  It checks whether a value can and should be cached and if so caches it.
sql/item_subselect.cc:
  Bug#34384: Slow down on constant conversion.
  Force immediate caching of subselect result.
sql/item_xmlfunc.cc:
  Bug#34384: Slow down on constant conversion.
sql/sp_rcontext.cc:
  Bug#34384: Slow down on constant conversion.
  Force immediate caching of values of an SP CASE function.

            only const tables

The problem was caused by two shortcuts in the optimizer that
are inapplicable in the ROLLUP case.

Normally in a case when only const tables are involved in a
query, DISTINCT clause can be safely optimized away since there
may be only one row produced by the join. Similarly, we don't
need to create a temporary table to resolve DISTINCT/GROUP
BY/ORDER BY. Both of these are inapplicable when the WITH
ROLLUP modifier is present.

Fixed by disabling the said optimizations for the WITH ROLLUP
case.

mysql-test/r/olap.result:
  Added a test case for bug #48475.
mysql-test/t/olap.test:
  Added a test case for bug #48475.
sql/sql_select.cc:
  Disabled const-only table optimizations for the WITH ROLLUP
  case.

Just change mysql_foo to mysql_cv_foo for one cache-id variable name.  There
was only one bad variable name, present in 5.0 and 5.1, but not in the -pe
branch.

The SE API requires mysql to notify the storage engine that
it's going to read certain tables at the beginning of the 
statement (by calling start_stmt(), store_lock() or
external_lock()).
These are typically called by the lock_tables(). 
However SHOW CREATE TABLE is not pre-locking the tables
because it's not expected to access the data at all.
But for some view definitions (that include comparing a
date/datetime/timestamp column to a string returning
scalar subquery) the JOIN::prepare may still access data
when materializing the scalar non-correlated subquery
in Arg_comparator::can_compare_as_dates().
Fixed by not materializing the subquery when the function
is called in a SHOW/EXPLAIN/CREATE VIEW

configure.in:
  Activate silent-rules if automake supports it.

If --log-bin is set to a directory name with the trailing 'FN_LIBCHAR', 
which will be '/' on Unix like systems, and '\\' on Windows like systems. 
the basename of the binlog is empty so that the created files named 
'.000001' and '.index'. It is not expected. 
The same thing happened to --log-bin-index, --relay-log and 
--relay-log-index options.

To resolve the problem, in these cases the program should report an error 
and abort.


sql/mysqld.cc:
  Added a check for the value of the --log-bin and --log-bin-index arguments, 
  if it's a directory, reports an error and aborts.
sql/rpl_rli.cc:
  Added a check for the value of the --relay-log and --relay-log-index arguments, 
  if it's a directory, reports an error and aborts.

BUG#47983: rpl_extraColmaster_myisam failed in PB2 with "Found
warnings!!"

BUG 45214 fixed the case when get_master_version_and_clock
function, used by the slave, would not report errors. The slave
now detects them and if related to transient network failures, it
prints some warnings and retries to connect. On the other hand,
if not network related, it just gives up and fails.

As such, sometimes, in PB2, the slave comes across some transient
communication issues between master and slave, while calling
get_master_version_and_clock, causing warnings print outs to the
error log. Nevertheless, in such cases slave retries to connect,
in which it succeeds, and the test case continues as it normally
would. But then, at the end of a successful test run, MTR checks
the error log, finds the unexpected warnings and considers them
harmful. This causes MTR to report error and, consequently, PB2
to report a failing test.

We fix this by adding to the global warnings suppress list the
warnings related to transient network failures only, which are
reported while in function get_master_version_and_clock.

  special chars

This script failed when the user tried passwords with multiple spaces, \, # or
' characters.  Now proper escaping and quoting is used in all contexts.

This problem occurs in the Perl version of this script, too, so fix it in both
places.

Remove a bash-ism (if ! ...).

Bug#41756 "Strange error messages about locks from InnoDB".
      
In JT_EQ_REF (join_read_key()) access method, 
don't try to unlock rows in the handler, unless certain that 
a) they were locked
b) they are not used.

Unlocking of rows is done by the logic of the nested join loop,
and is unaware of the possible caching that the access method may
have. This could lead to double unlocking, when a row
was unlocked first after reading into the cache, and then 
when taken from cache, as well as to unlocking of rows which
were actually used (but taken from cache).
      
Delegate part of the unlocking logic to the access method,
and in JT_EQ_REF count how many times a record was actually 
used in the join. Unlock it only if it's usage count is 0.

Implemented review comments.

mysql-test/r/innodb_lock_wait_timeout_1.result:
  Update results (Bug41756).
mysql-test/t/innodb_lock_wait_timeout_1.test:
  Add a test case (Bug#41756).
sql/item_subselect.cc:
  Complete struct READ_RECORD initialization with a new
  member to unlock records.
sql/records.cc:
  Extend READ_RECORD API with a method to unlock read records.
sql/sql_select.cc:
  In JT_EQ_REF (join_read_key()) access method, 
  don't try to unlock rows in the handler, unless certain that 
  a) they were locked
  b) they are not used.
sql/sql_select.h:
  Add members to TABLE_REF to count TABLE_REF buffer usage count.
sql/structs.h:
  Update declarations.