Some collations were causing IBMDB2I to report
inaccurate key range estimations to the optimizer
for LIKE clauses that select substrings. This can
be seen by running EXPLAIN. This problem primarily
affects multi-byte and unicode character sets.

This patch involves substantial changes to several
modules. There are a number of problems with the
character set and collation handling. These problems
have been or are being fixed,  and a comprehensive
test has been included which should provide much
better coverage than there was before. This test
is enabled only for IBM i 6.1, because that version
has support for the greatest number of collations.

mysql-test/suite/ibmdb2i/r/ibmdb2i_collations.result:
  Bug#45803 Inaccurate estimates for partial key values with IBMDB2I
  
  result file for test case.
mysql-test/suite/ibmdb2i/t/ibmdb2i_collations.test:
  Bug#45803 Inaccurate estimates for partial key values with IBMDB2I
  
  Tests for character sets and collations. This test
  is enabled only for IBM i 6.1, because that version
  has support for the greatest number of collations.
storage/ibmdb2i/db2i_conversion.cc:
  Bug#45803 Inaccurate estimates for partial key values with IBMDB2I
  
  - Added support in convertFieldChars to enable records_in_range
    to determine how many substitute characters were inserted and
    to suppress conversion warnings.
  
  - Fixed bug which was causing all multi-byte and Unicode fields
    to be created as UTF16 (CCSID 1200) fields in DB2. The corrected
    code will now create UCS2 fields as UCS2 (CCSID 13488), UTF8
    fields (except for utf8_general_ci) as UTF8 (CCSID 1208), and
    all other multi-byte or Unicode fields as UTF16.  This will only
    affect tables that are newly created through the IBMDB2I storage
    engine. Existing IBMDB2I tables will retain the original CCSID
    until recreated. The existing behavior is believed to be
    functionally correct, but it may negatively impact performance
    by causing unnecessary character conversion. Additionally, users
    accessing IBMDB2I tables through DB2 should be aware that mixing 
    tables created before and after this change may require extra type
    casts or other workarounds.  For this reason, users who have
    existing IBMDB2I tables using a Unicode collation other than
    utf8_general_ci are encouraged to recreate their tables (e.g.
    ALTER TABLE t1 ENGINE=IBMDB2I) in order to get the updated CCSIDs
    associated with their DB2 tables.
  
  - Improved error reporting for unsupported character sets by forcing
    a check for the iconv conversion table at table creation time,
    rather than at data access time.
storage/ibmdb2i/db2i_myconv.h:
  Bug#45803 Inaccurate estimates for partial key values with IBMDB2I
  
  Fix to set errno when iconv fails.
storage/ibmdb2i/db2i_rir.cc:
  Bug#45803 Inaccurate estimates for partial key values with IBMDB2I
  
  Significant improvements were made to the records_in_range code
  that handles partial length string data in keys for optimizer plan
  estimation. Previously, to obtain an estimate for a partial key
  value, the implementation would perform any necessary character
  conversion and then attempt to determine the unpadded length of
  the partial key by searching for the minimum or maximum sort
  character. While this algorithm was sufficient for most single-byte
  character sets, it did not treat Unicode and multi-byte strings
  correctly. Furthermore, due to an operating system limitation,
  partial keys having UTF8 collations (ICU sort sequences in DB2)
  could not be estimated with this method.
  
  With this patch, the code no longer attempts to explicitly determine
  the unpadded length of the key. Instead, the entire key is converted
  (if necessary), including padding, and then passed to the operating
  system for estimation. Depending on the source and target character
  sets and collations, additional logic is required to correctly
  handle cases in which MySQL uses unconvertible or differently
  -weighted values to pad the key. The bulk of the patch exists
  to implement this additional logic.
storage/ibmdb2i/ha_ibmdb2i.h:
  Bug#45803 Inaccurate estimates for partial key values with IBMDB2I
  
  The convertFieldChars declaration was updated to support additional 
  optional behaviors.

timeout
            
In STMT and MIXED modes, a statement that changes both non-transactional and
transactional tables must be written to the binary log whenever there are
changes to non-transactional tables. This means that the statement gets into the
binary log even when the changes to the transactional tables fail. In particular
, in the presence of a failure such statement is annotated with the error number
and wrapped in a begin/rollback. On the slave, while applying the statement, it
is expected the same failure and the rollback prevents the transactional changes
to be persisted.
            
Unfortunately, statements that fail due to concurrency issues (e.g. deadlocks,
timeouts) are logged in the same way causing the slave to stop as the statements
are applied sequentially by the SQL Thread. To fix this bug, we automatically
ignore concurrency failures on the slave. Specifically, the following failures
are ignored: ER_LOCK_WAIT_TIMEOUT, ER_LOCK_DEADLOCK and ER_XA_RBDEADLOCK.

          server

If the server connection was lost during repeated status commands,
the client would fail to detect this and the client output would be inconsistent.

This patch fixes this issue by making sure that the server is online
before the client attempts to execute the status command.


client/mysql.cc:
  * Replace variable "connected" with a call to mysql_real_query_for_lazy()
    will attempt to reconnect to server on if there is a failure.

The crash happend because for views which are joins
we have table_list->table == 0 and 
table_list->table->'any method' call leads to crash.
The fix is to perform table_list->table->file->extra()
method for all tables belonging to view.


mysql-test/r/view.result:
  test result
mysql-test/t/view.test:
  test case
sql/sql_insert.cc:
  added prepare_for_positional_update() function
  which updates extra info about primary key for
  tables belonging to view.

enabled message storing into error message list
for 'drop table' command


mysql-test/r/warnings.result:
  test result
mysql-test/t/warnings.test:
  test case
sql/sql_table.cc:
  We should skip error sending then we should return
  warnings to client as some functions may send its
  own errors, so we should set no_warnings_for_error= 0
  only in case of warning.
  The fix is to enable message storing into error message
  list for 'drop table' command(only for error case).
tests/mysql_client_test.c:
  test fix

 
Using DECIMAL constants with more than 65 digits in CREATE 
TABLE ... SELECT led to bogus errors in release builds or 
assertion failures in debug builds. 
 
The problem was in inconsistency in how DECIMAL constants and 
fields are handled internally. We allow arbitrarily long 
DECIMAL constants, whereas DECIMAL(M,D) columns are limited to 
M<=65 and D<=30. my_decimal_precision_to_length() was used in 
both Item and Field code and truncated precision to 
DECIMAL_MAX_PRECISION when calculating value length without 
adjusting precision and decimals. As a result, a DECIMAL 
constant with more than 65 digits ended up having length less 
than precision or decimals which led to assertion failures. 
 
Fixed by modifying my_decimal_precision_to_length() so that 
precision is truncated to DECIMAL_MAX_PRECISION only for Field 
object which is indicated by the new 'truncate' parameter. 
 
Another inconsistency fixed by this patch is how DECIMAL 
constants and expressions are handled for CREATE ... SELECT. 
create_tmp_field_from_item() (which is used for constants) was 
changed as a part of the bugfix for bug #24907 to handle long 
DECIMAL constants gracefully. Item_func::tmp_table_field() 
(which is used for expressions) on the other hand was still 
using a simplistic approach when creating a Field_new_decimal 
from a DECIMAL expression. 

mysql-test/r/type_newdecimal.result:
  Added a test case for bug #45262.
mysql-test/t/type_newdecimal.test:
  Added a test case for bug #45262.
sql/item.cc:
  Use the new 'truncate' parameter in 
  my_decimal_precision_to_length().
sql/item_cmpfunc.cc:
  Use the new 'truncate' parameter in 
  my_decimal_precision_to_length().
sql/item_func.cc:
  1. Use the new 'truncate' parameter in 
  my_decimal_precision_to_length().
  
  2. Do not truncate decimal precision to DECIMAL_MAX_PRECISION
  for additive expressions involving long DECIMAL constants.
  
  3. Fixed an incosistency in how DECIMAL constants and 
  expressions are handled for CREATE ... SELECT.
sql/item_func.h:
  Use the new 'truncate' parameter in 
  my_decimal_precision_to_length().
sql/item_sum.cc:
  Use the new 'truncate' parameter in 
  my_decimal_precision_to_length().
sql/my_decimal.h:
  Do not truncate precision to DECIMAL_MAX_PRECISION
  when calculating length in 
  my_decimal_precision_to_length() if 'truncate' parameter
  is FALSE.
sql/sql_select.cc:
  1. Use the new 'truncate' parameter in 
  my_decimal_precision_to_length().
  
  2. Use a more correct logic when adjusting value's length.

contains ONLY_FULL_GROUP_BY

The partitioning code needs to issue a Item::fix_fields()
on the partitioning expression in order to prepare 
it for being evaluated.
It does this by creating a special table and a table list 
for the scope of the partitioning expression.
But when checking ONLY_FULL_GROUP_BY the 
Item_field::fix_fields() was relying that there always be
cached_table set and was trying to use it to get the 
select_lex of the SELECT the field's table is in.
But the cached_table was not set by the partitioning code
that creates the artificial TABLE_LIST used to resolve the
partitioning expression and this resulted in a crash.
 
Fixed by rectifying the following errors :
1. Item_field::fix_fields() : the code that check for 
ONLY_FULL_GROUP_BY relies on having tables with 
cacheable_table set. This is mostly true, the only 
two exceptions being the partitioning context table
and the trigger context table.
Fixed by taking the current parsing context if no pointer
to the TABLE_LIST instance is present in the cached_table.

2. fix_fields_part_func() : 

2a. The code that adds the table being created to the 
scope for the partitioning expression is mostly a copy 
of the add_table_to_list and friends with one exception :
it was not marking the table as cacheable (something that
normal add_table_to_list is doing). This caused the 
problem in the check for ONLY_FULL_GROUP_BY in 
Item_field::fix_fields() to appear.
Fixed by setting the correct members to make the table
cacheable.
The ideal structural fix for this is to use a unified 
interface for adding a table to a table list 
(add_table_to_list?) : noted in a TODO comment

2b. The Item::fix_fields() was called with a NULL destination
pointer. This causes uninitalized memory reads in the 
overloaded ::fix_fields() function (namely 
Item_field::fix_fields()) as it expects a non-zero pointer 
there. Fixed by passing the source pointer similarly to how 
it's done in JOIN::prepare().

mysql-test/r/partition.result:
  Bug #45807: test case
mysql-test/t/partition.test:
  Bug #45807: test case
sql/item.cc:
  Bug #45807: fix the ONLY_FULL_GROUP_BY check code to 
  handle correctly non-cacheable tables.
sql/sql_partition.cc:
  Bug #45807: fix the Item::fix_fields() context
  initializatio for the partitioning expression in 
  CREATE TABLE.

Details:
- Add "sync_slave_with_master" at test end
- Restore the initial value of log_bin_trust_function_creators

Creating an IBMDB2I table with the macce character set
is successful, but any attempt to insert data into the
table was failing.

This was happening because the character set name "macce"
is not a valid iconv descriptor for IBM i PASE. This patch
adds an override to convertTextDesc to use the equivalent
valid iconv descriptor "IBM-1282" instead.

mysql-test/suite/ibmdb2i/r/ibmdb2i_bug_45793.result:
  Bug#45793 macce charset causes error with IBMDB2I
  
  Result file for the test case.
mysql-test/suite/ibmdb2i/t/ibmdb2i_bug_45793.test:
  Bug#45793 macce charset causes error with IBMDB2I
  
  Add a testcase for the macce charater set.
storage/ibmdb2i/db2i_charsetSupport.cc:
  Bug#45793 macce charset causes error with IBMDB2I
  
  The character set name "macce" is not a valid iconv
  descriptor for IBM i PASE. Add an override to convertTextDesc
  to use the equivalent valid iconv descriptor "IBM-1282" 
  instead.

without proper formatting
      
The problem is that a suitably crafted database identifier
supplied to COM_CREATE_DB or COM_DROP_DB can cause a SIGSEGV,
and thereby a denial of service. The database name is printed
to the log without using a format string, so potential
attackers can control the behavior of my_b_vprintf() by
supplying their own format string. A CREATE or DROP privilege
would be required.
      
This patch supplies a format string to the printing of the
database name. A test case is added to mysql_client_test.


sql/sql_parse.cc:
  Added format strings.
tests/mysql_client_test.c:
  Added new test case.

After applying Innodb snapshot 5.1-ss5282, build was broken
because of missing header file. 

Adding the header file to Makefile.am after informing the 
innodb developers.

Add new header to Makefile.am so it gets bundled.

storage/innobase/Makefile.am:
  Add header so it becomes part of a source distribution.

                               But Should Be "Zero Rows"


After applying the innodb snapshot 5.0-ss5406 for bug#40565, the windows push build
tests failed because of the missing cast of void * pointer in row0sel.c file

Informed the innodb developers and received patch by email.

innobase/row/row0sel.c:
  Cast the default_rec which is a void * pointer

Some collations--including cp1250_czech_cs,latin2_czech_cs,
ucs2/utf8_czech_ci, ucs2/utf8_danish_ci--are not being
sorted correctly by the IBMDB2I storage engine. This
was being caused because the sort order used by DB2 is
incompatible with the order expected by MySQL.

This patch removes support for the cp1250_czech_cs and
latin2_czech_cs collations because it has been determined
that the sort order used by DB2 is incompatible with the
order expected by MySQL. Users needing a czech collation
with IBMDB2I are encouraged to use a Unicode-based collation 
instead of these single-byte collations. This patch also
modifies the DB2 sort sequence used for ucs2/utf8_czech_ci
and ucs2/utf8_danish_ci collations to better match the
sorting expected by MySQL. This will only affect indexes
or tables that are newly created through the IBMDB2I storage
engine. Existing IBMDB2I tables will retain the old sort
sequence until recreated.

mysql-test/suite/ibmdb2i/r/ibmdb2i_bug_45196.result:
  Bug#45196  Some collations do not sort correctly with IBMDB2I
  
  Result file for the test case.
mysql-test/suite/ibmdb2i/t/ibmdb2i_bug_45196.test:
  Bug#45196  Some collations do not sort correctly with IBMDB2I
  
  Adding tests for testing the sort order with the modified collations.
storage/ibmdb2i/db2i_collationSupport.cc:
  Bug#45196  Some collations do not sort correctly with IBMDB2I
  
  Remove the support for the cp1250_czech_cs and latin2_czech_cs 
  collations because it has been determined that the sort order
  used by DB2 is incompatible with the order expected by MySQL.
  Users needing a czech collation with IBMDB2I are encouraged to
  use a Unicode-based collation instead of these single-byte
  collations. This patch also modifies the DB2 sort sequence
  used for ucs2/utf8_czech_ci and ucs2/utf8_danish_ci collations
  to better match the sorting expected by MySQL. This will only 
  affect indexes or tables that are newly created through the
  IBMDB2I storage engine. Existing IBMDB2I tables will retain
  the old sort sequence until recreated.

           format." warnings
      
Despite the fact that a statement would be filtered out from binlog, a
warning would still be thrown if it was issued with the LIMIT.
      
This patch addresses this issue by checking the filtering rules before
printing out the warning.


mysql-test/suite/binlog/t/binlog_stm_unsafe_warning-master.opt:
  Parameter to filter out database: "b42851".
mysql-test/suite/binlog/t/binlog_stm_unsafe_warning.test:
  Added a new test case.
sql/sql_class.cc:
  Added filtering rules check to condition used to decide whether to
  printout warning or not.

The TABLE::reginfo.impossible_range is used by the optimizer to indicate
that the condition applied to the table is impossible. It wasn't initialized
at table opening and this might lead to an empty result on complex queries:
a query might set the impossible_range flag on a table and when the query finishes,
all tables are returned back to the table cache. The next query that uses the table
with the impossible_range flag set and an index over the table will see the flag
and thus return an empty result.

The open_table function now initializes the TABLE::reginfo.impossible_range
variable.

mysql-test/r/select.result:
  A test case for the bug#45266: Uninitialized variable lead to an empty result.
mysql-test/t/select.test:
  A test case for the bug#45266: Uninitialized variable lead to an empty result.
sql/sql_base.cc:
  Bug#45266: Uninitialized variable lead to an empty result.
  The open_table function now initializes the TABLE::reginfo.impossible_range
  variable.
sql/sql_select.cc:
  Bug#45266: Uninitialized variable lead to an empty result.
  The open_table function now initializes the TABLE::reginfo.impossible_range
  variable.
sql/structs.h:
  Bug#45266: Uninitialized variable lead to an empty result.
  A comment is added.