replicating

Fixes to make mysqlbinlog work correctly and updates
to rpl_ndb result files.

sql/log_event.cc:
  The processing of the MYSQL_STRING_TYPE is now done
  in table_def::type(), so the support functions for
  mysqlbinlog are changed to work correctly.

Fixes to get it to compile on MacOSX.

Fixing error codes that changed after merge.

           for InnoDB

The class Field_bit_as_char stores the metadata for the
field incorrecly because bytes_in_rec and bit_len are set
to (field_length + 7 ) / 8 and 0 respectively, while
Field_bit has the correct values field_length / 8 and
field_length % 8.

Solved the problem by re-computing the values for the
metadata based on the field_length instead of using the
bytes_in_rec and bit_len variables.

To handle compatibility with old server, a table map
flag was added to indicate that the bit computation is
exact. If the flag is clear, the slave computes the
number of bytes required to store the bit field and
compares that instead, effectively allowing replication
*without conversion* from any field length that require
the same number of bytes to store.

mysql-test/suite/rpl/t/rpl_typeconv_innodb.test:
  Adding test to check compatibility for bit field
  replication when using InnoDB.
sql/field.cc:
  Extending compatible_field_size() with flags from
  table map to allow fields to check master info.
sql/field.h:
  Extending compatible_field_size() with flags from
  table map to allow fields to check master info.
sql/log.cc:
  Removing table map flags since they are not used
  outside table map class.
sql/log_event.cc:
  Removing flags parameter from table map constructor
  since it is not used and does not have to be exposed.
sql/log_event.h:
  Adding flag to denote that bit length for bit field type
  is exact and not potentially rounded to even bytes.
sql/rpl_utility.cc:
  Adding fields to table_def to store table map flags.
sql/rpl_utility.h:
  Removing obsolete comment and adding flags to store
  table map flags from master.

Fixing minor error when printing SQL types from master and cleaning some code.

Updating result files.

Row-based replication requires the types of columns on the
master and slave to be approximately the same (some safe
conversions between strings are allowed), but does not
allow safe conversions between fields of similar types such
as TINYINT and INT.

This patch implement type conversions between similar fields
on the master and slave.

The conversions are controlled using a new variable
SLAVE_TYPE_CONVERSIONS of type SET('ALL_LOSSY','ALL_NON_LOSSY').

Non-lossy conversions are any conversions that do not run the
risk of losing any information, while lossy conversions can
potentially truncate the value. The column definitions are
checked to decide if the conversion is acceptable.

If neither conversion is enabled, it is required that the
definitions of the columns are identical on master and slave.

Conversion is done by creating an internal conversion table,
unpacking the master data into it, and then copy the data to
the real table on the slave.

.bzrignore:
  New files added
client/Makefile.am:
  New files added
client/mysqlbinlog.cc:
  Functions in rpl_utility.cc is now needed by mysqlbinlog.cc.
libmysqld/Makefile.am:
  New files added
mysql-test/extra/rpl_tests/check_type.inc:
  Test include file to check a single type conversion.
mysql-test/extra/rpl_tests/rpl_extraSlave_Col.test:
  Switching to use INT instead of TEXT for column that should not have matching types.
mysql-test/extra/rpl_tests/rpl_row_basic.test:
  Adding code to enable type conversions for BIT tests since InnoDB
  cannot handle them properly due to incorrect information stored as
  metadata.
mysql-test/extra/rpl_tests/type_conversions.test:
  Test file to check a set of type conversions
  with current settings of slave_type_conversions.
mysql-test/suite/rpl/t/rpl_typeconv.test:
  Test file to test conversions from master to slave with
  all possible values for slave_type_conversions.
  
  The test also checks that the slave_type_conversions
  variable works as expected.
sql/field.cc:
  Changing definition of compatible_field_size to both check if 
  two field with identical base types are compatible and give an
  order between them if they are compatible.
  
  This only implement checking on the slave, so it will not affect
  replication from an old master to a new slave.
sql/field.h:
  Changing prototypes for functions:
  - compatible_field_size()
  - init_for_tmp_table()
  - row_pack_length()
sql/log_event.cc:
  Changing compability checks to build a conversion table if the fields
  are compatible, but does not have the same base type.
sql/log_event_old.cc:
  Changing compability checks to build a conversion table if the fields
  are compatible, but does not have the same base type.
sql/mysql_priv.h:
  Adding global option variable for SLAVE_TYPE_CONVERSIONS
sql/mysqld.cc:
  Adding SLAVE_TYPE_CONVERSIONS global server variable.
sql/rpl_record.cc:
  Changing unpack_row to use the conversion table if present.
sql/rpl_rli.h:
  Removing function get_tabledef and replacing it with get_table_data().
  This function retrieve data for table opened for replication, not just
  table definition.
sql/rpl_utility.cc:
  Function table_def::compatible_with is changed to compare table on master
  and slave for compatibility and generate a conversions table if they are
  compatible.
  
  Computing real type of fields from metadata for ENUM and SET types.
  Computing pack_length correctly for ENUM, SET, and BLOB types.
  
  Adding optimization to not check compatibility if no
  slave type conversions are enabled.
sql/rpl_utility.h:
  Changing prototypes since implementation has changed.
  
  Modifying table_def::type() to return real type instead of stored type.
sql/set_var.cc:
  Adding SLAVE_TYPE_CONVERSIONS variable.
sql/set_var.h:
  Adding SLAVE_TYPE_CONVERSIONS variable.
sql/share/errmsg.txt:
  Adding error messages for slave type conversions.
sql/sql_class.h:
  Adding SLAVE_TYPE_CONVERSIONS variable.
sql/sql_select.cc:
  Correcting create_virtual_tmp_table() to compute null bit positions
  correctly in the presence of bit fields.

Support for flushing individual logs, so that the user can
selectively flush a subset of the server logs.

Flush of individual logs is done according to the 
following syntax:

  FLUSH <log_category> LOGS;

The syntax is extended so that the user is able to flush a
subset of logs:

  FLUSH [log_category LOGS,];

where log_category is one of:
  SLOW
  ERROR
  BINARY
  ENGINE
  GENERAL
  RELAY.


mysql-test/suite/rpl/r/rpl_flush_logs.result:
  Test result for WL#5142.
mysql-test/suite/rpl/t/rpl_flush_logs.test:
  Added the test file to verify if the 'flush individual log' 
  statement works fine.
sql/log.cc:
  Added the two functions to flush slow and general log.
sql/sql_parse.cc:
  Added code to flush specified logs against the option.
sql/sql_yacc.yy:
  Added code to parse the 'flush * log' statement syntax and 
  set its option to Lex->type.

merging from 5.1 to rep+2 starting at gca(5.1, next-mr) == build@mysql.com-20091104182209-iui387z35159aoyw

NOTE: backporting BUG#42150 into next-mr
      Includes latest Andrei's patch (see [2 Feb 18:40] Bugs System)
      and merge.test post-push fix (see [3 Feb 18:04] Bugs System)

The reason of the bug appeared to be overreacting on absense of a
binlog file although the file name had been presented in in the master
binlog index file.
By convention, there should have been only a warning printed and the rest of
`reset master' logics completed.  This did not happen on windows
due to incorrect value of my_errno returned from nt_share_delete().
      
Fixed with correcting my_errno assignment in nt_share_delete() to be ENOENT in 
he event of no binlog file. Some minor refactoring has been made.

beyond unsigned long.
BUG#44779: binlog.binlog_max_extension may be causing failure on 
next test in PB
            
NOTE1: this is the backport to next-mr.
NOTE2: already includes patch for BUG#44779.
            
Binlog file extensions would turn into negative numbers once the
variable used to hold the value reached maximum for signed
long. Consequently, incrementing value to the next (negative) number
would lead to .000000 extension, causing the server to fail.
                        
This patch addresses this issue by not allowing negative extensions
and by returning an error on find_uniq_filename, when the limit is
reached. Additionally, warnings are printed to the error log when the
limit is approaching. FLUSH LOGS will also report warnings to the
user, if the extension number has reached the limit. The limit has been
set to 0x7FFFFFFF as the maximum.

mysql-test/suite/binlog/t/binlog_max_extension.test:
  Test case added that checks the maximum available number for
  binlog extensions.
sql/log.cc:
  Changes to find_uniq_filename and test_if_number.
sql/log.h:
  Added macros with values for MAX_LOG_UNIQUE_FN_EXT and
  LOG_WARN_UNIQUE_FN_EXT_LEFT, as suggested in review.

The mentioned on the bug report set of bugs fixes have not be pushed to the main trees.

Fixed with extracting commits done to 6.0-rpl tree and applying them to the main 5.1.
Notes.
1. part of changes - the mtr's specific - were packported to the main 5.0 tree for mtr v1
   as http://lists.mysql.com/commits/46562
   However, there is no that fix anymore in the mtr v2. (This fact was mailed to mtr maintaining
   people).

2. Bug@36929  crash in kill_zombie_dump_threads-> THD::awake() with replication tests
   is not backported because the base code of the patch is libevent and that was removed
   from the main trees due to its instability.

client/mysqlbinlog.cc:
  fixes for BUG#35546
mysql-test/suite/rpl/r/rpl_bug41902.result:
  the new tests result file is added.
mysql-test/suite/rpl/t/rpl_bug41902-slave.opt:
  conf file for bug41902 testing is added.
mysql-test/suite/rpl/t/rpl_bug41902.test:
  regression tests for Bug #41902 is added.
sql/log.cc:
  collection of changes due to Bug #48463.
sql/log.h:
  collection of changes due to Bug #48463.
sql/rpl_rli.h:
  collection of changes due to Bug #48463.
sql/slave.cc:
  collection of changes due to Bug #48463.
sql/sql_repl.cc:
  collection of changes due to Bug #48463.

Post-push fix: Removed MTRv1 arguments according to the
original patch. Although there is a version check, the patch
was pushed to a 5.1 GA staging tree, while the version check 
considers version 5.2. This makes the deprecated parameters 
to be used, despite the fact that they are not valid anymore.

Part of MTRv1 is currently used in RQG semisync test, and this
was causing the test to fail on slave startup.

It should be safe to uncomment when merging up to celosia.

Post-push fix: Reverting change in Makefile.am which was
causing windows not to build client/.

CONFLICTS
=========

Text conflict in sql/sql_yacc.yy
1 conflicts encountered. 

      
One statement that have more than one different tables to update with 
autoinc columns just was marked as unsafe in mixed mode, so the unsafe 
warning can't be produced in statement mode.
      
To fix the problem, mark the statement as unsafe in statement mode too.


mysql-test/extra/rpl_tests/rpl_insert_id.test:
  The test case is updated due to the patch of bug#45827.
mysql-test/suite/binlog/r/binlog_stm_mix_innodb_myisam.result:
  The test result is updated due to the patch of bug#45827.
mysql-test/suite/binlog/r/binlog_unsafe.result:
  Test result for bug#45827.
mysql-test/suite/binlog/t/binlog_unsafe.test:
  Added test to verify if stmt that have more than one
  different tables to update with autoinc columns will 
  produce unsafe warning
mysql-test/suite/rpl/r/rpl_stm_auto_increment_bug33029.result:
  The test result is updated due to the patch of bug#45827.
mysql-test/suite/rpl/t/rpl_trigger.test:
  The test case is updated due to the patch of bug#45827.
mysql-test/suite/rpl/t/rpl_variables_stm.test:
  The test case is updated due to the patch of bug#45827.
sql/sql_base.cc:
  Reomved the 'set_current_stmt_binlog_row_based_if_mixed' function 
  for producing unsafe warnings by executing 'decide_logging_format' 
  function later in statement mode

Removed test case that was left without significance after
backporting the deprecated constructs from 6.0 codebase.

NOTE: Backport of:

bzr log -r revid:sp1r-serg@sergbook.mysql.com-20070505200319-38337
------------------------------------------------------------
revno: 2469.263.4
committer: serg@sergbook.mysql.com
timestamp: Sat 2007-05-05 13:03:19 -0700
message:
  Removing deprecated features:
  --master-XXX command-line options
  log_bin_trust_routine_creators
  table_type
  BACKUP TABLE ...
  RESTORE TABLE ...
  SHOW PLUGIN
  LOAD TABLE ... FROM MASTER
  LOAD DATA FROM MASTER
  SHOW INNODB STATUS
  SHOW MUTEX STATUS
  SHOW TABLE TYPES
  ... TIMESTAMP(N)
  ... TYPE=engine
  
  RESET SLAVE don't reset connection parameters anymore
  LOAD DATA: check opt_secure_file_priv before access(filename)
  improved WARN_DEPRECATED macro

Bug#41756 "Strange error messages about locks from InnoDB".
      
In JT_EQ_REF (join_read_key()) access method, 
don't try to unlock rows in the handler, unless certain that 
a) they were locked
b) they are not used.

Unlocking of rows is done by the logic of the nested join loop,
and is unaware of the possible caching that the access method may
have. This could lead to double unlocking, when a row
was unlocked first after reading into the cache, and then 
when taken from cache, as well as to unlocking of rows which
were actually used (but taken from cache).
      
Delegate part of the unlocking logic to the access method,
and in JT_EQ_REF count how many times a record was actually 
used in the join. Unlock it only if it's usage count is 0.

Implemented review comments.

mysql-test/r/innodb_lock_wait_timeout_1.result:
  Update results (Bug41756).
mysql-test/t/innodb_lock_wait_timeout_1.test:
  Add a test case (Bug#41756).
sql/item_subselect.cc:
  Complete struct READ_RECORD initialization with a new
  member to unlock records.
sql/records.cc:
  Extend READ_RECORD API with a method to unlock read records.
sql/sql_select.cc:
  In JT_EQ_REF (join_read_key()) access method, 
  don't try to unlock rows in the handler, unless certain that 
  a) they were locked
  b) they are not used.
sql/sql_select.h:
  Add members to TABLE_REF to count TABLE_REF buffer usage count.
sql/structs.h:
  Update declarations.

Bug#41756 "Strange error messages about locks from InnoDB".

In JT_EQ_REF (join_read_key()) access method,
don't try to unlock rows in the handler, unless certain that
a) they were locked
b) they are not used.

Unlocking of rows is done by the logic of the nested join loop,
and is unaware of the possible caching that the access method may
have. This could lead to double unlocking, when a row
was unlocked first after reading into the cache, and then
when taken from cache, as well as to unlocking of rows which
were actually used (but taken from cache).

Delegate part of the unlocking logic to the access method,
and in JT_EQ_REF count how many times a record was actually
used in the join. Unlock it only if it's usage count is 0.

Implemented review comments.


mysql-test/r/bug41756.result:
  Add result file (Bug#41756)
mysql-test/t/bug41756-master.opt:
  Use --innodb-locks-unsafe-for-binlog, as in 5.0 just
  using read_committed isolation is not sufficient to 
  reproduce the bug.
mysql-test/t/bug41756.test:
  Add a test file (Bug#41756)
sql/item_subselect.cc:
  Complete struct READ_RECORD initialization with a new
  member to unlock records.
sql/records.cc:
  Extend READ_RECORD API with a method to unlock read records.
sql/sql_select.cc:
  In JT_EQ_REF (join_read_key()) access method,
  don't try to unlock rows in the handler, unless certain that
  a) they were locked
  b) they are not used.
sql/sql_select.h:
  Add members to TABLE_REF to count TABLE_REF buffer usage count.
sql/structs.h:
  Update declarations.

- re-enabled main.innodb_bug44369;
- re-enabled main.innodb_bug47777;
- re-enabled innodb.innodb_information_schema.

            values return too many records

WHERE clauses with "outer_value_list NOT IN subselect" were
handled incorrectly if the outer value list contained multiple 
items where at least one of these could be NULL. The first 
outer record with NULL value was handled correctly, but if a 
second record with NULL value existed, the optimizer would 
choose to reuse the result it got on the last execution of the 
subselect. This is incorrect if the outer value list has 
multiple items.
     
The fix is to make Item_in_optimizer::val_int (in 
item_cmpfunc.cc) reuse the result of the latest execution
for NULL values only if all values in the outer_value_list 
are NULL.


mysql-test/r/subselect3.result:
  Added test for BUG#48177
mysql-test/t/subselect3.test:
  Added test for BUG#48177
sql/item_cmpfunc.cc:
  Make Item_in_optimizer::val_int (in item_cmpfunc.cc) reuse the result of the latest execution for NULL values only if all values in the outer_value_list are NULL.