BUG#29549 (Endians: test failures on Solaris):

Refactoring code to add parameter to pack() and unpack() functions with
purpose of indicating if data should be packed in little-endian or
native order. Using new functions to always pack data for binary log
in little-endian order. The purpose of this refactoring is to allow
proper implementation of endian-agnostic pack() and unpack() functions.

Eliminating several versions of virtual pack() and unpack() functions
in favor for one single virtual function which is overridden in
subclasses.

Implementing pack() and unpack() functions for some field types that
packed data in native format regardless of the value of the
st_table_share::db_low_byte_first flag.

The field types that were packed in native format regardless are:
Field_real, Field_decimal, Field_tiny, Field_short, Field_medium,
Field_long, Field_longlong, and Field_blob.

Before the patch, row-based logging wrote the rows incorrectly on
big-endian machines where the storage engine defined its own
low_byte_first() to be FALSE on big-endian machines (the default
is TRUE), while little-endian machines wrote the fields in correct
order. The only known storage engine that does this is NDB. In effect,
this means that row-based replication from or to a big-endian
machine where the table was using NDB as storage engine failed if the
other engine was either non-NDB or on a little-endian machine.

With this patch, row-based logging is now always done in little-endian
order, while ORDER BY uses the native order if the storage engine
defines low_byte_first() to return FALSE for big-endian machines.

In addition, the max_data_length() function available in Field_blob
was generalized to the entire Field hierarchy to give the maximum
number of bytes that Field::pack() will write.


mysql-test/extra/rpl_tests/rpl_extraMaster_Col.test:
  Sorting by columns that produces deterministic order.
mysql-test/suite/rpl/r/rpl_extraColmaster_innodb.result:
  Result change.
mysql-test/suite/rpl/r/rpl_extraColmaster_myisam.result:
  Result change.
mysql-test/suite/rpl/r/rpl_row_extraColmaster_ndb.result:
  Result change.
mysql-test/suite/rpl/t/disabled.def:
  Enabling tests.
mysql-test/suite/rpl/t/rpl_row_mysqlbinlog.test:
  Adding missing sync_slave_with_master causing slave to keep tables
  after shutdown.
mysql-test/suite/rpl_ndb/t/disabled.def:
  Enabling tests.
mysql-test/suite/rpl_ndb/t/rpl_ndb_innodb2ndb-slave.opt:
  Adding --new option
mysql-test/suite/rpl_ndb/t/rpl_ndb_innodb2ndb.test:
  Adding have_log_bin.
mysql-test/suite/rpl_ndb/t/rpl_ndb_myisam2ndb-slave.opt:
  Adding --new option
mysql-test/suite/rpl_ndb/t/rpl_ndb_myisam2ndb.test:
  Adding have_log_bin
mysql-test/t/partition.test:
  Adding have_archive, since that is used in the test.
sql/field.cc:
  Eliminating all two-argument pack() and unpack() functions and moving
  functionality into the four-argument version. The four argument version
  is introduced so that it is possible to avoid using the storage engine
  default when writing and reading the packed format (the unpacked format
  still uses the storage engine's default). This is used by row-based
  replication to write the fields in a storage engine- and endian-agnostic
  format.
  
  Packing integral and floating-point numbers in little-endian format
  (if requested).
  
  Using pad_char for the field instead of spaces (0x20) when unpacking.
  
  Adding some Doxygen documentation.
  ---
  Adding max_data_length() to denote the maximum number of bytes that
  pack() will write.
  
  Adding casts to remove warnings for debug printouts.
sql/field.h:
  Eliminating all virtual pack() and unpack() functions except the four-
  argument version, which now is the function that shall be overridden.
  The two-argument versions are convenience functions, to prevent changes
  to code that uses these.
  
  Adding code to pack integer numbers and floating-point numbers in
  little-endian format, if requested.
  ---
  Adding max_data_length() to denote the maximum number of bytes that
  pack() will write.
sql/log.cc:
  Removing debug printout causing crash when starting NDB on Solaris.
sql/log_event.cc:
  Adding missing #ifndef causing compile failure. Adding debug printouts.
sql/rpl_record.cc:
  Debriding code. Using new pack() and unpack() functions to always pack
  fields little-endian. Adding debug printouts.
  ---
  Using max_data_length() when packing field into row.
  
  Adding casts to debug printouts.
sql/sql_show.cc:
  Adding code that causes crash on Solaris machines since printf() cannot
  handle NULL values for strings properly.
mysql-test/suite/rpl_ndb/r/rpl_ndb_innodb2ndb.result:
  New BitKeeper file ``mysql-test/suite/rpl_ndb/r/rpl_ndb_innodb2ndb.result''
mysql-test/suite/rpl_ndb/r/rpl_ndb_myisam2ndb.result:
  New BitKeeper file ``mysql-test/suite/rpl_ndb/r/rpl_ndb_myisam2ndb.result''

mysql-test/extra/rpl_tests/rpl_extraMaster_Col.test

@@ -466,7 +466,7 @@ binary data';
    select * from t2 order by f1;
    select * from t3 order by f1;
    select * from t4 order by f1;
-   select * from t31 order by f1;
+   select * from t31 order by f3;
    
 connection master;
 

mysql-test/suite/rpl/t/disabled.def

@@ -16,4 +16,3 @@ rpl_innodb_mixed_dml     : Bug #29363 rpl.rpl_innodb_mixed_* test failures
 rpl_invoked_features     : BUG#29020 2007-06-21 Lars Non-deterministic test case
 rpl_auto_increment_11932   : Bug#29809 2007-07-16 ingo Slave SQL errors in warnings file
 rpl_stm_extraColmaster_ndb : WL#3915 : Statement-based replication not supported in ndb. Enable test when supported.
-rpl_row_extraColmaster_ndb : BUG#29549 : Replication of BLOBs fail for NDB

mysql-test/suite/rpl/t/rpl_row_mysqlbinlog.test

@@ -344,5 +344,6 @@ FLUSH LOGS;
 --exec rm $MYSQLTEST_VARDIR/tmp/local.sql
 
 DROP TABLE IF EXISTS t1, t2, t3, t04, t05, t4, t5; 
+sync_slave_with_master;
 
 # End of 4.1 tests

mysql-test/suite/rpl_ndb/t/disabled.def

@@ -14,8 +14,6 @@
 rpl_ndb_2innodb          : BUG#19227 2006-04-20 pekka pk delete apparently not replicated
 rpl_ndb_2myisam          : BUG#19227 Seems to pass currently
 rpl_ndb_dd_partitions    : BUG#19259 2006-04-21 rpl_ndb_dd_partitions fails on s/AMD
-rpl_ndb_innodb2ndb       : Bug#29549 rpl_ndb_myisam2ndb,rpl_ndb_innodb2ndb failed on Solaris for pack_length issue
-rpl_ndb_myisam2ndb       : Bug#29549 rpl_ndb_myisam2ndb,rpl_ndb_innodb2ndb failed on Solaris for pack_length issue
 rpl_ndb_ddl              : BUG#28798 2007-05-31  lars  Valgrind failure in NDB
 rpl_ndb_mix_innodb       : BUG#28123 rpl_ndb_mix_innodb.test casue slave to core on sol10-sparc-a
 rpl_ndb_ctype_ucs2_def   : BUG#27404 util thd mysql_parse sig11 when mysqld default multibyte charset

mysql-test/suite/rpl_ndb/t/rpl_ndb_innodb2ndb-slave.opt

---default-storage-engine=ndbcluster
+--default-storage-engine=ndbcluster --new

mysql-test/suite/rpl_ndb/t/rpl_ndb_innodb2ndb.test

@@ -7,8 +7,10 @@
 # to be able to use the same code for all these different
 # test and to have control over the tests.
 ##############################################################
--- source include/have_ndb.inc
--- source include/have_innodb.inc
--- source include/ndb_master-slave.inc
+source include/have_ndb.inc;
+source include/have_innodb.inc;
+source include/have_log_bin.inc;
+source include/ndb_master-slave.inc;
+
 SET storage_engine=innodb;
---source extra/rpl_tests/rpl_ndb_2multi_eng.test
+source extra/rpl_tests/rpl_ndb_2multi_eng.test;

mysql-test/suite/rpl_ndb/t/rpl_ndb_myisam2ndb-slave.opt

---default-storage-engine=ndbcluster
+--default-storage-engine=ndbcluster --new

mysql-test/suite/rpl_ndb/t/rpl_ndb_myisam2ndb.test

@@ -7,6 +7,9 @@
 # to be able to use the same code for all these different
 # test and to have control over the tests.
 ##############################################################
--- source include/ndb_master-slave.inc
+source include/have_ndb.inc;
+source include/have_log_bin.inc;
+source include/ndb_master-slave.inc;
+
 SET storage_engine=myisam;
---source extra/rpl_tests/rpl_ndb_2multi_eng.test
+source extra/rpl_tests/rpl_ndb_2multi_eng.test;

mysql-test/t/partition.test

@@ -3,7 +3,8 @@
 # Simple test for the partition storage engine
 # Taken fromm the select test
 #
--- source include/have_partition.inc
+source include/have_partition.inc;
+source include/have_archive.inc;
 
 --disable_warnings
 drop table if exists t1;

sql/log.cc

@@ -3571,9 +3571,6 @@ bool MYSQL_BIN_LOG::write(Log_event *event_info)
 	(!binlog_filter->db_ok(local_db)))
     {
       VOID(pthread_mutex_unlock(&LOCK_log));
-      DBUG_PRINT("info",("OPTION_BIN_LOG is %s, db_ok('%s') == %d",
-                         (thd->options & OPTION_BIN_LOG) ? "set" : "clear",
-                         local_db, binlog_filter->db_ok(local_db)));
       DBUG_RETURN(0);
     }
 #endif /* HAVE_REPLICATION */

sql/log_event.cc

@@ -5697,7 +5697,9 @@ Rows_log_event::Rows_log_event(const char *buf, uint event_len,
                                *description_event)
   : Log_event(buf, description_event),
     m_row_count(0),
+#ifndef MYSQL_CLIENT
     m_table(NULL),
+#endif
     m_rows_buf(0), m_rows_cur(0), m_rows_end(0),
     m_curr_row(NULL), m_curr_row_end(NULL),
     m_key(NULL)
@@ -6168,6 +6170,8 @@ int Rows_log_event::do_apply_event(RELAY_LOG_INFO const *rli)
         unpack_current_row(rli);
   
       // at this moment m_curr_row_end should be set
+      DBUG_PRINT("debug", ("m_curr_row_end: 0x%lx; m_curr_row: 0x%lx; m_rows_end: 0x%lx",
+                           m_curr_row_end, m_curr_row, m_rows_end));
       DBUG_ASSERT(error || m_curr_row_end != NULL); 
       DBUG_ASSERT(error || m_curr_row < m_curr_row_end);
       DBUG_ASSERT(error || m_curr_row_end <= m_rows_end);

sql/rpl_record.cc

@@ -65,6 +65,8 @@ pack_row(TABLE *table, MY_BITMAP const* cols,
   my_ptrdiff_t const rec_offset= record - table->record[0];
   my_ptrdiff_t const def_offset= table->s->default_values - table->record[0];
 
+  DBUG_ENTER("pack_row");
+
   /*
     We write the null bits and the packed records using one pass
     through all the fields. The null bytes are written little-endian,
@@ -96,26 +98,14 @@ pack_row(TABLE *table, MY_BITMAP const* cols,
           For big-endian machines, we have to make sure that the
           length is stored in little-endian format, since this is the
           format used for the binlog.
-
-          We do this by setting the db_low_byte_first, which is used
-          inside some store_length() to decide what order to write the
-          bytes in.
-
-          In reality, db_log_byte_first is only set for legacy table
-          type Isam, but in the event of a bug, we need to guarantee
-          the endianess when writing to the binlog.
-
-          This is currently broken for NDB due to BUG#29549, so we
-          will fix it when NDB has fixed their way of handling BLOBs.
         */
-#if 0
-        bool save= table->s->db_low_byte_first;
-        table->s->db_low_byte_first= TRUE;
-#endif
-        pack_ptr= field->pack(pack_ptr, field->ptr + offset);
-#if 0
-        table->s->db_low_byte_first= save;
-#endif
+        const uchar *old_pack_ptr= pack_ptr;
+        pack_ptr= field->pack(pack_ptr, field->ptr + offset,
+                              field->max_data_length(), TRUE);
+        DBUG_PRINT("debug", ("field: %s; pack_ptr: 0x%lx;"
+                             " pack_ptr':0x%lx; bytes: %d",
+                             field->field_name, (ulong) old_pack_ptr,
+                             (ulong) pack_ptr, pack_ptr - old_pack_ptr));
       }
 
       null_mask <<= 1;
@@ -143,8 +133,8 @@ pack_row(TABLE *table, MY_BITMAP const* cols,
     packed data. If it doesn't, something is very wrong.
   */
   DBUG_ASSERT(null_ptr == row_data + null_byte_count);
-
-  return static_cast<size_t>(pack_ptr - row_data);
+  DBUG_DUMP("row_data", row_data, pack_ptr - row_data);
+  DBUG_RETURN(static_cast<size_t>(pack_ptr - row_data));
 }
 #endif
 
@@ -242,18 +232,14 @@ unpack_row(RELAY_LOG_INFO const *rli,
           Use the master's size information if available else call
           normal unpack operation.
         */
-#if 0
-        bool save= table->s->db_low_byte_first;
-        table->s->db_low_byte_first= TRUE;
-#endif
         uint16 const metadata= tabledef->field_metadata(i);
-        if (tabledef && metadata)
-          pack_ptr= f->unpack(f->ptr, pack_ptr, metadata);
-        else
-          pack_ptr= f->unpack(f->ptr, pack_ptr);
-#if 0
-        table->s->db_low_byte_first= save;
-#endif
+        uchar const *const old_pack_ptr= pack_ptr;
+        pack_ptr= f->unpack(f->ptr, pack_ptr, metadata, TRUE);
+	DBUG_PRINT("debug", ("field: %s; metadata: 0x%x;"
+                             " pack_ptr: 0x%lx; pack_ptr': 0x%lx; bytes: %d",
+                             f->field_name, metadata,
+                             (ulong) old_pack_ptr, (ulong) pack_ptr,
+                             pack_ptr - old_pack_ptr));
       }
 
       null_mask <<= 1;
@@ -265,6 +251,7 @@ unpack_row(RELAY_LOG_INFO const *rli,
     throw away master's extra fields
   */
   uint max_cols= min(tabledef->size(), cols->n_bits);
+  DBUG_PRINT("debug", ("Master has %u fields, slave %u", tabledef->size(), cols->n_bits));
   for (; i < max_cols; i++)
   {
     if (bitmap_is_set(cols, i))
@@ -279,6 +266,7 @@ unpack_row(RELAY_LOG_INFO const *rli,
 
       if (!((null_bits & null_mask) && tabledef->maybe_null(i)))
         pack_ptr+= tabledef->calc_field_size(i, (uchar *) pack_ptr);
+      DBUG_PRINT("debug", ("pack_ptr: 0x%lx", (ulong) pack_ptr));
       null_mask <<= 1;
     }
   }
@@ -289,6 +277,8 @@ unpack_row(RELAY_LOG_INFO const *rli,
    */
   DBUG_ASSERT(null_ptr == row_data + master_null_byte_count);
 
+  DBUG_DUMP("row_data", row_data, pack_ptr - row_data);
+
   *row_end = pack_ptr;
   if (master_reclength)
   {

sql/sql_show.cc

@@ -29,6 +29,8 @@
 #include "event_data_objects.h"
 #include <my_dir.h>
 
+#define STR_OR_NIL(S) ((S) ? (S) : "<nil>")
+
 #ifdef WITH_PARTITION_STORAGE_ENGINE
 #include "ha_partition.h"
 #endif
@@ -3096,8 +3098,8 @@ int get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond)
   schema_table_idx= get_schema_table_idx(schema_table);
   get_lookup_field_values(thd, cond, tables, &lookup_field_vals);
   DBUG_PRINT("INDEX VALUES",("db_name='%s', table_name='%s'",
-                             lookup_field_vals.db_value.str,
-                             lookup_field_vals.table_value.str));
+                             STR_OR_NIL(lookup_field_vals.db_value.str),
+                             STR_OR_NIL(lookup_field_vals.table_value.str)));
 
   if (!lookup_field_vals.wild_db_value && !lookup_field_vals.wild_table_value)
   {