Added my_uuid
Added pre-support for PAGE_CHECKSUM
Added syntax for CREATE ... PAGE_CHECKSUM=# TABLE_CHECKSUM=#
Reserved place for page checksums on index, bitmap and block pages
Added index number to header of index pages
Added linked list for free directory entries (speeds up inserts with BLOCK format)
Calculate checksums in original column order (fixes bug with checksum on rows with BLOCK format)
Cleaned up all index handling to use 'info->s->keypage_header' (variable size) as the header for index pages (before this was '2')
Added 0xffffffff to end of index and block data bases and 0xfffffffe at end of bitmap pages when page checksums are not enabled
Added _ma_get_page_used() and _ma_get_used_and_node() to simplify index page header handling
rec_per_key_part is now in double precision
Reserved place in index file for my_guid and nulls_per_key_part
Give error HA_ERR_NEW_FILE if trying to open a Maria file with new, not yet supported extensions

Lots of renames to increase readability:

randomize() -> my_rnd_init()
st_maria_info -> st_maria_handler
st_maria_info -> MARIA_HA
st_maria_isaminfo -> st_maria_info
rand_struct -> my_rand_struct
rec_per_key_rows -> records_at_analyze


client/mysqladmin.cc:
  rand_struct -> my_rrnd_struct
include/maria.h:
  st_maria_info -> MARIA_HA
  st_maria_isaminfo -> st_maria_info
  Changed analyze statistics to be of double precission
  Changed offset to field to be 32bits instead of 64 (safe as a record without blobs can't be that big)
include/my_base.h:
  Added HA_OPTION_PAGE_CHECKSUM & HA_CREATE_PAGE_CHECKSUM
  Fixed comments
  Added HA_ERR_NEW_FILE
include/my_sys.h:
  Added prototypes and structures for my_uuid() and my_rnd()
include/myisamchk.h:
  Changed some buffers to size_t
  Added possibility to have key statistics with double precission
include/mysql_com.h:
  Move rand functions to mysys
libmysql/Makefile.shared:
  Added my_rnd
mysql-test/r/maria.result:
  Updated results
mysql-test/t/maria.test:
  More tests for checksum
mysys/Makefile.am:
  Added my_rnd.c and my_uuid.c
server-tools/instance-manager/listener.cc:
  Fixed include order (my_global.h should always be first)
server-tools/instance-manager/mysql_connection.cc:
  Fixed include order (my_global.h should always be first)
  Use my_rnd_init()
server-tools/instance-manager/mysql_connection.h:
  rand_struct -> my_rand_struct
sql/handler.h:
  Added flag for page checksums
sql/item_func.cc:
  Use new my_rnd() interface
sql/item_func.h:
  Use new my_rnd() interface
sql/item_strfunc.cc:
  Use new my_rnd() interface
sql/lex.h:
  Added PAGE_CHECKSUM and TABLE_CHECKSUM
sql/mysql_priv.h:
  Use new my_rnd() interface
sql/mysqld.cc:
  Use new my_rnd() interface
sql/password.c:
  Move my_rnd() to mysys
  Use new my_rnd() interface
sql/sql_class.cc:
  Use new my_rnd() interface
sql/sql_class.h:
  Use new my_rnd() interface
sql/sql_crypt.cc:
  Use new my_rnd() interface
sql/sql_crypt.h:
  Use new my_rnd() interface
sql/sql_show.cc:
  Simpler handling of ha_choice_values
  Added PAGE_CHECKSUM
sql/sql_table.cc:
  Enable correct checksum handling (for now) if not running in compatible mode
sql/sql_yacc.yy:
  Added table option PAGE_CHECKSUM
  Added future compatible table option TABLE_CHECKSUM (alias for CHECKSUM)
  Added 'choice' target to simplify code
sql/table.cc:
  Store flag for PAGE_CHECKSUM
sql/table.h:
  Added support for PAGE_CHECKSUM
storage/maria/ha_maria.cc:
  Remove protection for incompatbile frm and MAI
  (Slow, not needed test)
  Rec_per_key is now in double
  Remember row type for table
  Give warning if one Maria uses another row type than requested
  Removed some old ASK_MONTY entries (added comments instead)
  Added handling of PAGE_CHECKSUM flags
storage/maria/ma_bitmap.c:
  Added page checksums to bitmap pages
  Added special bitmap marker for bitmap pages
  (Used to find bugs when running without page checksums)
storage/maria/ma_blockrec.c:
  Added a free-link list over directory entries. This makes insert of small rows faster as we don't
  have to scan the whole directory to find a not used entry.
  Moved SANITY_CHECKS to maria_def.h
  Simplify code by introducing dir_entry_pos()
  Added support for PAGE_CHECKSUM
storage/maria/ma_blockrec.h:
  Added DIR_FREE_SIZE (linked list of free directory entries)
  Added PAGE_CHECKSUM
  Added 'dir_entry_pos()'
storage/maria/ma_check.c:
  Check that index pages has correct index number
  Calculate rec_per_key with double precission
  Simplify code by using '_ma_get_used_and_node()'
  Check free directory list
  Remove wrong end \n from messages
  maria_data_on_page() -> _ma_get_page_used()
  maria_putint() -> _ma_store_page_used()
  rec_per_key_rows -> records_at_analyze
storage/maria/ma_checksum.c:
  Calculate checksum in original column order
storage/maria/ma_create.c:
  Store original column order in index file
  Reserve place for nulls_per_key_part (future)
  Added support for PAGE_CHECKSUM
storage/maria/ma_dbug.c:
  Fixed wrong debug output of key of type 'ulong'
storage/maria/ma_delete.c:
  maria_data_on_page() -> _ma_get_used_and_node()
  maria_data_on_page() -> _ma_get_page_used()
  maria_putint() -> _ma_store_page_used()
  Added page header (index key number) to all index pages
  Reserved page for checksum on index pages
  Use keypage_header
storage/maria/ma_ft_update.c:
  maria_putint() -> _ma_store_page_used()
  Store key number at start of page
storage/maria/ma_loghandler.h:
  st_maria_info -> MARIA_HA
storage/maria/ma_open.c:
  rec_per_key is now in double precission
  Added 'nulls_per_key_part'
  Added 'extra_options' (flags for future)
  Added support for PAGE_CHECKSUM
  Give error HA_ERR_NEW_FILE when using unsupported maria extensions
  Added comments
  Add maria_uuid to index file
  Added functions to store and read column_nr map.
  Changed some functions to return my_bool instead of uint
storage/maria/ma_page.c:
  Added checks that pages has correct key nr
  Store 0xffffffff in checksum position if page checksums are not enabled
  Moved key-page-delete link to take into account keypage header
storage/maria/ma_preload.c:
  Remove old MyISAM dependent code
  When scanning pages, only add pages to page cache for the requested index
storage/maria/ma_range.c:
  maria_data_on_page() -> _ma_get_used_and_node()
  Use keypage_header
storage/maria/ma_rt_index.c:
  Fixed indentation
storage/maria/ma_rt_index.h:
  Added support for dynamic index page header
  Reserved place for PAGE_CHECKSUM
storage/maria/ma_rt_key.c:
  Fixed indentation
  maria_data_on_page() -> _ma_get_page_used()
  maria_putint() -> maria_store_page_used()
storage/maria/ma_rt_mbr.c:
  Fixed indentation
storage/maria/ma_rt_split.c:
  Fixed indentation
  maria_data_on_page () -> _ma_get_page_used()
storage/maria/ma_rt_test.c:
  Fixed indentation
storage/maria/ma_search.c:
  Remove support of using -1 as 'last used index' to _ma_check_index()
  maria_data_on_page() -> _ma_get_page_used()
  maria_data_on_page() -> _ma_get_used_and_node()
  Use keypage_header
storage/maria/ma_sort.c:
  Changed some buffers to size_t
  Changed rec_per_key_part to double
storage/maria/ma_static.c:
  Removed NEAR
  Added maria_uuid
storage/maria/ma_test2.c:
  Moevd testflag == 2 to correct place
  Remove test of reading with index number -1 (not supported anymore)
storage/maria/ma_test_recovery.expected:
  Updated results
storage/maria/ma_test_recovery:
  Changed tmp table names so that one can run maria_chk on them
storage/maria/ma_write.c:
  Fixed indentation
  Use keypage_header
  Store index number on index pages
  maria_putint() -> _ma_store_page_used()
  maria_data_on_page() -> ma_get_used_and_node()
  maria_data_on_page() -> _ma_get_page_used()
  Added PAGE_CHECKSUM
  Added Maria handler to some functions
  Removed some not needed casts
storage/maria/maria_chk.c:
  Added error handling for HA_ERR_NEW_FILE
  Added information about page checksums
  rec_per_key_part changed to double
  maria_data_on_page() -> _ma_get_page_used()
  Use keypage_header
storage/maria/maria_def.h:
  Added IDENTICAL_PAGES_AFTER_RECOVERY and SANITY_CHECKS
  Changed rec_per_key_part to double
  Added nulls_per_key_part
  rec_per_key_rows -> records_at_analyze
  st_maria_info -> MARIA_HA
  Reserve place for new statistics variables, uuid, checksums per page etc.
  Removed NEAR tags
  Changed some prototypes to use my_bool and size_t
storage/maria/maria_pack.c:
  st_maria_info -> MARIA_HA
  Fixed indentation
storage/myisam/mi_dbug.c:
  Fix wrong debug output for ULONG
mysys/my_rnd.c:
  New BitKeeper file ``mysys/my_rnd.c''
mysys/my_uuid.c:
  New BitKeeper file ``mysys/my_uuid.c''

into  pwbook.sanja.is.com.ua:/home/bell/mysql/bk/work-maria-logref

Fixed compiler warning
PAGECACHE_PAGE_LINK -> PAGECACHE_BLOCK_LINK*


storage/maria/ma_loghandler.h:
  PAGECACHE_PAGE_LINK -> PAGECACHE_BLOCK_LINK*
storage/maria/ma_pagecache.c:
  PAGECACHE_PAGE_LINK -> PAGECACHE_BLOCK_LINK*
storage/maria/ma_pagecache.h:
  PAGECACHE_PAGE_LINK -> PAGECACHE_BLOCK_LINK*
storage/maria/maria_def.h:
  PAGECACHE_PAGE_LINK -> PAGECACHE_BLOCK_LINK*

BUILD/SETUP.sh:
  compile maria by default

REDO optimization (Bascily avoid moving blocks from/to pagecache)
More command line arguments to maria_read_log
Fixed recovery bug when recreating table


sql/opt_range.cc:
  Remove SAFE_MODE for opt_range as it disables UPDATE to use keys
storage/maria/ma_blockrec.c:
  REDO optimization
  Use new interface for pagecache_reads to avoid copying page buffers
storage/maria/ma_loghandler.c:
  Patch from Sanja:
  - Added new parameter to translog_get_page to use direct links to pagecache
  - Changed scanner to be able to use direct links
  
  This avoids a lot of calls to bmove512() in page cache.
storage/maria/ma_loghandler.h:
  Added direct link to pagecache objects
storage/maria/ma_open.c:
  Added const to parameter
  Added missing braces
storage/maria/ma_pagecache.c:
  From Sanja:
  - Added direct links to pagecache (from pagecache_read())
    Dirrect link means that on pagecache_read we get back a pointer to the pagecache buffer
  
  
  From Monty:
  - Fixed arguments to init_page_cache to handle big page caches
  - Fixed compiler warnings
  - Replaced PAGECACHE_PAGE_LINK with PAGECACHE_BLOCK_LINK * to catch errors
storage/maria/ma_pagecache.h:
  Changed block numbers from int to long to be able to handle big page caches
  Changed some PAGECACHE_PAGE_LINK to PAGECACHE_BLOCK_LINK
storage/maria/ma_recovery.c:
  Fixed recovery bug when recreating table (table was kept open)
  Moved some variables to function start (portability)
  Added space to some print messages
storage/maria/maria_chk.c:
  key_buffer_size -> page_buffer_size
storage/maria/maria_def.h:
  Changed default page_buffer_size to 10M
storage/maria/maria_read_log.c:
  Added more startup options:
  --version
  --undo (apply undo)
  --page_cache_size (to run with big cache sizes)
  --silent (to not get any output from --apply)
storage/maria/unittest/ma_control_file-t.c:
  Fixed compiler warning
storage/maria/unittest/ma_test_loghandler-t.c:
  Added new argument to translog_init_scanner()
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
  Added new argument to translog_init_scanner()
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
  Added new argument to translog_init_scanner()

Progress reports on stderr if doing recovery from ha_maria;
don't do checkpoints if activity since last checkpoint < 2MB
(no change in fact as background thread is disabled for now);
recovery trace is only if EXTRA_DEBUG now (better for benchmarks).


storage/maria/ma_checkpoint.c:
  don't do checkpoints if activity (log writes plus page flushes)
  since last checkpoint was < 2MB.
storage/maria/ma_recovery.c:
  progress reports in recovery (10%, transactions left to rollback etc);
  that is only if from ha_maria and is displayed on stderr.
  Recovery trace is now created only if EXTRA_DEBUG.
storage/maria/ma_test_recovery.expected:
  update (--debug gone)
storage/maria/ma_test_recovery:
  don't use --debug, as it can absent from binary

logging of insertions made by CREATE SELECT.


sql/sql_insert.cc:
  If error during the CREATE SELECT we drop the table, so no need for
  engines to do logging of the insertions (optimization). Engines
  require that disabling is done before locking and re-enabling is done
  before unlocking; as table creation and locking is done as one
  function (create_table_from_items()) we disable before calling
  this function and re-enable before unlocking, in send_eof() (called
  if success) and abort() (called if error).
  Question for reviewer: would it be better to do the disabling between
  creation and locking, so inside create_table_from_items(), given
  that this function is used only by CREATE SELECT?

into  pwbook.sanja.is.com.ua:/home/bell/mysql/bk/mysql-maria


storage/maria/ma_loghandler.c:
  Auto merged

record (for example in ma_test_loghandler-t). Reason was
wrongly matched () in DBUG.


storage/maria/ma_loghandler.c:
  Wrongly matched parenthesis:
  DBUG_PRINT(keyword, argslist) expands to roughly
  _db_doprnt arglist;
  So DBUG_PRINT("enter",(a);
  b;
  c);
  expands to roughly
  _db_doprnt(a);b;c;
  which is valid code. Except that in non-debug builds, DBUG_PRINT(
  expands to nothing so the wrongly "included" code is thrown away,
  leading to some members of "scanner" to not be initialized.

Note that non-debug build fails in log handler functions, mail sent.


storage/maria/ma_blockrec.c:
  fix for compiler warning
storage/maria/ma_checkpoint.c:
  Debug build does not catch this situation
  static int f();
  ...
  f(2);
  ...
  static int f(int a, int b);
  Maybe this is because it believes the declaration is K&R. Non-debug
  build catches it. Adding (void) as an habit to avoid such errors.
storage/maria/ma_checkpoint.h:
  adding (void)
storage/maria/ma_recovery.c:
  adding (void)
storage/maria/ma_recovery.h:
  adding (void)

BLOB whose size didn't change. Fix for probably impossible problem
in Recovery.


mysql-test/r/maria.result:
  result for new test
mysql-test/t/maria.test:
  testcase for a bug (used to segfault)
storage/maria/ma_blockrec.c:
  When writing a record, we put BLOBs into the head part if there is
  room for them. "Is there room" was first decided by
  !(tmp_data + length > end_of_data) (line 1894)
  but then was tested again as
  *blob_lengths < (ulong)(end_of_data - data). We see that in case of
  equality, the first condition was true but the second was not,
  so it was inconsistent and crashed later.
storage/maria/ma_recovery.c:
  When wondering if recovery should update the state (like state.records):
  if table was closed, its is_of_horizon was set to X, then
  table was reopened and a REDO was written. If this REDO had LSN
  X (as horizon is just a lower bound of the LSN of the next record),
  we have to apply it. In practice this equality probably could not
  happen because of LOGREC_FILE_ID would be written before the REDO.

* recovery from ha_maria now skips replaying DDLs (too dangerous)
* maria_read_log still replays DDLs, print warning about issues
* fixes to replaying of REDO_RENAME
* don't replay DDLs on corrupted tables (safer)
* print a one-line message when really doing a recovery (applies to
ha_maria, not maria_read_log) i.e. some REDOs or UNDOs are read.


storage/maria/ma_checkpoint.c:
  fix for assertion failure
storage/maria/ma_recovery.c:
  * Recovery from ha_maria now skips replaying DDLs (as the initial
  plan said) as this is unsafe in case of crashes during the DDL;
  applying the records may do harm (destroy important files)
  so we prefer to leave the "mess" of files untouched. A proper
  recovery of DDLs requires very careful thinking, probably testing
  separately the existence of the data and index file instead of
  using maria_open() which tests the existence of both, and maybe
  storing create_rename_lsn in the data file too.
  * maria_read_log still replays DDLs, we print a warning about dangers
  (due to ALTER TABLE not logging insertions into the tmp table; we
  will maybe need an option to have logging of those insertions).
  * fixes to replaying of REDO_RENAME (test create_rename_lsn of 'new_name'
  table if it exists; if that table exists and is more recent than the
  record, remove the 'old_name' table).
  * don't replay DDLs on corrupted tables (play safe)
  * fail also in non-debug builds if table is open when it should not be
  (when creating it for example, it should not be already open).
  * when the trace file is not stdout (i.e. when this is ha_maria),
  if really doing a recovery (reading REDOs or UNDOs), print a one-line
  message to stderr to inform about start and end of recovery
  (useful to know what mysqld is doing, especially if it takes long
  or crashes).
storage/maria/ma_recovery.h:
  parameter to replay DDLs or not
storage/maria/maria_read_log.c:
  replay DDLs in maria_read_log, to be able to recreate tables from
  scratch.

into  gbichot4.local:/home/mysql_src/mysql-maria-tmp

horizon addres.


storage/maria/ma_loghandler.c:
  New function to get correct LSN from chunk address.
storage/maria/ma_loghandler.h:
  New function to get correct LSN from chunk address.

storage/maria/ma_loghandler.c:
  Check of log initialization added.
  Function descriptions fixed.

the latter wrote into the settings of the key cache! Problem was that
--pagecache* options reused OPT_KEY_CACHE* enum values.


sql/mysqld.cc:
  --pagecache* options should not reuse the OPT_KEY_CACHE* enum values, it
  confuses my_getopt. Making handling of page caches similar to handling
  of key caches, in mysqld.cc.

storage/maria/ma_checkpoint.c:
  The new macro for easier printing LSN added.
storage/maria/ma_loghandler.c:
  The assertion returned.
  The new macro for easier printing LSN added.
storage/maria/ma_loghandler_lsn.h:
  The new macro for easier printing LSN added.
storage/maria/ma_pagecache.c:
  The new macro for easier printing LSN added.
storage/maria/ma_recovery.c:
  Recovery checks empty log state.
  RECHEADER_READ_ERROR means some real error.
storage/maria/maria_read_log.c:
  Read log starts from real beggining of the log and precess
  error and empty log states.
  The new macro for easier printing LSN added.
storage/maria/unittest/ma_test_loghandler-t.c:
  The new macro for easier printing LSN added.
storage/maria/unittest/ma_test_loghandler_first_lsn-t.c:
  The new macro for easier printing LSN added.
storage/maria/unittest/ma_test_loghandler_max_lsn-t.c:
  The new macro for easier printing LSN added.
storage/maria/unittest/ma_test_loghandler_multigroup-t.c:
  The new macro for easier printing LSN added.
storage/maria/unittest/ma_test_loghandler_multithread-t.c:
  The new macro for easier printing LSN added.
storage/maria/unittest/ma_test_loghandler_noflush-t.c:
  The new macro for easier printing LSN added.

storage/maria/ma_recovery.c:
  MY_ALLOW_ZERO_PTR needed as log_record_buffer.str is initially NULL.

mysql-test/r/rpl_row_flsh_tbls.result:
  result update
mysql-test/r/rpl_row_insert_delayed.result:
  result update
mysql-test/t/rpl_row_flsh_tbls.test:
  CREATE TABLE statement got an ENGINE clause so became longer

failed => Maria didn't start => tables were created as MyISAM).


storage/maria/ma_checkpoint.c:
  safemalloc complains if my_realloc() is passed NULL and
  MY_ALLOW_ZERO_PTR is not used.

* added replaying of REDO_REPAIR_TABLE, but disabled it as
mysterious linker errors appear.
* after replaying RENAME/REPAIR, we must bump create_rename_lsn
for idempotency of maria_read_log.


sql/mysqld.cc:
  typo
storage/maria/ma_checkpoint.c:
  silence compiler warning
storage/maria/ma_recovery.c:
  * added replaying of REDO_REPAIR_TABLE, but disabled it as
  mysterious linker errors appear.
  * after replaying RENAME/REPAIR, we must bump create_rename_lsn
  for idempotency of maria_read_log.

Finally this is the real checkpoint code.
It however exhibits unstabilities when a checkpoint runs concurrently
with data-modifying clients (table corruption, transaction log's
assertions) so for now a checkpoint is taken only at startup after
recovery and at shutdown, i.e. not in concurrent situations. Later
we will let it run periodically, as well as flush dirty pages
periodically (almost all needed code is there already, only pagecache
code is written but not committed).
WL#3072 Maria recovery
* replacing UNDO_ROW_PURGE with CLR_END; testing of those CLR_END via
ma_test2 which has INSERTs failing with duplicate keys.
* replaying of REDO_RENAME_TABLE
Now, off to test Recovery in ha_maria :)


BitKeeper/deleted/.del-ma_least_recently_dirtied.c:
  Delete: storage/maria/ma_least_recently_dirtied.c
BitKeeper/deleted/.del-ma_least_recently_dirtied.h:
  Delete: storage/maria/ma_least_recently_dirtied.h
storage/maria/Makefile.am:
  compile Checkpoint module
storage/maria/ha_maria.cc:
  When ha_maria starts, do a recovery from last checkpoint.
  Take a checkpoint when that recovery has ended and when ha_maria
  shuts down cleanly.
storage/maria/ma_blockrec.c:
  * even if my_sync() fails we have to my_close() (otherwise we leak
  a descriptor)
  * UNDO_ROW_PURGE is replaced by a simple CLR_END for UNDO_ROW_INSERT,
  as promised in the old comment; it gives us skipping during the
  UNDO phase.
storage/maria/ma_check.c:
  All REDOs before create_rename_lsn are ignored by Recovery. So
  create_rename_lsn must be set only after all data/index has been
  flushed and forced to disk. We thus move write_log_record_for_repair()
  to after _ma_flush_tables_files_after_repair().
storage/maria/ma_checkpoint.c:
  Checkpoint module.
storage/maria/ma_checkpoint.h:
  optional argument if caller wants a thread to periodically take
  checkpoints and flush dirty pages.
storage/maria/ma_create.c:
  * no need to init some vars as the initial bzero(share) takes care of this.
  * update to new function's name
  * even if we fail in my_sync() we have to my_close()
storage/maria/ma_extra.c:
  Checkpoint reads share->last_version under intern_lock, so we make
  maria_extra() update it under intern_lock. THR_LOCK_maria still needed
  because of _ma_test_if_reopen().
storage/maria/ma_init.c:
  destroy checkpoint module when Maria shuts down.
storage/maria/ma_loghandler.c:
  * UNDO_ROW_PURGE gone (see ma_blockrec.c)
  * we need to remember the LSN of the LOGREC_FILE_ID for a share,
  because this LSN is needed into the checkpoint record (Recovery wants
  to know the validity domain of an id->name mapping)
  * translog_get_horizon_no_lock() needed for Checkpoint
  * comment about failing assertion (Sanja knows)
  * translog_init_reader_data() thought that translog_read_record_header_scan()
  returns 0 in case of error, but 0 just means "0-length header".
  * translog_assign_id_to_share() now needs the MARIA_HA because
  LOGREC_FILE_ID uses a log-write hook.
  * Verify that (de)assignment of share->id happens only under intern_lock,
  as Checkpoint reads this id with intern_lock.
  * translog_purge() can accept TRANSLOG_ADDRESS, not necessarily
  a real LSN.
storage/maria/ma_loghandler.h:
  prototype updates
storage/maria/ma_open.c:
  no need to initialize "res"
storage/maria/ma_pagecache.c:
  When taking a checkpoint, we don't need to know the maximum rec_lsn
  of dirty pages; this LSN was intended to be used in the two-checkpoint
  rule, but last_checkpoint_lsn is as good.
  4 bytes for stored_list_size is enough as PAGECACHE::blocks (number
  of blocks which the pagecache can contain) is int.
storage/maria/ma_pagecache.h:
  new prototype
storage/maria/ma_recovery.c:
  * added replaying of REDO_RENAME_TABLE
  * UNDO_ROW_PURGE gone (see ma_blockrec.c), replaced by CLR_END
  * Recovery from the last checkpoint record now possible
  * In new_table() we skip the table if the id->name mapping is older than
  create_rename_lsn (mapping dates from lsn_of_file_id).
  * in get_MARIA_HA_from_REDO_record() we skip the record
  if the id->name mapping is newer than the record (can happen if processing
  a record which is before the checkpoint record).
  * parse_checkpoint_record() has to return a LSN, that's what caller expects
storage/maria/ma_rename.c:
  new function's name; log end zeroes of tables' names (ease recovery)
storage/maria/ma_test2.c:
  * equivalent of ma_test1's --test-undo added (named -u here).
  * -t=1 now stops right after creating the table, so that
  we can test undoing of INSERTs with duplicate keys (which tests the
  CLR_END logged by _ma_write_abort_block_record()).
storage/maria/ma_test_recovery.expected:
  Result of testing undoing of INSERTs with duplicate keys; there are
  some differences in maria_chk -dvv but they are normal (removing
  records does not shrink data/index file, does not put back the
  "analyzed, optimized keys"(etc) index state.
storage/maria/ma_test_recovery:
  Test undoing of INSERTs with duplicate keys, using ma_test2;
  when such INSERT happens, it logs REDO_INSERT, UNDO_INSERT, REDO_DELETE,
  CLR_END; we abort after that, and test that CLR_END causes recovery
  to jump over UNDO_INSERT.
storage/maria/ma_write.c:
  comment
storage/maria/maria_chk.c:
  comment
storage/maria/maria_def.h:
  * a new bit in MARIA_SHARE::in_checkpoint, used to build a list
  of unique shares during Checkpoint.
  * MARIA_SHARE::lsn_of_file_id added: the LSN of the last LOGREC_FILE_ID
  for this share; needed to know to which LSN domain the mappings
  found in the Checkpoint record apply (new mappings should not apply
  to old REDOs).
storage/maria/trnman.c:
  * small changes to how trnman_collect_transactions() fills its buffer;
  it also uses a non-dummy lsn_read_non_atomic() found in ma_checkpoint.h

into  gbichot4.local:/home/mysql_src/mysql-maria-no-flush-state

* testing of execution of UNDO_ROW_UPDATE
* when executing an UNDO_ROW_UPDATE, store "UNDO_ROW_UPDATE" as
"type of undone record" into the CLR_END record.


storage/maria/ma_blockrec.c:
  When logging a CLR_END in write_block_record(), it can be for
  a DELETE or for an UPDATE (now that Monty has coded execution of
  UNDO_UPDATE)
storage/maria/ma_loghandler.c:
  UNDO_ROW_UPDATE's execution coded, so no crash
storage/maria/ma_recovery.c:
  UNDO_ROW_UPDATE's execution now coded, so no crash
storage/maria/ma_test1.c:
  upper case letter
storage/maria/ma_test_recovery.expected:
  output of testing execution of UNDO_ROW_UPDATE. Table's checksum
  not recovered (known issue not specific to UPDATE).
storage/maria/ma_test_recovery:
  Test execution of UNDO_ROW_UPDATE: first we stop ma_test1 after
  deletes and commit, then we stop ma_test1 after updates and abort;
  we verify that updates are rolled back by comparing tables

into  mysql.com:/home/my/mysql-maria


storage/maria/maria_read_log.c:
  Auto merged

Fixed bug in undo_delete
Fixed wrong error output from maria_check


include/my_base.h:
  Added marker if we have null fields in table
mysql-test/r/maria.result:
  checksum in maria now ignore null fields that are null
sql/sql_table.cc:
  Ignore null fields that are now
  (Before enabling this, we have to change MyISAM to also skip null fields)
storage/maria/ma_blockrec.c:
  More logging
  After merge fixes
  Fixed some bugs when using undo of VARCHAR fields
  Fixed bug in undo_delete (We can't use info->rec_buff here as this is used in write_block_record())
storage/maria/ma_blockrec.h:
  ma_recordpos_to_dir_entry changed to return uint
storage/maria/ma_check.c:
  Fixed wrong output in case of errors
storage/maria/ma_create.c:
  Set share.base.pack_reclength more correct for block record
  Delete support for RAID
storage/maria/ma_open.c:
  Don't calculate checksum fields with value NULL
storage/maria/ma_test1.c:
  Fixed output from -v for VARCHAR keys
storage/maria/ma_test_recovery.expected:
  Update results after adding new printf
  New checksums (because we now ignore nulls)
  Some file lengths are different, but think they are ok (didn't have time to investigate)
storage/myisam/ha_myisam.cc:
  Fixed comment
storage/myisam/mi_test1.c:
  Fixed bug

storage/maria/Makefile.am:
  include trnman_public.h in source tarball

into  mysql.com:/home/my/mysql-maria


storage/maria/ma_check.c:
  Auto merged
storage/maria/ma_locking.c:
  Auto merged
storage/maria/ma_loghandler.c:
  Auto merged
storage/maria/ma_open.c:
  Auto merged
storage/maria/ma_recovery.c:
  Auto merged
storage/maria/maria_def.h:
  Auto merged
storage/maria/maria_read_log.c:
  Auto merged
storage/maria/ma_blockrec.c:
  Manual merge
storage/maria/ma_test1.c:
  Manual merge (using Guilhems code)

Fixed bug in duplicate key handling for block records during repair
All read-row methods now return error number in case of error
Don't calculate checksum for null fields
Fixed bug when running maria_read_log with -o


BUILD/SETUP.sh:
  Added STACK_DIRECTION
BUILD/compile-pentium-debug-max:
  Moved STACK_DIRECTION to SETUP
include/myisam.h:
  Added extra parameter to write_key
storage/maria/ma_blockrec.c:
  Added applying of undo for updates
  Fixed indentation
  Removed some not needed casts
  Fixed wrong logging of CLR record
  Split ma_update_block_record to two functions to be able to reuse it from undo-applying
  Simplify filling of packed fields
  ma_record_block_record) now returns error number on failure
  Sligtly changed log record information for undo-update
storage/maria/ma_check.c:
  Fixed bug in duplicate key handling for block records during repair
storage/maria/ma_checksum.c:
  Don't calculate checksum for null fields
storage/maria/ma_dynrec.c:
  _ma_read_dynamic_reocrd() now returns error number on error
  Rest of the changes are code simplification and indentation fixes
storage/maria/ma_locking.c:
  Added comment
storage/maria/ma_loghandler.c:
  More debugging
  Removed printing of total_record_length as this was always same as record_length
storage/maria/ma_open.c:
  Allocate bitmap for changed fields
storage/maria/ma_packrec.c:
  read_record now returns error number on error
storage/maria/ma_recovery.c:
  Fixed wrong arguments to undo_row_update
storage/maria/ma_statrec.c:
  read_record now returns error number on error (not 1)
  Code simplification
storage/maria/ma_test1.c:
  Added exit possibility after update phase (to test undo of updates)
storage/maria/maria_def.h:
  Include bitmap header file
storage/maria/maria_read_log.c:
  Fixed bug when running with -o

end of translog_flush() when datadir was in /dev/shm.


storage/maria/ma_loghandler.c:
  directory syncing can fail on shared memory devices (/dev/shm on Linux
  in this case); see my_sync_dir().

for compiler errors in pushbuild). Small bugfix.


sql/handler.h:
  don't use pagecaches if no Maria
storage/maria/ma_check.c:
  correcting mistake in previous push; need to call this function
  otherwise create_rename_lsn would not be updated at end of REPAIR.

At the end of recovery, we initialize the transaction manager's
trid generator with the maximum trid seen during the REDO phase.
This ensures that trids always grow (needed for versioning),
even after a crash.
This patch is only preparation, as ma_recover() is not called
from ha_maria yet.


storage/maria/ha_maria.cc:
  trnman_init() needs argument now (soon trnman_init() will rather
  be done via ma_recover() and thus it will not be 0)
storage/maria/ma_recovery.c:
  During the REDO phase, remember the max long trid of transactions
  which we have seen (both in the checkpoint record and the
  LOGREC_LONG_TRANSACTION_ID records)
storage/maria/ma_test1.c:
  trnman_init() needs argument now
storage/maria/ma_test2.c:
  trnman_init() needs argument now
storage/maria/trnman.c:
  new argument to trnman_init() so that caller can decide which
  value the generator of trids starts from.
storage/maria/trnman_public.h:
  trnman_init() needs argument now
storage/maria/unittest/trnman-t.c:
  trnman_init() needs argument now

Recovery of state.records (the count of records which is stored into
the header of the index file). For that, state.is_of_lsn is introduced;
logic is explained in ma_recovery.c (look for "Recovery of the state").
The net gain is that in case of crash, we now recover state.records,
and it is idempotent (ma_test_recovery tests it).
state.checksum is not recovered yet, mail sent for discussion.
- WL#3071 Maria Checkpoint: preparation for it, by protecting
all modifications of the state in memory or on disk with intern_lock
(with the exception of the really-often-modified state.records,
which is now protected with the log's lock, see ma_recovery.c
(look for "Recovery of the state"). Also, if maria_close() sees that
Checkpoint is looking at this table it will not my_free() the share.
- don't compute row's checksum twice in case of UPDATE (correction
to a bugfix I made yesterday).


storage/maria/ha_maria.cc:
  protect state write with intern_lock (against Checkpoint)
storage/maria/ma_blockrec.c:
  * don't reset trn->rec_lsn in _ma_unpin_all_pages(), because it
  should wait until we have corrected the allocation in the bitmap
  (as the REDO can serve to correct the allocation during Recovery);
  introducing _ma_finalize_row() for that.
  * In a changeset yesterday I moved computation of the checksum
  into write_block_record(), to fix a bug in UPDATE. Now I notice
  that maria_update() already computes the checksum, it's just that
  it puts it into info->cur_row while _ma_update_block_record()
  uses info->new_row; so, removing the checksum computation from
  write_block_record(), putting it back into allocate_and_write_block_record()
  (which is called only by INSERT and UNDO_DELETE), and copying
  cur_row->checksum into new_row->checksum in _ma_update_block_record().
storage/maria/ma_check.c:
  new prototypes, they will take intern_lock when writing the state;
  also take intern_lock when changing share->kfile. In both cases
  this is to protect against Checkpoint reading/writing the state or reading
  kfile at the same time.
  Not updating create_rename_lsn directly at end of write_log_record_for_repair()
  as it wouldn't have intern_lock.
storage/maria/ma_close.c:
  Checkpoint builds a list of shares (under THR_LOCK_maria), then it
  handles each such share (under intern_lock) (doing flushing etc);
  if maria_close() freed this share between the two, Checkpoint
  would see a bad pointer. To avoid this, when building the list Checkpoint
  marks each share, so that maria_close() knows it should not free it
  and Checkpoint will free it itself.
  Extending the zone covered by intern_lock to protect against
  Checkpoint reading kfile, writing state.
storage/maria/ma_create.c:
  When we update create_rename_lsn, we also update is_of_lsn to
  the same value: it is logical, and allows us to test in maria_open()
  that the former is not bigger than the latter (the contrary is a sign
  of index header corruption, or severe logging bug which hinders
  Recovery, table needs a repair).
  _ma_update_create_rename_lsn_on_disk() also writes is_of_lsn;
  it now operates under intern_lock (protect against Checkpoint),
  a shortcut function is available for cases where acquiring
  intern_lock is not needed (table's creation or first open).
storage/maria/ma_delete.c:
  if table is transactional, "records" is already decremented
  when logging UNDO_ROW_DELETE.
storage/maria/ma_delete_all.c:
  comments
storage/maria/ma_extra.c:
  Protect modifications of the state, in memory and/or on disk,
  with intern_lock, against a concurrent Checkpoint.
  When state goes to disk, update it's is_of_lsn (by calling
  the new _ma_state_info_write()).
  In HA_EXTRA_FORCE_REOPEN, don't set share->changed to 0 (undoing
  a change I made a few days ago) and ASK_MONTY
storage/maria/ma_locking.c:
  no real code change here.
storage/maria/ma_loghandler.c:
  Log-write-hooks for updating "state.records" under log's mutex
  when writing/updating/deleting a row or deleting all rows.
storage/maria/ma_loghandler_lsn.h:
  merge (make LSN_ERROR and LSN_REPAIRED_BY_MARIA_CHK different)
storage/maria/ma_open.c:
  When opening a table verify that is_of_lsn >= create_rename_lsn; if
  false the header must be corrupted.
  _ma_state_info_write() is split in two: _ma_state_info_write_sub()
  which is the old _ma_state_info_write(), and _ma_state_info_write()
  which additionally takes intern_lock if requested (to protect
  against Checkpoint) and updates is_of_lsn.
  _ma_open_keyfile() should change kfile.file under intern_lock
  to protect Checkpoint from reading a wrong kfile.file.
storage/maria/ma_recovery.c:
  Recovery of state.records: when the REDO phase sees UNDO_ROW_INSERT
  which has a LSN > state.is_of_lsn it increments state.records.
  Same for UNDO_ROW_DELETE and UNDO_ROW_PURGE.
  When closing a table during Recovery, we know its state is at least
  as new as the current log record we are looking at, so increase
  is_of_lsn to the LSN of the current log record.
storage/maria/ma_rename.c:
  update for new behaviour of _ma_update_create_rename_lsn_on_disk().
storage/maria/ma_test1.c:
  update to new prototype
storage/maria/ma_test2.c:
  update to new prototype (actually prototype was changed days ago,
  but compiler does not complain about the extra argument??)
storage/maria/ma_test_recovery.expected:
  new result file of ma_test_recovery. Improvements: record
  count read from index's header is now always correct.
storage/maria/ma_test_recovery:
  "rm" fails if file does not exist. Redirect stderr of script.
storage/maria/ma_write.c:
  if table is transactional, "records" is already incremented when
  logging UNDO_ROW_INSERT. Comments.
storage/maria/maria_chk.c:
  update is_of_lsn too
storage/maria/maria_def.h:
  - MARIA_STATE_INFO::is_of_lsn which is used by Recovery. It is stored
  into the index file's header.
  - Checkpoint can now mark a table as "don't free this", and maria_close()
  can reply "ok then you will free it".
  - new functions
storage/maria/maria_pack.c:
  update for new name

minimize writes to transactional Maria tables: don't write
data pages, state, and open_count at the end of each statement.
Data pages will be written by a background thread periodically.
State will be written by Checkpoint periodically.
open_count serves to detect when a table is potentially damaged
due to an unclean mysqld stop, but thanks to recovery an unclean
mysqld stop will be corrected and so open_count becomes useless.
As state is written less often, it is often obsolete on disk,
we thus should avoid to read it from disk.
- by removing the data page writes above, it is necessary to put
it back at the start of some statements like check, repair and
delete_all. It was already necessary in fact (see ma_delete_all.c).
- disabling CACHE INDEX on Maria tables for now (fixes crash
of test 'key_cache' when run with --default-storage-engine=maria).
- correcting some fishy code in maria_extra.c (we possibly could lose
index pages when doing a DROP TABLE under Windows, in theory).


storage/maria/ha_maria.cc:
  disable CACHE INDEX in Maria for now (there is a single cache for now),
  it crashes and it's not a priority
storage/maria/ma_bitmap.c:
  debug message
storage/maria/ma_check.c:
  The statement before maria_repair() may not flush state,
  so it needs to be done by maria_repair() (indeed this function
  uses maria_open(HA_OPEN_COPY) so reads state from disk,
  so needs to find it up-to-date on disk).
  For safety (but normally this is not needed) we remove index blocks
  out of the cache before repairing.
  _ma_flush_blocks() becomes _ma_flush_table_files_after_repair():
  it now additionally flushes the data file and state and syncs files.
  As a side effect, the assertion "no WRITE_CACHE_USED" from
  _ma_flush_table_files() fired so we move all end_io_cache() done
  at the end of repair to before the calls to _ma_flush_table_files_after_repair().
storage/maria/ma_close.c:
  when closing a transactional table, we fsync it. But we need to
  do this only after writing its state.
  We need to write the state at close time only for transactional
  tables (the other tables do that at last unlock).
  Putting back the O_RDONLY||crashed condition which I had
  removed earlier.
  Unmap the file before syncing it (does not matter now as Maria
  does not use mmap)
storage/maria/ma_delete_all.c:
  need to flush data pages before chsize-ing it. Was needed even when
  we flushed data pages at the end of each statement, because we didn't
  anyway do it if under LOCK TABLES: the change here thus fixes this bug:
  create table t(a int) engine=maria;lock tables t write;
  insert into t values(1);delete from t;unlock tables;check table t;
  "Size of datafile is: 16384       Should be: 8192"
  (an obsolete page went to disk after the chsize(), at unlock time).
storage/maria/ma_extra.c:
  When doing share->last_version=0, we make the MARIA_SHARE-in-memory
  invisible to future openers, so need to have an up-to-date state
  on disk for them. The same way, future openers will reopen the data
  and index file, so they will not find our cached blocks, so we
  need to flush them to disk.
  In HA_EXTRA_FORCE_REOPEN, this probably happens naturally as all
  tables normally get closed, we however add a safety flush.
  In HA_EXTRA_PREPARE_FOR_RENAME, we need to do the flushing. On
  Windows we additionally need to close files.
  In HA_EXTRA_PREPARE_FOR_DROP, we don't need to flush anything but
  remove dirty cached blocks from memory. On Windows we need to close
  files.
  Closing files forces us to sync them before (requirement for transactional
  tables).
  For mutex reasons (don't lock intern_lock twice), we move
  maria_lock_database() and _ma_decrement_open_count() first in the list
  of operations.
  Flush also data file in HA_EXTRA_FLUSH.
storage/maria/ma_locking.c:
  For transactional tables:
    - don't write data pages / state at unlock time;
    as a consequence, "share->changed=0" cannot be done.
    - don't write state in _ma_writeinfo()
    - don't maintain open_count on disk (Recovery corrects the table in case of crash
    anyway, and we gain speed by not writing open_count to disk),
  For non-transactional tables, flush the state at unlock only
  if the table was changed (optimization).
  Code which read the state from disk is relevant only with
  external locking, we disable it (if want to re-enable it, it shouldn't
  for transactional tables as state on disk may be obsolete (such tables
  does not flush state at unlock anymore).
  The comment "We have to flush the write cache" is now wrong because
  maria_lock_database(F_UNLCK) now happens before thr_unlock(), and
  we are not using external locking.
storage/maria/ma_open.c:
  _ma_state_info_read() is only used in ma_open.c, making it static
storage/maria/ma_recovery.c:
  set MARIA_SHARE::changed to TRUE when we are going to apply a
  REDO/UNDO, so that the state gets flushed at close.
storage/maria/ma_test_recovery.expected:
  Changes introduced by this patch:
  - good: the "open" (table open, not properly closed) is gone,
  it was pointless for a recovered table
  - bad: stemming from different moments of writing the index's state
  probably (_ma_writeinfo() used to write the state after every row
  write in ma_test* programs, doesn't anymore as the table is
  transactional): some differences in indexes (not relevant as we don't
  yet have recovery for them); some differences in count of records
  (changed from a wrong value to another wrong value) (not relevant
  as we don't recover this count correctly yet anyway, though
  a patch will be pushed soon).
storage/maria/ma_test_recovery:
  for repeatable output, no names of varying directories.
storage/maria/maria_chk.c:
  function renamed
storage/maria/maria_def.h:
  Function became local to ma_open.c. Function renamed.