Fix for LP#634943 "marked as crashed", "zerofilling" and "wrong data in bitmap" when recovering Aria tables
This was an interaction of several bugs:
- Tables marked as opened was not properly unmarked on recovery if there was not changes since checkpoint
- zerofill of tables put wrong data in bitmap if directory for page was full
- Tables was thought as 'moved' during recovery if they had a create_lsn bigger than the lsn in the control file.


storage/maria/ha_maria.cc:
  If table is moved and crashed, threat it as crashed.
  (Not a related to this bug, but still good to have fixed)
storage/maria/ma_blockrec.c:
  Make enough_free_entries_on_page() global
storage/maria/ma_blockrec.h:
  Make enough_free_entries_on_page() global
storage/maria/ma_check.c:
  If directory is full, mark page full. Fixes bug in zerofill
storage/maria/ma_open.c:
  Don't marke files as MOVED during recovery if create_trid > trnman_max_trid, as this fails for tables created after checkpoint.
storage/maria/ma_recovery.c:
  Reset open_count in file that was open during crash and was part of checkpoint.
  Fixed wrong warning of 'open count' after recovery of files that was not touched since checkpoint.
storage/maria/maria_chk.c:
  Changed not documented option --log-dir to --logdir
  Document more of the options.
  Clean up output for --help
storage/maria/trnman.c:
  Added DBUG_PRINT

When not-exists optimization was applied to a table that
happened to be an inner table of two outer joins, one
embedded into another, then setting the match flag for
the embedding outer join on could be skipped. This caused
generation of extra null complemented rows.
Made sure that the match flags are set correctly in all cases
when not-exists optimization is used.  

Extend remove_function_from_trace.pl to work with many threads (patch from Sergei)


dbug/remove_function_from_trace.pl:
  Extend remove_function_from_trace.pl to work with many threads (patch from Sergei)
storage/maria/ma_bitmap.c:
  Added marker that table had changed since last checkpoint.
  This ensures that we will flush all bitmap pages from cache at checkpoint.
  This fixes bug LP#605798 "wrong data in bitmap" after recovery.
storage/maria/ma_check.c:
  Cleaned up error output
storage/maria/ma_recovery.c:
  Cleaned up error output
storage/maria/maria_def.h:
  Added changed_not_flushed

Don't give warning about block_insert if table is crashed.

scripts/mysql_install_db.sh:
  Added link to KB.
storage/maria/ma_recovery.c:
  Don't give warning about block_insert if table is crashed.

Write message to stderr if recovery is not likely to succeed because we don't log records for batch inserts.
Don't do UNDO or REDO on a crashed table.
More DBUG_PRINT

sql/sql_parse.cc:
  Remove display of 'packet' which is not useful
storage/maria/ma_blockrec.c:
  More DBUG_PRINT()
storage/maria/ma_key_recover.c:
  Write page number instead of page position
storage/maria/ma_recovery.c:
  Write message to stderr if recovery is not likely to succeed because we don't log records for batch inserts.
  In normal cases this should never be an issue as we would recreate an empty table if MariaDB dies under batch insert.
  This warning will be given if you remove all tables and try to recreate them with maria_read_log, which can't be done as
  the log doesn't contain all data.
  Don't do UNDO or REDO on a crashed table.
storage/maria/ma_write.c:
  Write page number instead of page position.

Fixed typo that caused warnings from mysql-test-run

mysql-test/mysql-test-run.pl:
  Fixed typo
sql/mysqld.cc:
  Give a more precise warning why something fails.

mysql-test/mysql-test-run.pl:
  Added suppression
mysql-test/t/upgrade.test:
  Added missing path

Fixed test failures in buildbot
Don't write errors when failing to send ok packet

mysql-test/suite/pbxt/r/range.result:
  Don't write number of rows as it varies.
mysql-test/suite/pbxt/t/range.test:
  Don't write number of rows as it varies.
sql/mysqld.cc:
  Don't write errors when failing to send ok packet
storage/maria/ma_bitmap.c:
  Added DBUG_ASSERT to detect wrong bitmap pages
storage/maria/ma_blockrec.c:
  Don't reset BLOCKUSED_USE_ORG_BITMAP flag. This fixed a bug where bitmap could be wrong after UNDO of row with blobs

storage/maria/ha_maria.cc:
  Don't use 'trn' variable to avoid warning about not used variable
storage/maria/ma_delete.c:
  Added __attribute__((unused))
storage/maria/ma_key_recover.c:
  Added __attribute__((unused))
  Fixed wrong placement of #endif
storage/maria/ma_key_recover.h:
  Fixed typo

(It is not legal C to do pointer arithmetics on void *).

Fixed bug in Aria when replacing short keys with long keys and a key tree both overflow and underflow at same time.
Fixed several bugs when generating recovery logs when using RGQ with replacing long keys with short keys and vice versa.
Lots of new DBUG_ASSERT()'s
Added more information to recovery log to make it easier to know from where log entry orginated.
Introduced MARIA_PAGE->org_size that tells what the size of the page was in last log entry. This allows us to find out if all key changes for index page was logged.
Small code cleanups:
- Introduced _ma_log_key_changes() to log crc of key page changes
- Added share->max_index_block_size as max size of data one can put in key block (block_size - KEYPAGE_CHECKSUM_SIZE)
  This will later simplify adding a directory to index pages.
- Write page number instead of page postition to DBUG log



mysql-test/lib/v1/mysql-test-run.pl:
  Use --general-log instead of --log to disable warning when using RQG
sql/mysqld.cc:
  If we have already sent ok to client when we get an error, log this to stderr
  Don't disable option --log-output if CSV engine is not supported.
storage/maria/ha_maria.cc:
  Log queries to recovery log also in LOCK TABLES
storage/maria/ma_check.c:
  If param->max_trid is set, use this value instead of max_trid_in_system().
  This is used by recovery to set max_trid to max seen trid so far.
  keyinfo->block_length - KEYPAGE_CHECKSUM_SIZE -> max_index_block_size (Style optimization)
storage/maria/ma_delete.c:
  Mark tables crashed early
  Write page number instead of page position to debug log.
  Added parameter to ma_log_delete() and ma_log_prefix() that is logged so that we can find where wrong log entries where generated.
  Fixed bug where a page was not proplerly written when same key tree had both an overflow and underflow when deleting a key.
  keyinfo->block_length - KEYPAGE_CHECKSUM_SIZE => max_index_block_size (Style optimization)
  ma_log_delete() now has extra parameter of how many bytes from end of page should be appended to log for page (for page overflows)
storage/maria/ma_key_recover.c:
  Added extra parameter to ma_log_prefix() to indicate what caused log entry.
  Update MARIA_PAGE->org_size when logging info about page.
  Much more DBUG_ASSERT()'s.
  Fix some bugs in maria_log_add() to handle page overflows.
  Added _ma_log_key_changes() to log crc of key page changes.
  If EXTRA_STORE_FULL_PAGE_IN_KEY_CHANGES is defines, log the resulting pages to log so one can trivally
  see how the resulting page should have looked like (for errors in CRC values)
storage/maria/ma_key_recover.h:
  Added _ma_log_key_changes() which is only called if EXTRA_DEBUG_KEY_CHANGES is defined.
  Updated function prototypes.
storage/maria/ma_loghandler.h:
  Added more values to en_key_debug, to get more exact location where things went wrong when logging to recovery log.
storage/maria/ma_open.c:
  Initialize share->max_index_block_size
storage/maria/ma_page.c:
  Added updating and testing of MARIA_PAGE->org_size
  Write page number instead of page postition to DBUG log
  Generate error if we read page with wrong data.
  Removed wrong assert: key_del_current != share->state.key_del.
  Simplify _ma_log_compact_keypage()
storage/maria/ma_recovery.c:
  Set param.max_trid to max seen trid before running repair table (used for alter table to create index)
storage/maria/ma_rt_key.c:
  Update call to _ma_log_delete()
storage/maria/ma_rt_split.c:
  Use _ma_log_key_changes()
  Update MARIA_PAGE->org_size
storage/maria/ma_unique.c:
  Remove casts
storage/maria/ma_write.c:
  keyinfo->block_length - KEYPAGE_CHECKSUM_SIZE => share->max_index_block_length.
  Updated calls to _ma_log_prefix()
  Changed code to use _ma_log_key_changes()
  Update ma_page->org_size
  Fixed bug in _ma_log_split() for pages that overflow
  Added KEY_OP_DEBUG logging to functions
  Log KEYPAGE_FLAG in all log entries
storage/maria/maria_def.h:
  Added SHARE->max_index_block_size
  Added MARIA_PAGE->org_size
storage/maria/trnman.c:
  Reset flags for new transaction.

Cleaned up mysql_upgrade --help and mysqlcheck --help

client/mysql_upgrade.c:
  Increased version number.
  Marked all options that are not used 'Not used'.
  Don't write 'Looking for tool' if not using --verbose
client/mysqlcheck.c:
  Cleanup output for --help
  Reset not initialzed variable
mysql-test/r/log_tables_upgrade.result:
  Updated results
mysql-test/r/mysql_upgrade.result:
  Updated results
mysql-test/r/mysqlcheck.result:
  Updated results
mysql-test/t/log_tables_upgrade.test:
  mysql_upgrade is now run without --skip-verbose
mysql-test/t/mysql_upgrade.test:
  mysql_upgrade is now run without --skip-verbose

mysql-test/r/not_partition.result:
  Test result changed after I fixed the error message for not existing engine
mysql-test/suite/funcs_1/r/is_columns_is.result:
  Updated results
mysql-test/suite/funcs_1/r/is_engines_innodb.result:
  Updated results
mysql-test/suite/funcs_1/r/is_tables_is.result:
  Updated results
mysql-test/suite/funcs_1/t/is_tables_is.test:
  Test requires innodb as results depends on innodb
mysql-test/suite/innodb_plugin/t/disabled.def:
  Disable test as it shows errors in valgrind
mysql-test/suite/innodb_plugin/t/innodb-use-sys-malloc.test:
  Test can't be run under valgrind as mysql-test-run resets the innodb_use_sys_malloc flag
storage/xtradb/buf/buf0buf.c:
  Fixed compiler warning by adding casts

Added --silent option to mysql_upgrade so that one can only get errors printed
Don't write unnecessary warning about log tables during upgrade



client/mysql_upgrade.c:
  Don't print connect arguments if not using --verbose --verbose
  Added option --silent to only print errors. This options is passed to mysqlcheck.
  Write out phase names
  The 'verbose' code is a bit strange as I wanted to keep compatibility with old mysql_upgrade
client/mysqlcheck.c:
  Don't upgrade log tables (to avoid confusing warning message that they can't be locked)

Made long file names from previous patch shorter

mysql-test/r/archive.result:
  Added testing of repair (for upgrade) of 5.0 tables.
mysql-test/std_data/archive_5_0.ARM:
  Archive table created in MySQL 5.0
mysql-test/std_data/archive_5_0.ARZ:
  Archive table created in MySQL 5.0
mysql-test/std_data/archive_5_0.frm:
  Archive table created in MySQL 5.0
mysql-test/std_data/long_table_name.MYD:
  Made long file names shorter
mysql-test/std_data/long_table_name.MYI:
  Made long file names shorter
mysql-test/std_data/long_table_name.frm:
  Made long file names shorter
mysql-test/t/archive.test:
  Added testing of repair (for upgrade) of 5.0 tables.
sql/sql_table.cc:
  Allow recreate to open crashed tables.
sql/table.cc:
  Fix error message if storage engine doesn't exists.
storage/archive/azio.c:
  Reset status values in case archive is of old versions
storage/archive/ha_archive.cc:
  Fix to allow one to open old versions of table during repair
  Reset status variables for old version tables
  If the the table is of old version, force upgrade with ALTER TABLE when doing repair
storage/archive/ha_archive.h:
  Added variables to detect old versions

Fall back to use ALTER TABLE for engines that doesn't support REPAIR when doing repair for upgrade.
Nicer output from mysql_upgrade and mysql_check
Updated all arrays that used NAME_LEN to use SAFE_NAME_LEN to ensure that we don't break things accidently as names can now have a #mysql50# prefix.

client/mysql_upgrade.c:
  If we are using verbose, also run mysqlcheck in verbose mode.
client/mysqlcheck.c:
  Add more information if running in verbose mode
  Print 'Needs upgrade' instead of complex error if table needs to be upgraded
  Don't write connect information if verbose is not 2 or above
mysql-test/r/drop.result:
  Updated test and results as we now support full table names
mysql-test/r/grant.result:
  Now you get a correct error message if using #mysql with paths
mysql-test/r/show_check.result:
  Update results as table names can temporarly be bigger than NAME_LEN (during upgrade)
mysql-test/r/upgrade.result:
  Test upgrade for long table names.
mysql-test/suite/funcs_1/r/is_tables_is.result:
  Updated old test result (had note been updated in a while)
mysql-test/t/drop.test:
  Updated test and results as we now support full table names
mysql-test/t/grant.test:
  Now you get a correct error message if using #mysql with paths
mysql-test/t/upgrade.test:
  Test upgrade for long table names.
sql/ha_partition.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/item.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/log_event.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/mysql_priv.h:
  Added SAFE_NAME_LEN
sql/rpl_filter.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/sp.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/sp_head.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/sql_acl.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/sql_base.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/sql_connect.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/sql_parse.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/sql_prepare.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/sql_select.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/sql_show.cc:
  NAME_LEN -> SAFE_NAME_LEN
  Enlarge table names for SHOW TABLES to also include optional #mysql50#
sql/sql_table.cc:
  Fall back to use ALTER TABLE for engines that doesn't support REPAIR when doing repair for upgrade.
sql/sql_trigger.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/sql_udf.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/sql_view.cc:
  NAME_LEN -> SAFE_NAME_LEN
sql/table.cc:
  Fixed check_table_name() to not count #mysql50# as part of name
  If #mysql50# is part of the name, don't allow path characters in name.

sql/item_sum.cc:
  Fixed alignment problem that caused crases on sparc. (ORDER needs an aligment of 8 as it includes longlongs)
storage/federatedx/ha_federatedx.cc:
  Fixed crash on sparc as 'pos' is not aligned on 4/8.
storage/maria/ma_page.c:
  Removed wrong assert

mysql-test/t/loaddata.test:
  Don't run test if ucs2 is not compiled

to work in cygnin perl too

mysql-test/t/bug46080-master.opt:
  Lower limits to be able to run tests
regex/main.c:
  Fixed compiler warnings
storage/maria/ma_key_recover.c:
  Fixed compiler warnings
storage/maria/ma_recovery.c:
  Fixed compiler warnings
storage/maria/ma_unique.c:
  Fixed compiler warnings
strings/ctype-uca.c:
  Added comment
strings/xml.c:
  Fixed compiler warnings
support-files/compiler_warnings.supp:
  Added suppressions for windows
unittest/strings/strings-t.c:
  Added ifdef to fix compilation failure when compiling without UCA

mysql-test/suite/binlog/t/binlog_row_binlog.test:
  Don't run test if utf8_unicode_ci is not available
mysql-test/suite/binlog/t/binlog_stm_binlog.test:
  Don't run test if utf8_unicode_ci is not available
mysql-test/suite/funcs_1/r/is_columns_is.result:
  Update result
mysql-test/suite/innodb/t/innodb_misc1.test:
  Don't run test if utf8_unicode_ci is not available
mysql-test/suite/innodb/t/innodb_mysql.test:
  Don't run test if utf8_unicode_ci is not available

remove duplicates from the --plugin-load list.
$ENV{TERM} can be undefined (on Windows)

- Changed to still use bcmp() in certain cases becasue
  - Faster for short unaligneed strings than memcmp()
  - Bettern when using valgrind
- Changed to use my_sprintf() instead of sprintf() to get higher portability for old systems
- Changed code to use MariaDB version of select->skip_record()
- Removed -%::SCCS/s.% from Makefile.am:s to remove automake warnings

* expanding unknown env variable does not abort mtr
* have_archive, blackhole, innodb - preload the corresponding engine
* all options from .opt files get equal treatment, all are searched for special options,
  not only -{master,slave}.opt as before (which ignored suite.opt and --mysqld=...)
* --plugin-load gets special treatment too - all instances are merged into one
* federated test fixed to preload federated

- Need to get autopush to work to avoid bad pushes like this....