Ability for flush_pagecache_blocks() to flush only certain pages of
a file, as instructed by an option "filter" pointer-to-function argument;
Checkpoint and background dirty page flushing use that to flush only
pages which have been dirty for long enough and bitmap pages.
Fix for a bug in flush_cached_blocks() (no idea if it could produce
a bug in real life, but theoretically it is).
Testing checkpoint in ma_test_recovery via ma_test1 and ma_test2.
Background checkpoint & dirty pages flush thread is still disabled
by default in ha_maria.


mysql-test/r/maria.result:
  result update
storage/maria/ha_maria.cc:
  blank after function comment
storage/maria/ma_checkpoint.c:
  Using an enum instead of 0/1/2 (applying Sanja's review comments).
  The comment about "this is an horizon" can be removed as Sanja
  created translog_next_LSN() which parse_checkpoint_record() uses.
  Variables in ma_checkpoint_background() cannot be declared in the
  for() as their value must not be reset at each iteration!
storage/maria/ma_pagecache.c:
  adding to flush_pagecache_blocks() optional arguments 'filter'
  (pointer to function) and 'filter_arg'; if filter!=NULL this function
  will be called for each block of the file and will reply if this
  block and following ones should be flushed or not (3 possible
  replies).
  Fixing a bug when flush_cached_blocks() skips a pinned page: it has
  to unset PCBLOCK_IN_FLUSH set by flush_pagecache_blocks_int().
storage/maria/ma_pagecache.h:
  flush_pagecache_blocks() is changed to take "filter" and "filter_arg"
  arguments. "filter", if it is not NULL, may return one value
  among enum pagecache_flush_filter_result.
storage/maria/ma_recovery.c:
  open_count=0 when closing tables at the end of recovery.
storage/maria/ma_test1.c:
  Optional checkpoints (-H#) at various stages (stages similar
  to --testflag), for testing of checkpoints.
storage/maria/ma_test2.c:
  Optional checkpoints (-H#) at various stages (stages similar
  to -t), for testing of checkpoints.
storage/maria/ma_test_recovery.expected:
  Result update: the results of the additional test run with -H#
  (checkpoints) are added here. They are exactly identical to without
  checkpoints except that the index's Root (printed by maria_chk)
  is more correct when using checkpoints. This is because checkpoint
  flushed the state, so it happens to be correct, while no-checkpoint
  does not flush the state, and recovery does not recover indexes
  so Root is never fixed. When we recover indices, this will go away.
storage/maria/ma_test_recovery:
  We duplicate the loop of tests to add an additional run with
  checkpoints at various stages, to see if maria_read_log
  uses them fine.

storage/maria/ma_delete_all.c:
  comment (@todo gone, nothing we can do)
storage/maria/ma_recovery.c:
  - fail if LOGREC_CREATE_TABLE contains some symlink info (that
  will be handled in a future version)
  - don't do buffer overrun when parsing a REDO record
  - out-of-date @todo

into  gbichot4.local:/home/mysql_src/mysql-maria-tmp

- fixes (in recovery, checkpoint, log handler) of bugs found
during testing.
- new option --check for maria_read_log: with --only-display (which only
reads the header) it reads the full record, for debugging.


storage/maria/ma_loghandler.c:
  importing patch from Sanja for bug of translog_next_LSN() found
  during recovery
storage/maria/ma_loghandler_lsn.h:
  better types (0L is 4 bytes on some platforms, it causes problems
  when used into lsn_store(): right shift >= width of type.
storage/maria/ma_pagecache.c:
  work around infamous "PAGECACHE_PLAIN_PAGE used for transactional
  tables in specialm case"; REDO phase disables logging and this causes
  pages to be PAGECACHE_PLAIN_PAGE, thus ignored wrongly by the
  checkpoint taken at the end of the REDO phase.
storage/maria/ma_recovery.c:
  - a #ifdef which broke maria_read_log in non-debug builds (no output!)
  - support for maria_read_log --check
  - detect record corruption before opening the table
  - updating is_of_horizon requires writing the state
  - fix for wrong parsing of checkpoint record by recovery
storage/maria/ma_recovery.h:
  support for maria_read_log --check
storage/maria/maria_read_log.c:
  Option --check: --only-display only looks at the header;
  adding --check tries a translog_read_record() to see if record can
  be fully read (this is to find bugs).

  bugfixes


dbug/dbug_add_tags.pl:
  bugfixes

client/mysqltest.c:
  fix my_option's with incorrect defaults
mysql-test/r/maria.result:
  update results
mysql-test/t/variables.test:
  update results
sql/mysqld.cc:
  fix my_option's with incorrect defaults

1. available_stack_size() was getting the direction wrong,
   so alloca() was never used
2. (char*) casts added to kill "break strict-aliasing rules" warnings
3. s/node/node=0/ to kill "pointer casted to integer" warning
4. added volatiles as appropriate to prevent gcc from moving
   assignment out of the loop


mysys/lf_alloc-pin.c:
  bugs/warnings fixed:
  1. available_stack_size() was getting the direction wrong,
     so alloca() was never used
  2. (char*) casts added to kill "break strict-aliasing rules" warnings
  3. s/node/node=0/ to kill "pointer casted to integer" warning
  4. added volatiles as appropriate to prevent gcc from moving
     assignment out of the loop

into  janus.mylan:/usr/home/serg/Abk/mysql-maria

mysql-test/r/maria.result:
  test for maria variables
mysql-test/t/maria.test:
  test for maria variables
BitKeeper/etc/ignore:
  Added sql/link_sources ylwrap libmysql_r/link_sources to the ignore list
sql/sql_delete.cc:
  fix incorrect check
storage/maria/ha_maria.cc:
  maria status and pagecache variables

Mutex owner checks added.

  which can happen when you save address of horizon

into  desktop.sanja.is.com.ua:/home/bell/mysql/bk/work-maria-filler

Added ability top change empty space filler of the loghandler.
Fixed end of log reaction.
Fixed memory corruprion bug caused by reading non-filled hage header.
Added debug output.


storage/maria/ma_bitmap.c:
  Compiler warnings removed.
storage/maria/ma_blockrec.c:
  Compiler warnings removed.
storage/maria/ma_loghandler.c:
  Added ability top change empty space filler of the loghandler.
  Fixed end of log reaction.
  Fixed memory corruprion bug caused by reading non-filled hage header.
  Added debug output.
storage/maria/ma_loghandler_lsn.h:
  Compiler warnings removed.

storage/maria/ma_recovery.c:
  bugfix ("count" is meaningul only if printing messages)

plus printing a downcount of tables to flush).


storage/maria/ma_recovery.c:
  improving the progress message of recovery; it used to assume that
  if there is a non-zero UNDO phase then there was a non-REDO phase,
  and if there are tables to flush there was an UNDO phase,
  which is not always true. We now print a downcount of tables to flush
  (4 3 2 1 0).

* Preparation for having a background checkpoint thread:
frequency of checkpoint taken by that thread is now configurable
by the user: global variable maria_checkpoint_frequency, in seconds,
default 30 (checkpoint every 30th second); 0 means no checkpoints
(and thus no background thread, thus no background flushing, that
will probably only be used for testing).
* Don't take checkpoints in Recovery if it didn't do anything
significant; thus no checkpoint after a clean shutdown/restart. The
only checkpoint which is never skipped is the one at shutdown.
* fix for a test failure (after-merge fix)


include/maria.h:
  new variable
mysql-test/suite/rpl/r/rpl_row_flsh_tbls.result:
  result update
mysql-test/suite/rpl/t/rpl_row_flsh_tbls.test:
  position update (=after merge fix, as this position was already changed
  into 5.1 and not merged here, causing test to fail)
storage/maria/ha_maria.cc:
  Checkpoint's frequency is now configurable by the user:
  global variable maria_checkpoint_frequency. Changing it on the fly
  requires us to shutdown/restart the background checkpoint thread,
  as the loop done in that thread assumes a constant checkpoint
  interval. Default value is 30: a checkpoint every 30 seconds (yes, I
  know, physicists will remind that it should be named "period" then).
  ha_maria now asks for a background checkpoint thread when it starts,
  but this is still overruled (disabled) in ma_checkpoint_init().
storage/maria/ma_checkpoint.c:
  Checkpoint's frequency is now configurable by the user: background thread
  takes a checkpoint every maria_checkpoint_interval-th second.
  If that variable is 0, no checkpoints are taken.
  Note, I will enable the background thread only in a later changeset.
storage/maria/ma_recovery.c:
  Don't take checkpoints at the end of the REDO phase and at the end of
  Recovery if Recovery didn't make anything significant (didn't open
  any tables, didn't rollback any transactions).
  With this, after a clean shutdown, Recovery shouldn't take any
  checkpoint, which makes starting faster (we save a few fsync()s of
  the log and control file).

into  gbichot4.local:/home/mysql_src/mysql-maria-tmp


storage/maria/ma_loghandler.c:
  Auto merged

into  desktop.sanja.is.com.ua:/home/bell/mysql/bk/work-maria-bug

into  janus.mylan:/usr/home/serg/Abk/mysql-maria1

into  janus.mylan:/usr/home/serg/Abk/mysql-maria1


Makefile.am:
  Auto merged
client/mysqldump.c:
  Auto merged
configure.in:
  Auto merged
libmysqld/Makefile.am:
  Auto merged
mysql-test/mysql-test-run.pl:
  Auto merged
mysql-test/r/mysqldump.result:
  Auto merged
mysql-test/t/disabled.def:
  Auto merged
mysql-test/t/mysqldump.test:
  Auto merged
mysys/mf_tempfile.c:
  Auto merged
sql/Makefile.am:
  Auto merged
sql/ha_partition.cc:
  Auto merged
sql/ha_partition.h:
  Auto merged
sql/handler.cc:
  Auto merged
sql/lock.cc:
  Auto merged
sql/sql_parse.cc:
  Auto merged
sql/sql_plugin.cc:
  Auto merged
sql/sql_select.cc:
  Auto merged
sql/sql_table.cc:
  Auto merged
sql/table.cc:
  Auto merged
sql/table.h:
  Auto merged
BUILD/compile-dist:
  SCCS merged

  fixed uninit memory access in SET pluginvar=DEFAULT
innodb_mysql.test, innodb_mysql.result:
  test case for SET pluginvar=DEFAULT


mysql-test/r/innodb_mysql.result:
  test case for SET pluginvar=DEFAULT
mysql-test/t/innodb_mysql.test:
  test case for SET pluginvar=DEFAULT
sql/sql_plugin.cc:
  fixed uninit memory access in SET pluginvar=DEFAULT

  use mysqltest builtins instead of --exec. Fixes a failing test too :)


mysql-test/suite/rpl/include/rpl_mixed_dml.inc:
  use mysqltest builtins instead of --exec. Fixes a failing test too :)

  do link_sources only once


sql/Makefile.am:
  do link_sources only once
libmysql_r/Makefile.am:
  do link_sources only once

moved debugging defines.
Fixed buffer flushing (we copied last page
content before it was actually written, now
we abvance pointer in new buffer and unlock
it while waiting for filling old buffer)

Misc changes:
- fix for benign Valgrind error, compiler warnings
- fix for a segfault in execution of maria_delete_all_rows() and one
when taking multiple checkpoints
- fix for too paranoid assertion
- adding ability to take checkpoints at the end of the REDO phase
and at the end of recovery.
- other minor changes


storage/maria/ha_maria.cc:
  The checkpoint done after Recovery is finished, is moved to
  maria_recover().
storage/maria/ma_bitmap.c:
  fix for Valgrind error: the "shadow debug copy" of the bitmap page
  started unitialized and so ma_print_bitmap() would use it uninitialized
storage/maria/ma_checkpoint.c:
  * reset pointers to NULL after freeing them, or we segfault at
  next checkpoint in my_realloc().
  * fix for compiler warnings.
storage/maria/ma_delete_all.c:
  info->trn is NULL for non-transactional tables
storage/maria/ma_locking.c:
  correct assertion (it fired wrongly in execution of REDO_DROP_TABLE
  due to the maria_extra(HA_PREPARE_FOR_DROP)->_ma_decrement_open_count()
  ->maria_lock_database(F_UNLCK); another solution would have been to
  not call _ma_decrement_open_count() (it's ok to have a wrong open
  count in a table which we are dropping), but the same problem
  would still exist for REDO_RENAME_TABLE.
storage/maria/ma_loghandler.c:
  fail early if UNRECOVERABLE_ERROR
storage/maria/ma_recovery.c:
  * new argument to maria_apply_log(): should it take checkpoints
  (at end of REDO phase and at the very end) or no.
  * moving the call to translog_next_LSN() into
  parse_checkpoint_record() ("hide the details").
  * Refining an error detection for something which could happen
  if there is a checkpoint record in the log.
  * Using close_one_table() instead of maria_extra(HA_EXTRA_PREPARE_FOR_DROP|RENAME),
  as it looks safer, and also changing how close_one_table() works:
  it now limits itself to scanning all_tables[], thus having one loopp
  instead of two, which should be faster (as a result, it does not
  close tables not registered in this array, which is ok as there
  should not be any).
storage/maria/ma_recovery.h:
  new parameter
storage/maria/maria_read_log.c:
  update to new prototype

Debug output information fixed.
Fixed direct page referencing in write mode.


storage/maria/ma_loghandler.c:
  TODO added.
storage/maria/ma_pagecache.c:
  Debug output information fixed.
  Fixed direct page referencing in write mode.

note: this does *not* move pagecache* variables.


sql/mysqld.cc:
  remove maria_* variables from the server, declare them in ha_maria.cc
sql/set_var.cc:
  remove maria_* variables from the server, declare them in ha_maria.cc
sql/sql_class.h:
  remove maria_* variables from the server, declare them in ha_maria.cc
sql/sql_plugin.cc:
  bugfix: support for ENUM/SET thd local plugin variables
storage/maria/ha_maria.cc:
  remove maria_* variables from the server, declare them in ha_maria.cc

storage/maria/ma_test1.c:
  typo (?) fixed

into  janus.mylan:/usr/home/serg/Abk/mysql-5.1

into  a88-113-38-195.elisa-laajakaista.fi:/home/my/bk/mysql-maria.prod


BUILD/SETUP.sh:
  Auto merged
sql/opt_range.cc:
  Auto merged
sql/sql_insert.cc:
  Auto merged
storage/maria/ha_maria.cc:
  Auto merged
storage/maria/ma_check.c:
  Auto merged
storage/maria/ma_create.c:
  Auto merged
storage/maria/ma_delete.c:
  Auto merged
storage/maria/ma_open.c:
  Auto merged
storage/maria/ma_update.c:
  Auto merged
storage/maria/ma_write.c:
  Auto merged
storage/maria/maria_chk.c:
  Auto merged
storage/maria/maria_def.h:
  Auto merged
storage/maria/ma_pagecache.c:
  Manual merge with maria tree.
storage/myisam/sort.c:
  Manual merge with maria tree.

storage/maria/ma_test1.c:
  After merge fixes. Applied changes made in myisam manually to maria.
  
  Fixed memory leak.
storage/maria/ma_test2.c:
  After merge fixes. Applied changes made in myisam manually to maria.
  
  Fixed memory leak.

into  trift2.:/MySQL/M51/push-5.1


storage/ndb/test/ndbapi/testScanFilter.cpp:
  Auto merged

into  trift2.:/MySQL/M50/push-5.0

into  production.mysql.com:/usersnfs/jperkin/bk/build/5.1


mysql-test/mysql-test-run.pl:
  Auto merged
scripts/mysql_install_db.sh:
  Auto merged

into  production.mysql.com:/usersnfs/jperkin/bk/bug-27692/5.0

were accidentally removed during a previous rototill of this
code.  Fixes bug#27692.
  
While it can be argued we should strive to provide a 'secure by
default' installation, this happens to be the setup currently
documented in the manual as the default, so defer changes that
improve security out of the box to a co-ordinated effort later
on.
  
For now, make a note about the test databases and anonymous user
in mysql_install_db and recommend that mysql_secure_installation
be ran for users wishing to remove these defaults.

[..re-commit of previously lost change..]


scripts/mysql_system_tables_data.sql:
  Add anonymous accounts.
scripts/mysql_install_db.sh:
  Point users at the mysql_secure_installation script.
mysql-test/mysql-test-run.pl:
  Add a comment where removing anonymous users.

* Thanks to Serg's tip, we fix here the compilation issue of
REDO_REPAIR_TABLE's execution, by defining versions of
_ma_killed_ptr() and _ma_check_print_info|warning|error()
in maria_read_log.c (we move those of maria_chk.c into an include
file and include it in maria_chk.c and maria_read_log.c).
Execution of such record looks like working from my tests (it only
happens in maria_read_log; recovery-from-mysqld skips DDLs and
REPAIR is considered DDL here as it bypasses logging): tested
ALTER TABLE ENABLE KEYS and then remove table, apply log: that
did a repair.
* Recent changes broke maria_read_log a bit: -a became default
and -o caused error; fixing this.


storage/maria/Makefile.am:
  addind new file
storage/maria/ma_recovery.c:
  enable execution of REDO_REPAIR_TABLE by maria_read_log now that
  it compiles. Now reason to keep only T_QUICK from testflag.
storage/maria/maria_chk.c:
  moving these functions to ma_check_standalone.h for reusability
storage/maria/maria_def.h:
  comment
storage/maria/maria_read_log.c:
  ma_check_standalone.h needs my_progname_short.
  Fixing bug where "maria_read_log" would default to -a and
  "maria_read_log -o" would throw an error. Implemented behaviour is:
  - no options: usage()
  - -a : applys, applys UNDOs by default unless --disable-undo
  - -o : only prints
storage/maria/ma_check_standalone.h:
  All standalone programs which need to use functions from ma_check.c
  (like maria_repair()) must define their version of _ma_killed_ptr()
  and _ma_check_print_info|warning|error(). Indeed, linking with ma_check.o
  brings in the dependencies of ma_check.o which are definitions of the above
  functions; if the program does not define them then the ones of
  ha_maria.o are used i.e. ha_maria.o is linked into the program, and this
  brings dependencies of ha_maria.o on mysqld.o into the program's linking
  which thus fails, as the program is not linked with mysqld.o.
  We put in this file the functions which maria_chk.c uses, so that
  they can be reused by maria_read_log (when it replays REDO_REPAIR_TABLE)
  as they are good enough (they just print to stdout/stderr like
  maria_read_log already does).