Fixed lp:814231 Aria post-recovery error "Bitmap at page 0 has pages reserved outside of data file length"
The bug was a wrong check in aria_chk;  The table was fine.


storage/maria/ma_bitmap.c:
  Print whole bitmap to find errors in last bitmap
storage/maria/ma_check.c:
  Fixed wrong test if bitmap was overallocated.

Fixed bug lp:814054 'Assertion `block->hash_link == hash_link && hash_link->block == block' in ma_pagecache.c:2275 with Aria'
- Replaced old DBUG_ASSERT with a new correct one + a comment.

storage/maria/ma_pagecache.c:
  Replaced old DBUG_ASSERT with a new correct one + a comment.

This fixes a bug that when you use mysqldump --no-create-info to generate a dump that you want to merge with an existing table,
you can get an innodb table with duplicated unique keys.
Patch orignally by Eric Bergen.


client/mysqldump.c:
  Only use UNIQUE_CHECKS=0 for tables that are created.
  This solves the issue that you can't get duplicate unique keys when merging two dumps.
mysql-test/r/mysqldump.result:
  Test for mysqldump --no-create-info

storage/maria/ma_blockrec.c:
  Unlock bitmaps earlier (no reason to have them unlocked over _ma_write_clr())
storage/maria/ma_extra.c:
  Don't lock THR_LOCK_maria for HA_EXTRA_PREPARE_FOR_RENAME (upper level ensures that we are not opening the same table during this call)
  We don't need to have share->intern_lock locked over _ma_flush_table_files()
storage/maria/ma_open.c:
  Update comment

Problem was the parsing of test suite files for various tags and options.
This was done inefficiently, and include files were re-parsed for every
place they were included. This caused a delay of 20 seconds or so before
the first test started to run.

By parsing more efficiently and re-using first parse for subsequent
inclusion of the same file, time spent parsing is reduced to less than
1 second, and start appears instantaneous.

(With this patch, full ./mtr runs in 3 minutes on my laptop (release
build.)

mysql-test/suite/innodb_plugin/t/innodb_bug52663.test:
  Test is fairly slow, so try to avoid getting stuck with it at the end
  while other workers are idle.

Fixed lp:814237: Wrong mutex usage in Aria


storage/maria/ma_bitmap.c:
  Added call to _ma_bitmap_mark_file_changed() when flushing bitmap. This fixes lp:814237

scripts/mysqldumpslow.sh:
  Remove not used --basedir option
  Ensure that the last --datadir option is used from the my.cnf files.

The bug happens when one uses MAX_ROWS=# with Aria & row_format=page and one insert more than # rows.


mysql-test/mysql-test-run.pl:
  Ignore table is full error messages
mysql-test/suite/maria/r/max_length.result:
  Test case for 'Table is full'
mysql-test/suite/maria/t/max_length.test:
  Test case for 'Table is full'
storage/maria/ma_bitmap.c:
  Ensure that we don't allocate bits outside of max_data_file_size.
  Adjust max_data_file_size based on bitmap alignments.
  Backport fix to adjust wrong first_bitmap_with_space.
storage/maria/ma_blockrec.c:
  Calculate value of max_data_file_length
storage/maria/ma_blockrec.h:
  Updated prototype for _ma_bitmap_init()
storage/maria/ma_check.c:
  Give warnings if file sizes are above max file sizes.
  Give more warnings in case of errors.
  Have maria_chk write if table is recreated.
storage/maria/ma_create.c:
  Better calculation of max_data_file_length and thus data pointer length.
  Fixes some wrong pointer lengths when using MAX_ROWS=#
storage/maria/ma_open.c:
  Removed duplicate assigment.
  Use block size from file instead of global variable.
storage/maria/maria_chk.c:
  Remove -1 from printed file length
storage/maria/maria_def.h:
  Update struct st_maria_file_bitmap

Fixed explains of previous patch.

mysql-test/r/explain.result:
  Fixed explains of previous patch.
mysql-test/r/join_outer.result:
  Fixed explains of previous patch.
mysql-test/r/negation_elimination.result:
  Fixed explains of previous patch.
mysql-test/r/view.result:
  Fixed explains of previous patch.
mysql-test/suite/innodb/r/innodb_mysql.result:
  Removed duplicate test suite.
mysql-test/suite/innodb/t/innodb_mysql.test:
  Removed duplicate test suite.
mysql-test/suite/innodb_plugin/r/innodb_mysql.result:
  Removed duplicate test suite.
mysql-test/suite/innodb_plugin/t/innodb_mysql.test:
  Removed duplicate test suite.
sql/opt_range.h:
  Removed incorrect fix.
sql/records.cc:
  Removed incorrect fix.

There are 2 volatile condition constructions AND/OR constructions and fields(references) when first
good supported to be top elements of conditions because it is normal practice
(see copy_andor_structure for example) fields without any expression in the condition is really rare
and mostly useless case however it could lead to problems when optimiser changes/moves them unaware
of other variables referring to them. An easy solution of this problem is just to replace single field
in a condition with equivalent expression well supported by the server (<field> -> <field> != 0).

mysql-test/r/view.result:
  New test added.
mysql-test/t/view.test:
  New test added.
sql/sql_parse.cc:
  <field> -> <field> != 0
sql/sql_yacc.yy:
  <field> -> <field> != 0

Fix is to replace uint in public header with unsigned int.  uint is not guaranteed to be defined by system headers.

before strlen(db) we need to be sure that
db lies within packet boundaries.
same for client_plugin.

before strlen(db) we need to be sure that
db lies within packet boundaries

- Fixed multi-user problem with one thread doing inserts and another doing scans that gave error 175
- Fixed bug that caused assert in move_to_next_bitmap() & _ma_read_bitmap_page()
- Much more DBUG_ASSERT(!maria_assert_if_crashed_table) to detect errors early
- EXTERNAL_LOCKING -> MARIA_EXTERNAL_LOCKING (to use same define everywhere

storage/maria/ma_bitmap.c:
  More secure handling of first_bitmap_with_space (now we also take care of wrong values)
  Don't set page to -1; This fixed unlikely bug that caused assert in move_to_next_bitmap() & _ma_read_bitmap_page()
storage/maria/ma_blockrec.c:
  More DBUG_ASSERT()'s
  Fixed multi-user problem with one thread doing inserts and another doing scans that gave error 175
  (We should use data_file_length from start of scan, not new value as new bitmap page may not yet be in page cache)
storage/maria/ma_check.c:
  EXTERNAL_LOCKING -> MARIA_EXTERNAL_LOCKING (to use same define everywhere)
storage/maria/ma_checkpoint.c:
  Made maria_checkpoint_min_activity static so that one can change it in debugger while testing.
  Fixed long standing performance problem that caused write of state info at checkpoint for any file that was ever changed since open.
storage/maria/ma_create.c:
  EXTERNAL_LOCKING -> MARIA_EXTERNAL_LOCKING (to use same define everywhere
storage/maria/ma_dynrec.c:
  Added missing MARIA_EXTERNAL_LOCKING (minor performance improvement)
storage/maria/ma_locking.c:
  EXTERNAL_LOCKING -> MARIA_EXTERNAL_LOCKING (to use same define everywhere
storage/maria/ma_open.c:
  EXTERNAL_LOCKING -> MARIA_EXTERNAL_LOCKING (to use same define everywhere
storage/maria/ma_pagecache.c:
  Added assert to detect reads outside of data file
storage/maria/maria_def.h:
  Added checkpoint_state to cache state writes in checkpoint

- Fixed performance bug in alter table with Aria; Aria didn't use disable keys + enable keys
- Fixed wrong warning about 'Wrong CRC on datapage' from REPAIR TABLE with aria block tables.
- Fixed bug in aria_chk that disabled performance counters.
- Added --translog_buffer_size to maria_read_log.

storage/maria/ha_maria.cc:
  Fixed performance bug in alter table with Aria; Aria didn't use disable keys + enable keys
storage/maria/ma_bitmap.c:
  Added some DBUG_ASSERT()'s and made code more uniform
storage/maria/ma_check.c:
  Fixed wrong warning about 'Wrong CRC on datapage' from REPAIR TABLE with aria block tables.
storage/maria/ma_pagecache.c:
  Copy read pages to buffer even if there was an error (to be able to detect zero filled pages)
storage/maria/maria_chk.c:
  Fixed bug in aria_chk that disabled performance counters.
storage/maria/maria_read_log.c:
  Added option to set translog_buffer_size

Fixed several errors in Aria discovered by test case for lp:727869 ma_pagecache.c:2103: find_block: Assertion `block->rlocks == 0
- Fixed assert in transaction log handler when aria_check was run on block-record table that was much bigger than expected.
- Fixed warnings about wrong mutex order between bitmap and intern_lock
- Fixed error in bitmap that could cause two rows to use same block for a block record.
- Fixed wrong test that could cause error if last page for a bitmap was used by a blob.
- Fixed several bugs in pagecache for the case where pagecase had very few blocks and there was a lot of threads competing to get the blocks (very unlikely case).


mysql-test/suite/maria/r/maria-recovery3.result:
  Updated results
sql/mysqld.cc:
  Allow mi_check() to send information messages for log file
storage/maria/ma_bitmap.c:
  Fixed problem with wrong mutex order when bitmap was the first page that was flushed out of page cache
  - Fixed by introducing _ma_bitmap_mark_file_changed() that marks file changed without a bitmap lock.
  - Fixed one case in _ma_change_bitmap_page() where we didn't mark the bitmap changed. This could cause to rows to reuse same block if this was the only change to the bitmap.
  - Split _ma_bitmap_get_page_bits() in two parts to not take a bitmap lock when we already have it
  - Fixed bug in _ma_bitmap_set_full_page_bits() that caused an error if last page for a bitmap was used by a blob
storage/maria/ma_check.c:
  Better handling of wrong file length.
  Fixed bug when we tried to write to transaction log when it was not opened (happened when block record file was bigger than expected)
storage/maria/ma_pagecache.c:
  Fixed several bugs in pagecache for the case where pagecase had very few blocks and there was a lot of threads competing to get the blocks:
  - In link_block() mark a block given to another thread with PCBLOCK_REASSIGNED to ensure that no other threads can start re-using the block
    before the thread that requsted a block.
  - In free_block(), don't reset status for a block that is in re-assign by link_block() (we don't want to loose the PCBLOCK_REASSIGNED flag).
  - Added call to wait_for_flush() when we got a new block in find_block() to ensure that we don't use a block that is beeing flushed by another thread.
  - Moved setting of hits_left and last_hit_time in find_block() to where we assign the block.
  
  
  Code cleanup and making code uniform:
  - Changed a lot of KEYCACHE_DBUG_PRINT to use DBUG_PRINT
  - Streamlined all reporting of 'signal' and 'wait' between threads to be identical.
  - Use thread name instead of thread number (for each match against --debug)
  - Added more DBUG_ENTER, DBUG_PRINT and DBUG_ASSERT()
  - Added more comments
storage/myisam/ha_myisam.cc:
  Only print information about that we make a backup if we are really making a backup.
storage/myisam/mi_check.c:
  Inform mysqld that we are creating a backup of the data file (for inclusion in error log).

- Fixed error when writing a blob to the last page on the bitmap.
- Marked bitmap changed in once case that could cause two rows to use the same blob page.

mysql-test/r/join.result:
  Test case for LP:798597
mysql-test/t/join.test:
  Test case for LP:798597
sql/sql_select.cc:
  In simplify_joins we reset table->maybe_null for outer join tables that can't ever be NULL.
  This caused a conflict between the previously calculated items and the group_buffer against the fields
  in the temporary table that are created as not null thanks to the optimization.
  The fix is to correct the group by items to also be not_null so that they match the used fields and keys.

initialize plugins earlier, to support, for example, non-MyISAM mysql.plugin table.

Define USE_MARIA_FOR_TMP_TABLES preprocessor constant by default if Aria engine is compiled in.
Use CMake variable WITH_ARIA_TMP_TABLES to control the temp table engine setting.