LP bug #1035225 / MySQL bug #66301: INSERT ... ON DUPLICATE KEY UPDATE +
innodb_autoinc_lock_mode=1 is broken

The problem was that when certain INSERT ... ON DUPLICATE KEY UPDATE
were executed concurrently on a table containing an AUTO_INCREMENT
column as a primary key, InnoDB would correctly reserve non-overlapping
AUTO_INCREMENT intervals for each statement, but when the server
encountered the first duplicate key error on the secondary key in one of
the statements and performed an UPDATE, it also updated the internal
AUTO_INCREMENT value to the one from the existing row that caused a
duplicate key error, even though the AUTO_INCREMENT value was not
specified explicitly in the UPDATE clause. It would then proceed with
using AUTO_INCREMENT values the range reserved previously by another
statement, causing duplicate key errors on the AUTO_INCREMENT column.

Fixed by changing write_record() to ensure that in case of a duplicate
key error the internal AUTO_INCREMENT counter is only updated when the
AUTO_INCREMENT value was explicitly updated by the UPDATE
clause. Otherwise it is restored to what it was before the duplicate key
error, as that value is unused and can be reused for subsequent
successfully inserted rows.

sql/sql_insert.cc:
  Don't update next_insert_id to the value of a row found during ON DUPLICATE KEY UPDATE.
sql/sql_parse.cc:
  Added DBUG_SYNC
sql/table.h:
  Added next_number_field_updated flag to detect changing of auto increment fields.
  Moved fields a bit to get bool fields after each other (better alignment)

create table t1 (a smallint primary key auto_increment);
insert into t1 values(32767);
insert into t1 values(NULL);
ERROR 1062 (23000): Duplicate entry '32767' for key 'PRIMARY

Now on always gets error HA_ERR_AUTOINC_RANGE=167 "Out of range value for column", independent of
store engine, SQL Mode or number of inserted rows. This is an unique error that is easier to test for in replication.

Another bug fix is that we now get an error when trying to insert a too big auto-generated value, even in non-strict mode.
Before one get insted the max column value inserted.
This patch also fixes some issues with inserting negative numbers in an auto-increment column.
Fixed the ER_DUP_ENTRY and HA_ERR_AUTOINC_ERANGE are compared the same between master and slave.
This ensures that replication works between an old server to a new slave for auto-increment overflow errors.
Added SQLSTATE errors for handler errors

Smaller bug fixes:
* Added warnings for duplicate key errors when using INSERT IGNORE
* Fixed bug when using --skip-log-bin followed by --log-bin, which did set log-bin to "0"
* Allow one to see how cmake is called by using --just-print --just-configure


BUILD/FINISH.sh:
  --just-print --just-configure now shows how cmake would be invoked. Good for understanding parameters to cmake.
cmake/configure.pl:
  --just-print --just-configure now shows how cmake would be invoked. Good for understanding parameters to cmake.
include/CMakeLists.txt:
  Added handler_state.h
include/handler_state.h:
  SQLSTATE for handler error messages.
  Required for HA_ERR_AUTOINC_ERANGE, but solves also some other cases.
mysql-test/extra/binlog_tests/binlog.test:
  Fixed old wrong behaviour
  Added more tests
mysql-test/extra/binlog_tests/binlog_insert_delayed.test:
  Reset binary log to only print what's necessary in show_binlog_events
mysql-test/extra/rpl_tests/rpl_auto_increment.test:
  Update to new error codes
mysql-test/extra/rpl_tests/rpl_insert_delayed.test:
  Ignore warnings as this depends on how the test is run
mysql-test/include/strict_autoinc.inc:
  On now gets an error on overflow
mysql-test/r/auto_increment.result:
  Update results after fixing error message
mysql-test/r/auto_increment_ranges_innodb.result:
  Test new behaviour
mysql-test/r/auto_increment_ranges_myisam.result:
  Test new behaviour
mysql-test/r/commit_1innodb.result:
  Added warnings for duplicate key error
mysql-test/r/create.result:
  Added warnings for duplicate key error
mysql-test/r/insert.result:
  Added warnings for duplicate key error
mysql-test/r/insert_select.result:
  Added warnings for duplicate key error
mysql-test/r/insert_update.result:
  Added warnings for duplicate key error
mysql-test/r/mix2_myisam.result:
  Added warnings for duplicate key error
mysql-test/r/myisam_mrr.result:
  Added warnings for duplicate key error
mysql-test/r/null_key.result:
  Added warnings for duplicate key error
mysql-test/r/replace.result:
  Update to new error codes
mysql-test/r/strict_autoinc_1myisam.result:
  Update to new error codes
mysql-test/r/strict_autoinc_2innodb.result:
  Update to new error codes
mysql-test/r/strict_autoinc_3heap.result:
  Update to new error codes
mysql-test/r/trigger.result:
  Added warnings for duplicate key error
mysql-test/r/xtradb_mrr.result:
  Added warnings for duplicate key error
mysql-test/suite/binlog/r/binlog_innodb_row.result:
  Updated result
mysql-test/suite/binlog/r/binlog_row_binlog.result:
  Out of range data for auto-increment is not inserted anymore
mysql-test/suite/binlog/r/binlog_statement_insert_delayed.result:
  Updated result
mysql-test/suite/binlog/r/binlog_stm_binlog.result:
  Out of range data for auto-increment is not inserted anymore
mysql-test/suite/binlog/r/binlog_unsafe.result:
  Updated result
mysql-test/suite/innodb/r/innodb-autoinc.result:
  Update to new error codes
mysql-test/suite/innodb/r/innodb-lock.result:
  Updated results
mysql-test/suite/innodb/r/innodb.result:
  Updated results
mysql-test/suite/innodb/r/innodb_bug56947.result:
  Updated results
mysql-test/suite/innodb/r/innodb_mysql.result:
  Updated results
mysql-test/suite/innodb/t/innodb-autoinc.test:
  Update to new error codes
mysql-test/suite/maria/maria3.result:
  Updated result
mysql-test/suite/maria/mrr.result:
  Updated result
mysql-test/suite/optimizer_unfixed_bugs/r/bug43617.result:
  Updated result
mysql-test/suite/rpl/r/rpl_auto_increment.result:
  Update to new error codes
mysql-test/suite/rpl/r/rpl_insert_delayed,stmt.rdiff:
  Updated results
mysql-test/suite/rpl/r/rpl_loaddatalocal.result:
  Updated results
mysql-test/t/auto_increment.test:
  Update to new error codes
mysql-test/t/auto_increment_ranges.inc:
  Test new behaviour
mysql-test/t/auto_increment_ranges_innodb.test:
  Test new behaviour
mysql-test/t/auto_increment_ranges_myisam.test:
  Test new behaviour
mysql-test/t/replace.test:
  Update to new error codes
mysys/my_getopt.c:
  Fixed bug when using --skip-log-bin followed by --log-bin, which did set log-bin to "0"
sql/handler.cc:
  Ignore negative values for signed auto-increment columns
  Always give an error if we get an overflow for an auto-increment-column (instead of inserting the max value)
  Ensure that the row number is correct for the out-of-range-value error message.
  
  
  ******
  Fixed wrong printing of column namn for "Out of range value" errors
  Fixed that INSERT_ID is correctly replicated also for out-of-range autoincrement values
  Fixed that print_keydup_error() can also be used to generate warnings
  ******
  Return HA_ERR_AUTOINC_ERANGE (167) instead of ER_WARN_DATA_OUT_OF_RANGE for auto-increment overflow
sql/handler.h:
  Allow INSERT IGNORE to continue also after out-of-range inserts.
  Fixed that print_keydup_error() can also be used to generate warnings
sql/log_event.cc:
  Added DBUG_PRINT
  Fixed the ER_AUTOINC_READ_FAILED, ER_DUP_ENTRY and HA_ERR_AUTOINC_ERANGE are compared the same between master and slave.
  This ensures that replication works between an old server to a new slave for auto-increment overflow errors.
sql/sql_insert.cc:
  Add warnings for duplicate key errors when using INSERT IGNORE
sql/sql_state.c:
  Added handler errors
sql/sql_table.cc:
  Update call to print_keydup_error()
storage/innobase/handler/ha_innodb.cc:
  Fixed increment handling of auto-increment columns to be consistent with rest of MariaDB.
storage/xtradb/handler/ha_innodb.cc:
  Fixed increment handling of auto-increment columns to be consistent with rest of MariaDB.

mysql-test/t/features.test:
  --ps-protocol counts both prepare and execute
mysql-test/t/last_value.test:
  Disable ps-protocol for metadata tests

- Performed some refactoring and simplification that was enabled and required by the merge.

Analysis:
The queries in question use the [unique | index]_subquery execution methods.
These methods reuse the ref keys constructed by create_ref_for_key(). The
way create_ref_for_key() works is that it doesn't store in ref.key_copy[]
store_key elements that represent constants. In particular it doesn't store
the store_key for NULL constants.

The execution of [unique | index]_subquery calls
subselect_uniquesubquery_engine::copy_ref_key, which in addition to copy
the left IN argument into a index lookup key, is supposed to detect if
the left IN argument contains NULLs. Since the store_key for the NULL
constant is not copied into the key array, the null is not detected, and
execution erroneously proceeds as if it should look for a complete match.

Solution:
The solution (unlike MySQL) is to reuse already computed information about
NULL presence. Item_in_optimizer::val_int already finds out if the left IN
operand contains NULLs. The fix propagates this to the execution methods
subselect_[unique | index]subquery_engine::exec so it knows if there were
NULL values independent of the presence of keys.

In addition the patch siplifies copy_ref_key() and the logic that hanldes
the case of NULLs in the left IN operand.

Increment long_query_count also if thd->variables.log_slow_rate_limit is used
Added new state "Writing to binlog"


sql/sql_class.h:
  Added THD::utime_after_query to avoid calling current_utime() twice for every end-of-query
sql/sql_parse.cc:
  Increment long_query_count also if thd->variables.log_slow_rate_limit is used
  Removed extra calls to thd_proc_info(thd, "logging slow query") and thd->current_utime();
sql/sql_table.cc:
  Added new state "Writing to binlog"

mysql-test/include/index_merge2.inc:
  InnoDB did report 9 or 7 rows in explain

mysql-test/suite/heap/heap_hash.result:
  Added test case
mysql-test/suite/heap/heap_hash.test:
  Added test case
storage/heap/hp_hash.c:
  Limit key data length to max key length

feature_dynamic_columns,feature_fulltext,feature_gis,feature_locale,feature_subquery,feature_timezone,feature_trigger,feature_xml
Opened_views, Executed_triggers, Executed_events
Added new process status 'updating status' as part of 'freeing items'

mysql-test/r/features.result:
  Test of feature_xxx status variables
mysql-test/r/mysqld--help.result:
  Removed duplicated 'language' variable.
mysql-test/r/view.result:
  Test of opened_views
mysql-test/suite/rpl/t/rpl_start_stop_slave.test:
  Write more information on failure
mysql-test/t/features.test:
  Test of feature_xxx status variables
mysql-test/t/view.test:
  Test of opened_views
sql/event_scheduler.cc:
  Increment executed_events status variable
sql/field.cc:
  Increment status variable
sql/item_func.cc:
  Increment status variable
sql/item_strfunc.cc:
  Increment status variable
sql/item_subselect.cc:
  Increment status variable
sql/item_xmlfunc.cc:
  Increment status variable
sql/mysqld.cc:
  Add new status variables to 'show status'
sql/mysqld.h:
  Added executed_events
sql/sql_base.cc:
  Increment status variable
sql/sql_class.h:
  Add new status variables
sql/sql_parse.cc:
  Added new process status 'updating status' as part of 'freeing items'
sql/sql_trigger.cc:
  Increment status variable
sql/sys_vars.cc:
  Increment status variable
sql/tztime.cc:
  Increment status variable

Simplify code

sql/derror.cc:
  Better error message
sql/mysqld.cc:
  Simplify usage of lc_messages_dir_ptr.
  Don't give warnings that --log-bin should be used if --log-bin or --skip-log-bin is used.

As far as we reopen tables so TABLE become invalid we should remove the pointer on cleanup().

As far as we reopen tables so TABLE become invalid we should remove the pointer on cleanup().

The DELETE for emplicitly emptied MEMORY tables should be written directly to binlog.

Link view/derived table fields to a real table to check turning the table record to null row.

Item_direct_view_ref wrapper now checks if table is turned to null row.

Added function last_value() which returns the last value but evalutes all arguments as a side effect.
Original patch by Eric Herman

client/mysql.cc:
  Added LAST_VALUE
mysql-test/r/last_value.result:
  Testing of LAST_VALUE
mysql-test/t/last_value.test:
  Testing of LAST_VALUE
sql/item_func.cc:
  Added LAST_VALUE()
sql/item_func.h:
  Added LAST_VALUE()
sql/lex.h:
  Added LAST_VALUE()
sql/sql_yacc.yy:
  Added LAST_VALUE()

MDEV-481 Assertion `pins->pin[i] == 0' failed in _lf_pinbox_put_pins on concurrent OPTIMIZE TABLE and DML with Aria tables

A bug in the lock-free hash implementation!
when lsearch() has not found the key, the caller needs to unpin all the three pins,
because lsearch() was using all the three.

   not every time.
2. Increase purgatory size.

include/lf.h:
  allocate larger purgatory
mysys/lf_alloc-pin.c:
  typo.

        Autointersections of an object were treated as nodes, so the wrong result.

per-file comments:
  mysql-test/r/gis.result
Bug #1043845 st_distance() results are incorrect depending on variable order.
        test result updated.
  mysql-test/t/gis.test
Bug #1043845 st_distance() results are incorrect depending on variable order.
        test case added.
  sql/item.cc
        small fix to make compilers happy.
  sql/item_geofunc.cc
Bug #1043845 st_distance() results are incorrect depending on variable order.
        Skip intersection points when calculate distance.

MDEV-414 Depending on indexes or execution plans, a warning on incorrect or out of range values in WHERE condition is sometimes produced and sometimes not

use the same method that disables warnings in all relevant places, remove redundant function

When we append data to the binlog file, we use fdatasync() to ensure
the data gets to disk so that crash recovery can work.

Unfortunately there seems to be a bug in ext3/ext4 on linux, so that
fdatasync() does not correctly sync all data when the size of a file
is increased. This causes crash recovery to not work correctly (it
loses transactions from the binlog).

As a work-around, use fsync() for the binlog, not fdatasync(). Since
we are increasing the file size, (correct) fdatasync() will most
likely not be faster than fsync() on any file system, and fsync()
does work correctly on ext3/ext4. This avoids the need to try to
detect if we are running on buggy ext3/ext4.

store the precision in uint, not uint8

MDEV-438 Microseconds: Precision is ignored in CURRENT_TIMESTAMP(N) when it is given as a default column value

The syntax for specifying precision in the DEFAULT clause is unintentional and unsupported.
Don't allow it anymore.

1. Field_newdate::get_date should refuse to return a date with zeros when
   TIME_NO_ZERO_IN_DATE is set, not when TIME_FUZZY_DATE is unset
2. Item_func_to_days and Item_date_add_interval can only work with valid dates,
   no zeros allowed.

fix Item_func_add_time::get_date() to generate valid dates.
Move the validity check inside get_date_from_daynr()
instead of relying on callers
(5 that had it, and 2 that did not, but should've)