- In QUICK_INDEX_MERGE_SELECT::read_keys_and_merge: when we got table->sort from Unique,
  tell init_read_record() not to use rr_from_cache() because a) rowids are already sorted
  and b) it might be that the the data is used by filesort(), which will need record rowids
  (which rr_from_cache() cannot provide).
- Fully de-initialize the table->sort read in QUICK_INDEX_MERGE_SELECT::get_next(). This fixes BUG#35477.
(bk trigger: file as fix for BUG#35478).

sql/filesort.cc:
  BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
  - make find_all_keys() use quick->get_next() instead of init_read_record(r)/r.read_record() calls
  - added dbug printout
sql/mysql_priv.h:
  BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
  - Added parameter to init_read_record
sql/opt_range.cc:
  BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
  - In QUICK_INDEX_MERGE_SELECT::read_keys_and_merge: when we got table->sort from Unique,
    tell init_read_record() not to use rr_from_cache() because a) rowids are already sorted
    and b) it might be that the the data is used by filesort(), which will need record rowids
    (which rr_from_cache() cannot provide).
  - Fully de-initialize the table->sort read in QUICK_INDEX_MERGE_SELECT::get_next().
sql/records.cc:
  BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
  - Added disable_rr_cache parameter to init_read_record
  - Added comment
sql/sql_acl.cc:
  BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
  - Added parameter to init_read_record
sql/sql_delete.cc:
  BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
  - Added parameter to init_read_record
sql/sql_help.cc:
  BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
  - Added parameter to init_read_record
sql/sql_select.cc:
  BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
  - Added parameter to init_read_record
sql/sql_table.cc:
  BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
  - Added parameter to init_read_record
sql/sql_udf.cc:
  BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
  - Added parameter to init_read_record
sql/sql_update.cc:
  BUG#35478: sort_union() returns bad data when sort_buffer_size is hit
  - Added parameter to init_read_record

build)

The crash was caused by freeing the internal parser stack during the parser
execution.
This occured only for complex stored procedures, after reallocating the parser
stack using my_yyoverflow(), with the following C call stack:
- MYSQLparse()
- any rule calling sp_head::restore_lex()
- lex_end()
- x_free(lex->yacc_yyss), xfree(lex->yacc_yyvs)

The root cause is the implementation of stored procedures, which breaks the
assumption from 4.1 that there is only one LEX structure per parser call.

The solution is to separate the LEX structure into:
- attributes that represent a statement (the current LEX structure),
- attributes that relate to the syntax parser itself (Yacc_state),
so that parsing multiple statements in stored programs can create multiple
LEX structures while not changing the unique Yacc_state.

Now, Yacc_state and the existing Lex_input_stream are aggregated into
Parser_state, a structure that represent the complete state of the (Lexical +
Syntax) parser.


mysql-test/r/parser_stack.result:
  Bug#35577 (CREATE PROCEDURE causes either crash or syntax error depending on
  build)
mysql-test/t/parser_stack.test:
  Bug#35577 (CREATE PROCEDURE causes either crash or syntax error depending on
  build)
sql/sp.cc:
  Bug#35577 (CREATE PROCEDURE causes either crash or syntax error depending on
  build)
sql/sp_head.cc:
  Bug#35577 (CREATE PROCEDURE causes either crash or syntax error depending on
  build)
sql/sql_class.cc:
  Bug#35577 (CREATE PROCEDURE causes either crash or syntax error depending on
  build)
sql/sql_class.h:
  Bug#35577 (CREATE PROCEDURE causes either crash or syntax error depending on
  build)
sql/sql_lex.cc:
  Bug#35577 (CREATE PROCEDURE causes either crash or syntax error depending on
  build)
sql/sql_lex.h:
  Bug#35577 (CREATE PROCEDURE causes either crash or syntax error depending on
  build)
sql/sql_parse.cc:
  Bug#35577 (CREATE PROCEDURE causes either crash or syntax error depending on
  build)
sql/sql_prepare.cc:
  Bug#35577 (CREATE PROCEDURE causes either crash or syntax error depending on
  build)
sql/sql_trigger.cc:
  Bug#35577 (CREATE PROCEDURE causes either crash or syntax error depending on
  build)
sql/sql_view.cc:
  Bug#35577 (CREATE PROCEDURE causes either crash or syntax error depending on
  build)
sql/sql_yacc.yy:
  Bug#35577 (CREATE PROCEDURE causes either crash or syntax error depending on
  build)

            and value-list

The server returns unexpected results if a right side of the 
NOT IN clause consists of NULL value and some constants of
the same type, for example:

  SELECT * FROM t WHERE NOT t.id IN (NULL, 1, 2) 
  
may return 3, 4, 5 etc if a table contains these values.


The Item_func_in::val_int method has been modified:
unnecessary resets of an Item_func_case::has_null field 
value has been moved outside of an argument comparison
loop. (Also unnecessary re-initialization of the null_value
field has been moved).


mysql-test/r/func_in.result:
  Added test case for bug #37761.
mysql-test/t/func_in.test:
  Added test case for bug #37761.
sql/item_cmpfunc.cc:
  Bug #37761: IN handles NULL differently for table-subquery 
              and value-list
  
  The Item_func_in::val_int method has been modified:
  unnecessary resets of an Item_func_case::has_null field 
  value has been moved outside of an argument comparison
  loop. (Also unnecessary re-initialization of the null_value
  field has been moved).

libmysqld/emb_qcache.cc:
  Fixed build warning on Windows:
  ported the fix from 6.0 in 5.1

offset for time part in UUIDs was 1/1000 of what it
should be. In other words, offset was off.

Also handle the case where we count into the future
when several UUIDs are generated in one "tick", and
then the next call is late enough for us to unwind
some but not all of those borrowed ticks.

Lastly, handle the case where we keep borrowing and
borrowing until the tick-counter overflows by also
changing into a new "numberspace" by creating a new
random suffix.

mysql-test/r/func_misc.result:
  Show that time-part of UUIDs is correct now.
mysql-test/t/func_misc.test:
  Show that time-part of UUIDs is correct now
  by replicating the C-code's resultin SQL.
  Results also decode to expect date-data on
  command-line (external validation).
  
  No test for unwinding of borrowed ticks as
  this a) is a race and b) depends on what timer
  we get.
sql/item_strfunc.cc:
  correct offset for date/time-part of UUID.
  also make sure that when we counted into
  the future earlier (several UUIDs generated
  in same tick), we only give back as many
  "borrowed" ticks as we can without duplicating
  past timestamps. If our tick-counter overflows
  before we can give back, or if the system-clock
  is set back (by user or Daylight Saving Time),
  we create a new random suffix to avoid
  collisions and clear the tick-counter.

offset for time part in UUIDs was 1/1000 of what it
should be. In other words, offset was off.

Also handle the case where we count into the future
when several UUIDs are generated in one "tick", and
then the next call is late enough for us to unwind
some but not all of those borrowed ticks.

Lastly, handle the case where we keep borrowing and
borrowing until the tick-counter overflows by also
changing into a new "numberspace" by creating a new
random suffix.


mysql-test/r/func_misc.result:
  Show that time-part of UUIDs is correct now.
mysql-test/t/func_misc.test:
  Show that time-part of UUIDs is correct now
  by replicating the C-code's resultin SQL.
  Results also decode to expect date-data on
  command-line (external validation).
  
  No test for unwinding of borrowed ticks as
  this a) is a race and b) depends on what timer
  we get.
sql/item_strfunc.cc:
  correct offset for date/time-part of UUID.
  also make sure that when we counted into
  the future earlier (several UUIDs generated
  in same tick), we only give back as many
  "borrowed" ticks as we can without duplicating
  past timestamps. If our tick-counter overflows
  before we can give back, or if the system-clock
  is set back (by user or Daylight Saving Time),
  we create a new random suffix to avoid
  collisions and clear the tick-counter.

Problem: reading/writing data from/to an address without proper
alignment leads to SIGBUS on some platforms.

Fix: use the correct data type when dereferencing variable values.


sql/sql_plugin.cc:
  Fix for bug#37669: Server crash by setting "@@session.innodb_table_locks"
  
  Use *(my_bool *) typecast to avoid misaligned reading/writing and 
  to get/set proper boolean values.

   "funcs_2: The tests do not check if optional character
               sets exist."
2. Minor cleanup

The problem is that relying on the output of the 'ls' command is not
portable as its behavior is not the same between systems and it might
even not be available at all in (Windows).

So I added list_files that relies on the portable mysys library instead.
(and also list_files_write_file and list_files_append_file,
since the test was using '--exec ls' in that way.)

client/mysqltest.c:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  To be able to replace the use of '--exec ls' I have added
  list_files, list_files_write_file and list_files_append_file.
  
  list_files <dirname> [<filename incl. wild-cards>]
  is equivalent to 'ls <dirname>/[<filename incl. wild-cards>]'
  
  list_files_write_file creates/overwrites a file with the content
  list_files_append_file creates/appends a file with the content
  list_files* return a sorted output.
mysql-test/r/mysqltest.result:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  result file change, due to added test of the new list_files command.
mysql-test/suite/parts/inc/partition_check_drop.inc:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Using the new list_files instead of 'ls'.
  
  Changed the use of local variables (ls_file, file_list)
  and server variable (@aux).
mysql-test/suite/parts/inc/partition_layout.inc:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Using the new list_files instead of 'ls'.
mysql-test/suite/parts/inc/partition_layout_check1.inc:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Using the new list_files instead of 'ls'.
mysql-test/suite/parts/inc/partition_layout_check2.inc:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Replaces '--exec ls' with list_files.
  Removal of the directory part of file listing.
mysql-test/suite/parts/r/partition_alter1_1_2_myisam.result:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Replaces '--exec ls' with list_files.
  Removal of the directory part of file listing.
mysql-test/suite/parts/r/partition_alter1_1_myisam.result:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Replaces '--exec ls' with list_files.
  Removal of the directory part of file listing.
mysql-test/suite/parts/r/partition_alter1_2_myisam.result:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Replaces '--exec ls' with list_files.
  Removal of the directory part of file listing.
mysql-test/suite/parts/r/partition_alter2_myisam.result:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Replaces '--exec ls' with list_files.
  Removal of the directory part of file listing.
mysql-test/suite/parts/r/partition_alter3_innodb.result:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Replaces '--exec ls' with list_files.
  Removal of the directory part of file listing.
mysql-test/suite/parts/r/partition_alter3_myisam.result:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Replaces '--exec ls' with list_files.
  Removal of the directory part of file listing.
mysql-test/suite/parts/r/partition_basic_innodb.result:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Replaces '--exec ls' with list_files.
  Removal of the directory part of file listing.
mysql-test/suite/parts/r/partition_basic_myisam.result:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Replaces '--exec ls' with list_files.
  Removal of the directory part of file listing.
mysql-test/suite/parts/r/partition_basic_symlink_myisam.result:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Replaces '--exec ls' with list_files.
  Removal of the directory part of file listing.
mysql-test/suite/parts/r/partition_engine_myisam.result:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Replaces '--exec ls' with list_files.
  Removal of the directory part of file listing.
mysql-test/suite/parts/r/partition_syntax_myisam.result:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Replaces '--exec ls' with list_files.
  Removal of the directory part of file listing.
mysql-test/t/mysqltest.test:
  Bug#35736 Test 'parts.partition_basic_symlink_myisam' depends on output of 'ls'
  
  Adding test for the new mysqltest commands list_files,
  list_files_write_file and list_files_append_file.

Fixed the test to expect the correct result.
The previous test script was in fact affected by 26030,
and wrongly expected a ER_PARSE_ERROR error.

problem was that ha_partition::records was not implemented, thus
using the default handler::records, which is not correct if the engine
does not support HA_STATS_RECORDS_IS_EXACT.
Solution was to implement ha_partition::records as a wrapper around
the underlying partitions records.

The rows column in explain partitions will now include the total
number of records in the partitioned table.

(recommit after removing out-commented code)

enabled)

Before this fix, the lexer and parser would treat the ';' character as a
different token (either ';' or END_OF_INPUT), based on convoluted logic,
which failed in simple cases where a stored procedure is implemented as a
single statement, and used in a multi query.

With this fix:
- the character ';' is always parsed as a ';' token in the lexer,
- parsing multi queries is implemented in the parser, in the 'query:' rules,
- the value of thd->client_capabilities, which is the capabilities
  negotiated between the client and the server during bootstrap,
  is immutable and not arbitrarily modified during parsing (which was the
  root cause of the bug)

 - moved the test into a separate file to check for presence of the test variable

mysql-test/r/subselect.result:
  Bug#37627: moved the test to a separate file
mysql-test/r/subselect_debug.result:
  Bug#37627: moved the test to a separate file
mysql-test/t/subselect.test:
  Bug#37627: moved the test to a separate file
mysql-test/t/subselect_debug.test:
  Bug#37627: moved the test to a separate file