can crash under load

Backport from 5.1.
Does also include key cache fixes from:
Bug 44068 (RESTORE can disable the MyISAM Key Cache)
Bug 40944 (Backup: crash after myisampack)



include/keycache.h:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Added KEY_CACHE components in_resize and waiting_for_resize_cnt.
myisam/mi_preload.c:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Added code to allow LOAD INDEX to load indexes of different block size.
mysys/mf_keycache.c:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  .
  Changed resize_key_cache() to not disable the key cache
  after the flush phase. Changed queue handling to use
  standard functions. Wake all threads waiting on resize_queue.
  We can now have read/write threads waiting there (see below).
  .
  Combined add_to_queue() and the wait loops that were always
  following it to the new function wait_on_queue().
  Combined release_queue() and the condition that was always
  preceding it to the new function release_whole_queue().
  .
  Added code to flag and respect the exceptional situation
  BLOCK_IN_EVICTION.
  .
  Rewrote the resize branch of find_key_block().
  .
  Added code to the eviction handling in find_key_block()
  to catch more exceptional cases.
  .
  Changed key_cache_read(), key_cache_insert() and key_cache_write()
  so that they lock keycache->cache_lock whenever the key cache is
  initialized. Checking for a disabled cache and incrementing and
  decrementing the "resize counter" is always done within the lock.
  Locking and unlocking as well as counting the "resize counter" is
  now done once outside the loop. All three functions can now handle
  a NULL return from find_key_block. This happens in the flush phase
  of a resize and demands direct file I/O. Care is taken for
  secondary requests (PAGE_WAIT_TO_BE_READ) to wait in any case.
  Moved block status changes behind the copying of buffer data.
  key_cache_insert() does now read the block if the caller did
  supply less data than a full cache block.
  key_cache_write() does now take care of parallel running flushes
  (BLOCK_FOR_UPDATE, BLOCK_IN_FLUSHWRITE).
  .
  Changed free_block() to un-initialize block variables in the
  correct order and respect an exceptional BLOCK_IN_EVICTION state.
  .
  Changed flushing to take care for parallel running writes.
  Changed flushing to avoid freeing blocks in eviction.
  Changed flushing to consider that parallel writes can move blocks
  from the file_blocks hash to the changed_blocks hash.
  Changed flushing to take care for other parallel flushes.
  Changed flushing to assure that it ends with everything flushed.
  Optimized normal flush at end of statement (FLUSH_KEEP),
  but let other flush types be stringent.
  .
  Added some comments and debugging statements.
mysys/my_static.c:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Removed an unused global variable.
sql/ha_myisam.cc:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Moved an automatic (stack) variable to the scope where it is used.
sql/sql_table.cc:
  Bug#17332 - changing key_buffer_size on a running server
              can crash under load
  Changed TL_READ to TL_READ_NO_INSERT in mysql_preload_keys.

Memory allocated in TMP_TABLE_PARAM::copy_field is not cleaned up.
The fix is to clean up TMP_TABLE_PARAM::copy_field array in JOIN::destroy.


mysql-test/r/explain.result:
  test result
mysql-test/t/explain.test:
  test case
sql/sql_select.cc:
  Memory allocated in TMP_TABLE_PARAM::copy_field is not cleaned up.
  The fix is to clean up TMP_TABLE_PARAM::copy_field array in JOIN::destroy.

                    name as existing view

When trying to create a table with the same name as existing view with
join, mysql server crashes.

The problem is when create table is issued with the same name as view, while
verifying with the existing tables, we assume that base table object is 
created always.

In this case, since it is a view over multiple tables, we don't have the 
mysql derived table object.

Fixed the logic which checks if there is an existing table to not to assume
that table object is created when the base table is view over multiple 
tables.

mysql-test/r/create.result:
  BUG#46384 - mysqld segfault when trying to create table with same 
              name as existing view
  
  Testcase for the bug
mysql-test/t/create.test:
  BUG#46384 - mysqld segfault when trying to create table with same 
              name as existing view
  
  Testcase for the bug
sql/sql_insert.cc:
  BUG#46384 - mysqld segfault when trying to create table with same 
                  name as existing view
      
  Fixed create_table_from_items() method to properly check, if the base table 
  is a view over multiple tables.

                                for CREATE TABLE...LIKE...

Add my_sync.c to mysqltest sources list in CMakeLists.txt

client/CMakeLists.txt:
  BUG#46591 - .frm file isn't sync'd with sync_frm enabled
              for CREATE TABLE...LIKE...
  
  Add my_sync.c to mysqltest sources list

myisamchk tool generates warnings when run on an myisam files (.MYI or .MYD)
This is because of the conversion of max_value for certain options in myisamchk 
from singed long to unsigned long

The max value for the options key_buffer_size, read_buffer_size, write_buffer
_size and sort_buffer_size is given as (long) ~0L which becomes -1  when casted
from signed long to longlong and then casted to ulonglong. When (ulonglong) -1 
is compared with maximal value for GET_ULONG data type, we adjust it to 
(ulonglong) ULONG_MAX and throw the warning.

Fixed by using the right max size.

Max values for the variables (from mysqld.cc)
----------------------------
1. key_buffer_size
   5.0: ULONG_MAX
   5.1: SIZE_T_MAX
   6.0: SIZE_T_MAX

2. read_buffer_size and write_buffer_size
   5.0: INT_MAX32
   5.1: INT_MAX32
   6.0: INT_MAX32

3. sort_buffer_size (aka myisam_sort_buffer_size)
   5.0: UINT_MAX32
   5.1: ULONG_MAX
   6.0: ULONG_MAX

Note: testcase not attached

myisam/myisamchk.c:
  Bug#33785 - myisamchk show warning message
      
  Fixed the Max value for key_buffer_size, read_buffer_size, write_buffer_size and
  sort_buffer_size options

                    CREATE TABLE...LIKE...
      
The mysql server option 'sync_frm' is ignored when table is created with 
syntax CREATE TABLE .. LIKE.. 
      
Fixed by adding the MY_SYNC flag and calling my_sync() from my_copy() when
the flag is set.

In mysql_create_table(), when the 'sync_frm' is set, MY_SYNC flag is passed 
to my_copy(). 
      
Note: TestCase is not attached and can be tested manually using debugger.

client/Makefile.am:
  BUG#46591 - .frm file isn't sync'd with sync_frm enabled for 
              CREATE TABLE...LIKE...
      
  add my_sync to sources as it is used in my_copy() method
include/my_sys.h:
  BUG#46591 - .frm file isn't sync'd with sync_frm enabled for 
              CREATE TABLE...LIKE...
      
  MY_SYNC flag is added to call my_sync() method
mysys/my_copy.c:
  BUG#46591 - .frm file isn't sync'd with sync_frm enabled for 
              CREATE TABLE...LIKE...
      
  my_sync() is method is called when MY_SYNC is set in my_copy()
sql/sql_table.cc:
  BUG#46591 - .frm file isn't sync'd with sync_frm enabled for 
              CREATE TABLE...LIKE...
      
  Fixed mysql_create_like_table() to call my_sync() when opt_sync_frm variable
  is set

with the newer pb2 testing environments

Failing to connect would release parts of the MYSQL struct.
We would then proceed to try again to connect without re-
initializing the struct.

We prevent the unwanted freeing of data we'll still need now.


client/mysqladmin.cc:
  Losing a connection (or not even getting on in the first place) should
  not trash the MYSQL-struct.
  
  Add a lot of comments.
  
  Rewrite re-connection fu.
sql-common/client.c:
  Assert against bad parameters usually caused by de-initing a
  MYSQL-struct without re-initing it again before re-use.

with gcc 4.3.2
      
This patch fixes a number of GCC warnings about variables used
before initialized. A new macro UNINIT_VAR() is introduced for
use in the variable declaration, and LINT_INIT() usage will be
gradually deprecated. (A workaround is used for g++, pending a
patch for a g++ bug.)
      
GCC warnings for unused results (attribute warn_unused_result)
for a number of system calls (present at least in later
Ubuntus, where the usual void cast trick doesn't work) are
also fixed.


client/mysqlmanager-pwgen.c:
  A fix for warn_unused_result, adding fallback to use of
  srand()/rand() if /dev/random cannot be used. Also actually
  adds calls to rand() in the second branch so that it actually
  creates a random password.

When a connection is dropped any remaining temporary table is also automatically
dropped and the SQL statement of this operation is written to the binary log in
order to drop such tables on the slave and keep the slave in sync. Specifically,
the current code base creates the following type of statement:
DROP /*!40005 TEMPORARY */ TABLE IF EXISTS `db`.`table`;

Unfortunately, appending the database to the table name in this manner circumvents
the replicate-rewrite-db option (and any options that check the current database).
To solve the issue, we started writing the statement to the binary as follows:
use `db`; DROP /*!40005 TEMPORARY */ TABLE IF EXISTS `table`;

  field references

This error requires a combination of factors : 
1. An "impossible where" in the outermost SELECT
2. An aggregate in the outermost SELECT
3. A correlated subquery with a WHERE clause that includes an outer 
field reference as a top level WHERE sargable predicate

When JOIN::optimize detects an "impossible WHERE" it will bail out
without doing the rest of the work and initializations. It will not
call make_join_statistics() as well.  And make_join_statistics fills 
in various structures for each table referenced.
When processing the result of the "impossible WHERE" the query must
send a single row of data if there are aggregate functions in it.
In this case the server marks all the aggregates as having received 
no rows and calls the relevant Item::val_xxx() method on the SELECT
list. However if this SELECT list happens to contain a correlated 
subquery this subquery is evaluated in a normal evaluation mode.
And if this correlated subquery has a reference to a field from the 
outermost "impossible where" SELECT the add_key_fields will mistakenly
consider the outer field reference as a "local" field reference when 
looking for sargable predicates.
But since the SELECT where the outer field reference refers to is not
completely initialized due to the "impossible WHERE" in this level
we'll get a NULL pointer reference.
Fixed by making a better condition for discovering if a field is "local"
to the SELECT level being processed. 
It's not enough to look for OUTER_REF_TABLE_BIT in this case since 
for outer references to constant tables the Item_field::used_tables() 
will return 0 regardless of whether the field reference is from the 
local SELECT or not.

The crash happens because select_union object is used as result set
for queries which have derived tables.
select_union use temporary table as data storage and if
fields count exceeds 10(count of values for procedure ANALYSE())
then we get a crash on fill_record() function.


mysql-test/r/analyse.result:
  test result
mysql-test/r/subselect.result:
  result fix
mysql-test/t/analyse.test:
  test case
mysql-test/t/subselect.test:
  test fix
sql/sql_yacc.yy:
  The crash happens because select_union object is used as result set
  for queries which have derived tables.
  select_union use temporary table as data storage and if
  fields count exceeds 10(count of values for procedure ANALYSE())
  then we get a crash on fill_record() function.

The code was using a special global buffer for the value of IS NULL ranges.
This was not always long enough to be copied by a regular memcpy. As a 
result read buffer overflows may occur.
Fixed by setting the null byte to 1 and setting the rest of the field disk image
to NULL with a bzero (instead of relying on the buffer and memcpy()).

         decrease for INSERTs


Bulk inserts (multiple row, CREATE ... SELECT, INSERT ... SELECT) into
MyISAM tables were performed inefficiently. This was mainly affecting
use cases where read_buffer_size was considerably large (>256K) and low
number of rows was inserted (e.g. 30-100).

The problem was that during I/O cache initialization (this happens
before each bulk insert) allocated I/O buffer was unnecessarily
initialized to '\0'.

This was happening because of mess in flag values. MyISAM informs I/O
cache to wait for free space (if out of disk space) by passing
MY_WAIT_IF_FULL flag. Since MY_WAIT_IF_FULL and MY_ZEROFILL have the
same values, memory allocator was initializing memory to '\0'.

The performance gain provided with this patch may only be visible with
non-debug binaries, since safemalloc always initializes allocated memory
to 0xA5A5...

mysys/mf_iocache.c:
  Remove MY_WAIT_IF_FULL from myflags before calling my_malloc
  to prevent conflict with MY_ZEROFILL.

This is a partial correction to the original fix for bug#37098
   Get rid of "Installed (but unpackaged)" files in the RPM build
which used a wrong variable.

man/Makefile.am:
  Correction to the original fix:
  The variable to use is "$(mandir)", "$(manlibdir)" was wrong.

The check for stack overflow was independent of the size of the 
structure stored in the heap. 
Fixed by adding sizeof(PARAM) to the requested free heap size.

view manipulations
      
The bespoke flag was not properly reset after last call to 
fill_record. Fixed by resetting in caller mysql_update.

mysql-test/r/auto_increment.result:
  Bug#46616: Test result.
mysql-test/t/auto_increment.test:
  Bug#46616: Test case.
sql/sql_update.cc:
  Bug#46616: Fix.

view that has Group By
      
Table access rights checking function check_grant() assumed
that no view is opened when it's called.
This is not true with nested views where the inner view
needs materialization. In this case the view is already 
materialized when check_grant() is called for it.
This caused check_grant() to not look for table level
grants on the materialized view table.
Fixed by checking if a view is already materialized and if 
it is check table level grants using the original table name
(not the ones of the materialized temp table).

Bug#45243: crash on win in sql thread clear_tables_to_lock() -> free()
Bug#45242: crash on win in mysql_close() -> free()
Bug#45238: rpl_slave_skip, rpl_change_master failed (lost connection) for STOP SLAVE
Bug#46030: rpl_truncate_3innodb causes server crash on windows
Bug#46014: rpl_stm_reset_slave crashes the server sporadically in pb2

When killing a user session on the server, it's necessary to
interrupt (notify) the thread associated with the session that
the connection is being killed so that the thread is woken up
if waiting for I/O. On a few platforms (Mac, Windows and HP-UX)
where the SIGNAL_WITH_VIO_CLOSE flag is defined, this interruption
procedure is to asynchronously close the underlying socket of
the connection.

In order to enable this schema, each connection serving thread
registers its VIO (I/O interface) so that other threads can
access it and close the connection. But only the owner thread of
the VIO might delete it as to guarantee that other threads won't
see freed memory (the thread unregisters the VIO before deleting
it). A side note: closing the socket introduces a harmless race
that might cause a thread attempt to read from a closed socket,
but this is deemed acceptable.

The problem is that this infrastructure was meant to only be used
by server threads, but the slave I/O thread was registering the
VIO of a mysql handle (a client API structure that represents a
connection to another server instance) as a active connection of
the thread. But under some circumstances such as network failures,
the client API might destroy the VIO associated with a handle at
will, yet the VIO wouldn't be properly unregistered. This could
lead to accesses to freed data if a thread attempted to kill a
slave I/O thread whose connection was already broken.

There was a attempt to work around this by checking whether
the socket was being interrupted, but this hack didn't work as
intended due to the aforementioned race -- attempting to read
from the socket would yield a "bad file descriptor" error.

The solution is to add a hook to the client API that is called
from the client code before the VIO of a handle is deleted.
This hook allows the slave I/O thread to detach the active vio
so it does not point to freed memory.

server-tools/instance-manager/mysql_connection.cc:
  Add stub method required for linking.
sql-common/client.c:
  Invoke hook.
sql/client_settings.h:
  Export hook.
sql/slave.cc:
  Introduce hook that clears the active VIO before it is freed
  by the client API.

        
Replication SQL thread does not set database default charset to 
thd->variables.collation_database properly, when executing LOAD DATA binlog.
This bug can be repeated by using "LOAD DATA" command in STATEMENT mode.
        
This patch adds code to find the default character set of the current database 
then assign it to thd->db_charset when slave server begins to execute a relay log.
The test of this bug is added into rpl_loaddata_charset.test 

automerge didn't handle this correctly,
so test "view" was corrected manually.

The test for the 45806 entry in our bug DB got applied twice,
in different places for the "view.test" and "view.result" files.

The fix is to simply remove the erroneous insertion.