Result File Correction For Bug Fix

Problem: a byte behind the end of input string was read
in case of a broken XML not having a quote or doublequote
character closing a string value.

Fix: changing condition not to read behind the end of input string

  @ mysql-test/r/xml.result
  @ mysql-test/t/xml.test
  Adding tests

  @ strings/xml.c
  When checking if the closing quote/doublequote was found,
  using p->cur[0] us unsafe, as p->cur can point to the byte after the value.
  Comparing p->cur to p->beg instead.

and compressed tables

buf_LRU_drop_page_hash_for_tablespace(): after releasing and
reacquiring the buffer pool mutex, do not dereference any block
descriptor pointer that is not known to be a pointer to an
uncompressed page frame (type buf_block_t; state ==
BUF_BLOCK_FILE_PAGE). Also, defer the acquisition of the block_mutex
until it is needed.

buf_page_get_gen(): Add mode == BUF_GET_IF_IN_POOL_PEEK for
buffer-fixing a block without making it young in the LRU list.

buf_page_get_gen(), buf_page_init(), buf_LRU_block_remove_hashed_page():
Set bpage->state = BUF_BLOCK_ZIP_FREE before buf_buddy_free(bpage),
so that similar race conditions might be detected a little easier.

btr_search_drop_page_hash_when_freed(): Use BUF_GET_IF_IN_POOL_PEEK
when dropping the hash indexes.

rb://528 approved by Jimmy Yang

IN OVERFLOW

Do not assign the result of the difference to a signed variable and
checking whether it is negative afterwards because this limits the max diff
to 2G on 32 bit systems. E.g. "signed = 3.5G - 1G" would be negative and the
code would assume that 3.5G < 1G. Instead compare the two variables directly
and assign to unsigned only if we know that the result of the subtraction
will be positive.

Discussed with:	Jimmy and Sunny (via IRC)

               ("-") IN DATABASE NAMES IN ALTER DATABASE.

mysqldump did not quote database name in 'ALTER DATABASE'
statements in its output. This can further cause a failure
while loading if database name contains a hyphen '-'.

This happened as, while printing the 'ALTER DATABASE'
statements, the database name was not quoted.

Fixed by quoting the database name.


client/mysqldump.c:
  Bug#11766310 : 59398: MYSQLDUMP 5.1 CAN'T HANDLE A DASH
                 ("-") IN DATABASE NAMES IN ALTER DATABASE.
  
  Modified the print statement in order to print the quoted
  database name for 'ALTER DATABASE' statements.
mysql-test/r/mysqldump.result:
  Added a test case for bug#11766310.
mysql-test/t/mysqldump.test:
  Added a test case for bug#11766310.

A separate fix for 5.1 (as 5.1 and 5.5 have seriously
differged in the related pieces of the code).
A patch for 5.5 was approved earlier.

Problem: ucs2 was correctly disallowed in "SET NAMES" only,
while mysql_real_connect() and mysql_change_user() still allowed
to use ucs2, which made server crash.

Fix: disallow ucs2 in mysql_real_connect() and mysql_change_user().

  @ sql/sql_priv.h
    - changing return type for thd_init_client_charset() to bool,
      to return errors to the caller

  @ sql/sql_var.cc
    - using new function

  @ sql/sql_connect.cc
    - thd_client_charset_init:
      in case of unsupported client character set send error and return true;
      in case of success return false
    - check_connection:
      Return error if character set initialization failed

  @ sql/sql_parse.cc
    - check charset in the very beginnig of the CMD_CHANGE_USER handling code

  @ tests/mysql_client_test.c
    - adding tests

The loop that was looping over subqueries' references to outer field used a
local boolean variable to tell whether the field was grouped or not. But the
implementor failed to reset the variable after each iteration. Thus a field
that was not directly aggregated appeared to be.

Fixed by resetting the variable upon each new iteration.

Thanks to Kristian Nielsen for finding out the root cause for the
failure, see: https://bugs.launchpad.net/maria/+bug/677407

          memory reference

There are two issues present here.
  1) There is a possibility that we test a byte beyond the
     allocated buffer

  2) We compare a byte that might never have been
     initalized to see if it's 0.

The first issue is not triggered by existing code, but an
ASSERT has been added to safe-guard against introducing
new code that triggers it.

The second issue is what triggers the Valgrind warnings
reported in the bug report. A buffer is allocated in
class String to hold the value. This buffer is populated
by the character data constituting the string, but is not
zero-terminated in most cases.  Testing if it is indeed
zero-terminated means that we check a byte that has never
been explicitly set, thus causing Valgrind to trigger.

Note that issue 2 is not a serious problem.  The variable
is read, and if it's not zero, we will set it to zero.
There are no further consequences.

Note that this patch does not fix the underlying problems
with issue 1, as it is deemed too risky to fix at this
point (as noted in the bug report).  As discussed in
the report, the c_ptr() method should probably be
replaced, but this requires a thorough analysis of the
~200 calls to the method.


sql/set_var.cc:
  These two cases have been reported to fail
  with Valgrind.

Assertion `bitmap_is_set_all(&table->s->all_set)' failed in
handler::ha_reset

This followup fixes the compilation warning
  'test_bit' may be used uninitialized in this function
introduced by the previous patch.

Assertion `bitmap_is_set_all(&table->s->all_set)' failed in
handler::ha_reset

This assertion could be triggered if two connections simultaneously
executed two bitmap test functions on the same bitmap. For example,
the assertion could be triggered if one connection executed UPDATE
while a second connection executed SELECT on the same table.

Even if bitmap test functions have read-only semantics and have
const bitmaps as parameter, several of them modified the internal
state of the bitmap. With interleaved execution of two such functions
it was possible for one function to modify the state of the same
bitmap that the other function had just modified. This lead to an
inconsistent state and could trigger the assert.

Internally the bitmap uses 32 bit words for storage. Since bitmaps
can contain any number of bits, the last word in the bitmap may
not be fully used. A 32 bit mask is maintained where a bit is set
if the corresponding bit in the last bitmap word is unused.
The problem was that several test functions applied this mask to
the last word. Sometimes the mask was negated and used to zero out
the remainder of the last word and sometimes the mask was used as-is
to fill the remainder of the last word with 1's. This meant that if
a function first used the negated mask and another function then
used the mask as-is (or vice-versa), the first function would then
get the wrong result.

This patch fixes the problem by changing the implementation of
9 bitmap functions that modified the bitmap state even if the 
bitmap was declared const. These functions now preserve the
internal state of the bitmap. This makes it possible for
two connections to concurrently execute two of these functions
on the same bitmap without issues.

The patch also removes dead testing code from my_bitmap.c.
These tests have already been moved to unittest/mysys/bitmap-t.c.
Existing test coverage of my_bitmap has been extended.

No MTR test case added as this would require adding several sync
points to the bitmap functions. The patch has been tested with
a non-deterministic test case posted on the bug report.


include/my_bit.h:
  Removed my_count_bits_ushort() which is not needed anymore.
  Added my_count_bits_uint32().
unittest/mysys/bitmap-t.c:
  Extended test coverage of my_bitmap.

attempt to create spatial index on char > 31 bytes".

Attempt to create spatial index on char field with length
greater than 31 byte led to assertion failure on server
compiled with safemutex support.

The problem occurred in mi_create() function which was called
to create a new version of table being altered. This function
failed since it detected an attempt to create a spatial key
on non-binary column and tried to return an error.
On its error path it tried to unlock THR_LOCK_myisam mutex
which has not been not locked at this point. Indeed such an
incorrect behavior was caught by safemutex wrapper and caused
assertion failure.

This patch fixes the problem by ensuring that mi_create()
doesn't releases THR_LOCK_myisam mutex on error path if it was
not acquired.

mysql-test/r/gis.result:
  Added test for bug @59888 "debug assertion when attempt to
  create spatial index on char > 31 bytes".
mysql-test/t/gis.test:
  Added test for bug @59888 "debug assertion when attempt to
  create spatial index on char > 31 bytes".
storage/myisam/mi_create.c:
  Changed mi_create() not to release THR_LOCK_myisam mutex on
  error path if it was not acquired.

rw_lock_create_func(): Initialize lock->writer_thread, so that Valgrind
will not complain even when Valgrind instrumentation is not enabled.
Flag lock->writer_thread uninitialized, so that Valgrind can complain
when it is used uninitialized.

rw_lock_set_writer_id_and_recursion_flag(): Revert the bogus Valgrind
instrumentation that was pushed in the first attempt to fix this bug.

index gives failures.

Approved by Marko

by silencing a bogus Valgrind warning:

==4392== Conditional jump or move depends on uninitialised value(s)
==4392==    at 0x5A18416: rw_lock_set_writer_id_and_recursion_flag (sync0rw.ic:283)
==4392==    by 0x5A1865C: rw_lock_x_lock_low (sync0rw.c:558)
==4392==    by 0x5A18481: rw_lock_x_lock_func (sync0rw.c:617)
==4392==    by 0x597EEE6: mtr_x_lock_func (mtr0mtr.ic:271)
==4392==    by 0x597EBBD: fsp_header_init (fsp0fsp.c:970)
==4392==    by 0x5A15E78: innobase_start_or_create_for_mysql (srv0start.c:1508)
==4392==    by 0x598B789: innobase_init(void*) (ha_innodb.cc:2282)

os_compare_and_swap_thread_id() is defined as
__sync_bool_compare_and_swap(). From the GCC doc:

`bool __sync_bool_compare_and_swap (TYPE *ptr, TYPE oldval TYPE newval, ...)'
  ...
  The "bool" version returns true if the comparison is successful and
  NEWVAL was written.

So it is not possible that the return value is uninitialized, no matter what
the arguments to os_compare_and_swap_thread_id() are. Probably Valgrind gets
confused by the implementation of the GCC internal function
__sync_bool_compare_and_swap().

InnoDB Plugin 1.0.15 has been released with MySQL 5.1.55.

> ------------------------------------------------------------
> revno: 3545
> revision-id: holyfoot@mysql.com-20110112130241-50lwmhq562otxw31
> parent: dmitry.lenev@oracle.com-20110112130830-csanweanbny2ua3n
> committer: Alexey Botchkov <holyfoot@mysql.com>
> branch nick: 51mrg
> timestamp: Wed 2011-01-12 17:02:41 +0400
> message:
>   Bug #57321 crashes and valgrind errors from spatial types 
>           Item_func_spatial_collection::fix_length_and_dec didn't call parent's method, so
>           the maybe_null was set to '0' after it. But in this case the result was
>           just NULL, that caused wrong behaviour.
>   
>   per-file comments:
>     mysql-test/r/gis.result
>   Bug #57321 crashes and valgrind errors from spatial types 
>           test result updated.
>   
>     mysql-test/t/gis.test
>   Bug #57321 crashes and valgrind errors from spatial types 
>           test case added.
>     sql/item_geofunc.h
>   Bug #57321 crashes and valgrind errors from spatial types 
>           Item_func_geometry::fix_length_and_dec() called in
>           Item_func_spatial_collection::fix_length_and_dec().

> ------------------------------------------------------------
> revno: 3537
> revision-id: davi.arnaut@oracle.com-20110107192806-rmvvxwhk10sy0219
> parent: davi.arnaut@oracle.com-20110107183336-kp8niwm2hz3wb4c3
> committer: Davi Arnaut <davi.arnaut@oracle.com>
> branch nick: mysql-5.1
> timestamp: Fri 2011-01-07 17:28:06 -0200
> message:
>   Bug#51023: Mysql server crashes on SIGHUP and destroys InnoDB files
>   
>   WIN32 compilation fixes: define ETIMEDOUT only if not available and
>   fix typos and add a missing parameter.

> ------------------------------------------------------------
> revno: 3536
> revision-id: davi.arnaut@oracle.com-20110107183336-kp8niwm2hz3wb4c3
> parent: saikumar.v@sun.com-20110106103945-rhsek9uy6f63db44
> committer: Davi Arnaut <davi.arnaut@oracle.com>
> branch nick: 51023-5.1
> timestamp: Fri 2011-01-07 16:33:36 -0200
> message:
>   Bug#51023: Mysql server crashes on SIGHUP and destroys InnoDB files
>   
>   From a user perspective, the problem is that a FLUSH LOGS or SIGHUP
>   signal could end up associating the stdout and stderr to random
>   files. In the case of this bug report, the streams would end up
>   associated to InnoDB ibd files.
>   
>   The freopen(3) function is not thread-safe on FreeBSD. What this
>   means is that if another thread calls open(2) during freopen()
>   is executing that another thread's fd returned by open(2) may get
>   re-associated with the file being passed to freopen(3). See FreeBSD
>   PR number 79887 for reference:
>   
>     http://www.freebsd.org/cgi/query-pr.cgi?pr=79887
>   
>   This problem is worked around by substituting a internal hook within
>   the FILE structure. This avoids the loss of atomicity by not having
>   the original fd closed before its duplicated.
>   
>   Patch based on the original work by Vasil Dimov.

> ------------------------------------------------------------
> revno: 3527.3.2
> revision-id: mattias.jonsson@oracle.com-20101222144517-cbv3l5jlbd0mq5s8
> parent: mattias.jonsson@oracle.com-20101222095036-2lpx0gqu4i45jtkz
> committer: Mattias Jonsson <mattias.jonsson@oracle.com>
> branch nick: b54483-51-bt_2
> timestamp: Wed 2010-12-22 15:45:17 +0100
> message:
>   Bug#54483: valgrind errors when making warnings for
>   multiline inserts into partition
>   Bug#57071: EXTRACT(WEEK from date_col) cannot be
>   allowed as partitioning function
>   
>   Renamed function according to reviewers comments.

> ------------------------------------------------------------
> revno: 3527.3.1
> revision-id: mattias.jonsson@oracle.com-20101222095036-2lpx0gqu4i45jtkz
> parent: sven.sandberg@oracle.com-20101220090735-psae11j9nwuj8vzl
> committer: Mattias Jonsson <mattias.jonsson@oracle.com>
> branch nick: b54483-51-bt_2
> timestamp: Wed 2010-12-22 10:50:36 +0100
> message:
>   Bug#54483: valgrind errors when making warnings for multiline inserts into partition
>   Bug#57071: EXTRACT(WEEK from date_col) cannot be allowed as partitioning function
>   
>   There were functions allowed as partitioning functions
>   that implicit allowed cast. That could result in unacceptable
>   behaviour.
>   
>   Solution was to check that the arguments of date and time functions
>   have allowed types (field and date/datetime/time depending on function).

> ------------------------------------------------------------
> revno: 3520
> revision-id: sergey.glukhov@oracle.com-20101214093303-wmo9mqcb8rz0wv9f
> parent: tor.didriksen@oracle.com-20101213161301-81lprlbune7r98dl
> committer: Sergey Glukhov <sergey.glukhov@oracle.com>
> branch nick: mysql-5.1-bugteam
> timestamp: Tue 2010-12-14 12:33:03 +0300
> message:
>   Fixed following problems:
>   --Bug#52157 various crashes and assertions with multi-table update, stored function
>   --Bug#54475 improper error handling causes cascading crashing failures in innodb/ndb
>   --Bug#57703 create view cause Assertion failed: 0, file .\item_subselect.cc, line 846
>   --Bug#57352 valgrind warnings when creating view
>   --Recently discovered problem when a nested materialized derived table is used
>     before being populated and it leads to incorrect result
>   
>   We have several modes when we should disable subquery evaluation.
>   The reasons for disabling are different. It could be
>   uselessness of the evaluation as in case of 'CREATE VIEW'
>   or 'PREPARE stmt', or we should disable subquery evaluation
>   if tables are not locked yet as it happens in bug#54475, or
>   too early evaluation of subqueries can lead to wrong result
>   as it happened in Bug#19077.
>   Main problem is that if subquery items are treated as const
>   they are evaluated in ::fix_fields(), ::fix_length_and_dec()
>   of the parental items as a lot of these methods have
>   Item::val_...() calls inside.
>   We have to make subqueries non-const to prevent unnecessary
>   subquery evaluation. At the moment we have different methods
>   for this. Here is a list of these modes:
>   
>   1. PREPARE stmt;
>   We use UNCACHEABLE_PREPARE flag.
>   It is set during parsing in sql_parse.cc, mysql_new_select() for
>   each SELECT_LEX object and cleared at the end of PREPARE in
>   sql_prepare.cc, init_stmt_after_parse(). If this flag is set
>   subquery becomes non-const and evaluation does not happen.
>   
>   2. CREATE|ALTER VIEW, SHOW CREATE VIEW, I_S tables which
>      process FRM files
>   We use LEX::view_prepare_mode field. We set it before
>   view preparation and check this flag in
>   ::fix_fields(), ::fix_length_and_dec().
>   Some bugs are fixed using this approach,
>   some are not(Bug#57352, Bug#57703). The problem here is
>   that we have a lot of ::fix_fields(), ::fix_length_and_dec()
>   where we use Item::val_...() calls for const items.
>   
>   3. Derived tables with subquery = wrong result(Bug19077)
>   The reason of this bug is too early subquery evaluation.
>   It was fixed by adding Item::with_subselect field
>   The check of this field in appropriate places prevents
>   const item evaluation if the item have subquery.
>   The fix for Bug19077 fixes only the problem with
>   convert_constant_item() function and does not cover
>   other places(::fix_fields(), ::fix_length_and_dec() again)
>   where subqueries could be evaluated.
>   
>   Example:
>   CREATE TABLE t1 (i INT, j BIGINT);
>   INSERT INTO t1 VALUES (1, 2), (2, 2), (3, 2);
>   SELECT * FROM (SELECT MIN(i) FROM t1
>   WHERE j = SUBSTRING('12', (SELECT * FROM (SELECT MIN(j) FROM t1) t2))) t3;
>   DROP TABLE t1;
>   
>   4. Derived tables with subquery where subquery
>      is evaluated before table locking(Bug#54475, Bug#52157)
>   
>   Suggested solution is following:
>   
>   -Introduce new field LEX::context_analysis_only with the following
>    possible flags:
>    #define CONTEXT_ANALYSIS_ONLY_PREPARE 1
>    #define CONTEXT_ANALYSIS_ONLY_VIEW    2
>    #define CONTEXT_ANALYSIS_ONLY_DERIVED 4
>   -Set/clean these flags when we perform
>    context analysis operation
>   -Item_subselect::const_item() returns
>    result depending on LEX::context_analysis_only.
>    If context_analysis_only is set then we return
>    FALSE that means that subquery is non-const.
>    As all subquery types are wrapped by Item_subselect
>    it allow as to make subquery non-const when
>    it's necessary.

> ------------------------------------------------------------
> revno: 3518
> revision-id: sergey.glukhov@oracle.com-20101213114812-kaq7sh0s623128lv
> parent: sergey.glukhov@oracle.com-20101213103926-okypkn10adeeyns8
> committer: Sergey Glukhov <sergey.glukhov@oracle.com>
> branch nick: mysql-5.1-bugteam
> timestamp: Mon 2010-12-13 14:48:12 +0300
> message:
>   Bug#39828 : Autoinc wraps around when offset and increment > 1
>   Auto increment value wraps when performing a bulk insert with
>   auto_increment_increment and auto_increment_offset greater than
>   one.
>   The fix:
>   If overflow happened then return MAX_ULONGLONG value as an
>   indication of overflow and check this before storing the
>   value into the field in update_auto_increment().