Before this fix, the parser would sometime change where a token starts by
altering Lex_input_string::tok_start, which later confused the code in
sql_yacc.yy that needs to capture the source code of a SQL statement,
like to represent the body of a stored procedure.

This line of code in sql_lex.cc :

case MY_LEX_USER_VARIABLE_DELIMITER:
  lip->tok_start= lip->ptr; // Skip first `

would <skip the first back quote> ... and cause the bug reported.

In general, the responsibility of sql_lex.cc is to *find* where token are
in the SQL text, but is *not* to make up fake or incomplete tokens.
With a quoted label like `my_label`, the token starts on the first quote.
Extracting the token value should not change that (it did).

With this fix, the lexical analysis has been cleaned up to not change
lip->tok_start (in the case found for this bug).

The functions get_token() and get_quoted_token() now have an extra
parameters, used when some characters from the beginning of the token need
to be skipped when extracting a token value, like when extracting 'AB' from
'0xAB', for example, for a HEX_NUM token.

This exposed a bad assumption in Item_hex_string and Item_bin_string,
which has been fixed:

The assumption was that the string given, 'AB', was in fact preceded in
memory by '0x', which might be false (it can be preceded by "x'" and
followed by "'" -- or not be preceded by valid memory at all)

If a name is needed for Item_hex_string or Item_bin_string, the name is
taken from the original and true source code ('0xAB'), and assigned in
the select_item rule, instead of relying on assumptions related to how
memory is used.


mysql-test/r/sp.result:
  Lex_input_stream::tok_start must point at the real start of a token.
mysql-test/t/sp.test:
  Lex_input_stream::tok_start must point at the real start of a token.
sql/item.cc:
  Lex_input_stream::tok_start must point at the real start of a token.
sql/sql_lex.cc:
  Lex_input_stream::tok_start must point at the real start of a token.
sql/sql_yacc.yy:
  Lex_input_stream::tok_start must point at the real start of a token.

into  weblab.(none):/home/marcsql/TREE/mysql-5.0-25411_d


sql/item_func.cc:
  Auto merged
sql/log_event.cc:
  Auto merged
sql/mysql_priv.h:
  Auto merged
sql/slave.cc:
  Auto merged
sql/sp.cc:
  Auto merged
sql/sp_head.cc:
  Auto merged
sql/sql_class.cc:
  Auto merged
sql/sql_class.h:
  Auto merged
sql/sql_lex.cc:
  Auto merged
sql/sql_lex.h:
  Auto merged
sql/sql_parse.cc:
  Auto merged
sql/sql_prepare.cc:
  Auto merged
sql/sql_view.cc:
  Auto merged

The issue found with bug 25411 is due to the function skip_rear_comments()
which damages the source code while implementing a work around.
The root cause of the problem is in the lexical analyser, which does not
process special comments properly.
For special comments like :
[1] aaa /*!50000 bbb */ ccc
since 5.0 is a version older that the current code, the parser is in lining
the content of the special comment, so that the query to process is
[2] aaa bbb ccc
However, the text of the query captured when processing a stored procedure,
stored function or trigger (or event in 5.1), can be after rebuilding it:
[3] aaa bbb */ ccc
which is wrong.

To fix bug 25411 properly, the lexical analyser needs to return [2] when
in lining special comments.
In order to implement this, some preliminary cleanup is required in the code,
which is implemented by this patch.

Before this change, the structure named LEX (or st_lex) contains attributes
that belong to lexical analysis, as well as attributes that represents the
abstract syntax tree (AST) of a statement.
Creating a new LEX structure for each statements (which makes sense for the
AST part) also re-initialized the lexical analysis phase each time, which
is conceptually wrong.

With this patch, the previous st_lex structure has been split in two:
- st_lex represents the Abstract Syntax Tree for a statement. The name "lex"
has not been changed to avoid a bigger impact in the code base.
- class lex_input_stream represents the internal state of the lexical
  analyser, which by definition should *not* be reinitialized when parsing
  multiple statements from the same input stream.

This change is a pre-requisite for bug 25411, since the implementation of
lex_input_stream will later improve to deal properly with special comments,
and this processing can not be done with the current implementation of
sp_head::reset_lex and sp_head::restore_lex, which interfere with the lexer.

This change set alone does not fix bug 25411.


sql/item_func.cc:
  Refactoring, separate lex_input_stream from st_lex.
sql/log_event.cc:
  Refactoring, separate lex_input_stream from st_lex.
sql/mysql_priv.h:
  Refactoring, separate lex_input_stream from st_lex.
sql/slave.cc:
  Refactoring, separate lex_input_stream from st_lex.
sql/sp.cc:
  Refactoring, separate lex_input_stream from st_lex.
sql/sp_head.cc:
  Refactoring, separate lex_input_stream from st_lex.
sql/sp_head.h:
  Refactoring, separate lex_input_stream from st_lex.
sql/sql_class.cc:
  Refactoring, separate lex_input_stream from st_lex.
sql/sql_class.h:
  Refactoring, separate lex_input_stream from st_lex.
sql/sql_lex.cc:
  Refactoring, separate lex_input_stream from st_lex.
sql/sql_lex.h:
  Refactoring, separate lex_input_stream from st_lex.
sql/sql_parse.cc:
  Refactoring, separate lex_input_stream from st_lex.
sql/sql_prepare.cc:
  Refactoring, separate lex_input_stream from st_lex.
sql/sql_trigger.cc:
  Refactoring, separate lex_input_stream from st_lex.
sql/sql_view.cc:
  Refactoring, separate lex_input_stream from st_lex.
sql/sql_yacc.yy:
  Refactoring, separate lex_input_stream from st_lex.

into  blade09.mysql.com:/tmp/mmj/mysql-5.0-runtime

into  bk-internal.mysql.com:/data0/bk/mysql-5.0-opt


sql/item_cmpfunc.cc:
  Auto merged
sql/sql_select.cc:
  Auto merged
sql/sql_show.cc:
  Auto merged
sql/sql_view.cc:
  Auto merged

into  weblab.(none):/home/marcsql/TREE/mysql-5.0-rt-merge


mysql-test/r/sp.result:
  Auto merged
mysql-test/t/sp.test:
  Auto merged

mysql-test/r/rpl_critical_errors.result:
  New BitKeeper file ``mysql-test/r/rpl_critical_errors.result''

into  gshchepa.loc:/home/uchum/work/bk-trees/mysql-5.0-opt-27704

into  gshchepa.loc:/home/uchum/work/bk-trees/mysql-5.0-opt-27704


mysql-test/r/subselect.result:
  Auto merged
mysql-test/t/row.test:
  Auto merged
mysql-test/r/row.result:
  Test case updated for Bug#27704 (incorrect comparison
  of rows with NULL components).
sql/item_cmpfunc.cc:
  Bug#27704: incorrect comparison of rows with NULL components.
sql/item_cmpfunc.h:
  Bug#27704: incorrect comparison of rows with NULL components.
  Cosmetic fix.

Support for NULL components was incomplete for row comparison,
fixed.  Added support for abort_on_null at compare_row() like
in 5.x


sql/item_cmpfunc.h:
  Bug#27704: incorrect comparison of rows with NULL components
  Added support for abort_on_null at Item_bool_func2
  like in 5.x
sql/item_cmpfunc.cc:
  Bug#27704: incorrect comparison of rows with NULL components
  Support for NULL components was incomplete for row comparison,
  fixed. Added support for abort_on_null at compare_row() like
  in 5.x
mysql-test/t/row.test:
  Test case updated for Bug#27704 (incorrect comparison 
  of rows with NULL components)
mysql-test/r/row.result:
  Test case updated for Bug#27704 (incorrect comparison 
  of rows with NULL components)
mysql-test/r/subselect.result:
  Test case updated for Bug#27704 (incorrect comparison 
  of rows with NULL components)

into  magare.gmz:/home/kgeorge/mysql/autopush/B27786-5.0-opt

This bug was intruduced by the fix for bug#17212 (in 4.1). It is not 
ok to call test_if_skip_sort_order since this function will 
alter the execution plan. By contract it is not ok to call 
test_if_skip_sort_order in this context.

This bug appears only in the case when the optimizer has chosen 
an index for accessing a particular table but finds a covering 
index that enables it to skip ORDER BY. This happens in 
test_if_skip_sort_order.


mysql-test/r/key.result:
  Bug#24778
  
  test case.
  
  The bug causes the result to be the empty set.
mysql-test/t/key.test:
  Bug#24778
  
  The minimal test case that reveals the bug. The reason for such a 
  complicated schema is that we have to convince the optimizer to 
  pick one index, then discard it in order to be able to skip 
  ORDER BY.
sql/sql_select.cc:
  bug#24778
  
  Removed the call to test_if_skip_sort_order that constituted the
  bug.

into  romeo.(none):/home/bk/merge-mysql-5.0


client/mysql.cc:
  Auto merged
sql/mysql_priv.h:
  Auto merged
sql/mysqld.cc:
  Auto merged
sql/set_var.cc:
  Auto merged
sql/slave.cc:
  Auto merged
sql/sql_class.cc:
  Auto merged

into  romeo.(none):/home/bkroot/mysql-5.0-rpl

When merging views into the enclosing statement
the ORDER BY clause of the view is merged to the
parent's ORDER BY clause.
However when the VIEW is merged into an UNION
branch the ORDER BY should be ignored. 
Use of ORDER BY for individual SELECT statements
implies nothing about the order in which the rows
appear in the final result because UNION by default
produces unordered set of rows.
Fixed by ignoring the ORDER BY clause from the merge
view when expanded in an UNION branch.


mysql-test/r/view.result:
  Bug #27786: test case
mysql-test/t/view.test:
  Bug #27786: test case
sql/sql_lex.h:
  Bug #27786: add a is_union() inlined function
  Returns true if the unit represents an UNION.
sql/sql_view.cc:
  Bug #27786: ignore ORDER BY in mergeable views when in UNION context

into  magare.gmz:/home/kgeorge/mysql/autopush/B27530-5.0-opt

They can drop table after table names list creation and before table opening.
We open non existing table and get ER_NO_SUCH_TABLE error.
In this case we do not store the record into I_S table and clear error.


sql/sql_show.cc:
  Hide error for not existing table

conditions when executing an equijoin query with WHERE condition
containing a subquery predicate of the form join_attr NOT IN (SELECT ...).

To resolve a problem of the correct evaluation of the expression
  attr NOT IN (SELECT ...)
an array of guards is created to make it possible to filter out some 
predicates of the EXISTS subquery into which the original subquery 
predicate is transformed, in the cases when a takes the NULL value. 
If attr is defined as a field that cannot be NULL than such an array 
is not needed and is not created. 
However if the field a occurred also an an equijoin predicate t2.a=t1.b
and table t1 is accessed before table t2 then it may happen that the 
the EXISTS subquery is pushed down to the condition evaluated just after
table t1 has been accessed. In this case any occurrence of t2.a is 
substituted for t1.b. When t1.b takes the value of NULL an attempt is 
made to turn on the corresponding guard. This action caused a crash as 
no guard array had been created.

Now the code of Item_in_subselect::set_cond_guard_var checks that the guard
array has been created before setting a guard variable on. Otherwise the
method does nothing. It cannot results in returning a row that could be
rejected as the condition t2.a=t1.b will be checked later anyway.        


mysql-test/r/subselect3.result:
  Added a test case for bug #27870.
mysql-test/t/subselect3.test:
  Added a test case for bug #27870.
sql/item_subselect.h:
  Fixed bug #27870. The bug that causes crashes manifests itself at some
  conditions when executing an equijoin query with WHERE condition
  containing a subquery predicate of the form join_attr NOT IN (SELECT ...).
  
  Forced Item_in_subselect::set_cond_guard_var to check that the guard
  array has been created before setting a guard variable on. Otherwise the
  method does nothing.

into  bk-internal.mysql.com:/data0/bk/mysql-5.0-marvel


sql/ha_ndbcluster.cc:
  Auto merged
sql/item.cc:
  Auto merged
sql/item_cmpfunc.cc:
  Auto merged
sql/sql_delete.cc:
  Auto merged
sql/sql_select.cc:
  Auto merged
sql/sql_table.cc:
  Auto merged

into  weblab.(none):/home/marcsql/TREE/mysql-5.0-rt-merge

  Fix warning after fix for bug#27321.


sql/item.cc:
  Fix warning after fix for bug#27321.

  After merge fix.


mysql-test/r/subselect.result:
  After merge fix.
mysql-test/t/subselect.test:
  After merge fix.

into  moonbone.local:/mnt/gentoo64/work/27321-bug-5.0-opt-mysql


mysql-test/r/subselect3.result:
  Auto merged
sql/item.cc:
  Auto merged
sql/item.h:
  Auto merged
sql/sql_select.cc:
  Auto merged
mysql-test/r/subselect.result:
  Manually merged
mysql-test/t/subselect.test:
  Manually merged
mysql-test/t/subselect3.test:
  Manually merged

The Item_outer_ref class based on the Item_direct_ref class was always used
to represent an outer field. But if the outer select is a grouping one and the 
outer field isn't under an aggregate function which is aggregated in that
outer select an Item_ref object should be used to represent such a field.
If the outer select in which the outer field is resolved isn't grouping then
the Item_field class should be used to represent such a field.
This logic also should be used for an outer field resolved through its alias
name.

Now the Item_field::fix_outer_field() uses Item_outer_field objects to
represent aliased and non-aliased outer fields for grouping outer selects
only.
Now the fix_inner_refs() function chooses which class to use to access outer
field - the Item_ref or the Item_direct_ref. An object of the chosen class
substitutes the original field in the Item_outer_ref object.
The direct_ref and the found_in_select_list fields were added to the
Item_outer_ref class.


mysql-test/t/subselect3.test:
  Some test cases were corrected after the fix for the bug#27321.
mysql-test/r/subselect3.result:
  Some test cases were corrected after the fix for the bug#27321.
mysql-test/t/subselect.test:
  Added a test case for the bug#27321: Wrong subquery result in a grouping select.
mysql-test/r/subselect.result:
  Added a test case for the bug#27321: Wrong subquery result in a grouping select.
  Some test cases were corrected after this fix.
sql/sql_union.cc:
  Bug#27321: Wrong subquery result in a grouping select.
  Cleanup of the inner_refs_list.
sql/sql_select.cc:
  Bug#27321: Wrong subquery result in a grouping select.
  Now the fix_inner_refs() function chooses which class to use to access outer
  field - the Item_ref or the Item_direct_ref. An object of the chosen class
  substitutes the original field in the Item_outer_ref object.
  A comment is corrected.
sql/item.cc:
  Bug#27321: Wrong subquery result in a grouping select.
  Now the Item_field::fix_outer_field() uses Item_outer_field objects to
  represent aliased and non-aliased outer fields for grouping outer selects
  only.
sql/item.h:
  Bug#27321: Wrong subquery result in a grouping select.
  The direct_ref and the found_in_select_list fields were added to the
  Item_outer_ref class.

into  vajra.(none):/opt/local/work/mysql-5.0-runtime

into  mysql.com:/d2/hf/mrg/mysql-5.0-opt

into  mysql.com:/d2/hf/mrg/mysql-5.0-opt

into  mysql.com:/d2/hf/mrg/mysql-4.1-opt

into  whalegate.ndb.mysql.com:/home/tomas/mysql-5.0-opt

into  a88-113-38-195.elisa-laajakaista.fi:/home/my/new/mysql-5.0-marvel


heap/hp_write.c:
  Auto merged
sql/item_cmpfunc.cc:
  Auto merged
sql-common/my_time.c:
  Auto merged
sql/mysqld.cc:
  Auto merged
sql/opt_range.cc:
  Auto merged
sql/table.cc:
  Auto merged

into  whalegate.ndb.mysql.com:/home/tomas/mysql-5.0-opt

into  a88-113-38-195.elisa-laajakaista.fi:/home/my/new/mysql-5.0-marvel

into  a88-113-38-195.elisa-laajakaista.fi:/home/my/new/mysql-4.1-main

into  mysql.com:/home/bar/mysql-5.0.b20095

Problem: single byte do_varstring1() function was called, which didn't
check limit on "number of character", and checked only "number of bytes".
Fix: adding a multi-byte aware function do_varstring1_mb(),
to limit on "number of characters"


mysql-test/r/ctype_utf8.result:
  Adding test case
mysql-test/t/ctype_utf8.test:
  Adding test case
sql/field_conv.cc:
  Adding missing function to copy VARCHAR strings
  having one length byte.

- autoincrement column of size medium int not handled in ndb_restore
- added testcase also for other types, tiny, small... etc

into  gshchepa.localdomain:/home/uchum/work/bk-trees/5507-ci

into  whalegate.ndb.mysql.com:/home/tomas/mysql-5.0-opt