CERTAIN MAX_HEAP_TABLE_SIZE VALUES

Followup patch to fix failure on Window machine.

BACKGROUND:
This bug is a followup on Bug#16368875.
The assertion failure happens because in SQL layer the key
does not get promoted to PRIMARY KEY but InnoDB takes it
as PRIMARY KEY.

ANALYSIS:
Here we are trying to create an index on POINT (GEOMETRY)
data type which is a type of BLOB (since GEOMETRY is a
subclass of BLOB).
In general, we can't create an index over GEOMETRY family
type field unless we specify the length of the
keypart (similar to BLOB fields).
Only exception is the POINT field type. The POINT column
max size is 25. The problem is that the field is not treated
as PRIMARY KEY when we create a index on POINT column using
its max column size as key part prefix. The fix would allow
index on POINT column to be treated as PRIMARY KEY.

FIX:
Patch for Bug#16368875 is extended to take into account
GEOMETRY datatype, POINT in particular to consider it
as PRIMARY KEY in SQL layer.

Fix:
---
The issue reported is same as the BUG#14117018.
Hence backporting the patch from mysql-trunk
to mysql-5.5 and mysql-5.6

Bug#16415173 CRLF INSTEAD OF LF IN SQL-BENCH SCRIPTS
      
Correct perms and converts from Windows style to UNIX style line endings on some files.
Fix perms on installed ini files.

(MySQL 5.5 version)

                WITH CERTAIN MAX_HEAP_TABLE_SIZE VALUES

Description:
When the  system variable 'max_heap_table_size'
is set to 20GB, the server crashes on creation of a
temporary tables or tables using MEMORY storage engine.

Analysis:
The variable 'max_record' determines the amount heap
allocated for the records of the table. This value
is determined using the 'max_heap_table_size' variable.
'records_in_block' in turn uses the max_records to
determine the number of records per block.

When the 'max_heap_table_size' is set to 20GB, then
the 'records_in_block' is calculated to a value of
2^28.

The size of the block determined by multiplying the
'records_in_block' and 'recbuffer' results in overflow
and hence the value becomes zero. As a result, zero bytes
of the heap is allocated for the table. This will
result in a server crash when the table is accessed.

Fix:
The variables 'records_in_block' and 'recbuffer' are
typecasted to 'unsigned long' while calculating the
size of the block.

                                        PRIMARY_KEY_NO == 0 

This bug is a backport of the following revision of 5.6 source tree:
# committer: Gopal Shankar <gopal.shankar@oracle.com>
# branch nick: priKey56
# timestamp: Wed 2013-05-29 11:11:46 +0530
# message:
#   Bug#16368875 INNODB: FAILING ASSERTION:

Set CMP0026 and CMP0045 policies when using CMake 
version 3 or higher to restore old CMake behavior.

              CORRUPTS FRM

Analysis:
---------
ALTER TABLE on a partitioned table resulted in the wrong
engine being written into the table's FRM file and displayed
in SHOW CREATE TABLE.

The prep_alter_part_table() modifies the partition_info object
for TABLE instance representing the old version of table.
If the ALTER TABLE ENGINE statement fails, the partition_info
object for the TABLE contains the altered storage engine name.
The SHOW CREATE TABLE uses the TABLE object to display the table
information, hence displays incorrect storage engine for the table.
Also a subsequent successful ALTER TABLE operation will write the
incorrect engine information into the FRM file.

Fix:
---
A copy of the partition_info object is created before modification so
that any changes would not cause the the original partition_info object
to be modified if the ALTER TABLE fails.(Backported part of the code
provided as fix for bug#14156617 in mysql-5.6.6).

Backport of the fix:

: Bug 18017820: BISON 3 BREAKS MYSQL BUILD
: ========================================    
: 
: The source of the reported problem is a removal of a few deprecated
: things from Bison 3.x: 
: * YYPARSE_PARAM macro (use the %parse-param bison directive instead),
: * YYLEX_PARAM macro (use %lex-param instead),
: 
: The fix removes obsolete macro calls and introduces use of
: %parse-param and %lex-param directives.

This is the 5.5/5.6 version of the patch.

Add deprecation warning for timed_mutexes.

             DISCONNECT CON1 AND CON2
  
Problem:
The test suite/binlog/t/binlog_killed.test makes the connections
con1 and con2 but forgets to disconnect them + wait till that
operation is finished at test end.
This mistake has the potential to harm subsequent tests in
case these tests depend on the content of the processlist.
 
Solution:
Added disconnect + wait_until_disconnected.inc 
within the test cleanup.

No commit message

NON-EXISTS RECORDS

Problem:
========
In RBR replication, master deletes a record but the record
don't exist on slave. when slave tries to apply the
Delete_row_log_event from master, it will result in an
assert on slave.

Analysis:
========
This problem exists not only with Delete_rows event but also
with Update_rows event as well. Trying to update a non
existing row on the slave from the master will cause the
same assert.  This assert occurs only for the tables that
doesn't have primary keys and which basically require
sequential scan to be done to locate a record. This bug
occurs only with innodb engine not with myisam.

When update or delete rows is executed on a slave on a table
which doesn't have primary key the updated record is stored
in a buffer named table->record[0] and the same is copied to
table->record[1] so that during sequential scan
table->record[0] can reloaded with fetched data from the
table and compared against table->record[1].  In a special
case where there is no record on the slave side scan will
result in EOF in that case we reinit the scan and we try to
compare record[0]  with record[1] which are basically the
same. This comparison is incorrect. Since they both are the
same record_compare() will report that record is found and
we try to go ahead and try to update/delete non existing
row. Ideally if the scan results in EOF means no data found
hence no need to do a record_compare() at all.

Fix:
===
Avoid comparision of records on EOF.

sql/log_event.cc:
  Avoid record comparison on end of file.
sql/log_event_old.cc:
  Avoid record comparison on end of file.

SLOW/CRASHES SEMAPHORE

Problem:

There are 2 lakh tables - fk_000001, fk_000002 ... fk_200000.  All of them
are related to the same parent_table through a foreign key constraint.
When the parent_table is loaded into the dictionary cache, all the child table
will also be loaded.  This is taking lot of time.  Since this operation happens
when the dictionary latch is taken, the scenario leads to "long semaphore wait"
situation and the server gets killed.

Analysis:

A simple performance analysis showed that the slowness is because of the
dict_foreign_find() function.  It does a linear search on two linked list
table->foreign_list and table->referenced_list, looking for a particular
foreign key object based on foreign->id as the key.  This is called two
times for each foreign key object.

Solution:

Introduce a rb tree in table->foreign_rbt and table->referenced_rbt, which
are some sort of index on table->foreign_list and table->referenced_list
respectively, using foreign->id as the key.  These rbt structures will be
solely used by dict_foreign_find().  

rb#5599 approved by Vasil

For charsets with no binary collation: use my_charset_bin.

                                IN
                                SSL_CTX_LOAD_VERIFY_
                                LOCATIONS()
                                and
                                OFF-BY-ONE PROBLEM IN
                                VOID CERTDECODER::
                                GETDATE(DATETYPE DT)
                                IN ASN.CPP

Description : Fixes corner cases in yassl code.
              Refer to bug page for details.

Item_func_ltrim::val_str did not handle multibyte charsets.
Fix: factor out common code for Item_func_trim and Item_func_ltrim.

              MYSQLADMIN IN PROCESSES LIST

Description: Checking the process status (with ps -ef)  
while executing "mysqladmin" with old password and new 
password via command-line will show the new password in the
process list sporadically.

Analysis: The old password is being masked by "mysqladmin".
So masking the new password in the similar manner would 
reduce hitting the bug. But this would not completely fix
the bug, because if "ps -ef " command hits the mysqladmin
before it masks the passwords it will show both the old and
new passwords in the process list. But the chances of
hitting this is very less.

Fix: The new password also masked in the similar manner
that of the --password argument.

Problem:
Load_log_event::print_query() function does not put escape character in file name 
for "LOAD DATA INFILE" statement.

Analysis:
When we have "'" in our file name for "LOAD DATA INFILE" statement,
Load_log_event::print_query() function does not put escape character 
in our file name.

This one result that when we show binary-log, we get file name without 
escape character.

Solution:
To put escape character when we have "'" in file name, for this instead of using 
simple memcpy() to put file-name, we will use pretty_print_str().

               "HAVING SUM(DISTINCT)": WRONG RESULTS.
ISSUE:
------
If a query uses loose index scan and it has both
AGG(DISTINCT) and MIN()/MAX()functions. Then, result values
of MIN/MAX() is set improperly.
When query has AGG(DISTINCT) then end_select is set to
end_send_group. "end_send_group" keeps doing aggregation
until it sees a record from next group. And, then it will
send out the result row of that group.
Since query also has MIN()/MAX() and loose index scan is
used, values of MIN/MAX() are set as part of loose index
scan itself. Setting MIN()/MAX() values as part of loose
index scan overwrites values computed in end_send_group.
This caused invalid result.
For such queries to work loose index scan should stop
performing MIN/MAX() aggregation. And, let end_send_group to
do the same. But according to current design loose index
scan can produce only one row per group key. If we have both
MIN() and MAX() then it has to give two records out. This is
not possible as interface has to use common buffer
record[0]! for both records at a time.

SOLUTIONS:
----------
For such queries to work we need a new interface for loose
index scan. Hence, do not choose loose_index_scan for such
cases. So a new rule SA7 is introduced to take care of the
same.

SA7: "If Q has both AGG_FUNC(DISTINCT ...) and
      MIN/MAX() functions then loose index scan access
      method is not used."

mysql-test/r/group_min_max.result:
  Expected result.
mysql-test/t/group_min_max.test:
  1. Test with various combination of AGG(DISTINCT) and
  MIN(), MAX() functions.
  2. Corrected the plan for old queries.
sql/opt_range.cc:
  A new rule SA7 is introduced.

Bug #18382225 MYSQL_CONFIG CAN'T HANDLE RELOCABLE PACKAGES THAT USES "LIB64" OR "-64" SUFFIX
  
'lib' is hardcoded into mysql_config, so 'cmake -DINSTALL_LIBDIR=lib64' will not work.
Use INSTALL_LIBDIR when generating mysql_config.
  
mysql_config may be renamed to e.g. mysql_config-32, fix the basedir pattern matching.

SHOW PROCESSLIST, SHOW BINLOGS

Fixing post push test failure (MTR does not like giving
127.0.0.1 for localhost incase of --embedded run, it thinks
it is an external ip address)

SHOW PROCESSLIST, SHOW BINLOGS

Problem:  A deadlock was occurring when 4 threads were
involved in acquiring locks in the following way
Thread 1: Dump thread ( Slave is reconnecting, so on
              Master, a new dump thread is trying kill
              zombie dump threads. It acquired thread's
              LOCK_thd_data and it is about to acquire
              mysys_var->current_mutex ( which LOCK_log)
Thread 2: Application thread is executing show binlogs and
               acquired LOCK_log and it is about to acquire
               LOCK_index.
Thread 3: Application thread is executing Purge binary logs
               and acquired LOCK_index and it is about to
               acquire LOCK_thread_count.
Thread 4: Application thread is executing show processlist
               and acquired LOCK_thread_count and it is
               about to acquire zombie dump thread's
               LOCK_thd_data.
Deadlock Cycle:
     Thread 1 -> Thread 2 -> Thread 3-> Thread 4 ->Thread 1

The same above deadlock was observed even when thread 4 is
executing 'SELECT * FROM information_schema.processlist' command and
acquired LOCK_thread_count and it is about to acquire zombie
dump thread's LOCK_thd_data.

Analysis:
There are four locks involved in the deadlock.  LOCK_log,
LOCK_thread_count, LOCK_index and LOCK_thd_data.
LOCK_log, LOCK_thread_count, LOCK_index are global mutexes
where as LOCK_thd_data is local to a thread.
We can divide these four locks in two groups.
Group 1 consists of LOCK_log and LOCK_index and the order
should be LOCK_log followed by LOCK_index.
Group 2 consists of other two mutexes
LOCK_thread_count, LOCK_thd_data and the order should
be LOCK_thread_count followed by LOCK_thd_data.
Unfortunately, there is no specific predefined lock order defined
to follow in the MySQL system when it comes to locks across these
two groups. In the above problematic example,
there is no problem in the way we are acquiring the locks
if you see each thread individually.
But If you combine all 4 threads, they end up in a deadlock.

Fix: 
Since everything seems to be fine in the way threads are taking locks,
In this patch We are changing the duration of the locks in Thread 4
to break the deadlock. i.e., before the patch, Thread 4
('show processlist' command) mysqld_list_processes()
function acquires LOCK_thread_count for the complete duration
of the function and it also acquires/releases
each thread's LOCK_thd_data.

LOCK_thread_count is used to protect addition and
deletion of threads in global threads list. While show
process list is looping through all the existing threads,
it will be a problem if a thread is exited but there is no problem
if a new thread is added to the system. Hence a new mutex is
introduced "LOCK_thd_remove" which will protect deletion
of a thread from global threads list. All threads which are
getting exited should acquire LOCK_thd_remove
followed by LOCK_thread_count. (It should take LOCK_thread_count
also because other places of the code still thinks that exit thread
is protected with LOCK_thread_count. In this fix, we are changing
only 'show process list' query logic )
(Eg: unlink_thd logic will be protected with
LOCK_thd_remove).

Logic of mysqld_list_processes(or file_schema_processlist)
will now be protected with 'LOCK_thd_remove' instead of
'LOCK_thread_count'.

Now the new locking order after this patch is:
LOCK_thd_remove -> LOCK_thd_data -> LOCK_log ->
LOCK_index -> LOCK_thread_count

ISSUE:
------
For UNION of selects, rows examined by the query will be sum
of rows examined by individual select operations and rows
examined for union operation. The value of session level
global counter that is used to count the rows examined by a
select statement should be accumulated and reset before it
is used for next select statement. But we have missed to
reset the same. Because of this examined row count of a
select query is accounted more than once.

SOLUTION:
---------
In union reset the session level global counter used to
accumulate count of examined rows after its value is saved.

mysql-test/r/union.result:
  Expected output of testcase added.
mysql-test/t/union.test:
  Test to verify examined row count of Union operations.
sql/sql_union.cc:
  Reset the value of thd->examined_row_count after
  accumulating the value.

Description: Using the temporary file vulnerability an
attacker can create a file with arbitrary content at a
location of his choice. This can be used to create the
file /var/lib/mysql/my.cnf, which will be read as a
configuration file by MySQL, because it is located in the
home directory of the mysql user. With this configuration
file, the attacker can specify his own plugin_dir variable,
which then allows him to load arbitrary code via
"INSTALL PLUGIN...".

Analysis: While creating the ".TMD" file we are not checking
if the file is already exits or not in mi_repair() function.
And we are truncating if the ".TMD" file exits and going ahead
This is creating the security breach.

Fix: We need to use O_EXCL flag along with O_RDWR and O_TRUNC
which will make sure if any user creates ".TMD" file, will
fails the repair table with "cannot create ".TMD" file error".
Actually we are initialing "param.tmpfile_createflag" member
with O_RDWR | O_TRUNC | O_EXCL in myisamchk_init(). And we
are modifying it in ha_myisam::repair() to O_RDWR | O_TRUNC.
So, we need to remove the line which is modifying the
"param.tmpfile_createflag".

Bug#18187290 ISSUE WITH BUILDING MYSQL USING CMAKE 2.8.12

We want to upgrade to VS2013 on Windows.
In order to do this, we need to upgrade to cmake 2.8.12
This has introduced some incompatibilities for .pdb files,
and "make install" no longer works.

To reproduce:
  cmake --build . --target package --config debug

The fix:
Rather than installing .pdb files for static libraries, we use the /Z7 flag
to store symbolic debugging information in the .obj files.

Problem:
If there is a predicate on a column referenced by MIN/MAX and
that predicate is not present in all the disjunctions on
keyparts earlier in the compound index, Loose Index Scan will
not return correct result.

Analysis:
When loose index scan is chosen, range optimizer currently
groups all the predicates that contain group parts separately
and minmax parts separately. It therefore applies all the
conditions on the group parts first to the fetched row.
Then in the call to next_max, it processes the conditions
which have min/max keypart.

For ex in the following query:
Select f1, max(f2) from t1 where (f1 = 10 and f2 = 13) or
(f1 = 3) group by f1;
Condition (f2 = 13) would be applied even for rows that
satisfy (f1 = 3) thereby giving wrong results.

Solution:
Do not choose loose_index_scan for such cases. So a new rule
WA2 is introduced to take care of the same.

WA2: "If there are predicates on C, these predicates must
be in conjuction to all predicates on all earlier keyparts
in I."

Todo the same, fix reuses the function get_constant_key_infix().
Since this funciton will fail for all multi-range conditions, it
is re-written to recognize that if the sub-conditions are
equivalent across the disjuncts: it will now succeed.
And to achieve this a new helper function is introduced called
all_same().

The fix also moves the test of NGA3 up to the former only
caller, get_constant_key_infix().


mysql-test/r/group_min_max_innodb.result:
  Added test result change for Bug#17909656
mysql-test/t/group_min_max_innodb.test:
  Added test cases for Bug#17909656
sql/opt_range.cc:
  Introduced Rule WA2 because of Bug#17909656

Fixing post push failure

Typo leading to not including the last list values (partition).

Also improved pruning to skip last partition if not used.

rb#4762 approved by Aditya and Marko.

Fixing post push failure

Problem: Uninstallation of semi sync plugin causes replication to
break.

Analysis: A semisync enabled replication is mutual agreement between
Master and Slave when the connection (I/O thread) is established.
Once I/O thread is started and if semisync is enabled on both
master and slave, master appends special magic header to events
using semisync plugin functions and sends it to slave. And slave
expects that each event will have that special magic header format
and reads those bytes using semisync plugin functions.

When semi sync replication is in use if users execute
uninstallation of the plugin on master, slave gets confused while
interpreting that event's content because it expects special 
magic header at the beginning of the event. Slave SQL thread will
be stopped with "Missing magic number in the header" error.

Similar problem will happen if uninstallation of the plugin happens
on slave when semi sync replication is in in use. Master sends
the events with magic header and slave does not know about the
added magic header and thinks that it received a corrupted event.
Hence slave SQL thread stops with "Found  corrupted event" error.

Fix: Uninstallation of semisync plugin will be blocked when semisync
replication is in use and will throw 'ER_UNKNOWN_ERROR' error.
To detect that semisync replication is in use, this patch uses
semisync status variable values.
 > On Master, it checks for 'Rpl_semi_sync_master_status' to be OFF
    before allowing the uninstallation of rpl_semi_sync_master plugin.
    >> Rpl_semi_sync_master_status is OFF when
        >>> there is no dump thread running
        >>> there are no semisync slaves
 > On Slave, it checks for 'Rpl_semi_sync_slave_status' to be OFF
    before allowing the uninstallation of rpl_semi_sync_slave plugin.
    >> Rpl_semi_sync_slave_status is OFF when
       >>> there is no I/O thread running
       >>> replication is asynchronous replication.

Bug #18593044 COMPILE FLAGS NOT PASSED TO DTRACE, BREAKS CROSS BUILD