MySQL bug number 90264

Contribution by Yura Sorokin.

Problem:

File mysys/mf_iocache2.c contains non instrumented file io operations.
This causes inaccurate statistics in PERFORMANCE_SCHEMA.

Solution:

Use the instrumentation apis (mysql_file_tell instead of my_tell, etc).

               PARTITION

Issue:
------
ALTER TABLE REORGANIZE PARTITION .... can result in
incorrect behavior if any partition other than the last
one misses the "VALUES LESS THAN..." part of the syntax.

Root cause:
-----------
Currently ALTER TABLE with changes to partitions is handled
incorrectly by the parser.

Fix:
----
The if condition which handles partition management
differently for ALTER TABLE in the parser should be removed.
 Change the code to handle the case in the parser.

    - Added Perl Data dumper as a dependency for server and test sub packages.

              LOSS

ANALYSIS:
=========
When converting from a BLOB/TEXT type to a smaller
BLOB/TEXT type, no warning/error is reported to the user
informing about the truncation/data loss.

FIX:
====
We are now reporting a warning in non-strict mode and an
appropriate error in strict mode.

Switch to Exuberant Ctags when generating TAGS, since it is much
better at parsing modern C++

Change-Id: I9652012708df7e7edf93161097a547f60fb0cf79
(cherry picked from commit 125b2804fbbb8662632f761f39aeef0a7f9cebb3)

Problem: Can't locate lib/mtr_process.pl in @INC

The fix is to push current directory to @INC.

client/mysql.cc: In function void build_completion_hash(bool, bool):
client/mysql.cc:2674:37: error: invalid conversion from char to char* [-fpermissive]
       field_names[i][num_fields*2]= '\0';

Upgrading the zlib lib to 1.2.11

               --BINARY-AS-HEX, --NO-BEEP

DESCRIPTION:
============
mysql uses -b as the short-option form for two different
long options i.e. no-beep and binary-as-hex. This can
result in unintended results if the short form -b is used
instead of the specific long option name.

FIX:
====
-b will now be used for one long option only i.e --no-beep.
The option binary-as-hex will not have any short option and
should be provided as a complete name.

              INFO (HP_INFO)

Description:- Server crashes due to memory overflow.

Analysis:- Bytes for storing key length is wrongly set
for HEAP tables.

Fix:- Bytes used to store the key length is properly set
inside "heap_create()".

Shutdown server as mysql user to avoid accidentally sending signal to
wrong process.

(cherry picked from commit b5b986b2cbd9a7848dc3f48e5c42b6d4e1e5fb22)

client/mysql.cc: In function void build_completion_hash(bool, bool):
client/mysql.cc:2674:37: error: invalid conversion from char to char* [-fpermissive]
       field_names[i][num_fields*2]= '\0';
                                     ^~~~

Description:- MyISAM table gets corrupted with concurrent
executions of INSERT, DELETE statements in a particular
sequence.

Analysis:- Due to the inappropriate manipulation of w_lock
and r_lock associated with a MyISAM table, there arises a
scenario where the table's state information becomes
invalid.

Fix:- A lock is introduced to resolve this issue.

Description:- Client applications establishes connection to
server, which does not support SSL, via TCP even when SSL is
enforced via MYSQL_OPT_SSL_MODE or MYSQL_OPT_SSL_ENFORCE or
MYSQL_OPT_SSL_VERIFY_SERVER_CERT.

Analysis:- There exist no error handling for catching client
applications which enforces SSL connection to connect to a
server which does not support SSL.

Fix:- Error handling is done to catch above mentioned
scenarios.

              SHOW_ROUTINE_GRANTS


Description :- Server crashes in show_routine_grants().

Analysis :- When "grant_reload_procs_priv" encounters
an error, the grant structures (structures with column,
function and procedure privileges) are freed. Server
crashes when trying to access these structures later.

Fix :- Grant structures are retained even when
"grant_reload_procs_priv()" encounters an error while
reloading column, function and procedure privileges.

              INSERT PRIVILEGES FOR MYSQL.USER TABLE

Description:- Incorrect granting of EXECUTE and ALTER
ROUTINE privileges when the 'automatic_sp_privileges'
variable is set.

Fix:- EXECUTE and ALTER ROUTINE privileges are correctly
granted to the creator of the procedure when the
'automatic_sp_privileges' is SET.

                          WRONG VALUES

User variables will have the default session collation
associated with it. And a select which uses it as part of a
union may infer the collation while type merging.
This leads to problems when the result is of DECIMAL type.
Setting the appropriate collation of DECIMAL result type
is missing in 5.7 code base.

Added code to set appropriate collation when the result is
of DECIMAL type during Item_type_holder::join_types().

Description: Mysqldump utility fails for specific clauses
used with the option, 'where'.

Analysis:- Method, "fix_identifier_with_newline()" that
prefixes all occurrences of newline char ('\n') in incoming
buffer does not verify the size of the buffer. The buffer in
which the incoming buffer is copied is limited to 2048 bytes
and the method does not try to allocate additional memory
for larger incoming buffers.

Fix:- Method, "fix_identifier_with_newline()" is modified
to fix this issue.

              PREBUILT->TABLE->N_MYSQL_HANDLES_OPENED == 1

ANALYSIS:
=========

Adding unique index to a InnoDB table which is locked as
mutliple instances may trigger an InnoDB assert.

When we add a primary key or an unique index, we need to
drop the original table and rebuild all indexes. InnoDB
expects that only the instance of the table that is being
rebuilt, is open during the process. In the current
scenario we have opened multiple instances of the table.
This triggers an assert during table rebuild.
'Locked_tables_list' encapsulates a list of all
instances of tables locked by LOCK TABLES statement.

FIX:
===
We are now temporarily closing all the instances of the
table except the one which is being altered and later
reopen them via Locked_tables_list::reopen_tables().

Change the file to refer to published git repository directly

No commit message

Description:- Mysql client crashes when trying to connect
to a fake server which is sending incorrect packets.

Analysis:- Mysql client crashes when it tries to read
server version details.

Fix:- A check is added in "red_one_row()".

              PREVIOUS TO MYSQL 8.0

Description :
-------------
The mysqld--defaults-file test fails when the test suite is run from a
non-canonical path, which happens when the current working directory
when mysql-test-run.pl is started contains a symbolic link.

The problem is that this test case uses --replace-result with
$MYSQL_TEST_DIR. This variable is a potentially non-canonical path
based on the current working directory when mtr is started. However,
the path in the expected error message from mysqld contains a
canonical path. This means it does not contain $MYSQL_TEST_DIR if
mtr's working directory is not the canonical path of the working
directory.

Because other tests produce output that may contain non-canonical
paths, making $MYSQL_TEST_DIR always canonical is not a fix.

Fix :
-----
Introduced a new environment variable '$ABS_MYSQL_TEST_DIR' which will
contin the canonical path to the test directory and replaced
$MYSQL_TEST_DIR with the new variable in main.mysqld--defaults-file
test file.

This is a back-port of BUG#24579973.

Change-Id: I3b8df6f2d7ce2b04e188a896d76250cc1addbbc1

Problem
=======

When facing decoding of corrupt binary log files, server may misbehave
without detecting the events corruption.

This patch makes MySQL server more resilient to binary log decoding.

Fixes for events de-serialization and apply
===========================================

@sql/log_event.cc

Query_log_event::Query_log_event: added a check to ensure query length
is respecting event buffer limits.

Query_log_event::do_apply_event: extended a debug print, added a check
to character set to determine if it is "parseable" or not, verified if
database name is valid for system collation.

Start_log_event_v3::do_apply_event: report an error on applying a
non-supported binary log version.

Load_log_event::copy_log_event: added a check to table_name length.

User_var_log_event::User_var_log_event: added checks to avoid reading
out of buffer limits.

User_var_log_event::do_apply_event: reported an sanity check error
properly and added individual sanity checks for variable types that
expect fixed (or minimum) amount of bytes to be read.

Rows_log_event::Rows_log_event: added checks to avoid reading out of
buffer limits.

@sql/log_event_old.cc

Old_rows_log_event::Old_rows_log_event: added a sanity check to avoid
reading out of buffer limits.

@sql/sql_priv.h

Added a sanity check to available_buffer() function.

              STATEMENTS

ANALYSIS:
=========
A user not having FILE privilege is not allowed to create
custom data/index directories for a table or for its
partitions via CREATE TABLE but is allowed to do so via
ALTER TABLE statement.

ALTER TABLE ignores DATA DIRECTORY and INDEX DIRECTORY when
given as table options. The issue occurs during the
creation of partitions for a table via ALTER TABLE
statement with the DATA DIRECTORY and/or INDEX DIRECTORY
options. The issue exists because of the absence of FILE
privilege check for the user.

FIX:
====
A FILE privilege check has been introduced for resolving
the above scenario.

 Followup: now that the man pages have actually been removed,
 we no longer need to take deliberate action to ignore them.
 Thus we can remove that part of the original change.

 RPM: drop the conditional removal
 DEB: remove from the exclude list

No commit message

No commit message

              IS DROPPED

ANALYSIS:
=========
It is advised not to tamper with the system tables.
When primary key is dropped from a system table, certain
operations on the table which tries to access the table key
information may lead to server exit.

FIX:
====
An appropriate error is now reported in such a case.

               ROOT

DESCRIPTION
===========
If the .pid file is created at a world-writable location,
it can be compromised by replacing the server's pid with
another running server's (or some other non-mysql process)
PID causing abnormal behaviour.

ANALYSIS
========
In such a case, user should be warned that .pid file is
being created at a world-writable location.

FIX
===
A new function is_file_or_dir_world_writable() is defined
and it is called in create_pid_file() before .pid file
creation. If the location is world-writable, a relevant
warning is thrown.

NOTE
====
1. PID file is always created with permission bit 0664, so
for outside world its read-only.
2. Ignoring the case when permission is denied to get the
dir stats since the .pid file creation would fail anyway in
such a case.

              MDL_CONTEXT::TRY_ACQUIRE_LOCK_IMPL

ANALYSIS:
=========
Server sometimes exited when multiple threads tried to
acquire and release metadata locks simultaneously (for
example, necessary to access a table). The same problem
could have occurred when new objects were registered/
deregistered in Performance Schema.

The problem was caused by a bug in LF_HASH - our lock free
hash implementation which is used by metadata locking
subsystem in 5.7 branch. In 5.5 and 5.6 we only use LF_HASH
in Performance Schema Instrumentation implementation. So
for these versions, the problem was limited to P_S.

The problem was in my_lfind() function, which searches for
the specific hash element by going through the elements
list. During this search it loads information about element
checked such as key pointer and hash value into local
variables. Then it confirms that they are not corrupted by
concurrent delete operation (which will set pointer to 0)
by checking if element is still in the list. The latter
check did not take into account that compiler (and
processor) can reorder reads in such a way that load of key
pointer will happen after it, making result of the check
invalid.

FIX:
====
This patch fixes the problem by ensuring that no such
reordering can take place. This is achieved by using
my_atomic_loadptr() which contains compiler and processor
memory barriers for the check mentioned above and other
similar places.

The default (for non-Windows systems) implementation of
my_atomic*() relies on old __sync intrisics and implements
my_atomic_loadptr() as read-modify operation. To avoid
scalability/performance penalty associated with addition of
my_atomic_loadptr()'s we change the my_atomic*() to use
newer __atomic intrisics when available. This new default
implementation doesn't have such a drawback.

PROBLEM
-------

This warning message is printed when trx_sys->rseg_history_len is greater than some
arbitrary magic number (2000000). By seeing the reproducing scenario where we keep
a read view open and do a lot of transactions on table which increases the hitsory
length it is entirely possible that trx_sys->rseg_history_len can exceed 2000000.
So this is not a bug due to corruption of history length.The warning message was
just added to test some scenario and not removed.

FIX
---

1.Print this warning message only for debug versions.
2.Modified the warning message with more detailed information.
3.Don't crash even in debug versions.

[#rb 17929 Reviewed by jimmy and satya]

Issue:
------
VALUES doesn't have a type() function and is considered a
Item_field.

Solution for 5.7:
-----------------
Add a new type() function for Item_values_insert.

On 8.0 and trunk it was fixed by Mithun's Bug#19601973.

Solution for 5.6:
-----------------
Additionally Bug#17458914 is backported.

This will address the problem of using VALUES() in
INSERT ... ON DUPLICATE KEY UPDATE. Create a field object
only if it is in the UPDATE clause, else return a NULL
item.

This will also address the problems mentioned in
Bug#14789787 and Bug#16756402.

Solution for 5.5:
-----------------
As mentioned above Bug#17458914 is backported.

Additionally Bug#14786324 is also backported.

When VALUES() is detected outside its meaningful place,
it should be treated as NULL and is thus replaced with a
Field_null object, with the same name as the original
field.

Fields with type NULL are generally not handled well inside
the server (e.g Innodb will not accept them and it is
impossible to create them in regular tables). So create a
new const NULL item instead.