MDEV-20325: Assertion `outer_context || !*from_field || *from_field ==...

MDEV-20325: Assertion `outer_context || !*from_field || *from_field == not_found_field' failed in Item_field::fix_outer_field | `!derived->is_excluded()' failed in TABLE_LIST::set_check_materialized | SIGEGV in st_select_lex::mark_as_dependent (optimized builds)

Re-execution of a query containing subquery in the FROM clause results
in assert failure in case the query is run as part of a stored routine or
as a prepared statement AND derived table merge optimization is off.
As an example, the following test case
  CREATE TABLE t1 (a INT) ;
  CREATE PROCEDURE sp() SELECT * FROM (SELECT a FROM t1) tb;
  CALL sp();
  SET optimizer_switch='derived_merge=off';
  CALL sp();
results in assert failure on the second invocation of the 'sp' stored routine.

The reason for assertion failure is that the expression
  derived->is_excluded()
returns the value true where the value false expected.

The method is_excluded() returns the value true for a derived table
that has been merged to a parent select. Such transformation happens as part
of Derived Table Merge Optimization that is performed on first invocation of
a stored routine or a prepared statement containing a query with subquery
in the FROM clause of the main SELECT.

When the same routine or prepared statement is run the second time and
Derived Table Merge Optimization is OFF the MariaDB server tries to materialize
a derived table specified by the subquery that fails since this subquery
has already been merged to the top-most SELECT. This transformation is permanent
and can't be reverted. That is the reason why the assert
  DBUG_ASSERT(!derived->is_excluded());
fails inside the function TABLE_LIST::set_check_materialized().

Similar behaviour can be observed in case a stored routine or prepared statement
containing a SELECT statement with subquery in the FROM clause, first is run
with the optimizer_switch option set to derived_merge=off and re-run after this
option has been switched to derived_merge=on. In this case a derived table for
subquery is materialized on the first execution and marked as merged derived
table on the second execution that results in error with misleading error
message:

MariaDB [test]> CALL sp1();
ERROR 1030 (HY000): Got error 1 "Operation not permitted" from storage engine MEMORY

To fix the issue, a derived table that has been already optimized shouldn't be
re-marked for one more round of optimization.

One significant consequence following from suggested change is that the data
member TABLE_LIST::derived_type is not updated once the table optimization
has been done. This fact should be taken into account when Prepared Statement
being handled since once a table listed in a query has been optimized on
execution of the statement PREPARE FROM it won't be touched anymore on handling
the statement EXECUTE.

One side effect caused by this change could be observed for the following
test case:
  CREATE TABLE t1 (s1 INT);
  CREATE VIEW v1 AS
    SELECT s1,s2 FROM (SELECT s1 as s2 FROM t1 WHERE s1 <100) x, t1 WHERE t1.s1=x.s2;
  INSERT INTO v1 (s1) VALUES (-300);

  PREPARE stmt FROM "INSERT INTO v1 (s1) VALUES (-300)";
  EXECUTE stmt;

Execution of the above EXECUTE statement results in issuing the error
ER_COLUMNACCESS_DENIED_ERROR since table_ref->is_merged_derived() is false
and check_column_grant_in_table_ref() called for a temporary table that
shouldn't be. To fix this issue the function find_field_in_tables has been
modified in such a way that the function check_column_grant_in_table_ref()
is not called for a temporary table.

mysql-test/r/derived.result

@@ -1204,5 +1204,58 @@ REPLACE INTO v2 ( SELECT * FROM v4 ) UNION ( SELECT f FROM v2 );
 drop view v1,v2,v3,v4;
 drop table t1,t2,t3;
 #
+# MDEV-20325: Assertion `outer_context || !*from_field || *from_field == not_found_field' failed in Item_field::fix_outer_field | `!derived->is_excluded()' failed in TABLE_LIST::set_check_materialized | SIGEGV in st_select_lex::mark_as_dependent (optimized builds)
+#
+CREATE TABLE t1 (a INT);
+# Check that re-execution of a stored routine containing
+# a query with subquery in the FROM clause doesn't result in
+# assert failure in case the 'derived_merge' optimizer option
+# has been turned on/off
+CREATE PROCEDURE sp() SELECT * FROM (SELECT a FROM t1) tb;
+CALL sp();
+a
+SET optimizer_switch='derived_merge=off';
+# Without the patch the following statement would result in assert
+# failure
+CALL sp();
+a
+# Check the same test case for Prepared Statement
+SET optimizer_switch='derived_merge=on';
+PREPARE stmt FROM "SELECT * FROM (SELECT a FROM t1) tb";
+EXECUTE stmt;
+a
+SET optimizer_switch='derived_merge=off';
+# Without the patch the following statement would result in assert
+# failure
+EXECUTE stmt;
+a
+DEALLOCATE PREPARE stmt;
+# Here check the reverse test case - first turn off the 'derived_merge'
+# optimizer option, run the stored routine containing a query with
+# subquery in the FROM clause, then turn on the 'derived_merge'
+# optimizer option and re-execute the same stored routine to check that
+# the routine is finished successfully.
+CREATE PROCEDURE sp1() SELECT * FROM (SELECT a FROM t1) tb;
+SET optimizer_switch='derived_merge=off';
+CALL sp1();
+a
+SET optimizer_switch='derived_merge=on';
+CALL sp1();
+a
+# Check the same test case for Prepared Statement
+SET optimizer_switch='derived_merge=off';
+PREPARE stmt FROM "SELECT * FROM (SELECT a FROM t1) tb";
+EXECUTE stmt;
+a
+SET optimizer_switch='derived_merge=on';
+# Without the patch the following statement would result in assert
+# failure
+EXECUTE stmt;
+a
+DEALLOCATE PREPARE stmt;
+DROP PROCEDURE sp;
+DROP PROCEDURE sp1;
+DROP TABLE t1;
+#
 # End of 10.2 tests
 #

mysql-test/t/derived.test

@@ -1036,6 +1036,57 @@ REPLACE INTO v2 ( SELECT * FROM v4 ) UNION ( SELECT f FROM v2 );
 drop view v1,v2,v3,v4;
 drop table t1,t2,t3;
 
+--echo #
+--echo # MDEV-20325: Assertion `outer_context || !*from_field || *from_field == not_found_field' failed in Item_field::fix_outer_field | `!derived->is_excluded()' failed in TABLE_LIST::set_check_materialized | SIGEGV in st_select_lex::mark_as_dependent (optimized builds)
+--echo #
+CREATE TABLE t1 (a INT);
+
+--echo # Check that re-execution of a stored routine containing
+--echo # a query with subquery in the FROM clause doesn't result in
+--echo # assert failure in case the 'derived_merge' optimizer option
+--echo # has been turned on/off
+CREATE PROCEDURE sp() SELECT * FROM (SELECT a FROM t1) tb;
+CALL sp();
+SET optimizer_switch='derived_merge=off';
+--echo # Without the patch the following statement would result in assert
+--echo # failure
+CALL sp();
+
+--echo # Check the same test case for Prepared Statement
+SET optimizer_switch='derived_merge=on';
+PREPARE stmt FROM "SELECT * FROM (SELECT a FROM t1) tb";
+EXECUTE stmt;
+SET optimizer_switch='derived_merge=off';
+--echo # Without the patch the following statement would result in assert
+--echo # failure
+EXECUTE stmt;
+DEALLOCATE PREPARE stmt;
+
+--echo # Here check the reverse test case - first turn off the 'derived_merge'
+--echo # optimizer option, run the stored routine containing a query with
+--echo # subquery in the FROM clause, then turn on the 'derived_merge'
+--echo # optimizer option and re-execute the same stored routine to check that
+--echo # the routine is finished successfully.
+CREATE PROCEDURE sp1() SELECT * FROM (SELECT a FROM t1) tb;
+SET optimizer_switch='derived_merge=off';
+CALL sp1();
+SET optimizer_switch='derived_merge=on';
+CALL sp1();
+
+--echo # Check the same test case for Prepared Statement
+SET optimizer_switch='derived_merge=off';
+PREPARE stmt FROM "SELECT * FROM (SELECT a FROM t1) tb";
+EXECUTE stmt;
+SET optimizer_switch='derived_merge=on';
+--echo # Without the patch the following statement would result in assert
+--echo # failure
+EXECUTE stmt;
+DEALLOCATE PREPARE stmt;
+
+DROP PROCEDURE sp;
+DROP PROCEDURE sp1;
+DROP TABLE t1;
+
 --echo #
 --echo # End of 10.2 tests
 --echo #

sql/sql_base.cc

@@ -5980,7 +5980,7 @@ find_field_in_tables(THD *thd, Item_ident *item,
                                  TRUE, &(item->cached_field_index));
 #ifndef NO_EMBEDDED_ACCESS_CHECKS
       /* Check if there are sufficient access rights to the found field. */
-      if (found && check_privileges &&
+      if (found && check_privileges && !is_temporary_table(table_ref) &&
           check_column_grant_in_table_ref(thd, table_ref, name, length))
         found= WRONG_GRANT;
 #endif

sql/table.cc

@@ -8231,6 +8231,24 @@ void TABLE_LIST::wrap_into_nested_join(List<TABLE_LIST> &join_list)
 }
 
 
+/**
+  Check whether optimization has been performed and a derived table either
+  been merged to upper select level or materialized.
+
+  @param table  a TABLE_LIST object containing a derived table
+
+  @return true in case the derived table has been merged to surrounding select,
+          false otherwise
+*/
+
+static inline bool derived_table_optimization_done(TABLE_LIST *table)
+{
+  return table->derived &&
+      (table->derived->is_excluded() ||
+       table->is_materialized_derived());
+}
+
+
 /**
   @brief
   Initialize this derived table/view
@@ -8267,13 +8285,15 @@ bool TABLE_LIST::init_derived(THD *thd, bool init_view)
     set_multitable();
 
   unit->derived= this;
-  if (init_view && !view)
+  if (init_view && !view &&
+      !derived_table_optimization_done(this))
   {
     /* This is all what we can do for a derived table for now. */
     set_derived();
   }
 
-  if (!is_view())
+  if (!is_view() &&
+      !derived_table_optimization_done(this))
   {
     /* A subquery might be forced to be materialized due to a side-effect. */
     if (!is_materialized_derived() && first_select->is_mergeable() &&