• Alexander Barkov's avatar
    MDEV-32879 Server crash in my_decimal::operator= or unexpected ER_DUP_ENTRY... · 20b0ec9a
    Alexander Barkov authored
    MDEV-32879 Server crash in my_decimal::operator= or unexpected ER_DUP_ENTRY upon comparison with INET6 and similar types
    
    This is the 10.6 version of the patch.
    
    Item_bool_rowready_func2, Item_func_between, Item_func_in
    did not check if a not-NULL argument of an arbitrary data type
    can produce a NULL value on conversion to INET6.
    
    This caused a crash on DBUG_ASSERT() in conversion failures,
    because the function returned SQL NULL for something that
    has Item::maybe_null() equal to false.
    
    Adding setting NULL-ability in such cases.
    
    Details:
    
    - Removing the code in Item_func::setup_args_and_comparator()
      performing character set aggregation with optional narrowing.
      This aggregation is done inside Arg_comparator::set_cmp_func_string().
      So this code was redundant
    
    - Removing Item_func::setup_args_and_comparator() as it git simplified to
      just to two lines:
        convert_const_compared_to_int_field(thd);
        return cmp->set_cmp_func(thd, this, &args[0], &args[1], true);
      Using these lines directly in:
        - Item_bool_rowready_func2::fix_length_and_dec()
        - Item_func_nullif::fix_length_and_dec()
    
    - Adding a new virtual method:
      - Type_handler::Item_bool_rowready_func2_fix_length_and_dec().
    
    - Adding tests detecting if the data type conversion can return SQL NULL into
      the following methods of Type_handler_fbt:
      - Item_bool_rowready_func2_fix_length_and_dec
      - Item_func_between_fix_length_and_dec
      - Item_func_in_fix_comparator_compatible_types
    20b0ec9a
sql_type.h 286 KB