• Alexander Barkov's avatar
    MDEV-30932 UBSAN: negation of -X cannot be represented in type .. · 67657a01
    Alexander Barkov authored
      'long long int'; cast to an unsigned type to negate this value ..
      to itself in Item_func_mul::int_op and Item_func_round::int_op
    
    Problems:
    
      The code in multiple places in the following methods:
        - Item_func_mul::int_op()
        - longlong Item_func_int_div::val_int()
        - Item_func_mod::int_op()
        - Item_func_round::int_op()
    
      did not properly check for corner values LONGLONG_MIN
      and (LONGLONG_MAX+1) before doing negation.
      This cuased UBSAN to complain about undefined behaviour.
    
    Fix summary:
    
      - Adding helper classes ULonglong, ULonglong_null, ULonglong_hybrid
        (in addition to their signed couterparts in sql/sql_type_int.h).
    
      - Moving the code performing multiplication of ulonglong numbers
        from Item_func_mul::int_op() to ULonglong_hybrid::ullmul().
    
      - Moving the code responsible for extracting absolute values
        from negative numbers to Longlong::abs().
        It makes sure to perform negation without undefinite behavior:
        LONGLONG_MIN is handled in a special way.
    
      - Moving negation related code to ULonglong::operator-().
        It makes sure to perform negation without undefinite behavior:
        (LONGLONG_MAX + 1) is handled in a special way.
    
      - Moving signed<=>unsigned conversion code to
        Longlong_hybrid::val_int() and ULonglong_hybrid::val_int().
    
      - Reusing old and new sql_type_int.h classes in multiple
        places in Item_func_xxx::int_op().
    
    Fix details (explain how sql_type_int.h classes are reused):
    
      - Instead of straight negation of negative "longlong" arguments
        *before* performing unsigned multiplication,
        Item_func_mul::int_op() now calls ULonglong_null::ullmul()
        using Longlong_hybrid_null::abs() to pass arguments.
        This fixes undefined behavior N1.
    
      - Instead of straight negation of "ulonglong" result
        *after* performing unsigned multiplication,
        Item_func_mul::int_op() now calls ULonglong_hybrid::val_int(),
        which recursively calls ULonglong::operator-().
        This fixes undefined behavior N2.
    
      - Removing duplicate negating code from Item_func_mod::int_op().
        Using ULonglong_hybrid::val_int() instead.
        This fixes undefinite behavior N3.
    
      - Removing literal "longlong" negation from Item_func_round::int_op().
        Using Longlong::abs() instead, which correctly handler LONGLONG_MIN.
        This fixes undefinite behavior N4.
    
      - Removing the duplicate (negation related) code from
        Item_func_int_div::val_int(). Reusing class ULonglong_hybrid.
        There were no undefinite behavior in here.
        However, this change allowed to reveal a bug in
        "-9223372036854775808 DIV 1".
        The removed negation code appeared to be incorrect when
        negating +9223372036854775808. It returned the "out of range" error.
        ULonglong_hybrid::operator-() now handles all values correctly
        and returns +9223372036854775808 as a negation for -9223372036854775808.
    
        Re-recording wrong results for
          SELECT -9223372036854775808 DIV  1;
        Now instead of "out of range", it returns -9223372036854775808,
        which is the smallest possible value for the expression data type
        (signed) BIGINT.
    
      - Removing "no UBSAN" branch from Item_func_splus::int_opt()
        and Item_func_minus::int_opt(), as it made UBSAN happy but
        in RelWithDebInfo some MTR tests started to fail.
    67657a01
func_math.test 54.1 KB