• Xiaotong Niu's avatar
    MDEV-26494 Fix buffer overflow of string lib on Arm64 · 8f2f8f31
    Xiaotong Niu authored
    In the hexlo function, the element type of the array hex_lo_digit is not
    explicitly declared as signed char, causing elements with a value of -1
    to be converted to 255 on Arm64. The problem occurs because "char" is
    unsigned by default on Arm64 compiler, but signed on x86 compiler. This
    problem can be seen in https://godbolt.org/z/rT775xshj
    
    The above issue causes "use-after-poison" exception in my_mb_wc_filename
    function. The code snippet where the error occurred is shown below,
    copied from below link.
    https://github.com/MariaDB/server/blob/5fc19e71375fb39eb85354321bf852d998aecf81/strings/ctype-utf8.c#L2728
    
    2728    if ((byte1= hexlo(byte1)) >= 0 &&
    2729     (byte2= hexlo(byte2)) >= 0)
      	{
    2731    	int byte3= hexlo(s[3]);
        		…
      	}
    
    At line 2729, when byte2 is 0, which indicates the end of the string s.
    (1) On x86, hexlo(0) return -1 and line 2731 is skipped, as expected.
    (2) On Arm64, hexlo(0) return 255 and line 2731 is executed, not as
    expected, accessing s[3] after the null character of string s, thus
    raising the "user-after-poison" error.
    
    The problem was discovered when executing the main.mysqlcheck test.
    Signed-off-by: default avatarXiaotong Niu <xiaotong.niu@arm.com>
    8f2f8f31
ctype-utf8.c 368 KB