Commit 8f2f8f31 authored by Xiaotong Niu's avatar Xiaotong Niu Committed by Daniel Black

MDEV-26494 Fix buffer overflow of string lib on Arm64

In the hexlo function, the element type of the array hex_lo_digit is not
explicitly declared as signed char, causing elements with a value of -1
to be converted to 255 on Arm64. The problem occurs because "char" is
unsigned by default on Arm64 compiler, but signed on x86 compiler. This
problem can be seen in https://godbolt.org/z/rT775xshj

The above issue causes "use-after-poison" exception in my_mb_wc_filename
function. The code snippet where the error occurred is shown below,
copied from below link.
https://github.com/MariaDB/server/blob/5fc19e71375fb39eb85354321bf852d998aecf81/strings/ctype-utf8.c#L2728

2728    if ((byte1= hexlo(byte1)) >= 0 &&
2729     (byte2= hexlo(byte2)) >= 0)
  	{
2731    	int byte3= hexlo(s[3]);
    		…
  	}

At line 2729, when byte2 is 0, which indicates the end of the string s.
(1) On x86, hexlo(0) return -1 and line 2731 is skipped, as expected.
(2) On Arm64, hexlo(0) return 255 and line 2731 is executed, not as
expected, accessing s[3] after the null character of string s, thus
raising the "user-after-poison" error.

The problem was discovered when executing the main.mysqlcheck test.
Signed-off-by: default avatarXiaotong Niu <xiaotong.niu@arm.com>
parent e467e8d8
......@@ -6872,7 +6872,7 @@ static const uint16 uni_FF20_FF5F[64]=
static int hexlo(int x)
{
static const char hex_lo_digit[256]=
static const signed char hex_lo_digit[256]=
{
-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1, /* ................ */
-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1, /* ................ */
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment