Bug #13889741: HANDLE_FATAL_SIGNAL IN _DB_ENTER_ |

HANDLE_FATAL_SIGNAL IN STRNLEN

Fixed the following bounds checking problems :
1. in check_if_legal_filename() make sure the null terminated
string is long enough before accessing the bytes in it.
Prevents pottential read-past-buffer-end
2. in my_wc_mb_filename() of the filename charset check
for the end of the destination buffer before sending single
byte characters into it.
Prevents write-past-end-of-buffer (and garbaling stack in
the cases reported here) errors.

Added test cases.

mysys/my_access.c

@@ -148,7 +148,8 @@ static char reserved_map[256]=
 int check_if_legal_tablename(const char *name)
 {
   DBUG_ENTER("check_if_legal_tablename");
-  DBUG_RETURN((reserved_map[(uchar) name[0]] & 1) &&
+  DBUG_RETURN(name[0] != 0 && name[1] != 0 &&
+              (reserved_map[(uchar) name[0]] & 1) &&
               (reserved_map[(uchar) name[1]] & 2) &&
               (reserved_map[(uchar) name[2]] & 4) &&
               str_list_find(&reserved_names[1], name));

strings/ctype-utf8.c

@@ -4326,6 +4326,10 @@ my_wc_mb_filename(CHARSET_INFO *cs __attribute__((unused)),
 {
   int code;
   char hex[]= "0123456789abcdef";
+
+  if (s >= e)
+    return MY_CS_TOOSMALL;
+
   if (wc < 128 && filename_safe_char[wc])
   {
     *s= (uchar) wc;