• Venkata Sidagam's avatar
    Bug #13955256: KEYCACHE CRASHES, CORRUPTIONS/HANGS WITH, · 6b05e434
    Venkata Sidagam authored
                   FULLTEXT INDEX AND CONCURRENT DML.
    
    Problem Statement:
    ------------------
    1) Create a table with FT index.
    2) Enable concurrent inserts.
    3) In multiple threads do below operations repeatedly
       a) truncate table
       b) insert into table ....
       c) select ... match .. against .. non-boolean/boolean mode
    
    After some time we could observe two different assert core dumps
    
    Analysis:
    --------
    1)assert core dump at key_read_cache():
    Two select threads operating in-parallel on same key 
    root block.
    1st select thread block->status is set to BLOCK_ERROR 
    because the my_pread() in read_block() is returning '0'. 
    Truncate table made the index file size as 1024 and pread 
    was asked to get the block of count bytes(1024 bytes) 
    from offset of 1024 which it cannot read since its 
    "end of file" and retuning '0' setting 
    "my_errno= HA_ERR_FILE_TOO_SHORT" and the key_file_length, 
    key_root[0] is same i.e. 1024. Since block status has BLOCK_ERROR 
    the 1st select thread enter into the free_block() and will 
    be under wait on conditional mutex by making status as 
    BLOCK_REASSIGNED and goes for wait_on_readers(). Other select 
    thread will also work on the same block and sees the status as 
    BLOCK_ERROR and enters into free_block(), checks for BLOCK_REASSIGNED 
    and asserting the server.
    
    2)assert core dump at key_write_cache():
    One select thread and One insert thread.
    Select thread gets the unlocks the 'keycache->cache_lock', 
    which allows other threads to continue and gets the pread() 
    return value as'0'(please see the explanation above) and 
    tries to get the lock on 'keycache->cache_lock' and waits 
    there for the lock.
    Insert thread requests for the block, block will be assigned 
    from the hash list and makes the page_status as 
    'PAGE_WAIT_TO_BE_READ' and goes for the read_block(), waits 
    in the queue since there are some other threads performing 
    reads on the same block.
    Select thread which was waiting for the 'keycache->cache_lock' 
    mutex in the read_block() will continue after getting the my_pread() 
    value as '0' and sets the block status as BLOCK_ERROR and goes to 
    the free_block() and go to the wait_for_readers().
    Now the insert thread will awake and continues. and checks 
    block->status as not BLOCK_READ and it asserts.  
    
    Fix:
    ---
    In the full text code, multiple readers of index file is not guarded. 
    Hence added below below code in _ft2_search() and walk_and_match().
    
    to lock the key_root I have used below code in _ft2_search()
     if (info->s->concurrent_insert)
        mysql_rwlock_rdlock(&share->key_root_lock[0]);
    
    and to unlock 
     if (info->s->concurrent_insert)
       mysql_rwlock_unlock(&share->key_root_lock[0]);
    
    storage/myisam/ft_boolean_search.c:
      Since its a recursion function, to avoid confusion in taking and 
      releasing the locks, renamed _ft2_search() to _ft2_search_internal() 
      function. And _ft2_search() will take the lock, call 
      _ft2_search_internal() and release the lock in case of concurrent 
      inserts.
    storage/myisam/ft_nlq_search.c:
      Added read locks code in walk_and_match()
    6b05e434
ft_nlq_search.c 10.2 KB