• mariadb-DebarunBanerjee's avatar
    MDEV-34689 Redo log corruption at high load · e515e807
    mariadb-DebarunBanerjee authored
    Issue: During mtr_t:commit, if there is not enough space available in
    redo log buffer, we flush the buffer. During flush, the LSN lock is
    released allowing other concurrent mtr to commit. After flush we
    reacquire the lock but use the old LSN obtained before check. It could
    lead to redo log corruption. As the LSN moves backwards with the
    possibility of data loss and unrecoverable server if the server aborts
    for any reason or if server is shutdown with innodb_fast_shutdown=2.
    With normal shutdown, recovery fails to map the checkpoint LSN to
    correct offset.
    
    In debug mode it hits log0log.cc:863: lsn_t log_t::write_buf()
    Assertion `new_buf_free == ((lsn - first_lsn) & write_size_1)' failed.
    
    In release mode, after normal shutdown, restart fails.
    [ERROR] InnoDB: Missing FILE_CHECKPOINT(8416546) at 8416546
    [ERROR] InnoDB: Log scan aborted at LSN 8416546
    
    Backup fails reading the corrupt redo log.
    [00] 2024-07-31 20:59:10 Retrying read of log at LSN=7334851
    [00] FATAL ERROR: 2024-07-31 20:59:11 Was only able to copy log from
    7334851 to 7334851, not 8416446; try increasing innodb_log_file_size
    
    Unless a backup is tried or the server is shutdown or killed
    immediately, the corrupt redo part is eventually truncated and there
    may not be any visible issues seen in release mode.
    
    This issue was introduced by the following commit.
    
    commit a635c406
        MDEV-27774 Reduce scalability bottlenecks in mtr_t::commit()
    
    Fix: If we need to release latch and flush redo before writing mtr
    logs, make sure to get the latest system LSN after reacquiring the
    redo system latch.
    e515e807
mtr0mtr.cc 50.8 KB