• Sujatha's avatar
    MDEV-20645: Replication consistency is broken as workers miss the error... · 9b80f930
    Sujatha authored
    MDEV-20645: Replication consistency is broken as workers miss the error notification from an earlier failed group.
    
    Analysis:
    ========
    In general if there are three groups.
    1 - Inserts 32 which fails due to local entry '32' on slave.
    2 - Inserts 33
    3 - Inserts 34
    
    Each group considers itself as a waiter and it waits for prior group 'waitee'.
    This is done in 'register_wait_for_prior_event_group_commit'. If there is no
    other parallel group being scheduled then no waitee will be there.
    
    Let us assume 3 groups are being scheduled in parallel.
    
    3-> waits for 2-> waits for->1
    
    '1' upon completion it checks is there any registered subsequent waiter. If
    so it wakes up the subsequent waiter with its execution status. This execution
    status is stored in wakeup_error.
    
    If '1' failed then it sends corresponding wakeup_error to 2. Then '2' aborts
    and it propagates error to '3'.  So all further commits are aborted.  This
    mechanism works only when all transactions reach a stage where they are
    waiting for their prior commit to complete.
    
    In case of optimistic following scenario occurs.
    
    1,2,3 are scheduled in parallel.
    
    3 - Reaches group_commit_code waits for 2 to complete.
    1 - errors out sets stop_on_error_sub_id=1.
    
    When a group execution results in error its corresponding sub_id is set to
    'stop_on_error_sub_id'. Any new groups queued for execution will check if
    their sub_id is > stop_on_error_sub_id.  If it is true their execution will be
    skipped as prior group execution failed.  'skip_event_group=1' will be set.
    Since the execution of SQL thread is about to stop we just skip execution of
    all the following event groups.  We still do all the normal waiting and wakeup
    processing between the event groups as a simple way to ensure that everything
    is stopped and cleaned up correctly.
    
    Upon error '1' transaction checks for registered waiters. Since no one is
    there it simply goes away.
    
    2 - Starts the execution. It checks do I have a waitee.
    
    Since wait_commit_sub_id == entry->last_committed_sub_id no waitee is set.
    
    Secondly: 'entry->stop_on_error_sub_id' is set by '1'st execution.  Now
    'handle_parallel_thread' code checks if the current group 'sub_id' is greater
    than the 'sub_id' set within 'stop_on_error_sub_id'.
    
    Since the above is true 'skip_event_group=true' is set.  Simply call
    'wait_for_prior_commit' to wakeup all waiters.  Group '2' didn't had any
    waitee and its execution is skipped.  Hence its wakeup_error=0.It sends a
    positive wakeup signal to '3'. Which commits. This results in a missed
    transaction. i.e 33 is missed and 34 is committed.
    
    Fix:
    ===
    When a worker learns that an earlier transaction execution has failed, and it
    should not proceed for further execution, it should mark its own execution
    status as failed so that it alerts its followers to abort as well.
    9b80f930
rpl_parallel.cc 88.4 KB