• Peter Zijlstra's avatar
    sched/core: Fix an SMP ordering race in try_to_wake_up() vs. schedule() · ecf7d01c
    Peter Zijlstra authored
    Oleg noticed that its possible to falsely observe p->on_cpu == 0 such
    that we'll prematurely continue with the wakeup and effectively run p on
    two CPUs at the same time.
    
    Even though the overlap is very limited; the task is in the middle of
    being scheduled out; it could still result in corruption of the
    scheduler data structures.
    
            CPU0                            CPU1
    
            set_current_state(...)
    
            <preempt_schedule>
              context_switch(X, Y)
                prepare_lock_switch(Y)
                  Y->on_cpu = 1;
                finish_lock_switch(X)
                  store_release(X->on_cpu, 0);
    
                                            try_to_wake_up(X)
                                              LOCK(p->pi_lock);
    
                                              t = X->on_cpu; // 0
    
              context_switch(Y, X)
                prepare_lock_switch(X)
                  X->on_cpu = 1;
                finish_lock_switch(Y)
                  store_release(Y->on_cpu, 0);
            </preempt_schedule>
    
            schedule();
              deactivate_task(X);
              X->on_rq = 0;
    
                                              if (X->on_rq) // false
    
                                              if (t) while (X->on_cpu)
                                                cpu_relax();
    
              context_switch(X, ..)
                finish_lock_switch(X)
                  store_release(X->on_cpu, 0);
    
    Avoid the load of X->on_cpu being hoisted over the X->on_rq load.
    Reported-by: default avatarOleg Nesterov <oleg@redhat.com>
    Signed-off-by: default avatarPeter Zijlstra (Intel) <peterz@infradead.org>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Mike Galbraith <efault@gmx.de>
    Cc: Peter Zijlstra <peterz@infradead.org>
    Cc: Thomas Gleixner <tglx@linutronix.de>
    Signed-off-by: default avatarIngo Molnar <mingo@kernel.org>
    ecf7d01c
core.c 206 KB