• Tejun Heo's avatar
    sched_ext: Implement core-sched support · 7b0888b7
    Tejun Heo authored
    The core-sched support is composed of the following parts:
    
    - task_struct->scx.core_sched_at is added. This is a timestamp which can be
      used to order tasks. Depending on whether the BPF scheduler implements
      custom ordering, it tracks either global FIFO ordering of all tasks or
      local-DSQ ordering within the dispatched tasks on a CPU.
    
    - prio_less() is updated to call scx_prio_less() when comparing SCX tasks.
      scx_prio_less() calls ops.core_sched_before() if available or uses the
      core_sched_at timestamp. For global FIFO ordering, the BPF scheduler
      doesn't need to do anything. Otherwise, it should implement
      ops.core_sched_before() which reflects the ordering.
    
    - When core-sched is enabled, balance_scx() balances all SMT siblings so
      that they all have tasks dispatched if necessary before pick_task_scx() is
      called. pick_task_scx() picks between the current task and the first
      dispatched task on the local DSQ based on availability and the
      core_sched_at timestamps. Note that FIFO ordering is expected among the
      already dispatched tasks whether running or on the local DSQ, so this path
      always compares core_sched_at instead of calling into
      ops.core_sched_before().
    
    qmap_core_sched_before() is added to scx_qmap. It scales the
    distances from the heads of the queues to compare the tasks across different
    priority queues and seems to behave as expected.
    
    v3: Fixed build error when !CONFIG_SCHED_SMT reported by Andrea Righi.
    
    v2: Sched core added the const qualifiers to prio_less task arguments.
        Explicitly drop them for ops.core_sched_before() task arguments. BPF
        enforces access control through the verifier, so the qualifier isn't
        actually operative and only gets in the way when interacting with
        various helpers.
    Signed-off-by: default avatarTejun Heo <tj@kernel.org>
    Reviewed-by: default avatarDavid Vernet <dvernet@meta.com>
    Reviewed-by: default avatarJosh Don <joshdon@google.com>
    Cc: Andrea Righi <andrea.righi@canonical.com>
    7b0888b7
core.c 268 KB