• Lukas Wunner's avatar
    PCI: pciehp: Fix indefinite wait on sysfs requests · 3e487d2e
    Lukas Wunner authored
    David Hoyer reports that powering pciehp slots up or down via sysfs may
    hang:  The call to wait_event() in pciehp_sysfs_enable_slot() and
    _disable_slot() does not return because ctrl->ist_running remains true.
    
    This flag, which was introduced by commit 157c1062 ("PCI: pciehp: Avoid
    returning prematurely from sysfs requests"), signifies that the IRQ thread
    pciehp_ist() is running.  It is set to true at the top of pciehp_ist() and
    reset to false at the end.  However there are two additional return
    statements in pciehp_ist() before which the commit neglected to reset the
    flag to false and wake up waiters for the flag.
    
    That omission opens up the following race when powering up the slot:
    
    * pciehp_ist() runs because a PCI_EXP_SLTSTA_PDC event was requested
      by pciehp_sysfs_enable_slot()
    
    * pciehp_ist() turns on slot power via the following call stack:
      pciehp_handle_presence_or_link_change() -> pciehp_enable_slot() ->
      __pciehp_enable_slot() -> board_added() -> pciehp_power_on_slot()
    
    * after slot power is turned on, the link comes up, resulting in a
      PCI_EXP_SLTSTA_DLLSC event
    
    * the IRQ handler pciehp_isr() stores the event in ctrl->pending_events
      and returns IRQ_WAKE_THREAD
    
    * the IRQ thread is already woken (it's bringing up the slot), but the
      genirq code remembers to re-run the IRQ thread after it has finished
      (such that it can deal with the new event) by setting IRQTF_RUNTHREAD
      via __handle_irq_event_percpu() -> __irq_wake_thread()
    
    * the IRQ thread removes PCI_EXP_SLTSTA_DLLSC from ctrl->pending_events
      via board_added() -> pciehp_check_link_status() in order to deal with
      presence and link flaps per commit 6c35a1ac ("PCI: pciehp:
      Tolerate initially unstable link")
    
    * after pciehp_ist() has successfully brought up the slot, it resets
      ctrl->ist_running to false and wakes up the sysfs requester
    
    * the genirq code re-runs pciehp_ist(), which sets ctrl->ist_running
      to true but then returns with IRQ_NONE because ctrl->pending_events
      is empty
    
    * pciehp_sysfs_enable_slot() is finally woken but notices that
      ctrl->ist_running is true, hence continues waiting
    
    The only way to get the hung task going again is to trigger a hotplug
    event which brings down the slot, e.g. by yanking out the card.
    
    The same race exists when powering down the slot because remove_board()
    likewise clears link or presence changes in ctrl->pending_events per commit
    3943af9d ("PCI: pciehp: Ignore Link State Changes after powering off a
    slot") and thereby may cause a re-run of pciehp_ist() which returns with
    IRQ_NONE without resetting ctrl->ist_running to false.
    
    Fix by adding a goto label before the teardown steps at the end of
    pciehp_ist() and jumping to that label from the two return statements which
    currently neglect to reset the ctrl->ist_running flag.
    
    Fixes: 157c1062 ("PCI: pciehp: Avoid returning prematurely from sysfs requests")
    Link: https://lore.kernel.org/r/cca1effa488065cb055120aa01b65719094bdcb5.1584530321.git.lukas@wunner.deReported-by: default avatarDavid Hoyer <David.Hoyer@netapp.com>
    Signed-off-by: default avatarLukas Wunner <lukas@wunner.de>
    Signed-off-by: default avatarBjorn Helgaas <bhelgaas@google.com>
    Reviewed-by: default avatarKeith Busch <kbusch@kernel.org>
    Cc: stable@vger.kernel.org	# v4.19+
    3e487d2e
pciehp_hpc.c 26.9 KB