Documentation: PCI: Add background on Boot Interrupts

Improve understanding of the PCI quirks for this legacy PCI interrupt
behavior to the benefit of developers and users alike.

Link: https://lore.kernel.org/r/20200220192930.64820-3-sean.v.kelley@linux.intel.comSigned-off-by: Sean V Kelley <sean.v.kelley@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>

Documentation/PCI/boot-interrupts.rst

+.. SPDX-License-Identifier: GPL-2.0
+
+===============
+Boot Interrupts
+===============
+
+:Author: - Sean V Kelley <sean.v.kelley@linux.intel.com>
+
+Overview
+========
+
+On PCI Express, interrupts are represented with either MSI or inbound
+interrupt messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in a
+given Core IO converts the legacy interrupt messages from PCI Express to
+MSI interrupts.  If the IO-APIC is disabled (via the mask bits in the
+IO-APIC table entries), the messages are routed to the legacy PCH. This
+in-band interrupt mechanism was traditionally necessary for systems that
+did not support the IO-APIC and for boot. Intel in the past has used the
+term "boot interrupts" to describe this mechanism. Further, the PCI Express
+protocol describes this in-band legacy wire-interrupt INTx mechanism for
+I/O devices to signal PCI-style level interrupts. The subsequent paragraphs
+describe problems with the Core IO handling of INTx message routing to the
+PCH and mitigation within BIOS and the OS.
+
+
+Issue
+=====
+
+When in-band legacy INTx messages are forwarded to the PCH, they in turn
+trigger a new interrupt for which the OS likely lacks a handler. When an
+interrupt goes unhandled over time, they are tracked by the Linux kernel as
+Spurious Interrupts. The IRQ will be disabled by the Linux kernel after it
+reaches a specific count with the error "nobody cared". This disabled IRQ
+now prevents valid usage by an existing interrupt which may happen to share
+the IRQ line.
+
+  irq 19: nobody cared (try booting with the "irqpoll" option)
+  CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #1
+  Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020
+  Call Trace:
+  <IRQ>
+   ? dump_stack+0x46/0x5e
+   ? __report_bad_irq+0x2e/0xb0
+   ? note_interrupt+0x242/0x290
+   ? nNIKAL100_memoryRead16+0x8/0x10 [nikal]
+   ? handle_irq_event_percpu+0x55/0x70
+   ? handle_irq_event+0x4f/0x80
+   ? handle_fasteoi_irq+0x81/0x180
+   ? handle_irq+0x1c/0x30
+   ? do_IRQ+0x41/0xd0
+   ? common_interrupt+0x84/0x84
+  </IRQ>
+
+  handlers:
+  irq_default_primary_handler threaded usb_hcd_irq
+  Disabling IRQ #19
+
+
+Conditions
+==========
+
+The use of threaded interrupts is the most likely condition to trigger
+this problem today. Threaded interrupts may not be reenabled after the IRQ
+handler wakes. These "one shot" conditions mean that the threaded interrupt
+needs to keep the interrupt line masked until the threaded handler has run.
+Especially when dealing with high data rate interrupts, the thread needs to
+run to completion; otherwise some handlers will end up in stack overflows
+since the interrupt of the issuing device is still active.
+
+Affected Chipsets
+=================
+
+The legacy interrupt forwarding mechanism exists today in a number of
+devices including but not limited to chipsets from AMD/ATI, Broadcom, and
+Intel. Changes made through the mitigations below have been applied to
+drivers/pci/quirks.c
+
+Starting with ICX there are no longer any IO-APICs in the Core IO's
+devices.  IO-APIC is only in the PCH.  Devices connected to the Core IO's
+PCIe Root Ports will use native MSI/MSI-X mechanisms.
+
+Mitigations
+===========
+
+The mitigations take the form of PCI quirks. The preference has been to
+first identify and make use of a means to disable the routing to the PCH.
+In such a case a quirk to disable boot interrupt generation can be
+added.[1]
+
+  Intel® 6300ESB I/O Controller Hub
+  Alternate Base Address Register:
+   BIE: Boot Interrupt Enable
+	  0 = Boot interrupt is enabled.
+	  1 = Boot interrupt is disabled.
+
+  Intel® Sandy Bridge through Sky Lake based Xeon servers:
+  Coherent Interface Protocol Interrupt Control
+   dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2:
+	  When this bit is set. Local INTx messages received from the
+	  Intel® Quick Data DMA/PCI Express ports are not routed to legacy
+	  PCH - they are either converted into MSI via the integrated IO-APIC
+	  (if the IO-APIC mask bit is clear in the appropriate entries)
+	  or cause no further action (when mask bit is set)
+
+In the absence of a way to directly disable the routing, another approach
+has been to make use of PCI Interrupt pin to INTx routing tables for
+purposes of redirecting the interrupt handler to the rerouted interrupt
+line by default.  Therefore, on chipsets where this INTx routing cannot be
+disabled, the Linux kernel will reroute the valid interrupt to its legacy
+interrupt. This redirection of the handler will prevent the occurrence of
+the spurious interrupt detection which would ordinarily disable the IRQ
+line due to excessive unhandled counts.[2]
+
+The config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable (or
+disable) the redirection of the interrupt handler to the PCH interrupt
+line. The option can be overridden by either pci=ioapicreroute or
+pci=noioapicreroute.[3]
+
+
+More Documentation
+==================
+
+There is an overview of the legacy interrupt handling in several datasheets
+(6300ESB and 6700PXH below). While largely the same, it provides insight
+into the evolution of its handling with chipsets.
+
+Example of disabling of the boot interrupt
+------------------------------------------
+
+Intel® 6300ESB I/O Controller Hub (Document # 300641-004US)
+	5.7.3 Boot Interrupt
+	https://www.intel.com/content/dam/doc/datasheet/6300esb-io-controller-hub-datasheet.pdf
+
+Intel® Xeon® Processor E5-1600/2400/2600/4600 v3 Product Families
+Datasheet - Volume 2: Registers (Document # 330784-003)
+	6.6.41 cipintrc Coherent Interface Protocol Interrupt Control
+	https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf
+
+Example of handler rerouting
+----------------------------
+
+Intel® 6700PXH 64-bit PCI Hub (Document # 302628)
+	2.15.2 PCI Express Legacy INTx Support and Boot Interrupt
+	https://www.intel.com/content/dam/doc/datasheet/6700pxh-64-bit-pci-hub-datasheet.pdf
+
+
+If you have any legacy PCI interrupt questions that aren't answered, email me.
+
+Cheers,
+    Sean V Kelley
+    sean.v.kelley@linux.intel.com
+
+[1] https://lore.kernel.org/r/12131949181903-git-send-email-sassmann@suse.de/
+[2] https://lore.kernel.org/r/12131949182094-git-send-email-sassmann@suse.de/
+[3] https://lore.kernel.org/r/487C8EA7.6020205@suse.de/

Documentation/PCI/index.rst

@@ -16,3 +16,4 @@ Linux PCI Bus Subsystem
    pci-error-recovery
    pcieaer-howto
    endpoint/index
+   boot-interrupts