Commit fb4da215 authored by Linus Torvalds's avatar Linus Torvalds

Merge tag 'pci-v5.3-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci

Pull PCI updates from Bjorn Helgaas:
 "Enumeration changes:

   - Evaluate PCI Boot Configuration _DSM to learn if firmware wants us
     to preserve its resource assignments (Benjamin Herrenschmidt)

   - Simplify resource distribution (Nicholas Johnson)

   - Decode 32 GT/s link speed (Gustavo Pimentel)

  Virtualization:

   - Fix incorrect caching of VF config space size (Alex Williamson)

   - Fix VF driver probing sysfs knobs (Alex Williamson)

  Peer-to-peer DMA:

   - Fix dma_virt_ops check (Logan Gunthorpe)

  Altera host bridge driver:

   - Allow building as module (Ley Foon Tan)

  Armada 8K host bridge driver:

   - add PHYs support (Miquel Raynal)

  DesignWare host bridge driver:

   - Export APIs to support removable loadable module (Vidya Sagar)

   - Enable Relaxed Ordering erratum workaround only on Tegra20 &
     Tegra30 (Vidya Sagar)

  Hyper-V host bridge driver:

   - Fix use-after-free in eject (Dexuan Cui)

  Mobiveil host bridge driver:

   - Clean up and fix many issues, including non-identify mapped
     windows, 64-bit windows, multi-MSI, class code, INTx clearing (Hou
     Zhiqiang)

  Qualcomm host bridge driver:

   - Use clk bulk API for 2.4.0 controllers (Bjorn Andersson)

   - Add QCS404 support (Bjorn Andersson)

   - Assert PERST for at least 100ms (Niklas Cassel)

  R-Car host bridge driver:

   - Add r8a774a1 DT support (Biju Das)

  Tegra host bridge driver:

   - Add support for Gen2, opportunistic UpdateFC and ACK (PCIe protocol
     details) AER, GPIO-based PERST# (Manikanta Maddireddy)

   - Fix many issues, including power-on failure cases, interrupt
     masking in suspend, UPHY settings, AFI dynamic clock gating,
     pending DLL transactions (Manikanta Maddireddy)

  Xilinx host bridge driver:

   - Fix NWL Multi-MSI programming (Bharat Kumar Gogada)

  Endpoint support:

   - Fix 64bit BAR support (Alan Mikhak)

   - Fix pcitest build issues (Alan Mikhak, Andy Shevchenko)

  Bug fixes:

   - Fix NVIDIA GPU multi-function power dependencies (Abhishek Sahu)

   - Fix NVIDIA GPU HDA enablement issue (Lukas Wunner)

   - Ignore lockdep for sysfs "remove" (Marek Vasut)

  Misc:

   - Convert docs to reST (Changbin Du, Mauro Carvalho Chehab)"

* tag 'pci-v5.3-changes' of git://git.kernel.org/pub/scm/linux/kernel/git/helgaas/pci: (107 commits)
  PCI: Enable NVIDIA HDA controllers
  tools: PCI: Fix installation when `make tools/pci_install`
  PCI: dwc: pci-dra7xx: Fix compilation when !CONFIG_GPIOLIB
  PCI: Fix typos and whitespace errors
  PCI: mobiveil: Fix INTx interrupt clearing in mobiveil_pcie_isr()
  PCI: mobiveil: Fix infinite-loop in the INTx handling function
  PCI: mobiveil: Move PCIe PIO enablement out of inbound window routine
  PCI: mobiveil: Add upper 32-bit PCI base address setup in inbound window
  PCI: mobiveil: Add upper 32-bit CPU base address setup in outbound window
  PCI: mobiveil: Mask out hardcoded bits in inbound/outbound windows setup
  PCI: mobiveil: Clear the control fields before updating it
  PCI: mobiveil: Add configured inbound windows counter
  PCI: mobiveil: Fix the valid check for inbound and outbound windows
  PCI: mobiveil: Clean-up program_{ib/ob}_windows()
  PCI: mobiveil: Remove an unnecessary return value check
  PCI: mobiveil: Fix error return values
  PCI: mobiveil: Refactor the MEM/IO outbound window initialization
  PCI: mobiveil: Make some register updates more readable
  PCI: mobiveil: Reformat the code for readability
  dt-bindings: PCI: mobiveil: Change gpio_slave and apb_csr to optional
  ...
parents 2a3c389a 7b4b0f6b
......@@ -5,7 +5,7 @@ Contact: linux-pm@vger.kernel.org
Description:
The powercap/ class sub directory belongs to the power cap
subsystem. Refer to
Documentation/power/powercap/powercap.txt for details.
Documentation/power/powercap/powercap.rst for details.
What: /sys/class/powercap/<control type>
Date: September 2013
......
ACPI considerations for PCI host bridges
.. SPDX-License-Identifier: GPL-2.0
========================================
ACPI considerations for PCI host bridges
========================================
The general rule is that the ACPI namespace should describe everything the
OS might use unless there's another way for the OS to find it [1, 2].
......@@ -131,12 +135,13 @@ address always corresponds to bus 0, even if the bus range below the bridge
[4] ACPI 6.2, sec 6.4.3.5.1, 2, 3, 4:
QWord/DWord/Word Address Space Descriptor (.1, .2, .3)
General Flags: Bit [0] Ignored
General Flags: Bit [0] Ignored
Extended Address Space Descriptor (.4)
General Flags: Bit [0] Consumer/Producer:
1–This device consumes this resource
0–This device produces and consumes this resource
General Flags: Bit [0] Consumer/Producer:
* 1 – This device consumes this resource
* 0 – This device produces and consumes this resource
[5] ACPI 6.2, sec 19.6.43:
ResourceUsage specifies whether the Memory range is consumed by
......
.. SPDX-License-Identifier: GPL-2.0
======================
PCI Endpoint Framework
======================
.. toctree::
:maxdepth: 2
pci-endpoint
pci-endpoint-cfs
pci-test-function
pci-test-howto
CONFIGURING PCI ENDPOINT USING CONFIGFS
Kishon Vijay Abraham I <kishon@ti.com>
.. SPDX-License-Identifier: GPL-2.0
=======================================
Configuring PCI Endpoint Using CONFIGFS
=======================================
:Author: Kishon Vijay Abraham I <kishon@ti.com>
The PCI Endpoint Core exposes configfs entry (pci_ep) to configure the
PCI endpoint function and to bind the endpoint function
with the endpoint controller. (For introducing other mechanisms to
configure the PCI Endpoint Function refer to [1]).
*) Mounting configfs
Mounting configfs
=================
The PCI Endpoint Core layer creates pci_ep directory in the mounted configfs
directory. configfs can be mounted using the following command.
directory. configfs can be mounted using the following command::
mount -t configfs none /sys/kernel/config
*) Directory Structure
Directory Structure
===================
The pci_ep configfs has two directories at its root: controllers and
functions. Every EPC device present in the system will have an entry in
the *controllers* directory and and every EPF driver present in the system
will have an entry in the *functions* directory.
::
/sys/kernel/config/pci_ep/
.. controllers/
.. functions/
/sys/kernel/config/pci_ep/
.. controllers/
.. functions/
*) Creating EPF Device
Creating EPF Device
===================
Every registered EPF driver will be listed in controllers directory. The
entries corresponding to EPF driver will be created by the EPF core.
::
/sys/kernel/config/pci_ep/functions/
.. <EPF Driver1>/
... <EPF Device 11>/
... <EPF Device 21>/
.. <EPF Driver2>/
... <EPF Device 12>/
... <EPF Device 22>/
/sys/kernel/config/pci_ep/functions/
.. <EPF Driver1>/
... <EPF Device 11>/
... <EPF Device 21>/
.. <EPF Driver2>/
... <EPF Device 12>/
... <EPF Device 22>/
In order to create a <EPF device> of the type probed by <EPF Driver>, the
user has to create a directory inside <EPF DriverN>.
......@@ -44,34 +54,37 @@ Every <EPF device> directory consists of the following entries that can be
used to configure the standard configuration header of the endpoint function.
(These entries are created by the framework when any new <EPF Device> is
created)
.. <EPF Driver1>/
... <EPF Device 11>/
... vendorid
... deviceid
... revid
... progif_code
... subclass_code
... baseclass_code
... cache_line_size
... subsys_vendor_id
... subsys_id
... interrupt_pin
*) EPC Device
::
.. <EPF Driver1>/
... <EPF Device 11>/
... vendorid
... deviceid
... revid
... progif_code
... subclass_code
... baseclass_code
... cache_line_size
... subsys_vendor_id
... subsys_id
... interrupt_pin
EPC Device
==========
Every registered EPC device will be listed in controllers directory. The
entries corresponding to EPC device will be created by the EPC core.
/sys/kernel/config/pci_ep/controllers/
.. <EPC Device1>/
... <Symlink EPF Device11>/
... <Symlink EPF Device12>/
... start
.. <EPC Device2>/
... <Symlink EPF Device21>/
... <Symlink EPF Device22>/
... start
::
/sys/kernel/config/pci_ep/controllers/
.. <EPC Device1>/
... <Symlink EPF Device11>/
... <Symlink EPF Device12>/
... start
.. <EPC Device2>/
... <Symlink EPF Device21>/
... <Symlink EPF Device22>/
... start
The <EPC Device> directory will have a list of symbolic links to
<EPF Device>. These symbolic links should be created by the user to
......@@ -81,7 +94,7 @@ The <EPC Device> directory will also have a *start* field. Once
"1" is written to this field, the endpoint device will be ready to
establish the link with the host. This is usually done after
all the EPF devices are created and linked with the EPC device.
::
| controllers/
| <Directory: EPC name>/
......@@ -102,4 +115,4 @@ all the EPF devices are created and linked with the EPC device.
| interrupt_pin
| function
[1] -> Documentation/PCI/endpoint/pci-endpoint.txt
[1] :doc:`pci-endpoint`
PCI ENDPOINT FRAMEWORK
Kishon Vijay Abraham I <kishon@ti.com>
.. SPDX-License-Identifier: GPL-2.0
:Author: Kishon Vijay Abraham I <kishon@ti.com>
This document is a guide to use the PCI Endpoint Framework in order to create
endpoint controller driver, endpoint function driver, and using configfs
interface to bind the function driver to the controller driver.
1. Introduction
Introduction
============
Linux has a comprehensive PCI subsystem to support PCI controllers that
operates in Root Complex mode. The subsystem has capability to scan PCI bus,
......@@ -19,26 +21,30 @@ add endpoint mode support in Linux. This will help to run Linux in an
EP system which can have a wide variety of use cases from testing or
validation, co-processor accelerator, etc.
2. PCI Endpoint Core
PCI Endpoint Core
=================
The PCI Endpoint Core layer comprises 3 components: the Endpoint Controller
library, the Endpoint Function library, and the configfs layer to bind the
endpoint function with the endpoint controller.
2.1 PCI Endpoint Controller(EPC) Library
PCI Endpoint Controller(EPC) Library
------------------------------------
The EPC library provides APIs to be used by the controller that can operate
in endpoint mode. It also provides APIs to be used by function driver/library
in order to implement a particular endpoint function.
2.1.1 APIs for the PCI controller Driver
APIs for the PCI controller Driver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This section lists the APIs that the PCI Endpoint core provides to be used
by the PCI controller driver.
*) devm_pci_epc_create()/pci_epc_create()
* devm_pci_epc_create()/pci_epc_create()
The PCI controller driver should implement the following ops:
* write_header: ops to populate configuration space header
* set_bar: ops to configure the BAR
* clear_bar: ops to reset the BAR
......@@ -51,110 +57,116 @@ by the PCI controller driver.
The PCI controller driver can then create a new EPC device by invoking
devm_pci_epc_create()/pci_epc_create().
*) devm_pci_epc_destroy()/pci_epc_destroy()
* devm_pci_epc_destroy()/pci_epc_destroy()
The PCI controller driver can destroy the EPC device created by either
devm_pci_epc_create() or pci_epc_create() using devm_pci_epc_destroy() or
pci_epc_destroy().
*) pci_epc_linkup()
* pci_epc_linkup()
In order to notify all the function devices that the EPC device to which
they are linked has established a link with the host, the PCI controller
driver should invoke pci_epc_linkup().
*) pci_epc_mem_init()
* pci_epc_mem_init()
Initialize the pci_epc_mem structure used for allocating EPC addr space.
*) pci_epc_mem_exit()
* pci_epc_mem_exit()
Cleanup the pci_epc_mem structure allocated during pci_epc_mem_init().
2.1.2 APIs for the PCI Endpoint Function Driver
APIs for the PCI Endpoint Function Driver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This section lists the APIs that the PCI Endpoint core provides to be used
by the PCI endpoint function driver.
*) pci_epc_write_header()
* pci_epc_write_header()
The PCI endpoint function driver should use pci_epc_write_header() to
write the standard configuration header to the endpoint controller.
*) pci_epc_set_bar()
* pci_epc_set_bar()
The PCI endpoint function driver should use pci_epc_set_bar() to configure
the Base Address Register in order for the host to assign PCI addr space.
Register space of the function driver is usually configured
using this API.
*) pci_epc_clear_bar()
* pci_epc_clear_bar()
The PCI endpoint function driver should use pci_epc_clear_bar() to reset
the BAR.
*) pci_epc_raise_irq()
* pci_epc_raise_irq()
The PCI endpoint function driver should use pci_epc_raise_irq() to raise
Legacy Interrupt, MSI or MSI-X Interrupt.
*) pci_epc_mem_alloc_addr()
* pci_epc_mem_alloc_addr()
The PCI endpoint function driver should use pci_epc_mem_alloc_addr(), to
allocate memory address from EPC addr space which is required to access
RC's buffer
*) pci_epc_mem_free_addr()
* pci_epc_mem_free_addr()
The PCI endpoint function driver should use pci_epc_mem_free_addr() to
free the memory space allocated using pci_epc_mem_alloc_addr().
2.1.3 Other APIs
Other APIs
~~~~~~~~~~
There are other APIs provided by the EPC library. These are used for binding
the EPF device with EPC device. pci-ep-cfs.c can be used as reference for
using these APIs.
*) pci_epc_get()
* pci_epc_get()
Get a reference to the PCI endpoint controller based on the device name of
the controller.
*) pci_epc_put()
* pci_epc_put()
Release the reference to the PCI endpoint controller obtained using
pci_epc_get()
*) pci_epc_add_epf()
* pci_epc_add_epf()
Add a PCI endpoint function to a PCI endpoint controller. A PCIe device
can have up to 8 functions according to the specification.
*) pci_epc_remove_epf()
* pci_epc_remove_epf()
Remove the PCI endpoint function from PCI endpoint controller.
*) pci_epc_start()
* pci_epc_start()
The PCI endpoint function driver should invoke pci_epc_start() once it
has configured the endpoint function and wants to start the PCI link.
*) pci_epc_stop()
* pci_epc_stop()
The PCI endpoint function driver should invoke pci_epc_stop() to stop
the PCI LINK.
2.2 PCI Endpoint Function(EPF) Library
PCI Endpoint Function(EPF) Library
----------------------------------
The EPF library provides APIs to be used by the function driver and the EPC
library to provide endpoint mode functionality.
2.2.1 APIs for the PCI Endpoint Function Driver
APIs for the PCI Endpoint Function Driver
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This section lists the APIs that the PCI Endpoint core provides to be used
by the PCI endpoint function driver.
*) pci_epf_register_driver()
* pci_epf_register_driver()
The PCI Endpoint Function driver should implement the following ops:
* bind: ops to perform when a EPC device has been bound to EPF device
......@@ -166,50 +178,54 @@ by the PCI endpoint function driver.
The PCI Function driver can then register the PCI EPF driver by using
pci_epf_register_driver().
*) pci_epf_unregister_driver()
* pci_epf_unregister_driver()
The PCI Function driver can unregister the PCI EPF driver by using
pci_epf_unregister_driver().
*) pci_epf_alloc_space()
* pci_epf_alloc_space()
The PCI Function driver can allocate space for a particular BAR using
pci_epf_alloc_space().
*) pci_epf_free_space()
* pci_epf_free_space()
The PCI Function driver can free the allocated space
(using pci_epf_alloc_space) by invoking pci_epf_free_space().
2.2.2 APIs for the PCI Endpoint Controller Library
APIs for the PCI Endpoint Controller Library
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This section lists the APIs that the PCI Endpoint core provides to be used
by the PCI endpoint controller library.
*) pci_epf_linkup()
* pci_epf_linkup()
The PCI endpoint controller library invokes pci_epf_linkup() when the
EPC device has established the connection to the host.
2.2.2 Other APIs
Other APIs
~~~~~~~~~~
There are other APIs provided by the EPF library. These are used to notify
the function driver when the EPF device is bound to the EPC device.
pci-ep-cfs.c can be used as reference for using these APIs.
*) pci_epf_create()
* pci_epf_create()
Create a new PCI EPF device by passing the name of the PCI EPF device.
This name will be used to bind the the EPF device to a EPF driver.
*) pci_epf_destroy()
* pci_epf_destroy()
Destroy the created PCI EPF device.
*) pci_epf_bind()
* pci_epf_bind()
pci_epf_bind() should be invoked when the EPF device has been bound to
a EPC device.
*) pci_epf_unbind()
* pci_epf_unbind()
pci_epf_unbind() should be invoked when the binding between EPC device
and EPF device is lost.
PCI TEST
Kishon Vijay Abraham I <kishon@ti.com>
.. SPDX-License-Identifier: GPL-2.0
=================
PCI Test Function
=================
:Author: Kishon Vijay Abraham I <kishon@ti.com>
Traditionally PCI RC has always been validated by using standard
PCI cards like ethernet PCI cards or USB PCI cards or SATA PCI cards.
......@@ -23,65 +28,76 @@ The PCI endpoint test device has the following registers:
8) PCI_ENDPOINT_TEST_IRQ_TYPE
9) PCI_ENDPOINT_TEST_IRQ_NUMBER
*) PCI_ENDPOINT_TEST_MAGIC
* PCI_ENDPOINT_TEST_MAGIC
This register will be used to test BAR0. A known pattern will be written
and read back from MAGIC register to verify BAR0.
*) PCI_ENDPOINT_TEST_COMMAND:
* PCI_ENDPOINT_TEST_COMMAND
This register will be used by the host driver to indicate the function
that the endpoint device must perform.
Bitfield Description:
Bit 0 : raise legacy IRQ
Bit 1 : raise MSI IRQ
Bit 2 : raise MSI-X IRQ
Bit 3 : read command (read data from RC buffer)
Bit 4 : write command (write data to RC buffer)
Bit 5 : copy command (copy data from one RC buffer to another
RC buffer)
======== ================================================================
Bitfield Description
======== ================================================================
Bit 0 raise legacy IRQ
Bit 1 raise MSI IRQ
Bit 2 raise MSI-X IRQ
Bit 3 read command (read data from RC buffer)
Bit 4 write command (write data to RC buffer)
Bit 5 copy command (copy data from one RC buffer to another RC buffer)
======== ================================================================
*) PCI_ENDPOINT_TEST_STATUS
* PCI_ENDPOINT_TEST_STATUS
This register reflects the status of the PCI endpoint device.
Bitfield Description:
Bit 0 : read success
Bit 1 : read fail
Bit 2 : write success
Bit 3 : write fail
Bit 4 : copy success
Bit 5 : copy fail
Bit 6 : IRQ raised
Bit 7 : source address is invalid
Bit 8 : destination address is invalid
*) PCI_ENDPOINT_TEST_SRC_ADDR
======== ==============================
Bitfield Description
======== ==============================
Bit 0 read success
Bit 1 read fail
Bit 2 write success
Bit 3 write fail
Bit 4 copy success
Bit 5 copy fail
Bit 6 IRQ raised
Bit 7 source address is invalid
Bit 8 destination address is invalid
======== ==============================
* PCI_ENDPOINT_TEST_SRC_ADDR
This register contains the source address (RC buffer address) for the
COPY/READ command.
*) PCI_ENDPOINT_TEST_DST_ADDR
* PCI_ENDPOINT_TEST_DST_ADDR
This register contains the destination address (RC buffer address) for
the COPY/WRITE command.
*) PCI_ENDPOINT_TEST_IRQ_TYPE
* PCI_ENDPOINT_TEST_IRQ_TYPE
This register contains the interrupt type (Legacy/MSI) triggered
for the READ/WRITE/COPY and raise IRQ (Legacy/MSI) commands.
Possible types:
- Legacy : 0
- MSI : 1
- MSI-X : 2
*) PCI_ENDPOINT_TEST_IRQ_NUMBER
====== ==
Legacy 0
MSI 1
MSI-X 2
====== ==
* PCI_ENDPOINT_TEST_IRQ_NUMBER
This register contains the triggered ID interrupt.
Admissible values:
- Legacy : 0
- MSI : [1 .. 32]
- MSI-X : [1 .. 2048]
====== ===========
Legacy 0
MSI [1 .. 32]
MSI-X [1 .. 2048]
====== ===========
PCI TEST USERGUIDE
Kishon Vijay Abraham I <kishon@ti.com>
.. SPDX-License-Identifier: GPL-2.0
===================
PCI Test User Guide
===================
:Author: Kishon Vijay Abraham I <kishon@ti.com>
This document is a guide to help users use pci-epf-test function driver
and pci_endpoint_test host driver for testing PCI. The list of steps to
be followed in the host side and EP side is given below.
1. Endpoint Device
Endpoint Device
===============
1.1 Endpoint Controller Devices
Endpoint Controller Devices
---------------------------
To find the list of endpoint controller devices in the system:
To find the list of endpoint controller devices in the system::
# ls /sys/class/pci_epc/
51000000.pcie_ep
If PCI_ENDPOINT_CONFIGFS is enabled
If PCI_ENDPOINT_CONFIGFS is enabled::
# ls /sys/kernel/config/pci_ep/controllers
51000000.pcie_ep
1.2 Endpoint Function Drivers
To find the list of endpoint function drivers in the system:
Endpoint Function Drivers
-------------------------
To find the list of endpoint function drivers in the system::
# ls /sys/bus/pci-epf/drivers
pci_epf_test
If PCI_ENDPOINT_CONFIGFS is enabled
If PCI_ENDPOINT_CONFIGFS is enabled::
# ls /sys/kernel/config/pci_ep/functions
pci_epf_test
1.3 Creating pci-epf-test Device
Creating pci-epf-test Device
----------------------------
PCI endpoint function device can be created using the configfs. To create
pci-epf-test device, the following commands can be used
pci-epf-test device, the following commands can be used::
# mount -t configfs none /sys/kernel/config
# cd /sys/kernel/config/pci_ep/
......@@ -42,7 +55,7 @@ The "mkdir func1" above creates the pci-epf-test function device that will
be probed by pci_epf_test driver.
The PCI endpoint framework populates the directory with the following
configurable fields.
configurable fields::
# ls functions/pci_epf_test/func1
baseclass_code interrupt_pin progif_code subsys_id
......@@ -51,67 +64,83 @@ configurable fields.
The PCI endpoint function driver populates these entries with default values
when the device is bound to the driver. The pci-epf-test driver populates
vendorid with 0xffff and interrupt_pin with 0x0001
vendorid with 0xffff and interrupt_pin with 0x0001::
# cat functions/pci_epf_test/func1/vendorid
0xffff
# cat functions/pci_epf_test/func1/interrupt_pin
0x0001
1.4 Configuring pci-epf-test Device
Configuring pci-epf-test Device
-------------------------------
The user can configure the pci-epf-test device using configfs entry. In order
to change the vendorid and the number of MSI interrupts used by the function
device, the following commands can be used.
device, the following commands can be used::
# echo 0x104c > functions/pci_epf_test/func1/vendorid
# echo 0xb500 > functions/pci_epf_test/func1/deviceid
# echo 16 > functions/pci_epf_test/func1/msi_interrupts
# echo 8 > functions/pci_epf_test/func1/msix_interrupts
1.5 Binding pci-epf-test Device to EP Controller
Binding pci-epf-test Device to EP Controller
--------------------------------------------
In order for the endpoint function device to be useful, it has to be bound to
a PCI endpoint controller driver. Use the configfs to bind the function
device to one of the controller driver present in the system.
device to one of the controller driver present in the system::
# ln -s functions/pci_epf_test/func1 controllers/51000000.pcie_ep/
Once the above step is completed, the PCI endpoint is ready to establish a link
with the host.
1.6 Start the Link
Start the Link
--------------
In order for the endpoint device to establish a link with the host, the _start_
field should be populated with '1'.
field should be populated with '1'::
# echo 1 > controllers/51000000.pcie_ep/start
2. RootComplex Device
2.1 lspci Output
RootComplex Device
==================
lspci Output
------------
Note that the devices listed here correspond to the value populated in 1.4 above
Note that the devices listed here correspond to the value populated in 1.4
above::
00:00.0 PCI bridge: Texas Instruments Device 8888 (rev 01)
01:00.0 Unassigned class [ff00]: Texas Instruments Device b500
2.2 Using Endpoint Test function Device
Using Endpoint Test function Device
-----------------------------------
pcitest.sh added in tools/pci/ can be used to run all the default PCI endpoint
tests. To compile this tool the following commands should be used:
tests. To compile this tool the following commands should be used::
# cd <kernel-dir>
# make -C tools/pci
or if you desire to compile and install in your system:
or if you desire to compile and install in your system::
# cd <kernel-dir>
# make -C tools/pci install
The tool and script will be located in <rootfs>/usr/bin/
2.2.1 pcitest.sh Output
pcitest.sh Output
~~~~~~~~~~~~~~~~~
::
# pcitest.sh
BAR tests
......
.. SPDX-License-Identifier: GPL-2.0
=======================
Linux PCI Bus Subsystem
=======================
.. toctree::
:maxdepth: 2
:numbered:
pci
picebus-howto
pci-iov-howto
msi-howto
acpi-info
pci-error-recovery
pcieaer-howto
endpoint/index
The MSI Driver Guide HOWTO
Tom L Nguyen tom.l.nguyen@intel.com
10/03/2003
Revised Feb 12, 2004 by Martine Silbermann
email: Martine.Silbermann@hp.com
Revised Jun 25, 2004 by Tom L Nguyen
Revised Jul 9, 2008 by Matthew Wilcox <willy@linux.intel.com>
Copyright 2003, 2008 Intel Corporation
.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>
1. About this guide
==========================
The MSI Driver Guide HOWTO
==========================
:Authors: Tom L Nguyen; Martine Silbermann; Matthew Wilcox
:Copyright: 2003, 2008 Intel Corporation
About this guide
================
This guide describes the basics of Message Signaled Interrupts (MSIs),
the advantages of using MSI over traditional interrupt mechanisms, how
......@@ -15,7 +18,8 @@ to change your driver to use MSI or MSI-X and some basic diagnostics to
try if a device doesn't support MSIs.
2. What are MSIs?
What are MSIs?
==============
A Message Signaled Interrupt is a write from the device to a special
address which causes an interrupt to be received by the CPU.
......@@ -29,7 +33,8 @@ Devices may support both MSI and MSI-X, but only one can be enabled at
a time.
3. Why use MSIs?
Why use MSIs?
=============
There are three reasons why using MSIs can give an advantage over
traditional pin-based interrupts.
......@@ -61,14 +66,16 @@ Other possible designs include giving one interrupt to each packet queue
in a network card or each port in a storage controller.
4. How to use MSIs
How to use MSIs
===============
PCI devices are initialised to use pin-based interrupts. The device
driver has to set up the device to use MSI or MSI-X. Not all machines
support MSIs correctly, and for those machines, the APIs described below
will simply fail and the device will continue to use pin-based interrupts.
4.1 Include kernel support for MSIs
Include kernel support for MSIs
-------------------------------
To support MSI or MSI-X, the kernel must be built with the CONFIG_PCI_MSI
option enabled. This option is only available on some architectures,
......@@ -76,14 +83,15 @@ and it may depend on some other options also being set. For example,
on x86, you must also enable X86_UP_APIC or SMP in order to see the
CONFIG_PCI_MSI option.
4.2 Using MSI
Using MSI
---------
Most of the hard work is done for the driver in the PCI layer. The driver
simply has to request that the PCI layer set up the MSI capability for this
device.
To automatically use MSI or MSI-X interrupt vectors, use the following
function:
function::
int pci_alloc_irq_vectors(struct pci_dev *dev, unsigned int min_vecs,
unsigned int max_vecs, unsigned int flags);
......@@ -101,12 +109,12 @@ any possible kind of interrupt. If the PCI_IRQ_AFFINITY flag is set,
pci_alloc_irq_vectors() will spread the interrupts around the available CPUs.
To get the Linux IRQ numbers passed to request_irq() and free_irq() and the
vectors, use the following function:
vectors, use the following function::
int pci_irq_vector(struct pci_dev *dev, unsigned int nr);
Any allocated resources should be freed before removing the device using
the following function:
the following function::
void pci_free_irq_vectors(struct pci_dev *dev);
......@@ -126,7 +134,7 @@ The typical usage of MSI or MSI-X interrupts is to allocate as many vectors
as possible, likely up to the limit supported by the device. If nvec is
larger than the number supported by the device it will automatically be
capped to the supported limit, so there is no need to query the number of
vectors supported beforehand:
vectors supported beforehand::
nvec = pci_alloc_irq_vectors(pdev, 1, nvec, PCI_IRQ_ALL_TYPES)
if (nvec < 0)
......@@ -135,7 +143,7 @@ vectors supported beforehand:
If a driver is unable or unwilling to deal with a variable number of MSI
interrupts it can request a particular number of interrupts by passing that
number to pci_alloc_irq_vectors() function as both 'min_vecs' and
'max_vecs' parameters:
'max_vecs' parameters::
ret = pci_alloc_irq_vectors(pdev, nvec, nvec, PCI_IRQ_ALL_TYPES);
if (ret < 0)
......@@ -143,23 +151,24 @@ number to pci_alloc_irq_vectors() function as both 'min_vecs' and
The most notorious example of the request type described above is enabling
the single MSI mode for a device. It could be done by passing two 1s as
'min_vecs' and 'max_vecs':
'min_vecs' and 'max_vecs'::
ret = pci_alloc_irq_vectors(pdev, 1, 1, PCI_IRQ_ALL_TYPES);
if (ret < 0)
goto out_err;
Some devices might not support using legacy line interrupts, in which case
the driver can specify that only MSI or MSI-X is acceptable:
the driver can specify that only MSI or MSI-X is acceptable::
nvec = pci_alloc_irq_vectors(pdev, 1, nvec, PCI_IRQ_MSI | PCI_IRQ_MSIX);
if (nvec < 0)
goto out_err;
4.3 Legacy APIs
Legacy APIs
-----------
The following old APIs to enable and disable MSI or MSI-X interrupts should
not be used in new code:
not be used in new code::
pci_enable_msi() /* deprecated */
pci_disable_msi() /* deprecated */
......@@ -174,9 +183,11 @@ number of vectors. If you have a legitimate special use case for the count
of vectors we might have to revisit that decision and add a
pci_nr_irq_vectors() helper that handles MSI and MSI-X transparently.
4.4 Considerations when using MSIs
Considerations when using MSIs
------------------------------
4.4.1 Spinlocks
Spinlocks
~~~~~~~~~
Most device drivers have a per-device spinlock which is taken in the
interrupt handler. With pin-based interrupts or a single MSI, it is not
......@@ -188,7 +199,8 @@ acquire the spinlock. Such deadlocks can be avoided by using
spin_lock_irqsave() or spin_lock_irq() which disable local interrupts
and acquire the lock (see Documentation/kernel-hacking/locking.rst).
4.5 How to tell whether MSI/MSI-X is enabled on a device
How to tell whether MSI/MSI-X is enabled on a device
----------------------------------------------------
Using 'lspci -v' (as root) may show some devices with "MSI", "Message
Signalled Interrupts" or "MSI-X" capabilities. Each of these capabilities
......@@ -196,7 +208,8 @@ has an 'Enable' flag which is followed with either "+" (enabled)
or "-" (disabled).
5. MSI quirks
MSI quirks
==========
Several PCI chipsets or devices are known not to support MSIs.
The PCI stack provides three ways to disable MSIs:
......@@ -205,7 +218,8 @@ The PCI stack provides three ways to disable MSIs:
2. on all devices behind a specific bridge
3. on a single device
5.1. Disabling MSIs globally
Disabling MSIs globally
-----------------------
Some host chipsets simply don't support MSIs properly. If we're
lucky, the manufacturer knows this and has indicated it in the ACPI
......@@ -219,7 +233,8 @@ on the kernel command line to disable MSIs on all devices. It would be
in your best interests to report the problem to linux-pci@vger.kernel.org
including a full 'lspci -v' so we can add the quirks to the kernel.
5.2. Disabling MSIs below a bridge
Disabling MSIs below a bridge
-----------------------------
Some PCI bridges are not able to route MSIs between busses properly.
In this case, MSIs must be disabled on all devices behind the bridge.
......@@ -230,7 +245,7 @@ as the nVidia nForce and Serverworks HT2000). As with host chipsets,
Linux mostly knows about them and automatically enables MSIs if it can.
If you have a bridge unknown to Linux, you can enable
MSIs in configuration space using whatever method you know works, then
enable MSIs on that bridge by doing:
enable MSIs on that bridge by doing::
echo 1 > /sys/bus/pci/devices/$bridge/msi_bus
......@@ -244,7 +259,8 @@ below this bridge.
Again, please notify linux-pci@vger.kernel.org of any bridges that need
special handling.
5.3. Disabling MSIs on a single device
Disabling MSIs on a single device
---------------------------------
Some devices are known to have faulty MSI implementations. Usually this
is handled in the individual device driver, but occasionally it's necessary
......@@ -252,7 +268,8 @@ to handle this with a quirk. Some drivers have an option to disable use
of MSI. While this is a convenient workaround for the driver author,
it is not good practice, and should not be emulated.
5.4. Finding why MSIs are disabled on a device
Finding why MSIs are disabled on a device
-----------------------------------------
From the above three sections, you can see that there are many reasons
why MSIs may not be enabled for a given device. Your first step should
......@@ -260,8 +277,8 @@ be to examine your dmesg carefully to determine whether MSIs are enabled
for your machine. You should also check your .config to be sure you
have enabled CONFIG_PCI_MSI.
Then, 'lspci -t' gives the list of bridges above a device. Reading
/sys/bus/pci/devices/*/msi_bus will tell you whether MSIs are enabled (1)
Then, 'lspci -t' gives the list of bridges above a device. Reading
`/sys/bus/pci/devices/*/msi_bus` will tell you whether MSIs are enabled (1)
or disabled (0). If 0 is found in any of the msi_bus files belonging
to bridges between the PCI root and the device, MSIs are disabled.
......
PCI Express I/O Virtualization Howto
Copyright (C) 2009 Intel Corporation
Yu Zhao <yu.zhao@intel.com>
.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>
Update: November 2012
-- sysfs-based SRIOV enable-/disable-ment
Donald Dutile <ddutile@redhat.com>
====================================
PCI Express I/O Virtualization Howto
====================================
1. Overview
:Copyright: |copy| 2009 Intel Corporation
:Authors: - Yu Zhao <yu.zhao@intel.com>
- Donald Dutile <ddutile@redhat.com>
1.1 What is SR-IOV
Overview
========
What is SR-IOV
--------------
Single Root I/O Virtualization (SR-IOV) is a PCI Express Extended
capability which makes one physical device appear as multiple virtual
......@@ -23,9 +28,11 @@ Memory Space, which is used to map its register set. VF device driver
operates on the register set so it can be functional and appear as a
real existing PCI device.
2. User Guide
User Guide
==========
2.1 How can I enable SR-IOV capability
How can I enable SR-IOV capability
----------------------------------
Multiple methods are available for SR-IOV enablement.
In the first method, the device driver (PF driver) will control the
......@@ -43,105 +50,123 @@ checks, e.g., check numvfs == 0 if enabling VFs, ensure
numvfs <= totalvfs.
The second method is the recommended method for new/future VF devices.
2.2 How can I use the Virtual Functions
How can I use the Virtual Functions
-----------------------------------
The VF is treated as hot-plugged PCI devices in the kernel, so they
should be able to work in the same way as real PCI devices. The VF
requires device driver that is same as a normal PCI device's.
3. Developer Guide
Developer Guide
===============
3.1 SR-IOV API
SR-IOV API
----------
To enable SR-IOV capability:
(a) For the first method, in the driver:
(a) For the first method, in the driver::
int pci_enable_sriov(struct pci_dev *dev, int nr_virtfn);
'nr_virtfn' is number of VFs to be enabled.
(b) For the second method, from sysfs:
'nr_virtfn' is number of VFs to be enabled.
(b) For the second method, from sysfs::
echo 'nr_virtfn' > \
/sys/bus/pci/devices/<DOMAIN:BUS:DEVICE.FUNCTION>/sriov_numvfs
To disable SR-IOV capability:
(a) For the first method, in the driver:
(a) For the first method, in the driver::
void pci_disable_sriov(struct pci_dev *dev);
(b) For the second method, from sysfs:
(b) For the second method, from sysfs::
echo 0 > \
/sys/bus/pci/devices/<DOMAIN:BUS:DEVICE.FUNCTION>/sriov_numvfs
To enable auto probing VFs by a compatible driver on the host, run
command below before enabling SR-IOV capabilities. This is the
default behavior.
::
echo 1 > \
/sys/bus/pci/devices/<DOMAIN:BUS:DEVICE.FUNCTION>/sriov_drivers_autoprobe
To disable auto probing VFs by a compatible driver on the host, run
command below before enabling SR-IOV capabilities. Updating this
entry will not affect VFs which are already probed.
::
echo 0 > \
/sys/bus/pci/devices/<DOMAIN:BUS:DEVICE.FUNCTION>/sriov_drivers_autoprobe
3.2 Usage example
Usage example
-------------
Following piece of code illustrates the usage of the SR-IOV API.
::
static int dev_probe(struct pci_dev *dev, const struct pci_device_id *id)
{
pci_enable_sriov(dev, NR_VIRTFN);
static int dev_probe(struct pci_dev *dev, const struct pci_device_id *id)
{
pci_enable_sriov(dev, NR_VIRTFN);
...
return 0;
}
...
static void dev_remove(struct pci_dev *dev)
{
pci_disable_sriov(dev);
return 0;
}
...
}
static void dev_remove(struct pci_dev *dev)
{
pci_disable_sriov(dev);
static int dev_suspend(struct pci_dev *dev, pm_message_t state)
{
...
...
}
return 0;
}
static int dev_suspend(struct pci_dev *dev, pm_message_t state)
{
...
static int dev_resume(struct pci_dev *dev)
{
...
return 0;
}
return 0;
}
static int dev_resume(struct pci_dev *dev)
{
...
static void dev_shutdown(struct pci_dev *dev)
{
...
}
return 0;
}
static int dev_sriov_configure(struct pci_dev *dev, int numvfs)
{
if (numvfs > 0) {
...
pci_enable_sriov(dev, numvfs);
static void dev_shutdown(struct pci_dev *dev)
{
...
return numvfs;
}
if (numvfs == 0) {
....
pci_disable_sriov(dev);
...
return 0;
static int dev_sriov_configure(struct pci_dev *dev, int numvfs)
{
if (numvfs > 0) {
...
pci_enable_sriov(dev, numvfs);
...
return numvfs;
}
if (numvfs == 0) {
....
pci_disable_sriov(dev);
...
return 0;
}
}
}
static struct pci_driver dev_driver = {
.name = "SR-IOV Physical Function driver",
.id_table = dev_id_table,
.probe = dev_probe,
.remove = dev_remove,
.suspend = dev_suspend,
.resume = dev_resume,
.shutdown = dev_shutdown,
.sriov_configure = dev_sriov_configure,
};
static struct pci_driver dev_driver = {
.name = "SR-IOV Physical Function driver",
.id_table = dev_id_table,
.probe = dev_probe,
.remove = dev_remove,
.suspend = dev_suspend,
.resume = dev_resume,
.shutdown = dev_shutdown,
.sriov_configure = dev_sriov_configure,
};
The PCI Express Port Bus Driver Guide HOWTO
Tom L Nguyen tom.l.nguyen@intel.com
11/03/2004
.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>
1. About this guide
===========================================
The PCI Express Port Bus Driver Guide HOWTO
===========================================
:Author: Tom L Nguyen tom.l.nguyen@intel.com 11/03/2004
:Copyright: |copy| 2004 Intel Corporation
About this guide
================
This guide describes the basics of the PCI Express Port Bus driver
and provides information on how to enable the service drivers to
register/unregister with the PCI Express Port Bus Driver.
2. Copyright 2004 Intel Corporation
3. What is the PCI Express Port Bus Driver
What is the PCI Express Port Bus Driver
=======================================
A PCI Express Port is a logical PCI-PCI Bridge structure. There
are two types of PCI Express Port: the Root Port and the Switch
......@@ -30,7 +37,8 @@ support (AER), and virtual channel support (VC). These services may
be handled by a single complex driver or be individually distributed
and handled by corresponding service drivers.
4. Why use the PCI Express Port Bus Driver?
Why use the PCI Express Port Bus Driver?
========================================
In existing Linux kernels, the Linux Device Driver Model allows a
physical device to be handled by only a single driver. The PCI
......@@ -51,28 +59,31 @@ PCI Express Ports and distributes all provided service requests
to the corresponding service drivers as required. Some key
advantages of using the PCI Express Port Bus driver are listed below:
- Allow multiple service drivers to run simultaneously on
a PCI-PCI Bridge Port device.
- Allow multiple service drivers to run simultaneously on
a PCI-PCI Bridge Port device.
- Allow service drivers implemented in an independent
staged approach.
- Allow service drivers implemented in an independent
staged approach.
- Allow one service driver to run on multiple PCI-PCI Bridge
Port devices.
- Allow one service driver to run on multiple PCI-PCI Bridge
Port devices.
- Manage and distribute resources of a PCI-PCI Bridge Port
device to requested service drivers.
- Manage and distribute resources of a PCI-PCI Bridge Port
device to requested service drivers.
5. Configuring the PCI Express Port Bus Driver vs. Service Drivers
Configuring the PCI Express Port Bus Driver vs. Service Drivers
===============================================================
5.1 Including the PCI Express Port Bus Driver Support into the Kernel
Including the PCI Express Port Bus Driver Support into the Kernel
-----------------------------------------------------------------
Including the PCI Express Port Bus driver depends on whether the PCI
Express support is included in the kernel config. The kernel will
automatically include the PCI Express Port Bus driver as a kernel
driver when the PCI Express support is enabled in the kernel.
5.2 Enabling Service Driver Support
Enabling Service Driver Support
-------------------------------
PCI device drivers are implemented based on Linux Device Driver Model.
All service drivers are PCI device drivers. As discussed above, it is
......@@ -89,9 +100,11 @@ header file /include/linux/pcieport_if.h, before calling these APIs.
Failure to do so will result an identity mismatch, which prevents
the PCI Express Port Bus driver from loading a service driver.
5.2.1 pcie_port_service_register
pcie_port_service_register
~~~~~~~~~~~~~~~~~~~~~~~~~~
::
int pcie_port_service_register(struct pcie_port_service_driver *new)
int pcie_port_service_register(struct pcie_port_service_driver *new)
This API replaces the Linux Driver Model's pci_register_driver API. A
service driver should always calls pcie_port_service_register at
......@@ -99,69 +112,76 @@ module init. Note that after service driver being loaded, calls
such as pci_enable_device(dev) and pci_set_master(dev) are no longer
necessary since these calls are executed by the PCI Port Bus driver.
5.2.2 pcie_port_service_unregister
pcie_port_service_unregister
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
::
void pcie_port_service_unregister(struct pcie_port_service_driver *new)
void pcie_port_service_unregister(struct pcie_port_service_driver *new)
pcie_port_service_unregister replaces the Linux Driver Model's
pci_unregister_driver. It's always called by service driver when a
module exits.
5.2.3 Sample Code
Sample Code
~~~~~~~~~~~
Below is sample service driver code to initialize the port service
driver data structure.
::
static struct pcie_port_service_id service_id[] = { {
.vendor = PCI_ANY_ID,
.device = PCI_ANY_ID,
.port_type = PCIE_RC_PORT,
.service_type = PCIE_PORT_SERVICE_AER,
}, { /* end: all zeroes */ }
};
static struct pcie_port_service_id service_id[] = { {
.vendor = PCI_ANY_ID,
.device = PCI_ANY_ID,
.port_type = PCIE_RC_PORT,
.service_type = PCIE_PORT_SERVICE_AER,
}, { /* end: all zeroes */ }
};
static struct pcie_port_service_driver root_aerdrv = {
.name = (char *)device_name,
.id_table = &service_id[0],
static struct pcie_port_service_driver root_aerdrv = {
.name = (char *)device_name,
.id_table = &service_id[0],
.probe = aerdrv_load,
.remove = aerdrv_unload,
.probe = aerdrv_load,
.remove = aerdrv_unload,
.suspend = aerdrv_suspend,
.resume = aerdrv_resume,
};
.suspend = aerdrv_suspend,
.resume = aerdrv_resume,
};
Below is a sample code for registering/unregistering a service
driver.
::
static int __init aerdrv_service_init(void)
{
int retval = 0;
static int __init aerdrv_service_init(void)
{
int retval = 0;
retval = pcie_port_service_register(&root_aerdrv);
if (!retval) {
/*
* FIX ME
*/
}
return retval;
}
retval = pcie_port_service_register(&root_aerdrv);
if (!retval) {
/*
* FIX ME
*/
}
return retval;
}
static void __exit aerdrv_service_exit(void)
{
pcie_port_service_unregister(&root_aerdrv);
}
static void __exit aerdrv_service_exit(void)
{
pcie_port_service_unregister(&root_aerdrv);
}
module_init(aerdrv_service_init);
module_exit(aerdrv_service_exit);
module_init(aerdrv_service_init);
module_exit(aerdrv_service_exit);
6. Possible Resource Conflicts
Possible Resource Conflicts
===========================
Since all service drivers of a PCI-PCI Bridge Port device are
allowed to run simultaneously, below lists a few of possible resource
conflicts with proposed solutions.
6.1 MSI and MSI-X Vector Resource
MSI and MSI-X Vector Resource
-----------------------------
Once MSI or MSI-X interrupts are enabled on a device, it stays in this
mode until they are disabled again. Since service drivers of the same
......@@ -179,7 +199,8 @@ driver. Service drivers should use (struct pcie_device*)dev->irq to
call request_irq/free_irq. In addition, the interrupt mode is stored
in the field interrupt_mode of struct pcie_device.
6.3 PCI Memory/IO Mapped Regions
PCI Memory/IO Mapped Regions
----------------------------
Service drivers for PCI Express Power Management (PME), Advanced
Error Reporting (AER), Hot-Plug (HP) and Virtual Channel (VC) access
......@@ -188,7 +209,8 @@ registers accessed are independent of each other. This patch assumes
that all service drivers will be well behaved and not overwrite
other service driver's configuration settings.
6.4 PCI Config Registers
PCI Config Registers
--------------------
Each service driver runs its PCI config operations on its own
capability structure except the PCI Express capability structure, in
......
......@@ -13,7 +13,7 @@
For ARM64, ONLY "acpi=off", "acpi=on" or "acpi=force"
are available
See also Documentation/power/runtime_pm.txt, pci=noacpi
See also Documentation/power/runtime_pm.rst, pci=noacpi
acpi_apic_instance= [ACPI, IOAPIC]
Format: <int>
......@@ -223,7 +223,7 @@
acpi_sleep= [HW,ACPI] Sleep options
Format: { s3_bios, s3_mode, s3_beep, s4_nohwsig,
old_ordering, nonvs, sci_force_enable, nobl }
See Documentation/power/video.txt for information on
See Documentation/power/video.rst for information on
s3_bios and s3_mode.
s3_beep is for debugging; it makes the PC's speaker beep
as soon as the kernel's real-mode entry point is called.
......@@ -4119,7 +4119,7 @@
Specify the offset from the beginning of the partition
given by "resume=" at which the swap header is located,
in <PAGE_SIZE> units (needed only for swap files).
See Documentation/power/swsusp-and-swap-files.txt
See Documentation/power/swsusp-and-swap-files.rst
resumedelay= [HIBERNATION] Delay (in seconds) to pause before attempting to
read the resume files
......
......@@ -95,7 +95,7 @@ flags - flags of the cpufreq driver
3. CPUFreq Table Generation with Operating Performance Point (OPP)
==================================================================
For details about OPP, see Documentation/power/opp.txt
For details about OPP, see Documentation/power/opp.rst
dev_pm_opp_init_cpufreq_table -
This function provides a ready to use conversion routine to translate
......
......@@ -10,8 +10,10 @@ Required properties:
interrupt source. The value must be 1.
- compatible: Should contain "mbvl,gpex40-pcie"
- reg: Should contain PCIe registers location and length
Mandatory:
"config_axi_slave": PCIe controller registers
"csr_axi_slave" : Bridge config registers
Optional:
"gpio_slave" : GPIO registers to control slot power
"apb_csr" : MSI registers
......
......@@ -65,6 +65,14 @@ Required properties:
- afi
- pcie_x
Optional properties:
- pinctrl-names: A list of pinctrl state names. Must contain the following
entries:
- "default": active state, puts PCIe I/O out of deep power down state
- "idle": puts PCIe I/O into deep power down state
- pinctrl-0: phandle for the default/active state of pin configurations.
- pinctrl-1: phandle for the idle state of pin configurations.
Required properties on Tegra124 and later (deprecated):
- phys: Must contain an entry for each entry in phy-names.
- phy-names: Must include the following entries:
......
......@@ -24,6 +24,9 @@ driver implementation may support the following properties:
unsupported link speed, for instance, trying to do training for
unsupported link speed, etc. Must be '4' for gen4, '3' for gen3, '2'
for gen2, and '1' for gen1. Any other values are invalid.
- reset-gpios:
If present this property specifies PERST# GPIO. Host drivers can parse the
GPIO and apply fundamental reset to endpoints.
PCI-PCI Bridge properties
-------------------------
......
......@@ -10,6 +10,7 @@
- "qcom,pcie-msm8996" for msm8996 or apq8096
- "qcom,pcie-ipq4019" for ipq4019
- "qcom,pcie-ipq8074" for ipq8074
- "qcom,pcie-qcs404" for qcs404
- reg:
Usage: required
......@@ -116,6 +117,15 @@
- "ahb" AHB clock
- "aux" Auxiliary clock
- clock-names:
Usage: required for qcs404
Value type: <stringlist>
Definition: Should contain the following entries
- "iface" AHB clock
- "aux" Auxiliary clock
- "master_bus" AXI Master clock
- "slave_bus" AXI Slave clock
- resets:
Usage: required
Value type: <prop-encoded-array>
......@@ -167,6 +177,17 @@
- "ahb" AHB Reset
- "axi_m_sticky" AXI Master Sticky reset
- reset-names:
Usage: required for qcs404
Value type: <stringlist>
Definition: Should contain the following entries
- "axi_m" AXI Master reset
- "axi_s" AXI Slave reset
- "axi_m_sticky" AXI Master Sticky reset
- "pipe_sticky" PIPE sticky reset
- "pwr" PWR reset
- "ahb" AHB reset
- power-domains:
Usage: required for apq8084 and msm8996/apq8096
Value type: <prop-encoded-array>
......@@ -195,12 +216,12 @@
Definition: A phandle to the PCIe endpoint power supply
- phys:
Usage: required for apq8084
Usage: required for apq8084 and qcs404
Value type: <phandle>
Definition: List of phandle(s) as listed in phy-names property
- phy-names:
Usage: required for apq8084
Usage: required for apq8084 and qcs404
Value type: <stringlist>
Definition: Should contain "pciephy"
......
......@@ -3,6 +3,7 @@
Required properties:
compatible: "renesas,pcie-r8a7743" for the R8A7743 SoC;
"renesas,pcie-r8a7744" for the R8A7744 SoC;
"renesas,pcie-r8a774a1" for the R8A774A1 SoC;
"renesas,pcie-r8a774c0" for the R8A774C0 SoC;
"renesas,pcie-r8a7779" for the R8A7779 SoC;
"renesas,pcie-r8a7790" for the R8A7790 SoC;
......
......@@ -225,7 +225,7 @@ system-wide transition to a sleep state even though its :c:member:`runtime_auto`
flag is clear.
For more information about the runtime power management framework, refer to
:file:`Documentation/power/runtime_pm.txt`.
:file:`Documentation/power/runtime_pm.rst`.
Calling Drivers to Enter and Leave System Sleep States
......@@ -728,7 +728,7 @@ it into account in any way.
Devices may be defined as IRQ-safe which indicates to the PM core that their
runtime PM callbacks may be invoked with disabled interrupts (see
:file:`Documentation/power/runtime_pm.txt` for more information). If an
:file:`Documentation/power/runtime_pm.rst` for more information). If an
IRQ-safe device belongs to a PM domain, the runtime PM of the domain will be
disallowed, unless the domain itself is defined as IRQ-safe. However, it
makes sense to define a PM domain as IRQ-safe only if all the devices in it
......@@ -795,7 +795,7 @@ so on) and the final state of the device must reflect the "active" runtime PM
status in that case.
During system-wide resume from a sleep state it's easiest to put devices into
the full-power state, as explained in :file:`Documentation/power/runtime_pm.txt`.
the full-power state, as explained in :file:`Documentation/power/runtime_pm.rst`.
[Refer to that document for more information regarding this particular issue as
well as for information on the device runtime power management framework in
general.]
......
......@@ -46,7 +46,7 @@ device is turned off while the system as a whole remains running, we
call it a "dynamic suspend" (also known as a "runtime suspend" or
"selective suspend"). This document concentrates mostly on how
dynamic PM is implemented in the USB subsystem, although system PM is
covered to some extent (see ``Documentation/power/*.txt`` for more
covered to some extent (see ``Documentation/power/*.rst`` for more
information about system PM).
System PM support is present only if the kernel was built with
......
......@@ -103,6 +103,7 @@ needed).
vm/index
bpf/index
usb/index
PCI/index
misc-devices/index
Architecture-specific documentation
......
============
APM or ACPI?
------------
============
If you have a relatively recent x86 mobile, desktop, or server system,
odds are it supports either Advanced Power Management (APM) or
Advanced Configuration and Power Interface (ACPI). ACPI is the newer
......@@ -28,5 +30,7 @@ and be sure that they are started sometime in the system boot process.
Go ahead and start both. If ACPI or APM is not available on your
system the associated daemon will exit gracefully.
apmd: http://ftp.debian.org/pool/main/a/apmd/
acpid: http://acpid.sf.net/
===== =======================================
apmd http://ftp.debian.org/pool/main/a/apmd/
acpid http://acpid.sf.net/
===== =======================================
=================================
Debugging hibernation and suspend
=================================
(C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL
1. Testing hibernation (aka suspend to disk or STD)
===================================================
To check if hibernation works, you can try to hibernate in the "reboot" mode:
To check if hibernation works, you can try to hibernate in the "reboot" mode::
# echo reboot > /sys/power/disk
# echo disk > /sys/power/state
# echo reboot > /sys/power/disk
# echo disk > /sys/power/state
and the system should create a hibernation image, reboot, resume and get back to
the command prompt where you have started the transition. If that happens,
......@@ -15,20 +19,21 @@ test at least a couple of times in a row for confidence. [This is necessary,
because some problems only show up on a second attempt at suspending and
resuming the system.] Moreover, hibernating in the "reboot" and "shutdown"
modes causes the PM core to skip some platform-related callbacks which on ACPI
systems might be necessary to make hibernation work. Thus, if your machine fails
to hibernate or resume in the "reboot" mode, you should try the "platform" mode:
systems might be necessary to make hibernation work. Thus, if your machine
fails to hibernate or resume in the "reboot" mode, you should try the
"platform" mode::
# echo platform > /sys/power/disk
# echo disk > /sys/power/state
# echo platform > /sys/power/disk
# echo disk > /sys/power/state
which is the default and recommended mode of hibernation.
Unfortunately, the "platform" mode of hibernation does not work on some systems
with broken BIOSes. In such cases the "shutdown" mode of hibernation might
work:
work::
# echo shutdown > /sys/power/disk
# echo disk > /sys/power/state
# echo shutdown > /sys/power/disk
# echo disk > /sys/power/state
(it is similar to the "reboot" mode, but it requires you to press the power
button to make the system resume).
......@@ -37,6 +42,7 @@ If neither "platform" nor "shutdown" hibernation mode works, you will need to
identify what goes wrong.
a) Test modes of hibernation
----------------------------
To find out why hibernation fails on your system, you can use a special testing
facility available if the kernel is compiled with CONFIG_PM_DEBUG set. Then,
......@@ -44,36 +50,38 @@ there is the file /sys/power/pm_test that can be used to make the hibernation
core run in a test mode. There are 5 test modes available:
freezer
- test the freezing of processes
- test the freezing of processes
devices
- test the freezing of processes and suspending of devices
- test the freezing of processes and suspending of devices
platform
- test the freezing of processes, suspending of devices and platform
global control methods(*)
- test the freezing of processes, suspending of devices and platform
global control methods [1]_
processors
- test the freezing of processes, suspending of devices, platform
global control methods(*) and the disabling of nonboot CPUs
- test the freezing of processes, suspending of devices, platform
global control methods [1]_ and the disabling of nonboot CPUs
core
- test the freezing of processes, suspending of devices, platform global
control methods(*), the disabling of nonboot CPUs and suspending of
platform/system devices
- test the freezing of processes, suspending of devices, platform global
control methods\ [1]_, the disabling of nonboot CPUs and suspending
of platform/system devices
.. [1]
(*) the platform global control methods are only available on ACPI systems
the platform global control methods are only available on ACPI systems
and are only tested if the hibernation mode is set to "platform"
To use one of them it is necessary to write the corresponding string to
/sys/power/pm_test (eg. "devices" to test the freezing of processes and
suspending devices) and issue the standard hibernation commands. For example,
to use the "devices" test mode along with the "platform" mode of hibernation,
you should do the following:
you should do the following::
# echo devices > /sys/power/pm_test
# echo platform > /sys/power/disk
# echo disk > /sys/power/state
# echo devices > /sys/power/pm_test
# echo platform > /sys/power/disk
# echo disk > /sys/power/state
Then, the kernel will try to freeze processes, suspend devices, wait a few
seconds (5 by default, but configurable by the suspend.pm_test_delay module
......@@ -108,11 +116,12 @@ If the "devices" test fails, most likely there is a driver that cannot suspend
or resume its device (in the latter case the system may hang or become unstable
after the test, so please take that into consideration). To find this driver,
you can carry out a binary search according to the rules:
- if the test fails, unload a half of the drivers currently loaded and repeat
(that would probably involve rebooting the system, so always note what drivers
have been loaded before the test),
(that would probably involve rebooting the system, so always note what drivers
have been loaded before the test),
- if the test succeeds, load a half of the drivers you have unloaded most
recently and repeat.
recently and repeat.
Once you have found the failing driver (there can be more than just one of
them), you have to unload it every time before hibernation. In that case please
......@@ -146,6 +155,7 @@ indicates a serious problem that very well may be related to the hardware, but
please report it anyway.
b) Testing minimal configuration
--------------------------------
If all of the hibernation test modes work, you can boot the system with the
"init=/bin/bash" command line parameter and attempt to hibernate in the
......@@ -165,14 +175,15 @@ Again, if you find the offending module(s), it(they) must be unloaded every time
before hibernation, and please report the problem with it(them).
c) Using the "test_resume" hibernation option
---------------------------------------------
/sys/power/disk generally tells the kernel what to do after creating a
hibernation image. One of the available options is "test_resume" which
causes the just created image to be used for immediate restoration. Namely,
after doing:
after doing::
# echo test_resume > /sys/power/disk
# echo disk > /sys/power/state
# echo test_resume > /sys/power/disk
# echo disk > /sys/power/state
a hibernation image will be created and a resume from it will be triggered
immediately without involving the platform firmware in any way.
......@@ -190,6 +201,7 @@ to resume may be related to the differences between the restore and image
kernels.
d) Advanced debugging
---------------------
In case that hibernation does not work on your system even in the minimal
configuration and compiling more drivers as modules is not practical or some
......@@ -200,9 +212,10 @@ kernel messages using the serial console. This may provide you with some
information about the reasons of the suspend (resume) failure. Alternatively,
it may be possible to use a FireWire port for debugging with firescope
(http://v3.sk/~lkundrak/firescope/). On x86 it is also possible to
use the PM_TRACE mechanism documented in Documentation/power/s2ram.txt .
use the PM_TRACE mechanism documented in Documentation/power/s2ram.rst .
2. Testing suspend to RAM (STR)
===============================
To verify that the STR works, it is generally more convenient to use the s2ram
tool available from http://suspend.sf.net and documented at
......@@ -230,7 +243,8 @@ you will have to unload them every time before an STR transition (ie. before
you run s2ram), and please report the problems with them.
There is a debugfs entry which shows the suspend to RAM statistics. Here is an
example of its output.
example of its output::
# mount -t debugfs none /sys/kernel/debug
# cat /sys/kernel/debug/suspend_stats
success: 20
......@@ -248,6 +262,7 @@ example of its output.
-16
last_failed_step: suspend
suspend
Field success means the success number of suspend to RAM, and field fail means
the failure number. Others are the failure number of different steps of suspend
to RAM. suspend_stats just lists the last 2 failed devices, error number and
......
===============
Charger Manager
===============
(C) 2011 MyungJoo Ham <myungjoo.ham@samsung.com>, GPL
Charger Manager provides in-kernel battery charger management that
......@@ -55,41 +58,39 @@ Charger Manager supports the following:
notification to users with UEVENT.
2. Global Charger-Manager Data related with suspend_again
========================================================
=========================================================
In order to setup Charger Manager with suspend-again feature
(in-suspend monitoring), the user should provide charger_global_desc
with setup_charger_manager(struct charger_global_desc *).
with setup_charger_manager(`struct charger_global_desc *`).
This charger_global_desc data for in-suspend monitoring is global
as the name suggests. Thus, the user needs to provide only once even
if there are multiple batteries. If there are multiple batteries, the
multiple instances of Charger Manager share the same charger_global_desc
and it will manage in-suspend monitoring for all instances of Charger Manager.
The user needs to provide all the three entries properly in order to activate
in-suspend monitoring:
struct charger_global_desc {
The user needs to provide all the three entries to `struct charger_global_desc`
properly in order to activate in-suspend monitoring:
char *rtc_name;
: The name of rtc (e.g., "rtc0") used to wakeup the system from
`char *rtc_name;`
The name of rtc (e.g., "rtc0") used to wakeup the system from
suspend for Charger Manager. The alarm interrupt (AIE) of the rtc
should be able to wake up the system from suspend. Charger Manager
saves and restores the alarm value and use the previously-defined
alarm if it is going to go off earlier than Charger Manager so that
Charger Manager does not interfere with previously-defined alarms.
bool (*rtc_only_wakeup)(void);
: This callback should let CM know whether
`bool (*rtc_only_wakeup)(void);`
This callback should let CM know whether
the wakeup-from-suspend is caused only by the alarm of "rtc" in the
same struct. If there is any other wakeup source triggered the
wakeup, it should return false. If the "rtc" is the only wakeup
reason, it should return true.
bool assume_timer_stops_in_suspend;
: if true, Charger Manager assumes that
`bool assume_timer_stops_in_suspend;`
if true, Charger Manager assumes that
the timer (CM uses jiffies as timer) stops during suspend. Then, CM
assumes that the suspend-duration is same as the alarm length.
};
3. How to setup suspend_again
=============================
......@@ -109,26 +110,28 @@ if the system was woken up by Charger Manager and the polling
=============================================
For each battery charged independently from other batteries (if a series of
batteries are charged by a single charger, they are counted as one independent
battery), an instance of Charger Manager is attached to it.
battery), an instance of Charger Manager is attached to it. The following
struct charger_desc {
struct charger_desc elements:
char *psy_name;
: The power-supply-class name of the battery. Default is
`char *psy_name;`
The power-supply-class name of the battery. Default is
"battery" if psy_name is NULL. Users can access the psy entries
at "/sys/class/power_supply/[psy_name]/".
enum polling_modes polling_mode;
: CM_POLL_DISABLE: do not poll this battery.
CM_POLL_ALWAYS: always poll this battery.
CM_POLL_EXTERNAL_POWER_ONLY: poll this battery if and only if
an external power source is attached.
CM_POLL_CHARGING_ONLY: poll this battery if and only if the
battery is being charged.
unsigned int fullbatt_vchkdrop_ms;
unsigned int fullbatt_vchkdrop_uV;
: If both have non-zero values, Charger Manager will check the
`enum polling_modes polling_mode;`
CM_POLL_DISABLE:
do not poll this battery.
CM_POLL_ALWAYS:
always poll this battery.
CM_POLL_EXTERNAL_POWER_ONLY:
poll this battery if and only if an external power
source is attached.
CM_POLL_CHARGING_ONLY:
poll this battery if and only if the battery is being charged.
`unsigned int fullbatt_vchkdrop_ms; / unsigned int fullbatt_vchkdrop_uV;`
If both have non-zero values, Charger Manager will check the
battery voltage drop fullbatt_vchkdrop_ms after the battery is fully
charged. If the voltage drop is over fullbatt_vchkdrop_uV, Charger
Manager will try to recharge the battery by disabling and enabling
......@@ -136,50 +139,52 @@ unsigned int fullbatt_vchkdrop_uV;
condition) is needed to be implemented with hardware interrupts from
fuel gauges or charger devices/chips.
unsigned int fullbatt_uV;
: If specified with a non-zero value, Charger Manager assumes
`unsigned int fullbatt_uV;`
If specified with a non-zero value, Charger Manager assumes
that the battery is full (capacity = 100) if the battery is not being
charged and the battery voltage is equal to or greater than
fullbatt_uV.
unsigned int polling_interval_ms;
: Required polling interval in ms. Charger Manager will poll
`unsigned int polling_interval_ms;`
Required polling interval in ms. Charger Manager will poll
this battery every polling_interval_ms or more frequently.
enum data_source battery_present;
: CM_BATTERY_PRESENT: assume that the battery exists.
CM_NO_BATTERY: assume that the battery does not exists.
CM_FUEL_GAUGE: get battery presence information from fuel gauge.
CM_CHARGER_STAT: get battery presence from chargers.
char **psy_charger_stat;
: An array ending with NULL that has power-supply-class names of
`enum data_source battery_present;`
CM_BATTERY_PRESENT:
assume that the battery exists.
CM_NO_BATTERY:
assume that the battery does not exists.
CM_FUEL_GAUGE:
get battery presence information from fuel gauge.
CM_CHARGER_STAT:
get battery presence from chargers.
`char **psy_charger_stat;`
An array ending with NULL that has power-supply-class names of
chargers. Each power-supply-class should provide "PRESENT" (if
battery_present is "CM_CHARGER_STAT"), "ONLINE" (shows whether an
external power source is attached or not), and "STATUS" (shows whether
the battery is {"FULL" or not FULL} or {"FULL", "Charging",
"Discharging", "NotCharging"}).
int num_charger_regulators;
struct regulator_bulk_data *charger_regulators;
: Regulators representing the chargers in the form for
`int num_charger_regulators; / struct regulator_bulk_data *charger_regulators;`
Regulators representing the chargers in the form for
regulator framework's bulk functions.
char *psy_fuel_gauge;
: Power-supply-class name of the fuel gauge.
`char *psy_fuel_gauge;`
Power-supply-class name of the fuel gauge.
int (*temperature_out_of_range)(int *mC);
bool measure_battery_temp;
: This callback returns 0 if the temperature is safe for charging,
`int (*temperature_out_of_range)(int *mC); / bool measure_battery_temp;`
This callback returns 0 if the temperature is safe for charging,
a positive number if it is too hot to charge, and a negative number
if it is too cold to charge. With the variable mC, the callback returns
the temperature in 1/1000 of centigrade.
The source of temperature can be battery or ambient one according to
the value of measure_battery_temp.
};
5. Notify Charger-Manager of charger events: cm_notify_event()
=========================================================
==============================================================
If there is an charger event is required to notify
Charger Manager, a charger device driver that triggers the event can call
cm_notify_event(psy, type, msg) to notify the corresponding Charger Manager.
......
====================================================
Testing suspend and resume support in device drivers
====================================================
(C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL
1. Preparing the test system
============================
Unfortunately, to effectively test the support for the system-wide suspend and
resume transitions in a driver, it is necessary to suspend and resume a fully
......@@ -14,19 +18,20 @@ the machine's BIOS.
Of course, for this purpose the test system has to be known to suspend and
resume without the driver being tested. Thus, if possible, you should first
resolve all suspend/resume-related problems in the test system before you start
testing the new driver. Please see Documentation/power/basic-pm-debugging.txt
testing the new driver. Please see Documentation/power/basic-pm-debugging.rst
for more information about the debugging of suspend/resume functionality.
2. Testing the driver
=====================
Once you have resolved the suspend/resume-related problems with your test system
without the new driver, you are ready to test it:
a) Build the driver as a module, load it and try the test modes of hibernation
(see: Documentation/power/basic-pm-debugging.txt, 1).
(see: Documentation/power/basic-pm-debugging.rst, 1).
b) Load the driver and attempt to hibernate in the "reboot", "shutdown" and
"platform" modes (see: Documentation/power/basic-pm-debugging.txt, 1).
"platform" modes (see: Documentation/power/basic-pm-debugging.rst, 1).
c) Compile the driver directly into the kernel and try the test modes of
hibernation.
......@@ -34,12 +39,12 @@ c) Compile the driver directly into the kernel and try the test modes of
d) Attempt to hibernate with the driver compiled directly into the kernel
in the "reboot", "shutdown" and "platform" modes.
e) Try the test modes of suspend (see: Documentation/power/basic-pm-debugging.txt,
e) Try the test modes of suspend (see: Documentation/power/basic-pm-debugging.rst,
2). [As far as the STR tests are concerned, it should not matter whether or
not the driver is built as a module.]
f) Attempt to suspend to RAM using the s2ram tool with the driver loaded
(see: Documentation/power/basic-pm-debugging.txt, 2).
(see: Documentation/power/basic-pm-debugging.rst, 2).
Each of the above tests should be repeated several times and the STD tests
should be mixed with the STR tests. If any of them fails, the driver cannot be
......
====================
Energy Model of CPUs
====================
====================
Energy Model of CPUs
====================
1. Overview
-----------
......@@ -20,7 +20,7 @@ kernel, hence enabling to avoid redundant work.
The figure below depicts an example of drivers (Arm-specific here, but the
approach is applicable to any architecture) providing power costs to the EM
framework, and interested clients reading the data from it.
framework, and interested clients reading the data from it::
+---------------+ +-----------------+ +---------------+
| Thermal (IPA) | | Scheduler (EAS) | | Other |
......@@ -58,15 +58,17 @@ micro-architectures.
2. Core APIs
------------
2.1 Config options
2.1 Config options
^^^^^^^^^^^^^^^^^^
CONFIG_ENERGY_MODEL must be enabled to use the EM framework.
2.2 Registration of performance domains
2.2 Registration of performance domains
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Drivers are expected to register performance domains into the EM framework by
calling the following API:
calling the following API::
int em_register_perf_domain(cpumask_t *span, unsigned int nr_states,
struct em_data_callback *cb);
......@@ -80,7 +82,8 @@ callback, and kernel/power/energy_model.c for further documentation on this
API.
2.3 Accessing performance domains
2.3 Accessing performance domains
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Subsystems interested in the energy model of a CPU can retrieve it using the
em_cpu_get() API. The energy model tables are allocated once upon creation of
......@@ -99,46 +102,46 @@ More details about the above APIs can be found in include/linux/energy_model.h.
This section provides a simple example of a CPUFreq driver registering a
performance domain in the Energy Model framework using the (fake) 'foo'
protocol. The driver implements an est_power() function to be provided to the
EM framework.
-> drivers/cpufreq/foo_cpufreq.c
01 static int est_power(unsigned long *mW, unsigned long *KHz, int cpu)
02 {
03 long freq, power;
04
05 /* Use the 'foo' protocol to ceil the frequency */
06 freq = foo_get_freq_ceil(cpu, *KHz);
07 if (freq < 0);
08 return freq;
09
10 /* Estimate the power cost for the CPU at the relevant freq. */
11 power = foo_estimate_power(cpu, freq);
12 if (power < 0);
13 return power;
14
15 /* Return the values to the EM framework */
16 *mW = power;
17 *KHz = freq;
18
19 return 0;
20 }
21
22 static int foo_cpufreq_init(struct cpufreq_policy *policy)
23 {
24 struct em_data_callback em_cb = EM_DATA_CB(est_power);
25 int nr_opp, ret;
26
27 /* Do the actual CPUFreq init work ... */
28 ret = do_foo_cpufreq_init(policy);
29 if (ret)
30 return ret;
31
32 /* Find the number of OPPs for this policy */
33 nr_opp = foo_get_nr_opp(policy);
34
35 /* And register the new performance domain */
36 em_register_perf_domain(policy->cpus, nr_opp, &em_cb);
37
38 return 0;
39 }
EM framework::
-> drivers/cpufreq/foo_cpufreq.c
01 static int est_power(unsigned long *mW, unsigned long *KHz, int cpu)
02 {
03 long freq, power;
04
05 /* Use the 'foo' protocol to ceil the frequency */
06 freq = foo_get_freq_ceil(cpu, *KHz);
07 if (freq < 0);
08 return freq;
09
10 /* Estimate the power cost for the CPU at the relevant freq. */
11 power = foo_estimate_power(cpu, freq);
12 if (power < 0);
13 return power;
14
15 /* Return the values to the EM framework */
16 *mW = power;
17 *KHz = freq;
18
19 return 0;
20 }
21
22 static int foo_cpufreq_init(struct cpufreq_policy *policy)
23 {
24 struct em_data_callback em_cb = EM_DATA_CB(est_power);
25 int nr_opp, ret;
26
27 /* Do the actual CPUFreq init work ... */
28 ret = do_foo_cpufreq_init(policy);
29 if (ret)
30 return ret;
31
32 /* Find the number of OPPs for this policy */
33 nr_opp = foo_get_nr_opp(policy);
34
35 /* And register the new performance domain */
36 em_register_perf_domain(policy->cpus, nr_opp, &em_cb);
37
38 return 0;
39 }
=================
Freezing of tasks
(C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL
=================
(C) 2007 Rafael J. Wysocki <rjw@sisk.pl>, GPL
I. What is the freezing of tasks?
=================================
The freezing of tasks is a mechanism by which user space processes and some
kernel threads are controlled during hibernation or system-wide suspend (on some
architectures).
II. How does it work?
=====================
There are three per-task flags used for that, PF_NOFREEZE, PF_FROZEN
and PF_FREEZER_SKIP (the last one is auxiliary). The tasks that have
......@@ -41,7 +46,7 @@ explicitly in suitable places or use the wait_event_freezable() or
wait_event_freezable_timeout() macros (defined in include/linux/freezer.h)
that combine interruptible sleep with checking if the task is to be frozen and
calling try_to_freeze(). The main loop of a freezable kernel thread may look
like the following one:
like the following one::
set_freezable();
do {
......@@ -65,7 +70,7 @@ order to clear the PF_FROZEN flag for each frozen task. Then, the tasks that
have been frozen leave __refrigerator() and continue running.
Rationale behind the functions dealing with freezing and thawing of tasks:
Rationale behind the functions dealing with freezing and thawing of tasks
-------------------------------------------------------------------------
freeze_processes():
......@@ -86,6 +91,7 @@ thaw_processes():
III. Which kernel threads are freezable?
========================================
Kernel threads are not freezable by default. However, a kernel thread may clear
PF_NOFREEZE for itself by calling set_freezable() (the resetting of PF_NOFREEZE
......@@ -93,37 +99,39 @@ directly is not allowed). From this point it is regarded as freezable
and must call try_to_freeze() in a suitable place.
IV. Why do we do that?
======================
Generally speaking, there is a couple of reasons to use the freezing of tasks:
1. The principal reason is to prevent filesystems from being damaged after
hibernation. At the moment we have no simple means of checkpointing
filesystems, so if there are any modifications made to filesystem data and/or
metadata on disks, we cannot bring them back to the state from before the
modifications. At the same time each hibernation image contains some
filesystem-related information that must be consistent with the state of the
on-disk data and metadata after the system memory state has been restored from
the image (otherwise the filesystems will be damaged in a nasty way, usually
making them almost impossible to repair). We therefore freeze tasks that might
cause the on-disk filesystems' data and metadata to be modified after the
hibernation image has been created and before the system is finally powered off.
The majority of these are user space processes, but if any of the kernel threads
may cause something like this to happen, they have to be freezable.
hibernation. At the moment we have no simple means of checkpointing
filesystems, so if there are any modifications made to filesystem data and/or
metadata on disks, we cannot bring them back to the state from before the
modifications. At the same time each hibernation image contains some
filesystem-related information that must be consistent with the state of the
on-disk data and metadata after the system memory state has been restored
from the image (otherwise the filesystems will be damaged in a nasty way,
usually making them almost impossible to repair). We therefore freeze
tasks that might cause the on-disk filesystems' data and metadata to be
modified after the hibernation image has been created and before the
system is finally powered off. The majority of these are user space
processes, but if any of the kernel threads may cause something like this
to happen, they have to be freezable.
2. Next, to create the hibernation image we need to free a sufficient amount of
memory (approximately 50% of available RAM) and we need to do that before
devices are deactivated, because we generally need them for swapping out. Then,
after the memory for the image has been freed, we don't want tasks to allocate
additional memory and we prevent them from doing that by freezing them earlier.
[Of course, this also means that device drivers should not allocate substantial
amounts of memory from their .suspend() callbacks before hibernation, but this
is a separate issue.]
memory (approximately 50% of available RAM) and we need to do that before
devices are deactivated, because we generally need them for swapping out.
Then, after the memory for the image has been freed, we don't want tasks
to allocate additional memory and we prevent them from doing that by
freezing them earlier. [Of course, this also means that device drivers
should not allocate substantial amounts of memory from their .suspend()
callbacks before hibernation, but this is a separate issue.]
3. The third reason is to prevent user space processes and some kernel threads
from interfering with the suspending and resuming of devices. A user space
process running on a second CPU while we are suspending devices may, for
example, be troublesome and without the freezing of tasks we would need some
safeguards against race conditions that might occur in such a case.
from interfering with the suspending and resuming of devices. A user space
process running on a second CPU while we are suspending devices may, for
example, be troublesome and without the freezing of tasks we would need some
safeguards against race conditions that might occur in such a case.
Although Linus Torvalds doesn't like the freezing of tasks, he said this in one
of the discussions on LKML (http://lkml.org/lkml/2007/4/27/608):
......@@ -132,7 +140,7 @@ of the discussions on LKML (http://lkml.org/lkml/2007/4/27/608):
Linus: In many ways, 'at all'.
I _do_ realize the IO request queue issues, and that we cannot actually do
I **do** realize the IO request queue issues, and that we cannot actually do
s2ram with some devices in the middle of a DMA. So we want to be able to
avoid *that*, there's no question about that. And I suspect that stopping
user threads and then waiting for a sync is practically one of the easier
......@@ -150,17 +158,18 @@ thawed after the driver's .resume() callback has run, so it won't be accessing
the device while it's suspended.
4. Another reason for freezing tasks is to prevent user space processes from
realizing that hibernation (or suspend) operation takes place. Ideally, user
space processes should not notice that such a system-wide operation has occurred
and should continue running without any problems after the restore (or resume
from suspend). Unfortunately, in the most general case this is quite difficult
to achieve without the freezing of tasks. Consider, for example, a process
that depends on all CPUs being online while it's running. Since we need to
disable nonboot CPUs during the hibernation, if this process is not frozen, it
may notice that the number of CPUs has changed and may start to work incorrectly
because of that.
realizing that hibernation (or suspend) operation takes place. Ideally, user
space processes should not notice that such a system-wide operation has
occurred and should continue running without any problems after the restore
(or resume from suspend). Unfortunately, in the most general case this
is quite difficult to achieve without the freezing of tasks. Consider,
for example, a process that depends on all CPUs being online while it's
running. Since we need to disable nonboot CPUs during the hibernation,
if this process is not frozen, it may notice that the number of CPUs has
changed and may start to work incorrectly because of that.
V. Are there any problems related to the freezing of tasks?
===========================================================
Yes, there are.
......@@ -172,11 +181,12 @@ may be undesirable. That's why kernel threads are not freezable by default.
Second, there are the following two problems related to the freezing of user
space processes:
1. Putting processes into an uninterruptible sleep distorts the load average.
2. Now that we have FUSE, plus the framework for doing device drivers in
userspace, it gets even more complicated because some userspace processes are
now doing the sorts of things that kernel threads do
(https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html).
userspace, it gets even more complicated because some userspace processes are
now doing the sorts of things that kernel threads do
(https://lists.linux-foundation.org/pipermail/linux-pm/2007-May/012309.html).
The problem 1. seems to be fixable, although it hasn't been fixed so far. The
other one is more serious, but it seems that we can work around it by using
......@@ -201,6 +211,7 @@ requested early enough using the suspend notifier API described in
Documentation/driver-api/pm/notifiers.rst.
VI. Are there any precautions to be taken to prevent freezing failures?
=======================================================================
Yes, there are.
......@@ -226,6 +237,8 @@ So, to summarize, use [un]lock_system_sleep() instead of directly using
mutex_[un]lock(&system_transition_mutex). That would prevent freezing failures.
V. Miscellaneous
================
/sys/power/pm_freeze_timeout controls how long it will cost at most to freeze
all user space processes or all freezable kernel threads, in unit of millisecond.
The default value is 20000, with range of unsigned integer.
:orphan:
================
Power Management
================
.. toctree::
:maxdepth: 1
apm-acpi
basic-pm-debugging
charger-manager
drivers-testing
energy-model
freezing-of-tasks
interface
opp
pci
pm_qos_interface
power_supply_class
runtime_pm
s2ram
suspend-and-cpuhotplug
suspend-and-interrupts
swsusp-and-swap-files
swsusp-dmcrypt
swsusp
video
tricks
userland-swsusp
powercap/powercap
regulator/consumer
regulator/design
regulator/machine
regulator/overview
regulator/regulator
.. only:: subproject and html
Indices
=======
* :ref:`genindex`
===========================================
Power Management Interface for System Sleep
===========================================
Copyright (c) 2016 Intel Corp., Rafael J. Wysocki <rafael.j.wysocki@intel.com>
......@@ -11,10 +13,10 @@ mounted at /sys).
Reading from it returns a list of supported sleep states, encoded as:
'freeze' (Suspend-to-Idle)
'standby' (Power-On Suspend)
'mem' (Suspend-to-RAM)
'disk' (Suspend-to-Disk)
- 'freeze' (Suspend-to-Idle)
- 'standby' (Power-On Suspend)
- 'mem' (Suspend-to-RAM)
- 'disk' (Suspend-to-Disk)
Suspend-to-Idle is always supported. Suspend-to-Disk is always supported
too as long the kernel has been configured to support hibernation at all
......@@ -32,18 +34,18 @@ Specifically, it tells the kernel what to do after creating a hibernation image.
Reading from it returns a list of supported options encoded as:
'platform' (put the system into sleep using a platform-provided method)
'shutdown' (shut the system down)
'reboot' (reboot the system)
'suspend' (trigger a Suspend-to-RAM transition)
'test_resume' (resume-after-hibernation test mode)
- 'platform' (put the system into sleep using a platform-provided method)
- 'shutdown' (shut the system down)
- 'reboot' (reboot the system)
- 'suspend' (trigger a Suspend-to-RAM transition)
- 'test_resume' (resume-after-hibernation test mode)
The currently selected option is printed in square brackets.
The 'platform' option is only available if the platform provides a special
mechanism to put the system to sleep after creating a hibernation image (ACPI
does that, for example). The 'suspend' option is available if Suspend-to-RAM
is supported. Refer to Documentation/power/basic-pm-debugging.txt for the
is supported. Refer to Documentation/power/basic-pm-debugging.rst for the
description of the 'test_resume' option.
To select an option, write the string representing it to /sys/power/disk.
......@@ -71,7 +73,7 @@ If /sys/power/pm_trace contains '1', the fingerprint of each suspend/resume
event point in turn will be stored in the RTC memory (overwriting the actual
RTC information), so it will survive a system crash if one occurs right after
storing it and it can be used later to identify the driver that caused the crash
to happen (see Documentation/power/s2ram.txt for more information).
to happen (see Documentation/power/s2ram.rst for more information).
Initially it contains '0' which may be changed to '1' by writing a string
representing a nonzero integer into it.
PM Quality Of Service Interface.
===============================
PM Quality Of Service Interface
===============================
This interface provides a kernel and user mode interface for registering
performance expectations by drivers, subsystems and user space applications on
......@@ -11,6 +13,7 @@ memory_bandwidth.
constraints and PM QoS flags.
Each parameters have defined units:
* latency: usec
* timeout: usec
* throughput: kbs (kilo bit / sec)
......@@ -18,6 +21,7 @@ Each parameters have defined units:
1. PM QoS framework
===================
The infrastructure exposes multiple misc device nodes one per implemented
parameter. The set of parameters implement is defined by pm_qos_power_init()
......@@ -37,38 +41,39 @@ reading the aggregated value does not require any locking mechanism.
From kernel mode the use of this interface is simple:
void pm_qos_add_request(handle, param_class, target_value):
Will insert an element into the list for that identified PM QoS class with the
target value. Upon change to this list the new target is recomputed and any
registered notifiers are called only if the target value is now different.
Clients of pm_qos need to save the returned handle for future use in other
pm_qos API functions.
Will insert an element into the list for that identified PM QoS class with the
target value. Upon change to this list the new target is recomputed and any
registered notifiers are called only if the target value is now different.
Clients of pm_qos need to save the returned handle for future use in other
pm_qos API functions.
void pm_qos_update_request(handle, new_target_value):
Will update the list element pointed to by the handle with the new target value
and recompute the new aggregated target, calling the notification tree if the
target is changed.
Will update the list element pointed to by the handle with the new target value
and recompute the new aggregated target, calling the notification tree if the
target is changed.
void pm_qos_remove_request(handle):
Will remove the element. After removal it will update the aggregate target and
call the notification tree if the target was changed as a result of removing
the request.
Will remove the element. After removal it will update the aggregate target and
call the notification tree if the target was changed as a result of removing
the request.
int pm_qos_request(param_class):
Returns the aggregated value for a given PM QoS class.
Returns the aggregated value for a given PM QoS class.
int pm_qos_request_active(handle):
Returns if the request is still active, i.e. it has not been removed from a
PM QoS class constraints list.
Returns if the request is still active, i.e. it has not been removed from a
PM QoS class constraints list.
int pm_qos_add_notifier(param_class, notifier):
Adds a notification callback function to the PM QoS class. The callback is
called when the aggregated value for the PM QoS class is changed.
Adds a notification callback function to the PM QoS class. The callback is
called when the aggregated value for the PM QoS class is changed.
int pm_qos_remove_notifier(int param_class, notifier):
Removes the notification callback function for the PM QoS class.
Removes the notification callback function for the PM QoS class.
From user mode:
Only processes can register a pm_qos request. To provide for automatic
cleanup of a process, the interface requires the process to register its
parameter requests in the following way:
......@@ -89,6 +94,7 @@ node.
2. PM QoS per-device latency and flags framework
================================================
For each device, there are three lists of PM QoS requests. Two of them are
maintained along with the aggregated targets of resume latency and active
......@@ -107,73 +113,80 @@ the aggregated value does not require any locking mechanism.
From kernel mode the use of this interface is the following:
int dev_pm_qos_add_request(device, handle, type, value):
Will insert an element into the list for that identified device with the
target value. Upon change to this list the new target is recomputed and any
registered notifiers are called only if the target value is now different.
Clients of dev_pm_qos need to save the handle for future use in other
dev_pm_qos API functions.
Will insert an element into the list for that identified device with the
target value. Upon change to this list the new target is recomputed and any
registered notifiers are called only if the target value is now different.
Clients of dev_pm_qos need to save the handle for future use in other
dev_pm_qos API functions.
int dev_pm_qos_update_request(handle, new_value):
Will update the list element pointed to by the handle with the new target value
and recompute the new aggregated target, calling the notification trees if the
target is changed.
Will update the list element pointed to by the handle with the new target
value and recompute the new aggregated target, calling the notification
trees if the target is changed.
int dev_pm_qos_remove_request(handle):
Will remove the element. After removal it will update the aggregate target and
call the notification trees if the target was changed as a result of removing
the request.
Will remove the element. After removal it will update the aggregate target
and call the notification trees if the target was changed as a result of
removing the request.
s32 dev_pm_qos_read_value(device):
Returns the aggregated value for a given device's constraints list.
Returns the aggregated value for a given device's constraints list.
enum pm_qos_flags_status dev_pm_qos_flags(device, mask)
Check PM QoS flags of the given device against the given mask of flags.
The meaning of the return values is as follows:
PM_QOS_FLAGS_ALL: All flags from the mask are set
PM_QOS_FLAGS_SOME: Some flags from the mask are set
PM_QOS_FLAGS_NONE: No flags from the mask are set
PM_QOS_FLAGS_UNDEFINED: The device's PM QoS structure has not been
initialized or the list of requests is empty.
Check PM QoS flags of the given device against the given mask of flags.
The meaning of the return values is as follows:
PM_QOS_FLAGS_ALL:
All flags from the mask are set
PM_QOS_FLAGS_SOME:
Some flags from the mask are set
PM_QOS_FLAGS_NONE:
No flags from the mask are set
PM_QOS_FLAGS_UNDEFINED:
The device's PM QoS structure has not been initialized
or the list of requests is empty.
int dev_pm_qos_add_ancestor_request(dev, handle, type, value)
Add a PM QoS request for the first direct ancestor of the given device whose
power.ignore_children flag is unset (for DEV_PM_QOS_RESUME_LATENCY requests)
or whose power.set_latency_tolerance callback pointer is not NULL (for
DEV_PM_QOS_LATENCY_TOLERANCE requests).
Add a PM QoS request for the first direct ancestor of the given device whose
power.ignore_children flag is unset (for DEV_PM_QOS_RESUME_LATENCY requests)
or whose power.set_latency_tolerance callback pointer is not NULL (for
DEV_PM_QOS_LATENCY_TOLERANCE requests).
int dev_pm_qos_expose_latency_limit(device, value)
Add a request to the device's PM QoS list of resume latency constraints and
create a sysfs attribute pm_qos_resume_latency_us under the device's power
directory allowing user space to manipulate that request.
Add a request to the device's PM QoS list of resume latency constraints and
create a sysfs attribute pm_qos_resume_latency_us under the device's power
directory allowing user space to manipulate that request.
void dev_pm_qos_hide_latency_limit(device)
Drop the request added by dev_pm_qos_expose_latency_limit() from the device's
PM QoS list of resume latency constraints and remove sysfs attribute
pm_qos_resume_latency_us from the device's power directory.
Drop the request added by dev_pm_qos_expose_latency_limit() from the device's
PM QoS list of resume latency constraints and remove sysfs attribute
pm_qos_resume_latency_us from the device's power directory.
int dev_pm_qos_expose_flags(device, value)
Add a request to the device's PM QoS list of flags and create sysfs attribute
pm_qos_no_power_off under the device's power directory allowing user space to
change the value of the PM_QOS_FLAG_NO_POWER_OFF flag.
Add a request to the device's PM QoS list of flags and create sysfs attribute
pm_qos_no_power_off under the device's power directory allowing user space to
change the value of the PM_QOS_FLAG_NO_POWER_OFF flag.
void dev_pm_qos_hide_flags(device)
Drop the request added by dev_pm_qos_expose_flags() from the device's PM QoS list
of flags and remove sysfs attribute pm_qos_no_power_off from the device's power
directory.
Drop the request added by dev_pm_qos_expose_flags() from the device's PM QoS list
of flags and remove sysfs attribute pm_qos_no_power_off from the device's power
directory.
Notification mechanisms:
The per-device PM QoS framework has a per-device notification tree.
int dev_pm_qos_add_notifier(device, notifier):
Adds a notification callback function for the device.
The callback is called when the aggregated value of the device constraints list
is changed (for resume latency device PM QoS only).
Adds a notification callback function for the device.
The callback is called when the aggregated value of the device constraints list
is changed (for resume latency device PM QoS only).
int dev_pm_qos_remove_notifier(device, notifier):
Removes the notification callback function for the device.
Removes the notification callback function for the device.
Active state latency tolerance
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This device PM QoS type is used to support systems in which hardware may switch
to energy-saving operation modes on the fly. In those systems, if the operation
......
==========================
Regulator API design notes
==========================
......@@ -14,7 +15,9 @@ Safety
have different power requirements, and not all components with power
requirements are visible to software.
=> The API should make no changes to the hardware state unless it has
.. note::
The API should make no changes to the hardware state unless it has
specific knowledge that these changes are safe to perform on this
particular system.
......@@ -28,6 +31,8 @@ Consumer use cases
- Many of the power supplies in the system will be shared between many
different consumers.
=> The consumer API should be structured so that these use cases are
.. note::
The consumer API should be structured so that these use cases are
very easy to handle and so that consumers will work with shared
supplies without any additional effort.
==================================
Regulator Machine Driver Interface
===================================
==================================
The regulator machine driver interface is intended for board/machine specific
initialisation code to configure the regulator subsystem.
Consider the following machine :-
Consider the following machine::
Regulator-1 -+-> Regulator-2 --> [Consumer A @ 1.8 - 2.0V]
|
......@@ -13,31 +14,31 @@ Consider the following machine :-
The drivers for consumers A & B must be mapped to the correct regulator in
order to control their power supplies. This mapping can be achieved in machine
initialisation code by creating a struct regulator_consumer_supply for
each regulator.
each regulator::
struct regulator_consumer_supply {
struct regulator_consumer_supply {
const char *dev_name; /* consumer dev_name() */
const char *supply; /* consumer supply - e.g. "vcc" */
};
};
e.g. for the machine above
e.g. for the machine above::
static struct regulator_consumer_supply regulator1_consumers[] = {
static struct regulator_consumer_supply regulator1_consumers[] = {
REGULATOR_SUPPLY("Vcc", "consumer B"),
};
};
static struct regulator_consumer_supply regulator2_consumers[] = {
static struct regulator_consumer_supply regulator2_consumers[] = {
REGULATOR_SUPPLY("Vcc", "consumer A"),
};
};
This maps Regulator-1 to the 'Vcc' supply for Consumer B and maps Regulator-2
to the 'Vcc' supply for Consumer A.
Constraints can now be registered by defining a struct regulator_init_data
for each regulator power domain. This structure also maps the consumers
to their supply regulators :-
to their supply regulators::
static struct regulator_init_data regulator1_data = {
static struct regulator_init_data regulator1_data = {
.constraints = {
.name = "Regulator-1",
.min_uV = 3300000,
......@@ -46,7 +47,7 @@ static struct regulator_init_data regulator1_data = {
},
.num_consumer_supplies = ARRAY_SIZE(regulator1_consumers),
.consumer_supplies = regulator1_consumers,
};
};
The name field should be set to something that is usefully descriptive
for the board for configuration of supplies for other regulators and
......@@ -57,9 +58,9 @@ name is provided then the subsystem will choose one.
Regulator-1 supplies power to Regulator-2. This relationship must be registered
with the core so that Regulator-1 is also enabled when Consumer A enables its
supply (Regulator-2). The supply regulator is set by the supply_regulator
field below and co:-
field below and co::
static struct regulator_init_data regulator2_data = {
static struct regulator_init_data regulator2_data = {
.supply_regulator = "Regulator-1",
.constraints = {
.min_uV = 1800000,
......@@ -69,11 +70,11 @@ static struct regulator_init_data regulator2_data = {
},
.num_consumer_supplies = ARRAY_SIZE(regulator2_consumers),
.consumer_supplies = regulator2_consumers,
};
};
Finally the regulator devices must be registered in the usual manner.
Finally the regulator devices must be registered in the usual manner::
static struct platform_device regulator_devices[] = {
static struct platform_device regulator_devices[] = {
{
.name = "regulator",
.id = DCDC_1,
......@@ -88,9 +89,9 @@ static struct platform_device regulator_devices[] = {
.platform_data = &regulator2_data,
},
},
};
/* register regulator 1 device */
platform_device_register(&regulator_devices[0]);
};
/* register regulator 1 device */
platform_device_register(&regulator_devices[0]);
/* register regulator 2 device */
platform_device_register(&regulator_devices[1]);
/* register regulator 2 device */
platform_device_register(&regulator_devices[1]);
====================================
System Suspend and Device Interrupts
====================================
Copyright (C) 2014 Intel Corp.
Author: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
......
This diff is collapsed.
This diff is collapsed.
swsusp/S3 tricks
~~~~~~~~~~~~~~~~
================
swsusp/S3 tricks
================
Pavel Machek <pavel@ucw.cz>
If you want to trick swsusp/S3 into working, you might want to try:
......
......@@ -117,7 +117,7 @@ PM support:
implemented") error. You should also try to make sure that your
driver uses as little power as possible when it's not doing
anything. For the driver testing instructions see
Documentation/power/drivers-testing.txt and for a relatively
Documentation/power/drivers-testing.rst and for a relatively
complete overview of the power management issues related to
drivers see :ref:`Documentation/driver-api/pm/devices.rst <driverapi_pm_devices>`.
......
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment