- 05 Sep, 2024 5 commits
-
-
Harald Freudenberger authored
Add support for deriving protected keys from clear key token for AES xts and HMAC keys via PCKMO instruction. Add support for protected key generation and unwrap of protected key tokens for these key types. Furthermore 4 new sysfs attributes are introduced: - /sys/devices/virtual/misc/pkey/protkey/protkey_aes_xts_128 - /sys/devices/virtual/misc/pkey/protkey/protkey_aes_xts_256 - /sys/devices/virtual/misc/pkey/protkey/protkey_hmac_512 - /sys/devices/virtual/misc/pkey/protkey/protkey_hmac_1024 Signed-off-by:
Harald Freudenberger <freude@linux.ibm.com> Reviewed-by:
Ingo Franzki <ifranzki@linux.ibm.com> Signed-off-by:
Heiko Carstens <hca@linux.ibm.com>
-
Harald Freudenberger authored
Add the defines for the new PCKMO functions covering MSA 10 (AES XTS "double" keys) and MSA 11 (HMAC keys) support. Signed-off-by:
-
Gerald Schaefer authored
Adding/removing large amount of pages at once to/from the CMM balloon can result in rcu_sched stalls or workqueue lockups, because of busy looping w/o cond_resched(). Prevent this by adding a cond_resched(). cmm_free_pages() holds a spin_lock while looping, so it cannot be added directly to the existing loop. Instead, introduce a wrapper function that operates on maximum 256 pages at once, and add it there. Signed-off-by:
Gerald Schaefer <gerald.schaefer@linux.ibm.com> Reviewed-by:
-
Thomas Richter authored
Update the internal array of PAI extension 1 NNPA counter string table to support specialized processor instrumentation assist instructions. Signed-off-by:
Thomas Richter <tmricht@linux.ibm.com> Acked-by:
Sumanth Korikkar <sumanthk@linux.ibm.com> Signed-off-by:
-
Thomas Richter authored
Update the internal array of PAI crypto counter string table with new counters supported with Message Security Assist extension (MSA) 10 and MSA 11. Signed-off-by:
Finn Callies <fcallies@linux.ibm.com> Tested-by:
-
- 29 Aug, 2024 35 commits
-
-
Mete Durlu authored
Add three counters to follow and understand hiperdispatch behavior; * adjustment_count (amount of capacity adjustments triggered) * greedy_time_ms (time spent while all cpus are on high capacity) * conservative_time_ms (time spent while only entitled cpus are on high capacity) These counters can be found under /sys/kernel/debug/s390/hiperdispatch/ Time counters are in <msec> format and only cover the time spent when hiperdispatch is active. Acked-by:
Vasily Gorbik <gor@linux.ibm.com> Signed-off-by:
Mete Durlu <meted@linux.ibm.com> Signed-off-by:
-
Mete Durlu authored
Add two attributes for debug purposes. They can be found under; /sys/devices/system/cpu/hiperdispatch/ * hd_stime_threshold : allows user to adjust steal time threshold * hd_delay_factor : allows user to adjust delay factor of hiperdispatch work (after topology updates, delayed work is always delayed extra by this factor) hd_stime_threshold can have values between 0-100 as it represents a percentage value. hd_delay_factor can have values greater than 1. It is multiplied with the default delay to achieve a longer interval, pushing back the next hiperdispatch adjustment after a topology update. Ex: if delay interval is 250ms and the delay factor is 4; delayed interval is now 1000ms(1sec). After each capacity adjustment or topology change, work has a delayed interval of 1 sec for one interval. Acked-by: -
Mete Durlu authored
Expose hiperdispatch controls via sysctl. The user can now toggle hiperdispatch via assigning 0 or 1 to s390.hiperdispatch attribute. When hiperdipatch is toggled on, it tries to adjust CPU capacities, while system is in vertical polarization to gain performance benefits from different CPU polarizations. Disabling hiperdispatch reverts the CPU capacities to their default (HIGH_CAPACITY) and stops the dynamic adjustments. Introduce a kconfig option HIPERDISPATCH_ON which allows users to use hiperdispatch by default on vertical polarization. Using the sysctl attribute s390.hiperdispatch would overwrite this behavior. Acked-by:
-
Mete Durlu authored
Add trace events to debug hiperdispatch behavior and track domain rebuilding. Two events provide information about the decision making of hiperdispatch and the adjustments made. Acked-by:
Tobias Huschle <huschle@linux.ibm.com> Signed-off-by:
-
Mete Durlu authored
The measurements done by hiperdispatch can have sudden spikes and dips during run time. To prevent these outliers effecting the decision making process and causing adjustment overhead, use weighted average of the steal time. Acked-by:
-
Mete Durlu authored
When LPAR is in vertical polarization, CPUs get different polarization values, namely vertical high, vertical medium and vertical low. These values represent the likelyhood of the CPU getting physical runtime. Vertical high CPUs will always get runtime and others get varying runtime depending on the load the CEC is under. Vertical high and vertical medium CPUs are considered the CPUs which the current LPAR has the entitlement to run on. The vertical lows are on the other hand are borrowed CPUs which would only be given to the LPAR by hipervisor when the other LPARs are not utilizing them. Using the CPU capacities, hint linux scheduler when it should prioritise vertical high and vertical medium CPUs over vertical low CPUs. By tracking various system statistics hiperdispatch determines when to adjust cpu capacities. After each adjustment, rebuilding of scheduler domains is necessary to notify the scheduler about capacity changes but since this operation is costly it should be done as sparsely as possible. Acked-by:
-
Mete Durlu authored
Linux scheduler allows architectures to assign capacity values to individual CPUs. This hints scheduler the performance difference between CPUs and allows more efficient task distribution them. Implement helper methods to set and get CPU capacities for s390. This is particularly helpful in vertical polarization configurations of LPARs. On vertical polarization an LPARs CPUs can get different polarization values depending on the CEC configuration. CPUs with different polarization values can perform different from each other, using CPU capacities this can be reflected to linux scheduler. Acked-by:
-
Tobias Huschle authored
By default, all systems on s390 start in horizontal cpu polarization. Selecting the new config option SCHED_TOPOLOGY_VERTICAL allows to build a kernel that switches to vertical polarization during boot. Acked-by:
-
Tobias Huschle authored
Provide an additional path to set the polarization of the system, such that a user no longer relies on the sysfs interface only and is able configure the polarization for every reboot via sysctl control files. The new sysctl can be set as follows: - s390.polarization=0 for horizontal polarization - s390.polarization=1 for vertical polarization Acked-by:
-
Tobias Huschle authored
Introduce a new debug file which allows to determine how many warning track grace periods were missed on each CPU. The new file can be found as /sys/kernel/debug/s390/wti It is formatted as: CPU0 CPU1 [...] CPUx xyz xyz [...] xyz Acked-by: -
Tobias Huschle authored
A virtual CPU that has received a warning-track interrupt may fail to acknowledge the interrupt within the warning-track grace period. While this is usually not a problem, it will become necessary to investigate if there is a large number of such missed warning-track interrupts. Therefore, it is necessary to track these events. The information is tracked through the s390 debug facility and can be found under /sys/kernel/debug/s390dbf/wti/. The hex_ascii output is formatted as: <pid> <symbol> The values pid and current psw are collected when a warning track interrupt is received. Symbol is either the kernel symbol matching the collected psw or redacted to <user> when running in user space. Each line represents the currently executing process when a warning track interrupt was received which was then not acknowledged within its grace period. Acked-by:
-
Tobias Huschle authored
When a warning track interrupt is received, the kernel has only a very limited amount of time to make sure, that the CPU can be yielded as gracefully as possible before being pre-empted by the hypervisor. The interrupt handler for the wti therefore unparks a kernel thread which has being created on boot re-using the CPU hotplug kernel thread infrastructure. These threads exist per CPU and are assigned the highest possible real-time priority. This makes sure, that said threads will execute as soon as possible as the scheduler should pre-empt any other running user tasks to run the real-time thread. Furthermore, the interrupt handler disables all I/O interrupts to prevent additional interrupt processing on the soon-preempted CPU. Interrupt handlers are likely to take kernel locks, which in the worst case, will be kept while the interrupt handler is pre-empted from itself underlying physical CPU. In that case, all tasks or interrupt handlers on other CPUs would have to wait for the pre-empted CPU being dispatched again. By preventing further interrupt processing, this risk is minimized. Once the CPU gets dispatched again, the real-time kernel thread regains control, reenables interrupts and parks itself again. Acked-by:
-
Tobias Huschle authored
The warning-track interrupt (wti) provides a notification that the receiving CPU will be pre-empted from its physical CPU within a short time frame. This time frame is called grace period and depends on the machine type. Giving up the CPU on time may prevent a task to get stuck while holding a resource. Reviewed-by:
-
Gaosheng Cui authored
The hypfs_dbfs_exit() have been removed since commit 3325b4d8 ("s390/hypfs: factor out filesystem code"), and now it is useless, so remove it. Signed-off-by:
Gaosheng Cui <cuigaosheng1@huawei.com> Acked-by:
-
Vasily Gorbik authored
The only context where ftrace_enable_ftrace_graph_caller() or ftrace_disable_ftrace_graph_caller() is called also calls ftrace_arch_code_modify_post_process(), which already performs text_poke_sync_lock(). ftrace_run_update_code() arch_ftrace_update_code() ftrace_modify_all_code() ftrace_enable_ftrace_graph_caller()/ftrace_disable_ftrace_graph_caller() ftrace_arch_code_modify_post_process() text_poke_sync_lock() Remove the redundant serialization. Reviewed-by:
-
Vasily Gorbik authored
Use get/copy_from_kernel_nofault to access the kernel text consistently. Replace memcmp() in ftrace_init_nop() to ensure that in case of inconsistencies in the 'mcount' table, the kernel reports a failure instead of potentially crashing. Reviewed-by:
-
Vasily Gorbik authored
When a sequential instruction fetching facility is present, it is safe to patch ftrace NOPs in function prologues. All of them are 8-byte aligned. Reviewed-by:
-
Vasily Gorbik authored
Avoid stop machine on kprobes arm/disarm when sequential instruction fetching is present. Reviewed-by:
-
Vasily Gorbik authored
When sequential instruction fetching facility is present, certain guarantees are provided for code patching. In particular, atomic overwrites within 8 aligned bytes is safe from an instruction-fetching point of view. Reviewed-by:
-
Sven Schnelle authored
In the past two save areas existed because interrupt handlers and system call / program check handlers where entered with interrupts enabled. To prevent a handler from overwriting the save areas from the previous handler, interrupts used the async save area, while system call and program check handler used the sync save area. Since the removal of critical section cleanup from entry.S, handlers are entered with interrupts disabled. When the interrupts are re-enabled, the save area is no longer need. Therefore merge both save areas into one. Reviewed-by:
Alexander Gordeev <agordeev@linux.ibm.com> Signed-off-by:
Sven Schnelle <svens@linux.ibm.com> Signed-off-by:
-
Harald Freudenberger authored
There is a possibility to deadlock with an recursive lock of the AP bus scan mutex ap_scan_bus_mutex: ... kernel: ============================================ ... kernel: WARNING: possible recursive locking detected ... kernel: 5.14.0-496.el9.s390x #3 Not tainted ... kernel: -------------------------------------------- ... kernel: kworker/12:1/130 is trying to acquire lock: ... kernel: 0000000358bc1510 (ap_scan_bus_mutex){+.+.}-{3:3}, at: ap_bus_force_rescan+0x92/0x108 ... kernel: but task is already holding lock: ... kernel: 0000000358bc1510 (ap_scan_bus_mutex){+.+.}-{3:3}, at: ap_scan_bus_wq_callback+0x28/0x60 ... kernel: other info that might help us debug this: ... kernel: Possible unsafe locking scenario: ... kernel: CPU0 ... kernel: ---- ... kernel: lock(ap_scan_bus_mutex); ... kernel: lock(ap_scan_bus_mutex); ... kernel: *** DEADLOCK *** Here is how the callstack looks like: ... [<00000003576fe9ce>] process_one_work+0x2a6/0x748 ... [<0000000358150c00>] ap_scan_bus_wq_callback+0x40/0x60 <- mutex locked ... [<00000003581506e2>] ap_scan_bus+0x5a/0x3b0 ... [<000000035815037c>] ap_scan_adapter+0x5b4/0x8c0 ... [<000000035814fa34>] ap_scan_domains+0x2d4/0x668 ... [<0000000357d989b4>] device_add+0x4a4/0x6b8 ... [<0000000357d9bb54>] bus_probe_device+0xb4/0xc8 ... [<0000000357d9daa8>] __device_attach+0x120/0x1b0 ... [<0000000357d9a632>] bus_for_each_drv+0x8a/0xd0 ... [<0000000357d9d548>] __device_attach_driver+0xc0/0x140 ... [<0000000357d9d3d8>] driver_probe_device+0x40/0xf0 ... [<0000000357d9cec2>] really_probe+0xd2/0x460 ... [<000000035814d7b0>] ap_device_probe+0x150/0x208 ... [<000003ff802a5c46>] zcrypt_cex4_queue_probe+0xb6/0x1c0 [zcrypt_cex4] ... [<000003ff7fb2d36e>] zcrypt_queue_register+0xe6/0x1b0 [zcrypt] ... [<000003ff7fb2c8ac>] zcrypt_rng_device_add+0x94/0xd8 [zcrypt] ... [<0000000357d7bc52>] hwrng_register+0x212/0x228 ... [<0000000357d7b8c2>] add_early_randomness+0x102/0x110 ... [<000003ff7fb29c94>] zcrypt_rng_data_read+0x94/0xb8 [zcrypt] ... [<0000000358150aca>] ap_bus_force_rescan+0x92/0x108 ... [<0000000358177572>] mutex_lock_interruptible_nested+0x32/0x40 <- lock again Note this only happens when the very first random data providing crypto card appears via hot plug in the system AND is in disabled state ("deconfig"). Then the initial pull of random data fails and a re-scan of the AP bus is triggered while already in the middle of an AP bus scan caused by the appearing new hardware. The fix is relatively simple once the scenario us understood: The AP bus force rescan function will immediately return if there is currently an AP bus scan running with the very same thread id. Fixes: eacf5b36 ("s390/ap: introduce mutex to lock the AP bus scan") Signed-off-by: -
Joerg Schmidbauer authored
On newer machines the SHA3 performance of CPACF instructions KIMD and KLMD can be enhanced by using additional modifier bits. This allows the application to omit initializing the ICV, but also affects the internal processing of the instructions. Performance is mostly gained when processing short messages. The new CPACF feature is backwards compatible with older machines, i.e. the new modifier bits are ignored on older machines. However, to save the ICV initialization, the application must detect the MSA level and omit the ICV initialization only if this feature is supported. Reviewed-by:
Holger Dengler <dengler@linux.ibm.com> Signed-off-by:
Joerg Schmidbauer <jschmidb@de.ibm.com> Signed-off-by:
-
Harald Freudenberger authored
There is a use case during early boot with an secure key encrypted root file system where the paes cipher may try to derive a protected key from secure key while the AP bus is still in the process of scanning the bus and building up the zcrypt device drivers. As the detection of CEX cards also triggers the modprobe of the pkey handler modules, these modules may come into existence too late. Yet another use case happening during early boot is for use of an protected key encrypted swap file(system). There is an ephemeral protected key read via sysfs to set up the swap file. But this only works when the pkey_pckmo module is already in - which may happen at a later time as the load is triggered via CPU feature. This patch introduces a new function pkey_handler_request_modules() and invokes it which unconditional tries to load in the pkey handler modules. This function is called for the in-kernel API to derive a protected key from whatever and in the sysfs API when the first attempt to simple invoke the handler function failed. Signed-off-by:
-
Harald Freudenberger authored
For some keys there exists an alternative but usually slower path to convert the key material into a protected key. This patch introduces a new handler function slowpath_key_to_protkey() which provides this alternate path for the CCA and EP11 handler code. With that even the knowledge about how and when this can be used within the pkey API code can be removed. So now the pkey API just tries the primary way and if that fails simple tries the alternative way. Signed-off-by:
-
Harald Freudenberger authored
Introduce pkey base kernel code with a simple pkey handler registry. Regroup the pkey code into these kernel modules: - pkey is the pkey api supporting the ioctls, sysfs and in-kernel api. Also the pkey base code which offers the handler registry and handler wrapping invocation functions is integrated there. This module is automatically loaded in via CPU feature if the MSA feature is available. - pkey-cca is the CCA related handler code kernel module a offering CCA specific implementation for pkey. This module is loaded in via MODULE_DEVICE_TABLE when a CEX[4-8] card becomes available. - pkey-ep11 is the EP11 related handler code kernel module offering an EP11 specific implementation for pkey. This module is loaded in via MODULE_DEVICE_TABLE when a CEX[4-8] card becomes available. - pkey-pckmo is the PCKMO related handler code kernel module. This module is loaded in via CPU feature if the MSA feature is available, but on init a check for availability of the pckmo instruction is performed. The handler modules register via a pkey_handler struct at the pkey base code and the pkey customer (that is currently the pkey api code fetches a handler via pkey handler registry functions and calls the unified handler functions via the pkey base handler functions. As a result the pkey-cca, pkey-ep11 and pkey-pckmo modules get independent from each other and it becomes possible to write new handlers which offer another kind of implementation without implicit dependencies to other handler implementations and/or kernel device drivers. For each of these 4 kernel modules there is an individual Kconfig entry: CONFIG_PKEY for the base and api, CONFIG_PKEY_CCA for the PKEY CCA support handler, CONFIG_PKEY_EP11 for the EP11 support handler and CONFIG_PKEY_PCKMO for the pckmo support. The both CEX related handler modules (PKEY CCA and PKEY EP11) have a dependency to the zcrypt api of the zcrypt device driver. Signed-off-by:
-
Harald Freudenberger authored
As a preparation step for introducing a common function API between the pkey API module and the handlers (that is the cca, ep11 and pckmo code) this patch unifies the functions signatures exposed by the handlers and reworks all the invocation code of these functions. Signed-off-by:
-
Harald Freudenberger authored
This is a huge rework of all the pkey kernel module code. The goal is to split the code into individual parts with a dedicated calling interface: - move all the sysfs related code into pkey_sysfs.c - all the CCA related code goes to pkey_cca.c - the EP11 stuff has been moved to pkey_ep11.c - the PCKMO related code is now in pkey_pckmo.c The CCA, EP11 and PCKMO code may be seen as "handlers" with a similar calling interface. The new header file pkey_base.h declares this calling interface. The remaining code in pkey_api.c handles the ioctl, the pkey module things and the "handler" independent code on top of the calling interface invoking the handlers. This regrouping of the code will be the base for a real pkey kernel module split into a pkey base module which acts as a dispatcher and handler modules providing their service. Signed-off-by:
-
Harald Freudenberger authored
Split the very huge ioctl handling function pkey_unlocked_ioctl() into individual functions per each IOCTL command. There is no change in functional code coming with this patch. The work is a simple copy-and-paste with the goal to have the functionality absolutely untouched. Signed-off-by:
-
Holger Dengler authored
Add new shash exploiting the HMAC hardware accelerations for SHA224, SHA256, SHA384 and SHA512 introduced with message-security assist extension 11. Reviewed-by:
-
Holger Dengler authored
Add new cipher exploiting the full AES-XTS hardware acceleration introduced with message-security assist extension 10. The full AES-XTS cipher is registered as preferred cipher in addition to the discrete AES-XTS variant. Reviewed-by:
-
Mete Durlu authored
Remove leftover dentry variable after hypfs refactoring. Before 2fcb3686, hypfs_diag.c and other hypfs files were using debugfs_create_file() explicitly for creating debugfs files and were storing the returned pointer. After the refactor, common debugfs file operations and also the related dentry pointers have been moved into hypfs_dbfs.c and redefined as new common mechanisms. Therefore the dentry variable and the debugfs_remove() function calls in hypfs_diag.c are now redundant. Current code is not effected since the dentry pointer in hypfs_diag is implicitly assigned to NULL and debugfs_remove() returns without an error if the passed pointer is NULL. Acked-by:
-
Vasily Gorbik authored
Add more instructions to the kernel disassembler. Reviewed-by:
Jens Remus <jremus@linux.ibm.com> Reviewed-by:
-
Vasily Gorbik authored
Since commit ba05b39d ("s390/expoline: Make modules use kernel expolines"), there is no longer any reason not to use CONFIG_EXPOLINE_EXTERN when supported by the compiler. On the positive side: - there is only a single set of expolines generated and used by both the kernel code and modules, - it eliminates expolines "comdat" sections, which can confuse tools like kpatch. Always enable EXPOLINE_EXTERN if supported by the compiler. Suggested-by:
-
Jens Remus authored
Over the course of CPU generations a few instructions got extended, changing their base mnemonic, while keeping the former as an extended mnemonic. Update the instruction mnemonics in the disassembler to their latest base mnemonic as documented in the latest IBM z/Architecture Principles of Operation specification [1]. With the IBM z14 the base mnemonics of the following vector instructions have been changed: - Vector FP Load Lengthened (VFLL) - Vector FP Load Rounded (VFLR) With Message-Security-Assist Extension 5 Perform Pseudorandom Number Operation (PPNO) has been renamed to Perform Random Number Operation (PRNO). With Vector Enhancements Facility 2 the base mnemonics of the following vector instructions have been changed: - Vector FP Convert from Fixed (VCFPS) - Vector FP Convert from Logical (VCFPL) - Vector FP Convert to Fixed (VCSFP) - Vector FP Convert to Logical (VCLFP) [1] IBM z/Architecture Principles of Operation, SA22-7832-13, IBM z16, https://publibfp.dhe.ibm.com/epubs/pdf/a227832d.pdfAcked-by: -
Jens Remus authored
Treat register numbers as unsigned. Treat signed operand values as signed. This resolves multiple instances of the Cppcheck warning: warning: %i in format string (no. 1) requires 'int' but the argument type is 'unsigned int'. [invalidPrintfArgType_sint] Acked-by:
-
