- 23 Dec, 2018 14 commits
-
-
Nathan Chancellor authored
Clang warns when one enumerated type is implicitly converted to another: drivers/crypto/ux500/hash/hash_core.c:169:4: warning: implicit conversion from enumeration type 'enum dma_data_direction' to different enumeration type 'enum dma_transfer_direction' [-Wenum-conversion] direction, DMA_CTRL_ACK | DMA_PREP_INTERRUPT); ^~~~~~~~~ 1 warning generated. dmaengine_prep_slave_sg expects an enum from dma_transfer_direction. We know that the only direction supported by this function is DMA_TO_DEVICE because of the check at the top of this function so we can just use the equivalent value from dma_transfer_direction. DMA_TO_DEVICE = DMA_MEM_TO_DEV = 1 Signed-off-by:Nathan Chancellor <natechancellor@gmail.com> Reviewed-by:
Nick Desaulniers <ndesaulniers@google.com> Signed-off-by:
Herbert Xu <herbert@gondor.apana.org.au>
-
Nathan Chancellor authored
Clang warns when one enumerated type is implicitly converted to another: drivers/crypto/ux500/cryp/cryp_core.c:559:5: warning: implicit conversion from enumeration type 'enum dma_data_direction' to different enumeration type 'enum dma_transfer_direction' [-Wenum-conversion] direction, DMA_CTRL_ACK); ^~~~~~~~~ drivers/crypto/ux500/cryp/cryp_core.c:583:5: warning: implicit conversion from enumeration type 'enum dma_data_direction' to different enumeration type 'enum dma_transfer_direction' [-Wenum-conversion] direction, ^~~~~~~~~ 2 warnings generated. dmaengine_prep_slave_sg expects an enum from dma_transfer_direction. Because we know the value of the dma_data_direction enum from the switch statement, we can just use the proper value from dma_transfer_direction so there is no more conversion. DMA_TO_DEVICE = DMA_MEM_TO_DEV = 1 DMA_FROM_DEVICE = DMA_DEV_TO_MEM = 2 Signed-off-by: -
Dave Watson authored
Add the appropriate scatter/gather stubs to the avx asm. In the C code, we can now always use crypt_by_sg, since both sse and asm code now support scatter/gather. Introduce a new struct, aesni_gcm_tfm, that is initialized on startup to point to either the SSE, AVX, or AVX2 versions of the four necessary encryption/decryption routines. GENX_OPTSIZE is still checked at the start of crypt_by_sg. The total size of the data is checked, since the additional overhead is in the init function, calculating additional HashKeys. Signed-off-by:
Dave Watson <davejwatson@fb.com> Signed-off-by:
-
Dave Watson authored
Before this diff, multiple calls to GCM_ENC_DEC will succeed, but only if all calls are a multiple of 16 bytes. Handle partial blocks at the start of GCM_ENC_DEC, and update aadhash as appropriate. The data offset %r11 is also updated after the partial block. Signed-off-by:
-
Dave Watson authored
Introduce READ_PARTIAL_BLOCK macro, and use it in the two existing partial block cases: AAD and the end of ENC_DEC. In particular, the ENC_DEC case should be faster, since we read by 8/4 bytes if possible. This macro will also be used to read partial blocks between enc_update and dec_update calls. Signed-off-by:
-
Dave Watson authored
Prepare to handle partial blocks between scatter/gather calls. For the last partial block, we only want to calculate the aadhash in GCM_COMPLETE, and a new partial block macro will handle both aadhash update and encrypting partial blocks between calls. Signed-off-by:
-
Dave Watson authored
Fill in aadhash, aadlen, pblocklen, curcount with appropriate values. pblocklen, aadhash, and pblockenckey are also updated at the end of each scatter/gather operation, to be carried over to the next operation. Signed-off-by:
-
Dave Watson authored
The precompute functions differ only by the sub-macros they call, merge them to a single macro. Later diffs add more code to fill in the gcm_context_data structure, this allows changes in a single place. Signed-off-by:
-
Dave Watson authored
AAD hash only needs to be calculated once for each scatter/gather operation. Move it to its own macro, and call it from GCM_INIT instead of INITIAL_BLOCKS. Signed-off-by:
-
Dave Watson authored
Merge encode and decode tag calculations in GCM_COMPLETE macro. Scatter/gather routines will call this once at the end of encryption or decryption. Signed-off-by:
-
Dave Watson authored
Add support for 192/256-bit keys using the avx gcm/aes routines. The sse routines were previously updated in e31ac32d (Add support for 192 & 256 bit keys to AESNI RFC4106). Instead of adding an additional loop in the hotpath as in e31ac32d, this diff instead generates separate versions of the code using macros, and the entry routines choose which version once. This results in a 5% performance improvement vs. adding a loop to the hot path. This is the same strategy chosen by the intel isa-l_crypto library. The key size checks are removed from the c code where appropriate. Note that this diff depends on using gcm_context_data - 256 bit keys require 16 HashKeys + 15 expanded keys, which is larger than struct crypto_aes_ctx, so they are stored in struct gcm_context_data. Signed-off-by:
-
Dave Watson authored
Macro-ify function save and restore. These will be used in new functions added for scatter/gather update operations. Signed-off-by:
-
Dave Watson authored
Add the gcm_context_data structure to the avx asm routines. This will be necessary to support both 256 bit keys and scatter/gather. The pre-computed HashKeys are now stored in the gcm_context_data struct, which is expanded to hold the greater number of hashkeys necessary for avx. Loads and stores to the new struct are always done unlaligned to avoid compiler issues, see e5b954e8 "Use unaligned loads from gcm_context_data" Signed-off-by:
-
Dave Watson authored
The GCM_ENC_DEC routines for AVX and AVX2 are identical, except they call separate sub-macros. Pass the macros as arguments, and merge them. This facilitates additional refactoring, by requiring changes in only one place. The GCM_ENC_DEC macro was moved above the CONFIG_AS_AVX* ifdefs, since it will be used by both AVX and AVX2. Signed-off-by:
-
- 13 Dec, 2018 20 commits
-
-
Eric Biggers authored
crypto_alg_mod_lookup() takes a reference to the hash algorithm but crypto_init_shash_spawn() doesn't take ownership of it, hence the reference needs to be dropped in adiantum_create(). Fixes: 059c2a4d ("crypto: adiantum - add Adiantum support") Signed-off-by:
Eric Biggers <ebiggers@google.com> Signed-off-by:
-
Eric Biggers authored
CRYPTO_MSG_GETALG in NLM_F_DUMP mode sometimes doesn't return all registered crypto algorithms, because it doesn't support incremental dumps. crypto_dump_report() only permits itself to be called once, yet the netlink subsystem allocates at most ~64 KiB for the skb being dumped to. Thus only the first recvmsg() returns data, and it may only include a subset of the crypto algorithms even if the user buffer passed to recvmsg() is large enough to hold all of them. Fix this by using one of the arguments in the netlink_callback structure to keep track of the current position in the algorithm list. Then userspace can do multiple recvmsg() on the socket after sending the dump request. This is the way netlink dumps work elsewhere in the kernel; it's unclear why this was different (probably just an oversight). Also fix an integer overflow when calculating the dump buffer size hint. Fixes: a38f7907 ("crypto: Add userspace configuration API") Signed-off-by:
-
Eric Biggers authored
The 2018-11-28 revision of the Adiantum paper has revised some notation: - 'M' was replaced with 'L' (meaning "Left", for the left-hand part of the message) in the definition of Adiantum hashing, to avoid confusion with the full message - ε-almost-∆-universal is now abbreviated as ε-∆U instead of εA∆U - "block" is now used only to mean block cipher and Poly1305 blocks Also, Adiantum hashing was moved from the appendix to the main paper. To avoid confusion, update relevant comments in the code to match. Signed-off-by:
-
Eric Biggers authored
The kernel's ChaCha20 uses the RFC7539 convention of the nonce being 12 bytes rather than 8, so actually I only appended 12 random bytes (not 16) to its test vectors to form 24-byte nonces for the XChaCha20 test vectors. The other 4 bytes were just from zero-padding the stream position to 8 bytes. Fix the comments above the test vectors. Signed-off-by:
-
Eric Biggers authored
There is a draft specification for XChaCha20 being worked on. Add the XChaCha20 test vector from the appendix so that we can be extra sure the kernel's implementation is compatible. I also recomputed the ciphertext with XChaCha12 and added it there too, to keep the tests for XChaCha20 and XChaCha12 in sync. Signed-off-by:
-
Eric Biggers authored
To improve responsiveness, yield the FPU (temporarily re-enabling preemption) every 4 KiB encrypted/decrypted, rather than keeping preemption disabled during the entire encryption/decryption operation. Alternatively we could do this for every skcipher_walk step, but steps may be small in some cases, and yielding the FPU is expensive on x86. Suggested-by:
Martin Willi <martin@strongswan.org> Signed-off-by:
-
Eric Biggers authored
Now that the x86_64 SIMD implementations of ChaCha20 and XChaCha20 have been refactored to support varying the number of rounds, add support for XChaCha12. This is identical to XChaCha20 except for the number of rounds, which is 12 instead of 20. This can be used by Adiantum. Reviewed-by:
-
Eric Biggers authored
In preparation for adding XChaCha12 support, rename/refactor the x86_64 SIMD implementations of ChaCha20 to support different numbers of rounds. Reviewed-by:
-
Eric Biggers authored
Add an XChaCha20 implementation that is hooked up to the x86_64 SIMD implementations of ChaCha20. This can be used by Adiantum. An SSSE3 implementation of single-block HChaCha20 is also added so that XChaCha20 can use it rather than the generic implementation. This required refactoring the ChaCha permutation into its own function. Signed-off-by:
-
Eric Biggers authored
Add a 64-bit AVX2 implementation of NHPoly1305, an ε-almost-∆-universal hash function used in the Adiantum encryption mode. For now, only the NH portion is actually AVX2-accelerated; the Poly1305 part is less performance-critical so is just implemented in C. Signed-off-by:
-
Eric Biggers authored
Add a 64-bit SSE2 implementation of NHPoly1305, an ε-almost-∆-universal hash function used in the Adiantum encryption mode. For now, only the NH portion is actually SSE2-accelerated; the Poly1305 part is less performance-critical so is just implemented in C. Signed-off-by:
-
Eric Biggers authored
If the stream cipher implementation is asynchronous, then the Adiantum instance must be flagged as asynchronous as well. Otherwise someone asking for a synchronous algorithm can get an asynchronous algorithm. There are no asynchronous xchacha12 or xchacha20 implementations yet which makes this largely a theoretical issue, but it should be fixed. Fixes: 059c2a4d ("crypto: adiantum - add Adiantum support") Signed-off-by:
-
Ard Biesheuvel authored
To some degree, most known AArch64 micro-architectures appear to be able to issue ALU instructions in parellel to SIMD instructions without affecting the SIMD throughput. This means we can use the ALU to process a fifth ChaCha block while the SIMD is processing four blocks in parallel. Signed-off-by:
Ard Biesheuvel <ard.biesheuvel@linaro.org> Signed-off-by:
-
Ard Biesheuvel authored
Update the 4-way NEON ChaCha routine so it can handle input of any length >64 bytes in its entirety, rather than having to call into the 1-way routine and/or memcpy()s via temp buffers to handle the tail of a ChaCha invocation that is not a multiple of 256 bytes. On inputs that are a multiple of 256 bytes (and thus in tcrypt benchmarks), performance drops by around 1% on Cortex-A57, while performance for inputs drawn randomly from the range [64, 1024) increases by around 30%. Signed-off-by:
-
Ard Biesheuvel authored
In order to have better coverage of algorithms operating on block sizes that are in the ballpark of a VPN packet, add 1472 to the block_sizes array. Signed-off-by:
-
Srikanth, Jampala authored
Enabled the PF->VF Mailbox support. Mailbox message are interpreted as {type, opcode, data}. Supported message types are REQ, ACK and NACK. Signed-off-by:Srikanth Jampala <Jampala.Srikanth@cavium.com> Signed-off-by:
-
Eric Biggers authored
Now that the ARM64 NEON implementation of ChaCha20 and XChaCha20 has been refactored to support varying the number of rounds, add support for XChaCha12. This is identical to XChaCha20 except for the number of rounds, which is 12 instead of 20. This can be used by Adiantum. Reviewed-by:
-
Eric Biggers authored
In preparation for adding XChaCha12 support, rename/refactor the ARM64 NEON implementation of ChaCha20 to support different numbers of rounds. Reviewed-by:
-
Eric Biggers authored
Add an XChaCha20 implementation that is hooked up to the ARM64 NEON implementation of ChaCha20. This can be used by Adiantum. A NEON implementation of single-block HChaCha20 is also added so that XChaCha20 can use it rather than the generic implementation. This required refactoring the ChaCha20 permutation into its own function. Signed-off-by:
-
Eric Biggers authored
Add an ARM64 NEON implementation of NHPoly1305, an ε-almost-∆-universal hash function used in the Adiantum encryption mode. For now, only the NH portion is actually NEON-accelerated; the Poly1305 part is less performance-critical so is just implemented in C. Reviewed-by:
-
- 07 Dec, 2018 6 commits
-
-
Yangtao Li authored
Use DEFINE_SHOW_ATTRIBUTE macro to simplify the code. Signed-off-by:
Yangtao Li <tiny.windzz@gmail.com> Signed-off-by:
-
Atul Gupta authored
Send SPI, 64b seq nos and 64b IV with aadiv drop for inline crypto. This information is added in outgoing packet after the CPL TX PKT XT and removed by hardware. The aad, auth and cipher offsets are then adjusted for ESN enabled tunnel. Signed-off-by:
Atul Gupta <atul.gupta@chelsio.com> Signed-off-by:
-
Atul Gupta authored
Immediate packets sent to hardware should include the work request length in calculating the flits. WR occupy one flit and if not accounted result in invalid request which stalls the HW queue. Cc: stable@vger.kernel.org Signed-off-by:
-
Corentin Labbe authored
This patch add the crypto_stats_init() function. This will permit to remove some ifdef from __crypto_register_alg(). Signed-off-by:
Corentin Labbe <clabbe@baylibre.com> Signed-off-by:
-
Corentin Labbe authored
Since now all crypto stats are on their own structures, it is now useless to have the algorithm name in the err_cnt member. Signed-off-by:
-
Corentin Labbe authored
Like for userspace, this patch splits stats into multiple structures, one for each algorithm class. Signed-off-by:
-
