This patch avoids copy of buffered data to hash from bufnext to buf
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
SEC1 doesn't support S/G in descriptors so for hash operations,
the CPU has to build a buffer containing the buffered block and
the incoming data. This generates a lot of memory copies which
represents more than 50% of CPU time of a md5sum operation as
shown below with a 'perf record'.
|--86.24%-- kcapi_md_digest
| |
| |--86.18%-- _kcapi_common_vmsplice_chunk_fd
| | |
| | |--83.68%-- splice
| | | |
| | | |--83.59%-- ret_from_syscall
| | | | |
| | | | |--83.52%-- sys_splice
| | | | | |
| | | | | |--83.49%-- splice_from_pipe
| | | | | | |
| | | | | | |--83.04%-- __splice_from_pipe
| | | | | | | |
| | | | | | | |--80.67%-- pipe_to_sendpage
| | | | | | | | |
| | | | | | | | |--78.25%-- hash_sendpage
| | | | | | | | | |
| | | | | | | | | |--60.08%-- ahash_process_req
| | | | | | | | | | |
| | | | | | | | | | |--56.36%-- sg_copy_buffer
| | | | | | | | | | | |
| | | | | | | | | | | |--55.29%-- memcpy
| | | | | | | | | | | |
However, unlike SEC2+, SEC1 offers the possibility to chain
descriptors. It is therefore possible to build a first descriptor
pointing to the buffered data and a second descriptor pointing to
the incoming data, hence avoiding the memory copy to a single
buffer.
With this patch, the time necessary for a md5sum on a 90Mbytes file
is approximately 3 seconds. Without the patch it takes 6 seconds.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
At every request, we map and unmap the same hash hw_context.
This patch moves the dma mapping/unmapping in functions ahash_init()
and ahash_import().
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
dma_map_single() is an heavy operation which doesn't need to
be done at each request as the key doesn't change.
Instead of DMA mapping the key at every request, this patch maps it
once in setkey()
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Do (desc->hdr & DESC_HDR_TYPE_IPSEC_ESP) only once.
Limit number of if/else paths
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
to_talitos_ptr() and to_talitos_ptr_len() are always called together
in order to fully set a ptr, so lets merge them into a single
helper.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The number of channels is known from the beginning, no need to
test it everytime.
This patch defines two additional done functions handling only channel 0.
Then the probe registers the correct one based on the number of channels.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use of_property_read_u32() to simplify DT read
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
talitos_handle_buggy_hash() and talitos_sg_map() are only used
locally, make them static
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch zeroize the descriptor at allocation using memset().
This has two advantages:
- It reduces the number of places where data has to be set to 0
- It avoids reading memory and loading the cache with data that
will be entirely replaced.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
sg_link_tbl_len shall be used instead of cryptlen, otherwise
SECs which perform HW CICV verification will fail.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
sha224 AEAD test fails with:
[ 2.803125] talitos ff020000.crypto: DEUISR 0x00000000_00000000
[ 2.808743] talitos ff020000.crypto: MDEUISR 0x80100000_00000000
[ 2.814678] talitos ff020000.crypto: DESCBUF 0x20731f21_00000018
[ 2.820616] talitos ff020000.crypto: DESCBUF 0x0628d64c_00000010
[ 2.826554] talitos ff020000.crypto: DESCBUF 0x0631005c_00000018
[ 2.832492] talitos ff020000.crypto: DESCBUF 0x0628d664_00000008
[ 2.838430] talitos ff020000.crypto: DESCBUF 0x061b13a0_00000080
[ 2.844369] talitos ff020000.crypto: DESCBUF 0x0631006c_00000080
[ 2.850307] talitos ff020000.crypto: DESCBUF 0x0631006c_00000018
[ 2.856245] talitos ff020000.crypto: DESCBUF 0x063100ec_00000000
[ 2.884972] talitos ff020000.crypto: failed to reset channel 0
[ 2.890503] talitos ff020000.crypto: done overflow, internal time out, or rngu error: ISR 0x20000000_00020000
[ 2.900652] alg: aead: encryption failed on test 1 for authenc-hmac-sha224-cbc-3des-talitos: ret=22
This is due to SHA224 not being supported by the HW. Allthough for
hash we are able to init the hash context by SW, it is not
possible for AEAD. Therefore SHA224 AEAD has to be deactivated.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Crypto manager test report the following failures:
[ 3.061081] alg: skcipher: setkey failed on test 5 for ecb-des-talitos: flags=100
[ 3.069342] alg: skcipher-ddst: setkey failed on test 5 for ecb-des-talitos: flags=100
[ 3.077754] alg: skcipher-ddst: setkey failed on test 5 for ecb-des-talitos: flags=100
This is due to setkey being expected to detect weak keys.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Today, md5sum fails with error -ENOKEY because a setkey
function is set for non hmac hashing algs, see strace output below:
mmap(NULL, 378880, PROT_READ, MAP_SHARED, 6, 0) = 0x77f50000
accept(3, 0, NULL) = 7
vmsplice(5, [{"bin/\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 378880}], 1, SPLICE_F_MORE|SPLICE_F_GIFT) = 262144
splice(4, NULL, 7, NULL, 262144, SPLICE_F_MORE) = -1 ENOKEY (Required key not available)
write(2, "Generation of hash for file kcap"..., 50) = 50
munmap(0x77f50000, 378880) = 0
This patch ensures that setkey() function is set only
for hmac hashing.
Cc: <stable@vger.kernel.org>
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
md5sum on some files gives wrong result
Exemple:
With the md5sum from libkcapi:
c15115c05bad51113f81bdaee735dd09 test
With the original md5sum:
bbdf41d80ba7e8b2b7be3a0772be76cb test
This patch fixes this issue
Cc: <stable@vger.kernel.org>
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
An updated patch that also handles the additional key length requirements
for the AEAD algorithms.
The max keysize is not 96. For SHA384/512 it's 128, and for the AEAD
algorithms it's longer still. Extend the max keysize for the
AEAD size for AES256 + HMAC(SHA512).
Cc: <stable@vger.kernel.org> # 3.6+
Fixes: 357fb60502 ("crypto: talitos - add sha224, sha384 and sha512 to existing AEAD algorithms")
Signed-off-by: Martin Hicks <mort@bork.org>
Acked-by: Horia Geantă <horia.geanta@nxp.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Trivial fix to spelling mistake "pointeur" to "pointer"
in dev_err message
Signed-off-by: Colin Ian King <colin.king@canonical.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
SEC1 doesn't have IPSEC_ESP descriptor type but it is able to perform
IPSEC using HMAC_SNOOP_NO_AFEU, which is also existing on SEC2
In order to be able to define descriptors templates for SEC1 without
breaking SEC2+, we have to give lower priority to HMAC_SNOOP_NO_AFEU
so that SEC2+ selects IPSEC_ESP and not HMAC_SNOOP_NO_AFEU which is
less performant.
This is done by adding a priority field in the template. If the field
is 0, we use the default priority, otherwise we used the one in the
field.
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patchs enhances the IPSEC_ESP related functions for them to
also supports the same operations with descriptor type
HMAC_SNOOP_NO_AFEU.
The differences between the two descriptor types are:
* pointeurs 2 and 3 are swaped (Confidentiality key and
Primary EU Context IN)
* HMAC_SNOOP_NO_AFEU has CICV out in pointer 6
* HMAC_SNOOP_NO_AFEU has no primary EU context out so we get it
from the end of data out
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
In preparation of IPSEC for SEC1, first step is to make the mapping
helpers more generic so that they can also be used by AEAD functions.
First, the functions are moved before IPSEC functions in talitos.c
talitos_sg_unmap() and unmap_sg_talitos_ptr() are merged as they
are quite similar, the second one handling the SEC1 case an calling
the first one for SEC2
map_sg_in_talitos_ptr() and map_sg_out_talitos_ptr() are merged
into talitos_sg_map() and enhenced to support offseted zones
as used for AEAD. The actual mapping is now performed outside that
helper. The DMA sync is also done outside to not make it several
times.
talitos_edesc_alloc() size calculation are fixed to also take into
account AEAD specific parts also for SEC1
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
In order to be able to use the mapping/unmapping helpers for IPSEC
it needs to be move upper in the file
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Use helper for all modifications to talitos_ptr in preparation to
the implementation of AEAD for SEC1
to_talitos_ptr_extent_clear() has been removed in favor of
to_talitos_ptr_ext_set() to set any value and
to_talitos_ptr_ext_or() to or the extent field with a value
name has been shorten to help keeping single lines of 80 chars
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Provide hardware state import/export functionality, as mandated by
commit 8996eafdcb ("crypto: ahash - ensure statesize is non-zero")
Cc: <stable@vger.kernel.org> # 4.3+
Reported-by: Jonas Eymann <J.Eymann@gmx.net>
Signed-off-by: Horia Geant? <horia.geanta@nxp.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
After conversion to new AEAD interface, tcrypt tests fail as follows:
[...]
[ 1.145414] alg: aead: Test 1 failed on encryption for authenc-hmac-sha1-cbc-aes-talitos
[ 1.153564] 00000000: 53 69 6e 67 6c 65 20 62 6c 6f 63 6b 20 6d 73 67
[ 1.160041] 00000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 1.166509] 00000020: 00 00 00 00
[...]
Fix them by providing the correct cipher in & cipher out pointers,
i.e. must skip over associated data in src and dst S/G.
While here, fix a problem with the HW S/G table index usage:
tbl_off must be updated after the pointer to the table entries is set.
Cc: <stable@vger.kernel.org> # 4.3+
Fixes: aeb4c132f3 ("crypto: talitos - Convert to new AEAD interface")
Reported-by: Jonas Eymann <J.Eymann@gmx.net>
Signed-off-by: Horia Geant? <horia.geanta@nxp.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch adds the following algorithms to the talitos driver:
* ecb(aes)
* ctr(aes)
* ecb(des)
* cbc(des)
* ecb(des3_ede)
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The sg_nents_for_len() function could fail, this patch add a check for
its return value.
Signed-off-by: LABBE Corentin <clabbe.montjoie@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Using non-constant time memcmp() makes the verification of the authentication
tag in the decrypt path vulnerable to timing attacks. Fix this by using
crypto_memneq() instead.
Cc: stable@vger.kernel.org
Signed-off-by: David Gstir <david@sigma-star.at>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The talitos driver use two dma_map_sg path
according to SG are chained or not.
Since dma_map_sg can handle both case, clean the code with all
references to sg chained.
Thus removing talitos_map_sg, talitos_unmap_sg_chain
and sg_count functions.
Signed-off-by: LABBE Corentin <clabbe.montjoie@gmail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Pull SG updates from Jens Axboe:
"This contains a set of scatter-gather related changes/fixes for 4.3:
- Add support for limited chaining of sg tables even for
architectures that do not set ARCH_HAS_SG_CHAIN. From Christoph.
- Add sg chain support to target_rd. From Christoph.
- Fixup open coded sg->page_link in crypto/omap-sham. From
Christoph.
- Fixup open coded crypto ->page_link manipulation. From Dan.
- Also from Dan, automated fixup of manual sg_unmark_end()
manipulations.
- Also from Dan, automated fixup of open coded sg_phys()
implementations.
- From Robert Jarzmik, addition of an sg table splitting helper that
drivers can use"
* 'for-4.3/sg' of git://git.kernel.dk/linux-block:
lib: scatterlist: add sg splitting function
scatterlist: use sg_phys()
crypto/omap-sham: remove an open coded access to ->page_link
scatterlist: remove open coded sg_unmark_end instances
crypto: replace scatterwalk_sg_chain with sg_chain
target/rd: always chain S/G list
scatterlist: allow limited chaining without ARCH_HAS_SG_CHAIN
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
[hch: split from a larger patch by Dan]
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Jens Axboe <axboe@fb.com>
The probe error path for this driver, for all intents and purposes,
is the talitos_remove() function due to the common "goto err_out".
Without this patch applied, talitos_remove() will panic under these
two conditions:
1. If the RNG device hasn't been registered via
talitos_register_rng() prior to entry into talitos_remove(),
then the attempt to unregister the RNG "device" will cause a panic.
2. If the priv->chan array has not been allocated prior to entry
into talitos_remove(), then the per-channel FIFO cleanup will panic
because of the dereference of that NULL "array".
Both of the above scenarios occur if talitos_probe_irq() fails.
This patch resolves issue #1 by introducing a boolean to mask the
hwrng_unregister() call in talitos_unregister_rng() if RNG device
registration was unsuccessful.
It resolves issue #2 by checking that priv->chan is not NULL in the
per-channel FIFO cleanup for loop.
Signed-off-by: Aaron Sierra <asierra@xes-inc.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch converts talitos to the new AEAD interface. IV generation
has been removed since it's equivalent to a software implementation.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
-change req_ctx->nbuf from u64 to unsigned int to silence checker
warnings; this is safe since nbuf value is <= HASH_MAX_BLOCK_SIZE
-remove unused value read from TALITOS_CCPSR; there is no requirement
to read upper 32b before reading lower 32b of a 64b register;
SEC RM mentions: "reads can always be done by byte, word, or dword"
-remove unused return value of sg_to_link_tbl()
-change "len" parameter of map_single_talitos_ptr() and
to_talitos_ptr_len() to unsigned int; later, cpu_to_be16 will __force
downcast the value to unsigned short without any checker warning
Signed-off-by: Horia Geanta <horia.geanta@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Check return value of scatterlist_sg_next(), i.e. don't rely solely
on number of bytes to be processed or number of scatterlist entries.
Signed-off-by: Horia Geanta <horia.geanta@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This reverts commit 7291a932c6.
The conversion to be16_add_cpu() is incorrect in case cryptlen is
negative due to premature (i.e. before addition / subtraction)
implicit conversion of cryptlen (int -> u16) leading to sign loss.
Cc: <stable@vger.kernel.org> # 3.10+
Cc: Wei Yongjun <yongjun_wei@trendmicro.com.cn>
Signed-off-by: Horia Geanta <horia.geanta@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Cc: <stable@vger.kernel.org> # 3.2+
Fixes: 1d11911a8c ("crypto: talitos - fix warning: 'alg' may be used uninitialized in this function")
Signed-off-by: Horia Geanta <horia.geanta@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The + operation has higher precedence than ?: so we need parentheses
here. Otherwise we may end up allocating a max of only one "cryptlen"
instead of two.
Fixes: 6f65f6ac5f ('crypto: talitos - implement scatter/gather copy for SEC1')
Signed-off-by: Dan Carpenter <dan.carpenter@oracle.com>
Acked-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
We add a specific compatible for SEC1, to handle the differences
between SEC1 and SEC2+
Signed-off-by: Christophe Leroy <christophe.leroy@c-s.fr>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
LEROY Christophe