there is no noticeable benefit for multiple cores to process one
job ring's output ring: in fact, we can benefit from cache effects
of having the back-half stay on the core that receives a particular
ring's interrupts, and further relax general contention and the
locking involved with reading outring_used, since tasklets run
atomically.
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Memory barriers are implied by the i/o register write implementation
(at least on Power). So we can remove the redundant wmb() in
caam_jr_enqueue, and, in dequeue(), hoist the h/w done notification
write up to before we need to increment the head of the ring, and
save an smp_mb.
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Code was needlessly checking the s/w job ring when there
would be nothing to process if the h/w's output completion
ring were empty anyway.
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The enqueue lock isn't used in any interrupt context, and
the dequeue lock isn't used in the h/w interrupt context,
only in bh context.
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
It has been observed that in zero-loss benchmarks, when a
slow traffic rate is being tested, the IRQ timer coalescing
parameter was set too high, and the ethernet controller
would start dropping packets because the job ring back half
wouldn't be executed in time before the ethernet controller
would fill its buffers, thereby significantly reducing the
zero-loss performance figures.
Empirical testing has shown that the best zero-loss performance
is achieved when IRQ coalescing is set to minimum values and/or
turned off, since apparently the job ring driver already implements
an adequately-performing general-purpose IRQ mitigation strategy
in software.
Whilst we could go with minimal count (2-8) and timing settings
(192-256), we prefer just turning h/w coalescing altogether off
to minimize setkey latency (due to split key generation), and
for consistent cross-SoC performance (the SEC vs. core clock
ratio changes).
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The SEC v4.x' RNGB h/w block self-initialized. RNG4, available
on SEC versions 5 and beyond, is based on a different standard
that requires manual initialization.
Also update any new errors From the SEC v5.2 reference manual:
The SEC v5.2's RNG4 unit reuses some error IDs, thus the addition
of rng_err_id_list over the CHA-independent err_id_list.
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
SEC v4.x were only 36-bit, SEC v5+ are 40-bit capable.
Also set a DMA mask for any job ring devices created.
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
caam_read copies random bytes from two buffers into output.
caam rng can fill empty buffer 0xffff bytes at a time,
but the buffer sizes are rounded down to multiple of cacheline size.
Signed-off-by: Yuan Kang <Yuan.Kang@freescale.com>
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
support chained scatterlists for aead, ablkcipher and ahash.
Signed-off-by: Yuan Kang <Yuan.Kang@freescale.com>
- fix dma unmap leak
- un-unlikely src == dst, due to experience with AF_ALG
Signed-off-by: Kudupudi Ugendreshwar <B38865@freescale.com>
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
caam supports and registers unkeyed sha algorithms and md5.
Signed-off-by: Yuan Kang <Yuan.Kang@freescale.com>
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
caam supports ahash hmac with sha algorithms and md5.
Signed-off-by: Yuan Kang <Yuan.Kang@freescale.com>
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
- rename scatterlist and link_tbl functions
- link_tbl changed to sec4_sg
- sg_to_link_tbl_one changed to dma_to_sec4_sg_one,
since no scatterlist is use
Signed-off-by: Yuan Kang <Yuan.Kang@freescale.com>
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
create separate files for split key generation and scatterlist functions.
Signed-off-by: Yuan Kang <Yuan.Kang@freescale.com>
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
remove caam_jr_register and caam_jr_deregister
to allow sharing of job rings.
Signed-off-by: Yuan Kang <Yuan.Kang@freescale.com>
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
functions for external storage of seq in/out lengths,
i.e., for 32-bit lengths.
These type-dependent functions automatically determine whether to
store the length internally (embedded in the command header word) or
externally (after the address pointer), based on size of the type
given.
Signed-off-by: Yuan Kang <Yuan.Kang@freescale.com>
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Add a PDB header file to support building protocol descriptors.
Signed-off-by: Steve Cornelius <sec@pobox.com>
Signed-off-by: Hemant Agrawal <hemant@freescale.com>
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
init_desc, by always ORing with 1 for the descriptor header inclusion
into the descriptor length, and init_sh_desc_pdb, by always specifying
the descriptor length modification for the PDB via options, would not
allow for odd length PDBs to be embedded in the constructed descriptor
length. Fix this by simply changing the OR to an addition.
also round-up pdb_bytes to the next SEC command unit size, to
allow for, e.g., optional packet header bytes that aren't a
multiple of CAAM_CMD_SZ.
Reported-by: Radu-Andrei BULIE <radu.bulie@freescale.com>
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Cc: Yashpal Dutta <yashpal.dutta@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
In case of protocol acceleration descriptors, Shared descriptor header must
carry size of header length + PDB length in words which will be skipped by
DECO while processing descriptor to provide first command word offset
Signed-off-by: Yashpal Dutta <yashpal.dutta@freescale.com>
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
SEC4 h/w gets configured in 32- vs. 36-bit physical
addressing modes depending on the size of dma_addr_t,
which is not always equal to sizeof(u32 *).
Also fixed alignment of a dma_unmap call whilst in there.
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
presumably leftovers from possible macro development.
Signed-off-by: Kim Phillips <kim.phillips@freescale.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Move AES header to the new asm/crypto directory.
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Move serpent crypto headers to the new asm/crypto/ directory.
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Now that shared glue code is available, convert twofish-avx to use it.
Cc: Johannes Goetzfried <Johannes.Goetzfried@informatik.stud.uni-erlangen.de>
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Now that shared glue code is available, convert twofish-x86_64-3way to use it.
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Now that shared glue code is available, convert camellia-x86_64 to use it.
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Now that shared glue code is available, convert serpent-avx to use it.
Cc: Johannes Goetzfried <Johannes.Goetzfried@informatik.stud.uni-erlangen.de>
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Now that serpent-sse2 glue code has been made generic, it can be split to
separate module.
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Block cipher implementations in arch/x86/crypto/ contain common glue code that
is currently duplicated in each module (camellia-x86_64, twofish-x86_64-3way,
twofish-avx, serpent-sse2 and serpent-avx). This patch prepares serpent-sse2
glue into generic glue code for all 128bit block ciphers to use in
arch/x86/crypto.
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Remove duplicate ablk_* functions and make use of ablk_helper module instead.
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Remove duplicate ablk_* functions and make use of ablk_helper module instead.
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Move ablk-* functions to separate module to share common code between cipher
implementations.
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
When the nx driver was pulled, the Makefile that actually
builds it is arch/powerpc/Makefile. This is unnatural.
This patch moves the line that builds the nx driver from
arch/powerpc/Makefile to drivers/crypto/Makefile where it
belongs.
Signed-off-by: Seth Jennings <sjenning@linux.vnet.ibm.com>
Acked-by: Kent Yoder <key@linux.vnet.ibm.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Commit 45001e9, which added support for RNGA, ignored the previous commit
984e976, which changed the data_present API.
Cc: Matt Mackall <mpm@selenic.com>
Cc: Sascha Hauer <kernel@pengutronix.de>
Cc: Alan Carvalho de Assis <acassis@gmail.com>
Cc: <linux-arm-kernel@lists.infradead.org>
Signed-off-by: Benoît Thébaudeau <benoit.thebaudeau@advansee.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
It has been observed that sometimes the crypto allocation code
will get stuck for 60 seconds or multiples thereof. This is
usually caused by an algorithm failing to pass the self-test.
If an algorithm fails to be constructed, we will immediately notify
all larval waiters. However, if it succeeds in construction, but
then fails the self-test, we won't notify anyone at all.
This patch fixes this by merging the notification in the case
where the algorithm fails to be constructed with that of the
the case where it pases the self-test. This way regardless of
what happens, we'll give the larval waiters an answer.
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Rename serpent-avx assembler functions so that they do not collide with
serpent-sse2 assembler functions when linking both versions in to same
kernel image.
Reported-by: Randy Dunlap <rdunlap@xenotime.net>
Cc: Johannes Goetzfried <Johannes.Goetzfried@informatik.stud.uni-erlangen.de>
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
This patch changes u8 in struct arc4_ctx and variables to u32 (as AMD seems
to have problem with u8 array). Below are tcrypt results of old 1-byte block
cipher versus ecb(arc4) with u8 and ecb(arc4) with u32.
tcrypt results, x86-64 (speed ratios: new-u32/old, new-u8/old):
u32 u8
AMD Phenom II : x3.6 x2.7
Intel Core 2 : x2.0 x1.9
tcrypt results, i386 (speed ratios: new-u32/old, new-u8/old):
u32 u8
Intel Atom N260 : x1.5 x1.4
Cc: Jon Oberheide <jon@oberheide.org>
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Currently arc4.c provides simple one-byte blocksize cipher which is wrapped
by ecb() module, giving function call overhead on every encrypted byte. This
patch adds ecb(arc4) directly into arc4.c for higher performance.
tcrypt results (speed ratios: new/old):
AMD Phenom II, x86-64 : x2.7
Intel Core 2, x86-64 : x1.9
Intel Atom N260, i386 : x1.4
Cc: Jon Oberheide <jon@oberheide.org>
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Commit 98971f8439 ("crypto: s390 - cleanup
DES code") should have also removed crypto_des.h. That file is unused
and unneeded since that commit. So let's clean up that file too.
Signed-off-by: Paul Bolle <pebolle@tiscali.nl>
Acked-by: Jan Glauber <jang@linux.vnet.ibm.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The AVX implementation of the twofish cipher processes 8 blocks parallel, so we
need to make test vectors larger to check parallel code paths. Test vectors are
also large enough to deal with 16 block parallel implementations which may occur
in the future.
Signed-off-by: Johannes Goetzfried <Johannes.Goetzfried@informatik.stud.uni-erlangen.de>
Signed-off-by: Jussi Kivilinna <jussi.kivilinna@mbnet.fi>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Since mv_hash_final_fallback() uses ctx->state, read out the digest
state register before calling it.
Signed-off-by: Phil Sutter <phil.sutter@viprinet.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The timer triggers when 500ms have gone by after triggering the engine
and no completion interrupt was received. The callback then tries to
sanitise things as well as possible.
Signed-off-by: Phil Sutter <phil.sutter@viprinet.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
The CRC peripheral is a hardware block used to compute the CRC of the block
of data. This is based on a CRC32 engine which computes the CRC value of 32b
data words presented to it. For data words of < 32b in size, this driver
pack 0 automatically into 32b data units. This driver implements the async
hash crypto framework API.
Signed-off-by: Sonic Zhang <sonic.zhang@analog.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Sonic Zhang <sonic.zhang@analog.com>
Acked-by: Mike Frysinger <vapier@gentoo.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Commit ea4d26ae ("raid5: add AVX optimized RAID5 checksumming")
introduced x86/ arch wide defines for AFLAGS and CFLAGS indicating AVX
support in binutils based on the same test we have in x86/crypto/ right
now. To minimize duplication drop our implementation in favour to the
one in x86/.
Signed-off-by: Mathias Krause <minipli@googlemail.com>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Pull crypto fixes from Herbert Xu:
"This push fixes an unaligned fault on x86-32 with aesni-intel and an
RNG failure with atmel-rng (repeated bits)."
* git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6:
crypto: aesni-intel - fix unaligned cbc decrypt for x86-32
hwrng: atmel-rng - fix race condition leading to repeated bits
Kim Phillips