linux-sg2042/arch/x86/crypto
Martin Willi c70f4abef0 crypto: poly1305 - Add a SSE2 SIMD variant for x86_64
Implements an x86_64 assembler driver for the Poly1305 authenticator. This
single block variant holds the 130-bit integer in 5 32-bit words, but uses
SSE to do two multiplications/additions in parallel.

When calling updates with small blocks, the overhead for kernel_fpu_begin/
kernel_fpu_end() negates the perfmance gain. We therefore use the
poly1305-generic fallback for small updates.

For large messages, throughput increases by ~5-10% compared to
poly1305-generic:

testing speed of poly1305 (poly1305-generic)
test  0 (   96 byte blocks,   16 bytes per update,   6 updates): 4080026 opers/sec,  391682496 bytes/sec
test  1 (   96 byte blocks,   32 bytes per update,   3 updates): 6221094 opers/sec,  597225024 bytes/sec
test  2 (   96 byte blocks,   96 bytes per update,   1 updates): 9609750 opers/sec,  922536057 bytes/sec
test  3 (  288 byte blocks,   16 bytes per update,  18 updates): 1459379 opers/sec,  420301267 bytes/sec
test  4 (  288 byte blocks,   32 bytes per update,   9 updates): 2115179 opers/sec,  609171609 bytes/sec
test  5 (  288 byte blocks,  288 bytes per update,   1 updates): 3729874 opers/sec, 1074203856 bytes/sec
test  6 ( 1056 byte blocks,   32 bytes per update,  33 updates):  593000 opers/sec,  626208000 bytes/sec
test  7 ( 1056 byte blocks, 1056 bytes per update,   1 updates): 1081536 opers/sec, 1142102332 bytes/sec
test  8 ( 2080 byte blocks,   32 bytes per update,  65 updates):  302077 opers/sec,  628320576 bytes/sec
test  9 ( 2080 byte blocks, 2080 bytes per update,   1 updates):  554384 opers/sec, 1153120176 bytes/sec
test 10 ( 4128 byte blocks, 4128 bytes per update,   1 updates):  278715 opers/sec, 1150536345 bytes/sec
test 11 ( 8224 byte blocks, 8224 bytes per update,   1 updates):  140202 opers/sec, 1153022070 bytes/sec

testing speed of poly1305 (poly1305-simd)
test  0 (   96 byte blocks,   16 bytes per update,   6 updates): 3790063 opers/sec,  363846076 bytes/sec
test  1 (   96 byte blocks,   32 bytes per update,   3 updates): 5913378 opers/sec,  567684355 bytes/sec
test  2 (   96 byte blocks,   96 bytes per update,   1 updates): 9352574 opers/sec,  897847104 bytes/sec
test  3 (  288 byte blocks,   16 bytes per update,  18 updates): 1362145 opers/sec,  392297990 bytes/sec
test  4 (  288 byte blocks,   32 bytes per update,   9 updates): 2007075 opers/sec,  578037628 bytes/sec
test  5 (  288 byte blocks,  288 bytes per update,   1 updates): 3709811 opers/sec, 1068425798 bytes/sec
test  6 ( 1056 byte blocks,   32 bytes per update,  33 updates):  566272 opers/sec,  597984182 bytes/sec
test  7 ( 1056 byte blocks, 1056 bytes per update,   1 updates): 1111657 opers/sec, 1173910108 bytes/sec
test  8 ( 2080 byte blocks,   32 bytes per update,  65 updates):  288857 opers/sec,  600823808 bytes/sec
test  9 ( 2080 byte blocks, 2080 bytes per update,   1 updates):  590746 opers/sec, 1228751888 bytes/sec
test 10 ( 4128 byte blocks, 4128 bytes per update,   1 updates):  301825 opers/sec, 1245936902 bytes/sec
test 11 ( 8224 byte blocks, 8224 bytes per update,   1 updates):  153075 opers/sec, 1258896201 bytes/sec

Benchmark results from a Core i5-4670T.

Signed-off-by: Martin Willi <martin@strongswan.org>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
2015-07-17 21:20:27 +08:00
..
sha-mb Merge git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6 2015-06-22 21:04:48 -07:00
Makefile crypto: poly1305 - Add a SSE2 SIMD variant for x86_64 2015-07-17 21:20:27 +08:00
aes-i586-asm_32.S crypto: x86/aes - assembler clean-ups: use ENTRY/ENDPROC, localize jump targets 2013-01-20 10:16:47 +11:00
aes-x86_64-asm_64.S crypto: x86/aes - assembler clean-ups: use ENTRY/ENDPROC, localize jump targets 2013-01-20 10:16:47 +11:00
aes_ctrby8_avx-x86_64.S crypto: aesni - fix "by8" variant for 128 bit keys 2015-01-05 21:35:02 +11:00
aes_glue.c crypto: prefix module autoloading with "crypto-" 2014-11-24 22:43:57 +08:00
aesni-intel_asm.S crypto: aesni - Add support for 192 & 256 bit keys to AESNI RFC4106 2015-01-14 21:56:51 +11:00
aesni-intel_avx-x86_64.S crypto: aesni - fix build on x86 (32bit) 2014-01-15 11:36:34 +08:00
aesni-intel_glue.c crypto: aesni - Use new IV convention 2015-07-14 14:56:47 +08:00
blowfish-x86_64-asm_64.S crypto: blowfish-x86_64: use ENTRY()/ENDPROC() for assembler functions and localize jump targets 2013-01-20 10:16:48 +11:00
blowfish_glue.c crypto: prefix module autoloading with "crypto-" 2014-11-24 22:43:57 +08:00
camellia-aesni-avx-asm_64.S crypto: x86/camellia-aesni-avx - add more optimized XTS code 2013-04-25 21:01:52 +08:00
camellia-aesni-avx2-asm_64.S crypto: camellia-aesni-avx2 - tune assembly code for more performance 2013-06-21 14:44:23 +08:00
camellia-x86_64-asm_64.S crypto: camellia-x86_64/aes-ni: use ENTRY()/ENDPROC() for assembler functions and localize jump targets 2013-01-20 10:16:48 +11:00
camellia_aesni_avx2_glue.c x86/fpu, crypto: Fix AVX2 feature tests 2015-05-22 10:58:45 +02:00
camellia_aesni_avx_glue.c x86/fpu, crypto x86/camellia_aesni_avx: Simplify the camellia_aesni_init() xfeature checks 2015-05-19 15:47:56 +02:00
camellia_glue.c crypto: prefix module autoloading with "crypto-" 2014-11-24 22:43:57 +08:00
cast5-avx-x86_64-asm_64.S crypto: cast5-avx: use ENTRY()/ENDPROC() for assembler functions and localize jump targets 2013-01-20 10:16:48 +11:00
cast5_avx_glue.c x86/fpu, crypto x86/cast5_avx: Simplify the cast5_init() xfeature checks 2015-05-19 15:47:58 +02:00
cast6-avx-x86_64-asm_64.S crypto: cast6-avx: use new optimized XTS code 2013-04-25 21:01:52 +08:00
cast6_avx_glue.c x86/fpu, crypto x86/cast6_avx: Simplify the cast6_init() xfeature checks 2015-05-19 15:47:58 +02:00
chacha20-avx2-x86_64.S crypto: chacha20 - Add an eight block AVX2 variant for x86_64 2015-07-17 21:20:25 +08:00
chacha20-ssse3-x86_64.S crypto: chacha20 - Add a four block SSSE3 variant for x86_64 2015-07-17 21:20:25 +08:00
chacha20_glue.c crypto: chacha20 - Add an eight block AVX2 variant for x86_64 2015-07-17 21:20:25 +08:00
crc32-pclmul_asm.S x86, crc32-pclmul: Fix build with older binutils 2013-05-30 16:36:23 -07:00
crc32-pclmul_glue.c x86/fpu: Rename i387.h to fpu/api.h 2015-05-19 15:47:30 +02:00
crc32c-intel_glue.c x86/fpu: Rename fpu-internal.h to fpu/internal.h 2015-05-19 15:47:31 +02:00
crc32c-pcl-intel-asm_64.S x86/asm: Replace "MOVQ $imm, %reg" with MOVL 2015-04-01 13:17:39 +02:00
crct10dif-pcl-asm_64.S Reinstate "crypto: crct10dif - Wrap crc_t10dif function all to use crypto transform framework" 2013-09-07 12:56:26 +10:00
crct10dif-pclmul_glue.c x86/fpu: Rename i387.h to fpu/api.h 2015-05-19 15:47:30 +02:00
des3_ede-asm_64.S crypto: des_3des - add x86-64 assembly implementation 2014-06-20 21:27:58 +08:00
des3_ede_glue.c crypto: x86/des3_ede - drop bogus module aliases 2015-01-13 22:30:52 +11:00
fpu.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6 2015-06-22 21:04:48 -07:00
ghash-clmulni-intel_asm.S crypto: ghash-clmulni-intel - Use u128 instead of be128 for internal key 2014-04-04 21:06:14 +08:00
ghash-clmulni-intel_glue.c x86/fpu: Rename i387.h to fpu/api.h 2015-05-19 15:47:30 +02:00
glue_helper-asm-avx.S crypto: x86 - add more optimized XTS-mode for serpent-avx 2013-04-25 21:01:51 +08:00
glue_helper-asm-avx2.S crypto: twofish - add AVX2/x86_64 assembler implementation of twofish cipher 2013-04-25 21:09:05 +08:00
glue_helper.c crypto: don't export static symbol 2015-03-13 21:37:15 +11:00
poly1305-sse2-x86_64.S crypto: poly1305 - Add a SSE2 SIMD variant for x86_64 2015-07-17 21:20:27 +08:00
poly1305_glue.c crypto: poly1305 - Add a SSE2 SIMD variant for x86_64 2015-07-17 21:20:27 +08:00
salsa20-i586-asm_32.S crypto: x86/salsa20 - assembler cleanup, use ENTRY/ENDPROC for assember functions and rename ECRYPT_* to salsa20_* 2013-01-20 10:16:50 +11:00
salsa20-x86_64-asm_64.S crypto: x86/salsa20 - assembler cleanup, use ENTRY/ENDPROC for assember functions and rename ECRYPT_* to salsa20_* 2013-01-20 10:16:50 +11:00
salsa20_glue.c crypto: prefix module autoloading with "crypto-" 2014-11-24 22:43:57 +08:00
serpent-avx-x86_64-asm_64.S crypto: x86 - add more optimized XTS-mode for serpent-avx 2013-04-25 21:01:51 +08:00
serpent-avx2-asm_64.S crypto: serpent - add AVX2/x86_64 assembler implementation of serpent cipher 2013-04-25 21:09:07 +08:00
serpent-sse2-i586-asm_32.S crypto: x86/serpent - use ENTRY/ENDPROC for assember functions and localize jump targets 2013-01-20 10:16:50 +11:00
serpent-sse2-x86_64-asm_64.S crypto: x86/serpent - use ENTRY/ENDPROC for assember functions and localize jump targets 2013-01-20 10:16:50 +11:00
serpent_avx2_glue.c x86/fpu, crypto: Fix AVX2 feature tests 2015-05-22 10:58:45 +02:00
serpent_avx_glue.c x86/fpu, crypto x86/serpent_avx: Simplify the serpent_init() xfeature checks 2015-05-19 15:47:57 +02:00
serpent_sse2_glue.c crypto: serpent_sse2 - mark Serpent SSE2 helper ciphers 2015-03-31 21:21:10 +08:00
sha1_avx2_x86_64_asm.S crypto: x86/sha1 - reduce size of the AVX2 asm implementation 2014-03-25 20:25:43 +08:00
sha1_ssse3_asm.S crypto: x86/sha1 - assembler clean-ups: use ENTRY/ENDPROC 2013-01-20 10:16:51 +11:00
sha1_ssse3_glue.c x86/fpu, crypto x86/sha1_ssse3: Simplify the sha1_ssse3_mod_init() xfeature checks 2015-05-19 15:47:59 +02:00
sha256-avx-asm.S crypto: x86/sha256_ssse3 - move SHA-224/256 SSSE3 implementation to base layer 2015-04-10 21:39:47 +08:00
sha256-avx2-asm.S crypto: x86/sha256_ssse3 - move SHA-224/256 SSSE3 implementation to base layer 2015-04-10 21:39:47 +08:00
sha256-ssse3-asm.S crypto: x86/sha256_ssse3 - move SHA-224/256 SSSE3 implementation to base layer 2015-04-10 21:39:47 +08:00
sha256_ssse3_glue.c x86/fpu, crypto x86/sha256_ssse3: Simplify the sha256_ssse3_mod_init() xfeature checks 2015-05-19 15:47:57 +02:00
sha512-avx-asm.S crypto: x86/sha512_ssse3 - move SHA-384/512 SSSE3 implementation to base layer 2015-04-10 21:39:48 +08:00
sha512-avx2-asm.S crypto: x86/sha512_ssse3 - fixup for asm function prototype change 2015-04-24 20:09:01 +08:00
sha512-ssse3-asm.S crypto: x86/sha512_ssse3 - move SHA-384/512 SSSE3 implementation to base layer 2015-04-10 21:39:48 +08:00
sha512_ssse3_glue.c x86/fpu, crypto x86/sha512_ssse3: Simplify the sha512_ssse3_mod_init() xfeature checks 2015-05-19 15:47:58 +02:00
twofish-avx-x86_64-asm_64.S crypto: x86/twofish-avx - use optimized XTS code 2013-04-25 21:01:51 +08:00
twofish-i586-asm_32.S crypto: x86/twofish - assembler clean-ups: use ENTRY/ENDPROC, localize jump labels 2013-01-20 10:16:51 +11:00
twofish-x86_64-asm_64-3way.S crypto: x86/twofish - assembler clean-ups: use ENTRY/ENDPROC, localize jump labels 2013-01-20 10:16:51 +11:00
twofish-x86_64-asm_64.S x86/asm: Replace "MOVQ $imm, %reg" with MOVL 2015-04-01 13:17:39 +02:00
twofish_avx_glue.c x86/fpu, crypto x86/twofish_avx: Simplify the twofish_init() xfeature checks 2015-05-19 15:47:57 +02:00
twofish_glue.c crypto: prefix module autoloading with "crypto-" 2014-11-24 22:43:57 +08:00
twofish_glue_3way.c crypto: prefix module autoloading with "crypto-" 2014-11-24 22:43:57 +08:00