OSchip
/
OpenBLAS
mirror of https://github.com/xianyi/OpenBLAS.git
0
0
Fork 2
Code Issues Packages Projects Releases Wiki Activity
9,676 Commits 58 Branches 70 Tags 121 MiB
Commit Graph

2619 Commits

Author SHA1 Message Date
Martin Kroeker 995a990e24
Make AVX512 BFLOAT16 kernels conditional on compiler capability 2024-01-12 00:12:46 +01:00
Martin Kroeker 7df363e1e2
temporarily disable the MSA C/ZSCAL kernels 2024-01-12 00:08:52 +01:00
Chip-Kerchner 058dd2a4cb Replace two vector loads with one vector pair load and fix endianess of stores - DGEMM versions. 2024-01-08 14:16:09 -06:00
Martin Kroeker 1c31f56e5a
Handle NAN 2024-01-08 16:11:25 +01:00
Martin Kroeker 7ee1ee38e2
Handle NaN in input 2024-01-08 14:20:07 +01:00
Martin Kroeker f637e12713
Handle INF and NAN 2024-01-08 09:52:38 +01:00
Martin Kroeker 25b0c48082
Update zscal.c 2024-01-08 09:49:18 +01:00
Martin Kroeker 5e7f714e93
Update zscal.c 2024-01-08 08:17:40 +01:00
Martin Kroeker cf8b03ae8b
Use NAN rather than SNAN for portability 2024-01-07 23:09:57 +01:00
Martin Kroeker f0808d856b
Handle NAN in input 2024-01-07 20:27:29 +01:00
Martin Kroeker acf17a825d
Handle NAN in input 2024-01-07 20:26:16 +01:00
Martin Kroeker c9df62e883
Fix handling of NAN 2024-01-07 17:49:40 +01:00
Martin Kroeker def4996170
Fix handling of NAN and INF arguments 2024-01-07 15:29:42 +01:00
Martin Kroeker 519b40fad9
Merge pull request #4398 from yinshiyou/la-dev 
Add Optimizations for LoongArch.
 2023-12-30 19:51:08 +01:00
pengxu a5d0d21378 loongarch64: Add zgemm and cgemm optimization 2023-12-29 18:06:26 +08:00
gxw 546f13558c loongarch64: Add {c/z}swap and {c/z}sum optimization 2023-12-29 17:30:57 +08:00
Hao Chen edabb93668 loongarch64: Refine axpby optimization functions. 2023-12-29 17:30:57 +08:00
Hao Chen 1ec5dded43 loongarch64: Add c/zrot optimization functions. 
Signed-off-by: Hao Chen <chenhao@loongson.cn>
 2023-12-29 17:30:57 +08:00
Hao Chen 3c53ded315 loongarch64: Add c/znrm2 optimization functions. 2023-12-29 17:30:57 +08:00
Hao Chen fbd612f8c4 loongarch64: Add ic/zamin optimization functions. 2023-12-29 17:30:57 +08:00
Hao Chen d97272cb35 loongarch64: Add c/zdot optimization functions. 2023-12-29 17:30:57 +08:00
Hao Chen 65a0aeb128 loongarch64: Add c/zcopy optimization functions. 
Signed-off-by: Hao Chen <chenhao@loongson.cn>
 2023-12-29 17:30:57 +08:00
Hao Chen 2a34fb4b80 loongarch64: Add and refine scal optimization functions. 
Signed-off-by: Hao Chen <chenhao@loongson.cn>
 2023-12-29 17:30:57 +08:00
Hao Chen 8785e948b5 loongarch64: Add camin optimization function. 2023-12-29 17:30:57 +08:00
Hao Chen 0753848e03 loongarch64: Refine and add axpy optimization functions. 
Signed-off-by: Hao Chen <chenhao@loongson.cn>
 2023-12-29 17:30:57 +08:00
Hao Chen 06fd5b5995 loongarch64: Add and Refine asum optimization functions. 2023-12-29 17:30:57 +08:00
guxiwei e771be185e Optimize copy functions with lsx. 
Signed-off-by: Hao Chen <chenhao@loongson.cn>
 2023-12-29 17:30:57 +08:00
Hao Chen 179ed51d3b Add dgemm_kernel_8x4.S file. 2023-12-29 17:30:57 +08:00
Hao Chen 173a65d4e6 loongarch64: Add and refine iamax optimization functions. 2023-12-29 17:30:57 +08:00
zhoupeng ea70e165c7 loongarch64: Refine rot optimization. 2023-12-29 17:30:57 +08:00
zhoupeng 116aee7527 loongarch64: Refine imin optimization. 2023-12-29 17:30:57 +08:00
zhoupeng 8be2654193 loongarch64: Refine imax optimization. 2023-12-29 17:30:57 +08:00
zhoupeng 154baad454 loongarch64: Refine iamin optimization. 2023-12-29 17:30:57 +08:00
Shiyou Yin 36c12c4971 loongarch64: Refine copy,swap,nrm2,sum optimization. 2023-12-29 17:30:57 +08:00
Shiyou Yin c6996a80e9 loongarch64: Refine amax,amin,max,min optimization. 2023-12-29 17:30:57 +08:00
Chris Sidebottom ecae1389df Reduce duplication in kernel definitions 
These files are exactly the same, so I believe we can reduce these files
down. Other files require a slightly more complex unpicking.
 2023-12-23 12:39:53 +00:00
Chris Sidebottom 60e66725e4 Use numeric labels to allow repeated inlining 2023-12-19 13:11:06 +00:00
Chris Sidebottom 7a4fef4f60 Tweak SVE dot kernel 
This changes the SVE dot kernel to only predicate when necessary as well
as streamlining the assembly a bit. The benchmarks seem to indicate this
can improve performance by ~33%.
 2023-12-19 12:08:54 +00:00
Martin Kroeker f06b535566
Use C kernel for dgemv_t due to limitations of the old assembly one 2023-12-15 09:58:44 +01:00
barracuda156 d9653af018 KERNEL.PPC970, KERNEL.PPCG4: unbreak CMake parsing 
Fixes: https://github.com/OpenMathLib/OpenBLAS/issues/4366
 2023-12-14 12:00:11 +08:00
Chip-Kerchner 93747fb377 Merge remote-tracking branch 'origin/develop' into power10Copies 2023-12-12 09:32:49 -06:00
Chip-Kerchner 4e738e561a Replace two vector loads with one vector pair load and fix endianess of stores. 2023-12-08 12:36:08 -06:00
yancheng d32f38fb37 loongarch64: Add optimizations for nrm2. 2023-12-07 14:36:26 +08:00
yancheng f9b468990e loongarch64: Add optimizations for rot. 2023-12-07 14:36:26 +08:00
yancheng c80e7e27d1 loongarch64: Add optimizations for sum and asum. 2023-12-07 14:36:26 +08:00
yancheng d4c96a35a8 loongarch64: Add optimizations for axpy and axpby. 2023-12-07 14:36:26 +08:00
yancheng 360acc0a41 loongarch64: Add optimizations for swap. 2023-12-07 14:36:26 +08:00
yancheng 174c25766b loongarch64: Add optimizations for copy. 2023-12-07 14:36:26 +08:00
yancheng 49829b2b7d loongarch64: Add optimizations for iamin. 2023-12-07 14:36:07 +08:00
yancheng be83f5e4e0 loongarch64: Add optimizations for iamax. 2023-12-07 14:36:07 +08:00
yancheng e3fb2b5afa loongarch64: Add optimizations for imin. 2023-12-07 14:36:07 +08:00
yancheng e46b48e372 loongarch64: Add optimizations for imax. 2023-12-07 14:36:07 +08:00
yancheng 702fc1d56d loongarch64: Add optimization for min. 2023-12-07 14:36:07 +08:00
yancheng 346b384d1c loongarch64: Add optimization for max. 2023-12-07 14:36:07 +08:00
yancheng ff2ecc6cda loongarch64: Add optimization for amin. 2023-12-07 14:36:07 +08:00
yancheng 265b5f2e80 loongarch64: Add optimizations for amax. 2023-12-07 14:36:07 +08:00
yancheng 993ede7c70 loongarch64: Add optimizations for scal. 2023-12-07 14:36:07 +08:00
Octavian Maghiar 4a12cf53ec [RISC-V] Improve RVV kernel generator LMUL usage 
The RVV kernel generation script uses the provided LMUL to increase the number of accumulator registers.
Since the effect of the LMUL is to group together the vector registers into larger ones, it actually should be used as a multiplier in the calculation of vlenmax.
At the moment, no matter what LMUL is provided, the generated kernels would only set the maximum number of vector elements equal to VLEN/SEW.
Commit changes the use of LMUL to properly adjust vlenmax. Note that an increase in LMUL results in a decrease in the number of effective vector registers.
 2023-12-04 11:13:35 +00:00
Octavian Maghiar e4586e81b8 [RISC-V] Add RISC-V Vector 128-bit target 
Current RVV x280 target depends on vlen=512-bits for Level 3 operations.
Commit adds generic target that supports vlen=128-bits.
New target uses the same scalable kernels as x280 for Level 1&2 operations, and autogenerated kernels for Level 3 operations.
Functional correctness of Level 3 operations tested on vlen=128-bits using QEMU v8.1.1 for ctests and BLAS-Tester.
 2023-12-04 11:02:18 +00:00
Martin Kroeker 39bf8ece20
Merge pull request #4340 from yinshiyou/la-dev 
Add some refines and optimizations for LoongArch.
 2023-11-29 08:22:25 +01:00
Shiyou Yin 9fe07d82fd loongarch: Add LSX optimization for dot. 2023-11-28 20:24:18 +08:00
Shiyou Yin 13b8c44b44 loongarch: Add optimization for dsdot kernel. 2023-11-28 20:24:16 +08:00
Shiyou Yin 3def6a8143 loongarch: Add LASX optimization for dot. 2023-11-28 20:24:14 +08:00
Bart Oldeman c34e2cf380 Use _mm_set1_epi{32,64x} to init mask in x86-64 [cz]asum 
for skylake kernels. This is the same method as used in [sd]asum.
_mm_set1_epi64x was commented out for zasum, but has the advantage
of avoiding possible undefined behaviour (using an uninitialized
variable), optimized out by NVHPC and icx. The new code works
fine with those compilers.

For GCC 12.3 the generated code is identical; no matter what method
you use, the compiler optimizes the code into a compile-time
constant, there is no performance benefit using mm_cmpeq_epi8
since the corresponding instruction (VPCMPEQB) isn't actually
generated!
 2023-11-19 21:28:35 +00:00
Martin Kroeker 22aa401656
Temporarily disable the AVX512 CASUM/ZASUM microkernels for any version of NVIDIA HPC (#4327) 
* Temporarily disable the C/ZASUM microkernels for any version of NVHPC
 2023-11-19 00:04:31 +01:00
Bart Oldeman f8ad5344c2 Fix casum fallback kernel. 
This kernel is only used on Skylake+ if the kernel with AVX512
intrinsics can't be used, but used the variable x1 incorrectly
in the tail end of the loop, as it is still at the initial
value instead of where x points to.

This caused 55 "other error"s in the LAPACK tests
(https://github.com/OpenMathLib/OpenBLAS/issues/4282)

This change makes casum.c as similar as possible as zasum.c,
because zasum.c does this correctly.
 2023-11-17 23:53:56 +00:00
Martin Kroeker 04bc801999
(Re)apply fixes for supporting only a subset of precision types from PR 3915 2023-11-04 23:48:59 +01:00
Martin Kroeker 9019bc4945
Use SkylakeX ?ASUM microkernel for Cooperlake/Sapphirerapids as well 2023-11-04 22:10:06 +01:00
Martin Kroeker 3bfa4d4dcc
Fix outdated SVE kernel definitions for Cortex cpus by aliasing to ARMV8SVE 2023-11-03 14:55:31 +01:00
Rajalakshmi Srinivasaraghavan 980f702f72 POWER: AIX: Make use of power10 optimization 
POWER10 optimizations are disabled when using default AIX assembler.
As we have fixed many issues recently, enabling optimization path
for default assembler.
 2023-10-19 18:48:19 -05:00
Rajalakshmi Srinivasaraghavan 9f42570e33 POWER: Increase macro size limit for AIX 
This patch increases the macro size limit from 4096 to 16384 to
allow compiling larger assembly files in AIX.
Tested with GCC and IBM Open XL C.
 2023-10-12 12:37:40 -05:00
Martin Kroeker 9f49aef91b
Merge pull request #4255 from RajalakshmiSR/AIX-P10 
POWER10: Fix compilation issues with Open XL C
 2023-10-12 18:59:17 +02:00
Martin Kroeker e7d05402e0
Fix up S/D GEMM copy function definitions after #4009 2023-10-12 14:24:53 +02:00
Rajalakshmi Srinivasaraghavan 71d733e5f7 POWER: Avoid m4 conversions for C files 
This patch removes intermediate m4 conversions used in sbgemm
compilation as it is not needed for .c files.
Tested on AIX with gcc and IBM Open XL C.
 2023-10-11 17:18:42 -05:00
Rajalakshmi Srinivasaraghavan 82fc29a57a POWER10: Fallback to POWER8 functions 
As cgemm and zgemm kernels are not optimized for big endian falling
back to POWER8 versions.  Tested on AIX using gcc and Open XL C.
 2023-10-11 17:04:42 -05:00
Rajalakshmi Srinivasaraghavan db0805906b powerpc: Fix build errors with Open XL C 
This patch fixes errors when using Open XL C compiler on AIX.
Tested with gcc/xlf and ibm-clang/xlf compiler combinations.
 2023-10-04 14:04:03 -05:00
Martin Kroeker 675cd551da
fix improper function prototypes (empty parentheses) 2023-09-30 12:56:38 +02:00
gxw d15e0a055c LoongArch64: Fixed compilation issues when enable DYNAMIC_ARCH 2023-09-27 10:05:27 +08:00
gxw 4670eb1462 LoongArch64: Add dtrsm kernel 2023-09-26 15:45:14 +08:00
gxw f2cf929374 LoongArch64: Add sgemv kernel 2023-09-04 14:28:37 +08:00
Martin Kroeker 8e6d93359d
Merge pull request #4196 from TiborGY/obsolete_inlines 
Modernize obsolete inline order
 2023-09-03 14:12:42 +02:00
gxw 394a1fd1bf LoongArch64: Compatible with early internal toolchain 
__loongarch_grlen and __loongarch_frlen were introduced in gcc version 8.3.0
(Loongnix 8.3.0-6.lnd.vec.31) internally within Loongson to standardize the
general and floating-point register widths. However, previous versions did
not have them, requiring additional checks to be added.
 2023-08-31 16:55:29 +08:00
Martin Kroeker 9c4ae4d4fb
Merge pull request #4206 from martin-frbg/issue4201-2 
Work around miscompilation of zdot_thunderx2t99 by the current NVIDIA HPC compiler
 2023-08-26 10:17:27 +02:00
Martin Kroeker 88435104c8
Merge pull request #4204 from martin-frbg/llvm17-2 
Work around LLVM17 miscompiling the AVX512 microkernels for CASUM/ZASUM
 2023-08-26 00:32:18 +02:00
Martin Kroeker fc8894dd98
Workaround miscompilation by NVIDIA nvc 2023-08-26 00:30:17 +02:00
Martin Kroeker 7a6203ffa1
restore default Neoverse SVE build instructions for non-NVIDIA compilers 2023-08-25 18:25:51 +02:00
Martin Kroeker 2c3034ff7f
Disable the C/ZASUM AVX512 microkernels when compiling with LLVM17 as well 2023-08-25 17:22:51 +02:00
Martin Kroeker 8794544b43
Add support for compiling the Neoverse SVE kernels with the NVIDIA HPC compiler 2023-08-25 16:47:32 +02:00
gxw 553cc1372f LoongArch64: Add sgemm_kernel 2023-08-23 16:08:43 +08:00
Martin Kroeker 12ede72ab7
Merge pull request #4192 from imciner2/im/clangfix 
Fix cooperlake and sapphire rapids march flags on clang
 2023-08-21 15:46:35 +02:00
Ian McInerney 79c15db348 Fix power10 gcc intrinsic check 
__builtin_vsx_assemble_pair was only in GCC 10-11.2 and was replaced by
__builtin_vsx_build_pair thereafter.
 2023-08-17 15:05:29 +01:00
TGY b5ba95a6c0 Modernize obsolete inline order 2023-08-16 00:48:40 +02:00
Ian McInerney 8a8a8479be Fix cooperlake and sapphire rapids march flags on clang 
The march=cooperlake and march=sapphirerapids flags were never getting
added when building with Clang targetting those architectures. Instead
it was falling back to the skylake AVX512 implementation.

Clang added support for these two architectures in Clang 9 and Clang 12,
so introduce new checks for those versions to enable the appropriate
march flag, and fallback to skylake otherwise.
 2023-08-14 16:12:35 +01:00
Martin Kroeker 34da1a067d
Allow negative INCX (API change from version 3.10 of the reference implementation) 2023-08-10 17:01:50 +02:00
Martin Kroeker 07e32c4cb8
Allow negative INCX (API change from version 3.10 of the reference implementation) 2023-08-10 17:00:18 +02:00
Martin Kroeker c211da0688
Allow negative INCX (API change from version 3.10 of the reference implementation) 2023-08-10 16:58:57 +02:00
Martin Kroeker a34a0a7abc
Allow negative INCX (API change from version 3.10 of the reference implementation) 2023-08-10 16:56:52 +02:00
Martin Kroeker 54d3246fc6
Allow negative INCX (API change from version 3.10 of the reference implementation) 2023-08-10 16:55:17 +02:00
Martin Kroeker 7dd441d5db
Allow negative INCX (API change from version 3.10 of the reference implementation) 2023-08-10 16:53:33 +02:00
Martin Kroeker f692178792
Allow negative INCX (API change from version 3.10 of the reference implementation) 2023-08-10 16:52:09 +02:00
Martin Kroeker d15ffb7fdf
Allow negative INCX (API change from version 3.10 of the reference implementation) 2023-08-10 16:50:44 +02:00
Martin Kroeker a2d867f4d1
Allow negative iNCX (API change from version 3.10 of the reference implementation) 2023-08-10 16:49:05 +02:00
gxw 4d0f000db6 MIPS: Enable MSA 2023-08-07 21:00:10 +08:00
Martin Kroeker afdc56a421
Merge pull request #4158 from XiWeiGu/loongarch64_update_dgemm_kernel 
LoongArch64: Update dgemm kernel
 2023-08-07 12:44:09 +02:00
gxw e8b571d245 LoongArch64: Add dgemv_t_8_lasx.S and dgemv_n_8_lasx.S V2 2023-08-07 11:20:42 +08:00
gxw 71fcee6eef LoongArch64: Update dgemm kernel 2023-08-07 11:06:52 +08:00
Martin Kroeker 0f521ece25
Merge pull request #4183 from martin-frbg/issue4181 
Apply USE_TRMM to MIPS64_GENERIC as to GENERIC in gmake builds
 2023-08-06 18:59:50 +02:00
Martin Kroeker 41c31bc1d4
Revert "LoongArch64: Add dgemv_t_8_lasx.S and dgemv_n_8_lasx.S" 2023-08-06 16:00:03 +02:00
Martin Kroeker 61d803547a
Apply USE_TRMM to MIPS64_GENERIC as to GENERIC 2023-08-06 15:17:38 +02:00
Martin Kroeker f8ee309402
Merge pull request #4153 from XiWeiGu/dgemv 
LoongArch64: Add dgemv_t_8_lasx.S and dgemv_n_8_lasx.S
 2023-08-06 08:49:16 +02:00
gxw ec1e96aac8 LoongArch64: Add dgemv_t_8_lasx.S and dgemv_n_8_lasx.S 2023-08-05 10:24:17 +08:00
gxw d46772e037 LoongArch64: Add compiler feature checks 2023-08-05 10:21:43 +08:00
Martin Kroeker 4664b57e6e
use shortcut only when both incx and incy are zero 2023-08-04 12:25:34 +02:00
Martin Kroeker 09131f79a6
Merge pull request #4164 from martin-frbg/issue4162 
Enable use of AVX512 microkernels with NVIDIA HPC from version 22.3
 2023-07-29 15:07:20 +02:00
Martin Kroeker 6a428b5629
Update casum_microk_skylakex-2.c 2023-07-29 12:24:30 +02:00
Martin Kroeker ebb447e32e
Update zasum_microk_skylakex-2.c 2023-07-29 12:23:57 +02:00
Martin Kroeker 9f6847583a
nvc currently miscompiles this, hopefully fixed in release 23.09 2023-07-29 11:50:16 +02:00
Martin Kroeker fe54ee3d15
nvc currently miscompiles this, hopefully fixed in release 23.09 2023-07-29 11:48:38 +02:00
Martin Kroeker 5720fa02c5
Merge pull request #4168 from Mousius/sve-zgemm-cgemm 
Use SVE zgemm/cgemm on Arm(R) Neoverse(TM) V1 core
 2023-07-27 17:41:45 +02:00
Chris Sidebottom 84a268b6ca Use SVE zgemm/cgemm on Arm(R) Neoverse(TM) V1 core 
This patch removes the prefetches from cgemm/zgemm which improves the performance similar to sgemm/dgemm did in #3868, this means I'm happy to enable this on any applicable cores.

I also replicated the unrolling the copies from sgemm and dgemm.
 2023-07-27 14:12:20 +01:00
Chris Sidebottom 730ca04b48 Fix ZHEMM copy for SVE 
Whilst disambiguating whilelt, I inadvertantly used the wrong datatype
for offsets, which can be negative. This rectifies that.
 2023-07-27 13:27:28 +01:00
Martin Kroeker 2a62d2df96
Enable use of AVX512 microkernels with NVIDIA HPC from version 22.3 2023-07-26 19:39:11 +02:00
Martin Kroeker 849c8806b8
Merge pull request #4161 from Mousius/non-sve-kernels 
Use latest non-SVE kernels in ARMV8SVE
 2023-07-26 15:49:40 +02:00
Chris Sidebottom 24586bc4ff Disambiguate whilelt 2023-07-25 20:15:44 +01:00
Chris Sidebottom aea2a4622b Use latest non-SVE kernels in ARMV8SVE 
These are generally better and, in some cases, include threading which helps in the cores we're targeting here.
 2023-07-25 14:12:26 +01:00
Octavian Maghiar 826a9d5fa4 Adds tail undisturbed for RVV Level 2 operations 
During the last iteration of some RVV operations, accumulators can get overwritten when VL < VLMAX and tail policy is agnostic.
Commit changes intrinsics tail policy to undistrubed.
 2023-07-25 11:36:23 +01:00
martin-frbg 7976deff80 Fix file permissions (issue 4095) 2023-07-23 20:37:07 +02:00
Octavian Maghiar 8df0289db6 Adds tail undisturbed for RVV Level 1 operations 
During the last iteration of some RVV operations, accumulators can get overwritten when VL < VLMAX and tail policy is agnostic.
Commit changes intrinsics tail policy to undistrubed.
 2023-07-20 15:28:35 +01:00
Martin Kroeker 76ef1672f8
Override DSDOT with generic code to get rid of qemu precision error 2023-07-19 22:31:07 +02:00
Martin Kroeker 49077e7bde
Merge pull request #4145 from martin-frbg/issue4144 
Restore zero-initialization of variables in generic ztrsm_utcopy
 2023-07-14 12:44:05 +02:00
Martin Kroeker 3d31191b0f
Work around Clang failing to disambiguate SVE intrinsics and add AppleClang crossbuild to MacOS/arm64 DYNAMIC_ARCH in AzureCI (#4140) 
* Add AppleClang crossbuild to MacOS/arm64 DYNAMIC_ARCH

* add casts to disambiguate svwhilelt for clang
 2023-07-14 11:06:48 +02:00
Martin Kroeker cfa0a80664
Restore initialization of data variables 2023-07-13 23:23:12 +02:00
Martin Kroeker 9567305e4c
Restore initialization of data01,data02 2023-07-13 23:21:18 +02:00
Octavian Maghiar 1e4a3a2b5e Fixes RVV masked intrinsics for izamax/izamin kernels 2023-07-12 12:55:50 +01:00
Octavian Maghiar e1958eb705 Fixes RVV masked intrinsics for iamax/iamin/imax/imin kernels 
Changes masked intrinsics from _m to _mu and reintroduces maskedoff argument.
 2023-07-05 11:34:00 +01:00
Xianyi Zhang e14a025bb1 Temporily walk around zaxpy vector kernel bug. 2023-06-28 11:17:38 +00:00
Martin Kroeker 772b0cc715
Fix early bailout 2023-06-27 16:12:27 +02:00
Martin Kroeker d6be5036d7
Fix IDAMAX 2023-06-26 21:19:33 +02:00
Martin Kroeker 1fe96f8da7
Fix failures to handle increments of zero 2023-06-25 22:36:57 +02:00
Martin Kroeker 73b30b1dec
Fix VLEV_FLOAT/VSEV_FLOAT macros to compile with t-head 2.6.1 2023-06-18 17:46:29 +02:00
Martin Kroeker c3a2d407a0
Merge pull request #4048 from imzhuhl/spr_sbgemm_fix 
Sapphire Rapids sbgemm fix
 2023-06-17 20:47:09 +02:00
Manjul Mohan 58b88aa5f0 POWER10: Fix compiler warnings 
This patch removes the warning messages related to unused variables in
sbgemm_kernel_power10.c.

Signed-off-by: Manjul Mohan <manjul@linux.vnet.ibm.com>
 2023-06-12 01:08:59 -04:00
ZhengSh 2a8bc38cdc
Merge branch 'xianyi:risc-v' into risc-v 2023-06-09 20:01:03 +08:00
Heller Zheng 0954746380 remove argument unused during compilation. 
fix wrong vr = VFMVVF_FLOAT(0, vl);
 2023-06-04 20:06:58 -07:00
sh-zheng d3bf5a5401 Combine two reduction operations of zhe/symv into one, with tail undisturbed setted. 2023-05-22 22:39:45 +08:00
Honglin Zhu 9e80a194d6 Fix dynamic_list build and gcc version check error 2023-05-21 19:52:58 +08:00
Honglin Zhu a76afdc047 Compatible with older version of GNU make 2023-05-20 13:58:23 +08:00
sh-zheng 18d7afe69d Add rvv support for zsymv and active rvv support for zhemv 2023-05-20 01:19:44 +08:00
Honglin Zhu 90f041e348 Invoke the syscall to allow the use of amx tiles 2023-05-19 10:48:18 +08:00
Honglin Zhu 0b83088887 spr dynamic arch support 2023-05-19 10:48:18 +08:00
Honglin Zhu f249ccb741 Fix spr sbgemm error 2023-05-19 10:48:18 +08:00
Martin Kroeker e9a8d5b45f
Merge pull request #4015 from martin-frbg/issue4013-2 
[WIP] Disable gcc's tree-vectorizer for x86_64 CGEMV
 2023-04-23 18:51:12 +02:00
Martin Kroeker 72caceb324
Merge pull request #4009 from Mousius/sve-gemm 
Use SVE kernel for SGEMM/DGEMM on Arm(R) Neoverse(TM) V1
 2023-04-22 13:56:45 +02:00
Martin Kroeker 84bcf6639f
Disable gcc's tree-vectorizer pass on all operating systems 2023-04-20 23:24:52 +02:00
Martin Kroeker c9174ae8d7
Disable gcc's tree-vectorizer pass on all operating systems 2023-04-19 23:45:44 +02:00
Martin Kroeker c2fe9cb91f
Disable gcc's tree-vectorizer pass on all operating systems 2023-04-19 23:45:14 +02:00
Martin Kroeker 66b39b835c
Disable gcc's tree-vectorizer pass on all operating systems 2023-04-19 23:44:45 +02:00
Martin Kroeker bb6d6735bf
Disable gcc's tree-vectorizer pass on all operating systems 2023-04-19 23:44:15 +02:00
Martin Kroeker d18efaed20
Disable gcc's tree-vectorizer pass on all operating systems 2023-04-19 23:43:43 +02:00
Martin Kroeker 99f6d31ed5
Disable gcc's tree-vectorizer pass on all operating systems 2023-04-19 23:42:55 +02:00
Martin Kroeker 7de9335c56
Disable gcc's tree-vectorizer pass on all operating systems 2023-04-19 23:42:09 +02:00
Martin Kroeker 437c0bf2b4
Merge pull request #3843 from Mousius/switch-ratio 
Propagate SWITCH_RATIO to DYNAMIC_ARCH builds
 2023-04-19 11:51:54 +02:00
Chris Sidebottom ec334e69dc Use SVE kernel for SGEMM/DGEMM on Arm(R) Neoverse(TM) V1 
This re-spins #3869 with some additional copy unrolling which helps maintain SYRK performance.

After #3868, the SVE kernels represent a pretty good boost.

This re-uses ARMV8SVE as a base and I'm going to incrementally move everything to use ARMV8SVE in additional patches (as well as fix up anything that's not already in ARMV8SVE).
 2023-04-17 17:38:42 +01:00
Chris Sidebottom 32f2fafde7 Propagate SWITCH_RATIO to DYNAMIC_ARCH builds 
Previously dynamic builds were either using the default SWITCH_RATIO
or one from the higher level architecture; this patch ensures the
dynamic builds can use this parameter as well.
 2023-04-17 15:34:12 +01:00
Martin Kroeker 44164e3a3d
revert "move alpha out of register 18" (out of PR scope, no SVE on Apple hw) 2023-04-17 14:23:13 +02:00
Martin Kroeker 8be68fa7f4
move declaration of sca to really keep the compiler from throwing it out (for now) 2023-04-15 12:02:39 +02:00
Martin Kroeker 3727672a74
Improve workaround and keep compilers from optimizing it out 2023-04-13 18:07:52 +02:00
Martin Kroeker 108a21e47a
Move ALPHA out of register 18 (reserved on OSX) 2023-04-13 18:05:14 +02:00
Martin Kroeker 0b1acb0ba3
Move ALPHA_I out of register 18 (reserved on OSX) 2023-04-13 18:03:35 +02:00
Martin Kroeker c7bbad09ad
Move ALPHA_I out of register 18 (reserved on OSX) 2023-04-13 18:00:47 +02:00
Martin Kroeker cda29633a3
move ALPHA_I out of register 18 (reserved on OSX) 2023-04-13 17:59:48 +02:00
Martin Kroeker 09ace3cf23
Merge pull request #3846 from lilh9598/sbgemm_opt 
Improve the performance of sbgemm_tcopy on neoversen2
 2023-03-26 19:04:57 +02:00
Heller Zheng 1374a2d08b This PR adapts latest spec changes 
Add prefix (_riscv) for all riscv intrinsics
Update some intrinsics' parameter, like vfredxxxx, vmerge
 2023-03-19 23:59:03 -07:00
Zhang Xianyi 19f17c8bc6
Merge pull request #3893 from HellerZheng/develop 
add riscv level3 C,Z kernel functions.
 2023-03-15 10:17:13 +08:00
Sergei Lewis cb0a70e0e2 dot.c early bail fix 2023-03-02 09:51:10 +00:00
Sergei Lewis 9b61be4545 factoring riscv64/dot.c fix into separate PR as requested 2023-03-01 17:40:42 +00:00
Sergei Lewis 2406958629 * update intrinsics to match latest spec at https://github.com/riscv-non-isa/rvv-intrinsic-doc (in particular, __riscv_ prefixes for rvv intrinsics) 
* fix multiple numerical stability and corner case issues
* add a script to generate arbitrary gemm kernel shapes
* add a generic zvl256b target to demonstrate large gemm kernel unrolls
 2023-02-24 10:45:03 +00:00
Martin Kroeker 38d6fb4225
Fix dependencies in builds with specified subsets of precision types 2023-02-23 23:12:06 +01:00
Martin Kroeker e412bee313
fix GEMM kernel dependencies in builds that use only a subset of precisions 2023-02-22 00:37:14 +01:00
Martin Kroeker d80adf253e
make SSYMV available to BUILD_DOUBLE-only builds 2023-02-22 00:30:20 +01:00
Martin Kroeker 5481c328e8
fix DYNAMIC_ARCH builds that use only a subset of precisions 2023-02-22 00:28:25 +01:00
Heller Zheng 63cf4d0166 add riscv level3 C,Z kernel functions. 2023-02-01 19:13:44 -08:00
Xianyi Zhang c19dff0a31 Fix T-Head RVV intrinsic API changes. 2023-01-25 19:33:32 +08:00
Martin Kroeker 5a9cd87794
Merge pull request #3868 from Mousius/sve-prefetch 
Remove prefetches from SVE kernels
 2022-12-24 10:52:29 +01:00
Chris Sidebottom 1361229291 Remove prefetches from SVE kernels 
This is a precursor to enabling the SVE kernels for Arm(R) Neoverse(TM)
V1 which has 256-bit SVE. Testing revealed that the SVE kernel was
actually worse in some cases than the existing kernel which seemed odd -
removing these prefetches the underlying architecture seems to do a better job
😸
 2022-12-16 14:43:09 +00:00
Bart Oldeman 60e49b851c Fix typo in clobber list, should be xmm14 instead of ymm14. 2022-12-06 16:30:46 -05:00
Bart Oldeman 4afe1439a1 Fix skylake fallback kernel name for old compilers. 2022-12-06 16:09:54 -05:00
Bart Oldeman 5ceca1a4d8 Add sscal.c + microkernels for Haswell, Zen, Skylake and newer. 
Unlike [dcz]scal, sscal still used the original GotoBLAS SSE code from scal_sse.S.
This code follows dscal as closely as possible, except for the inc_x > 1 code
for which a plain C loop is used much like the one in cscal.c, instead of an
adaptation of the SSE2 asm code of dscal.c (I tried but the performance wasn't
better than the plain C loop).
 2022-12-06 14:05:49 -05:00
lilianhuang 729af6406f bugfix for sbgemm_ncopy_8_neoversen2 2022-12-05 05:10:18 -05:00
Martin Kroeker 042e3c0e7c
Merge pull request #3848 from bartoldeman/dscal-haswell-ymm 
dscal: use ymm registers in Haswell microkernel
 2022-12-05 08:56:08 +01:00
Xianyi Zhang e5313f53d5 Merge branch 'develop' of https://github.com/HellerZheng/OpenBLAS_riscv_x280 into HellerZheng-develop 2022-12-03 12:00:52 +08:00
Bart Oldeman 5c3169ecd8 dscal: use ymm registers in Haswell microkernel 
Using 256-bit registers in dscal makes this microkernel consistent with
cscal and zscal, and generally doubles performance if the vector fits in
L1 cache.
 2022-12-01 07:48:05 -05:00
Chris Sidebottom eea006a688 Wrap SVE header with __has_include check 2022-12-01 12:07:55 +00:00
Chris Sidebottom fd4f52c797 Add SVE implementation for sdot/ddot 
This adds an SVE implementation to sdot/ddot when available, falling back to the previous Advanced SIMD kernel where there's no SVE implementation for the kernel.

All the targets were essentially treating `dot_thunderx2t99.c` as the Advanced SIMD implementation so I've renamed it to better fit with the feature detection.
 2022-12-01 12:07:50 +00:00
lilianhuang fdac8a97c1 Add sbgemm_ncopy_8 and sbgemm_tcopy_4 2022-11-29 04:46:14 -05:00
lilianhuang 135718eafc Improve the performance of sbgemm_tcopy on neoversen2 2022-11-28 04:17:54 -05:00
Chris Sidebottom 4f7b77e08a Remove unnecessary instructions from Advanced SIMD dot 
The existing kernel was issuing extra instructions to organise the arguments into the same registers they would usually be in and similarly to put the result into the appropriate register.

This has an impact on smaller sized dots and seemed like a quick fix
 2022-11-25 16:19:03 +00:00
Heller Zheng 3918d8504e nrm2 simple optimization 2022-11-21 19:06:07 -08:00
HellerZheng 943372bdf5
Merge branch 'develop' into develop 2022-11-18 10:12:46 +08:00
Martin Kroeker f73cfb7e2c
change line endings from CRLF to LF 2022-11-17 09:39:56 +01:00
Martin Kroeker 1688c7da43
change line endings from CRLF to LF 2022-11-16 22:24:01 +01:00
Heller Zheng 5d0d1c5551 Remove redundant files 2022-11-15 18:22:21 -08:00
Heller Zheng bef47917bd Initial version for riscv sifive x280 2022-11-15 00:06:25 -08:00
Bart Oldeman 6c1043eb41 Add [cz]scal microkernels for SKYLAKEX 
These are as similar to dscal_microk_skylakex-2.c as possible
for consistency.

Note that before this change SKYLAKEX+ uses generic C functions for
cscal/zscal via commit 2271c350 from #2610 (which is masked by
commit 086d87a30). However now #3799 disables FMAs (in turn enabled
by `-march=skylake-avx512`) in the plain C code which fixes excessive
LAPACK test failures more nicely.
 2022-11-09 08:57:03 -05:00
Martin Kroeker c9d78dc3b2
Remove excess initializer (leftover from rework of PR 3793) 2022-10-31 16:57:03 +01:00
Martin Kroeker 65338a9493
Merge pull request #3799 from bartoldeman/cscal-zscal-no-fma 
x86_64: prevent GCC and Clang from generating FMAs in cscal/zscal.
 2022-10-30 18:56:10 +01:00
Honglin Zhu 79066b6bf3 Change file name to match the norm and delete useless code. 2022-10-28 17:09:39 +08:00
Bart Oldeman e7e3aa2948 x86_64: prevent GCC and Clang from generating FMAs in cscal/zscal. 
If e.g. -march=haswell is set in CFLAGS, GCC generates FMAs by default, which
is inconsistent with the microkernels, none of which use FMAs. These
inconsistencies cause a few failures in the LAPACK testcases, where
eigenvalue results with/without eigenvectors are compared.

Moreover using FMAs for multiplication of complex numbers can give surprising
results, see 22aa81f for more information.

This uses the same syntax as used in 22aa81f for zarch (s390x).
 2022-10-27 18:16:43 -04:00
Honglin Zhu 4989e039a5 Define SBGEMM_ALIGN_K for DYNAMIC_ARCH build 2022-10-27 14:10:26 +08:00
Honglin Zhu 843e9fd0b9 Fix typo error 2022-10-26 17:06:33 +08:00
Honglin Zhu b00d5b9746 New sbgemm implementation for Neoverse N2 
1. Use UZP instructions but not gather load and scatter store instructions to get lower latency.
    2. Padding k to a power of 4.
 2022-10-26 15:09:41 +08:00
Martin Kroeker f6f35a4288
fix copyobj declarations to work with DYNAMIC_ARCH 2022-09-29 08:47:14 +02:00
Martin Kroeker b1d69fb3ac
Add MIPS64_GENERIC as a copy of GENERIC 2022-09-17 23:52:32 +02:00
gxw edea1bcfaf MIPS64: Fixed failed utest dsdot:dsdot_n_1 when TARGET=I6500 2022-09-17 16:43:22 +08:00
Martin Kroeker 101a2c77c3
Fix warnings 2022-09-15 09:19:19 +02:00
Martin Kroeker 23d59baaf1
Add -mfma to -mavx2 for Apple clang, and set AVX2 options for Zen as well 2022-09-13 22:39:27 +02:00
gxw 365936ae1b MIPS64: Using the macro MTC rather than MTC1 2022-09-13 16:39:40 +08:00
Martin Kroeker 739c3c44a7
Work around windows/osx gcc12 x86_64 tree-optimizer problem and add an osx/gcc12 build to Azure CI (#3745) 
Add pragma to disable the gcc tree-optimizer for some x86_64 S and Z kernels with gcc12 on OSX or Windows
 2022-09-03 15:01:22 +02:00
Martin Kroeker bd30120ba7
Merge pull request #3720 from FlyGoat/mips64 
Make it work on general MIPS64 processors
 2022-08-19 20:24:27 +02:00
Jiaxun Yang a50b29c540 Provide a fallback MIPS64_GENERIC target 
It is really dangerous to fallback to Loongson core on other
MIPS64 processors.

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
 2022-08-12 13:13:28 +01:00
Jiaxun Yang 50c4eeb97d alpha: Remove include of version.h 
It will be defined by preprocessor argument.

Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
 2022-08-11 15:02:58 +01:00
Ivan Pribec 802e71bf05 Add const attribute to lsame 2022-08-08 15:15:52 +02:00
gxw fbfe1daf6e LoongArch64: Add DYNAMIC_ARCH support 2022-07-28 14:28:45 +08:00
Martin Kroeker cd8e57040c
Merge pull request #3691 from martin-frbg/issue3679-sparc 
SPARC: fix DNRM2 returning INF instead of zero due to intermediate overflow
 2022-07-25 15:41:15 +02:00
Martin Kroeker 6c118b7977
Fix DNRM2 returning INF instead of zero due to intermediate overflow 2022-07-24 17:42:31 +02:00
Martin Kroeker c43ec53bdd
Merge pull request #3690 from RajalakshmiSR/cdotp10 
POWER: Fix complex dot function failures
 2022-07-19 13:59:16 +02:00
Martin Kroeker b7c65d08cb
Merge pull request #3689 from RajalakshmiSR/dgemvgcc10 
POWER10: dgemv builtin rename
 2022-07-19 10:25:01 +02:00
Martin Kroeker 06ef015234
fix DNRM2 returning INF instead of zero due to intermediate overflow 2022-07-19 10:19:27 +02:00
Rajalakshmi Srinivasaraghavan a612e78a97 POWER: Fix complex dot function failures 
There are some test failures in complex dot functions when compiling with gcc12.
The machine constraints used now do not update all the four elements in the
expected result array. Fixing this with a reduced level of optimization.
This is not changing any performance numbers but will be converted to C code in future.
 2022-07-18 14:48:43 -05:00
Rajalakshmi Srinivasaraghavan 432fd99445 POWER10: dgemv builtin rename 
Add check to use correct builtin name for older versions
of gcc10 compilers.
 2022-07-18 09:48:01 -05:00
gxw 4dd05e526b LoongArch64: Fix dnrm2_tiny testcase failure 2022-07-15 11:18:59 +08:00
gxw cce4b1d956 MIPS64: Fix dnrm2_tiny testcase failure 2022-07-11 19:18:38 +08:00
Martin Kroeker e12d474780
Eliminate uses of CREAL on left-hand side of assignments 2022-07-05 00:01:09 +02:00
Martin Kroeker 9e29598575
workaround fault with ssq=inf,scale=0 2022-07-02 23:47:17 +02:00
Honglin Zhu 123e0dfb62 Neoverse N2 sbgemm: 
1. Modify the algorithm to resolve multithreading failures
    2. No memory allocation in sbgemm kernel
    3. Optimize when alpha == 1.0f
 2022-06-29 10:14:21 +08:00
Honglin Zhu bc3728475f format code 2022-06-29 10:14:21 +08:00
Honglin Zhu 55d686d41e neoverse n2 sbgemm: 
implement ncopy tcopy kernel_8x4
 2022-06-29 10:14:21 +08:00
Honglin Zhu 04593bb27c neoverse n2 sbgemm: init file 2022-06-29 10:14:21 +08:00
Martin Kroeker be5500e704
Merge pull request #3669 from VFerrari/fix_small_matrix_kernel 
POWER: fix issues with the small matrix kernel
 2022-06-28 16:09:36 +02:00
Martin Kroeker 92275a7902
Merge pull request #3642 from nursik/develop 
Add ARM64 support for Windows
 2022-06-28 16:05:11 +02:00
VFerrari cac634fce3
POWER10: Fix multithreading check when USE_THREAD=0 
This patch fixes an issue when OpenBLAS is compiled for TARGET=POWER10
and the flag USE_THREAD is set to 0.

The function `num_cpu_avail` is only available when USE_THREAD=1,
so SMP is defined.
 2022-06-25 03:46:46 -03:00
Martin Kroeker 9283c7c0b5
Merge pull request #3655 from RajalakshmiSR/zgemmasmp10 
POWER10: Fix ZGEMM testcase failures
 2022-06-18 20:52:26 +02:00
Rajalakshmi Srinivasaraghavan f191bc652b POWER10: Fix ZGEMM testcase failures 
This patch fixes storing and restoring non volatile registers
in zgemm POWER10 kernel.
 2022-06-17 08:18:08 -05:00
Rajalakshmi Srinivasaraghavan 8419d538ff POWER10: convert dgemv inline assembly 
This patch makes use of compiler builtins and matches with assembly
performance. Tested with clang14 and gcc12.
 2022-06-09 10:42:57 -05:00
Xianyi Zhang 5e9a912591 Merge branch 'develop' into risc-v 2022-06-06 14:12:09 +08:00
Xianyi Zhang 968e1f51d8 Update RISC-V Intrinsic API. 2022-06-06 13:52:21 +08:00
Nursultan Zarlyk 1bb7993a97 Fix MSVC ARM64 build. Add generic kernel for ARM64 2022-06-02 16:53:54 +02:00
Martin Kroeker dc49edd4e6
Revert "roll back DGEMM kernel ... for DYNAMIC_ARCH" 2022-05-20 11:23:30 +02:00
Rajalakshmi Srinivasaraghavan b62173c5a0 POWER10: Changing store instructions for Level1 functions 
This patch changes 32 bytes stores to two 16 bytes stores
to fix a recent degradation due to 32 bytes stores.
 2022-05-12 11:17:33 -05:00
Martin Kroeker 84cb58b7fb
Fix generator rules for ?laswp_ncopy and ?neg_tcopy 2022-04-30 15:28:38 +02:00
Martin Kroeker 05dcfa176e
fix undefined prefetchsizes 2022-04-16 10:04:27 +02:00
Martin Kroeker 2bbb9f05c7
fix undefined prefetchsize 2022-04-16 10:00:10 +02:00
Martin Kroeker 115bc9b98f
CortexX1 is ARMV8 like A7x 2022-03-28 17:28:29 +02:00
Martin Kroeker b3b4672c30
Add initial support for Phytium FT2000 series and ARMV9 Cortex 510/710/X1/X2 2022-03-27 15:29:20 +02:00
Martin Kroeker 40302558ed
Remove extraneous (and wrong) definition of sbgemm_r on x86_64 2022-03-23 20:05:32 +01:00
Caroline Newcombe 5cc1111383 fix unsafe read of Y in assembly kernel 2022-03-11 11:56:33 -06:00
Xianyi Zhang 45786b05da Merge branch 'develop' into risc-v 2022-02-28 11:48:02 +08:00
Wangyang Guo 225683218c Small Matrix: use proper inline asm input constraint for AVX512 mask 2022-02-28 03:22:31 +00:00
Martin Kroeker 9c626e466e
really fix definition of SHUFFLE_MAGIC_NO 2022-02-25 15:36:02 +01:00
Martin Kroeker 0698212c8c
Remove stray $ 2022-02-25 15:33:02 +01:00
Martin Kroeker 9d7429406f
Declare SHUFFLE_MAGIC_NO as const to placate clang 2022-02-25 10:05:36 +01:00
Martin Kroeker d9894f45d3
Define sbgemm_r to fix DYNAMIC_ARCH builds 2022-02-25 10:04:00 +01:00
Martin Kroeker 522f809825
Merge pull request #3542 from martin-frbg/issue3540 
Fix compilation for CooperLake on Windows/clang
 2022-02-24 00:00:00 +01:00
Mosè Giordano abbc947edb Fix compilation of Skylake AVX512 kernels with GCC 6 2022-02-23 22:51:59 +00:00
Martin Kroeker c62f8e2c01
Prevent compiler attempts to use k0 as mask register 2022-02-23 20:12:20 +01:00
Martin Kroeker 80eb581c83
Fix non-portable u_int64_t 2022-02-23 20:10:59 +01:00
Martin Kroeker 73ffabe6ba
Guard uses of _mm512_reduce_add_p? 2022-02-23 20:06:14 +01:00
Martin Kroeker 7656aba00e
Merge pull request #3493 from martin-frbg/casts+cleanup 
WIP casts and cleanups
 2022-02-06 23:55:06 +01:00
Martin Kroeker addc2a7aaa
Add proper defaults for IMIN/IMAX 2022-01-27 19:56:32 +01:00
Martin Kroeker 299d4d70a3
Add default KERNEL file for Elbrus E2K arch 2022-01-22 18:59:36 +01:00
Martin Kroeker 3492bea602
Create Makefile 2022-01-22 18:57:28 +01:00
Martin Kroeker 898cf5faf3
Add Elbrus e2k architecture support 2022-01-22 18:55:10 +01:00
Martin Kroeker c1c0d5ce1d
Merge pull request #3492 from binebrank/arm_sve_zgemm 
SVE zgemm&cgemm (and other BLAS 3 complex)
 2022-01-18 21:36:33 +01:00
Bine Brank 19d435b1b3 update armv8sve + contributors 2022-01-18 08:28:31 +01:00
Bine Brank f158d59087 adapt CMake 2022-01-17 22:36:48 +01:00
Bine Brank b6a445cfd8 adapt Makefile for SVE trsm 2022-01-16 21:40:56 +01:00
Bine Brank 0fb6cc07bf fix ztrsm lt/ut copy 2022-01-16 21:39:57 +01:00
Bine Brank f1315288a8 add sve ztrsm 2022-01-15 22:27:25 +01:00
Bine Brank aaa2b1a861 fix sve dtrsm kernels 2022-01-15 21:02:14 +01:00
Bine Brank 8071e179f1 add remaining sve trsm copy kernels 2022-01-11 21:16:38 +01:00
Bine Brank f87468ac91 trsm_lncopy_sve 2022-01-10 21:45:37 +01:00
Bine Brank e8939b3d30 sve trsmRN and trsmRT 2022-01-10 20:42:20 +01:00
Bine Brank 098672b51b add trsm_kernel_LT_sve 2022-01-09 20:11:47 +01:00
Bine Brank be7e55880c sve trsm_kernel_LN 2022-01-09 19:40:04 +01:00
Martin Kroeker b6b024232d
Merge pull request #3508 from snadampal/v1_n2 
OpenBLAS: aarch64: Add neoverse-v1/n2 architecture specifics
 2022-01-09 14:50:26 +01:00
Sunita Nadampalli 19c8f615dc OpenBLAS: aarch64: Add neoverse-v1/n2 architecture specifics 2022-01-07 00:28:17 +00:00
Bine Brank bb33446b40 fix makefile.L3 2022-01-06 10:26:11 +01:00
Bine Brank f33543d029 combine zchemm into single file 2022-01-05 14:42:37 +01:00
Bine Brank 0c91d043ae adapt CMake for SVE 2022-01-05 14:36:39 +01:00
Bine Brank 39ab219704 sve copy functions for cgemm chemm zsymm 2022-01-05 09:12:22 +01:00
Bine Brank 18102ae8c3 add cgemm ctrmm sve kernels 2022-01-05 09:09:18 +01:00
Bine Brank 87537b8c55 modify sve zgemmcopy kernels 2022-01-05 09:07:28 +01:00
Bine Brank d30157d891 update configuration of kernels for A64FX and ARMV8SVE 2022-01-05 09:00:54 +01:00
Bine Brank 07fa6fa3b1 configure Makefile for sve 2022-01-05 08:57:51 +01:00
Bine Brank 2e2c02b762 fix sve ztrmm kernel 2022-01-04 14:42:07 +01:00
Bine Brank 68c414d3a6 ztrmm sve copy functions 2022-01-04 14:40:59 +01:00
Bine Brank ce329ab686 add sve zhemm copy routines 2022-01-03 15:56:05 +01:00
Bine Brank 0140373802 add sve ztrmm 2022-01-02 19:15:33 +01:00
Bine Brank f7b6912868 ztrmm sve copy kernels 2021-12-30 21:00:16 +01:00
Bine Brank 40b14e4957 fix zgemm kernel 2021-12-29 11:42:04 +01:00
Bine Brank 6ec4aab875 zgemm sve copy routines 2021-12-26 17:05:46 +01:00
Bine Brank 878064f394 sve zgemm kernel 2021-12-26 08:44:05 +01:00
Bine Brank 683a7548bf added macros for sve zgemm kernels 2021-12-25 11:46:41 +01:00
Martin Kroeker 7b146e590c
fix function typecast 2021-12-24 20:01:52 +01:00
Martin Kroeker e9a0e52201
fix function typecast 2021-12-24 20:00:50 +01:00
Martin Kroeker d1ee6ff73f
fix function typecasts 2021-12-21 18:45:28 +01:00
Bine Brank e3c9947c0f prepare kernel for sve zgemm 2021-12-21 11:19:27 +01:00
gxw 8d9b9c6b2a loongarch64: Optimize dgemm_kernel 2021-12-21 09:33:06 +08:00
Wu Zhigang 92b7b949dd fix bug in zscal function 
memset can not be used in zscal because of
the stride parameters.

Signed-off-by: Wu Zhigang <zhigang.wu@starfivetech.com>
 2021-12-15 01:23:30 -08:00
Martin Kroeker b0a590f4fe
Merge pull request #3475 from wjc404/optimize-A53-dgemm 
optimize cgemm on ARM cortex A53 & cortex A55
 2021-12-12 19:09:08 +01:00
Martin Kroeker f4d1f0333b
Merge pull request #3474 from rafaelcfsousa/rafael/cmake_power 
Add CMake support for Power
 2021-12-12 19:08:27 +01:00
Jia-Chen b610d2de37 optimize cgemm on ARM cortex A53 & cortex A55 2021-12-12 17:22:52 +08:00
Martin Kroeker 697e2752d7
Merge pull request #3464 from binebrank/arm_sve_sgemm 
Add sgemm part for Arm SVE
 2021-12-11 20:35:22 +01:00
Bine Brank a8f62a347b fix UNROLL_MN and add to targets for SVE 2021-12-11 16:37:23 +01:00
Bine Brank 774267fdac adjust Makefile.L3 for SVE 2021-12-11 16:35:08 +01:00
Rafael Cardoso Fernandes Sousa 23a7561353 Fix error cmake (small kernels) 2021-12-09 09:57:39 -06:00
Martin Kroeker 5378046abd
roll back DGEMM kernels to 4x8 when compiling for DYNAMIC_ARCH 2021-12-06 19:43:54 +01:00
Bine Brank a1fea1fe2a sgemm v2x8 SVE kernel 2021-12-05 18:47:29 +01:00
Bine Brank abe1ce3434 strmm sve v1x8 kernel 2021-12-05 14:03:08 +01:00
Martin Kroeker 54d321d742
Merge pull request #3466 from rafaelcfsousa/rafael/small_matrix_p10 
[POWER] Add small matrix for sgemm/dgemm on Power10
 2021-12-03 12:12:20 +01:00
Martin Kroeker 0882db30a2
Merge pull request #3455 from cenewcombe/develop 
Fix unsafe read during final iteration of zsymv_L_sse2.S
 2021-12-03 10:01:20 +01:00
Bine Brank 0de36f7b5c trmm sve copy fucntions for single precision 2021-11-29 21:25:05 +01:00
Rafael Cardoso Fernandes Sousa c78fdcc80d [POWER] Add support for SMALL_MATRIX_OPT 2021-11-28 12:41:16 -06:00
Bine Brank 86ae89bf33 add sgemm kernel and copy functions for sgemm and ssymm 2021-11-28 18:12:47 +01:00
Martin Kroeker 454edd741c
Merge pull request #3425 from binebrank/arm_sve_dgemm 
Add dgemm kernel for arm64 SVE
 2021-11-26 16:14:55 +01:00
Martin Kroeker bcfbdc81b2
Merge pull request #3459 from rafaelcfsousa/fix_cmake 
Fix issues when building OpenBLAS with cmake
 2021-11-26 15:19:24 +01:00
Bine Brank 1af73ce38e Adapt CMake for SVE 2021-11-26 10:35:01 +01:00
Martin Kroeker e7fca060db
Merge pull request #3457 from wjc404/optimize-A53-dgemm 
MOD: optimize zgemm on cortex-A53/cortex-A55
 2021-11-26 10:30:47 +01:00
Jia-Chen 5c1cd5e0c2 MOD: add comments to a53 zgemm kernel 2021-11-25 22:48:48 +08:00
Rafael Cardoso Fernandes Sousa d5c9353f1b Modify the order that cmake set the KERNEL variables (generic now is fallback) 2021-11-24 20:08:35 -06:00
Jia-Chen 9f59b19fcd MOD: optimize zgemm on cortex-A53/cortex-A55 2021-11-24 21:51:45 +08:00
Bine Brank 531a28b6a0 removed unused code (compiler warnings) 2021-11-22 10:12:34 +01:00
Bine Brank 9b9cb90bb1 modify Makefile for SVE copy 2021-11-22 09:54:20 +01:00
Bine Brank 9388f05a3c configure SVE Makefile 2021-11-21 18:33:43 +01:00
Bine Brank b58d4f31ab some clean-up & commentary 2021-11-21 14:56:27 +01:00
Martin Kroeker b7df500106
Add generic mips32 target 2021-11-20 17:31:51 +01:00
Bine Brank e6ed4be02e symm SVE copy rutines 2021-11-20 16:35:29 +01:00
Caroline Newcombe feeb8283a5 Fix unsafe read during final iteration of zsymv_L_sse2.S 2021-11-19 14:29:32 -06:00
Jia-Chen 302f22693a MOD: optimize normal DGEMM on ARMV8 cortex-A53 & cortex-A55 2021-11-18 21:14:43 +08:00
Bine Brank 3c7eed0e53 add remaining trmm copy rutines for SVE 2021-11-14 16:00:10 +01:00
Bine Brank 7d996b1c36 dtrmm_utcopy sve function 2021-11-13 18:48:53 +01:00
Bine Brank ab7917910d add v2x8 kernel + fix sve dtrmm 2021-11-07 20:37:51 +01:00
Bine Brank 7093372e32 add ARMV8SVE target 2021-11-01 22:53:21 +01:00
Bine Brank a8fbdbac34 fix sve dgemm kernel + sve dtrmm 2021-10-31 10:24:25 +01:00
Bine Brank 746b4f0f17 added SVE ncopy and tcopy 2021-10-30 12:11:44 +02:00
Bine Brank 1a10d3e09d add sve dgemm prototype 2021-10-27 16:37:18 +02:00
Martin Kroeker 22bf5c27ba
Add basic support for the Fujitsu A64FX (#3415) 
* Add initial support for Fujitsu A64FX as generic ARMV8
 2021-10-18 15:00:19 +02:00
Wangyang Guo 63a103ba6e sbgemm: spr: disable small matrix path by default 2021-10-17 19:08:03 -07:00
Wangyang Guo 82194ea9d2 sbgemm: spr: implement otcopy_16 2021-10-17 19:08:03 -07:00
Wangyang Guo 8632380a96 sbgemm: spr: reuse ncopy_16 from cooperlake as incopy 2021-10-17 19:08:03 -07:00
Wangyang Guo 6bc8204ce5 sbgemm: spr: optimization for tmp_c buffer 2021-10-17 19:08:03 -07:00
Wangyang Guo f018aa342a sbgemm: spr: kernel handle alpha != 1.0 2021-10-17 19:08:03 -07:00
Wangyang Guo a52456b168 sbgemm: spr: oncopy: use tile load/store instead 2021-10-17 19:08:03 -07:00
Wangyang Guo f2485352a6 sbgemm: spr: only load A once in tail_k handling 2021-10-17 19:08:03 -07:00
Wangyang Guo 9ab33228bb sbgemm: spr: process k2 and odd k at the same time 2021-10-17 19:08:03 -07:00
Wangyang Guo 10d52646e2 sbgemm: spr: oncopy: avoid handling too much pointer at a time 2021-10-17 19:08:03 -07:00
Wangyang Guo 88154ed02d sbgemm: spr: reduce tile conf loading by seperate tail k handling 2021-10-17 19:08:03 -07:00
Wangyang Guo a70bfb52d5 sbgemm: spr: kernel works for NN case when alpha is 1.0 2021-10-17 19:08:03 -07:00
Wangyang Guo 6051c86741 sbgemm: spr: kernel works for m32 in NN case 2021-10-17 19:08:03 -07:00
Wangyang Guo d0b253ac6e sbgemm: spr: implement oncopy_16 2021-10-17 19:08:03 -07:00
Wangyang Guo 1d48b7cb16 sbgemm: spr: add dummy source files 2021-10-17 19:08:03 -07:00
Wangyang Guo 3dc6052c7e initial support for Sapphire Rapids platform 2021-10-12 01:30:40 -07:00
Martin Kroeker 8c20ca345a
Use Neoverse's current mix of ThunderX2 kernels for Vortex as well 2021-10-06 11:06:43 +02:00
Martin Kroeker 8e4c209002
Merge pull request #3398 from kavanabhat/aix_p10_gnuas 
Big Endian Changes for Power10 kernels
 2021-10-05 18:59:47 +02:00
kavanabhat 9cc95e5657 AIX changes for P10 with GNU Compiler 2021-10-01 05:18:35 -05:00
kavanabhat fe3c778c51 AIX changes for P10 with GNU Compiler 2021-09-30 06:06:27 -05:00
Wangyang Guo ee5ca8a328 x86_64: BFLOAT16: fix build warning 2021-09-28 18:30:06 +08:00
Martin Kroeker 90cc944625
Move alphaI to x22 to leave x18 unused (reserved on OSX) 2021-09-17 09:53:18 +02:00
Martin Kroeker 590fbff06e
move alpha to x19/x20 to leave x18 unused for OSX 2021-09-17 09:42:17 +02:00
Martin Kroeker 380940271b
Move temp to x21 to leave x18 unused (reserved on OSX) 2021-09-17 09:28:19 +02:00
Martin Kroeker 7d75177446
Move temp to x21 to leave x18 unused (reserved on OSX) 2021-09-17 09:24:11 +02:00
Martin Kroeker 0a4ac4b585
Use x21 for I to leave x18 unused (reserved on OSX) 2021-09-17 09:19:51 +02:00
Martin Kroeker 7d4a221579
Remove unused TEMP2 and reshuffle to leave x18 unused (reserved on OSX) 2021-09-17 09:18:25 +02:00
Martin Kroeker d3a9c7ef7f
Merge pull request #3382 from rafaelcfsousa/rafael/cwarnings 
[POWER] Remove unused variable warnings.
 2021-09-17 09:15:16 +02:00
Martin Kroeker 8dfa61a61c
Initialize abs_mask1 with itself to silence a gcc warning 2021-09-15 22:11:35 +02:00
Martin Kroeker 99aa10b3ff
Initialize abs_mask1 with itself to silence a gcc warning 
actual initialization is via the _mm_cmpeq_ep18, which I've seen claimed to be the fastest way to set an xmm register to all 1s
 2021-09-15 22:10:43 +02:00
Rafael Cardoso Fernandes Sousa b751edf624 Fix unused variable warnings on Power 2021-09-15 13:36:07 -05:00
Martin Kroeker 80346b8813
Merge pull request #3379 from martin-frbg/issue3369-2 
Add casts to fix compiler warnings for SkylakeX sasum/dasum
 2021-09-15 07:18:57 +02:00
Martin Kroeker ce036a2fc0
Add casts 2021-09-14 21:41:53 +02:00
Martin Kroeker ddf106f769
Add dedicated entries for BFLOAT16 kernels 2021-09-14 16:17:18 +02:00
Martin Kroeker af8843875a
Merge pull request #3376 from martin-frbg/issue3370 
Fix a few harmless compiler warnings
 2021-09-12 00:01:31 +02:00
Martin Kroeker 0925dfe2c9
One instance of kernel_4x1 is used even on SKX 2021-09-11 15:30:19 +02:00
Martin Kroeker 7d873a329f
Add ifdefs around conditionally used functions 2021-09-11 14:38:47 +02:00
Martin Kroeker ef24712030
Move a conditionally used variable 2021-09-11 14:37:44 +02:00
Martin Kroeker d17238599b
Add casts 2021-09-11 13:38:28 +02:00
Wangyang Guo 59a1114d03 sbgemm: cooperlake: tuning for small matrix 2021-09-07 21:30:46 +08:00
Wangyang Guo 682d66555d sbgemm: cooperlake: implement ncopy_16 2021-09-07 21:30:46 +08:00
Wangyang Guo beccb83b16 sbgemm: cooperlake: add n24 kernel for tcopy_4 2021-09-07 21:30:46 +08:00
Wangyang Guo 5fcacad32b sbgemm: cooperlake: implement tcopy_4 2021-09-07 21:30:46 +08:00
Wangyang Guo bb1c4fa5bd sbgemm: cooperlake: prefetch A & B 2021-09-07 21:30:46 +08:00
Wangyang Guo 7a2d1601ec sbgemm: cooperlake: unroll core loop by 2 2021-09-07 21:30:46 +08:00
Wangyang Guo 45fdf951b6 sbgemm: cooperlake: reorder ptr increase for performance 2021-09-07 21:30:46 +08:00
Wangyang Guo cece3541ab sbgemm: cooperlake: fix bug in m64n12 2021-09-07 21:30:46 +08:00
Wangyang Guo 9df0953cde sbgemm: cooperlake: kernel works for NN 2021-09-07 21:30:45 +08:00
Wangyang Guo 2ec9f3a8aa sbgemm: cooperlake: change kernel size to 16x4 2021-09-07 21:30:45 +08:00
Wangyang Guo ef8f5fecc8 sbgemm: cooperlake: implement sbgemm_tcopy_32 2021-09-07 21:30:45 +08:00
Wangyang Guo 4c294336e6 sbgemm: cooperlake: add dummy source files 2021-09-07 21:30:45 +08:00
Martin Kroeker f1e3305974
Add workaround for Windows10 macro name clash 2021-09-01 21:36:50 +02:00
Wangyang Guo 619588fbab sbgemm: remove unnecessary b0 files 2021-08-30 17:55:01 +08:00
Wangyang Guo f39301935c sbgemm: cooperlake: make sure hot buffer aligned to 64 2021-08-30 17:40:30 +08:00
Wangyang Guo 7d27b182fc sbgemm: cooperlake: enable SBGEMM by small matrix path 2021-08-30 17:40:30 +08:00
Wangyang Guo 1d83ca4bca Small Matrix: support BFLOAT16 data type 2021-08-30 17:40:20 +08:00
Martin Kroeker bec9d9f63d
Merge pull request #3335 from guowangy/small-matrix-latest 
Add GEMM optimization for small matrix and single/double kernel for skylakex
 2021-08-29 22:33:33 +02:00
Wangyang Guo dbbb39199f sgemv: skylakex: fix build warning 2021-08-25 07:13:00 +00:00
Wangyang Guo e9acb46431 sgemv: skylakex: bug fix for sgemv_t kernel in corner case 2021-08-25 07:07:27 +00:00
Wangyang Guo f9dba63c28 Small Matrix: skylakex: remove unnecessary b0 source files 2021-08-13 03:28:44 +00:00
Wangyang Guo 989e6bbdd3 Small Matrix: reduce generic kernel source files 2021-08-13 03:17:38 +00:00
Martin Kroeker 04255be948
Merge pull request #3344 from gxw-loongson/develop 
Delete the macro instruction "li" and use "li.d" instead
 2021-08-12 15:16:46 +02:00
gxw a7bc8ec1f1 Delete the macro instruction "li" and use "li.d" instead 
Change-Id: Icff7981e2eb7df29ba5af1f8eb5be8443c67450f
 2021-08-12 17:02:54 +08:00
Rajalakshmi Srinivasaraghavan b06880c2cd POWER10: Improving dasum performance 
Unrolling a loop in dasum micro code to help in improving
POWER10 performance.
 2021-08-10 22:06:04 -05:00
Wangyang Guo 44d0032f3b Small Matrix: skylakex: fix build error in old compiler 2021-08-05 04:43:47 +00:00
Chen, Guobing 5d86becdae Add all SBGEMM kernels for IA AVX512-BF16 based platforms 
Added all SBGEMM kernels including NN/NT/TN/TT for both ColMajor and
RowMajor, based on AVX512-BF16 ISA set on IA.

Signed-off-by: Chen, Guobing <guobing.chen@intel.com>
 2021-08-05 11:11:29 +08:00
Wangyang Guo fee5abd84b Small Matrix: support cmake build 2021-08-04 08:50:15 +00:00
Wangyang Guo 478d1086c1 Small Matrix: support DYNAMIC_ARCH build 2021-08-04 03:12:41 +00:00
Wangyang Guo 6b58bca18b Small Matrix: disable low performance default kernel 2021-08-03 06:49:03 +00:00
Wangyang Guo fa777f5517 Small Matrix: skylakex: add DGEMM_SMALL_M_PERMIT and tune for TN kernel 2021-08-02 07:06:54 +00:00
Wangyang Guo 8592c21af4 Small Matrix: skylakex: dgemm nn: fix typo in idx load 2021-08-02 07:06:54 +00:00
Wangyang Guo 3e79f6d89a Small Matrix: skylakex: add dgemm tn kernel 2021-08-02 07:06:54 +00:00
Wangyang Guo 323d7da4f7 Small Matrix: skylakex: add dgemm tt kernel 2021-08-02 07:06:54 +00:00
Wangyang Guo f57fc932ac Small Matrix: skylakex: add dgemm nt kernel 2021-08-02 07:06:54 +00:00
Wangyang Guo 91ec21202b Small Matrix: skylakex: add dgemm nn kernel 2021-08-02 07:06:54 +00:00
Wangyang Guo 72e070539c Small Matrix: skylakex: add sgemm tt kernel 2021-08-02 07:06:54 +00:00
Wangyang Guo 02c6e764f2 Small Matrix: skylakex: add SGEMM_SMALL_M_PERMIT and tune for TN kernel 2021-08-02 07:06:54 +00:00
Wangyang Guo 5dc7c3c8e5 Small Matrix: add GEMM_SMALL_MATRIX_PERMIT to tune small matrics case 2021-08-02 07:06:54 +00:00
Wangyang Guo 642c393879 Small Matrix: skylakex: add sgemm tn kernel 2021-08-02 07:06:54 +00:00
Wangyang Guo ae3f5c737c Small Matrix: skylakex: sgemm nt: optimize for M < 12 2021-08-02 07:06:54 +00:00
Wangyang Guo 0d72d75bf9 Small Matrix: skylakex: add sgemm nt kernel 2021-08-02 07:06:54 +00:00
Wangyang Guo ca7682e3a3 Small Matrix: skylakex: sgemm nn: fix n6 conflicts with n4 2021-08-02 07:06:54 +00:00
Wangyang Guo 9967e61abb Small Matrix: skylakex: sgemm nn: fix error when beta not zero 2021-08-02 07:06:54 +00:00
Wangyang Guo a87736346f Small Matrix: skylakex: sgemm nn: add n6 to improve performance 2021-08-02 07:06:54 +00:00
Wangyang Guo 4c9d9940fd Small Matrix: skylakex: sgemm nn: reduce store 4 N at a time 2021-08-02 07:06:54 +00:00
Wangyang Guo 13b32f69b7 Small Matrix: skylakex: sgemm nn: reduce store 4 M at a time 2021-08-02 07:06:54 +00:00
Wangyang Guo 3d8c6d9607 Small Matrix: skylakex: sgemm nn: clean up unused code 2021-08-02 07:06:54 +00:00
Wangyang Guo 49b61a3f30 Small Matrix: skylakex: sgemm_nn: optimize for M <= 8 2021-08-02 07:06:54 +00:00
Wangyang Guo f88470323b Optimize M < 16 using AVX512 mask 2021-08-02 07:06:54 +00:00
Wangyang Guo 9186456a12 small matrix: SkylakeX: add SGEMM NN kernel 2021-08-02 07:06:54 +00:00
Xianyi Zhang 6022e5629c Refs #2587 fix small matrix c/zgemm bug. 2021-08-02 07:06:54 +00:00
Xianyi Zhang 57ed58cefe Refs #2587 Add small matrix optimization reference kernel for c/zgemm. 2021-08-02 07:06:54 +00:00
Xianyi Zhang 17d32a4a82 Change a1b0 gemm to b0 gemm. 2021-08-02 07:06:54 +00:00
Xianyi Zhang 59cb5de46b Refs #2587 Fix typos. 2021-08-02 07:06:54 +00:00
Xianyi Zhang be3349405d Add alpha=1.0 beta=0.0 for small gemm. 2021-08-02 07:01:47 +00:00
Xianyi Zhang 0a2077901c Add small marix optimization kernel interface. 
make SMALL_MATRIX_OPT=1
 2021-08-02 07:01:47 +00:00
gxw 0b8f7c8c10 Add cmake support for LOONGARCH64 2021-08-02 10:00:41 +08:00
gxw af0a69f355 Add support for LOONGARCH64 2021-07-27 15:29:12 +08:00
Martin Kroeker 49bbf330ca
Empirical workaround for numpy SVD NaN problem from issue 3318 2021-07-18 22:19:19 +02:00
Martin Kroeker 5b4b385ecf
Temporarily disable the SkylakeX sgemv_t microkernel due to LAPACK testsuite failures 2021-07-14 20:50:14 +02:00
User User-User 39ef0880ae copy conf 2021-06-19 21:49:58 +02:00
Martin Kroeker c4b464cac6
Merge pull request #3273 from austinpagan/sbgemm_gcc10_fix 
Power10: Fix for SBGEMM
 2021-06-15 22:58:48 +02:00
Gordon Fossum e6dd44d989 Power10: Fix for SBGEMM 
While testing bfloat16 sbgemm kernel, there are some failures for odd value inputs due to updating result for
additional bytes.
 2021-06-15 13:07:47 -05:00
Gilles Gouaillardet 9d292d37b2 arm64: add the missing d9 register to the clobber list 
Refs. numpy/numpy#18422

Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
 2021-06-14 17:01:28 +09:00
Martin Kroeker 2e8ff4a781
Merge pull request #3266 from martin-frbg/powerparam 
Remove spurious casts from PPC parameters and fix compilation for older targets
 2021-06-10 18:05:47 +02:00
Martin Kroeker dbba381dc3
Merge pull request #3260 from intelmy/sgemv_t_opt 
Optimized sgemv_t for small N based on AVX512
 2021-06-10 16:08:24 +02:00
Martin Kroeker efdbdd8f82
Add prefetch values for power3 2021-06-10 11:20:29 +02:00
Martin Kroeker 3906ef3b0f
Add prefetch values for power3 2021-06-10 11:19:40 +02:00
Martin Kroeker 8adf0971d8
Add prefetch values for power3 2021-06-10 11:18:22 +02:00
Martin Kroeker 08e2e60762
Add prefetch values for power3 2021-06-10 11:17:33 +02:00
Martin Kroeker fb9e678235
Fix caxpy/zaxpy for big-endian 2021-06-10 11:15:48 +02:00
Martin Kroeker dc4fcb48df
Fix inverted conditional for caxpy/zaxpy 2021-06-10 11:14:03 +02:00
Martin Kroeker 7a48247761
fix c/zrot and sgemv for POWER5 2021-06-10 11:11:56 +02:00
Rajalakshmi Srinivasaraghavan cbb70438df POWER10: Fixes for sbgemm kernel 
While testing bfloat16 sbgemm kernel, there are some failures
for odd value inputs due to array access beyond the boundary.
 2021-06-09 12:20:09 -05:00
Ma, Yu 706a08d4a0 Optimized sgemv_t for small N based on AVX512 2021-06-08 15:08:28 -04:00
Zhaofeng Li 590be3fae3 riscv64: Add Makefile 2021-06-07 22:55:56 +00:00
Zhaofeng Li 3521cd48cb RISCV64_GENERIC: Use generic kernel for DSDOT for better precision 
The implementation in `riscv64/dot.c` fails the `test_dsdot` test, and
the generic kernel seems to have better precision. Tested on SiFive
FU740 (HiFive Unmatched) and QEMU.

Also see #1469.
 2021-06-07 22:50:23 +00:00
Zhaofeng Li 1e0192a5cc riscv64/imin: Fix wrong comparison 
Same as #1990.
 2021-06-07 22:49:39 +00:00
Martin Kroeker 5f677e782e
Merge pull request #3196 from guowangy/skylakex-gemm-batch-k 
GEMM: skylake: improve the performance when m is small
 2021-05-22 19:25:28 +02:00
Martin Kroeker 02087a62e7
Merge pull request #3205 from intelmy/sgemv_n_opt 
optimize on sgemv_n for small n
 2021-05-17 17:49:01 +02:00
Martin Kroeker 4ecf631f95
Merge pull request #3228 from martin-frbg/issue3226 
filter out -mavx flag on Sandybridge zgemm/ztrmm kernels
 2021-05-15 09:06:12 +02:00
Martin Kroeker 310b76aad7
Merge pull request #3231 from martin-frbg/issue3227 
Support compilation with pre-C99 versions of MSVC
 2021-05-14 23:28:06 +02:00
Martin Kroeker c4da892ba0
Only filter out -mavx on Sandybridge ZGEMM/ZTRMM kernels 2021-05-14 23:19:10 +02:00
Martin Kroeker 8b90e5f202
Drop redundant inclusion of complex.h 2021-05-14 15:06:44 +02:00
Martin Kroeker bd60fb6ffc
filter out -mavx flag on zgemm kernels as it can cause problems with older gcc 2021-05-13 23:05:00 +02:00
Martin Kroeker 37ea8702ee
Merge pull request #3192 from damonyu1989/develop 
Update the intrinsic api to the offical name.
 2021-05-11 16:00:45 +02:00
Martin Kroeker c0ca63ea46
Fix missing conditionals for non-SKX kernels 2021-05-05 14:55:36 +02:00
pnp 3d4ccd2a13 fix for build error 2021-04-30 12:25:33 -04:00
pnp c59652f0ce optimize on sgemv_n for small n 2021-04-30 12:14:58 -04:00
Wangyang Guo aa7b3dc3db GEMM: skylake: improve the performance when m is small 2021-04-28 13:56:06 +00:00
damonyu ceb44bef14 update the intrinsic api to the offical name. 2021-04-27 11:12:29 +08:00
Martin Kroeker 3d511f0e66
replace spurious avx512 requirement with fma check 2021-04-26 21:55:30 +02:00
Rajalakshmi Srinivasaraghavan 2379abaa5e POWER10: Improve dgemm performance 
This patch uses vector pair pointer for input load operation
which helps to generate power10 lxvp instructions.
 2021-04-13 22:30:06 -05:00
Rajalakshmi Srinivasaraghavan 55bb9f639a POWER10: Optimized zgemv 
This patch makes use of Matrix-Multiply Assist (MMA)
feature introduced in POWER ISA v3.1 for zgemv_n and zgemv_t.
 2021-04-10 19:00:24 -05:00
Martin Kroeker 2dfb24730d
Use "old" compute(24) function with clang due to register limitations 2021-04-06 19:58:32 +02:00
Martin Kroeker 147e0a75fd
Merge pull request #3170 from CodesWithWolves/sgemm_tcopy_16-invalid-read 
Remove Unnecessary/Erroneous Adds/Reads In sgemm_tcopy_16.S COPY1x8 Macro
 2021-04-03 19:49:47 +02:00
Rajalakshmi Srinivasaraghavan 2dbcddd83d POWER10: Adding check for little endian 
This patch makes sure that recent POWER10 patches are used
only for little endian.
 2021-03-31 21:32:42 -05:00
CodesWithWolves d2bda3b56a Remove Unnecessary/Erroneous Reads In sgemm_tcopy_16.S COPY1x8 Macro 
There appears to have been some code leak when copying from the COPY2x8
macro above where we're reading 8 bytes into d4-d7 directly after
reading 4 bytes into s4-s7. These 32 bytes in d4-7 are unused and can
possibly overrun the boundary of allocated memory -- Valgrind detected
this which is what dragged my attention to it for a 128,1 copy.

Additionally, there is no need to update the addresses stored in A0-A7
as the only possible paths after running this macro will overwrite A0-7
if looping to the next 8 rows, or overwrite A0-3 if moving to 4 rows --
in which case A4-7 are unused.
 2021-03-31 15:44:25 -04:00
Martin Kroeker bdd6e3a153
Merge pull request #3157 from martin-frbg/issue3020-final 
Add workaround for LAPACK testsuite failures with the NVIDIA HPC compiler on PPC
 2021-03-19 15:23:12 +01:00
Martin Kroeker 7b8f580941
Merge pull request #3156 from martin-frbg/omatcopy_d 
Move x86_64 DOMATCOPY_RT back to the C implementation
 2021-03-19 15:22:48 +01:00
Martin Kroeker 86c5a0013f
Add workaround for LAPACK testsuite failures with the NVIDIA HPC compiler 2021-03-19 11:47:58 +01:00
Martin Kroeker ef85c22474
Add workaround for LAPACK test failures with the NVIDIA HPC compiler 2021-03-19 11:46:25 +01:00
Martin Kroeker d3555d2e50
Add workaround for LAPACK test failures with the NVIDIA HPC compiler 2021-03-19 11:44:31 +01:00
Martin Kroeker 0f5e86a0d9
Remove premature entry for DOMATCOPY_RT 2021-03-18 21:53:50 +01:00
Martin Kroeker 7b294a99fd
Move common.h back to the top of the file so that SKYLAKEX (from config.h) is defined in time 2021-03-18 21:28:19 +01:00
Martin Kroeker 0934568d9c
Move includes under the ifdef for compilers w/o intrinsics support 2021-03-12 12:42:05 +01:00
Rajalakshmi Srinivasaraghavan 09d47af2c0 Optimize zscal function for POWER10 
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
 2021-03-10 17:15:33 -06:00
Martin Kroeker ef0238ba2b
Merge pull request #3130 from martin-frbg/issue3128 
Replace spurious AVX512 requirement in the Haswell srot microkernel with an AVX2/FMA3 guard
 2021-03-06 19:15:53 +01:00
Martin Kroeker a9f6f7ad39
Remove spurious AVX512 requirement and add AVX2/FMA3 guard 2021-03-06 14:35:49 +01:00
Rajalakshmi Srinivasaraghavan 41646ed006 Optimize s/dasum function for POWER10 
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
 2021-03-05 16:22:36 -06:00
Rajalakshmi Srinivasaraghavan 0571c3187b POWER10: Rename mma builtins 
The LLVM and GCC teams agreed to rename the __builtin_mma_assemble_pair and
__builtin_mma_disassemble_pair built-ins to __builtin_vsx_assemble_pair and
__builtin_vsx_disassemble_pair respectively. This patch is to make
corresponding changes in dgemm kernel. Also made changes in
inputs to those builtins to avoid some potential typecasting issues.

Reference gcc commit id:77ef995c1fbcab76a2a69b9f4700bcfd005d8e62
 2021-02-26 20:56:34 -06:00
Martin Kroeker 292d1af1a0
Update omatcopy_rt.c 2021-02-24 09:34:14 +01:00
Martin Kroeker 325b398e3c
Update omatcopy_rt.c 2021-02-24 09:13:12 +01:00
Martin Kroeker 6f5667b4d4
Enable optimized S/D OMATCOPY_RT 2021-02-24 09:03:41 +01:00
Martin Kroeker cceeee7806
Add optimized omatcopy_rt 2021-02-24 09:00:54 +01:00
Martin Kroeker 0a4546b742
Typo fix 2021-02-23 13:14:35 +01:00
Martin Kroeker b1eed27a54
Replace naive omatcopy_rt with 4x4 blocked implementation 
as suggested by MigMuc in issue 2532
 2021-02-22 21:35:42 +01:00
Martin Kroeker 47691c031f
Use Haswell optimizations for Zen as well 2021-02-11 09:26:15 +01:00
Martin Kroeker ce7ddd8921
Use Haswell optimizations for Zen as well 2021-02-11 09:25:36 +01:00
Martin Kroeker 950c047b49
Use Haswell optimizations for Zen as well 2021-02-11 09:24:51 +01:00
Martin Kroeker 46509953a9
Use Haswell optimizations for Zen as well 2021-02-11 09:24:16 +01:00
Martin Kroeker db348dcff2
Enable optimized srot/drot kernels from Haswell 2021-02-11 09:23:05 +01:00
Rajalakshmi Srinivasaraghavan 2056ffc227 Optimize cscal function for POWER10 
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
 2021-01-29 13:51:43 -06:00
Rajalakshmi Srinivasaraghavan 3ede843d50 Optimize s/dscal function for POWER10 
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
 2021-01-24 07:48:28 -06:00
Martin Kroeker 69a5558203
Merge pull request #3059 from Guobing-Chen/BF16_gemm 
Initial code for Cooperlake BF16 GEMM kernel
 2021-01-23 19:08:05 +01:00
Martin Kroeker d6905403e3
Merge pull request #3068 from alexhenrie/scan-build 
scan-build fixes
 2021-01-23 19:06:29 +01:00
Rajalakshmi Srinivasaraghavan 439b93f6d2 Optimize s/drot function for POWER10 
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
 2021-01-21 13:24:45 -06:00
Rajalakshmi Srinivasaraghavan eff7c9166e Optimize cdot function for POWER10 
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
 2021-01-15 13:40:34 -06:00
Alex Henrie 202fc9e8ed Fix uninitialized argument value in dasum_k 2021-01-14 19:40:31 -07:00
Martin Kroeker e378b24487
Merge pull request #3067 from albertziegenhagel/fix-generic-cmake 
Fix building "generic" TRMM kernel with CMake
 2021-01-14 21:35:19 +01:00
Albert Ziegenhagel e3f4063683 Fix building "generic" TRMM kernel with CMake 
The CMake "TARGET_CORE" variables stores the "generic" target name in all lowercase letters, but gets compared to an all uppercase string, which results in the wrong TRMM kernel being selected.
This commit converts the TARGET_CORE to all uppercase before comparing its value to make sure case mismatches are not an issue in the future anymore.
 2021-01-14 10:00:49 +01:00
Martin Kroeker b716c0ef01
Add workaround for NVIDIA HPC 2021-01-12 16:51:35 +01:00
Martin Kroeker 2efa3b70dc
Add workaround for NVIDIA HPC 2021-01-12 16:49:39 +01:00
Martin Kroeker 49959d4f1c
Add workaround for NVIDIA HPC 2021-01-12 16:47:15 +01:00
Martin Kroeker 0f27a03607
Add workaround for NVIDIA HPC mishandling of the asm DOT kernels 2021-01-12 16:39:35 +01:00
Martin Kroeker c2a8ebfe69
Add workaround for NVIDIA HPC mishandling of the asm DOT kernels 2021-01-12 16:38:51 +01:00
Martin Kroeker 43aac5bacc
Support NVIDIA HPC compiler 2021-01-12 16:36:12 +01:00
Chen, Guobing b0beb0b1ca Initial code for Cooperlake BF16 GEMM kernel 2021-01-11 02:15:21 +08:00
Rajalakshmi Srinivasaraghavan 601b711c78 Optimize swap function for POWER10 
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
 2021-01-08 08:01:36 -06:00
Ashwin Sekhar T K 1b2508362b arm64: Fix nrm2 for input vectors with Inf 
Fix double precision nrm2 kernels returning NaN when the
input vectors contain Inf/-Inf.
 2021-01-01 02:49:37 -08:00
Martin Kroeker 3559c5d7a2
Merge pull request #3048 from martin-frbg/issue2998 
Temporarily revert to the old NRM2 kernels for ThunderX2/3 and NeoverseN1
 2020-12-21 13:30:08 +01:00
Martin Kroeker 8631e2976a
Temporarily revert to the old nrm2 kernels 2020-12-21 07:45:13 +01:00
Martin Kroeker 2768bc1764
Temporarily revert to the old nrm2 kernels 2020-12-21 07:42:51 +01:00
Martin Kroeker 6f4698ee1f
Temporarily revert to the old nrm2 kernel 2020-12-21 07:41:18 +01:00
Martin Kroeker 114eb159a4
Disable FMA intrinsics in the srot kernel when the compiler is PGI/NVIDIA 2020-12-19 22:15:58 +01:00
Martin Kroeker 005cce5507
Amend SkylakeX options to support the NVIDIA compiler 2020-12-19 22:11:49 +01:00
Xianyi Zhang a3cac9cca0 Update sgemm kernel 1x4 for C910. 2020-12-18 11:53:23 +08:00
Martin Kroeker c73d8ee40d
Conditionally add -mfma to compiler options where needed 2020-12-17 11:34:05 +01:00
Rajalakshmi Srinivasaraghavan 2fb11f873b POWER10: Improve copy performance 
This patch aligns the stores to 32 byte boundary for scopy and dcopy
before entering into vector pair loop. For ccopy, changed the store
instructions to stxv to improve performance of unaligned cases.
 2020-12-13 10:41:45 -06:00
Martin Kroeker 043128cbe5
Merge pull request #3029 from RajalakshmiSR/axpyp10 
POWER10: Improve axpy performance
 2020-12-10 22:49:28 +01:00
Martin Kroeker 3331ca492d
Merge pull request #3021 from austinpagan/trsm_p10 
POWER: Added special unrolled vectorized versions of "Solve" for specific si…
 2020-12-10 19:42:54 +01:00
Rajalakshmi Srinivasaraghavan 346e30a46a POWER10: Improve axpy performance 
This patch aligns the stores to 32 byte boundary for saxpy and daxpy
before entering into vector pair loop. Fox caxpy, changed the store
instructions to stxv to improve performance of unaligned cases.
 2020-12-10 11:51:42 -06:00
gxw 4b548857d6 Add msa support for loongson 
1. Using core loongson3r3 and loongson3r4 for loongson
2. Add DYNAMIC_ARCH for loongson

Change-Id: I1c6b54dbeca3a0cc31d1222af36a7e9bd6ab54c1
 2020-12-09 10:28:46 +08:00
Martin Kroeker 7f11e33e8d
Merge pull request #3025 from TiredNotTear/develop 
MIPS: Fix two bugs
 2020-12-08 09:39:27 +01:00
Martin Kroeker 53e0837809
Merge pull request #3022 from jinboson/develop 
Fix test errors reported by cblas_cgemm & cblas_ctrmm
 2020-12-07 08:09:11 +01:00
Hao Chen ad38bd0e89 Fix failed cgemv and zgemv test case after using msa optimization 
The cgemv and zgemv test case will call cgemv_n/t_msa.c zgemv_n/t_msa.c files in MIPS environment.
When the macro CONJ is defined, the calculation result will be wrong due to the wrong definition of OP2.
This patch updates the value of OP2 and passes the corresponding test.
 2020-12-07 10:25:01 +08:00
Hao Chen 47b639cc9b Fix failed sswap and dswap case by using msa optimization 
The swap test case will call sswap_msa.c and dswap_msa.c files in MIPS environmnet.
When inc_x or inc_y is equal to zero, the calculation result of the two functions will be wrong.
This patch adds the processing of inc_x or inc_y equal to zero, and the swap test case has passed.
 2020-12-07 10:24:49 +08:00
Martin Kroeker b660008c7e
Work around DOT and SWAP test failures 2020-12-06 19:15:37 +01:00
Martin Kroeker f8346603cf
Fix compilation with SolarisStudio 2020-12-06 19:14:16 +01:00
Jin Bo 65de6f5957 Fix test errors reported by cblas_cgemm & cblas_ctrmm 
The file cgemm_kernel_8x4_msa.c holds the MSA optimization
codes of cblas_cgemm and cblas_ctrmm. It defines two
macros: CGEMM_SCALE_1X2 and CGEMM_TRMM_SCALE_1X2. The pc1
array index in the two macros should be 0 and 1.
 2020-12-05 15:08:17 +08:00
Gordon Fossum 213c0e7abb Added special unrolled vectorized versions of "Solve" for specific sizes, 
in DTRSM and STRSM, to improve performance in Power9 and Power10.
 2020-12-04 17:07:06 -06:00
Martin Kroeker 441c08c9ff
Merge pull request #3016 from xiegengxin/complex-asum 
Improve the performance of zasum and casum with AVX512 intrinsic
 2020-12-04 22:07:16 +01:00
Gengxin Xie 0cb7a403b2 fix error declare function blas_level1_thread_with_return_value 2020-12-02 09:51:52 +08:00
Gengxin Xie b766c1e9bb Improve the performance of zasum and casum with AVX512 intrinsic 2020-12-01 16:49:26 +08:00