Commit Graph

364 Commits

Author SHA1 Message Date
Martin Kroeker edaa73fd24
Hide the local 2VLx2VL symbol as static is insufficient for this with gcc 2025-08-20 06:33:28 -07:00
Martin Kroeker 501728a354
adjust register 20 accesses to 21 after moving x18 2025-08-20 06:24:38 -07:00
Martin Kroeker 05dbb54362
Delete misplaced file 2025-08-19 05:12:09 -07:00
Martin Kroeker 0bc19a1335
Update SME kernel details 2025-08-18 14:38:16 -07:00
Martin Kroeker ca542f319f
Add VORTEXM4 2025-08-18 08:41:38 -07:00
Martin Kroeker 53d3bb50cc
Get symbol name from build system; change b.first to b.mi for AppleClang compatibility 2025-08-18 01:37:50 -07:00
Martin Kroeker 08a00326a4
Build symbol name from build system variables 2025-08-18 01:35:41 -07:00
Martin Kroeker 89898fc499
Add sgemm_direct_performant for switching between direct and regular kernels 2025-08-18 01:31:40 -07:00
Martin Kroeker 22c6607db9
Use ASMNAME to get symbol name from build system; leave x18 unused as reserved on MacOS 2025-08-18 01:30:10 -07:00
Martin Kroeker ca22e28ca1
Rename sgemm_direct_sme1.S to sgemm_direct_sme1_2VLx2VL.S 2025-08-18 01:25:44 -07:00
Martin Kroeker f3b2a15fad
Merge pull request #5420 from yuanjia111/develop
Move the value assignment of vector x in gemv_n_sve.c to the outermos…
2025-08-16 12:06:53 -07:00
yuanjia 803e8d4838 Move the value assignment of vector x in gemv_n_sve.c to the outermost loop to reduce the repeated data retrieval.
1.Verify correctness using BLAS-Tester
    2.Using the built-in benchmark to verify performance, the performance of float and doule type improved by about 60% and about 40% respectively.The test command is:
     export OMP_NUM_THREADS=1;numactl -C 10 -l ./sgemv.goto 3000 4000 100
     export OMP_NUM_THREADS=1;numactl -C 10 -l ./dgemv.goto 3000 4000 100
2025-08-12 18:03:16 +08:00
Chris Sidebottom 5f47b872f1 Remove older kernels for BGEMM on NEOVERSEV1 2025-08-11 09:25:19 +00:00
Chris Sidebottom 114316f361 Optimize SBGEMM / BGEMM for NEOVERSEV1 further
This changes the kernels to pack full SVE vectors and reduces the
overall complexity of the inner GEMM loop.
2025-08-11 09:25:13 +00:00
Martin Kroeker f1ee61ea30
Include NEON header for the bfloat conversion functions 2025-08-04 00:21:39 -07:00
Martin Kroeker b3ffd5524a
Include NEON header for the bfloat conversion functions 2025-08-04 00:20:28 -07:00
Martin Kroeker a5e7c0e3e0
Merge pull request #5396 from abhishek-iitmadras/abhishekk_bfloat16
ARM64: Enable bfloat16 kernels by default
2025-07-28 13:39:08 -07:00
abhishek-fujitsu 0bc79da587 add neon header 2025-07-25 11:10:20 +05:30
Chris Sidebottom ea2faf0c9a Add optimized BGEMM for NEOVERSEN2 target
This re-uses the existing NEOVERSEN2 8x4 `sbgemm` kernel to implement `bgemm`.
2025-07-24 10:59:28 +00:00
Chris Sidebottom 2c3cdaf74e Optimized BGEMV for NEOVERSEV1 target
- Adds bgemv T based off of sbgemv T kernel
- Adds bgemv N which is slightly alterated to not use Y as an
accumulator due to the output being bf16 which results in loss of
precision
- Enables BGEMM_GEMV_FORWARD to proxy BGEMM to BGEMV with new kernels
2025-07-23 10:51:41 +01:00
Martin Kroeker 39c90f9859
Merge pull request #5380 from quic/topic/sgemm_direct_sme1_alpha_beta
SME1 based direct kernel (with alpha and beta) for cblas_sgemm level 3
2025-07-18 23:23:39 +02:00
Rajendra Prasad Matcha eae0abfdb6 SME1 based direct kernel with alpha and beta for cblas_sgemm level 3 API. 2025-07-17 16:14:31 +05:30
Chris Sidebottom 740efd71c4 Add optimized BGEMM kernel for NEOVERSEV1 target
This also improves the testing and generic kernel by re-using the BF16
conversion functions.

Built on top of https://github.com/OpenMathLib/OpenBLAS/pull/5357 and derived from https://github.com/OpenMathLib/OpenBLAS/pull/5287

Co-authored-by: Ye Tao <ye.tao@arm.com>
2025-07-10 23:23:27 +00:00
Martin Kroeker fd37406817
Merge branch 'develop' into optimized_gemv_n_1x3 2025-07-08 21:05:30 +02:00
Iha, Taisei f7ad906b49 Performance improvements of [SD]DOT with loop-unrolling on A64FX 2025-07-04 22:57:44 +09:00
Martin Kroeker ee26caffb3
Merge pull request #5309 from davidz-ampere/dev-ampereone
Add support for Ampere AmpereOne processors
2025-06-24 12:27:08 +02:00
davidz-ampere aa90ab4142 Add support for Ampere AmpereOne processors 2025-06-24 00:12:34 -04:00
Ian McInerney badef1d32e Update sbgemm_tcopy_4_neoversev1 kernel to use standard C types 2025-06-19 14:26:16 +01:00
davidz-ampere 84730068af reduce duplicate kernel code 2025-06-17 03:05:34 -04:00
davidz-ampere be68ef03b4 Add support for Ampere processors 2025-06-15 22:00:40 -04:00
Martin Kroeker 58eeb9041c
fix handling of dummy2 2025-06-12 03:03:01 -07:00
Martin Kroeker 1589d0b21e
Merge pull request #5281 from martin-frbg/zscal_arm64
kernel/arm64: fixed cscal and zscal
2025-06-12 01:04:18 -07:00
Sharif Inamdar 8279e68805 Optimize gemv_n_sve_v1x3 kernel
- Calculate predicate outside the loop
- Divide matrix in blocks of 3
2025-06-11 10:16:56 +00:00
Arne Juul 5442aff218 Accumulate results in output register explicitly 2025-06-09 19:03:22 +00:00
Martin Kroeker 28f8fdaf0f
support flag for NaN/Inf handling and fix scaling of NaN/Inf values 2025-05-23 14:59:59 +02:00
Martin Kroeker 5141a90993
Fix ARMV9SME target in DYNAMIC_ARCH and add SME query code for MacOS (#5222)
* Fix ARMV9SME target and add support_sme1 code for MacOS
* make sgemm_direct unconditionally available on all arm64
* build a (dummy) sgemm_direct kernel on all arm64





* Update dynamic_arm64.c
2025-05-10 22:39:32 +02:00
Martin Kroeker 151b74284e
Merge pull request #5203 from quic/fix-sgemmdirect-sme1
Add vector registers to clobber list to prevent compiler optimization.
2025-05-09 05:39:47 -07:00
abhishek-fujitsu 9c02cdb073 optimise dot using thread throttling for NEOVERSE V1 2025-04-23 22:35:05 +05:30
Martin Kroeker d0e8fd6d40
Merge pull request #5239 from annop-w/gemv_n_sve
Use SVE kernel for S/DGEMVN for SVE machines
2025-04-22 10:19:49 -07:00
Iha, Taisei 08b5c18d70 fixed a potential out-of-bounds on gemv. 2025-04-22 19:56:44 +09:00
Annop Wongwathanarat e11744a411 Use SVE kernel for S/DGEMVN for SVE machines 2025-04-22 09:40:13 +00:00
Martin Kroeker dd38b4e811
Merge pull request #5225 from annop-w/gemv_n
Improve performance for SGEMVN on NEONVERSEN1
2025-04-17 01:54:10 -07:00
Martin Kroeker 0241d516f6
Merge pull request #5220 from iha-taisei/sdgemv_n_unroll
Further performance improvements to non-transposed [SD]GEMV kernels for A64FX and Neoverse V1.
2025-04-16 12:55:55 -07:00
Annop Wongwathanarat d535728803 Improve performance for SGEMVN on NEONVERSEN1 2025-04-16 09:54:30 +00:00
Usui, Tetsuzo d711906e3e Add symv kernels for arm64 2025-04-11 20:39:52 +09:00
Iha, Taisei f1e628b889 Further performance improvements to [SD]GEMV. 2025-04-11 20:00:33 +09:00
Annop Wongwathanarat ec146157d3 Use SVE kernel for S/DGEMVT for SVE machines 2025-04-09 20:38:14 +00:00
Vaisakh K V 04915be829 Add vector registers to clobber list to prevent compiler optimization.
SME based SGEMMDIRECT kernel uses the vector registers (z) and adding
    clobber list informs compiler not to optimize these registers.
2025-04-03 12:18:43 +05:30
Ye Tao f27ba5efd1 fix bugs in aarch64 sbgemv_n kernel 2025-03-14 17:55:40 +00:00
Annop Wongwathanarat edef2e4441 Fix bug in ARM64 sbgemv_t 2025-03-13 20:55:31 +00:00