Commit Graph

2721 Commits

Author SHA1 Message Date
Martin Kroeker 0f9f6e4be5
Merge pull request #5710 from martin-frbg/issue5708
Work around miscompilation of the ARM64 non-SVE DDOT kernel
2026-03-27 22:09:08 +01:00
Martin Kroeker b8dbc4a1fc
Merge pull request #5716 from yuanjia111/develop
[ARM64] Add optimized fp16 shgemm kernels for Neoverse N2
2026-03-27 13:36:25 +01:00
yuanjia e6eba9fa21 Add optimized FP16 shgemm for for NEOVERSEN2 target 2026-03-27 17:55:06 +08:00
Murray Steele f6d4fe703b Fix incorrect cast from BF16 to FP32 in SBGEMM
This change fixes a regression in SBGEMM where C is assumed to be BF16,
and so unconditionally casts the output to FP32 resulting in incorrect
outputs when beta=1.
2026-03-26 12:10:52 +00:00
Martin Kroeker e3ce4623c2
Use volatile attribute for SDOT only, to avoid creating new miscompilations 2026-03-24 23:08:02 +01:00
Martin Kroeker 4956446ca2
Merge pull request #5692 from teddygood/wasm-sum-followup
Enable DSUM SIMD path for WASM128_GENERIC
2026-03-21 12:39:30 +01:00
Martin Kroeker a89142fd5d
Merge pull request #5688 from martin-frbg/divlimit_dyn
Make PREFERRED_SIZE, GEMM_DIVIDE_LIMIT and _RATE available to DYNAMIC_ARCH builds
2026-03-20 22:23:15 +01:00
Martin Kroeker 0dd501d794
Add GEMM_DIVIDE_RATE and GEMM_PREFERRED_SIZE to parameters 2026-03-20 15:32:06 +01:00
Martin Kroeker 3f6e928d34
Declare result as volatile to keep compilers from optimizing it out 2026-03-20 11:32:23 +01:00
teddygood f0d142c4dd Enable DSUM SIMD path for WASM128_GENERIC 2026-03-20 18:39:53 +09:00
Martin Kroeker e9aab19bbc
Merge pull request #5689 from teddygood/wasm-sdot-followup
Use generic dot kernels for WASM128_GENERIC
2026-03-19 17:08:58 +01:00
Martin Kroeker 8f5e49556f
Add GEMM_DIVIDE_LIMIT to parameters 2026-03-19 08:26:33 +01:00
teddygood 8c3717f69a Add WASM SIMD widening path for DSDOT 2026-03-19 14:16:18 +09:00
teddygood 6f672df537 Use generic DDOT kernel for WASM128_GENERIC 2026-03-19 14:15:32 +09:00
teddygood 6bb0dbfd3c Use generic SDOT kernel for WASM128_GENERIC 2026-03-19 13:54:58 +09:00
Martin Kroeker adba2c3c02
Merge pull request #5685 from teddygood/wasm-intrin-backend-exp
Add a WebAssembly SIMD backend for reusable intrinsics kernels
2026-03-18 21:49:53 +01:00
teddygood 99d05575d0 Enable SAXPY for WebAssembly SIMD backend 2026-03-18 21:27:45 +09:00
teddygood 7ff3588833 Refine WebAssembly SIMD backend scope 2026-03-18 17:24:02 +09:00
Martin Kroeker 79a50d80d3
Fix potential over-optimization by gcc15 2026-03-17 23:13:58 +01:00
teddygood 53d0be88f8 Add WebAssembly SIMD backend for universal intrinsics 2026-03-18 03:23:31 +09:00
Martin Kroeker 7a95460bb1
Merge pull request #5680 from teddygood/wasm128-generic-target-exp
Add WebAssembly SIMD SGEMM and DGEMM kernels
2026-03-17 14:25:39 +01:00
Martin Kroeker a1fd7a4658
Merge pull request #5677 from CheryDan/riscv/zdrot
Optimize ZROT_RVV for the non-unit-stride case
2026-03-17 11:10:32 +01:00
teddygood 86d1451cbe Add WebAssembly SIMD GEMM kernels 2026-03-17 05:51:54 +09:00
daichengrong aa967ef6ba Optimize ZROT_RVV for the non-unit-stride case
Optimize the RVV implementation of ZROT when inc_x and inc_y are
non-unit strides (inc_x != 1, inc_y != 1).

Reorder several operations to reduce vector register pressure and
avoid unnecessary vector register spill to the stack. This helps GCC
keep vector values in registers and reduces redundant spill/reload
instructions, improving runtime performance.

No functional change.

Signed-off-by: daichengrong <daichengrong@iscas.ac.cn>
2026-03-16 14:22:54 +08:00
Martin Kroeker 4a888bcb73
set USE_TRMM for WASM 2026-03-15 23:07:16 +01:00
Martin Kroeker ef3315527f
Don't include the CPUID code in WebAssembly builds 2026-03-15 19:30:13 +01:00
Martin Kroeker 48f0a0f0ec
Generate WASM kernel including existing intrinsics-based kernels 2026-03-15 19:28:08 +01:00
Martin Kroeker cc64ce68c3
Create generic C KERNEL as baseline for WASM 2026-03-15 19:26:42 +01:00
Martin Kroeker 37262654d9
Merge pull request #5667 from fadara01/accelerate_sve128_sbgemm
Accelerate SVE128 SBGEMM/BGEMM
2026-03-06 09:14:44 +01:00
Fadi Arafeh f30202b705 Accelerate SVE128 SBGEMM/BGEMM
This accelerates SBGEMM/BGEMM by extending the existing 8x4 kernel to 8x8 (unrolling N by 8)

Not sure if it's a good idea to delete the previous 8x4 kernel?

Here are the speedups on single core Neoverse-V2 (SVE128) compared to prev state:

Per-shape speedup
  M=N=K=64: SBGEMM 1.164x (16.42%), BGEMM 1.133x (13.30%)
  M=N=K=128: SBGEMM 1.220x (22.02%), BGEMM 1.186x (18.56%)
  M=N=K=256: SBGEMM 1.241x (24.08%), BGEMM 1.235x (23.54%)
  M=N=K=512: SBGEMM 1.240x (23.95%), BGEMM 1.227x (22.75%)
  M=N=K=1024: SBGEMM 1.251x (25.11%), BGEMM 1.232x (23.23%)
  M=N=K=2048: SBGEMM 1.235x (23.47%), BGEMM 1.246x (24.64%)

Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>
2026-03-05 13:50:07 +00:00
Martin Kroeker df29cc0205
Use AVX2 in the tail loop too for consistent FMA rounding 2026-03-03 15:51:51 +01:00
Martin Kroeker ef27ec6bed
Add pragma to limit optimization level 2026-02-22 13:42:41 +01:00
Martin Kroeker 30cf14c548
Merge pull request #5640 from ChipKerchner/RVV_Narrow_Accumulate_FP16_GEMM
Added ability to accumulate in FP16.  Convert BF16 to FP32.  For FP16 and BF16 GEMM in RISC-V (BF16 now works for pre-RVA23)
2026-02-20 14:22:27 +01:00
Martin Kroeker 46b963b9a0
Use generic C kernels for SCAL on FreeBSD 2026-02-19 22:46:03 +01:00
Chip Kerchner efe63e7970 Add pre-RVA23 to BF16 GEMM. 2026-02-15 15:49:59 +00:00
Chip Kerchner 1d6aa0dc31 Add dummy memsets - just in case. 2026-02-13 20:03:35 +00:00
Chip Kerchner 7a1d23400f Add flag for not converting A & B - will be used in future to do conversion during packing. 2026-02-13 19:00:41 +00:00
Chip Kerchner 1cc377ef61 Only convert B if M is greater or equal to 4. 2026-02-13 18:14:11 +00:00
Chip Kerchner 0acb60aab3 Conversion from BF16 to FP32 only once. 2026-02-13 17:55:15 +00:00
Chip Kerchner 9701a80a9f One small change. 2026-02-12 20:35:41 +00:00
Chip Kerchner 4121a22c02 Convert BF16 values once (and vectorized). 2026-02-12 18:45:39 +00:00
Chip Kerchner 33560437f5 Convert inputs from BF16 to FP32 and use FP32 vector madds. 18% faster. 2026-02-11 19:50:48 +00:00
Chip Kerchner e3cb067bf4 Fixed MADD to use float16 values. Use LMUL = 2 in main loop. Now 1.85X faster on BananaPi. 2026-02-11 00:27:27 +00:00
Chip Kerchner 74d9fe2832 Forget to add defintion. 2026-02-10 19:00:26 +00:00
Chip Kerchner aa1cebd45b 128-bit versions. 2026-02-10 18:30:02 +00:00
Chip Kerchner b5f2a50fe9 Added ability to accumulate in FP16 for GEMM. Widens once at the end of loops. 2026-02-10 17:30:05 +00:00
Martin Kroeker 69d92490c1
move inclusion of sme_abi header into the conditional section 2026-01-29 22:24:00 +01:00
Martin Kroeker 601bdde8ec
fix stack location of dummy2 flag 2026-01-27 22:40:50 +01:00
Martin Kroeker d53d2b11a9
fix stack location of dummy2 flag 2026-01-27 22:39:37 +01:00
Martin Kroeker 861b3db733
Reuse ?SUM kernels from ThunderX2T99 2026-01-20 15:42:09 +01:00