The skylakex/cooperlake d/s/c/z asum and c/z sum microkernels peel leading
elements until the input pointer reaches a 64-byte boundary (a scalar loop in
dasum/sasum, a masked header load in the complex variants) before entering an
aligned-load accumulator loop. The peel count depends on the buffer address
mod 64, so the grouping of the sum into accumulators - and therefore the
rounding of the result - depends on where the caller's buffer happens to sit
in memory. The same data at a different address can give a bitwise-different
sum.
That address dependence surfaced as non-reproducibility in OrcaFlex: LAPACK's
dstein scales each inverse-iteration eigenvector by 1/dasum(...) over a heap
array whose alignment varies with allocation history, so eigenvectors from
identical inputs differed run to run in the last bits, which zero-tolerance
regression comparison flags.
Fix by dropping the alignment peel and using unaligned loads throughout, so
the summation order is a function of the length alone. On AVX-512 hardware
unaligned load instructions on addresses that happen to be aligned cost the
same as aligned loads; only genuinely split cache lines pay a small penalty,
negligible for these level-1 reductions.
DYNAMIC_ARCH builds CORTEXA72 as a separate kernel and pull
dneg_tcopy from generic/neg_tcopy_$(DGEMM_UNROLL_M).c. Width 6 was
missing (only 1/2/4/8/16 existed), which broke the arm64 Graviton
Cirun and Azure DYNAMIC_ARM64 jobs.
Co-authored-by: Cursor <cursoragent@cursor.com>
TARGET=CORTEXA72 previously reused the A57 8x4 path. Add a dedicated
6x8 NEON ukernel, contiguous MR=6 panel packers (stock gemm_*copy_6 is
4+2), and UNROLL_M=6-aware TRSM kernels so HPL/dtrsm does not corrupt
the heap. DTRMM falls back to generic 2x2 until a matching kernel exists.
Co-authored-by: Cursor <cursoragent@cursor.com>
Pure cosmetic cleanup; no functional changes.
Convert all tab-indented code to 4-space indent, align macro
continuation backslashes, switch brace placement to consistent K&R
style, reformat multi-line vector initialisers, and remove trailing
whitespace on SAVE4x2_ACC1_SCALAR.
Signed-off-by: Amrita H S <amritahs@linux.vnet.ibm.com>
Add BGEMM (BF16 input → BF16 output) for POWER10 by reusing the
existing SBGEMM kernel infrastructure. A -DBGEMM compile flag switches
only the store path; the xvbf16ger2pp MMA instruction and BF16 packing
routines are shared with SBGEMM unchanged.
Changes
-------
kernel/power/KERNEL.POWER10
- Register BGEMM kernel and copy-routine targets, reusing the
sbgemm_ncopy/tcopy sources (packing layout is identical).
kernel/power/sbgemm_kernel_power10.c
- Under BGEMM: force v4sf_t to float so accumulators stay in
float32; add STORE4_BF16/STORE2_BF16 macros (read BF16 C, widen,
apply alpha*acc, convert back via xvcvspbf16, store); add
f32_to_bf16_scalar() for m&1/n&1 tails; add BGEMM variants of
all SAVE_ACC macros covering the full m/n tile hierarchy.
param.h
- Add BGEMM_DEFAULT_UNROLL_M=16, UNROLL_N=8, P/Q/R blocking
parameters for POWER10, matching the 16x8 kernel tile.
Unit test (utest/test_extensions/test_bgemm.c)
- Uses SBGEMM as a trusted reference. 21 test cases cover all four
transpose combinations, all m/n remainder paths, odd-k, alpha=0,
beta=0/1, and the m>=32 fast path. Tolerance 0.01 to account for
the one extra BF16 rounding on the BGEMM store.
Performance (POWER10, single-threaded, Transa=N Transb=N)
----------------------------------------------------------
Step=1 (sizes 1–200):
- Sizes 1–80: baseline 0.1–128 MFlops (scalar fallback);
patch 0.1–39,656 MFlops (up to ~350x faster at M=80).
- Sizes 81–200: patch 3x–24x faster (avg ~10x) with 16×8 MMA tile
engaged.
Step=8 (sizes 8–1024):
- Baseline peaks at ~2,950 MFlops (avg ~1,746 MFlops).
- Patch sustains 75,000–1,009,866 MFlops for sizes 384–960,
exceeding 1 TFlops at large sizes.
- Median speedup ~99x; average speedup ~140x across all sizes.
Signed-off-by: Amrita H S <amritahs@linux.vnet.ibm.com>
Keep the logical target for dynamic dispatch names while selecting kernel sources and capabilities from the core chosen by getarch after ISA restrictions.
* Add transpose support for C in GEADD (fixes#4646)
Extends GEADD to support independent transposition of both A and C,
matching the behavior of cuBLAS's geam and Apple's Accelerate geadd.
Previously only A could be transposed.
- Add transc parameter across cblas.h, common_interface.h,
common_level3.h, common_param.h
- Add transc handling to interface/geadd.c and interface/zgeadd.c
- Extend kernel/generic/geadd.c and kernel/generic/zgeadd.c with
stride logic for transposed C
- Add transpose test coverage (hand-verified 2x2 cases and randomized
large-matrix tests) for sgeadd, dgeadd, cgeadd, zgeadd
* Add self to CONTRIBUTORS.md
Extends GEADD to support independent transposition of both A and C,
matching the behavior of cuBLAS's geam and Apple's Accelerate geadd.
Previously only A could be transposed.
- Add transc parameter across cblas.h, common_interface.h,
common_level3.h, common_param.h
- Add transc handling to interface/geadd.c and interface/zgeadd.c
- Extend kernel/generic/geadd.c and kernel/generic/zgeadd.c with
stride logic for transposed C
- Add transpose test coverage (hand-verified 2x2 cases and randomized
large-matrix tests) for sgeadd, dgeadd, cgeadd, zgeadd
Per review, the U74-specific DGEMM micro-kernel does not belong under
kernel/generic (reserved for portable, architecture-neutral C kernels).
Move both the hybrid dispatcher and its readable assembly source into
kernel/riscv64, next to KERNEL.U74:
- git mv kernel/generic/gemmkernel_4x4_u74.c -> kernel/riscv64/
- git mv kernel/generic/kern_u74.S -> kernel/riscv64/
- KERNEL.U74: DGEMMKERNEL ../generic/gemmkernel_4x4_u74.c
-> ../riscv64/gemmkernel_4x4_u74.c
- gemmkernel_4x4_u74.c: #include "conversion_macros.h"
-> #include "../generic/conversion_macros.h"
(that header stays under kernel/generic; same form kernel/wasm
already uses for it)
No functional change. The portable gemmkernel_4x4.c (SGEMMKERNEL) is
untouched and stays in kernel/generic. DGEMMKERNEL still resolves through
$(KERNELDIR), so both the Makefile ($(KERNELDIR)/$(DGEMMKERNEL)) and CMake
(GenerateNamedObjects "${KERNELDIR}/${DGEMMKERNEL}") builds pick the kernel
up from its new location.
Full-memory HPL N=27456 (asm hybrid + Q=256) measured at 5.99 GFLOPS
(residual PASSED, ~50% of the 12 GF peak, +10.7% over the tuned C kernel,
1.81x the stock 2x2) - the best clean figure. Doc-only header update.
Adds a hand-written RV64GC scalar 4x4 DGEMM micro-kernel (kern_u74.S) and
wires it as the U74 DGEMMKERNEL via a hybrid dispatcher
(gemmkernel_4x4_u74.c): the fast path (bm,bn multiples of 4, even bk,
non-TRMM) runs the asm; all other shapes, odd bk, and the TRMM builds fall
back to the portable C 4x4 kernel.
The asm uses a 4x4 register tile (16 accumulators), full operand double-
buffering (P/Q ping-pong) with one-iteration lookahead, and load-before-FMA
issue ordering matched to the U74's dual-issue in-order front end. Probes
show this reaches the FP-pipe peak (~16.5 cycles / 16 fmadd.d = 2.9 GF on
L1-resident data); the streaming plateau is memory-latency-bound, not the
schedule.
Measured on a VisionFive 2 (single-core, KC=256): micro-kernel 1.88 vs 1.54
GF (+22%), full blocked DGEMM 1.77 vs 1.48 GF (+20%); the advantage holds
under 4-core contention (+17%). End-to-end HPL N=10000 (4 cores, Q=256):
5.17 vs 4.97 GF (+4.0%), residual PASSED. Correctness validated against the
full BLAS Level-3 test suite (DGEMM 17,496 computational calls, 0 failures).
kern_u74.S is the readable source; the .c embeds it via top-level __asm__ so
it builds as a single OpenBLAS kernel object with no build-system changes.
The SiFive U74 (RV64GC; e.g. StarFive JH7110 / VisionFive 2) is a scalar,
in-order core with no RVV, so today it falls back to RISCV64_GENERIC whose
S/D GEMM uses the generic 2x2 C micro-kernel.
Per the U74 Core Complex Manual (Table 169) fmadd.d has a 7-cycle latency
at repeat rate 1 (fully pipelined). A 2x2 tile exposes only 4 independent
accumulator chains -- fewer than the FMA latency -- so the FP pipe stalls
on the accumulator dependency, and the 1:1 load:FMA ratio saturates the
single load/store pipe ("only one outstanding line fill", manual 8.2).
This adds a portable 4x4 GEMM micro-kernel and a dedicated U74 target:
- kernel/generic/gemmkernel_4x4.c: 16-accumulator 4x4 register tile. 16
independent chains exceed the 7-cycle latency, and the load:FMA ratio
drops to 1:2. 16 acc + 4 A + 4 B fit RV64G's 32 FP registers without
spilling. Full 4/2/1 edge handling in both M and N.
- U74 target wiring: getarch.c (FORCE_U74, 32 KiB/64 B L1D, 2 MiB L2),
param.h (S/D UNROLL 4/4; complex stays 2/2), kernel/riscv64/KERNEL.U74
(S/D GEMM -> gemmkernel_4x4 + gemm_[nt]copy_4; S/D TRMM -> existing
trmmkernel_4x4), Makefile.prebuild + Makefile.riscv64 (-mtune=sifive-u74),
TargetList.txt, cpuid_riscv64.c.
The 4x4 kernel was verified numerically against a naive reference GEMM,
driven through the real gemm_tcopy_4 / gemm_ncopy_4 packing routines,
across 27,436 M/N/K x alpha combinations covering every 4/2/1 tail case:
worst absolute error 0.
Build with: make TARGET=U74
Martin Kroeker