Extends GEADD to support independent transposition of both A and C,
matching the behavior of cuBLAS's geam and Apple's Accelerate geadd.
Previously only A could be transposed.
- Add transc parameter across cblas.h, common_interface.h,
common_level3.h, common_param.h
- Add transc handling to interface/geadd.c and interface/zgeadd.c
- Extend kernel/generic/geadd.c and kernel/generic/zgeadd.c with
stride logic for transposed C
- Add transpose test coverage (hand-verified 2x2 cases and randomized
large-matrix tests) for sgeadd, dgeadd, cgeadd, zgeadd
Per review, the U74-specific DGEMM micro-kernel does not belong under
kernel/generic (reserved for portable, architecture-neutral C kernels).
Move both the hybrid dispatcher and its readable assembly source into
kernel/riscv64, next to KERNEL.U74:
- git mv kernel/generic/gemmkernel_4x4_u74.c -> kernel/riscv64/
- git mv kernel/generic/kern_u74.S -> kernel/riscv64/
- KERNEL.U74: DGEMMKERNEL ../generic/gemmkernel_4x4_u74.c
-> ../riscv64/gemmkernel_4x4_u74.c
- gemmkernel_4x4_u74.c: #include "conversion_macros.h"
-> #include "../generic/conversion_macros.h"
(that header stays under kernel/generic; same form kernel/wasm
already uses for it)
No functional change. The portable gemmkernel_4x4.c (SGEMMKERNEL) is
untouched and stays in kernel/generic. DGEMMKERNEL still resolves through
$(KERNELDIR), so both the Makefile ($(KERNELDIR)/$(DGEMMKERNEL)) and CMake
(GenerateNamedObjects "${KERNELDIR}/${DGEMMKERNEL}") builds pick the kernel
up from its new location.
Full-memory HPL N=27456 (asm hybrid + Q=256) measured at 5.99 GFLOPS
(residual PASSED, ~50% of the 12 GF peak, +10.7% over the tuned C kernel,
1.81x the stock 2x2) - the best clean figure. Doc-only header update.
Adds a hand-written RV64GC scalar 4x4 DGEMM micro-kernel (kern_u74.S) and
wires it as the U74 DGEMMKERNEL via a hybrid dispatcher
(gemmkernel_4x4_u74.c): the fast path (bm,bn multiples of 4, even bk,
non-TRMM) runs the asm; all other shapes, odd bk, and the TRMM builds fall
back to the portable C 4x4 kernel.
The asm uses a 4x4 register tile (16 accumulators), full operand double-
buffering (P/Q ping-pong) with one-iteration lookahead, and load-before-FMA
issue ordering matched to the U74's dual-issue in-order front end. Probes
show this reaches the FP-pipe peak (~16.5 cycles / 16 fmadd.d = 2.9 GF on
L1-resident data); the streaming plateau is memory-latency-bound, not the
schedule.
Measured on a VisionFive 2 (single-core, KC=256): micro-kernel 1.88 vs 1.54
GF (+22%), full blocked DGEMM 1.77 vs 1.48 GF (+20%); the advantage holds
under 4-core contention (+17%). End-to-end HPL N=10000 (4 cores, Q=256):
5.17 vs 4.97 GF (+4.0%), residual PASSED. Correctness validated against the
full BLAS Level-3 test suite (DGEMM 17,496 computational calls, 0 failures).
kern_u74.S is the readable source; the .c embeds it via top-level __asm__ so
it builds as a single OpenBLAS kernel object with no build-system changes.
The SiFive U74 (RV64GC; e.g. StarFive JH7110 / VisionFive 2) is a scalar,
in-order core with no RVV, so today it falls back to RISCV64_GENERIC whose
S/D GEMM uses the generic 2x2 C micro-kernel.
Per the U74 Core Complex Manual (Table 169) fmadd.d has a 7-cycle latency
at repeat rate 1 (fully pipelined). A 2x2 tile exposes only 4 independent
accumulator chains -- fewer than the FMA latency -- so the FP pipe stalls
on the accumulator dependency, and the 1:1 load:FMA ratio saturates the
single load/store pipe ("only one outstanding line fill", manual 8.2).
This adds a portable 4x4 GEMM micro-kernel and a dedicated U74 target:
- kernel/generic/gemmkernel_4x4.c: 16-accumulator 4x4 register tile. 16
independent chains exceed the 7-cycle latency, and the load:FMA ratio
drops to 1:2. 16 acc + 4 A + 4 B fit RV64G's 32 FP registers without
spilling. Full 4/2/1 edge handling in both M and N.
- U74 target wiring: getarch.c (FORCE_U74, 32 KiB/64 B L1D, 2 MiB L2),
param.h (S/D UNROLL 4/4; complex stays 2/2), kernel/riscv64/KERNEL.U74
(S/D GEMM -> gemmkernel_4x4 + gemm_[nt]copy_4; S/D TRMM -> existing
trmmkernel_4x4), Makefile.prebuild + Makefile.riscv64 (-mtune=sifive-u74),
TargetList.txt, cpuid_riscv64.c.
The 4x4 kernel was verified numerically against a naive reference GEMM,
driven through the real gemm_tcopy_4 / gemm_ncopy_4 packing routines,
across 27,436 M/N/K x alpha combinations covering every 4/2/1 tail case:
worst absolute error 0.
Build with: make TARGET=U74
This change fixes a regression in SBGEMM where C is assumed to be BF16,
and so unconditionally casts the output to FP32 resulting in incorrect
outputs when beta=1.
This adds all the relevant bits and pieces to add a `shgemv` path as
well as a future `hgemm`/`hgemv` path in a similar model to `sb` and `b`
interfaces.
I've also fixed a few bits and pieces around `shgemm` which didn't build
in a few situations.
This fixes an issue originally introduced with the BGEMM kernel.
I've updated the tests to run with `beta=1.0` so as to test loading and
updating from C.
Alongside this, the tests now return sensible return values to reduce
the risk of them being ignored.
Also fixed a bug in `generic/gemv_t.c` resulting in weird outputs for
`bgemv`.
- Sets up all the various entrypoints for `bgemv`
- Adds `bscal` for use in the `bgemv` interface
- Adds test cases for comparing `sgemv` and `bgemv`
- Adds generic kernels for `bgemv_n` and `bgemv_t` which are accurate
enough to pass above tests
Setting up all the infrastructure for BGEMM support in OpenBLAS, hopefully I found all the right places.
Derived mostly from the previous work done in https://github.com/OpenMathLib/OpenBLAS/pull/5287
Co-authored-by: Ye Tao <ye.tao@arm.com>
* fix multiple numerical stability and corner case issues
* add a script to generate arbitrary gemm kernel shapes
* add a generic zvl256b target to demonstrate large gemm kernel unrolls
Martin Kroeker
Xianyi Zhang