Pure cosmetic cleanup; no functional changes.
Convert all tab-indented code to 4-space indent, align macro
continuation backslashes, switch brace placement to consistent K&R
style, reformat multi-line vector initialisers, and remove trailing
whitespace on SAVE4x2_ACC1_SCALAR.
Signed-off-by: Amrita H S <amritahs@linux.vnet.ibm.com>
Add BGEMM (BF16 input → BF16 output) for POWER10 by reusing the
existing SBGEMM kernel infrastructure. A -DBGEMM compile flag switches
only the store path; the xvbf16ger2pp MMA instruction and BF16 packing
routines are shared with SBGEMM unchanged.
Changes
-------
kernel/power/KERNEL.POWER10
- Register BGEMM kernel and copy-routine targets, reusing the
sbgemm_ncopy/tcopy sources (packing layout is identical).
kernel/power/sbgemm_kernel_power10.c
- Under BGEMM: force v4sf_t to float so accumulators stay in
float32; add STORE4_BF16/STORE2_BF16 macros (read BF16 C, widen,
apply alpha*acc, convert back via xvcvspbf16, store); add
f32_to_bf16_scalar() for m&1/n&1 tails; add BGEMM variants of
all SAVE_ACC macros covering the full m/n tile hierarchy.
param.h
- Add BGEMM_DEFAULT_UNROLL_M=16, UNROLL_N=8, P/Q/R blocking
parameters for POWER10, matching the 16x8 kernel tile.
Unit test (utest/test_extensions/test_bgemm.c)
- Uses SBGEMM as a trusted reference. 21 test cases cover all four
transpose combinations, all m/n remainder paths, odd-k, alpha=0,
beta=0/1, and the m>=32 fast path. Tolerance 0.01 to account for
the one extra BF16 rounding on the BGEMM store.
Performance (POWER10, single-threaded, Transa=N Transb=N)
----------------------------------------------------------
Step=1 (sizes 1–200):
- Sizes 1–80: baseline 0.1–128 MFlops (scalar fallback);
patch 0.1–39,656 MFlops (up to ~350x faster at M=80).
- Sizes 81–200: patch 3x–24x faster (avg ~10x) with 16×8 MMA tile
engaged.
Step=8 (sizes 8–1024):
- Baseline peaks at ~2,950 MFlops (avg ~1,746 MFlops).
- Patch sustains 75,000–1,009,866 MFlops for sizes 384–960,
exceeding 1 TFlops at large sizes.
- Median speedup ~99x; average speedup ~140x across all sizes.
Signed-off-by: Amrita H S <amritahs@linux.vnet.ibm.com>
Corrected the register constraints for the PowerPC dcbt (Data Cache Block
Touch) instruction in Power10 kernel implementations. The dcbt instruction
has special behavior where if the first operand (RA) is r0, it uses the
value 0 instead of the register contents. Therefore, RA must use the "b"
constraint (any GPR except r0), while RB can use "r" (any GPR including r0).
Changes:
- Changed first operand constraint from "r" to "b" to exclude r0
- Changed second operand constraint from "b" to "r" for flexibility
This ensures correct prefetch behavior and compliance with PowerPC ISA
specifications, preventing potential issues where r0 might be incorrectly
used as the base address register.
Signed-off-by: Amrita H S <amritahs@linux.vnet.ibm.com>
Replace normal vector pair pointer dereferences with the optimized
__builtin_vsx_lxvp builtin across DGEMM, ZGEMM, and DGEMV kernels.
Also done some identation corrections in dgemm_kernel_power10.c.
This is done as part of POWER code cleanup and may not have any
performance impact.
Signed-off-by: Amrita H S <amritahs@linux.vnet.ibm.com>
With GCC 14, unnecessary move and lxvp instructions appear when unrolling the inner loop for larger sizes.
Reducing the loop unroll factor restores performance to GCC 11.
Martin Kroeker