diff --git a/README.md b/README.md index c0254d5..4f7a0ea 100644 --- a/README.md +++ b/README.md @@ -35,8 +35,7 @@ Recent highlights on the Orange Pi RV2 (SpaceMiT X60, RVV): fixing an OpenBLAS ` Library-level probes — performance *and* numerical correctness: -- **[BLAS](scientific-libs/blas.html)** — OpenBLAS improvements (U74 kernel, X60 `gemv_n` fix) -- **[DGEMM](scientific-libs/dgemm.html)** — `bench_dgemm` + `difftest` performance and correctness probes +- **[BLAS](scientific-libs/blas.html)** — OpenBLAS improvements (U74 kernel, X60 `gemv_n` / TRSM fixes) and [`OpenBLAS/`](https://github.com/opensolvers/benchmarks/tree/main/OpenBLAS) verification (`bench_dgemm`, `difftest`, `verify_ctrsm`) - **[NumPy](scientific-libs/numpy.html)** — `bench_blas.py` DGEMM and `eigvalsh` through the SciPy stack - **[LAPACK](scientific-libs/lapack.html)** — LAPACK path via NumPy `eigvalsh` - **[ELPA](scientific-libs/elpa.html)** — dense eigensolver (CP2K / VASP class workloads) diff --git a/_config.yml b/_config.yml index 1e342f7..2e1fd94 100644 --- a/_config.yml +++ b/_config.yml @@ -37,8 +37,6 @@ navigation_apps: navigation_scientific_libs: - title: BLAS url: /scientific-libs/blas.html - - title: DGEMM - url: /scientific-libs/dgemm.html - title: NumPy url: /scientific-libs/numpy.html - title: LAPACK diff --git a/apps/gromacs.md b/apps/gromacs.md index b716f9a..93d99c2 100644 --- a/apps/gromacs.md +++ b/apps/gromacs.md @@ -39,7 +39,7 @@ The **`Force` row confirms the A/B** — 821.35 vs 819.97 s (0.17%, noise). RVV | Probe | Axis | Isolated speedup | Whole-app effect | | ----- | ---- | ---------------: | ---------------- | -| [DGEMM](../scientific-libs/dgemm.html) | BLAS | ~2.3× | (pure kernel) | +| [OpenBLAS verification](../scientific-libs/blas.html#verification) | BLAS | ~2.3× | (pure kernel) | | [QE](qe.html) (DFT SCF) | BLAS | ~1.5–2.0× on BLAS routines | ~1.2–1.3× (FFT half untouched) | | [QE + FFTW](../scientific-libs/fftw.html) | FFT | ~1.02× on `fftw` | **~0%** (`FFTW_ESTIMATE`) | | **GROMACS** (this page) | **FFT** | **~1.23× on `PME 3D-FFT`** | small (`Force` = 90%) | diff --git a/apps/qe.md b/apps/qe.md index 8425e6d..112df74 100644 --- a/apps/qe.md +++ b/apps/qe.md @@ -6,7 +6,7 @@ Benchmark source: [opensolvers/benchmarks/qe](https://github.com/opensolvers/ben ## Why QE as a probe -[`dgemm`](../scientific-libs/dgemm.html) isolates BLAS-3; [HPL](hpl.html) and [ELPA](../scientific-libs/elpa.html) probe single HPC solvers; [FFTW](../scientific-libs/fftw.html) covers the FFT half. A full DFT SCF mixes level-3 GEMM (`calbec`, subspace rotation), dense LAPACK diagonalization, latency-bound BLAS-2, MPI, **and a large FFT fraction** — so it shows both whether a buggy vector BLAS breaks a production code, and what fraction of a real run each backend swap actually moves. +[OpenBLAS verification](../scientific-libs/blas.html#verification) isolates BLAS-3; [HPL](hpl.html) and [ELPA](../scientific-libs/elpa.html) probe single HPC solvers; [FFTW](../scientific-libs/fftw.html) covers the FFT half. A full DFT SCF mixes level-3 GEMM (`calbec`, subspace rotation), dense LAPACK diagonalization, latency-bound BLAS-2, MPI, **and a large FFT fraction** — so it shows both whether a buggy vector BLAS breaks a production code, and what fraction of a real run each backend swap actually moves. ## Correctness — stock RVV `gemv_n` breaks a real DFT SCF @@ -56,7 +56,7 @@ Same X60, patched RVV vs scalar: | Probe | Speedup | Why | | ----- | ------: | --- | -| [DGEMM](../scientific-libs/dgemm.html) (pure level-3) | ~2.3× | all BLAS-3 | +| [OpenBLAS verification](../scientific-libs/blas.html#verification) (pure level-3) | ~2.3× | all BLAS-3 | | [HPL](hpl.html) (Linpack) | ~1.8× | BLAS-3 + `dgemv` panel factorization | | [ELPA](../scientific-libs/elpa.html) (eigensolver) | ~1.58× | BLAS-3 + BLAS-2 tridiagonalization | | **QE** (full DFT SCF) | **~1.2–1.3×** | BLAS + ~40–50% FFT + MPI | @@ -76,4 +76,4 @@ RVV_LIB=/path/to/patched/libopenblas.so ./run-qe-ab.sh si-scf.in 4 RVV_LIB=/path/to/patched/libopenblas.so ./run-perf-ab.sh si-super-64.in 4 ``` -**Toolchain:** QuantumESPRESSO 7.5 / foss-2025b (EESSI), FlexiBLAS 3.4.5, OpenBLAS 0.3.30. Patched vector backend = OpenBLAS 0.3.30 with the RISC-V `gemv_n` NaN fix backported — same build used by [DGEMM](../scientific-libs/dgemm.html), [HPL](hpl.html), and [ELPA](../scientific-libs/elpa.html). +**Toolchain:** QuantumESPRESSO 7.5 / foss-2025b (EESSI), FlexiBLAS 3.4.5, OpenBLAS 0.3.30. Patched vector backend = OpenBLAS 0.3.30 with the RISC-V `gemv_n` NaN fix backported — same build used by [OpenBLAS verification](../scientific-libs/blas.html#verification), [HPL](hpl.html), and [ELPA](../scientific-libs/elpa.html). diff --git a/boards/F3.md b/boards/F3.md index f60ea7b..3a0ce3f 100644 --- a/boards/F3.md +++ b/boards/F3.md @@ -34,8 +34,8 @@ See also the [HPL app overview](../apps/hpl.html). | Probe | Scalar | Patched RVV | Speedup | | ----- | ------ | ----------- | ------- | -| [DGEMM](../scientific-libs/dgemm.html) N=2048, 1 core | 1.26 GFLOP/s | 2.96 GFLOP/s | 2.35× | -| [DGEMM](../scientific-libs/dgemm.html) N=4096, 8 threads | — | 17.71 GFLOP/s | — | +| [DGEMM](../scientific-libs/blas.html#verification) N=2048, 1 core | 1.26 GFLOP/s | 2.96 GFLOP/s | 2.35× | +| [DGEMM](../scientific-libs/blas.html#verification) N=4096, 8 threads | — | 17.71 GFLOP/s | — | | [NumPy](../scientific-libs/numpy.html) DGEMM N=4096 | 4.91 GFLOP/s | 17.51 GFLOP/s | 3.6× | | [NumPy](../scientific-libs/numpy.html) `eigvalsh` N=2048 | 9.59 s | 5.94 s | 1.6× | | ELPA `na=3000` | 50.42 s | **34.83 s** | 1.45× | diff --git a/scientific-libs/blas.md b/scientific-libs/blas.md index 4f0ebe0..f51d8ef 100644 --- a/scientific-libs/blas.md +++ b/scientific-libs/blas.md @@ -2,7 +2,7 @@ Improvements to **OpenBLAS 0.3.30** on RISC-V boards — built via EasyBuild, deployed through [EESSI](https://www.eessi.io/) with **FlexiBLAS** runtime swapping. -Verification and A/B probes: [DGEMM](dgemm.html) (`bench_dgemm`, `difftest`), [NumPy](numpy.html) (`bench_blas.py`). Source repo: [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks). +Verification microbenchmarks live in [opensolvers/benchmarks/OpenBLAS](https://github.com/opensolvers/benchmarks/tree/main/OpenBLAS) (`bench_dgemm`, `difftest`, `verify_ctrsm`). Stack probes: [NumPy](numpy.html) (`bench_blas.py`). Full repo: [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks). **Base stack:** GCC 14.3.0, OpenBLAS 0.3.30, EESSI `2025.06-001` ([`dev.eessi.io/riscv`](https://www.eessi.io/docs/repositories/dev.eessi.io-riscv/)). @@ -11,8 +11,8 @@ Verification and A/B probes: [DGEMM](dgemm.html) (`bench_dgemm`, `difftest`), [N | Board / CPU | Problem (stock 0.3.30) | Fix | Result | | ----------- | ---------------------- | --- | ------ | | [VisionFive 2](../boards/VisionFive2.html) — SiFive **U74** (scalar) | No U74 kernel; falls back to generic `RISCV64_GENERIC` C `2×2` GEMM | New **4×4 DGEMM micro-kernel** in RV64 assembly, `TARGET=U74` | Single-core DGEMM **~1.4 → 1.77 GFLOP/s**; 4-core DGEMM **6.31 GFLOP/s**; [HPL **1.69×**](../apps/hpl.html) | -| [Orange Pi RV2](../boards/RV2.html) — SpacemiT **X60** (RVV VLEN=256) | RVV `gemv_n` zeroes an **uninitialized** vector register → `dgemv` returns NaN | Backport upstream `gemv_n` fix; `TARGET=RISCV64_ZVL256B` | [DGEMM](dgemm.html) 2.3×; [NumPy](numpy.html) DGEMM 2.4×; [HPL](../apps/hpl.html) **10.53 GFLOP/s** | -| [Banana Pi F3](../boards/F3.html) — same K1 / X60 SoC | Same RVV `gemv_n` bug as RV2 | Same [easyconfigs#26444](https://github.com/easybuilders/easybuild-easyconfigs/pull/26444) fix | [DGEMM](dgemm.html) 2.35×; [NumPy](numpy.html) DGEMM **3.6×**; HPL **11.52 GFLOP/s** | +| [Orange Pi RV2](../boards/RV2.html) — SpacemiT **X60** (RVV VLEN=256) | RVV `gemv_n` zeroes an **uninitialized** vector register → `dgemv` returns NaN | Backport upstream `gemv_n` fix; `TARGET=RISCV64_ZVL256B` | Verification 2.3×; [NumPy](numpy.html) DGEMM 2.4×; [HPL](../apps/hpl.html) **10.53 GFLOP/s** | +| [Banana Pi F3](../boards/F3.html) — same K1 / X60 SoC | Same RVV `gemv_n` bug as RV2 | Same [easyconfigs#26444](https://github.com/easybuilders/easybuild-easyconfigs/pull/26444) fix | Verification 2.35×; [NumPy](numpy.html) DGEMM **3.6×**; HPL **11.52 GFLOP/s** | ## Packages @@ -23,6 +23,60 @@ Verification and A/B probes: [DGEMM](dgemm.html) (`bench_dgemm`, `difftest`), [N Build with `eb --from-pr --robot` into an **EESSI-extend** user install, then `flexiblas add` / `flexiblas default` — no downstream rebuild. +## Verification {#verification} + +Small, self-contained programs in [opensolvers/benchmarks/OpenBLAS](https://github.com/opensolvers/benchmarks/tree/main/OpenBLAS) for **BLAS performance and per-routine correctness** — used to isolate broken RVV kernels before trusting downstream apps. + +| File | Purpose | +| ---- | ------- | +| [`bench_dgemm.c`](https://github.com/opensolvers/benchmarks/blob/main/OpenBLAS/bench_dgemm.c) | Times square `C = A×B` (3 reps), reports GFLOP/s, prints `C[0]` | +| [`difftest.c`](https://github.com/opensolvers/benchmarks/blob/main/OpenBLAS/difftest.c) | `dlopen`s a BLAS `.so`, runs level-1/2/3 routines, reports `sum` / NaN counts | +| [`verify_ctrsm.c`](https://github.com/opensolvers/benchmarks/blob/main/OpenBLAS/verify_ctrsm.c) | Full-parameter TRSM correctness sweep (localizes [OpenBLAS#5928](https://github.com/OpenMathLib/OpenBLAS/pull/5928) VLEN bug in `_rvv_v1` kernels) | + +Both switch backends at runtime via FlexiBLAS or `OPENBLAS_CORETYPE` — no recompile. + +```bash +gcc -O2 bench_dgemm.c -o bench_dgemm -lflexiblas +gcc -O2 difftest.c -o difftest -ldl -lm + +OPENBLAS_NUM_THREADS=8 ./bench_dgemm 4096 +./difftest /path/to/libopenblas.so +``` + +### Correctness — `difftest` (X60, RV2 & F3) + +Bit-identical on [Orange Pi RV2](../boards/RV2.html) and [Banana Pi F3](../boards/F3.html): + +| Backend | `dgemv` NaN | `dgemm` NaN | `dtrsm` NaN | `dgemv` sum | +| ------- | ----------- | ----------- | ----------- | ----------- | +| Stock EESSI, default RVV | **192** | 0 | 0 | 198.94 (wrong) | +| Forced scalar | 0 | 0 | 0 | 42.06549 (reference) | +| Patched RVV (`gemv_n` fix) | 0 | 0 | 0 | 42.06549 (matches) | + +Fault is in **`dgemv` only** — plain `dgemm` and `dtrsm` look fine on the broken `gemv_n` build, which is why [HPL](../apps/hpl.html) and [Quantum ESPRESSO](../apps/qe.html) can fail while a GEMM micro-benchmark passes. + +A second bug — RVV `_rvv_v1` TRSM kernels not VLEN-agnostic ([OpenBLAS#5928](https://github.com/OpenMathLib/OpenBLAS/pull/5928)) — is caught by `verify_ctrsm` on `ZVL128B` builds where `GEMM_UNROLL_M ≠ VSETVL_MAX`. + +### Performance — `bench_dgemm` + +#### Orange Pi RV2 (1 core, N=2048) + +| Backend | GFLOP/s | `C[0]` | +| ------- | ------- | ------ | +| Scalar | 1.16 | 245.24 | +| Patched RVV | 2.62 | 245.24 | + +**2.3×** faster, numerically identical. + +#### Banana Pi F3 (cross-board) + +| Backend | GFLOP/s | `C[0]` | +| ------- | ------- | ------ | +| Scalar | 1.26 | 245.24 | +| Patched RVV | 2.96 | 245.24 | + +**2.35×** at 1 core. Threaded (8 cores, N=4096): **17.71 GFLOP/s** on patched RVV. + ## Notes - **U74** — performance kernel; stock OpenBLAS works but leaves FP throughput on the table. diff --git a/scientific-libs/dgemm.md b/scientific-libs/dgemm.md deleted file mode 100644 index d829e62..0000000 --- a/scientific-libs/dgemm.md +++ /dev/null @@ -1,54 +0,0 @@ -# DGEMM - -Micro-benchmarks for **BLAS level-3 performance** and **per-routine correctness** — from [opensolvers/benchmarks/dgemm](https://github.com/opensolvers/benchmarks/tree/main/dgemm). Used to isolate the OpenBLAS 0.3.30 RVV `gemv_n` NaN bug before fixing downstream apps. - -See also the [BLAS overview](blas.html) for OpenBLAS packages and board context. - -## Tools - -| File | Purpose | -| ---- | ------- | -| [`bench_dgemm.c`](https://github.com/opensolvers/benchmarks/blob/main/dgemm/bench_dgemm.c) | Times square `C = A×B` (3 reps), reports GFLOP/s, prints `C[0]` | -| [`difftest.c`](https://github.com/opensolvers/benchmarks/blob/main/dgemm/difftest.c) | `dlopen`s a BLAS `.so`, runs level-1/2/3 routines, reports `sum` / NaN counts | - -Both switch backends at runtime via FlexiBLAS or `OPENBLAS_CORETYPE` — no recompile. - -```bash -gcc -O2 bench_dgemm.c -o bench_dgemm -lflexiblas -gcc -O2 difftest.c -o difftest -ldl -lm - -OPENBLAS_NUM_THREADS=8 ./bench_dgemm 4096 -./difftest /path/to/libopenblas.so -``` - -## Correctness — `difftest` (X60, RV2 & F3) - -Bit-identical on [Orange Pi RV2](../boards/RV2.html) and [Banana Pi F3](../boards/F3.html): - -| Backend | `dgemv` NaN | `dgemm` NaN | `dgemv` sum | -| ------- | ----------- | ----------- | ----------- | -| Stock EESSI, default RVV | **192** | 0 | 198.94 (wrong) | -| Forced scalar | 0 | 0 | 42.06549 (reference) | -| Patched RVV (`gemv_n` fix) | 0 | 0 | 42.06549 (matches) | - -Fault is in **`dgemv` only** — plain `dgemm` looks fine on the broken build, which is why [HPL](../apps/hpl.html) and [Quantum ESPRESSO](../apps/qe.html) can fail while a GEMM micro-benchmark passes. - -## Performance — `bench_dgemm` - -### Orange Pi RV2 (1 core, N=2048) - -| Backend | GFLOP/s | `C[0]` | -| ------- | ------- | ------ | -| Scalar | 1.16 | 245.24 | -| Patched RVV | 2.62 | 245.24 | - -**2.3×** faster, numerically identical. - -### Banana Pi F3 (cross-board) - -| Backend | GFLOP/s | `C[0]` | -| ------- | ------- | ------ | -| Scalar | 1.26 | 245.24 | -| Patched RVV | 2.96 | 245.24 | - -**2.35×** at 1 core. Threaded (8 cores, N=4096): **17.71 GFLOP/s** on patched RVV. diff --git a/scientific-libs/lapack.md b/scientific-libs/lapack.md index e0cdcf5..a0c1896 100644 --- a/scientific-libs/lapack.md +++ b/scientific-libs/lapack.md @@ -2,7 +2,7 @@ LAPACK performance and correctness probed via NumPy **`eigvalsh`** (`dsyevd`) — the LAPACK-heavy half of the [NumPy benchmark](numpy.html). -Benchmark source: [opensolvers/benchmarks/numpy](https://github.com/opensolvers/benchmarks/tree/main/numpy) (`bench_blas.py`). For DGEMM (`A @ B`) see the [NumPy](numpy.html) or [DGEMM](dgemm.html) pages. +Benchmark source: [opensolvers/benchmarks/numpy](https://github.com/opensolvers/benchmarks/tree/main/numpy) (`bench_blas.py`). For DGEMM (`A @ B`) see the [NumPy](numpy.html) or [BLAS verification](blas.html#verification) sections. ## Orange Pi RV2 (8 threads) @@ -12,7 +12,7 @@ Patched RVV OpenBLAS via FlexiBLAS (`SciPy-bundle` module, [easyconfigs#26444](h | ------ | ------ | ----------- | ------- | | EIGH (`eigvalsh`, N=2048) | 10.54 s | 6.72 s | **1.6×** | -Stock EESSI default RVV: `eigvalsh` returns NaN / fails to converge. See [DGEMM `difftest`](dgemm.html). +Stock EESSI default RVV: `eigvalsh` returns NaN / fails to converge. See [`difftest`](blas.html#verification). ## Banana Pi F3 (cross-board) diff --git a/scientific-libs/numpy.md b/scientific-libs/numpy.md index 57c7840..283fa54 100644 --- a/scientific-libs/numpy.md +++ b/scientific-libs/numpy.md @@ -7,7 +7,7 @@ A **SciPy-stack probe** for BLAS and LAPACK through NumPy — from [opensolvers/ | DGEMM | `A @ B` | `dgemm` | GFLOP/s | | EIGH | `np.linalg.eigvalsh(S)` | `dsyevd` | seconds | -Unlike raw [DGEMM](dgemm.html), `eigvalsh` exercises BLAS-2 tridiagonalization — a correctness gate for the same `gemv_n` bug that breaks [HPL](../apps/hpl.html) and [ELPA](elpa.html). +Unlike raw [OpenBLAS verification](blas.html#verification), `eigvalsh` exercises BLAS-2 tridiagonalization — a correctness gate for the same `gemv_n` bug that breaks [HPL](../apps/hpl.html) and [ELPA](elpa.html). ```bash module load SciPy-bundle/2025.07-gfbf-2025b @@ -33,6 +33,6 @@ Both patched results finite. See [LAPACK](lapack.html) for the LAPACK angle. | DGEMM N=4096 | 4.91 GFLOP/s | **17.51 GFLOP/s** | **3.6×** | | EIGH N=2048 | 9.59 s | 5.94 s | **1.6×** | -**Stock unpatched RVV:** `eigvalsh` raises `LinAlgError: Eigenvalues did not converge`; DGEMM alone stays finite at 10.94 GFLOP/s (only `gemv` is broken — see [DGEMM difftest](dgemm.html)). +**Stock unpatched RVV:** `eigvalsh` raises `LinAlgError: Eigenvalues did not converge`; DGEMM alone stays finite at 10.94 GFLOP/s (only `gemv` is broken — see [`difftest`](blas.html#verification)). Eigensolver speedup (~1.6×) is lower than DGEMM (~2.4–3.6×) because `eigvalsh` mixes BLAS-3 with latency-bound BLAS-2 — same pattern as [ELPA](elpa.html). diff --git a/website.md b/website.md index bce719d..d3a7fd9 100644 --- a/website.md +++ b/website.md @@ -20,7 +20,7 @@ Context file for structural and content decisions on [opensolvers.com](https://w 1. **Home** · **Videos** · **YouTube** (external) 2. **Apps** — HPL, Quantum ESPRESSO, ONNX Runtime, GROMACS -3. **Scientific libs** — BLAS, DGEMM, NumPy, LAPACK, ELPA, MLAS, FFTW, ScaLAPACK +3. **Scientific libs** — BLAS (incl. OpenBLAS verification), NumPy, LAPACK, ELPA, MLAS, FFTW, ScaLAPACK 4. **Boards** — VisionFive 2, OrangePi RV2, BananaPi F3 Nav config: `_config.yml` (`navigation`, `navigation_boards`, `navigation_apps`, `navigation_scientific_libs`). Rendered in `_includes/header.html`. Cayman theme requires `_layouts/default.html` override to include the header. @@ -44,7 +44,7 @@ Catalog: `_data/videos.yml` (newest first). Rendered by `_includes/video-grid.ht | Source | Used for | |--------|----------| -| [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks) | BLAS (`dgemm/`), HPL (`hpl/`), ELPA/ScaLAPACK (`elpa/`, `scalapack/`), QE/FFTW (`qe/`, `fftw/`), GROMACS (`gromacs/`), ONNX/MLAS (`onnx/`), NumPy (`numpy/`) | +| [opensolvers/benchmarks](https://github.com/opensolvers/benchmarks) | OpenBLAS (`OpenBLAS/`), HPL (`hpl/`), ELPA/ScaLAPACK (`elpa/`, `scalapack/`), QE/FFTW (`qe/`, `fftw/`), GROMACS (`gromacs/`), ONNX/MLAS (`onnx/`), NumPy (`numpy/`) | | [EESSI/docs#818](https://github.com/EESSI/docs/pull/818) | VisionFive 2 / U74 OpenBLAS + HPL | | [EESSI/docs#819](https://github.com/EESSI/docs/pull/819) | Orange Pi RV2 / X60 RVV `gemv_n` fix + HPL | | [easyconfigs#26436](https://github.com/easybuilders/easybuild-easyconfigs/pull/26436) | U74 OpenBLAS package | @@ -60,6 +60,7 @@ Catalog: `_data/videos.yml` (newest first). Rendered by `_includes/video-grid.ht | Date | Decision | |------|----------| +| 2026-07-19 | Sync benchmarks PR #18: merge OpenBLAS verification into BLAS page; remove `dgemm.html`; fix GitHub links | | 2026-07-18 | Videos page + `_data/videos.yml`; U74 YouTube link; SEO (`url`, JSON-LD, robots.txt, page descriptions) | | 2026-07-18 | Per-board compute-backend SVGs; fix invalid UTF-8 in VisionFive 2 / K1 diagrams | | 2026-07-18 | U74 video production moved to private repo `opensolvers/u74-video` (removed from site) |