GPU performance benchmarks against CUDA with a historical dashboard

[tetsuo-cpp]

Summary

Add reproducible GPU runtime benchmarks that compare WarpForth kernels with hand-written CUDA equivalents on the same GPU, persist the history, and publish a dashboard showing whether the gap is closing over time.

The primary metric should be:

gap = median WarpForth kernel time / median CUDA kernel time

A gap of 1.0× is parity; lower is better. Show both absolute kernel times as supporting metrics.

This should measure steady-state kernel execution only. Compilation, PTX upload, CUDA context creation, module loading/JIT, allocation, and host/device copies should be reported separately or excluded, not mixed into kernel timing.

Why the existing GPU tests are a good base

The repository already has most of the expensive infrastructure:

• gpu_test/conftest.py::VastSession safely searches, rents, connects to, and destroys one Vast.ai instance per pytest session.
• Compiler builds WarpForth source locally with warpforthc --arch.
• KernelRunner uploads PTX and launches it through the CUDA Driver API.
• tools/warpforth-runner/warpforth-runner.cpp is compiled remotely with nvcc and already owns CUDA context/module/buffer/launch handling.
• .github/workflows/gpu.yml serializes GPU jobs, enforces a cost ceiling, and runs on trusted canon pushes or an owner-applied gpu-test PR label.
• .github/workflows/pages.yml already deploys the Sphinx site with GitHub Pages.
• The existing algorithm tests provide starting WarpForth kernels for multi-parameter elementwise work, naive/tiled matmul, and f32/f64 attention.

The benchmark should reuse this provisioning and cleanup path. It should not rent a second instance after correctness tests finish.

Important comparability constraint

The current offer query accepts any GPU whose compute capability is at least the selected sm_XX, then takes the cheapest offer. Those results are valid for correctness but not comparable over time.

For canonical performance runs:

• Pin one exact GPU model and architecture. Start with gpu_name=RTX_4090 and sm_89 after confirming availability under the existing $0.50/hr ceiling.
• Add a gpu_name/benchmark-hardware option to VastSession; require an exact match and fail rather than silently falling back to another model.
• Keep the existing broad search available for correctness-only runs.
• Keep the CUDA toolkit image fixed (nvidia/cuda:12.4.0-devel-ubuntu22.04 initially).
• Record the actual GPU name/UUID, machine/offer ID, compute capability, driver version, CUDA toolkit/nvcc version, max power, clocks, temperature, PTX target, LLVM version, compiler flags, workflow run, commit SHA, and source hashes in every result.
• The dashboard must never combine different GPU models, benchmark case versions, or baseline source hashes into one trend.

Exact model pinning will not eliminate host cooling/clock variance, so the paired WarpForth/CUDA ratio is the primary historical signal.

Baseline policy

Each CUDA source should be a readable, reasonable CUDA C++ implementation of the same algorithm, datatype, problem size, launch decomposition, and numerical contract as the WarpForth source.

• Compile CUDA remotely to PTX with fixed flags, initially nvcc -O3 -ptx -arch=sm_89; do not enable fast math unless both implementations intentionally use the same relaxed contract.
• Load both PTX modules through the same CUDA Driver API path.
• Do not use cuBLAS/CUB/FlashAttention, tensor cores, or a different algorithm in the canonical comparison. Those can be added later as separately named “library ceiling” metrics.
• Do not force instruction-for-instruction equivalence: the point is to measure compiler/language overhead for an equivalent implementation.
• Version a case whenever its algorithm, launch geometry, datatype, problem size, correctness tolerance, WarpForth source, or CUDA baseline changes. Start a new chart rather than splicing incompatible measurements together.

Initial kernel suite

Use one canonical problem size per chart plus a small correctness configuration. Calibrate sizes/batching so measurements are long enough to be stable without making the suite expensive.

Priority	Case	What it diagnoses	Starting point
P0	copy_i64 or scale_i64	Address calculation, global load/store, baseline runtime-stack tax	Adapt test_multi_param / test_scalar_param; use a grid-stride loop so the runtime stack is not allocated for millions of threads
P0	axpy_f64	Memory bandwidth, scalar parameter ABI, float arithmetic	Existing f64 load/store/arithmetic words
P0	reduce_sum_f64	Loops, branches, shared memory, barriers	One-stage block reduction producing one value per block; keep CUDA algorithm/launch matched to what WarpForth can express
P0	matmul_f64_naive	Inner-loop and address arithmetic/codegen	Scale the existing naive matmul test to a useful fixed square size
P0	matmul_f64_tiled	Shared memory, synchronization, indexing	Scale the existing 2×2 tiled f64 test to a practical tile such as 16×16
P0	attention_f32_naive	End-to-end reduced-width access, exp/sqrt, divergent control flow, shared memory	Reuse the algorithm in demo/attention.forth / test_naive_attention_f32; use a fixed GPT-2-like (sequence, head_dim) case and compare to the same naive CUDA algorithm, not FlashAttention
P1	polynomial_f64 / stack chain	Pure arithmetic and stack manipulation, useful for #61	Fixed Horner-style expression per element
P1	transpose or stencil	Coalescing, boundary branches, shared-memory tiling	Add after the harness is stable

Future feature-specific cases should be added when they become expressible: warp reduction (#10), MMA matmul (#11), unrolled loops (#12), atomics/histogram (#43), and multi-kernel workloads (#79).

Measurement protocol

Extend the remote runner (or its eventual replacement in #50) with a structured benchmark mode:

1. Load both WarpForth and CUDA PTX into one CUDA context and create equivalent, independently initialized parameter buffers.
2. Generate large deterministic inputs remotely from a fixed seed or load compact binary inputs. Do not pass million-element arrays through the current CLI CSV format.
3. Run each implementation once and compare output to a reference and to each other before accepting timings. Use explicit per-case tolerances for floating point.
4. Warm up both implementations (for example 10 launches), excluding module load/JIT and warm-up from timing.
5. Use CUDA events on one explicit stream—not Python time, SSH wall time, or nvidia-smi—to time batches of launches.
6. Calibrate batch count to at least roughly 50 ms per sample, then collect at least 30 paired samples. Alternate WarpForth → CUDA and CUDA → WarpForth sample order to reduce thermal/order bias.
7. Require benchmark outputs to be overwritten deterministically on each launch; if a case mutates inputs, reset them outside the timed interval.
8. Emit JSON containing all raw samples plus median, p95, MAD, sample count, absolute times, ratio, correctness status, and environment metadata.
9. Mark noisy runs (initially MAD / median > 3%) as unstable. Persist/display them but exclude them from aggregate trend and alerts.

Keep a suite_version and stable case IDs in the JSON. Add unit tests for JSON parsing/statistics and fake-SDK tests for exact GPU selection; the real event timing is covered by the labeled/canonical GPU workflow.

Suggested repository shape

benchmarks/
  README.md                 # methodology, local command, cost/noise notes
  cases.toml                # case ID/version, dimensions, grid/block, types, seed, tolerance
  kernels/
    <case>.forth[.in]
    <case>.cu

gpu_test/
  infrastructure.py         # extracted Compiler/VastSession/transport shared by tests + benchmarks
  conftest.py               # pytest options/fixtures only
  test_benchmarks.py        # orchestration; remote runner performs the timing

tools/warpforth-runner/
  warpforth-runner.cpp      # correctness mode plus CUDA-event/JSON benchmark mode

The exact extraction can stay smaller if preferred, but the reusable Vast/session code should no longer live only in pytest configuration. Keep the timing/result contract transport-independent so #50 can replace the C++ runner later without invalidating the suite.

CI integration

Extend the existing GPU workflow rather than creating a second rental path:

• Add a benchmark pytest marker and an explicit --run-benchmarks / output-file option. Benchmark cases are skipped by default.
• Run correctness tests and benchmarks in one pytest process so they share the session-scoped VastSession fixture.
• On trusted pushes to canon, use the pinned benchmark GPU, run the P0 suite, upload raw JSON as a workflow artifact, and persist dashboard aggregates.
• Keep owner-applied gpu-test PR runs safe: they may run benchmarks and publish a job summary/artifact, but they must not write historical data or deploy Pages.
• Add workflow_dispatch for manual runs. A weekly scheduled calibration run can be added after the initial noise level/cost is known; it is not required for MVP because every canon push already rents a GPU.
• Retain the existing gpu-ci concurrency group, cleanup guarantees, 45-minute job timeout, and $0.50/hr cap. Add a benchmark-suite timeout/budget so an accidental large case cannot consume the entire job.
• Persist results only after correctness passes and only from trusted canon/scheduled runs.

Do not make noisy performance thresholds a merge gate initially. First collect enough repeated data to establish variance. A later PR mode can build base and head and run both on the same rented GPU; that is much more reliable than comparing a PR to a historical run from another Vast host.

Historical storage and dashboard

Use the existing GitHub Pages deployment; do not add a metrics service for MVP.

Recommended implementation:

1. Convert the runner JSON to the customSmallerIsBetter format accepted by benchmark-action/github-action-benchmark, with stable metrics for:
   • <case>/v<version>/warpforth in µs
   • <case>/v<version>/cuda in µs
   • <case>/v<version>/gap in × CUDA
   • suite geometric-mean gap across stable P0 cases
2. Let the action store its durable aggregate history and generated Chart.js dashboard on a dedicated orphan branch such as benchmark-data (not canon). Pin the action version/commit.
3. Include MAD/noise and the environment/source hashes in each metric's range/extra metadata. Keep the full raw JSON as a GitHub Actions artifact for diagnosis.
4. Update the Pages workflow to check out/copy the generated benchmark dashboard into build/docs/html/benchmarks/ after the Sphinx build, and add a Benchmark link to the docs navigation.
5. Trigger a Pages rebuild after a successful canonical GPU workflow (workflow_run is sufficient). Keep the existing push trigger so documentation still deploys when a GPU run fails or is unavailable.
6. Grant contents: write only to the trusted persistence step. PR code must never receive a path that can update benchmark-data.

This uses the existing Pages site and avoids another account/API key. Bencher remains an option later if statistical PR gating, richer branch comparisons, or hosted retention becomes worth the extra service and secret.

The dashboard should lead with the latest geometric-mean gap and per-case gaps, then show per-case WarpForth time, CUDA time, gap over commit/date, noise, commit/run links, and the pinned hardware/toolchain signature.

Implementation phases

Phase 1: stable paired harness

• Extract reusable Vast/compiler/SSH infrastructure from conftest.py without changing correctness behavior.
• Add exact GPU-model selection and environment metadata capture.
• Add remote CUDA baseline compilation and paired CUDA-event timing with JSON output.
• Support deterministic large buffers without command-line CSV payloads.
• Add result-schema/statistics and infrastructure unit tests.

Phase 2: benchmark cases

• Add P0 WarpForth/CUDA source pairs and a versioned manifest.
• Validate each implementation before timing against deterministic expected output.
• Document baseline fairness, fixed flags, dimensions, tolerances, and local Vast.ai invocation.
• Run repeated calibration jobs and tune sample count/problem sizes/noise threshold to stay within the current CI budget.

Phase 3: CI history and dashboard

• Reuse one Vast rental for correctness + benchmark collection.
• Upload raw result artifacts and expose a concise Actions job summary.
• Persist trusted canonical aggregates to benchmark-data with github-action-benchmark.
• Publish the generated dashboard under the existing Sphinx GitHub Pages site.
• Ensure incompatible hardware/case/toolchain versions cannot silently join one series.

Phase 4: regression feedback (after variance is known)

• Define an alert threshold from observed MAD/noise (for example, max(5%, 3 × historical MAD) rather than an arbitrary fixed gate).
• Alert on both the WarpForth/CUDA ratio and WarpForth absolute time; a CUDA timing shift can identify environmental drift.
• Optionally benchmark base and head in one labeled PR run on the same GPU and comment the comparison.

Acceptance criteria

• One trusted GPU CI run rents exactly one pinned GPU and runs correctness plus all P0 benchmarks.
• Kernel execution is timed with CUDA events and excludes setup/transfer/JIT.
• Every timed result has passed correctness validation.
• The result artifact contains raw paired samples, summary statistics, commit/source versions, and full GPU/toolchain metadata.
• The public Pages dashboard shows current and historical WarpForth time, CUDA time, per-case gap, and aggregate gap.
• A run on the wrong GPU model or incompatible suite version cannot contaminate the canonical series.
• PR-triggered runs cannot write benchmark history or deploy Pages.
• Vast.ai cleanup behavior and existing fake-SDK coverage remain intact.
• Methodology and a local reproduction command are documented.

Related

• Replace warpforth-runner C++ CLI with Python cuda-python + JSON protocol #50 — runner JSON/cuda-python redesign; compatible but not a prerequisite
• Compile-time stack flattening (promote stack slots to SSA/registers) #61 — stack flattening should use this suite to quantify improvement
• Algorithm-level GPU test suite (matmul, attention, shared tiling) #72 — completed algorithm-level GPU tests provide seed kernels
• Warp-level primitives: shuffle and reductions #10, Tensor core / MMA intrinsics #11, Loop unrolling hints #12, Atomic memory operations #43, Multi-kernel compilation units (multiple entry points per module) #79 — future benchmark coverage as capabilities land