Skip to content

Canonical benchmark suites + ideal-configuration guardrails#6

Merged
balegas merged 8 commits into
mainfrom
bench/canonical-suites
Jul 14, 2026
Merged

Canonical benchmark suites + ideal-configuration guardrails#6
balegas merged 8 commits into
mainfrom
bench/canonical-suites

Conversation

@balegas

@balegas balegas commented Jul 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Makes the cardinality-cliff campaign's findings operational — and carries the first full canonical campaign run (2026-07-14).

What's here

  • Canonical suite set (suites/canonical-*.json, 7 suites) — the maintained benchmarks covering the reported workloads: write saturation (durable-streams wal-ideal + memory; ursula v0.2.0), catch-up reads, SSE tail, mixed interference (cal / paced-readers / delivery). Each suite's _doc carries its reference numbers and regression gate. Legacy suites (80) and all previous results deleted — git history keeps them; the 2026-07-02 write numbers were found physics-inflated and are superseded.
  • AGENTS.md §0 — canonical list + the ideal-configuration invariants (stream data never on the boot PD, WAL lanes ≠ data lanes, Guaranteed QoS + static CPU manager, size-triggered checkpoints, --tier off on memory arms, removed-flag list) so the 37× write recovery doesn't get silently broken again.
  • REPORT_TEMPLATE.md — campaign report skeleton + a pre-publication caveat checklist encoding every past numbers-inflation failure (storage layout, window alignment, ladder-vs-plateau, digest provenance, physics sanity, cardinality shape, client-bound cells).
  • Campaign infrastructure: split-lane / 3×3 / guaranteed manifests, STATIC_CPU=1 node pools, SPLITLANE/GUARANTEED/SERVER_MANIFEST selectors, cluster-up hardening (fail-hard create, mktemp fix), 1M-stream barrier timeouts, plus all cliff-campaign suites/results/findings.

Canonical campaign results (2026-07-14, electric#4697 head — results/REPORT.md)

workload headline
write (wal-ideal) 417k @10k / 382k @100k — no cliff (−8%), gates passed
write (memory) 680k / 632k
write (ursula 0.2) mem 49–55k flat; disk 4.5–8.3k
reads catch-up wal 2.27–2.74 GiB/s, cardinality-flat; ursula errors @100 streams (historic ceiling)
reads SSE tail wal p99 2–3 ms to 2048 conns, flat; ursula degrades @100 streams
mixed 100k readers cost pinned 50k writes nothing; 2026-07-02 memory delivery collapse gone (126.7k del/s @ 127k writes)

All suites self-tore-down; zero clusters at campaign end (verified).

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk

balegas and others added 8 commits July 13, 2026 18:34
@balegas @claude
bench: split-lane + guaranteed-QoS manifests, STATIC_CPU pool, checkp…
6cfd6cb 
…oint-fix suites + findings

- gke/durable-streams-splitlane{,-guaranteed}.yaml: stream data on its own NVMe
  lane, WAL shards on dedicated lanes; guaranteed variant = requests==limits
  everywhere + integer server CPU for exclusive cores under static CPU manager.
  Recreate strategy + hostPath base volume (emptyDir teardown recursed through
  stale lane mounts and wiped the next server's dirs).
- cluster-up.sh: STATIC_CPU=1 -> kubelet cpuManagerPolicy=static on server pool.
- lib-bench.sh: SPLITLANE=1 / GUARANTEED=1 manifest selection.
- suites + results: wal-decomp-lane0, wal-splitlane, wal-sizetrigger, wal-cpubind
  (cliff eliminated: 10.4k -> 328k @100k streams; see reports + AGENTS.md).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk
@balegas @claude
bench: wal-stacked-1m results + cluster-up hardening (mktemp trailing…
98ab4ed 
…-X, fail-hard create)

Stacked ideal config (split-lane + syncfs + 1GiB size-trigger + pinned cores):
383k @100k (37x campaign total), 244k @500k, 56k @1m — a NEW mechanism appears
near 1M streams (fd count / dir scale / working set), next investigation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk
@balegas @claude
bench: stream-lanes 3x3 validation (1M wall broken: 68k -> 212k) + 1m…
5e46489 
…-diag telemetry harness

wal-1m-diag: root cause = checkpoint syncfs writeback of ~200k dirty files on ONE
data device (60-74s per barrier, ~40x metadata amplification of small appends);
fds peaked at 1,005,724 (96% of ceiling — adjacent risk, not the cause).
wal-streamlanes-1m: 3 stream lanes + 3 WAL shards -> 374k/285k/212k @100k/500k/1M,
syncfs 5.7-11.2s. New: splitlane3x3-guaranteed manifest, SERVER_MANIFEST override,
DS_BENCH_BARRIER_TIMEOUT_SECS=3600 for 1M setups.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk
@balegas @claude
bench: AGENTS.md stream-lanes guidance + wal-streamlanes-1m findings
73642cc 
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk
@balegas @claude
bench: canonical suite set + ideal-configuration invariants (AGENTS.m…
13a2d5f 
…d §0)

Six canonical suites replace the ad-hoc set as the maintained benchmarks:
canonical-write (durable-streams wal-ideal per WAL_TUNING.md + memory),
canonical-write-ursula (upstream v0.2.0), canonical-sustained,
canonical-reads-{catchup,live,sse}. Each _doc states reference numbers and the
regression gate (wal-ideal@100k < 250k / memory@100k < 400k / >30% cardinality
drop = the cliff is back). AGENTS.md §0 documents the do-not-break invariants:
stream data never on the boot PD, WAL lanes != data lanes, Guaranteed+static
CPU, size-triggered checkpoints, memory arms need --tier off (server refuses
memory+tier), removed-flag list, persisted layout knobs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk
@balegas @claude
bench: canonical-only — delete legacy suites + all previous results; …
1c67ad0 
…REPORT_TEMPLATE.md

- suites/: only the 7 canonical suites remain (write, write-ursula,
  reads-catchup, reads-sse, mixed-{cal,writes,delivery}), matching exactly the
  workloads we report (template: the 2026-07-02 report structure). Long-poll and
  sustained are not in the reported set and were dropped.
- results/ + results-2026-07-02/: deleted (git history keeps them). The
  2026-07-02 write numbers were later found inflated (physics-impossible
  2.05M @500k on 4 CPUs, wal > memory) — superseded by canonical references.
- REPORT_TEMPLATE.md: the report skeleton + a pre-publication caveat checklist
  encoding every past numbers-inflation failure (storage layout, window
  alignment, ladder-vs-plateau, digest provenance, physics sanity, cardinality
  shape, client-bound cells).
- AGENTS.md: §0/§1 aligned to the canonical set.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk
@balegas @claude
bench: canonical reads suites use stock per-mode clusters
2b0ada5 
A shared cluster_name override made the wal and ursula modes (which the harness
runs in PARALLEL, one cluster per mode by design) race to create the same
cluster; the loser had no kube context. Reverting to the derived per-mode
names/zones.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk
@balegas @claude
bench: canonical campaign 2026-07-14 — full results, REPORT.md + PROV…
2e9783b 
…ENANCE.md

Write: wal-ideal 417k/382k @10k/100k, memory 680k/632k (gates passed, no
cliff); ursula v0.2.0 baselines. Reads: wal cardinality-flat (2.3-2.7 GiB/s
catchup; SSE p99 2-3ms @2048 conns). Mixed: 100k readers cost writes nothing;
2026-07-02 memory delivery collapse is gone (126.7k del/s @ 127k writes).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk
@balegas

balegas commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

Canonical campaign complete (2026-07-14) — all gates passed ✅

Full results/REPORT.md + PROVENANCE.md now on the branch. Headlines:

Write (electric#4697 build, ideal config): wal-ideal 417k @10k / 382k @100k, memory 680k / 632k — no cardinality cliff (−8%/−7%), both regression gates passed with wide margin, physics-sane (memory > wal, unlike the retracted 2026-07-02 numbers). ursula v0.2.0: mem ~49–55k flat, disk 4.5–8.3k.

Reads: wal cardinality-flat — catch-up 2.3–2.7 GiB/s at 10 AND 100 streams; SSE tail p99 2–3 ms up to 2048 conns. ursula degrades at 100 streams in both (catch-up errors out — its historic ceiling).

Mixed interference: 100k concurrent catch-up readers cost the pinned 50k write path nothing (303 MiB/s of replay served alongside). And the 2026-07-02 memory-mode delivery collapse is gone: delivery tracks writes 1:1 to 127k/s.

Ops notes: every suite self-tore-down; zero clusters at campaign end. Two orchestration restarts happened (reads suites' cluster-name race — fixed in 2b0ada5; a collateral pkill of the mixed chain) — no measured cell was affected.

Ready for review/merge.

@balegas
balegas force-pushed the bench/canonical-suites branch from 9f7e2b8 to 2e9783b Compare July 14, 2026 08:50
@balegas
balegas merged commit 88c8a9e into main Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@balegas