perf(durable-streams-rust): eliminate the write cardinality cliff — syncfs checkpoints, per-shard triggers, stream lanes

[balegas]

Eliminates the WAL write cardinality cliff end-to-end. Formerly a 4-PR stack (#4704, #4705, #4707 folded in here). Full deployment guidance: WAL_TUNING.md (in this PR). Validated by the canonical benchmark campaign 2026-07-14 (ds-bench#6: results/REPORT.md + PROVENANCE.md) run against this PR's head.

The problem

WAL-durable write throughput collapsed with stream count: 10.4k ops/s @100k streams (vs ~110k @10k), degrading further under load. Two compounding causes:

1. The checkpoint issued one fdatasync per touched stream — an O(N) barrier storm sharing the device with commit fsyncs.
2. At extreme cardinality (~1M), checkpoint writeback amplification (~40× for small appends: dirty page + inode + extents + journal per file) saturated the single data device — one syncfs measured at 60–74 s.

The changes

commit	what	measured effect
syncfs checkpoint	one syncfs barrier per stream lane instead of O(N-touched) fdatasync (unconditional on Linux)	+51% at the 100k cliff
per-shard checkpoint triggers	--wal-checkpoint-interval-ms + --wal-checkpoint-wal-bytes (retained-WAL/replay budget; shards self-stagger, in-flight guard, panic-safe)	checkpoint cost ≈ 0
--stream-lanes N	hash stream data files across per-device lane dirs; N parallel per-lane syncfs; lane count persisted + validated on open	1M wall broken: 68k → 212k (syncfs 60–74s → 6–11s)
cleanup	dead knobs removed (--wal-fsync-parallel, meta gates, meta-sweep toggles, --tier local); memory×tier rejected at startup; flag-clear WAL records decode as torn	smaller, safer surface

Results — canonical campaign 2026-07-14 (this PR's head, ideal configuration)

Write saturation (canonical-write, 256 B appends, c4d-standard-64-lssd, 3 stream lanes + 3 WAL lanes, pinned cores):

streams	wal-ideal	memory (no-fsync ceiling)
10k	417k	680k
100k	382k (−8%, no cliff)	632k

vs 10.4k @100k pre-campaign = 37×. At extreme cardinality (same config): 285k @500k, 212k @1m.

Reads (unchanged by this PR, revalidated): catch-up 2.27–2.74 GiB/s, cardinality-flat 10→100 streams; SSE tail p99 2–3 ms up to 2048 connections.

Mixed interference: 100k concurrent catch-up readers cost a pinned 50k ops/s write load nothing (303 MiB/s replay served alongside). The 2026-07-02 memory-mode delivery collapse no longer reproduces on this build: delivery tracks writes 1:1 to 127k/s (wal: keeps pace to ~40k, caps ~64k del/s at full write saturation).

Correctness

• The recoverability contract is unchanged: ack = WAL group-commit fdatasync; a WAL segment is recycled only after its records' stream bytes are fsynced AND the durable-tail map is persisted. All knobs only change when/how that sequence runs.
• 105/105 tests: syncfs crash-recovery e2e, 3-lane crash-recovery + layout-spread + lane-mismatch rejection, memory-mode sidecar-skip regression, randomized crash sim.
• CI green including all four conformance configs (memory, wal-default, wal-read-offload-always, wal-resident-cache).
• Defaults preserve behavior except syncfs-on-Linux (extensively measured); --stream-lanes defaults to 1 = byte-identical layout. Changeset: patch.

Follow-ups (tracked)

• durable-streams-rust: log-structured per-stream data store to eliminate O(N) checkpoint fdatasync #4695 log-structured store (removes per-file writeback amplification structurally — the residual 100k→1M slope)
• durable-streams-rust: use the cold tier as the durable bulk store (make local per-stream files a non-durable read cache) #4696 cold tier as durable store
• durable-streams-rust: lazy stream-file fd management (1 fd per live stream ≈ nofile ceiling at 1M streams) #4706 lazy fd management (1 fd/live stream = 96% of the 1,048,576 ceiling at 1M streams)
• feat(durable-streams-rust): --io-uring write paths (experimental, default off — v1 measured as a regression) #4709 io_uring experiment (measured as a regression in v1 form; default off, not for merge)

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk

[github-actions]

Auto-approved: all changes are within the durable streams package.

[balegas]

Update: cliff eliminated — layout + syncfs validated end-to-end (2026-07-13)

Two follow-up benchmark campaigns (wal-decomp-lane0, wal-splitlane on c4d-standard-64-lssd raw-block NVMe) complete the picture:

1. A storage confound inflated the original cliff. In all earlier multilane runs the stream data files (<data-dir>/streams/) sat on the PD boot disk (raw-block node pools leave the base emptyDir on PD) while only WAL segments got NVMe. The checkpoint was hammering a network disk.

2. With correct layout, this PR's syncfs checkpoint is cardinality-flat:

config @100k streams	peak ops/s
original (streams on PD)	10.4k
everything on ONE shared NVMe lane, syncfs @3s	46k
split-lane (streams on own lane, 5 WAL lanes), syncfs @3s	271.6k (286k @10k, −5%)
split-lane, checkpoint deferred 1h	306k
memory mode (no durability, ceiling)	512k

• Commit fdatasync on dedicated lanes costs ~nothing (ckpt-off ≥ the no-fsync control — group commit amortizes better under fsync backpressure).
• Checkpoint @3s costs 7–11% on the right layout; the cardinality cliff (−90%+) is gone (−5%).
• Full decomposition + production recipe in ds-bench/results/wal-splitlane/report.md; layout guidance added to ds-bench AGENTS.md.

Follow-up branch perf/wal-checkpoint-sizetrigger (stacked on this PR): --wal-checkpoint-interval-ms + --wal-checkpoint-wal-bytes — per-shard dual trigger that turns checkpoint cadence into an explicit replay-time budget and self-staggers shards. 56/56 wal tests pass; cluster validation pending.

[balegas]

Self-review pass (full-diff review) — findings and fixes, all pushed:

Blocking, fixed: the branch had collaterally deleted packages/durable-streams-rust/Dockerfile (bench image-build flow's copy/rm cycle); docker_multiarch_image.yml builds from that exact path. Restored verbatim from main.

Robustness, fixed:

• --stream-lanes layout mismatch was a silent data-unavailability footgun (open a 3-lane dir with --stream-lanes 1 → every stream invisible). Now persisted in streams/.lanes and validated at open, mirroring the WAL shard-count contract; mismatch = clean startup error; enabling lanes over an existing flat layout is also refused. e2e-covered.
• A panicking checkpoint task could permanently wedge its shard's in-flight guard (→ unbounded retained WAL). The ticker now maps task-id → shard and clears the guard on JoinError too.

Quality, fixed: unused import + never-constructed-variant warnings (0 warnings now), orphaned/tombstone doc comments, whitespace.

Noting explicitly (semantic change already in the diff): WAL records with PAYLOAD_CHECKSUMMED clear now decode as Torn (end of durable log) instead of "bytes present = complete". Safe — no released writer ever cleared the flag (only the long-removed splice relay did) — but it is a decode-behavior change for on-disk bytes.

105/105 tests pass.

[balegas]

The conformance flake (should wait for new data with long-poll, 1500ms timeout on the wal-default row) was investigated and turned out to be a real latent bug, not CI jitter — present on main too.

Mechanism: the long-poll timeout arm read the fresh tail and returned an empty 204 advertising it as Stream-Next-Offset without delivering the bytes — so an append whose durable-tail publish lands inside the deadline window gets silently skipped by the client forever. The conformance test sits exactly on this boundary by construction (500ms pre-append sleep == the suite's --long-poll-timeout-ms 500). The wake path itself is provably race-free (watch subscribe → borrow_and_update re-check); only the timeout arm violated the delivery invariant. In production (30s timeout) the same interleaving = silent data skip for a long-poll consumer.

Fix (pushed): the sleep arm re-checks the tail like the closed-channel arm and serves observed data. New regression test races 200 appends against a 20ms deadline asserting no response advances the offset without delivering data — it reproduces the bug in <100 iterations without the fix, passes with it. 106/106 tests.