perf(durable-streams-rust): --stream-lanes — shard stream data files across per-device lane dirs#4705
Merged
Conversation
…cross per-device lane dirs The ~1M-stream wall (wal-1m-diag): with every stream file on ONE data device, a checkpoint's syncfs writeback of ~200k dirty files took 60-74s per shard — tiny appends to distinct files amplify to ~8-12KB of data+inode+journal writeback each, saturating the single device and starving append staging (batch_avg collapsed 53->2-6). --stream-lanes N hashes stream files across streams/<0..N>/ subdirs, one per device: N× writeback capacity, N parallel syncfs barriers (one per lane, threaded), and no single ext4 dir holds every stream. Default 1 = byte-identical historical layout. Recovery walks all lanes; e2e covers lanes=3 crash recovery + layout spread. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Base automatically changed from
perf/wal-checkpoint-sizetrigger
to
perf/wal-checkpoint-syncfs
July 13, 2026 21:54
balegas
added a commit
that referenced
this pull request
Jul 14, 2026
…yncfs checkpoints, per-shard triggers, stream lanes (#4697) Eliminates the WAL write cardinality cliff end-to-end. Formerly a 4-PR stack (#4704, #4705, #4707 folded in here). Full deployment guidance: `WAL_TUNING.md` (in this PR). **Validated by the canonical benchmark campaign 2026-07-14** (ds-bench#6: `results/REPORT.md` + `PROVENANCE.md`) run against this PR's head. ## The problem WAL-durable write throughput collapsed with stream count: **10.4k ops/s @100k streams** (vs ~110k @10k), degrading further under load. Two compounding causes: 1. The checkpoint issued **one `fdatasync` per touched stream** — an O(N) barrier storm sharing the device with commit fsyncs. 2. At extreme cardinality (~1M), checkpoint **writeback amplification** (~40× for small appends: dirty page + inode + extents + journal per file) saturated the single data device — one `syncfs` measured at 60–74 s. ## The changes | commit | what | measured effect | |---|---|---| | syncfs checkpoint | one `syncfs` barrier per stream lane instead of O(N-touched) `fdatasync` (unconditional on Linux) | +51% at the 100k cliff | | per-shard checkpoint triggers | `--wal-checkpoint-interval-ms` + `--wal-checkpoint-wal-bytes` (retained-WAL/replay budget; shards self-stagger, in-flight guard, panic-safe) | checkpoint cost ≈ 0 | | `--stream-lanes N` | hash stream data files across per-device lane dirs; N parallel per-lane syncfs; lane count persisted + validated on open | 1M wall broken: 68k → **212k** (syncfs 60–74s → 6–11s) | | cleanup | dead knobs removed (`--wal-fsync-parallel`, meta gates, meta-sweep toggles, `--tier local`); memory×tier rejected at startup; flag-clear WAL records decode as torn | smaller, safer surface | ## Results — canonical campaign 2026-07-14 (this PR's head, ideal configuration) **Write saturation** (`canonical-write`, 256 B appends, c4d-standard-64-lssd, 3 stream lanes + 3 WAL lanes, pinned cores): | streams | wal-ideal | memory (no-fsync ceiling) | |---|---|---| | 10k | 417k | 680k | | 100k | **382k (−8%, no cliff)** | 632k | vs 10.4k @100k pre-campaign = **37×**. At extreme cardinality (same config): 285k @500k, 212k @1m. **Reads** (unchanged by this PR, revalidated): catch-up 2.27–2.74 GiB/s, cardinality-flat 10→100 streams; SSE tail p99 2–3 ms up to 2048 connections. **Mixed interference**: 100k concurrent catch-up readers cost a pinned 50k ops/s write load nothing (303 MiB/s replay served alongside). The 2026-07-02 memory-mode delivery collapse no longer reproduces on this build: delivery tracks writes 1:1 to 127k/s (wal: keeps pace to ~40k, caps ~64k del/s at full write saturation). ## Correctness - The recoverability contract is unchanged: ack = WAL group-commit fdatasync; a WAL segment is recycled only after its records' stream bytes are fsynced AND the durable-tail map is persisted. All knobs only change *when/how* that sequence runs. - 105/105 tests: syncfs crash-recovery e2e, 3-lane crash-recovery + layout-spread + lane-mismatch rejection, memory-mode sidecar-skip regression, randomized crash sim. - CI green including all four conformance configs (`memory`, `wal-default`, `wal-read-offload-always`, `wal-resident-cache`). - Defaults preserve behavior except syncfs-on-Linux (extensively measured); `--stream-lanes` defaults to 1 = byte-identical layout. Changeset: patch. ## Follow-ups (tracked) - #4695 log-structured store (removes per-file writeback amplification structurally — the residual 100k→1M slope) - #4696 cold tier as durable store - #4706 lazy fd management (1 fd/live stream = 96% of the 1,048,576 ceiling at 1M streams) - #4709 io_uring experiment (measured as a regression in v1 form; default off, not for merge) 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #4704 (which stacks on #4697). Breaks the ~1M-stream writeback wall.
The wall (wal-1m-diag, 2026-07-13)
With every stream data file on ONE device, a checkpoint must write back ~200k dirty files per shard — and a 256 B append to a distinct file amplifies to ~8–12 KB of writeback (dirty page + inode + extent + ext4 journal, ~40×). Measured: one
syncfs= 60–74 s (WAL_CKPT fsync_us=73,446,860), the data device permanently saturated, append staging starved (group-commitbatch_avgcollapsed 53–88 → 2–6), throughput 68k @1m streams. Meanwhile the WAL lanes were loafing (33k fsync/s).The fix
--stream-lanes N(default 1 = byte-identical historical layout): stream data files hash (FNV-1a of the file name — stable, recomputable anywhere) acrossstreams/<0..N>/subdirs, one per device. N× writeback capacity, N parallel per-lanesyncfsbarriers (one call is per-filesystem), and no single ext4 dir holds every stream. A layout choice, not a runtime tunable — must match the on-disk layout across restarts, like the WAL shard count.Results (c4d-standard-64-lssd; 3 stream lanes + 3 WAL shards vs 1 data lane + 5 WAL shards, same image family/QoS; suite
wal-streamlanes-1m)Checkpoint
syncfs: 60–74 s → 5.7–11.2 s, with more touched files per shard.Verification
e2e_stream_lanes_recover_acked_records): 3-lane store, checkpoint + recycle, crash, byte-identical recovery + layout-spread assertion.🤖 Generated with Claude Code
https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk