perf(durable-streams-rust): eliminate the write cardinality cliff — syncfs checkpoints, per-shard triggers, stream lanes

.changeset/wal-cardinality-cliff.md

@@ -0,0 +1,13 @@
+---
+"@electric-ax/durable-streams-server-rust": patch
+---
+
+Eliminate the WAL write cardinality cliff (10.4k → 383k appends/s @100k streams; 212k @1M).
+
+- Checkpoint durability now uses one `syncfs` barrier per stream lane on Linux (was O(touched-streams) per-file `fdatasync` — the barrier storm that collapsed throughput at high stream counts).
+- New `--wal-checkpoint-interval-ms` (per-shard time trigger, default 3000) and `--wal-checkpoint-wal-bytes` (retained-WAL size budget, 0 = off): checkpoint cadence is an explicit crash-replay budget and shards self-stagger instead of storming together.
+- New `--stream-lanes N` (default 1 = unchanged layout): hash stream data files across `streams/<0..N>/` dirs, one per device, spreading checkpoint writeback over N devices with N parallel barriers. The lane count is persisted and validated on open.
+- New `--server-stats N` telemetry (`SRV_STATS`: cpu / inflight / service / lock / durability-wait per interval) — the dependency-free bottleneck diagnostics used to find all of the above. Memory-mode plain appends no longer queue redundant sidecar flushes.
+- Removed dead/diagnostic flags: `--wal-fsync-parallel`, `--wal-meta-gate`, `--mem-meta-gate`, `--meta-sweep-disable`, `--meta-sweep-stats`, `--tier local` / `--tier-local-dir` (tier is `off|s3`). `--durability memory` combined with `--tier` is now rejected at startup. WAL records without `PAYLOAD_CHECKSUMMED` decode as torn (no released writer ever emitted them).
+
+Deployment guidance (device layout, CPU pinning, checkpoint budgets): `WAL_TUNING.md`.

packages/durable-streams-rust/README.md

@@ -87,29 +87,28 @@ durable, single-node server on `127.0.0.1:4437` with its data dir under `$TMPDIR
 
 **Cold-storage tier** — off by default; see [Tiered storage](#tiered-storage-cold-offload). With `--tier off` the server is byte-identical to a single-file deployment.
 
-| Flag                                          | Default    | Description                                                                              |
-| --------------------------------------------- | ---------- | ---------------------------------------------------------------------------------------- |
-| `--tier`                                      | `off`      | `off` \| `local` (sealed segments to a local dir) \| `s3` (S3-compatible object storage) |
-| `--tier-local-dir`                            | —          | (`tier=local`) directory for sealed segments                                             |
-| `--tier-endpoint`                             | —          | (`tier=s3`) S3 endpoint URL                                                              |
-| `--tier-region`                               | —          | (`tier=s3`) region                                                                       |
-| `--tier-bucket`                               | —          | (`tier=s3`) bucket name                                                                  |
-| `--tier-key-prefix`                           | —          | object-key prefix for sealed segments                                                    |
-| `--tier-segment-bytes`                        | `8 MiB`    | sealed-segment size (fixed-size, CDN-friendly)                                           |
-| `--tier-compact-bytes`                        | `64 MiB`   | small-segment compaction threshold                                                       |
-| `--tier-path-style` / `--tier-virtual-hosted` | path-style | S3 addressing style                                                                      |
-| `--tier-allow-http`                           | off        | allow plain HTTP to the S3 endpoint (e.g. a local MinIO)                                 |
+| Flag                                          | Default    | Description                                              |
+| --------------------------------------------- | ---------- | -------------------------------------------------------- |
+| `--tier`                                      | `off`      | `off` \| `s3` (S3-compatible object storage)             |
+| `--tier-endpoint`                             | —          | (`tier=s3`) S3 endpoint URL                              |
+| `--tier-region`                               | —          | (`tier=s3`) region                                       |
+| `--tier-bucket`                               | —          | (`tier=s3`) bucket name                                  |
+| `--tier-key-prefix`                           | —          | object-key prefix for sealed segments                    |
+| `--tier-segment-bytes`                        | `8 MiB`    | sealed-segment size (fixed-size, CDN-friendly)           |
+| `--tier-compact-bytes`                        | `64 MiB`   | small-segment compaction threshold                       |
+| `--tier-path-style` / `--tier-virtual-hosted` | path-style | S3 addressing style                                      |
+| `--tier-allow-http`                           | off        | allow plain HTTP to the S3 endpoint (e.g. a local MinIO) |
 
 S3 credentials come from the **environment**, never flags: `DS_S3_ACCESS_KEY_ID` /
 `DS_S3_SECRET_ACCESS_KEY` (or the standard `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY`).
 
 ### Choosing a configuration
 
-| Your situation                                      | Use                                                                                                         |
-| --------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
-| **Bounded local disk with long history**            | `--tier s3` (or `local`) — seal cold segments to object storage; the recent tail stays local and zero-copy. |
-| **High fan-out p99 across many streams on Linux**   | try `--tail-cache-bytes 65536` (off by default there because `sendfile` already covers it).                 |
-| **Experimenting on macOS and writes take ~2–10 ms** | expected — see [macOS write latency](#macos-write-latency-f_fullfsync) below.                               |
+| Your situation                                      | Use                                                                                            |
+| --------------------------------------------------- | ---------------------------------------------------------------------------------------------- |
+| **Bounded local disk with long history**            | `--tier s3` — seal cold segments to object storage; the recent tail stays local and zero-copy. |
+| **High fan-out p99 across many streams on Linux**   | try `--tail-cache-bytes 65536` (off by default there because `sendfile` already covers it).    |
+| **Experimenting on macOS and writes take ~2–10 ms** | expected — see [macOS write latency](#macos-write-latency-f_fullfsync) below.                  |
 
 ### macOS write latency (`F_FULLFSYNC`)
 
@@ -179,7 +178,7 @@ Opt-in (`--tier`, off by default). Because streams are append-only and immutable
 
 - **S3-compatible, not AWS-only.** The `s3` backend works with any S3-compatible endpoint — Cloudflare R2, Fly/Tigris, MinIO, Backblaze B2 — via a configurable endpoint + path-style addressing. It's built on the `object_store` crate behind the **`tier` Cargo feature** (so a default build pulls no object-storage dependencies): `cargo build --release --features tier`. A `local` backend (sealed segments to a directory) needs no feature/deps and is handy for testing.
 - **How it stays correct.** Each stream keeps a manifest of its sealed segments. The lifecycle is seal → upload → `head`-verify → durably flip the manifest entry `local → remote` → only then unlink the staged chunk file (safe even under an in-flight read — Unix keeps an open fd readable after unlink). A read resolves each requested offset against the manifest — local ranges (live file or sealed chunk file) keep the zero-copy `sendfile` path; remote ranges are fetched by range-GET and streamed in. JSON seals always land on a value boundary (never inside a string). Durability is unchanged: an append still acks only after the local fsync — offload is strictly post-durability. Fully-sealed ranges are stamped `Cache-Control: immutable` for long-lived CDN caching, and remote objects are GC'd ref-count-aware with forks. The live data file's redundant sealed prefix is reclaimed by **compaction**: once it exceeds `--tier-compact-bytes` (default 64 MiB), the live file is rewritten to hold only the hot tail, under the appender lock and crash-safe via a `pending_compaction` intent. In-flight reads drain off the old fd, so reads stay lock-free — this is why compaction is used rather than `fallocate` hole-punching, which raced those reads. Set `--tier-compact-bytes 0` to disable.
-- **Flags:** `--tier {off|local|s3}`, `--tier-segment-bytes`, `--tier-compact-bytes` (live-file compaction threshold; `0` disables), `--tier-key-prefix`, `--tier-local-dir` (local), `--tier-endpoint` / `--tier-region` / `--tier-bucket`, `--tier-path-style` / `--tier-virtual-hosted`, `--tier-allow-http`. S3 credentials come from `DS_S3_ACCESS_KEY_ID` / `DS_S3_SECRET_ACCESS_KEY` (with `AWS_*` fallback), env only.
+- **Flags:** `--tier {off|s3}`, `--tier-segment-bytes`, `--tier-compact-bytes` (live-file compaction threshold; `0` disables), `--tier-key-prefix`, `--tier-endpoint` / `--tier-region` / `--tier-bucket`, `--tier-path-style` / `--tier-virtual-hosted`, `--tier-allow-http`. S3 credentials come from `DS_S3_ACCESS_KEY_ID` / `DS_S3_SECRET_ACCESS_KEY` (with `AWS_*` fallback), env only.
   Conformance passes with tiering on as well as off (verified manually with a small `--tier-segment-bytes` so streams seal and offload mid-suite; catch-up reads, ETag/304, closed-stream EOF, and forks are all served correctly from cold). Note this is a manual check, not a CI gate: the `rust-conformance` matrix in `.github/workflows/ci.yml` builds the default feature set (no `--features tier`) and never passes `--tier`, so the tier code path is not exercised in CI today.
 
 ## Observability (OpenTelemetry)

packages/durable-streams-rust/WAL_TUNING.md

@@ -0,0 +1,120 @@
+# WAL write-path tuning — the ideal configuration
+
+**Status:** validated 2026-07-13 on GCP `c4d-standard-64-lssd` (6× Titanium local
+NVMe, raw block), 256 B appends, 8 vCPU server, up to 100k streams. Campaign:
+ds-bench suites `wal-decomp-lane0`, `wal-splitlane`, `wal-sizetrigger`,
+`wal-cpubind` (see ds-bench `results/<suite>/report.md` + `AGENTS.md`).
+**Verdict: the write cardinality cliff is eliminated** — WAL-durable throughput
+is flat from 10k → 100k streams (−5%) at ~26–32× the pre-fix baseline.
+
+## The ladder (what each step bought, @100k streams)
+
+| configuration                                                             | ops/s    |
+| ------------------------------------------------------------------------- | -------- |
+| pre-fix: streams on network PD, per-stream checkpoint fdatasync storm     | 10.4k    |
+| + syncfs checkpoint barrier (PR #4697)                                    | 13.6k    |
+| + stream files on local NVMe (not the boot disk)                          | 46k      |
+| + split-lane layout (WAL shards on their own NVMe devices)                | 272k     |
+| + size-triggered checkpoint (PR #4704, `--wal-checkpoint-wal-bytes 1GiB`) | 303k     |
+| + exclusive pinned cores (Guaranteed QoS + static CPU manager)            | 328k     |
+| **all of the above stacked** (suite `wal-stacked-1m`)                     | **383k** |
+| reference: memory durability (no fsync anywhere)                          | 512k     |
+
+At extreme cardinality the stacked config on ONE data lane hits a second wall —
+checkpoint writeback (~40× metadata amplification of small appends) saturates
+the single data device (`syncfs` = 60–74 s at 1M streams): 244k @500k, 56–68k
+@1M. `--stream-lanes 3` (suite `wal-streamlanes-1m`, 3 data lanes + 3 WAL
+lanes) breaks it: **374k @100k, 285k @500k, 212k @1M** (`syncfs` 5.7–11 s).
+
+The residual ~1.6× gap to memory mode is WAL machinery (staging + double-write),
+not fsync — future work: io_uring segment writer (`wal/segment.rs` seam),
+batched mark-written.
+
+## The ideal configuration
+
+### 1. Hardware / storage layout (the #1 lever)
+
+Use an instance with **multiple physically attached NVMe devices** (GCP: 4th-gen
+`-lssd` types, raw block via `--local-nvme-ssd-block`; do NOT use
+`--ephemeral-storage-local-ssd`, which RAID0-stripes every device into one fsync
+barrier). Then:
+
+- **One device for stream data files** — mount it and point `--data-dir` at it.
+  The per-stream files and the checkpoint's `syncfs` domain live here.
+- **One device per WAL shard** — mount device _j_ at `<data-dir>/wal/<i>` (the
+  server opens shard _i_ at that path automatically). `--wal-shards` = number of
+  dedicated WAL devices.
+- **Never share a device between WAL and stream data.** Commit `fdatasync` vs
+  checkpoint writeback contention on one queue was worth 5× by itself
+  (55k → 272k). On dedicated lanes the commit-fsync cost is ~zero (checkpoint-off
+  measured _below_ an all-lanes-shared no-fsync control).
+- **Never leave stream data on the boot disk / network PD.** On Kubernetes
+  raw-block node pools the default emptyDir sits on the boot PD — this single
+  mistake mismeasures (and misdeploys) WAL mode by 5–26×.
+
+Split the box between data lanes and WAL lanes **by cardinality**:
+
+- ≤ ~100k streams: 1 data lane is enough — e.g. device 0 → data root,
+  devices 1–5 → 5 WAL shards: `--data-dir /data/wal/0 --wal-shards 5`
+- ≥ ~500k streams: checkpoint writeback dominates; give data more lanes — e.g.
+  device 0 → data root (stream lane 0), devices 1–2 → stream lanes 1–2
+  (mounted at `<data-dir>/streams/1`, `/2`), devices 3–5 → 3 WAL shards:
+  `--data-dir /data/wal/0 --wal-shards 3 --stream-lanes 3`
+  (`--stream-lanes` is a LAYOUT choice like the shard count: it must match the
+  on-disk layout across restarts.)
+
+### 2. Server flags
+
+```
+--wal-checkpoint-wal-bytes 1073741824 # checkpoint a shard when ITS retained WAL
+                                      # exceeds 1 GiB (PR #4704) — checkpoint cost ≈ 0,
+                                      # crash-replay bounded to ≤1 GiB/shard (<1 s NVMe)
+--wal-checkpoint-interval-ms 60000    # fallback timer so an idle shard still recycles
+--wal-shards <number of WAL devices>  # shards = fsync lanes; on a SINGLE shared
+                                      # device keep 2–4 (more only fragments batches)
+--stream-lanes <number of data devices> # hash stream files across per-device dirs
+                                      # (PR #4705); layout choice, default 1
+--worker-threads <vCPUs>
+```
+
+The syncfs checkpoint barrier (PR #4697) is unconditional on Linux — one
+`syncfs` per stream lane instead of O(N-touched) per-stream `fdatasync`.
+Non-Linux uses the serial per-file loop (no `syncfs` there).
+
+### 3. CPU binding (+21–24%)
+
+Give the server **exclusive pinned cores**. On Kubernetes: node pool with
+`kubeletConfig.cpuManagerPolicy: static` + a **Guaranteed QoS** pod (every
+container `requests == limits`, server CPU an integer). Measured 356k @10k /
+328k @100k vs 286k/272k on shared cores, same layout and flags. Now that WAL is
+no longer fsync-bound, it scales with cores again — don't starve it.
+
+### 4. What you do NOT need to worry about
+
+- **Read performance while tuning checkpoints.** Reads never touch the WAL and
+  are served zero-copy (`sendfile`) from the data file's page cache, which is
+  written _before_ the WAL ack barrier. Checkpoint cadence has zero read-path
+  cost.
+- **Ack latency vs checkpoints.** Acks gate only on the WAL group-commit
+  `fdatasync`; a checkpoint never blocks appends.
+- **Recoverability.** The contract is unchanged by any of these knobs: a WAL
+  segment is recycled only after its records' stream bytes are fsynced into
+  their files **and** the durable-tail map is persisted (`wal/shard.rs`
+  checkpoint ordering; crash-recovery e2e + randomized crash sim cover it).
+  The size trigger only changes _when_ that sequence runs.
+
+## Caveats / follow-ups
+
+- The 1M-stream writeback wall is diagnosed and broken (`--stream-lanes`, PR
+  #4705: 68k → 212k @1M). The residual slope (374k @100k → 212k @1M on 3 data
+  lanes) is per-file writeback amplification against total data-lane
+  capacity — add data lanes, or see #4695 (log-structured store) for the
+  structural end-state.
+- fd ceiling: the server holds one fd per live stream — 1,005,724 fds at 1M
+  streams = 96% of the default 1,048,576 limit. Not the throughput wall, but a
+  hard scale ceiling just above 1M: raise LimitNOFILE, or see #4706 (lazy fd
+  management).
+- The checkpoint size trigger is opt-in (`--wal-checkpoint-wal-bytes 0` default);
+  making ~1 GiB the default is a candidate after soak.
+- Larger retained WAL = longer replay: 1 GiB/shard ≈ sub-second on local NVMe,
+  but budget it consciously on slower disks.