fix(durable-streams-rust): recovery hardening — fail-stop barriers, crash-safe metadata, rollback-on-error

[balegas]

Deep failure-mode & recovery review of the WAL/store/tier stack (four focused audits: durability-barrier errors, recovery/boot robustness, runtime failures, cold tier), with every accepted finding fixed. No instrumentation in this PR — fault-injection tooling used to validate these paths lives on a separate disposable branch.

The central design correction: durability barriers are now fail-stop

Every barrier (committer fdatasync, checkpoint syncfs/per-file fsync, segment seal_to) was retryable-in-place. On Linux ≥4.13 a failed fsync marks the dirty pages clean and clears the fd's error state — the retry falsely succeeds, so:

• the committer would ack bytes that never reached disk (fsyncgate),
• the next checkpoint would recycle the WAL records that were the only durable copy,
• a retried seal could graft a zero tail that silently ends replay, discarding every later segment's acked records.

All barrier failures now abort the process. This is provably safe: nothing at risk was acked at first-error time (acks gate on the barrier), so restart replay is strictly better than the false-success path. The committer thread is also supervised — a panic aborts instead of freezing durable_lsn (which silently hung every append until connection-permit exhaustion = total DoS).

Data-loss fixes (each with the exact fault sequence in the commit message)

fix	previously
checkpoint error re-registers the drained dirty set	one transient checkpoint error + a later clean restart truncated acked, durable bytes (tails proofs dropped, WAL recycled anyway)
sidecar tmp always synced before rename; corrupt sidecars quarantined (.meta.corrupt), never deleted; sidecar read errors fail the boot	a crash-torn sidecar got the sidecar AND the whole data file deleted at boot — unrecoverable (stream identity is sidecar-only)
per-lane .lane mount markers; boot refuses a missing lane mount	an absent device mount booted "successfully", every stream on the lane vanished, and the WAL reset destroyed their acked records
recovery data-file open errors fail the boot	EMFILE silently skipped streams; their WAL records were skipped and then destroyed by the reset
dir fsyncs: segment create/roll/reset_after_recovery, tails rename before recycle; wal/shards + .lanes tmp+sync+rename+dir-fsync	lost dirents / same-name inode clobber / re-derived shard count silently skipping whole shard dirs
append rollback on stage failure (bytes + tail + producer/seq/close state); write_wire truncates partial writes	500'd bytes were durably resurrected by the next append; the client's retry was swallowed as a producer duplicate; ENOSPC partial writes desynced offsets for all later data
create undoes map entry + file + parent refcount on meta-write failure	phantom streams accepted (and durably acked) appends the next boot deleted
recovered compaction intents durably cleared before appends	double-crash re-derivation mis-set file_base, shifting every later read and replay write

Tier fixes

• Seal cuts at the durable frontier, not the writer tail — sealing un-acked bytes made them served-as-durable after a crash and permanently shadowed the retried acked bytes.
• Boot re-offload unlinks the local chunk only after the manifest flip is durable (was: unlink unconditionally → segment permanently unreadable).
• An unreadable sealed chunk poisons the read (new ResolvedSlice::Missing) — previously the slice was silently omitted and a well-formed 200 was served with interior bytes missing, advancing the client past them forever.
• Meta::capture snapshots segments + sealed_offset under one manifest lock (torn captures persisted manifest holes below the watermark).

Verified sound by the review (no changes needed)

Compaction single-crash matrix; torn-tail replay semantics for acked data; checkpoint-file optionality; steady-state offload ordering; fork/GC refcounting; accept-loop EMFILE backoff.

Deferred (tracked as follow-ups, availability/operational class)

Offload retry sweep + tier telemetry; mutex-poison strategy overhaul (~150 .lock().unwrap() sites — the fail-stop supervision covers the worst blast radius); wait_durable deadline; ENOSPC recycle-before-tails ordering; SSE reactor thread supervision.

Verification

109/109 tests, including 3 new e2e regressions: stage-failure rollback (bytes must not resurrect live or across recovery), corrupt-sidecar quarantine, missing-lane-mount boot refusal. The stage-failure injection hook moved pre-reservation to model the now-reachable semantics (post-reservation write failures retry bounded then abort). A follow-up fault-injection campaign (separate branch) exercises the fsyncgate/torn-write/ENOSPC paths that deterministic tests cannot.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Y3x7bcT9vLGiT4tXZeQpnk

[github-actions]

Auto-approved: all changes are within the durable streams package.

[balegas]

Fault-injection validation

The recovery-hardening fixes in this PR were validated against real I/O faults on a separate, disposable branch — fault/recovery-injection — which carries no instrumentation into this PR (this PR is the clean fixes only).

Approach: in-process failpoints at the six durability-critical seams (wal-write, wal-fsync, ckpt-barrier, seal, data-write, tails-write), driven by DS_FAULT="<site>:<behavior>[:N]". Because the fail-stop paths call std::process::abort() for real, each scenario runs its workload in a re-exec'd child process; the parent asserts the abort, then recovers the child's data dir in-process and checks the durability contract — every acked byte survives (prefix match; staged-but-unacked bytes may also survive, since an abort is not a power loss).

5 scenarios, all green:

Fault	Reaction validated	Outcome
committer fdatasync EIO	fail-stop abort() (F1 — no false-success retry)	acked records survive recovery
checkpoint barrier EIO	fail-stop abort() (F3)	acked records survive
seal-at-roll EIO	fail-stop abort() (F5)	acked records survive
tails-write EIO	plain error → dirty-set re-registration (F2)	checkpoint #2 re-covers the drained set; recovery intact
short (partial) data write	rollback-on-error (F7/1.2)	recovered bytes == acked exactly, no torn record

Alternatives (tikv fail-rs, LD_PRELOAD, dm-flakey/dm-error, FUSE) are compared in the branch's fault.rs module doc; in-process failpoints won for per-call-site determinism + portability. dm-flakey remains the right tool for a later full-system soak.

The branch is intentionally not for merge — it exists only as a record of the validation.

[balegas]

Performance validation — no write regression

Built a :dev image from this PR head (abca3a5f7, digest 93803a261706) and re-ran the two write/durability canonical ds-bench suites against the #4697 baseline. This PR's added durability barriers — fail-stop abort() on unretryable fsync/barrier failures, always-sync sidecar tmp writes, and extra parent-dir fsyncs after WAL renames — all land on the write path, so that's where a regression would surface.

Result: no measurable cost.

canonical-write (peak append/s, 256 B; Δ vs #4697):

streams	wal-ideal	memory
10k	413k (−1.0%)	669k (−1.6%)
100k	386k (+1.0%)	628k (−0.5%)

• No cardinality cliff (wal −6.5%, memory −6.1% across 10k→100k); regression gates cleared wide.
• memory is flat (no fsync barrier for the hardening to touch); wal is within ±1%. The two move exactly as the code predicts.

canonical-mixed-delivery (SSE delivery under a write ladder): memory delivery still tracks writes 1:1 to 140k/s (the 2026-07-02 collapse stays fixed); wal caps at ~64k del/s at full write saturation — same shape as #4697.

Full data, split provenance, and per-suite grids: electric-sql/ds-bench#7. Both suites self-tore-down; zero GKE clusters at run end.