fix(uring): harden cancel safety and degradation

CHANGELOG.md

@@ -23,7 +23,72 @@ aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ## [Unreleased]
 
-No unreleased changes.
+### Docs
+
+- Rewrote the README from the current public API — all three read entry points
+  (`read_at`, `read_at_direct`, `read_current`) and the degradation contract —
+  and removed the `docs/DESIGN.md` design notes. The per-invariant rationale now
+  lives in the module and function docs and in the README, so there is a single
+  source of truth.
+
+## [0.2.0] - 2026-07-11
+
+Hardening pass from the rustfs/backlog#1160 audit. All changes are on the read
+path and preserve the cancel-safety ownership model; both `run-docker.sh` legs
+(seccomp-blocked degradation and real io_uring) pass.
+
+### Added
+
+- **Test-only fault-injection seams** for the cancel-safety escape hatches
+  (driver-thread panic, stuck bounded drain, forced probe-drain failure), gated
+  behind the `fault-injection` feature and never present in a default build.
+  ([#11], backlog#1103)
+- `StatsSnapshot::submit_errors` — count of non-transient `ring.submit()`
+  failures, so a persistently failing `io_uring_enter` is observable.
+  (backlog#1162)
+
+### Fixed
+
+- **Bounded-drain bailout no longer hangs awaited handles.** It now fails every
+  stranded caller with a driver-gone error before leaking the pending table, and
+  leaks the ring-registered eventfd alongside the ring so the "cq_efd outlives
+  the ring" invariant holds on that exit too. (backlog#1161, #1167)
+- **Persistent `ring.submit()` errors are classified** instead of retried
+  forever in silence: EINTR/EBUSY stay transient; any other errno is counted and,
+  after a bounded run, shuts the shard down so callers fall back. (backlog#1162)
+- **Wakeup gaps closed:** short-read resubmits are flushed the same turn, and a
+  drop-cancel now signals the wake eventfd (after closing the receiver so the
+  reclaim is still counted as an orphan), removing up-to-50 ms stalls.
+  (backlog#1163)
+- **Driver-thread spawn failure degrades** to a `ProbeFailure` instead of
+  panicking out of disk init. (backlog#1164)
+- **The startup probe is time-bounded** (EXT_ARG timeout with a pre-5.11
+  fallback) and runs on the first shard only. (backlog#1165)
+- **EINTR/EAGAIN completions are retried** (bounded) rather than surfaced as the
+  read's final error, and the submit offset guard also rejects an aligned end
+  crossing `i64::MAX`. (backlog#1166)
+- **CQ overflow is reported as a NODROP backpressure warning**, not fatal loss,
+  and AsyncCancel SQEs are deduplicated per op. (backlog#1167)
+- **O_DIRECT tail short reads are disambiguated with `fstat`** instead of assuming
+  any non-block-multiple read is EOF, so a stacked filesystem cannot cause a
+  silent truncation. (backlog#1168)
+- **Idle churn cut**: the loop skips `io_uring_enter` on an empty SQ and uses a
+  longer heartbeat when idle; each ring caps its io-wq bounded workers.
+  (backlog#1169)
+
+### Changed
+
+- **Internal driver-loop refactor** folding the duplicated submit,
+  cancel-enqueue, and resubmit-SQE paths into single helpers (`submit_ring`,
+  `queue_cancel`, `Pending::resubmit_sqe`, `flush_backlog`). No behavior change;
+  the perf-sensitive submit → reap → conditional re-flush order is kept so a
+  cache-hit read is still reaped the same turn it completes inline.
+  (backlog#1160)
+
+### Tests
+
+- Deterministic sharded cancel-routing test proving a drop-cancel reaches the
+  ring that accepted the op. (backlog#1180)
 
 ## [0.1.0] - 2026-07-11
 
@@ -116,7 +181,7 @@ polished (`39018c0`).
 The cancel-safety invariants that survived that audit — the driver's pending table owning the
 buffer and fd from SQE submission until the CQE, drop-abandons-result-only, bounded shutdown
 drain, abort-before-free on a driver panic — are the ones every change above is measured
-against. See the [design notes](https://github.com/rustfs/uring/blob/v0.1.0/docs/DESIGN.md).
+against. See the [design notes](https://github.com/rustfs/uring/blob/0.1.0/docs/DESIGN.md).
 
 ## Decisions recorded, not implemented
 
@@ -145,8 +210,10 @@ built. They are listed so nobody re-opens them without new evidence.
 [#4]: https://github.com/rustfs/uring/pull/4
 [#5]: https://github.com/rustfs/uring/pull/5
 [#6]: https://github.com/rustfs/uring/pull/6
+[#11]: https://github.com/rustfs/uring/pull/11
 [rustfs/backlog#1051]: https://github.com/rustfs/backlog/issues/1051
 [rustfs/backlog#1144]: https://github.com/rustfs/backlog/issues/1144
 [rustfs/backlog#1159]: https://github.com/rustfs/backlog/issues/1159
-[Unreleased]: https://github.com/rustfs/uring/compare/v0.1.0...HEAD
-[0.1.0]: https://github.com/rustfs/uring/releases/tag/v0.1.0
+[Unreleased]: https://github.com/rustfs/uring/compare/0.2.0...HEAD
+[0.2.0]: https://github.com/rustfs/uring/compare/0.1.0...0.2.0
+[0.1.0]: https://github.com/rustfs/uring/releases/tag/0.1.0

Cargo.toml

@@ -14,7 +14,7 @@
 
 [package]
 name = "rustfs-uring"
-version = "0.1.0"
+version = "0.2.0"
 edition = "2024"
 rust-version = "1.96"
 license = "Apache-2.0"
@@ -25,7 +25,6 @@ readme = "README.md"
 homepage = "https://rustfs.com"
 keywords = ["io_uring", "async", "storage", "rustfs", "linux"]
 categories = ["asynchronous", "filesystem"]
-exclude = ["docs/**"]
 
 [features]
 # Test-only fault-injection seams (rustfs/backlog#1103). OFF by default and in

README.md

@@ -7,40 +7,49 @@
 
 Cancel-safe async `io_uring` read backend for [RustFS](https://github.com/rustfs/rustfs) storage.
 
-This crate is the io_uring integration that RustFS's read path is built on. It started as the Spike 0 cancel-safety
-prototype ([rustfs/backlog#894](https://github.com/rustfs/backlog/issues/894)) and was hardened per
-the [#1048/#1051 audit](https://github.com/rustfs/backlog/issues/1051). It lives in its own repository so it can be
+This crate is the io_uring integration that RustFS's read path is built on. It lives in its own repository so it can be
 verified in isolation — with a real io_uring CI leg that the main `rustfs/rustfs` workspace cannot run — before being
-wired into the storage layer.
+wired into the storage layer behind a runtime probe.
 
-> **Status:** read path only, Linux only. The read path is wired into `rustfs/rustfs` behind a runtime probe and is
-> **off by default** (`RUSTFS_IO_URING_READ_ENABLE`). See [`CHANGELOG.md`](CHANGELOG.md) for what has landed and the
-> [design notes](https://github.com/rustfs/uring/blob/v0.1.0/docs/DESIGN.md) for the invariants.
+> **Status:** read path only, Linux only. On any other target the crate compiles to an empty stub. The read path is
+> wired into `rustfs/rustfs` behind a runtime probe and is **off by default** (`RUSTFS_IO_URING_READ_ENABLE`). What has
+> landed is in [`CHANGELOG.md`](CHANGELOG.md); the per-invariant rationale lives in the module and function docs on
+> [docs.rs](https://docs.rs/rustfs-uring/) and inline in `src/`.
 
 ```toml
 [target.'cfg(target_os = "linux")'.dependencies]
-rustfs-uring = "0.1.0"
+rustfs-uring = "0.2.0"
 ```
 
 ## The ownership model it enforces
 
 When a caller drops the future of an in-flight read (EC quorum reached, timeout, disconnect), the kernel may still write
 into the read buffer at any point until the CQE. Freeing the buffer at future-drop is a use-after-free. This crate
-proves and enforces the invariants any production io_uring integration must follow:
+enforces the invariants any production io_uring integration must follow:
 
 - **The buffer and the file handle are owned by the driver's pending (orphan) table** from SQE submission until the
   CQE — never by the caller's future.
-- **Dropping the future abandons only the result**; reclamation always happens at the CQE (optionally accelerated by
-  `IORING_OP_ASYNC_CANCEL`).
-- **Shutdown drains in-flight ops to zero** (with a bounded escape hatch for hung disks) before the ring is unmapped.
-- A driver-thread panic **aborts before freeing in-flight buffers** (leak over UAF); backpressure permits are released
-  at the CQE, not at future drop; short reads on positioned reads are resubmitted to satisfy the whole-range contract;
+- **Dropping the future abandons only the result**; reclamation always happens at the CQE, optionally accelerated by an
+  `IORING_OP_ASYNC_CANCEL` sent on drop.
+- **Shutdown drains in-flight ops to zero** before the ring is unmapped, with a bounded escape hatch for a hung disk:
+  on timeout it fails the stranded callers with an error and *leaks* the ring and its buffers rather than free memory
+  the kernel may still touch (leak over UAF).
+- A driver-thread panic **aborts the process before freeing in-flight buffers**; backpressure permits are released at
+  the CQE, not at future drop; a short read on a positioned read is resubmitted to satisfy the whole-range contract;
   the probe file is opened via `O_TMPFILE`.
 
-Each invariant holds **per shard** (see below), because a shard is an independent instance of the same driver.
+It also has a **degradation contract** — the reason it can ship default-on-probe safely:
 
-The full invariant list, the corrected fd-reuse mechanism, and the design constraints are in
-the [design notes](https://github.com/rustfs/uring/blob/v0.1.0/docs/DESIGN.md).
+- The startup probe runs a real `IORING_OP_READ` under a wall-clock timeout. A restricted environment
+  (seccomp / gVisor / old kernel) fails with a `ProbeFailure` whose `is_expected_restriction()` tells the caller to
+  degrade to the std backend quietly; an unexpected failure is surfaced, not hidden.
+- A persistently failing `io_uring_enter` is classified (not retried forever in silence): the shard shuts down so
+  callers fall back, and the failure count is visible in `StatsSnapshot`.
+- Transient CQE errnos (`EINTR`/`EAGAIN`) are retried; the whole-range and EOF contracts are honoured even on stacked
+  filesystems by disambiguating a short read with `fstat` rather than assuming EOF.
+
+Each invariant holds **per shard** (see below), because a shard is an independent instance of the same driver. Every
+one is pinned by an acceptance test (see [Testing](#testing)).
 
 ## Usage
 
@@ -50,9 +59,8 @@ use std::sync::Arc;
 use rustfs_uring::UringDriver;
 
 # async fn demo() -> std::io::Result<()> {
-    // Probe a real IORING_OP_READ before accepting work. A restricted environment
-    // (seccomp/gVisor/old kernel) returns a ProbeFailure whose
-    // `is_expected_restriction()` tells you to degrade to the std backend quietly.
+    // Probe a real IORING_OP_READ before accepting work. On a restricted host the
+    // ProbeFailure's `is_expected_restriction()` tells you to degrade quietly.
     let driver = UringDriver::probe_and_start(64).expect("io_uring available");
 
     let file = Arc::new(File::open("/data/object")?);
@@ -70,6 +78,17 @@ use rustfs_uring::UringDriver;
 }
 ```
 
+Three read entry points, all returning an awaitable `ReadHandle`:
+
+- `read_at(file, offset, len)` — positioned (pread) read, whole-range: short reads are resubmitted until the range is
+  satisfied or a real EOF.
+- `read_at_direct(file, offset, len, align)` — the same, for an fd opened with `O_DIRECT` (see below).
+- `read_current(file, len)` — reads from the fd's current position with `read(2)` semantics: a short read is a valid
+  final result and is *not* resubmitted. Use it for pipes and other non-seekable fds.
+
+`ReadHandle::without_cancel_on_drop()` opts a handle out of the drop-time `ASYNC_CANCEL` when you know the op will
+complete on its own; and `driver.stats()` returns a `StatsSnapshot` at any time.
+
 ### Sharded rings
 
 A buffered read that hits the page cache completes *inline* inside `io_uring_enter`, so the thread driving a ring
@@ -135,15 +154,19 @@ cargo test -- --nocapture --test-threads=1
 
 # Two legs in Docker (also runs on macOS via Docker Desktop / OrbStack):
 #   leg 1 — io_uring blocked by an explicit seccomp profile → the suite MUST
-#           degrade to a graceful skip (reproduces the #4313 restricted env);
+#           degrade to a graceful skip (reproduces a restricted environment);
 #   leg 2 — seccomp=unconfined → real io_uring, and NO test may skip.
 ./run-docker.sh
 ```
 
 The harness fails on either a non-degrading leg 1 or a vacuous-pass leg 2, so a skipped suite can never masquerade as
-real coverage. The 15 acceptance tests are the cancel-safety contract: buffer conservation under a mixed
-drop/keep stress across shards, an orphaned op reclaimed only at its CQE, bounded shutdown drain, `O_DIRECT` returning
-exact unaligned ranges, and backpressure deferring rather than blocking a runtime worker.
+real coverage. The cancel-safety contract is pinned by 16 acceptance tests in `tests/cancel.rs` — buffer conservation
+under a mixed drop/keep stress across shards, an orphaned op reclaimed only at its CQE, sharded cancel routed to the
+ring that owns the op, bounded shutdown drain, `O_DIRECT` returning exact unaligned ranges, and backpressure deferring
+rather than blocking a runtime worker. Five deterministic fault-injection tests in `tests/fault_injection.rs` (behind
+the test-only `fault-injection` feature, never compiled into a release build) drive the escape hatches: the
+panic-abort barrier, the bounded-drain leak-and-error path, a forced probe-drain failure, and a driver-thread spawn
+failure that must degrade rather than panic.
 
 ## Benchmarks
 
@@ -168,10 +191,9 @@ alone cannot tell a correct strategy from one reading the wrong offsets.
 - **`SQPOLL`.** Eliminates `io_uring_enter` under sustained load, at the cost of a kernel polling thread per ring —
   which multiplies by shards and by disks. Only for high-end deployments.
 
-Closed by measurement, not built — see [`CHANGELOG.md`](CHANGELOG.md#decisions-recorded-not-implemented): streaming
-reads through io_uring (NO-GO), `AsyncFd` reaping without a driver thread (would break the public API), a process-wide
-singleton ring (conflicts with per-disk isolation), and registered buffers (conflicts with the `Vec<u8>` ownership
-model, and the bottleneck is elsewhere).
+Closed by measurement, not built (see the CHANGELOG): streaming reads through io_uring (NO-GO), `AsyncFd` reaping
+without a driver thread (would break the public API), a process-wide singleton ring (conflicts with per-disk
+isolation), and registered buffers (conflicts with the `Vec<u8>` ownership model, and the bottleneck is elsewhere).
 
 ## License