fix: gate Linux-only benchmark examples

CHANGELOG.md

@@ -0,0 +1,153 @@
+<!--
+Copyright 2024 RustFS Team
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Changelog
+
+All notable changes to `rustfs-uring` are documented here.
+
+The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project
+aims to follow [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+The crate is **not published to crates.io** (`publish = false`); consumers pin a git revision.
+Because of that, entries are grouped under `[Unreleased]` until the first published release,
+and each one names the commit that landed it so a pin can be matched to a feature.
+
+## [Unreleased]
+
+Everything below is in `main` and reachable by pinning the listed revision.
+
+### Added
+
+- **Sharded rings** — `UringDriver::probe_and_start_sharded(entries, shards)` runs `shards`
+  independent rings, each with its own driver thread, pending table, backpressure semaphore,
+  and eventfd. `probe_and_start(entries)` is unchanged and equals `..._sharded(entries, 1)`.
+  ([#6], `719b245`)
+
+  A buffered read that hits the page cache completes *inline* inside `io_uring_enter`, so the
+  thread driving a ring performs that read's `memcpy`. A single-ring driver is therefore capped
+  at one core's memory bandwidth. Measured on a 16-core host: one thread pinned at 100% CPU,
+  throughput flat at ~5 GB/s regardless of read size, while a blocking-pool baseline reached
+  50 GB/s. Independent rings scaled near-linearly (1/2/4 rings → 4890/8969/15806 MB/s), which is
+  what identified the driver thread as the ceiling. With 8 shards, 1 MiB reads went from
+  4911 MB/s to 47361 MB/s, and 64 KiB reads at concurrency 32 from 124k to 345k IOPS — while
+  *keeping* io_uring's tail-latency advantage rather than trading it away.
+
+  In-flight ops are capped at `entries` **per shard**, so the invariant that makes CQ overflow
+  structurally unreachable holds per ring; the driver admits up to `shards * entries`
+  concurrent reads. Every cancel-safety invariant holds per shard: a `ReadHandle` carries the
+  `tx`/`wake` of the shard that accepted it, so a cancel or a deferred submission necessarily
+  routes back to the ring whose pending table holds the op.
+
+- **Native aligned `O_DIRECT` positioned read** — `UringDriver::read_at_direct(file, offset, len, align)`.
+  The caller opens the fd with `O_DIRECT` and passes the device's logical block size; `offset`
+  and `len` need **not** be aligned. The driver reads a block-aligned superset range into a
+  block-aligned buffer and hands back exactly `[offset, offset + len)`. ([#3], `f577ba0`)
+
+  Alignment padding, the bytes preceding the range, and the block-aligned tail never escape — a
+  `BitrotReader` expecting an exact shard length would flag padded output as corruption. The
+  buffer remains a `Vec<u8>`, over-allocated by `align - 1` with the read region starting at the
+  first aligned byte inside the allocation, so the cancel-safety ownership model is untouched.
+  Buffered reads are the degenerate `align == 1` case of the same geometry.
+
+- **Benchmark harnesses** (`examples/` + runner scripts), both hardened so they cannot pass
+  vacuously: files are created only when absent (`create_new` + `O_NOFOLLOW`, never truncating a
+  caller-supplied path even though the sweeps run as root), content is offset-addressable, and
+  `BENCH_VERIFY=1` checks every delivered byte at its claimed offset.
+
+  - `streaming_bench` + `bench-streaming.sh` — sequential whole-file read across
+    `std_buffered` / `std_odirect` / pipelined `uring_read_at` / `uring_read_at_direct`, in warm
+    and cold page-cache legs. ([#4], `4299b92`)
+  - `concurrent_pread_bench` + `bench-concurrent-pread.sh` — many concurrent positioned reads on
+    one disk, the shape erasure-coded shard reads actually serve. Four strategies isolate the
+    cost of the per-read `open` and of the `spawn_blocking` hop. ([#5], `d03977a`)
+
+### Changed
+
+- **Async backpressure** — the driver's synchronous backpressure gate became a tokio
+  `Semaphore`. `submit` takes a permit with `try_acquire_owned()` on the fast path and, when
+  saturated, hands the acquire future to the `ReadHandle` to await on its first poll instead of
+  parking the calling runtime worker. ([#2], `2b7cae4`)
+
+  The permit is stored in the pending entry, so it is released exactly when that entry is
+  dropped at the final CQE — never at future drop. Public API is unchanged: `read_at`'s
+  signature and its "in flight the moment it returns" behaviour hold on the fast path.
+
+- **eventfd-driven reaping** — the 200 µs busy-poll loop was replaced by a `poll(2)` on two
+  eventfds: one registered with the ring so the kernel signals every CQE, one signalled by
+  `submit`/shutdown so a new message wakes the loop immediately. A bounded heartbeat still runs
+  a loop turn so the drain deadline is checked and the NODROP overflow list is flushed.
+  ([#1], `ea6b920`)
+
+### Fixed
+
+- `aligned_geometry` rejects an `align` above the kernel read cap, foreclosing the
+  `align_offset == usize::MAX` path that would have made the driver's pointer arithmetic
+  undefined, and the `region_len + align - 1` allocation overflow. The driver's aligned-buffer
+  setup replaced a release-mode-only `debug_assert!` with a checked allocation size and a
+  runtime guard: an unsatisfiable alignment now fails the read instead of doing UB pointer
+  arithmetic. ([#3], `f577ba0`)
+- `deliver()` skips the full-buffer `copy_within` on the buffered path (`align == 1`, where the
+  logical range already starts at byte 0), so a buffered read no longer pays a per-read
+  `memmove`. ([#3], `f577ba0`)
+- The fast-path `submit` now surfaces an explicit driver-gone error through the result channel
+  when the driver has exited, instead of letting the caller infer one from a dropped oneshot.
+  ([#2], `2b7cae4`)
+
+## Pre-history
+
+Before this repository existed, the code was the Spike 0 cancel-safety prototype in
+`rustfs/rustfs` under `experiments/`. It was audited line by line
+([rustfs/backlog#1051]) and every confirmed finding was remediated there. The library was then
+promoted into this repository with its history preserved (`60993c9`), given a real-io_uring CI
+leg that the main workspace cannot run (`d5b28fc`), and had its crate metadata and README
+polished (`39018c0`).
+
+The cancel-safety invariants that survived that audit — the driver's pending table owning the
+buffer and fd from SQE submission until the CQE, drop-abandons-result-only, bounded shutdown
+drain, abort-before-free on a driver panic — are the ones every change above is measured
+against. See [`docs/DESIGN.md`](docs/DESIGN.md).
+
+## Decisions recorded, not implemented
+
+These were on the roadmap and were **closed by measurement or by design conflict** rather than
+built. They are listed so nobody re-opens them without new evidence.
+
+- **Three read shapes / streaming reads through io_uring** — **NO-GO** ([rustfs/backlog#1144]).
+  io_uring's lever on a *sequential* stream is pipelining; kernel readahead already has it. Cold
+  reads are device-bound, so io_uring at best ties; on a warm page cache it *loses badly*
+  (11–41% of a buffered read) because a single sequential stream exploits neither batched
+  submission nor the absence of a blocking thread. Streaming reads stay on the std backend.
+- **`AsyncFd` reaping without a driver thread** — **not done**. `Drop` cannot `await`, so the
+  bounded shutdown drain would have to break the public API. The dedicated driver thread plus
+  eventfd reaping already removed the busy-poll, which was the actual cost.
+- **Process-wide singleton ring** — **redefined**. A singleton conflicts with per-disk
+  isolation: a stalled disk must not starve another disk's ring. Rings stay per-disk, and
+  sharding ([#6]) scales them within a disk instead.
+- **Registered buffers (`register_buffers`)** — **deprioritized**. It conflicts with the
+  `Vec<u8>` ownership model the cancel-safety proof rests on, and end-to-end profiling showed
+  the S3 GET bottleneck is userspace copies, not the disk read
+  ([rustfs/backlog#1159]). See "When this crate helps" in the README.
+
+[#1]: https://github.com/rustfs/uring/pull/1
+[#2]: https://github.com/rustfs/uring/pull/2
+[#3]: https://github.com/rustfs/uring/pull/3
+[#4]: https://github.com/rustfs/uring/pull/4
+[#5]: https://github.com/rustfs/uring/pull/5
+[#6]: https://github.com/rustfs/uring/pull/6
+[rustfs/backlog#1051]: https://github.com/rustfs/backlog/issues/1051
+[rustfs/backlog#1144]: https://github.com/rustfs/backlog/issues/1144
+[rustfs/backlog#1159]: https://github.com/rustfs/backlog/issues/1159
+[Unreleased]: https://github.com/rustfs/uring/commits/main

README.md

@@ -1,20 +1,25 @@
 # rustfs-uring
 
 [![CI](https://github.com/rustfs/uring/actions/workflows/ci.yml/badge.svg)](https://github.com/rustfs/uring/actions/workflows/ci.yml)
-[![crates](https://img.shields.io/crates/v/rustfs-uring.svg)](https://crates.io/crates/rustfs-uring)
-[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/rustfs/uring/blob/master/LICENSE)
-[![docs.rs](https://docs.rs/rustfs-uring/badge.svg)](https://docs.rs/rustfs-uring/)
+[![license](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://github.com/rustfs/uring/blob/main/LICENSE)
 
 Cancel-safe async `io_uring` read backend for [RustFS](https://github.com/rustfs/rustfs) storage.
 
-This crate is the io_uring integration that RustFS's P2 read path is built on. It started as the Spike 0 cancel-safety
+This crate is the io_uring integration that RustFS's read path is built on. It started as the Spike 0 cancel-safety
 prototype ([rustfs/backlog#894](https://github.com/rustfs/backlog/issues/894)) and was hardened per
 the [#1048/#1051 audit](https://github.com/rustfs/backlog/issues/1051). It lives in its own repository so it can be
 verified in isolation — with a real io_uring CI leg that the main `rustfs/rustfs` workspace cannot run — before being
 wired into the storage layer.
 
-> **Status:** read path only, Linux only. Not yet published to crates.io — depend on it via git until the P2 work (write
-> path, eventfd/`AsyncFd` reaping, O_DIRECT, `LocalIoBackend` integration) lands. See [`docs/DESIGN.md`](docs/DESIGN.md).
+> **Status:** read path only, Linux only. **Not published to crates.io** (`publish = false`) — depend on it by pinning a
+> git revision. The read path is wired into `rustfs/rustfs` behind a runtime probe and is **off by default**
+> (`RUSTFS_IO_URING_READ_ENABLE`). See [`CHANGELOG.md`](CHANGELOG.md) for what has landed and
+> [`docs/DESIGN.md`](docs/DESIGN.md) for the invariants.
+
+```toml
+[target.'cfg(target_os = "linux")'.dependencies]
+rustfs-uring = { git = "https://github.com/rustfs/uring", rev = "<pin a commit>" }
+```
 
 ## The ownership model it enforces
 
@@ -31,8 +36,10 @@ proves and enforces the invariants any production io_uring integration must foll
   at the CQE, not at future drop; short reads on positioned reads are resubmitted to satisfy the whole-range contract;
   the probe file is opened via `O_TMPFILE`.
 
-The full invariant list, the corrected fd-reuse mechanism, and the P2 design constraints are in [
-`docs/DESIGN.md`](docs/DESIGN.md).
+Each invariant holds **per shard** (see below), because a shard is an independent instance of the same driver.
+
+The full invariant list, the corrected fd-reuse mechanism, and the design constraints are in
+[`docs/DESIGN.md`](docs/DESIGN.md).
 
 ## Usage
 
@@ -42,25 +49,81 @@ use std::sync::Arc;
 use rustfs_uring::UringDriver;
 
 # async fn demo() -> std::io::Result<()> {
-    // Probe a real IORING_OP_READ before accepting work; a restricted environment
-    // (seccomp/gVisor/old kernel) returns a ProbeFailure you degrade to the std
-    // backend on.
+    // Probe a real IORING_OP_READ before accepting work. A restricted environment
+    // (seccomp/gVisor/old kernel) returns a ProbeFailure whose
+    // `is_expected_restriction()` tells you to degrade to the std backend quietly.
     let driver = UringDriver::probe_and_start(64).expect("io_uring available");
 
     let file = Arc::new(File::open("/data/object")?);
 
-    // Positioned read (pread semantics, whole-range).
+    // Positioned read (pread semantics, whole-range: short reads are resubmitted).
     let bytes = driver.read_at(Arc::clone(&file), 0, 65536).await?;
 
     // Dropping the returned future before it completes is safe — the driver owns
     // the buffer until the CQE.
 
-    driver.shutdown();
+    let snapshot = driver.shutdown();
+    assert_eq!(snapshot.delivered + snapshot.orphan_reclaimed, snapshot.submitted);
     # Ok(())
     #
 }
 ```
 
+### Sharded rings
+
+A buffered read that hits the page cache completes *inline* inside `io_uring_enter`, so the thread driving a ring
+performs that read's `memcpy`. One ring is therefore capped at a single core's memory bandwidth (~5 GB/s measured).
+Give a disk several rings when its reads hit the cache:
+
+```rust
+# use rustfs_uring::UringDriver;
+// Four independent rings, each with `entries` SQ slots and its own driver thread.
+// In-flight is capped per shard, so the driver admits up to `shards * entries` reads.
+let driver = UringDriver::probe_and_start_sharded(64, 4)?;
+# Ok::<_, rustfs_uring::ProbeFailure>(())
+```
+
+`probe_and_start(entries)` is exactly `probe_and_start_sharded(entries, 1)`, so nobody grows threads by upgrading.
+Rings stay per-disk: a stalled disk cannot starve another disk's rings.
+
+### `O_DIRECT`
+
+Open the fd with `O_DIRECT`, pass the device's logical block size, and let the driver do the alignment. `offset` and
+`len` need **not** be aligned — it reads a block-aligned superset into a block-aligned buffer and hands back exactly
+the range you asked for. Padding, the bytes before the range, and the block-aligned tail never escape.
+
+```rust
+# use rustfs_uring::UringDriver;
+# use std::{fs::File, sync::Arc};
+# async fn demo(driver: &UringDriver, file: Arc<File>) -> std::io::Result<()> {
+// `file` was opened with O_DIRECT; 4096 is the probed logical block size.
+let bytes = driver.read_at_direct(file, 8_191, 100, 4096).await?;
+assert_eq!(bytes.len(), 100);
+# Ok(())
+# }
+```
+
+## When this crate helps — and when it does not
+
+These numbers come from the harnesses in this repository and from end-to-end profiling of RustFS
+([rustfs/backlog#1159](https://github.com/rustfs/backlog/issues/1159)). They are reported as measured, including the
+cases where io_uring loses.
+
+| workload | result |
+| --- | --- |
+| **Many concurrent positioned reads on one disk** (erasure-coded shard reads) | **Where it wins.** With sharded rings and a cached fd: 64 KiB at concurrency 128 → 361k IOPS vs 125k for a blocking-pool baseline, and p999 3.0 ms vs 13.5 ms. |
+| **A single sequential stream** | **It loses.** Kernel readahead already does what pipelining would buy. Cold reads are device-bound; on a warm page cache io_uring reaches only 11–41% of a buffered read. Streaming reads should stay on the std backend. |
+| **One read at a time (low concurrency)** | **It loses.** Per-op submission overhead exceeds a page-cache `memcpy`. |
+| **End-to-end S3 GET** | **Roughly neutral today (−7% … +4%).** The disk read is not the bottleneck: a cached 1 MiB GET spends ~25% of CPU in `memcpy` and ~10% in `memset`, and 0% on device reads. Optimising the read path further only pays once those copies are gone. |
+
+Two traps this crate's own benchmarking fell into, documented so others do not repeat them:
+
+- A `76×` apparent speedup turned out to be a **behaviour regression**, not a win: the io_uring path had silently
+  stopped honouring RustFS's `fadvise(DONTNEED)` page-cache reclaim policy, so one leg served everything from cache
+  while the other read the device. Always check `disk_read` and page-cache deltas, not just throughput.
+- Microbenchmarks of the read path measured a page-cache-hit regime that production *deliberately avoids* for large
+  reads. Isolated-path gains do not transfer end-to-end for free.
+
 ## Testing
 
 This is a Linux-only crate; on a non-Linux host `cargo check` only builds the empty stub.
@@ -77,15 +140,37 @@ cargo test -- --nocapture --test-threads=1
 ```
 
 The harness fails on either a non-degrading leg 1 or a vacuous-pass leg 2, so a skipped suite can never masquerade as
-real coverage.
+real coverage. The 15 acceptance tests are the cancel-safety contract: buffer conservation under a mixed
+drop/keep stress across shards, an orphaned op reclaimed only at its CQE, bounded shutdown drain, `O_DIRECT` returning
+exact unaligned ranges, and backpressure deferring rather than blocking a runtime worker.
+
+## Benchmarks
+
+Both harnesses refuse to overwrite or follow a symlink at a caller-supplied path (the sweeps run as root), fill their
+files with an offset-addressable pattern, and check every delivered byte against it under `BENCH_VERIFY=1` — throughput
+alone cannot tell a correct strategy from one reading the wrong offsets.
+
+```bash
+# Sequential whole-file read: buffered vs O_DIRECT vs pipelined io_uring, warm and cold cache.
+./bench-streaming.sh
+
+# Many concurrent positioned reads on one disk — the shape shard reads actually serve.
+# Isolates the cost of the per-read open and of the spawn_blocking hop.
+./bench-concurrent-pread.sh
+```
+
+## Roadmap
 
-## Roadmap (P2)
+- **Write path.** Untouched today; PUT still goes through the blocking pool, and profiling suggests the win there may
+  exceed the read path's.
+- **`register_files`.** Would remove the per-op fd lookup. Lower value now that the consumer caches descriptors.
+- **`SQPOLL`.** Eliminates `io_uring_enter` under sustained load, at the cost of a kernel polling thread per ring —
+  which multiplies by shards and by disks. Only for high-end deployments.
 
-- eventfd + tokio `AsyncFd` reaping to replace the poll loop (must still flush the NODROP overflow list).
-- O_DIRECT aligned buffers and the three read shapes.
-- `LocalIoBackend` integration in `rustfs/rustfs` behind runtime probing.
-- Per-disk probe cache and runtime errno degradation latch.
-- Registered buffers (P3) with the content-hygiene invariant, and the write path (P4).
+Closed by measurement, not built — see [`CHANGELOG.md`](CHANGELOG.md#decisions-recorded-not-implemented): streaming
+reads through io_uring (NO-GO), `AsyncFd` reaping without a driver thread (would break the public API), a process-wide
+singleton ring (conflicts with per-disk isolation), and registered buffers (conflicts with the `Vec<u8>` ownership
+model, and the bottleneck is elsewhere).
 
 ## License