Skip to content

Refactor read driver as an explicit stream - #9395

Merged
joseph-isaacs merged 2 commits into
ji/read-rangesfrom
ji/read-ranges-state-machine
Aug 13, 2026
Merged

Refactor read driver as an explicit stream#9395
joseph-isaacs merged 2 commits into
ji/read-rangesfrom
ji/read-ranges-state-machine

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Summary

Stacked on #9384.

  • replace the nested async read-driver loop with an explicit ReadDriver<R>: Stream state machine
  • replace the nested async_stream result adapter with an explicit ReadRangeResults state machine
  • drain every immediately-ready completion and refill freed read slots from IoRequestStream before yielding
  • preserve the existing physical concurrency cap, validation, unknown/missing-result handling, metrics, tracing, and panic propagation

ReadRangeResults has explicit Reading and Missing states: it matches out-of-order range results while reading, then reports every unresolved request if the underlying stream ends early.

When several results become ready together, the driver now resolves all of them and submits one refill group for the freed slots in the same poll. It does not add another FileReadAt or ObjectStore queue.

Validation

  • cargo +nightly fmt --all -- --check
  • cargo test -p vortex-file (142 passed; 1 doctest ignored)
  • cargo clippy -p vortex-file --all-targets --all-features -- -D warnings
  • RUSTDOCFLAGS='-D warnings --cfg docsrs' cargo doc --profile ci --no-deps --document-private-items -p vortex-file
  • deterministic refill test verifies three simultaneous completions produce one three-range refill while retaining the configured active-read cap

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@codspeed-hq

codspeed-hq Bot commented Aug 13, 2026

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚡ 4 improved benchmarks
❌ 1 regressed benchmark
✅ 1957 untouched benchmarks
⏩ 121 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation optimize_or_chain[200] 546.8 µs 1,177.4 µs -53.56%
Simulation compact_sliced[(16384, 10)] 2 ms 1.5 ms +37.48%
Simulation compact_sliced[(4096, 10)] 528.6 µs 391.6 µs +34.99%
Simulation take[small_m/shuffled/primitive/nonnull/chunks=16384/indices=16] 1.2 ms 1 ms +13.21%
Simulation eq_pushdown_low_match 1.6 ms 1.4 ms +11.06%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/read-ranges-state-machine (c3f5d45) with ji/read-ranges (0a48a0c)

Open in CodSpeed

Footnotes

  1. 121 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs marked this pull request as ready for review August 13, 2026 13:50
@joseph-isaacs
joseph-isaacs merged commit e8759c7 into ji/read-ranges Aug 13, 2026
72 of 75 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/read-ranges-state-machine branch August 13, 2026 13:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant