Skip to content

test: add reader benchmark suite#150

Merged
chaliy merged 1 commit into
mainfrom
codex/reader-bench-suite
Jul 7, 2026
Merged

test: add reader benchmark suite#150
chaliy merged 1 commit into
mainfrom
codex/reader-bench-suite

Conversation

@chaliy

@chaliy chaliy commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

What

Adds a Mira reader benchmark example comparing FetchKit against Jina Reader, Firecrawl, Trafilatura/Readability, and Crawl4AI on fixed public pages.

Why

Gives the project a repeatable real-web quality benchmark for token count, link preservation, main-content precision, table/code retention, and metadata preservation.

How

  • Adds the reader_bench Mira example with Rust Learn, docs.rs reqwest, Wikipedia Rust, GitHub fetchkit, and Stack Overflow Python main pages.
  • Adds target adapters for FetchKit, Jina Reader, Firecrawl, and CLI-based Trafilatura/Crawl4AI.
  • Documents install/run configuration for optional tool targets.

Risk

  • Medium
  • Benchmark depends on live public pages and third-party services, so scores, latency, and availability can drift.

Checklist

  • Unit tests are passed
  • Smoke tests are passed
  • Documentation is updated
  • Specs are up to date and not in conflict

Validation

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace
  • RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps
  • cargo build --workspace --exclude fetchkit-python --release
  • doppler run --project everruns-dev --config dev -- bash -lc 'PATH="/Users/mykhailochalyi/.codex/worktrees/cc71/fetchkit/.bench-bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/opt/homebrew/bin:/Users/mykhailochalyi/.codex/tmp/arg0/codex-arg0DzDLth:/Users/mykhailochalyi/.cache/codex-runtimes/codex-primary-runtime/dependencies/bin:/opt/homebrew/opt/curl/bin:/Users/mykhailochalyi/.local/bin:/opt/homebrew/opt/rustup/bin:/Users/mykhailochalyi/.cargo/bin:/opt/homebrew/sbin:/Applications/Codex.app/Contents/Resources" mira --example reader_bench run' (25/25 passed)

@chaliy chaliy marked this pull request as ready for review July 7, 2026 00:30
@chaliy chaliy merged commit 5214403 into main Jul 7, 2026
11 checks passed
@chaliy chaliy deleted the codex/reader-bench-suite branch July 7, 2026 00:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant