Skip to content

fix: bound readable content extraction#151

Open
chaliy wants to merge 1 commit into
mainfrom
2026-07-07-propose-fix-for-cpu-and-memory-dos
Open

fix: bound readable content extraction#151
chaliy wants to merge 1 commit into
mainfrom
2026-07-07-propose-fix-for-cpu-and-memory-dos

Conversation

@chaliy

@chaliy chaliy commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Motivation

  • The readable/agent content_focus path previously collected and cloned every nested content-like subtree, allowing a bounded but malicious HTML response to amplify into quadratic CPU and memory use and potentially OOM or stall workers.
  • The change constrains work done during readability extraction to protect availability while preserving the goal of returning a dense article-like block for agents.

Description

  • Replace whole-list candidate accumulation with single best-candidate tracking via collect_best_readable_candidate, scoring borrowed &str slices and cloning only the winning subtree to avoid retaining overlapping copies.
  • Add a hard cap MAX_READABLE_CANDIDATES = 64 to stop scoring after a bounded number of candidates and stop scanning further low-priority candidates.
  • Change score_readable_candidate to accept &str and return a lightweight ReadableCandidateScore, and only construct ReadableCandidate for the current best match.
  • Add a regression unit test test_extract_readable_content_handles_deep_nested_content to ensure the extractor still selects readable content under deeply nested markup and preserve fallback behavior when confidence is low.

Testing

  • Ran formatting with cargo fmt --all and static linting with cargo clippy --workspace --all-targets -- -D warnings, both completed without warnings or failures.
  • Ran cargo test --workspace and all tests passed (300 passed; 0 failed), including the new regression test convert::tests::test_extract_readable_content_handles_deep_nested_content which passed.
  • Verified the readable extraction behavior remains compatible with existing fallbacks (strip_boilerplate) when confidence is low.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant