fix: bound readable content extraction#151
Open
chaliy wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
content_focuspath previously collected and cloned every nested content-like subtree, allowing a bounded but malicious HTML response to amplify into quadratic CPU and memory use and potentially OOM or stall workers.Description
collect_best_readable_candidate, scoring borrowed&strslices and cloning only the winning subtree to avoid retaining overlapping copies.MAX_READABLE_CANDIDATES = 64to stop scoring after a bounded number of candidates and stop scanning further low-priority candidates.score_readable_candidateto accept&strand return a lightweightReadableCandidateScore, and only constructReadableCandidatefor the current best match.test_extract_readable_content_handles_deep_nested_contentto ensure the extractor still selects readable content under deeply nested markup and preserve fallback behavior when confidence is low.Testing
cargo fmt --alland static linting withcargo clippy --workspace --all-targets -- -D warnings, both completed without warnings or failures.cargo test --workspaceand all tests passed (300 passed; 0 failed), including the new regression testconvert::tests::test_extract_readable_content_handles_deep_nested_contentwhich passed.strip_boilerplate) when confidence is low.Codex Task