v4.0: Rewrite on PHP 8.4's native DOM API, parity with Readability.js 0.6.0#41
Open
fivefilters wants to merge 13 commits into
Open
v4.0: Rewrite on PHP 8.4's native DOM API, parity with Readability.js 0.6.0#41fivefilters wants to merge 13 commits into
fivefilters wants to merge 13 commits into
Conversation
- Require PHP >= 8.4; drop masterminds/html5, psr/log, ext-xml - Remove src/Nodes (DOM subclasses, NodeTrait, NodeUtility): the new Dom\ classes have no registerNodeClass, and the hacks these existed for (node shifting, attribute-based state) are obsolete with Lexbor - New Configuration (readonly options, Readability.js 0.6.0 defaults), Article result object, RegExps (0.6.0 patterns), ParseException - Replace test corpus with Mozilla's 130 pages verbatim; keep 7 PHP-only pages with metadata converted to Mozilla's format; drop image fixtures and per-page config files - Test harness: DomCompare ports Mozilla's structural DOM comparison, ReadabilityTest mirrors Mozilla's jsdom test path; cross-check tools run Readability.js over the corpus for divergence attribution - CI matrix: PHP 8.4/8.5, no Docker/libxml pinning Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aWXZXD5qmLs5aCJ2HLkC2
Fresh method-by-method transcription of Readability.js v0.6.0 onto Dom\HTMLDocument (Lexbor). Method names and order mirror the JS prototype to keep upstream syncs mechanical. Notable PHP adaptations: - Scoring state and data-table flags live in per-parse SplObjectStorage maps (the new DOM API has no registerNodeClass or expando properties) - getAllNodesWithTag uses querySelectorAll snapshots, which is the only query path used while mutating - JS-compatible whitespace handling (NBSP etc.) for trim/normalize - toAbsoluteURI reproduces WHATWG URL parser behaviors that RFC 3986 resolvers reject or leave alone: control/tab/newline stripping, space encoding, empty-path normalization, colon-in-first-segment references - fdiv for JS division semantics where a zero score is possible - Readerable ports Readability-readerable.js (new to the PHP library) All 130 Mozilla test pages pass content, metadata and readerable comparisons; the 7 PHP-only pages await regenerated goldens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aWXZXD5qmLs5aCJ2HLkC2
- Regenerate expected.html for the 7 PHP-only pages via the golden-file workflow, after validating the algorithm against Mozilla's 130 reference fixtures (extracted text length matches the old goldens) - Add readerable keys for those pages, computed with Mozilla's own isProbablyReaderable via jsdom - WHATWG URL behaviors in toAbsoluteURI (empty-path normalization, whitespace stripping, colon-in-first-segment refs, absolute passthrough for opaque schemes) - Treat test sources as UTF-8 text like jsdom does, so fixture meta charsets don't trigger re-decoding - Document accepted divergences from the npm Readability.js release in test/tools/known-divergences.md: the port tracks git master, which is what Mozilla's fixtures are generated from Full suite green: 415 tests, 946 assertions, 0 skipped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aWXZXD5qmLs5aCJ2HLkC2
- ReadabilityUnitTest pins textSimilarity, unescapeHtmlEntities, toAbsoluteURI (including the WHATWG behaviors), isValidByline and getRowAndColumnCount against values verified with Readability.js - ConfigurationTest checks the 0.6.0 defaults and fromArray Also verified: E_ALL warning sweep over all 137 test pages is silent, parse state is released after each run, peak memory 16 MB for the whole corpus. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aWXZXD5qmLs5aCJ2HLkC2
- README: new Article/Configuration API, PHP 8.4 requirement, option reference with Readability.js mapping, v3 -> v4 migration table, cross-check tooling docs - CHANGELOG: v4.0.0 entry - CONTRIBUTING: how to sync with a new Readability.js release, and the intentional PHP differences that should not be 'fixed' Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aWXZXD5qmLs5aCJ2HLkC2
PHP 8.5 deprecates SplObjectStorage::contains() and ::attach() in favor of offsetExists()/offsetSet(). Also surface deprecation details in test output by default so these show up in CI logs. Suite is now a clean OK on both 8.4 and 8.5 (was: 10 deprecations on 8.5). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aWXZXD5qmLs5aCJ2HLkC2
actions/checkout@v4 runs on Node.js 20, which GitHub Actions runners are deprecating. v5 runs on Node 24. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aWXZXD5qmLs5aCJ2HLkC2
…l on 8.4 Readability.js resolves URLs with the browser's WHATWG new URL(); the port emulated its behaviors (control/whitespace stripping, space encoding, empty-path normalization, opaque-scheme passthrough, colon-in-first-segment references) on top of league/uri's RFC 3986 resolver. Replace all of that with the real thing: PHP 8.5's native Uri\WhatWg\Url when available, falling back to rowbot/url (a WHATWG-compliant, WPT-tested parser) on PHP 8.4, via a small internal Url wrapper. toAbsoluteURI is now a 1:1 mirror of the JS closure, and isUrl matches JS new URL(str) strictness exactly. Also untrack .phpunit.result.cache (committed by accident earlier). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aWXZXD5qmLs5aCJ2HLkC2
- vimeo/psalm ^6 at errorLevel 3 (strictBinaryOperands off — mixed int/float arithmetic mirrors JS's single number type), wired into CI and exposed as 'composer analyse'. A small stub under stubs/ covers PHP 8.5's native URI classes so analysis on 8.4 resolves them. - Fix everything Psalm found: per-parse state ($doc/$scores/$dataTables) is now non-nullable and reset to fresh empty instances after parse (same memory release, no null-juggling); preg_* false/null returns get explicit fallbacks; Dom\Node::remove() calls (not part of that class) become removeChild(); unwrapNoscriptImages guards its querySelector results; allowedVideoRegex is typed non-empty-string with a guarded assignment; array properties and params get shape docblocks. - phpunit/phpunit ^11 -> ^12 (dev-only; ^13 conflicts with Psalm's sebastian/diff constraint). rowbot/url already latest. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aWXZXD5qmLs5aCJ2HLkC2
UPGRADE.md covers the full migration: before/after code, a mapping table for every 3.x result getter and configuration option (verified against the actual 3.x API), replacement snippets for removed features (image extraction via contentElement->querySelectorAll, og:image via the document; PSR-3 -> debug flag), and the behavior changes (Article value object, page wrapper div, always-on byline, WHATWG URL fixing, encoding handling). All code snippets in the guide are executed and verified. README gains finer-control output and contentElement post-processing examples; its migration section now summarizes and links to UPGRADE.md, as does the CHANGELOG. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aWXZXD5qmLs5aCJ2HLkC2
…-3 logger Requested by the maintainer to ease 3.x upgrades — these were the removed features most likely to be missed. - Image extraction returns as readonly Article fields ($article->image, $article->images) rather than the old getter methods, fitting the value-object API. Lead image comes from og:image/twitter:image or <link rel=img_src|image_src>; the list prepends it to the content <img> srcs, de-duplicated. Both absolutized when fixRelativeURLs is on. - keepInlineByline (default false) replaces v3's articleByline. The byline is always extracted into Article::$byline now (as in JS); this option only controls whether an inline byline stays in the content. - PSR-3 logging returns via a Configuration $logger option (psr/log back as a dependency); messages go to the logger independently of the debug flag. log() dispatches to both. New PhpFeaturesTest covers all three. Docs (README options + Article fields, UPGRADE.md, CHANGELOG) updated. 472 tests green on 8.4/8.5, Psalm clean, corpus content output unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aWXZXD5qmLs5aCJ2HLkC2
Add a warning callout and a comparison table making explicit that a 3.x install using the default kept the inline byline in the content, whereas 4.0 removes it by default (keepInlineByline: true restores the old behavior). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011aWXZXD5qmLs5aCJ2HLkC2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ground-up rewrite of the library on PHP 8.4's native
Dom\HTMLDocument(the WHATWG-spec Lexbor parser), transcribed method-for-method from Readability.js v0.6.0 (git master). No third-party HTML parser, full parity with current upstream, and the test corpus is now Mozilla's own.Core port
src/Readability.phpis a fresh transcription: private methods mirror the Readability.js prototype in name and order, each doc-commented with the JS method it mirrors, so future upstream syncs are a mechanical diff (a how-to is in CONTRIBUTING.md).SplObjectStoragemaps — the new DOM API has noregisterNodeClass, which kills the oldsrc/Nodes/subclass layer, andWeakMapverifiably loses entries when node wrappers are GC'd.getAllNodesWithTag()usesquerySelectorAllsnapshots (the new API's static NodeLists), replacing the old shifting-aware iteration hack. No XPath anywhere.trim/\s).Uri\WhatWg\Urlwhen available, rowbot/url on PHP 8.4 — the same URL Standard Readability.js gets from the browser'snew URL().toAbsoluteURIis a 1:1 mirror of the JS closure; league/uri and the RFC 3986→WHATWG emulation layer are gone.New API
parse($html)returns a readonlyArticlevalue object mirroring the JS result:title,content,textContent,length,excerpt,byline,siteName,dir,lang,publishedTime, pluscontentElementfor DOM consumers. ThrowsParseExceptionwhere JS returns null.Configurationis a readonly options object with Readability.js 0.6.0 defaults, addingmaxElemsToParse,classesToPreserve,allowedVideoRegex,linkDensityModifier,debug.Readerable::isProbablyReaderable()— first port of Readability-readerable.js.parser,substituteEntities,normalizeEntities,summonCthulhu),articleByline(always on now, as upstream). Migration table in the README.Parity features previously missing
lang/publishedTimeoutputs; aria-modal dialog removal; ad/loading-word stripping; Unicode comma scoring; parsely/article:author/itemprop=namemetadata; JSON-LD@graph/@context-object/array handling; limited-set HTML entity unescaping; updated regexes (mathjax, bilibili, en/em-dash title separators); invalid-attribute resilience.Tests
test/DomCompare.php), so upstreamexpected.html/expected-metadata.jsonfiles are used unmodified — the goldens come from the reference implementation, not from this port.readerablekeys computed via Mozilla's own implementation under jsdom.test/tools/) runs Readability.js over the corpus and diffs field-by-field; remaining differences are all npm-release-vs-git-master lag, documented intest/tools/known-divergences.md.Infra
composer.json: PHP ≥ 8.4, dropsmasterminds/html5,psr/log,ext-xml,league/uri; addsrowbot/url(unused at runtime on PHP 8.5+, where the native URI extension takes over).🤖 Generated with Claude Code
https://claude.ai/code/session_011aWXZXD5qmLs5aCJ2HLkC2
Generated by Claude Code