v4.0: Rewrite on PHP 8.4's native DOM API, parity with Readability.js 0.6.0

.github/workflows/main.yml

@@ -1,46 +1,35 @@
-# This is a basic workflow to help you get started with Actions
-
 name: CI
 
-# Controls when the workflow will run
 on:
-  # Triggers the workflow on push or pull request events but only for the master branch
   push:
     branches: [master]
   pull_request:
     branches: [master]
-
-  # Allows you to run this workflow manually from the Actions tab
   workflow_dispatch:
 
-# A workflow run is made up of one or more jobs that can run sequentially or in parallel
 jobs:
-  # This workflow contains a single job called "build"
-  build:
-    # The type of runner that the job will run on
+  test:
     runs-on: ubuntu-latest
 
     strategy:
       matrix:
-        php: ['8.1', '8.2', '8.3', '8.4']
-        libxml: ['2.9.14']
+        php: ['8.4', '8.5']
 
-    # Steps represent a sequence of tasks that will be executed as part of the job
     steps:
-      # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v7
 
       - name: Set up PHP
         uses: shivammathur/setup-php@v2
         with:
           php-version: ${{matrix.php}}
+          extensions: dom, mbstring
           tools: composer:v2
 
       - name: Install dependencies
         run: composer install
 
-      # Runs a set of commands using the runners shell
       - name: Run tests
-        run: |
-          docker build --build-arg PHP_VERSION=${{matrix.php}} --build-arg LIBXML_VERSION=${{matrix.libxml}} -t gh-action - < ./docker/php/Dockerfile
-          docker run --volume $PWD:/app --workdir="/app" --env XDEBUG_MODE=coverage gh-action php ./vendor/bin/phpunit --coverage-clover /app/test/clover.xml
+        run: ./vendor/bin/phpunit
+
+      - name: Static analysis
+        run: ./vendor/bin/psalm --output-format=github --no-progress

.gitignore

@@ -2,4 +2,8 @@
 vendor
 composer.lock
 /test.*
-/test/changed/
+/test/changed/
+/test/tools/node_modules/
+/test/tools/js-output/
+/test/tools/package-lock.json
+/.phpunit.result.cache

CHANGELOG.md

@@ -1,6 +1,33 @@
 # Change Log
 All notable changes to this project will be documented in this file.
 
+## [v4.0.0](https://github.com/fivefilters/readability.php/releases/tag/v4.0.0)
+
+Ground-up rewrite on PHP 8.4's native DOM API, at parity with Readability.js v0.6.0 (git master). See [UPGRADE.md](UPGRADE.md) for the full 3.x → 4.0 migration guide.
+
+### Changed
+- Requires PHP >= 8.4; parsing and serialization use `Dom\HTMLDocument` (the WHATWG-spec Lexbor parser bundled with PHP), replacing HTML5-PHP and the legacy libxml path
+- `parse()` now returns a readonly `Article` value object (`title`, `content`, `textContent`, `length`, `excerpt`, `byline`, `siteName`, `dir`, `lang`, `publishedTime`, `image`, `images`, `contentElement`) and throws `ParseException` when no article is found
+- `Configuration` is a readonly object with named constructor arguments; `maxTopCandidates` renamed to `nbTopCandidates` (matching Readability.js)
+- Article output is wrapped in `<div id="readability-page-1" class="page">`, as in Readability.js
+- Byline is always extracted into `Article::$byline`; the `articleByline` option becomes `keepInlineByline`, which only controls whether an inline byline stays in the content (default removes it, as in Readability.js)
+- Image extraction moved onto the result object: `getImage()`/`getImages()` become `$article->image` / `$article->images`
+- PSR-3 logging kept: pass a `LoggerInterface` as the `logger` option instead of `setLogger()`
+- Relative URL resolution now uses a real WHATWG URL parser — PHP 8.5's native `Uri\WhatWg\Url` when available, [rowbot/url](https://github.com/TRowbotham/URL-Parser) on PHP 8.4 — matching the `new URL()` behavior Readability.js relies on; replaces league/uri
+- Test corpus replaced with Mozilla's 130 test pages verbatim; content comparison ports Mozilla's structural DOM comparison
+
+### Added
+- Parity with Readability.js 0.6.0: `lang` and `publishedTime` output; `maxElemsToParse`, `classesToPreserve`, `allowedVideoRegex`, `linkDensityModifier` and `debug` options; aria-modal dialog removal; ad/loading-indicator stripping; parsely/`article:author`/`itemprop` metadata sources; JSON-LD `@graph`, `@context`-object and array handling; Unicode comma scoring; updated regexes (mathjax, bilibili, en/em-dash title separators)
+- `Readerable::isProbablyReaderable()`, a port of Readability-readerable.js
+- `parseDocument()` for callers who already hold a `Dom\HTMLDocument`
+- Cross-check harness (`test/tools/`) that diffs this port's output against Readability.js over the whole corpus
+- Static analysis with [Psalm](https://psalm.dev/) (`composer analyse`), run in CI alongside the test suite
+
+### Removed
+- HTML5-PHP dependency; `ext-xml` requirement
+- Options that existed as libxml workarounds: `parser`, `substituteEntities`, `normalizeEntities`, `summonCthulhu`
+- The custom DOM subclass layer (`src/Nodes/`) and its workarounds (attribute-based state, shifting-aware iteration)
+
 ## [v3.3.3](https://github.com/fivefilters/readability.php/releases/tag/v3.3.3)
 - Fix type error - extends type support to add DOMProcessingInstruction in more method signatures (reported by @reinierkors)
 

CONTRIBUTING.md

@@ -21,10 +21,23 @@ For anything else, we accept contributions via Pull Requests on [Github](https:/
 - **Don't forget to add yourself to AUTHORS.md** - If you want to be credited, make sure you add your information (whatever you want to include) in `AUTHORS.md`.
 
 
+## Syncing with a new Readability.js release
+
+The private methods in `src/Readability.php` mirror the prototype methods of Readability.js in name (minus the underscore prefix) and order, so syncing is a mechanical diff exercise:
+
+1. Diff the new `Readability.js` against the previous synced version (git master at the time of the last sync — see `test/tools/known-divergences.md`).
+2. For each changed `_method`, apply the same change to the matching method in `src/Readability.php`. Regex changes go to `src/RegExps.php` (constants are the UPPER_SNAKE forms of the JS `REGEXPS` keys). `Readability-readerable.js` changes go to `src/Readerable.php`.
+3. Copy any added/changed directories from Mozilla's `test/test-pages/` into `test/test-pages/` verbatim (drop each page's `expected-images.json`-era leftovers if any; sources, `expected.html` and `expected-metadata.json` are used as-is).
+4. Run `./vendor/bin/phpunit`, then the cross-check harness in `test/tools/` (bump the `@mozilla/readability` version in its `package.json`), and update `known-divergences.md`.
+
+Things that intentionally differ from the JS (don't "fix" these): scoring state lives in `SplObjectStorage` maps instead of node expandos; `getAllNodesWithTag` materializes querySelectorAll snapshots; URL resolution goes through `src/Url.php`, which wraps a real WHATWG URL parser (PHP 8.5's native `Uri\WhatWg\Url`, or rowbot/url on PHP 8.4) and returns `null` where JS `new URL()` throws; JS `''`/`undefined` metadata maps to PHP `null`; `parse()` throws instead of returning null.
+
 ## Running Tests
 
 ``` bash
-$ make test-all #requires docker and docker-compose
+$ ./vendor/bin/phpunit      # requires PHP 8.4+
+$ ./vendor/bin/psalm        # static analysis; CI runs this too
+$ make test-all             # multiple PHP versions; requires docker and docker-compose
 ```
 
 

Makefile

@@ -1,30 +1,15 @@
-.PHONY: test-all
+.PHONY: test-all test-8.4 test-8.5 start stop
 
-test-all: start test-8.1 test-8.2 test-8.3 test-8.4 stop
-
-test-8.1:
-	docker-compose exec php-8.1-libxml-2.9.13 php /app/vendor/phpunit/phpunit/phpunit --configuration /app/phpunit.xml
-
-test-8.2:
-	docker-compose exec php-8.2-libxml-2.9.14 php /app/vendor/phpunit/phpunit/phpunit --configuration /app/phpunit.xml
-
-test-8.3:
-	docker-compose exec php-8.3-libxml-2.9.14 php /app/vendor/phpunit/phpunit/phpunit --configuration /app/phpunit.xml
+test-all: start test-8.4 test-8.5 stop
 
 test-8.4:
-	docker-compose exec php-8.4-libxml-2.9.14 php /app/vendor/phpunit/phpunit/phpunit --configuration /app/phpunit.xml
+	docker-compose exec php-8.4 php /app/vendor/phpunit/phpunit/phpunit --configuration /app/phpunit.xml
+
+test-8.5:
+	docker-compose exec php-8.5 php /app/vendor/phpunit/phpunit/phpunit --configuration /app/phpunit.xml
 
 start:
-	docker-compose up -d php-8.1-libxml-2.9.13 php-8.2-libxml-2.9.14 php-8.3-libxml-2.9.14 php-8.4-libxml-2.9.14
+	docker-compose up -d php-8.4 php-8.5
 
 stop:
 	docker-compose stop
-
-test-all-versions:
-	for php_version in 8.1 8.2 8.3 8.4; do \
-	    for libxml_version in 2.9.14; do \
-			docker-compose up -d php-$$php_version-libxml-$$libxml_version; \
-			docker-compose exec php-$$php_version-libxml-$$libxml_version php /app/vendor/phpunit/phpunit/phpunit --configuration /app/phpunit.xml; \
-		done \
-	done
-	docker-compose stop