Parallel test execution - #6784
Conversation
sebastianbergmann
added
type/enhancement
API Surface ChangesIf any of the additions below are not intended as public API, mark them with New API SurfaceClasses
Methods
Modified API SurfaceMethods
|
9a08909 to
efcd7ee
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #6784 +/- ##
===========================================
Coverage 99.48% 99.49%
- Complexity 9416 9859 +443
===========================================
Files 915 938 +23
Lines 28736 30069 +1333
===========================================
+ Hits 28589 29916 +1327
- Misses 147 153 +6 ☔ View full report in Codecov by Harness. |
sebastianbergmann
changed the title
|
My 2 cents on the topic:
Parallel execution can't and will never be deterministic, a tiny difference between how much a unit lasts results in the following test being run in worker X instead of Y. |
|
Thank you, @Slamdunk, for taking the time to write all of this up. There's a lot of hard-won experience in here, and I appreciate it. I want to set expectations honestly, though: for me, native parallel test execution in PHPUnit is still just an idea, and this pull request is a (remarkably well-working) proof of concept rather than a roadmap commitment. If I ever decide to be serious about this and actually ship it, the feature will have limitations. And those limitations must (and will) be very well documented. Complaints about them will then be kindly refused. 🙂 Even in that case, I neither intend nor expect this to replace dedicated solutions such as ParaTest or Paraunit. The scope I have in mind right now is roughly: PHPUnit supports parallel execution for well-architected test suites that properly deal with resource-usage conflicts like databases, etc. So several of the needs you describe, coordinated bootstrapping, functional/method-level distribution, deterministic replay of a run, are exactly the kind of thing that lives outside that scope and is better served by the tools built specifically for it. That said, your notes are genuinely useful for thinking about where the boundaries of that scope should sit, so thank you again. |
4e03c18 to
5053a2d
Compare
5053a2d to
cf8f973
Compare
|
I have briefly looked into whether the new I/O polling API that PHP 8.6 introduces (RFC: poll_api) could replace the file-based completion polling used by the parallel test runner. The conclusion is that it cannot, because it has the exact limitation that forced the file-based design in the first place: on Windows, it cannot wait on
However:
In other words, Therefore, the uniform file-based completion polling stays. This decision should be revisited only if a future PHP version ships a poll handle that can wait on pipes or process handles on Windows, which is what this feature actually needs. |
cf8f973 to
1e97c6b
Compare
|
I see that it's up to the user to select how many worker to use. The You might be interested in giving it a try, so PHPUnit can provide |
I am aware of |
04cb38d to
43341d8
Compare
908d59e to
4b04c6a
Compare
Generalize process isolation into a worker that boots PHPUnit once and then runs an arbitrary number of tests, each in response to a command on its control channel. Results are transported back using the same serialized envelope as process isolation, so ChildProcessResultProcessor reconstitutes them unchanged. Tests run by one worker share a process and therefore do not get per-test global-state isolation.
…ially, selected with the new --parallel=<n> CLI option (sequential remains the default). Builds on the persistent worker and async JobRunner from #6753. The distribution unit is one test class, run in a worker that reconstructs and runs it from the same serialized envelope process isolation uses. PHPT tests are distributed too, reconstructed in a worker from their file path. A ResultAggregator replays each unit's collected events into the parent in deterministic suite order, so output, logging, results, and coverage are produced exactly as in sequential mode. Tests that cannot run in a worker are run in the main process at their suite position instead: those marked #[DoNotRunInParallel] (new class- and method-level attribute), those requiring process isolation, those whose data cannot be serialized, and PHPT tests carrying a --DO_NOT_RUN_IN_PARALLEL-- section.
…em inside parallel workers (which hung on Windows)
…test through their IterativeTestSuite when tests are run in parallel
…has finished, instead of each time an outermost test suite finishes
… process so that parallel runs report progress live
…ests are run in parallel, so that only tests of the same suite run concurrently
…ool and the PHPT runner, so that --parallel N never executes more than N units at once
…uites when tests are run in parallel, so that loggers that reconstruct the suite hierarchy work
…estSuite to the worker, so that the suite's event envelope nests them in logger output as in a sequential run
…minating the units that are still executing, so that the --stop-on-* options work with --parallel
…ss, so that a transient crash does not fail a parallel run
…ain process, so that cross-class dependencies work when tests are run in parallel
…s, longest first, so that the longest-running tests do not become the stragglers a parallel run waits for
… after a grace period, so that a parallel run can neither deadlock on an unread pipe nor hang when stopping early
…ected as errored, ship only a unit's own passes in its envelope, and load test-suite bootstrap scripts in workers, so that a parallel run loses no test results and matches a sequential run
…o that the retry of its unit boots a fresh worker process instead of aborting the run or leaking the compromised one
… chunk runs, so that a unit that leads a later chunk is not run inside the test-suite envelope of the chunk before it
…atch with a telling exception when the worker command cannot be encoded, so that a data provider key that is not valid UTF-8 cannot abort a parallel run
…d, and do not start the FILE section of one terminated during SKIPIF, when a parallel run stops early, so that an abandoned test still cleans up after itself
…onflicts with every other test entirely on their own, draining the worker pool and the PHPT runner first, so that the exclusivity these declarations promise actually holds
…'s classes with named constructors, derivation methods, and shared helpers, so that each invariant lives in one place
…es in one pass through one metadata traversal, and poll a worker's event stream with a stat instead of a re-read, so that a parallel run wastes less work as suites grow
…e helper and two template fragments, writing the configuration and source map once per run for all of them, so that isolated processes and parallel workers cannot drift apart
…e the result envelope's decoding between its two consumers, and let a work unit report its own recorded duration, so that each half of the parallel runner's data model lives in exactly one place
…and the facade's dispatcher selection, and record at collection time which chunk envelopes the units emit, so that behavior the sequential and parallel runners must perform identically is expressed exactly once
…let those descriptors travel in a serialized command instead of a JSON-encoded one, so that a member's encoding and decoding live in one place and no value it carries needs a base64 shim to survive the trip to a worker
2c7a98a to
c034df8
Compare
|
2 participants
PHPUnit can now execute a test suite across several worker processes concurrently instead of one test after another. Parallel execution is opt-in via a new
--parallel=<n>command-line option and changes nothing when it is not used: the sequentialTextUI\TestRunnerremains the default andTextUI\ParallelTestRunneris selected only when<n> > 1.Native parallelism here is a generalization of process isolation rather than a new subsystem: a worker reconstructs and runs a unit of work and ships its outcome home in the very same serialized envelope that process isolation already uses, and the parent replays that envelope through its normal event pipeline. As a result the parent process remains the single source of truth for all output, logging, results, and code coverage, which are produced exactly as in sequential mode.
How it works
#[BeforeClass]/#[AfterClass]and intra-class ordering. ADataProviderTestSuite, and theIterativeTestSuitethat carries the repetitions of a repeated test or the attempts of a retried test, travel to the worker as atomic members of their class' unit, so their suite envelopes nest in logger output exactly as they do sequentially.<testsuite>elements of an XML configuration are run one after another, just as in sequential mode: only tests that belong to the same top-level test suite ever run concurrently.Runner\Parallel\WorkerPoolownsnpersistent workers and pulls from a dynamic work-stealing queue, so the load self-balances against stragglers. A worker signals completion through the filesystem, which the parent polls;stream_select()is not used, because it does not work on the workers' output pipes on Windows.Runner\Parallel\Schedulerdispatches units in the order of the durations recorded by the test run history, longest first, so that the longest-running work starts as early as possible instead of becoming the straggler the pool waits for. A unit with no recorded duration goes first. Only the dispatch order is affected; results are released in suite order either way.Runner\Parallel\ResultAggregatorbuffers each unit and releases it only once every preceding unit (in suite order) has been released, so the event stream, and therefore every output format, including the default progress output, is byte-for-byte what sequential mode produces. Streamed events of the unit that is next in suite order are forwarded immediately; those of later units are buffered until their turn.PHPUnit\Framework\TestCaseinstances and cannot be reconstructed in a worker (and nesting their child processes inside workers hung on Windows), soRunner\Parallel\PhptRunnerruns them side by side in the main process, each as its own child process.ProcessBudget, so--parallel <n>never executes more than<n>units at once, no matter how a chunk is composed.Running in the main process
Some tests cannot, or must not, run in a worker. These run in the main process at their correct suite position — the aggregator invokes them while releasing results, so global ordering is preserved — and their execution there is ordinary sequential execution:
#[DoNotRunInParallel]— a new attribute, valid on classes and methods, for tests that must not run alongside others (for instance because they share a machine-global resource). Such a unit runs alone: the worker pool and the PHPT runner first finish what they are executing and start nothing new until it is done.#[RunInSeparateProcess],#[RunTestsInSeparateProcesses], or global--process-isolation) — a shared worker cannot provide isolation, but the main process spawns the isolated child as usual.#[Depends]— the result they depend on is produced by a different unit and is only visible in the main process, once that unit has been released.serialize()throw) as well as resources (whichserialize()silently degrades to0).TestCasenor a PHPT test.Because a work unit is a whole test class, a single test method carrying
#[DoNotRunInParallel], requiring isolation, depending on another class, or providing non-serializable data takes its entire class out of the parallel phase.PHPT tests declare their concurrency constraints with a
--CONFLICTS--section instead, since a PHPT file cannot carry PHP attributes: while a test holding conflict keyKruns, no other test declaringKis started, and the reserved keyallmeans the test runs entirely on its own.Robustness
The worker process running X ended unexpectedly), the remaining units are redistributed across the surviving workers, and the run accounts for all of them.--stop-on-*. As soon as the collected results call for a stop, the aggregator releases nothing further and the units still executing are terminated (a grace period, then a forced kill). A PHPT test whoseFILEsection was terminated still runs itsCLEANsection, and one terminated duringSKIPIFdoes not start itsFILEsection.New public API
--parallel=<n>CLI option (also listed in--help); a value that is not a positive integer is ignored with a test runner warning.#[PHPUnit\Framework\Attributes\DoNotRunInParallel](TARGET_CLASS | TARGET_METHOD), with full metadata-layer support.--CONFLICTS--PHPT section, as supported by the PHP project'srun-tests.php: one conflict key per line,#starts a comment, blank lines are ignored, andallis reserved.PHPUNIT_WORKER_ID— the small, stable ordinal (0,1,2, ...), ideal for indexing a fixed set of pre-provisioned resources. A worker restarted after a crash keeps its ordinal.PHPUNIT_WORKER_TOKEN— a value of the form<id>_<random>that is unique across workers and across runs, for resources that must not collide with those left behind by a previous run. A restarted worker gets a fresh token.Event\TestRunner\ChildProcessReason::ParallelWorker, so that the events about child processes say why one was used.Shared with the sequential runner
Rather than duplicating the sequential runner, behavior the two must perform identically was extracted so it is expressed exactly once:
TextUI\TestRunnerLifecycle— the test runner lifecycle both runners drive.Event\Dispatcher\CollectionWindowand the facade's dispatcher selection — the window during which events are collected instead of dispatched.Framework\TestRunner\ChildProcessBootstrapplus two templates — the boot code every kind of child process shares; the configuration and source map are written once per run for all of them, so isolated processes and parallel workers cannot drift apart.Framework\TestRunner\ChildProcessResultEnvelope— the result envelope's encoding and decoding, shared by its consumers.TestRunner\ExecutionFinished, instead of every time an outermost test suite finishes.Results for PHPUnit's own test suite
Measured on a 6-core machine:
--parallel 10--testsuite unit(5813 tests)--testsuite end-to-end(1257 tests)Both suites report the same numbers of tests, failures, and skipped tests in both modes, stable across runs. The end-to-end suite benefits the most because it is dominated by child processes rather than by CPU work in the main process.
Which worker runs which test class is not deterministic; the work-stealing scheduler assigns classes by timing, but the aggregated output is identical regardless.
Notes and limitations (possible follow-ups)
#[DoNotRunInParallel].