fix(runner): fail the run when a test is aborted by a pipeline error

[viktorprogger]

Problem

When tests are Aborted (an interceptor throws during the pipeline — e.g. a DI container cannot resolve a dependency), the terminal renderer correctly prints FAILED and counts aborted tests as failures in the summary. However, the process exits with code 0.

Root cause: Status::isFailure() did not include Status::Aborted, so the failure flag was never propagated up the aggregation chain:

TestRunner (Aborted result)
  → CaseRunner: isFailure() == false → case stays Passed
    → SuiteRunner: isFailure() == false → suite stays Passed
      → Application: isFailure() == false → overall status stays Passed
        → Run command: isSuccessful() == true → exit 0

The renderer (TerminalLogger) handles this correctly already — it explicitly includes Aborted when computing the failure count for the final banner. The exit code logic does not.

Fix

Add self::Aborted to the isFailure() match in Status.php. This single change cascades through the entire aggregation chain and produces exit code 1 whenever any test is aborted.

Behaviour after fix

Scenario	Before	After
All tests pass	exit 0	exit 0
Some tests fail (assertion)	exit 1	exit 1
Some tests aborted (interceptor exception)	exit 0 ❌	exit 1 ✅

[roxblnfk]

Thanks for spotting this and pushing a fix! I pushed a revised approach to this branch — sharing the reasoning here.

We agree on the goal: an aborted test (a critical interceptor failure — unknown verdict) must fail the run and exit with code 1, matching what the terminal / JUnit / TeamCity reporters already render.

Why not add Aborted to Status::isFailure(): that predicate is overloaded. Besides driving pass/fail aggregation, it's what RetryPolicyRunInterceptor checks to decide whether to retry a test:

if ($result->status->isFailure()) { /* retry */ }

An aborted test must not be retried — retrying won't fix a broken pipeline, and the verdict is unknown. Adding Aborted to isFailure():

• makes retry start retrying aborts, breaking the statusTriggersRetry unit test (#[DataSet([Status::Aborted, false])] in plugin/retry/tests/Unit/RetryPolicyRunInterceptorTest.php);
• also changes DataProvider/Inline/Bench aggregation and JsonReport failure listing, and shifts the public meaning of isFailure() for plugin authors.

What this branch does instead — escalate at the aggregation point, leaving isFailure() untouched:

// CaseRunner::run()
($result->status->isFailure() || $result->status === Status::Aborted)
    and $status = Status::Failed;

An aborted test now fails its case, which propagates case → suite → run → exit code through the existing chain. Retry/repeat semantics stay correct (an abort is still not retried).

Extras included:

• Aborted gets its own red A glyph in the terminal (verbose + dots) so it reads distinctly from a genuine Error (E).
• A self-test PipelineFailureStatusTest + a small stub apparatus (an interceptor that throws at a chosen pipeline stage — before/after $next(), at test and case level) that locks in the behaviour for both test-level aborts and case-level failures.

Verified: the new self-test (5/5) plus the retry, repeat and output suites (275/275) are green; retry is not broken.

Note: the PR title/description still mention isFailure() — worth updating to reflect the CaseRunner approach.

[codecov]

Codecov Report

❌ Patch coverage is 0% with 6 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
core/Output/Terminal/Renderer/Formatter.php	0.00%	4 Missing ⚠️
core/Application/Internal/Runner/CaseRunner.php	0.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!