fix(#396): the zero-assertion reporter printed nothing in CI - #847
Merged
Conversation
PR #846 added a reporter that names any test finishing with zero assertions, wired it into `playwright.config.ts`, verified it locally, mutation-tested it three ways, and merged it. **It printed nothing in CI.** `playwright test --reporter=X` REPLACES the config's `reporter` array. It does not append. Every lane passes one — `line,json` in e2e-local, `list` and `blob` across e2e.yml — so the reporter ran in the two lanes that do not override (smoke, signup-mailer) and in NONE of the 25 shards on the required lane. That is the exact family #396 catalogues — a control whose presence was asserted and whose effect never was — committed by the tool built to detect it. It was caught by grepping the merged run's logs for the reporter's own output and getting nothing back, which is the only check that could have found it: everything else about the change was green. Fixed at all 8 overriding invocations (1 in e2e-local, 7 in e2e.yml), appending the reporter to each existing list rather than replacing it. `merge-reports` lines are left alone — merging a report is not running the suite. GUARDED, because the config and the workflows are different files with nothing relating them and the failure is silent in the worst way: the reporter is present, correct, unit-tested and mute, and nothing goes red. `scripts/__tests__/reporter-is-not-inert.test.js` asserts every `--reporter=` override still names it, with both halves of non-vacuity — the reporter file must exist, and several overrides must be found, since "every override includes it" is trivially true of zero overrides. Mutation-verified, mutant confirmed present first, two ways: reverting e2e-local to the exact string that shipped inert (caught, naming `e2e-local.yml:429`), and deleting the reporter file (caught by the existence check). Worth stating for the next person: the local verification was not wrong, it was incomplete. Running `playwright test --reporter=<the reporter>` by hand proves the reporter WORKS; it proves nothing about whether CI invokes it. Those are different questions and only the second one mattered here. test:scripts 428/428; type-check and lint clean. Refs #396
TortoiseWolfe
added a commit
that referenced
this pull request
Aug 20, 2026
…px submit shipped (#848) The zero-assertion reporter (#846/#847) produced its first real CI output, and it named `mobile-buttons.spec.ts:35 › All buttons meet 44x44px minimum on mobile` among 31 tests that ran no assertions across the required lane. Investigating that one found a live defect rather than a broken test. The test is fine. It visits `/`, whose three visible `.btn` elements all pass — so it correctly asserted nothing and went green. Meanwhile the CONTACT FORM'S PRIMARY ACTION renders at 40px: `btn btn-primary` with no height floor, measured 139x40 on production at 390px, below the 44px minimum CLAUDE.md mandates. A gate is only as wide as what it points at (#411). This one pointed at one route. Two changes: - `min-h-11` on the submit button. Measured 44px after, locally, on a restarted container with the page proven loaded (`h1 = "Get in Touch"`, not a 404). - `mobile-buttons.spec.ts` now sweeps `/`, `/contact/` and `/blog/`, each earning its place by holding buttons the others do not: nav/hero chrome, a form submit, card actions. AND THE ASSERTION IS NOW UNCONDITIONAL. It read: if (failures.length > 0) { expect(failures.length, ...).toBe(0); } which is why it reported zero assertions when passing. That shape is not wrong — it fails correctly when a button IS too small — but it makes "found nothing wrong" indistinguishable from "measured nothing" in every report. It now asserts `failures` is empty unconditionally, plus a COVERAGE FLOOR: at least one visible `.btn` per route, so a selector change or an empty route fails loudly instead of passing. Verified against production, which still carries the defect: Error: 1 button(s) below 44px: /contact/ — "Send Message": 139x40px and mutation-verified with the mutant confirmed present first: pointing the locator at `.btn-does-not-exist` fails on the coverage floor ("this gate measured nothing") rather than passing green, which is what the old shape would have done. Worth noting for the other 30 names on that list: they are NOT all defects. Several use `checkA11y()`, which throws rather than calling `expect`, so they assert without a countable step — the reporter's documented blind spot. The list is a lead, not a verdict. ContactForm 33/33; test:scripts 428/428; type-check and lint clean. Refs #396 Co-authored-by: TurtleWolfe <TurtleWolfe@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug is mine, from two hours ago
PR #846 added a reporter that names any test finishing with zero assertions. I wired it into
playwright.config.ts, verified it locally, mutation-tested it three ways, and merged it.It printed nothing in CI.
playwright test --reporter=Xreplaces the config'sreporterarray — it does not append. Every lane passes one (line,jsonin e2e-local,listandblobacross e2e.yml), so the reporter ran in the two lanes that don't override (smoke, signup-mailer) and in none of the 25 shards on the required lane.That's the exact family #396 catalogues — a control whose presence was asserted and whose effect never was — committed by the tool built to detect it.
How it was caught
By grepping the merged run's logs for the reporter's own output and getting nothing back. That was the only check that could have found it: the reporter file was present, correct, unit-tested at 5/5, type-clean, and merged green.
The local verification wasn't wrong, it was incomplete. Running
playwright test --reporter=<the reporter>by hand proves the reporter works. It proves nothing about whether CI invokes it. Two different questions, and only the second one mattered.The fix
All 8 overriding invocations — 1 in
e2e-local.yml, 7 ine2e.yml— now append the reporter to their existing list rather than replacing it.merge-reportslines are left alone; merging a report isn't running the suite.The guard
scripts/__tests__/reporter-is-not-inert.test.jsasserts every--reporter=override still names it.The config and the workflows are different files with nothing relating them, and the failure is silent in the worst way: present, correct, unit-tested, and mute, with nothing going red. This relates them.
Both halves of non-vacuity are asserted — the reporter file must exist, and several overrides must be found, because "every override includes it" is trivially true of zero overrides.
Verification
Mutation-verified with the mutant confirmed present first:
e2e-localto the exact string that shipped inert → caught, naminge2e-local.yml:429test:scripts428/428;type-checkandlintclean.Refs #396