feat(#396): name every test that finished having run zero assertions - #846
Merged
Conversation
#396's last unticked item, and the one it calls highest-value: "a spec that runs zero assertions is visibly different from one that runs twelve". Every instance in that catalogue shares one symptom — a green result that measured nothing — and every one was found by a person becoming suspicious of a specific test. This makes the symptom visible without anyone having to suspect anything first. Not hypothetical. Two specs found on 2026-08-20 guarded every assertion behind `if (elements.length > 0)` while visiting a page with none of those elements (#842); fixing them immediately exposed a real 40px touch target on production. Both would have printed here as zero-assertion tests the first time they ran. WHY A REPORTER AND NOT THE JSON OUTPUT. Checked before building: the `json` reporter's result objects carry annotations, attachments, duration, errors, status and stdout — and NO `steps`. Assertion counts are reachable only through the reporter API's step callbacks, so post-processing `results.json` cannot do this. IT ONLY PRINTS, deliberately, following this repo's own pattern of landing a gate in annotate mode first (E2E_BUDGET_MODE, FLAKY_GATE_MODE). A zero-assertion test is not always a defect — a spec may assert via `toPass`, a fixture, or a thrown helper — so failing on it today would redden the REQUIRED lane for reasons nobody has triaged. Turn it into a gate once the list is known and empty. Measured on a real batch before wiring it in: 19 passing tests across the mobile and colorblind specs, all of which asserted. The suite is in better shape than the catalogue implies, largely because #843 fixed the two that were not — so this ships producing a clean signal rather than a backlog of noise. THE REPORTER REFUSES TO GIVE A CLEAN BILL OF HEALTH HAVING OBSERVED NOTHING. If a run sees no tests — a bad shard filter, a crashed setup — it says so rather than printing "all good". A tool built to detect this family committing it would be its own #396 entry, and it is asserted, not just intended. Mutation-verified, mutant confirmed present in the file first, three ways: counting every step instead of only `expect` steps, disabling the observed-nothing guard, and not recording zeros so silent tests vanish from the map. Each is caught. Stated in the file so a green run is not over-read: it cannot see assertions made by a raw `throw` inside a helper, and a nonzero count does not prove a test asserts anything USEFUL. vitest 4724/4724 (447 files); test:scripts 424/424; type-check and lint clean. Refs #396
TortoiseWolfe
added a commit
that referenced
this pull request
Aug 20, 2026
PR #846 added a reporter that names any test finishing with zero assertions, wired it into `playwright.config.ts`, verified it locally, mutation-tested it three ways, and merged it. **It printed nothing in CI.** `playwright test --reporter=X` REPLACES the config's `reporter` array. It does not append. Every lane passes one — `line,json` in e2e-local, `list` and `blob` across e2e.yml — so the reporter ran in the two lanes that do not override (smoke, signup-mailer) and in NONE of the 25 shards on the required lane. That is the exact family #396 catalogues — a control whose presence was asserted and whose effect never was — committed by the tool built to detect it. It was caught by grepping the merged run's logs for the reporter's own output and getting nothing back, which is the only check that could have found it: everything else about the change was green. Fixed at all 8 overriding invocations (1 in e2e-local, 7 in e2e.yml), appending the reporter to each existing list rather than replacing it. `merge-reports` lines are left alone — merging a report is not running the suite. GUARDED, because the config and the workflows are different files with nothing relating them and the failure is silent in the worst way: the reporter is present, correct, unit-tested and mute, and nothing goes red. `scripts/__tests__/reporter-is-not-inert.test.js` asserts every `--reporter=` override still names it, with both halves of non-vacuity — the reporter file must exist, and several overrides must be found, since "every override includes it" is trivially true of zero overrides. Mutation-verified, mutant confirmed present first, two ways: reverting e2e-local to the exact string that shipped inert (caught, naming `e2e-local.yml:429`), and deleting the reporter file (caught by the existence check). Worth stating for the next person: the local verification was not wrong, it was incomplete. Running `playwright test --reporter=<the reporter>` by hand proves the reporter WORKS; it proves nothing about whether CI invokes it. Those are different questions and only the second one mattered here. test:scripts 428/428; type-check and lint clean. Refs #396 Co-authored-by: TurtleWolfe <TurtleWolfe@users.noreply.github.com>
TortoiseWolfe
added a commit
that referenced
this pull request
Aug 20, 2026
…px submit shipped (#848) The zero-assertion reporter (#846/#847) produced its first real CI output, and it named `mobile-buttons.spec.ts:35 › All buttons meet 44x44px minimum on mobile` among 31 tests that ran no assertions across the required lane. Investigating that one found a live defect rather than a broken test. The test is fine. It visits `/`, whose three visible `.btn` elements all pass — so it correctly asserted nothing and went green. Meanwhile the CONTACT FORM'S PRIMARY ACTION renders at 40px: `btn btn-primary` with no height floor, measured 139x40 on production at 390px, below the 44px minimum CLAUDE.md mandates. A gate is only as wide as what it points at (#411). This one pointed at one route. Two changes: - `min-h-11` on the submit button. Measured 44px after, locally, on a restarted container with the page proven loaded (`h1 = "Get in Touch"`, not a 404). - `mobile-buttons.spec.ts` now sweeps `/`, `/contact/` and `/blog/`, each earning its place by holding buttons the others do not: nav/hero chrome, a form submit, card actions. AND THE ASSERTION IS NOW UNCONDITIONAL. It read: if (failures.length > 0) { expect(failures.length, ...).toBe(0); } which is why it reported zero assertions when passing. That shape is not wrong — it fails correctly when a button IS too small — but it makes "found nothing wrong" indistinguishable from "measured nothing" in every report. It now asserts `failures` is empty unconditionally, plus a COVERAGE FLOOR: at least one visible `.btn` per route, so a selector change or an empty route fails loudly instead of passing. Verified against production, which still carries the defect: Error: 1 button(s) below 44px: /contact/ — "Send Message": 139x40px and mutation-verified with the mutant confirmed present first: pointing the locator at `.btn-does-not-exist` fails on the coverage floor ("this gate measured nothing") rather than passing green, which is what the old shape would have done. Worth noting for the other 30 names on that list: they are NOT all defects. Several use `checkA11y()`, which throws rather than calling `expect`, so they assert without a countable step — the reporter's documented blind spot. The list is a lead, not a verdict. ContactForm 33/33; test:scripts 428/428; type-check and lint clean. Refs #396 Co-authored-by: TurtleWolfe <TurtleWolfe@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
#396's last unticked item, and the one it calls highest-value: "a spec that runs zero assertions is visibly different from one that runs twelve."
Every instance in that catalogue shares one symptom — a green result that measured nothing — and every one was found by a person becoming suspicious of a specific test. This makes the symptom visible without anyone having to suspect anything first.
Not hypothetical. Two specs found yesterday guarded every assertion behind
if (elements.length > 0)while visiting a page with none of those elements (#842). Fixing them immediately exposed a real 40px touch target on production. Both would have printed here as zero-assertion tests the first time they ran.Why a reporter, not the JSON output
Checked before building it: the
jsonreporter's result objects carryannotations,attachments,duration,errors,status,stdout— and nosteps. Assertion counts are reachable only through the reporter API's step callbacks, so post-processingresults.jsoncan't do this.It only prints
Deliberately, following this repo's own pattern of landing a gate in annotate mode first (
E2E_BUDGET_MODE,FLAKY_GATE_MODE).A zero-assertion test isn't always a defect — a spec may assert via
toPass, a fixture, or a thrown helper — so failing on it today would redden the required lane for reasons nobody has triaged. Turn it into a gate once the list is known and empty.Measured on a real batch before wiring it in: 19 passing tests across the mobile and colorblind specs, all of which asserted. The suite is in better shape than the catalogue implies, largely because #843 fixed the two that weren't — so this ships producing a clean signal rather than a backlog of noise.
It refuses to give a clean bill of health having observed nothing
If a run sees no tests — a bad shard filter, a crashed setup — it says "proves nothing" rather than "all good". A tool built to detect this family committing it would be its own #396 entry. That's asserted, not just intended.
Verification
Mutation-verified with the mutant confirmed present in the file first, three ways:
expectsteps → caughtvitest 4724/4724 (447 files);
test:scripts424/424;type-checkandlintclean.Stated in the file so a green run isn't over-read: it cannot see assertions made by a raw
throwinside a helper, and a nonzero count does not prove a test asserts anything useful.Refs #396