Skip to content

feat(#396): name every test that finished having run zero assertions - #846

Merged
TortoiseWolfe merged 1 commit into
mainfrom
feat/396-assertion-count-reporter
Aug 20, 2026
Merged

feat(#396): name every test that finished having run zero assertions#846
TortoiseWolfe merged 1 commit into
mainfrom
feat/396-assertion-count-reporter

Conversation

@TortoiseWolfe

Copy link
Copy Markdown
Owner

What this does

#396's last unticked item, and the one it calls highest-value: "a spec that runs zero assertions is visibly different from one that runs twelve."

Every instance in that catalogue shares one symptom — a green result that measured nothing — and every one was found by a person becoming suspicious of a specific test. This makes the symptom visible without anyone having to suspect anything first.

Not hypothetical. Two specs found yesterday guarded every assertion behind if (elements.length > 0) while visiting a page with none of those elements (#842). Fixing them immediately exposed a real 40px touch target on production. Both would have printed here as zero-assertion tests the first time they ran.

Why a reporter, not the JSON output

Checked before building it: the json reporter's result objects carry annotations, attachments, duration, errors, status, stdout — and no steps. Assertion counts are reachable only through the reporter API's step callbacks, so post-processing results.json can't do this.

It only prints

Deliberately, following this repo's own pattern of landing a gate in annotate mode first (E2E_BUDGET_MODE, FLAKY_GATE_MODE).

A zero-assertion test isn't always a defect — a spec may assert via toPass, a fixture, or a thrown helper — so failing on it today would redden the required lane for reasons nobody has triaged. Turn it into a gate once the list is known and empty.

Measured on a real batch before wiring it in: 19 passing tests across the mobile and colorblind specs, all of which asserted. The suite is in better shape than the catalogue implies, largely because #843 fixed the two that weren't — so this ships producing a clean signal rather than a backlog of noise.

It refuses to give a clean bill of health having observed nothing

If a run sees no tests — a bad shard filter, a crashed setup — it says "proves nothing" rather than "all good". A tool built to detect this family committing it would be its own #396 entry. That's asserted, not just intended.

Verification

Mutation-verified with the mutant confirmed present in the file first, three ways:

  • count every step instead of only expect steps → caught
  • disable the observed-nothing guard → caught
  • stop recording zeros, so silent tests vanish from the map → caught (2 tests)

vitest 4724/4724 (447 files); test:scripts 424/424; type-check and lint clean.

Stated in the file so a green run isn't over-read: it cannot see assertions made by a raw throw inside a helper, and a nonzero count does not prove a test asserts anything useful.

Refs #396

#396's last unticked item, and the one it calls highest-value: "a spec that runs zero
assertions is visibly different from one that runs twelve". Every instance in that
catalogue shares one symptom — a green result that measured nothing — and every one
was found by a person becoming suspicious of a specific test. This makes the symptom
visible without anyone having to suspect anything first.

Not hypothetical. Two specs found on 2026-08-20 guarded every assertion behind
`if (elements.length > 0)` while visiting a page with none of those elements (#842);
fixing them immediately exposed a real 40px touch target on production. Both would
have printed here as zero-assertion tests the first time they ran.

WHY A REPORTER AND NOT THE JSON OUTPUT. Checked before building: the `json` reporter's
result objects carry annotations, attachments, duration, errors, status and stdout —
and NO `steps`. Assertion counts are reachable only through the reporter API's step
callbacks, so post-processing `results.json` cannot do this.

IT ONLY PRINTS, deliberately, following this repo's own pattern of landing a gate in
annotate mode first (E2E_BUDGET_MODE, FLAKY_GATE_MODE). A zero-assertion test is not
always a defect — a spec may assert via `toPass`, a fixture, or a thrown helper — so
failing on it today would redden the REQUIRED lane for reasons nobody has triaged.
Turn it into a gate once the list is known and empty.

Measured on a real batch before wiring it in: 19 passing tests across the mobile and
colorblind specs, all of which asserted. The suite is in better shape than the
catalogue implies, largely because #843 fixed the two that were not — so this ships
producing a clean signal rather than a backlog of noise.

THE REPORTER REFUSES TO GIVE A CLEAN BILL OF HEALTH HAVING OBSERVED NOTHING. If a run
sees no tests — a bad shard filter, a crashed setup — it says so rather than printing
"all good". A tool built to detect this family committing it would be its own #396
entry, and it is asserted, not just intended.

Mutation-verified, mutant confirmed present in the file first, three ways: counting
every step instead of only `expect` steps, disabling the observed-nothing guard, and
not recording zeros so silent tests vanish from the map. Each is caught.

Stated in the file so a green run is not over-read: it cannot see assertions made by a
raw `throw` inside a helper, and a nonzero count does not prove a test asserts
anything USEFUL.

vitest 4724/4724 (447 files); test:scripts 424/424; type-check and lint clean.

Refs #396
@TortoiseWolfe
TortoiseWolfe merged commit 6041ca1 into main Aug 20, 2026
37 of 38 checks passed
@TortoiseWolfe
TortoiseWolfe deleted the feat/396-assertion-count-reporter branch August 20, 2026 05:50
TortoiseWolfe added a commit that referenced this pull request Aug 20, 2026
PR #846 added a reporter that names any test finishing with zero assertions, wired it
into `playwright.config.ts`, verified it locally, mutation-tested it three ways, and
merged it. **It printed nothing in CI.**

`playwright test --reporter=X` REPLACES the config's `reporter` array. It does not
append. Every lane passes one — `line,json` in e2e-local, `list` and `blob` across
e2e.yml — so the reporter ran in the two lanes that do not override (smoke,
signup-mailer) and in NONE of the 25 shards on the required lane.

That is the exact family #396 catalogues — a control whose presence was asserted and
whose effect never was — committed by the tool built to detect it. It was caught by
grepping the merged run's logs for the reporter's own output and getting nothing back,
which is the only check that could have found it: everything else about the change was
green.

Fixed at all 8 overriding invocations (1 in e2e-local, 7 in e2e.yml), appending the
reporter to each existing list rather than replacing it. `merge-reports` lines are
left alone — merging a report is not running the suite.

GUARDED, because the config and the workflows are different files with nothing
relating them and the failure is silent in the worst way: the reporter is present,
correct, unit-tested and mute, and nothing goes red.
`scripts/__tests__/reporter-is-not-inert.test.js` asserts every `--reporter=` override
still names it, with both halves of non-vacuity — the reporter file must exist, and
several overrides must be found, since "every override includes it" is trivially true
of zero overrides.

Mutation-verified, mutant confirmed present first, two ways: reverting e2e-local to
the exact string that shipped inert (caught, naming `e2e-local.yml:429`), and deleting
the reporter file (caught by the existence check).

Worth stating for the next person: the local verification was not wrong, it was
incomplete. Running `playwright test --reporter=<the reporter>` by hand proves the
reporter WORKS; it proves nothing about whether CI invokes it. Those are different
questions and only the second one mattered here.

test:scripts 428/428; type-check and lint clean.

Refs #396

Co-authored-by: TurtleWolfe <TurtleWolfe@users.noreply.github.com>
TortoiseWolfe added a commit that referenced this pull request Aug 20, 2026
…px submit shipped (#848)

The zero-assertion reporter (#846/#847) produced its first real CI output, and it named
`mobile-buttons.spec.ts:35 › All buttons meet 44x44px minimum on mobile` among 31 tests
that ran no assertions across the required lane. Investigating that one found a live
defect rather than a broken test.

The test is fine. It visits `/`, whose three visible `.btn` elements all pass — so it
correctly asserted nothing and went green. Meanwhile the CONTACT FORM'S PRIMARY ACTION
renders at 40px: `btn btn-primary` with no height floor, measured 139x40 on production
at 390px, below the 44px minimum CLAUDE.md mandates.

A gate is only as wide as what it points at (#411). This one pointed at one route.

Two changes:

  - `min-h-11` on the submit button. Measured 44px after, locally, on a restarted
    container with the page proven loaded (`h1 = "Get in Touch"`, not a 404).
  - `mobile-buttons.spec.ts` now sweeps `/`, `/contact/` and `/blog/`, each earning its
    place by holding buttons the others do not: nav/hero chrome, a form submit, card
    actions.

AND THE ASSERTION IS NOW UNCONDITIONAL. It read:

    if (failures.length > 0) {
      expect(failures.length, ...).toBe(0);
    }

which is why it reported zero assertions when passing. That shape is not wrong — it
fails correctly when a button IS too small — but it makes "found nothing wrong"
indistinguishable from "measured nothing" in every report. It now asserts
`failures` is empty unconditionally, plus a COVERAGE FLOOR: at least one visible `.btn`
per route, so a selector change or an empty route fails loudly instead of passing.

Verified against production, which still carries the defect:

    Error: 1 button(s) below 44px:
    /contact/ — "Send Message": 139x40px

and mutation-verified with the mutant confirmed present first: pointing the locator at
`.btn-does-not-exist` fails on the coverage floor ("this gate measured nothing")
rather than passing green, which is what the old shape would have done.

Worth noting for the other 30 names on that list: they are NOT all defects. Several use
`checkA11y()`, which throws rather than calling `expect`, so they assert without a
countable step — the reporter's documented blind spot. The list is a lead, not a verdict.

ContactForm 33/33; test:scripts 428/428; type-check and lint clean.

Refs #396

Co-authored-by: TurtleWolfe <TurtleWolfe@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants