Work queue: E2E tests that finish having run zero assertions (#396) — 6 auth-gated defects left

[TortoiseWolfe]

DONE. All 15 defects in this queue are fixed, deleted or pinned. What still appears in
the reporter's output is the 8 already-triaged non-defects listed below — 4 checkA11y
blind spots and 4 correct assert-on-failure tests. Shipped across #848, #849, #851, #852,
#853, #854, #858 and #860.

The single most valuable thing this queue produced: #859. Making one test assert
revealed that "Delete Account Permanently" deletes nothing — user_profiles has no DELETE
RLS policy, so the browser's delete matches zero rows, returns no error, and the service
reports success while the profile, auth record and email address all remain.

Follow-ups carried forward: #861 (make the reporter a gate) and #862 (the one
messaging test this queue could never settle).

The #846 reporter (made to actually run in CI by #847) counts assertions per test and names the ones that finish having run zero. Its first full run across the required lane found 31; after seven fixes that night the regenerated list was 24. This issue is the queue, and the triage is now complete.

Status: 23 of 24 settled, 1 blocked

verdict	n	which
real defect	15	avatar-upload 321/394/411 · protected-routes 199/259 · gdpr-consent 148 · accessibility 437 · form-submission 69/177/222/262/282/316 · mobile-typography 216 · pwa-installation 120
false positive	4	accessibility 23/34/43/64 — checkA11y() throws instead of calling expect, so it asserts and the reporter cannot see it. Documented blind spot.
correct as written	4	blog-touch-targets 26 · cross-page-nav 166, 187 (says "if present", and there are no breadcrumbs) · mobile-typography 160
blocked	1	complete-user-workflow.spec.ts:232

Shipped so far: #848, #849, #851, #852, #853, #854 and #858.

All nine public-route defects are now fixed. The six that remain are auth-gated and are
the only unfixed defects left in this queue: avatar-upload 321/394/411, protected-routes
199/259, gdpr-consent 148. CI still names them every run, which is the queue working.

What the whole queue turned out to be

Every one of these tests asks the page about a state the test never puts it into, or about a feature the product never had. The assertions sit behind if (…) guards that are false on every run, so the test passes by never looking. That is one sentence, but it has three different cures, and picking the wrong one is how a real gap gets papered over.

Group 1 — the app HAS the behaviour, the test never triggers it

Real coverage holes. Fixed in #854.

• form-submission:69 — react-hook-form is mode:'onSubmit' (ContactForm.tsx:31) and every -error label renders under {errors.X && …}. No error exists until a submit is attempted. The test filled a field with '' and pressed Tab, so [id$="-error"] was always 0.
• accessibility:437 — same shape on /sign-in, plus worse: SignInForm has no field-level association at all (one form-level role="alert" with no id), and both inputs carry native required, so an empty submit never reaches the React handler. Repointed to /contact. The /sign-in gap is now Sign-in and sign-up errors are never associated with the field they describe #857.
• form-submission:282 — dead twice over. Waited for a response URL containing /api/ (static export, no API routes; Web3Forms is api.web3forms.com/submit, which contains //api. not /api/) and clicked submit on an empty form, so validation blocked the request first.
• mobile-typography:216 — guarded on box.width > 390, but boundingBox() returns the border box, so a container with overflow-x: auto can never trip it. Also backwards: it asked whether wide containers handle overflow, not whether content overflows.

Group 2 — the behaviour does not exist

The question worth asking here, and the one that was nearly skipped: was it ever planned? The answers differ, and so do the right responses.

feature	evidence	outcome
multi-step forms	Out of Scope in docs/specs/009-web3forms-integration/spec.md:39, docs/prp-docs/SPEC.md:3805, and features/integrations/022-web3forms-integration/spec.md:204	deleted — a test for something we decided not to build
form persistence	no issue, no spec, no form_* key in src/	deleted — and the real capability is the offline queue, a different mechanism
disabled fields	no spec; zero <input disabled> in src/	deleted — not a feature, a state; the real behaviour is the submit button, now covered by :282
help text	FormField.tsx renders ${name}-help and has shipped since the initial commit — imported by nobody, helpText passed nowhere	kept — it exposed a genuine UX gap, filed as #855; the component itself is #856

form-submission:177 was deliberately not deleted, and that judgement paid off: #855 shipped
the help text in #858, and the test now asserts a real subject — count exactly 2, and each control
must point at its own hint. Deleting it in #854 only to re-add it would have been churn, and the
UX gap it was pointing at would have gone unrecorded.

Group 3 — an honest skip wearing a pass

pwa-installation:120 returned early when the service worker didn't activate, which Playwright reports as passed. Now test.skip() with a reason.

The rule that prevents recurrence

A guard clause in a test should be an assertion. if (n > 0) { expect(…) } should be await expect(x).toHaveCount(n) followed by the assertions. That is the difference between "silently measured nothing" and "told you the page changed" — the same lesson as the coverage floors in #843 and #851.

Watch for tautologies too. expect(alt).toBeDefined() on a getAttribute result was one (#849): the call returns null for a missing attribute, and null is defined, so an image with no alt passed. The same shape hides in toBeTruthy() on a count and toBeGreaterThanOrEqual(0).

The one still blocked

tests/e2e/messaging/complete-user-workflow.spec.ts:232 fails in a statically-served harness with page.waitForSelector: Timeout 60000ms — messaging needs realtime plumbing a static build does not provide. Its assertion count is meaningless while the test is red, so no verdict is recorded.

It also runs under chromium-msg-iso — not chromium-gen, and not chromium-msg either, which explicitly ignores it via MSG_ISO_GLOBS. Running it under the wrong project yields "No tests found", which the reporter correctly reports as "observed no tests; this run proves nothing about assertion coverage" rather than as a pass. That non-vacuity line caught a wrong "acquitted" inference during this triage and is worth keeping.

Next: make the reporter a gate

The reporter is report-only. Once this queue is empty the natural endgame is failing the lane on a zero-assertion test — but that needs the four checkA11y cases handled first (an allowlist, or teaching the reporter to count a thrown helper), otherwise the required lane goes red on four correct tests. Filed separately when the queue reaches zero.

How to regenerate the list

gh run view <id> --log truncates badly — 48 KB for a 26-job run versus 2.2 MB for a single job. Never conclude from it. Fetch per job:

gh run view <rid> --json jobs --jq '.jobs[]|select(.conclusion=="success")|"\(.databaseId) \(.name)"'
gh api "repos/TortoiseWolfe/ScriptHammer/actions/jobs/<jid>/logs" \
  | sed -n '/\[assertion-count\] [0-9]* of [0-9]* passing/,/Reporting only/p'