Skip to content

Stop resolution.Empty racing a store scan, and stop tests writing rows into the reports - #153

Merged
kyleve merged 7 commits into
mainfrom
cursor/fix-resolution-empty-snapshot-race-b2df
Jul 29, 2026
Merged

Stop resolution.Empty racing a store scan, and stop tests writing rows into the reports#153
kyleve merged 7 commits into
mainfrom
cursor/fix-resolution-empty-snapshot-race-b2df

Conversation

@kyleve

@kyleve kyleve commented Jul 28, 2026

Copy link
Copy Markdown
Owner

Fixes the red main (run 30402846712), plus a reporting bug found while diagnosing it. The snapshot job's only real failure was ResolutionViewSnapshotTests/resolution(), on Empty_iPhone, at max channel delta 255 over 91.7% of the image. Everything else in that run was noise the new tolerance absorbs or the already-known swiftDataInspector issue.

Merged with #150 (demo mode) — see Merging alongside #150 for the one conflict and why both sides had to survive it.

1. resolution.Empty never rendered the empty state

PreviewSupport.resolveModel(seededWithIssues: false) skipped setDataIssues entirely, so the fixture came back with hasLoaded == false — which ResolutionView deliberately cannot distinguish from "the first scan hasn't landed yet". The view therefore held AppIconLoadingView until its .task(id:) scan of the empty in-memory store returned, which found the whole year missing. So the committed reference was that scan's output — a populated list titled "Missing days / Jan 1 – Jul 14 / 195 days" — not the all-clear state the case is named for, and every capture was a race against an async store read. CI lost that race and baked the placeholder.

The settle loop can't catch this, which is why the suite had been reporting green: a placeholder is perfectly pixel-stable, so the loop proves stability and the capture proceeds. Same failure class as root.LoggedIn baking the launch splash, already ledgered in Where/TODOs.md. It stayed hidden until now because drainInFlightAnimations used to waste ~1s per capture; removing that waste in #151 took away the accidental slack. The prior main run confirms the shape — resolution() passed there in 23.0s, and failed here in 12.1s.

Fix. Both fixture modes now seed. setDataIssues([]) is what marks the model loaded and isSeeded, so the view's load(...) is a no-op and the first rendered frame is already final. The case no longer depends on the store, on now, or on how fast the machine is — there is no phase transition left to race. The two references are re-recorded to the "All clear" state, coverage the suite never had (WithIssues already pins the populated list), and consistent with the app's other empty states.

ResolveModelTests gains two guards, both deterministic and independent of the image suite: the fixture comes back loaded in both modes, and load(...) leaves a seeded fixture alone against a store whose scan does find issues — the short-circuit the empty case's determinism now rests on.

2. The reporting tests were writing rows into the reports

Found while explaining a suspicious line in ./test --review. ./test recovers SNAPSHOT_DIFF and SNAPSHOT_TIMING by grepping them out of the run logs, and counts timing lines as captured images for the progress line. Both were encoded and printed by a single function, so the tests that pin those wire formats emitted genuine lines:

  • --review listed a reference that does not exist, at the top of the list, because the fixture's numbers were borrowed from the real swiftDataInspector regression (max delta 203, 7430 pixels, 0.235%). The row most worth investigating was the one that wasn't real.
  • --timings counted five captures that never happened. ./test --only SnapshotKitTestingTests --timings reported "5 captures, 0.1s total, 0.024s per image" for a run that captured nothing, naming fixtures (repeated, one-phase, mixed, …) in its slowest-captures list. On --everything --timings those blended into the aggregate the module's performance decisions are read off.

The SNAPSHOT_DIFF env gate never covered this: it's checked by the pipeline before calling report, not inside it.

Fix. Each channel is split so printing belongs to the pipeline alone and the payload is separately askable — SnapshotDiffReporting.line(describing:), SnapshotCaptureTiming.line(), and SnapshotSettleReporting.line(...) return the same JSON without emitting, and all four test sites call those. The settle channel gets the same treatment for symmetry; nothing aggregates it today, but it's one scraper away from the same bug. The rule is now an invariant in AGENTS.md, since it's enforced by convention rather than by the compiler.

Verified in both directions: a unit-only run reports no timing lines and no differing captures, while the full suite still prints the complete phase breakdown — now 264 captures rather than 269 — and the diff table.

Merging alongside #150

#150 edited the adjacent lines of the same function for an unrelated reason, so neither side could be taken wholesale:

The merge keeps both, and loadLeavesASeededFixtureAlone uses the makePreferences() helper #150 added to the test file.

Docs

SnapshotKitTesting's README claimed .settled "waits for pixel-stable renders so .task-driven content loads" — the over-promise that let the first bug land. It now says pixel stability gives async content time to load but cannot certify that it did. AGENTS.md gains two invariants: a settled capture is not a ready capture (seed the fixture or gate on onReadyToSnapshot), and only the pipeline may print a report channel.

Testing

Run locally on iPhone 17 / iOS 27.0, on the merged tree:

  • ./test --everything --review --timings — 1365 unit tests and 30 snapshot tests / 264 images, all passing; every remaining diff is max delta 1–2 sub-visible drift and no fabricated rows appear in either report
  • both reporting fixes verified from both sides: unit-only runs report nothing, real snapshot runs report fully
  • swift run bumper config/test/lint . — no architecture violations
  • ./swiftformat --lint and ./xcstrings --lint clean
Open in Web Open in Cursor 

kyleve added 7 commits July 28, 2026 15:37
`resolution.Empty` never rendered the empty state, and its capture raced a
live `DataIssueScanner` pass — which turned main red (run 30402846712) the
first time CI lost that race, baking the `AppIconLoadingView` placeholder
over 91.7% of `Empty_iPhone`.

`PreviewSupport.resolveModel(seededWithIssues: false)` skipped
`setDataIssues` entirely, so the fixture came back with `hasLoaded == false`
— which `ResolutionView` deliberately can't distinguish from "the first scan
hasn't landed" — and the view held the placeholder until its `.task(id:)`
scan of the empty in-memory store returned the whole year as missing days.
So the *reference* was that scan's output, a populated list titled "Missing
days", not the all-clear state the case is named for.

The settle loop can't catch this: a placeholder is pixel-stable, so the
capture settles clean and the suite reports green either way. It is the same
class as `root.LoggedIn` baking the launch splash, and it was masked until
now by the ~1s that `drainInFlightAnimations` wasted per capture.

Both fixture modes now seed. `setDataIssues([])` is what marks the model
loaded *and* `isSeeded`, so the view's `load(...)` is a no-op and the first
rendered frame is final — the case no longer depends on the store, on `now`,
or on how fast the machine is. The two references are re-recorded to the
"All clear" state, which the suite never covered before, since `WithIssues`
already pins the populated list.

`ResolveModelTests` gains two guards: the fixture is loaded up front in both
modes, and `load(...)` leaves a seeded fixture alone against a store whose
scan does find issues. The pipeline docs now say that a settled capture is
not a ready capture, rather than implying pixel stability certifies a
`.task` landed.
`PreviewSupport` documents that none of its fixtures touch disk, but these
two built `WherePreferences()`, which defaults to `UserDefaults.standard`.
`RemindersSettingsModel` is the one that matters: every toggle and time
picker on the alerts screen assigns through to the store, so a preview or a
hosted test could rewrite the running app's reminder settings.

Same treatment the onboarding fixture already got.
`./test --everything --review` lists the reporting fixture's synthetic
`/refs/thing.case_dark.png` among real differing captures, and it sorts
first because its numbers replay a real regression.
The only conflict is `PreviewSupport.resolveModel`, where both sides edited
adjacent lines for unrelated reasons. Kept both: main's
`previewPreferences()` (its sweep of fixtures that reached the host's real
`UserDefaults`) and this branch's `setDataIssues(seededWithIssues ? … : [])`
(seeding both modes, so the empty case stops racing a live store scan).

Taking either side alone would have regressed the other — main still carried
the race, and this branch had reverted its own narrower version of the
preferences fix in favor of main's. `loadLeavesASeededFixtureAlone` now uses
the `makePreferences()` helper main added to the file.
Records the unit-only reproduction (a run that captures no images still
reports one differing capture), why the SNAPSHOT_DIFF env gate doesn't
prevent it, and the cleaner fix of splitting the print out of report().
`./test` recovers `SNAPSHOT_DIFF` and `SNAPSHOT_TIMING` by grepping them out
of the run logs, and counts timing lines as captured images for the progress
line. Both were encoded and printed by one function, so the tests pinning
those wire formats emitted real lines and the reports listed things that
never happened.

The diff fixture sorted to the *top* of `./test --review`, because its
numbers were borrowed from the genuine swiftDataInspector regression (max
delta 203, 7430 pixels, 0.235%) — so the row most worth investigating was
the one that wasn't real, pointing at a reference that does not exist.
`SnapshotCaptureTimingTests` contributed five invented captures: a run of
that bundle alone reported "5 captures, 0.1s total, 0.024s per image" having
captured nothing, and `--everything --timings` blended them into the
aggregate the suite's performance decisions are read off.

The `SNAPSHOT_DIFF` env gate never covered this — it is checked by the
pipeline before calling `report`, not inside it.

Each channel is now split: `report(...)` / `emit()` print, and
`line(describing:)` / `line()` return the same JSON without emitting. All
four test sites ask for the payload. `SnapshotSettleReporting` gets the same
split for symmetry — nothing aggregates that channel today, but it is one
scraper away from the same bug.

Verified both directions: a unit-only run now reports no timing lines and no
differing captures, while the full suite still prints the phase breakdown
(264 captures, down from 269 with the fakes) and the diff table.
@cursor cursor Bot changed the title Seed the empty Resolve fixture so its snapshot stops racing a store scan Stop resolution.Empty racing a store scan, and stop tests writing rows into the reports Jul 28, 2026
@kyleve
kyleve merged commit 0f9a253 into main Jul 29, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant