Stop resolution.Empty racing a store scan, and stop tests writing rows into the reports - #153
Merged
Merged
Conversation
`resolution.Empty` never rendered the empty state, and its capture raced a live `DataIssueScanner` pass — which turned main red (run 30402846712) the first time CI lost that race, baking the `AppIconLoadingView` placeholder over 91.7% of `Empty_iPhone`. `PreviewSupport.resolveModel(seededWithIssues: false)` skipped `setDataIssues` entirely, so the fixture came back with `hasLoaded == false` — which `ResolutionView` deliberately can't distinguish from "the first scan hasn't landed" — and the view held the placeholder until its `.task(id:)` scan of the empty in-memory store returned the whole year as missing days. So the *reference* was that scan's output, a populated list titled "Missing days", not the all-clear state the case is named for. The settle loop can't catch this: a placeholder is pixel-stable, so the capture settles clean and the suite reports green either way. It is the same class as `root.LoggedIn` baking the launch splash, and it was masked until now by the ~1s that `drainInFlightAnimations` wasted per capture. Both fixture modes now seed. `setDataIssues([])` is what marks the model loaded *and* `isSeeded`, so the view's `load(...)` is a no-op and the first rendered frame is final — the case no longer depends on the store, on `now`, or on how fast the machine is. The two references are re-recorded to the "All clear" state, which the suite never covered before, since `WithIssues` already pins the populated list. `ResolveModelTests` gains two guards: the fixture is loaded up front in both modes, and `load(...)` leaves a seeded fixture alone against a store whose scan does find issues. The pipeline docs now say that a settled capture is not a ready capture, rather than implying pixel stability certifies a `.task` landed.
`PreviewSupport` documents that none of its fixtures touch disk, but these two built `WherePreferences()`, which defaults to `UserDefaults.standard`. `RemindersSettingsModel` is the one that matters: every toggle and time picker on the alerts screen assigns through to the store, so a preview or a hosted test could rewrite the running app's reminder settings. Same treatment the onboarding fixture already got.
`./test --everything --review` lists the reporting fixture's synthetic `/refs/thing.case_dark.png` among real differing captures, and it sorts first because its numbers replay a real regression.
This reverts commit fc4aace.
The only conflict is `PreviewSupport.resolveModel`, where both sides edited adjacent lines for unrelated reasons. Kept both: main's `previewPreferences()` (its sweep of fixtures that reached the host's real `UserDefaults`) and this branch's `setDataIssues(seededWithIssues ? … : [])` (seeding both modes, so the empty case stops racing a live store scan). Taking either side alone would have regressed the other — main still carried the race, and this branch had reverted its own narrower version of the preferences fix in favor of main's. `loadLeavesASeededFixtureAlone` now uses the `makePreferences()` helper main added to the file.
Records the unit-only reproduction (a run that captures no images still reports one differing capture), why the SNAPSHOT_DIFF env gate doesn't prevent it, and the cleaner fix of splitting the print out of report().
`./test` recovers `SNAPSHOT_DIFF` and `SNAPSHOT_TIMING` by grepping them out of the run logs, and counts timing lines as captured images for the progress line. Both were encoded and printed by one function, so the tests pinning those wire formats emitted real lines and the reports listed things that never happened. The diff fixture sorted to the *top* of `./test --review`, because its numbers were borrowed from the genuine swiftDataInspector regression (max delta 203, 7430 pixels, 0.235%) — so the row most worth investigating was the one that wasn't real, pointing at a reference that does not exist. `SnapshotCaptureTimingTests` contributed five invented captures: a run of that bundle alone reported "5 captures, 0.1s total, 0.024s per image" having captured nothing, and `--everything --timings` blended them into the aggregate the suite's performance decisions are read off. The `SNAPSHOT_DIFF` env gate never covered this — it is checked by the pipeline before calling `report`, not inside it. Each channel is now split: `report(...)` / `emit()` print, and `line(describing:)` / `line()` return the same JSON without emitting. All four test sites ask for the payload. `SnapshotSettleReporting` gets the same split for symmetry — nothing aggregates that channel today, but it is one scraper away from the same bug. Verified both directions: a unit-only run now reports no timing lines and no differing captures, while the full suite still prints the phase breakdown (264 captures, down from 269 with the fakes) and the diff table.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the red
main(run 30402846712), plus a reporting bug found while diagnosing it. The snapshot job's only real failure wasResolutionViewSnapshotTests/resolution(), onEmpty_iPhone, at max channel delta 255 over 91.7% of the image. Everything else in that run was noise the new tolerance absorbs or the already-knownswiftDataInspectorissue.Merged with #150 (demo mode) — see Merging alongside #150 for the one conflict and why both sides had to survive it.
1.
resolution.Emptynever rendered the empty statePreviewSupport.resolveModel(seededWithIssues: false)skippedsetDataIssuesentirely, so the fixture came back withhasLoaded == false— whichResolutionViewdeliberately cannot distinguish from "the first scan hasn't landed yet". The view therefore heldAppIconLoadingViewuntil its.task(id:)scan of the empty in-memory store returned, which found the whole year missing. So the committed reference was that scan's output — a populated list titled "Missing days / Jan 1 – Jul 14 / 195 days" — not the all-clear state the case is named for, and every capture was a race against an async store read. CI lost that race and baked the placeholder.The settle loop can't catch this, which is why the suite had been reporting green: a placeholder is perfectly pixel-stable, so the loop proves stability and the capture proceeds. Same failure class as
root.LoggedInbaking the launch splash, already ledgered inWhere/TODOs.md. It stayed hidden until now becausedrainInFlightAnimationsused to waste ~1s per capture; removing that waste in #151 took away the accidental slack. The priormainrun confirms the shape —resolution()passed there in 23.0s, and failed here in 12.1s.Fix. Both fixture modes now seed.
setDataIssues([])is what marks the model loaded andisSeeded, so the view'sload(...)is a no-op and the first rendered frame is already final. The case no longer depends on the store, onnow, or on how fast the machine is — there is no phase transition left to race. The two references are re-recorded to the "All clear" state, coverage the suite never had (WithIssuesalready pins the populated list), and consistent with the app's other empty states.ResolveModelTestsgains two guards, both deterministic and independent of the image suite: the fixture comes back loaded in both modes, andload(...)leaves a seeded fixture alone against a store whose scan does find issues — the short-circuit the empty case's determinism now rests on.2. The reporting tests were writing rows into the reports
Found while explaining a suspicious line in
./test --review../testrecoversSNAPSHOT_DIFFandSNAPSHOT_TIMINGby grepping them out of the run logs, and counts timing lines as captured images for the progress line. Both were encoded and printed by a single function, so the tests that pin those wire formats emitted genuine lines:--reviewlisted a reference that does not exist, at the top of the list, because the fixture's numbers were borrowed from the realswiftDataInspectorregression (max delta 203, 7430 pixels, 0.235%). The row most worth investigating was the one that wasn't real.--timingscounted five captures that never happened../test --only SnapshotKitTestingTests --timingsreported "5 captures, 0.1s total, 0.024s per image" for a run that captured nothing, naming fixtures (repeated,one-phase,mixed, …) in its slowest-captures list. On--everything --timingsthose blended into the aggregate the module's performance decisions are read off.The
SNAPSHOT_DIFFenv gate never covered this: it's checked by the pipeline before callingreport, not inside it.Fix. Each channel is split so printing belongs to the pipeline alone and the payload is separately askable —
SnapshotDiffReporting.line(describing:),SnapshotCaptureTiming.line(), andSnapshotSettleReporting.line(...)return the same JSON without emitting, and all four test sites call those. The settle channel gets the same treatment for symmetry; nothing aggregates it today, but it's one scraper away from the same bug. The rule is now an invariant inAGENTS.md, since it's enforced by convention rather than by the compiler.Verified in both directions: a unit-only run reports no timing lines and no differing captures, while the full suite still prints the complete phase breakdown — now 264 captures rather than 269 — and the diff table.
Merging alongside #150
#150 edited the adjacent lines of the same function for an unrelated reason, so neither side could be taken wholesale:
WherePreferences()with its newpreviewPreferences()helper, part of its sweep of fixtures that reached the host's realUserDefaults. This branch briefly carried a narrower version of that same fix; it is reverted here, because Add demo mode, and open nothing until the user picks a world #150's is better — it covers every fixture and removes the parameter defaults so omitting one is a compile error.if seededWithIssues { … }withsetDataIssues(seededWithIssues ? sampleDataIssues() : []). Add demo mode, and open nothing until the user picks a world #150 left that line alone, somaincarried the race until this merge.The merge keeps both, and
loadLeavesASeededFixtureAloneuses themakePreferences()helper #150 added to the test file.Docs
SnapshotKitTesting's README claimed.settled"waits for pixel-stable renders so.task-driven content loads" — the over-promise that let the first bug land. It now says pixel stability gives async content time to load but cannot certify that it did.AGENTS.mdgains two invariants: a settled capture is not a ready capture (seed the fixture or gate ononReadyToSnapshot), and only the pipeline may print a report channel.Testing
Run locally on iPhone 17 / iOS 27.0, on the merged tree:
./test --everything --review --timings— 1365 unit tests and 30 snapshot tests / 264 images, all passing; every remaining diff is max delta 1–2 sub-visible drift and no fabricated rows appear in either reportswift run bumper config/test/lint .— no architecture violations./swiftformat --lintand./xcstrings --lintclean