fix(test, frontend): raise Vitest timeouts for CI stalls - #7623
fix(test, frontend): raise Vitest timeouts for CI stalls#7623aglinxinyuan wants to merge 4 commits into
Conversation
The macOS leg of `build / frontend` goes red on a different unit test every few days -- always a timeout, never the same spec, always green on rerun. Three occurrences in the last four days: | Run | Test | Error | | --- | --- | --- | | 31665399757 | UserDatasetVersionCreatorComponent > onClickCreate ... | Test timed out in 5000ms | | 31630884042 | AdminUserComponent > sortByAffiliation ... | Hook timed out in 10000ms | | 31411656559 | WorkflowRuntimeStatisticsComponent > should create | Test timed out in 5000ms | The tests are not the problem: the runner stalls, and the stall lands on whichever test is executing. In run 31665399757 the offending spec file took 11727ms on macos-latest and 240ms on ubuntu-latest for the same commit; in an earlier run the same file took 219ms on macOS. Suite totals from that run show the same picture -- 252.88s wall on macOS vs 89.85s on ubuntu, with a cumulative test time of 307.69s vs 182.34s. macos-latest gives 3 cores and 7 GB against ubuntu's 4 and 16, so the jsdom + v8-coverage workers run under real memory pressure there. Raise testTimeout and hookTimeout to 30s in both Vitest configs, which absorbs a stall an order of magnitude worse than any observed so far. A spec that legitimately needs 30s is broken, and the job's own timeout still bounds a true hang. Per-test timeouts would be whack-a-mole: the next stall picks a different test. Also opt the frontend matrix out of fail-fast, as every other multi-leg matrix in build.yml already does. Today one flaky OS cancels the other two legs, which destroys exactly the evidence needed to tell a runner flake from a real break. Before: macOS stalls 5s -> that test fails -> ubuntu + windows cancelled After: macOS stalls 5s -> absorbed; a real break still fails all legs
Backport auto-label reportThis
|
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7623 +/- ##
============================================
- Coverage 90.93% 90.93% -0.01%
+ Complexity 4452 4451 -1
============================================
Files 1175 1175
Lines 47140 47140
Branches 5284 5284
============================================
- Hits 42869 42868 -1
Misses 2581 2581
- Partials 1690 1691 +1
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 370 | 0.226 | 24,509/40,047/40,047 us | 🔴 +18.0% / 🔴 +147.7% |
| 🟢 | bs=100 sw=10 sl=64 | 820 | 0.501 | 121,371/134,383/134,383 us | 🟢 -12.1% / 🔴 +27.2% |
| ⚪ | bs=1000 sw=10 sl=64 | 935 | 0.571 | 1,066,059/1,164,992/1,164,992 us | ⚪ within ±5% / 🔴 +15.1% |
Baseline details
Latest main 1011ff7 from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 370 tuples/sec | 427 tuples/sec | 784.16 tuples/sec | -13.3% | -52.8% |
| bs=10 sw=10 sl=64 | MB/s | 0.226 MB/s | 0.261 MB/s | 0.479 MB/s | -13.4% | -52.8% |
| bs=10 sw=10 sl=64 | p50 | 24,509 us | 22,901 us | 12,626 us | +7.0% | +94.1% |
| bs=10 sw=10 sl=64 | p95 | 40,047 us | 33,926 us | 16,169 us | +18.0% | +147.7% |
| bs=10 sw=10 sl=64 | p99 | 40,047 us | 33,926 us | 18,986 us | +18.0% | +110.9% |
| bs=100 sw=10 sl=64 | throughput | 820 tuples/sec | 822 tuples/sec | 1,023 tuples/sec | -0.2% | -19.9% |
| bs=100 sw=10 sl=64 | MB/s | 0.501 MB/s | 0.502 MB/s | 0.625 MB/s | -0.2% | -19.8% |
| bs=100 sw=10 sl=64 | p50 | 121,371 us | 117,604 us | 99,185 us | +3.2% | +22.4% |
| bs=100 sw=10 sl=64 | p95 | 134,383 us | 152,877 us | 105,616 us | -12.1% | +27.2% |
| bs=100 sw=10 sl=64 | p99 | 134,383 us | 152,877 us | 113,681 us | -12.1% | +18.2% |
| bs=1000 sw=10 sl=64 | throughput | 935 tuples/sec | 939 tuples/sec | 1,057 tuples/sec | -0.4% | -11.5% |
| bs=1000 sw=10 sl=64 | MB/s | 0.571 MB/s | 0.573 MB/s | 0.645 MB/s | -0.3% | -11.5% |
| bs=1000 sw=10 sl=64 | p50 | 1,066,059 us | 1,065,603 us | 965,435 us | +0.0% | +10.4% |
| bs=1000 sw=10 sl=64 | p95 | 1,164,992 us | 1,119,073 us | 1,012,135 us | +4.1% | +15.1% |
| bs=1000 sw=10 sl=64 | p99 | 1,164,992 us | 1,119,073 us | 1,042,088 us | +4.1% | +11.8% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,539.95,200,128000,370,0.226,24509.22,40047.23,40047.23
1,100,10,64,20,2438.74,2000,1280000,820,0.501,121370.54,134383.29,134383.29
2,1000,10,64,20,21383.00,20000,12800000,935,0.571,1066059.14,1164991.87,1164991.87
Yicong-Huang
left a comment
There was a problem hiding this comment.
🔴 1 must-fix · 2 advisory · 1 polish — well-diagnosed, correctly scoped CI-stability fix; the only blocker is the missing Closes linkage, the rest are accuracy fixes to comments and docs that would otherwise ship permanently.
Simplifications (2)
frontend/vitest.browser.config.ts:71— underbrowser.enabled: trueVitest's defaults are already 15s/30s, so this line is a no-op and TESTING.md's baseline is wrong here (advisory, see inline)frontend/vitest.config.ts:44— the "job's own timeout bounds a true hang" reassurance has no backing; the frontend job sets notimeout-minutes(advisory, see inline)
Conventions (1)
- Description: promote the
#6073mention toCloses #6073— no closing link now (linkage_ok: false), and #6073's follow-ups are exactly this PR's two changes, so it ships untracked (must-fix)
Polish: 1 quick touch-up (see inline comments).
Review follow-ups: the frontend job had no `timeout-minutes`, so the claim that "the job's own timeout bounds a true hang" rested on GitHub's implicit 6h cap; browser mode already resolves a 30s `hookTimeout` from `browser.enabled`, so setting it there restated the default; and the fail-fast rationale named "every other multi-leg matrix", which `amber-integration` and `pyamber` refute.
|
All four addressed in fb5ec5a, plus a description edit.
Description updated to match: per-config timeout rows, the new job cap, and the ceiling split (6x under jsdom, 2x in browser mode). |
Yicong-Huang
left a comment
There was a problem hiding this comment.
🟡 4 resolved · 0 open · 1 new (1 new = 1 newly introduced · 0 late catches)
All four round-1 findings verified fixed against the tree, not taken from the replies. The one new item arrived with the fix itself and blocks nothing.
Simplifications (1)
.github/workflows/build.yml:92— the job cap's sizing argument reads a step budget as an observed duration, and leaves no headroom (advisory, see inline)
Verification trace
Each resolved thread was re-checked against the working tree rather than the author's reply. grep -n hookTimeout frontend/vitest.browser.config.ts now returns only the comment line explaining the key's absence, so the no-op is genuinely gone. timeout-minutes: 30 sits at 4-space (job-level) indent at build.yml:94, which is what makes the vitest.config.ts:44 reassurance point at a real bound. Mapping each fail-fast: false line to its owning job confirms the opt-out set is exactly the four now named at :100-101.
TESTING.md's four baseline numbers were re-read from the pinned runtime, not from the docs: node_modules/vitest/dist/chunks/coverage.DM_a_rWm.js:538-539 in vitest 4.1.10 resolves testTimeout to 15s and hookTimeout to 30s under browser.enabled, and 5s / 10s otherwise — so the per-config row is correct on all four counts.
The timings behind the one new finding come from this PR's own run 31869822969 via the Actions API, per step: frontend totals 518s / 601s / 641s on ubuntu / macOS / windows, of which Install dependency is 37s / 49s / 74s.
The justification for `timeout-minutes: 30` cited the "Install dependency" step's own 20-minute budget as though it were an observed duration. It isn't: install measures 37s / 49s / 74s on ubuntu / macOS / windows. The conclusion did not follow either -- had install actually approached 20, the remaining steps still need 8-9.5 minutes, putting the job at 28-29.5 against the cap, which is the opposite of the absorption the sentence claimed. Size it from the legs instead. From this PR's run 31869822969: | Leg | Job | Install | Rest | | --- | --- | --- | --- | | ubuntu-latest | 8.6 min | 37s | 8.0 min | | macos-latest | 10.0 min | 49s | 9.2 min | | windows-latest | 10.7 min | 74s | 9.5 min | The value stays at 30 -- ~3x the slowest leg, ~19 minutes of slack.
|
This PR conflicts with the base branch; please resolve the conflicts before review can continue. |
What changes were proposed in this PR?
The macOS leg of
build / frontendgoes red on a different unit test every few days — always a timeout, never the same spec, always green on rerun. Three occurrences in the last four days:UserDatasetVersionCreatorComponent > onClickCreate creates a dataset with a sanitized name …Test timed out in 5000msAdminUserComponent > sortByAffiliation compares affiliations …Hook timed out in 10000msWorkflowRuntimeStatisticsComponent > should createTest timed out in 5000msRoot cause: the runner stalls, not the test. The stall lands on whichever test happens to be executing. From run
31665399757— one commit, one matrix, two OSes:That file is not systematically slow — it took 219 ms on macOS in an earlier run, and 443 ms locally. The 11.7 s is a stall. jsdom + v8-coverage workers on 3 cores / 7 GB run under real memory pressure, which is where multi-second pauses come from.
testTimeout5s → 30s,hookTimeout10s → 30sfrontend/vitest.config.tstestTimeout15s → 30s (hookTimeoutleft alone — browser mode already resolves 30s)frontend/vitest.browser.config.tsfail-fast: falseandtimeout-minutes: 30on the frontend job.github/workflows/build.ymlfrontend/TESTING.mdBumping the one test's timeout would be whack-a-mole — the next stall picks a different spec. 30 s absorbs a stall an order of magnitude worse than any observed; a spec that legitimately needs 30 s is broken, and the job now carries
timeout-minutes: 30to bound a true hang. Without it the job inherited GitHub's implicit 6h cap — billed at 10× on the macOS leg. Green legs run 4–12 minutes and the "Install dependency" step is already allowed 20 on its own, so 30 leaves room for a cold yarn cache without masking a hang.The
fail-fastopt-out followsplatform,platform-integration,agent-serviceandinfra, the jobs that already set it. Today one flaky OS cancels the other two legs, so the run no longer says whether the failure reproduces off that OS — exactly the evidence needed to tell a runner flake from a real break.If macOS keeps flaking after this, the next lever is capping
maxWorkerson that leg to cut memory pressure. Left out here: it trades wall clock for stability and can't be measured from a non-macOS box.amber-integrationandpyamberare also multi-leg without afail-fastopt-out; left alone to keep this PR to the frontend job.Any related issues, documentation, discussions?
Closes #6073
How was this PR tested?
No production code is touched; the change is to the test harness and CI config.
yarn test:ci(full jsdom suite, new config)beforeEach+ 8 s body passes (14 026 ms); the 8 s body fails on the old 5 s default. Removed before commitvitest@4.1.10(dist/chunks/coverage.DM_a_rWm.js:538-539):testTimeout ??= browser.enabled ? 15e3 : 5e3,hookTimeout ??= browser.enabled ? 3e4 : 1e4— so the droppedhookTimeoutline was setting the value it already resolved tobuild.ymlparsesyaml.safe_load→jobs.frontend['timeout-minutes'] === 30,strategy['fail-fast'] === false; the same pass over every job confirmsplatform,platform-integration,agent-service,infraare the ones opting outprettier-eslint --list-differentcleanThe flake itself can't be reproduced on demand — that's the nature of a runner stall. What this PR asserts is verifiable: the per-test ceiling the stalls blow past is 6× higher under jsdom (5 s → 30 s) and 2× in browser mode (15 s → 30 s), the surviving matrix legs still report their own results, and a genuine hang now ends at 30 minutes instead of 6 hours.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 5)