Skip to content

fix(test, frontend): raise Vitest timeouts for CI stalls - #7623

Open
aglinxinyuan wants to merge 4 commits into
apache:mainfrom
aglinxinyuan:fix/frontend-vitest-timeouts
Open

fix(test, frontend): raise Vitest timeouts for CI stalls#7623
aglinxinyuan wants to merge 4 commits into
apache:mainfrom
aglinxinyuan:fix/frontend-vitest-timeouts

Conversation

@aglinxinyuan

@aglinxinyuan aglinxinyuan commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

The macOS leg of build / frontend goes red on a different unit test every few days — always a timeout, never the same spec, always green on rerun. Three occurrences in the last four days:

Run Test Error
31665399757 UserDatasetVersionCreatorComponent > onClickCreate creates a dataset with a sanitized name … Test timed out in 5000ms
31630884042 AdminUserComponent > sortByAffiliation compares affiliations … Hook timed out in 10000ms
31411656559 WorkflowRuntimeStatisticsComponent > should create Test timed out in 5000ms

Root cause: the runner stalls, not the test. The stall lands on whichever test happens to be executing. From run 31665399757 — one commit, one matrix, two OSes:

Measure ubuntu-latest macos-latest
the spec file that failed (10 tests) 240 ms 11 727 ms
suite wall clock 89.85 s 252.88 s
cumulative test time 182.34 s 307.69 s
runner size 4 cores / 16 GB 3 cores / 7 GB

That file is not systematically slow — it took 219 ms on macOS in an earlier run, and 443 ms locally. The 11.7 s is a stall. jsdom + v8-coverage workers on 3 cores / 7 GB run under real memory pressure, which is where multi-second pauses come from.

Change File
testTimeout 5s → 30s, hookTimeout 10s → 30s frontend/vitest.config.ts
testTimeout 15s → 30s (hookTimeout left alone — browser mode already resolves 30s) frontend/vitest.browser.config.ts
fail-fast: false and timeout-minutes: 30 on the frontend job .github/workflows/build.yml
Timeouts row in the runner-setup table, scoped per config frontend/TESTING.md

Bumping the one test's timeout would be whack-a-mole — the next stall picks a different spec. 30 s absorbs a stall an order of magnitude worse than any observed; a spec that legitimately needs 30 s is broken, and the job now carries timeout-minutes: 30 to bound a true hang. Without it the job inherited GitHub's implicit 6h cap — billed at 10× on the macOS leg. Green legs run 4–12 minutes and the "Install dependency" step is already allowed 20 on its own, so 30 leaves room for a cold yarn cache without masking a hang.

The fail-fast opt-out follows platform, platform-integration, agent-service and infra, the jobs that already set it. Today one flaky OS cancels the other two legs, so the run no longer says whether the failure reproduces off that OS — exactly the evidence needed to tell a runner flake from a real break.

Before:  macOS stalls 5s -> that test fails -> ubuntu + windows cancelled
After:   macOS stalls 5s -> absorbed; a real break still fails all legs

If macOS keeps flaking after this, the next lever is capping maxWorkers on that leg to cut memory pressure. Left out here: it trades wall clock for stability and can't be measured from a non-macOS box. amber-integration and pyamber are also multi-leg without a fail-fast opt-out; left alone to keep this PR to the frontend job.

Any related issues, documentation, discussions?

Closes #6073

How was this PR tested?

No production code is touched; the change is to the test harness and CI config.

Check Result
yarn test:ci (full jsdom suite, new config) 200 files, 4433 passed / 1 skipped — same counts as CI's ubuntu leg
config actually wired a throwaway spec with a 6 s beforeEach + 8 s body passes (14 026 ms); the 8 s body fails on the old 5 s default. Removed before commit
browser-mode defaults read out of the pinned vitest@4.1.10 (dist/chunks/coverage.DM_a_rWm.js:538-539): testTimeout ??= browser.enabled ? 15e3 : 5e3, hookTimeout ??= browser.enabled ? 3e4 : 1e4 — so the dropped hookTimeout line was setting the value it already resolved to
build.yml parses yaml.safe_loadjobs.frontend['timeout-minutes'] === 30, strategy['fail-fast'] === false; the same pass over every job confirms platform, platform-integration, agent-service, infra are the ones opting out
formatting prettier-eslint --list-different clean

The flake itself can't be reproduced on demand — that's the nature of a runner stall. What this PR asserts is verifiable: the per-test ceiling the stalls blow past is 6× higher under jsdom (5 s → 30 s) and 2× in browser mode (15 s → 30 s), the surviving matrix legs still report their own results, and a genuine hang now ends at 30 minutes instead of 6 hours.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 5)

The macOS leg of `build / frontend` goes red on a different unit test
every few days -- always a timeout, never the same spec, always green
on rerun. Three occurrences in the last four days:

| Run | Test | Error |
| --- | --- | --- |
| 31665399757 | UserDatasetVersionCreatorComponent > onClickCreate ... | Test timed out in 5000ms |
| 31630884042 | AdminUserComponent > sortByAffiliation ... | Hook timed out in 10000ms |
| 31411656559 | WorkflowRuntimeStatisticsComponent > should create | Test timed out in 5000ms |

The tests are not the problem: the runner stalls, and the stall lands
on whichever test is executing. In run 31665399757 the offending spec
file took 11727ms on macos-latest and 240ms on ubuntu-latest for the
same commit; in an earlier run the same file took 219ms on macOS.
Suite totals from that run show the same picture -- 252.88s wall on
macOS vs 89.85s on ubuntu, with a cumulative test time of 307.69s vs
182.34s. macos-latest gives 3 cores and 7 GB against ubuntu's 4 and
16, so the jsdom + v8-coverage workers run under real memory pressure
there.

Raise testTimeout and hookTimeout to 30s in both Vitest configs, which
absorbs a stall an order of magnitude worse than any observed so far.
A spec that legitimately needs 30s is broken, and the job's own
timeout still bounds a true hang. Per-test timeouts would be
whack-a-mole: the next stall picks a different test.

Also opt the frontend matrix out of fail-fast, as every other
multi-leg matrix in build.yml already does. Today one flaky OS
cancels the other two legs, which destroys exactly the evidence
needed to tell a runner flake from a real break.

Before:  macOS stalls 5s -> that test fails -> ubuntu + windows cancelled
After:   macOS stalls 5s -> absorbed; a real break still fails all legs
Copilot AI lite review requested due to automatic review settings August 13, 2026 05:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Yicong-Huang Yicong-Huang added the release/v1.2 back porting to release/v1.2 label Aug 13, 2026
@github-actions github-actions Bot added fix frontend Changes related to the frontend GUI ci changes related to CI docs Changes related to documentations and removed release/v1.2 back porting to release/v1.2 labels Aug 13, 2026
@github-actions
github-actions Bot requested a review from xuang7 August 13, 2026 05:03
@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Backport auto-label report

This fix: PR was checked against each actively-supported release branch. release/* labels drive the post-merge backport, so add or remove one to change where this fix lands.

Release branch Analysis
🚫 release/v1.2 Label was removed earlier (opt-out); not re-added. Re-add it by hand if this fix should be backported here after all.

Auto-label run.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Committers with relevant context: @parshimers
    You can request their reviews formally with /request-review @parshimers.

  • Contributors with relevant context: @Yicong-Huang
    You can notify them by mentioning @Yicong-Huang in a comment.

@codecov-commenter

codecov-commenter commented Aug 13, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.93%. Comparing base (1011ff7) to head (d3b5442).
⚠️ Report is 5 commits behind head on main.
✅ All tests successful. No failed tests found.

Additional details and impacted files
@@             Coverage Diff              @@
##               main    #7623      +/-   ##
============================================
- Coverage     90.93%   90.93%   -0.01%     
+ Complexity     4452     4451       -1     
============================================
  Files          1175     1175              
  Lines         47140    47140              
  Branches       5284     5284              
============================================
- Hits          42869    42868       -1     
  Misses         2581     2581              
- Partials       1690     1691       +1     
Flag Coverage Δ
access-control-service 81.00% <ø> (ø)
agent-service 98.62% <ø> (ø)
amber 87.39% <ø> (-0.01%) ⬇️
computing-unit-managing-service 73.67% <ø> (ø)
config-service 86.73% <ø> (ø)
file-service 68.90% <ø> (ø)
frontend 92.44% <ø> (ø)
notebook-migration-service 83.74% <ø> (ø)
pyamber 97.57% <ø> (ø)
workflow-compiling-service 77.19% <ø> (ø)

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 5 worse · ⚪ 8 noise (<±5%) · 0 without baseline

Compared against main 1011ff7 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 370 0.226 24,509/40,047/40,047 us 🔴 +18.0% / 🔴 +147.7%
🟢 bs=100 sw=10 sl=64 820 0.501 121,371/134,383/134,383 us 🟢 -12.1% / 🔴 +27.2%
bs=1000 sw=10 sl=64 935 0.571 1,066,059/1,164,992/1,164,992 us ⚪ within ±5% / 🔴 +15.1%
Baseline details

Latest main 1011ff7 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 370 tuples/sec 427 tuples/sec 784.16 tuples/sec -13.3% -52.8%
bs=10 sw=10 sl=64 MB/s 0.226 MB/s 0.261 MB/s 0.479 MB/s -13.4% -52.8%
bs=10 sw=10 sl=64 p50 24,509 us 22,901 us 12,626 us +7.0% +94.1%
bs=10 sw=10 sl=64 p95 40,047 us 33,926 us 16,169 us +18.0% +147.7%
bs=10 sw=10 sl=64 p99 40,047 us 33,926 us 18,986 us +18.0% +110.9%
bs=100 sw=10 sl=64 throughput 820 tuples/sec 822 tuples/sec 1,023 tuples/sec -0.2% -19.9%
bs=100 sw=10 sl=64 MB/s 0.501 MB/s 0.502 MB/s 0.625 MB/s -0.2% -19.8%
bs=100 sw=10 sl=64 p50 121,371 us 117,604 us 99,185 us +3.2% +22.4%
bs=100 sw=10 sl=64 p95 134,383 us 152,877 us 105,616 us -12.1% +27.2%
bs=100 sw=10 sl=64 p99 134,383 us 152,877 us 113,681 us -12.1% +18.2%
bs=1000 sw=10 sl=64 throughput 935 tuples/sec 939 tuples/sec 1,057 tuples/sec -0.4% -11.5%
bs=1000 sw=10 sl=64 MB/s 0.571 MB/s 0.573 MB/s 0.645 MB/s -0.3% -11.5%
bs=1000 sw=10 sl=64 p50 1,066,059 us 1,065,603 us 965,435 us +0.0% +10.4%
bs=1000 sw=10 sl=64 p95 1,164,992 us 1,119,073 us 1,012,135 us +4.1% +15.1%
bs=1000 sw=10 sl=64 p99 1,164,992 us 1,119,073 us 1,042,088 us +4.1% +11.8%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,539.95,200,128000,370,0.226,24509.22,40047.23,40047.23
1,100,10,64,20,2438.74,2000,1280000,820,0.501,121370.54,134383.29,134383.29
2,1000,10,64,20,21383.00,20000,12800000,935,0.571,1066059.14,1164991.87,1164991.87

@Yicong-Huang Yicong-Huang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 1 must-fix · 2 advisory · 1 polish — well-diagnosed, correctly scoped CI-stability fix; the only blocker is the missing Closes linkage, the rest are accuracy fixes to comments and docs that would otherwise ship permanently.

Simplifications (2)

  • frontend/vitest.browser.config.ts:71 — under browser.enabled: true Vitest's defaults are already 15s/30s, so this line is a no-op and TESTING.md's baseline is wrong here (advisory, see inline)
  • frontend/vitest.config.ts:44 — the "job's own timeout bounds a true hang" reassurance has no backing; the frontend job sets no timeout-minutes (advisory, see inline)

Conventions (1)

  • Description: promote the #6073 mention to Closes #6073 — no closing link now (linkage_ok: false), and #6073's follow-ups are exactly this PR's two changes, so it ships untracked (must-fix)

Polish: 1 quick touch-up (see inline comments).

Comment thread frontend/vitest.browser.config.ts Outdated
Comment thread frontend/vitest.config.ts Outdated
Comment thread .github/workflows/build.yml Outdated
Review follow-ups: the frontend job had no `timeout-minutes`, so the
claim that "the job's own timeout bounds a true hang" rested on GitHub's
implicit 6h cap; browser mode already resolves a 30s `hookTimeout` from
`browser.enabled`, so setting it there restated the default; and the
fail-fast rationale named "every other multi-leg matrix", which
`amber-integration` and `pyamber` refute.
@aglinxinyuan

Copy link
Copy Markdown
Contributor Author

All four addressed in fb5ec5a, plus a description edit.

  • Closes #6073 — promoted from "Related to". The two follow-ups it lists are exactly what this ships. It's assigned to @rbelavadi for the Windows-side tracking, so say the word if you'd rather it stay open and I'll revert to a plain reference.
  • Browser hookTimeout — dropped. Confirmed in the pinned 4.1.10 that browser.enabled already resolves 15s/30s, so that line set the value it inherited; the remaining testTimeout is now documented as a 15s → 30s bump rather than 6x.
  • timeout-minutes: 30 on the frontend job, so the "the job's own timeout bounds a true hang" sentence is backed by a real key rather than GitHub's implicit 6h cap. Green legs run 4-12 min and the install step alone is allowed 20.
  • fail-fast rationale — now names platform, platform-integration, agent-service and infra instead of claiming every multi-leg matrix. amber-integration (2 OS legs) and pyamber (3 Python versions) are the counterexamples; left alone here, happy to open the follow-up.

Description updated to match: per-config timeout rows, the new job cap, and the ceiling split (6x under jsdom, 2x in browser mode).

@Yicong-Huang Yicong-Huang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 4 resolved · 0 open · 1 new (1 new = 1 newly introduced · 0 late catches)

All four round-1 findings verified fixed against the tree, not taken from the replies. The one new item arrived with the fix itself and blocks nothing.

Simplifications (1)

  • .github/workflows/build.yml:92 — the job cap's sizing argument reads a step budget as an observed duration, and leaves no headroom (advisory, see inline)
Verification trace

Each resolved thread was re-checked against the working tree rather than the author's reply. grep -n hookTimeout frontend/vitest.browser.config.ts now returns only the comment line explaining the key's absence, so the no-op is genuinely gone. timeout-minutes: 30 sits at 4-space (job-level) indent at build.yml:94, which is what makes the vitest.config.ts:44 reassurance point at a real bound. Mapping each fail-fast: false line to its owning job confirms the opt-out set is exactly the four now named at :100-101.

TESTING.md's four baseline numbers were re-read from the pinned runtime, not from the docs: node_modules/vitest/dist/chunks/coverage.DM_a_rWm.js:538-539 in vitest 4.1.10 resolves testTimeout to 15s and hookTimeout to 30s under browser.enabled, and 5s / 10s otherwise — so the per-config row is correct on all four counts.

The timings behind the one new finding come from this PR's own run 31869822969 via the Actions API, per step: frontend totals 518s / 601s / 641s on ubuntu / macOS / windows, of which Install dependency is 37s / 49s / 74s.

Comment thread .github/workflows/build.yml Outdated
The justification for `timeout-minutes: 30` cited the "Install dependency"
step's own 20-minute budget as though it were an observed duration. It
isn't: install measures 37s / 49s / 74s on ubuntu / macOS / windows. The
conclusion did not follow either -- had install actually approached 20,
the remaining steps still need 8-9.5 minutes, putting the job at 28-29.5
against the cap, which is the opposite of the absorption the sentence
claimed.

Size it from the legs instead. From this PR's run 31869822969:

| Leg | Job | Install | Rest |
| --- | --- | --- | --- |
| ubuntu-latest | 8.6 min | 37s | 8.0 min |
| macos-latest | 10.0 min | 49s | 9.2 min |
| windows-latest | 10.7 min | 74s | 9.5 min |

The value stays at 30 -- ~3x the slowest leg, ~19 minutes of slack.
@Yicong-Huang

Copy link
Copy Markdown
Contributor

This PR conflicts with the base branch; please resolve the conflicts before review can continue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci changes related to CI docs Changes related to documentations fix frontend Changes related to the frontend GUI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky frontend unit test: joint-ui.service per-port counts test times out on Windows CI

4 participants