ci: replace coop-shard wall-clock timeout with per-test budget enforcement - #170
Merged
Merged
Conversation
…ement
The per-shard 25-minute CI wall-clock timeout is the wrong bound - the suite
only grows, and in run 31999655291 it spuriously killed 2 of 4 shards ("has
timed out after 25 minutes" on shards 2 and 4, which carried test_coop_debrief_sync
at 529.8s and test_parallel_soak at 305.8s). Replace the shard wall with per-TEST
budget enforcement so the suite can grow while a single pathological test is still
bounded.
What changed:
- New tools/coop_test/slow_test_exceptions.json: single source of truth for
budgets. default 180s/test; hard_kill_multiplier 2.0; max_budget_s 900 (hard cap,
no unlimited entries - both runners error on load if an exception exceeds it).
7 seeded exceptions with observed CI durations in each "reason".
- tools/ci/run_coop_suite.ps1 (the CI shard runner) and tools/coop_test/run_parallel.py
(the local K-lane runner) both read that JSON and enforce it:
* a test that FINISHES over budget FAILS the suite even if it passed
("BUDGET EXCEEDED: <t> took Xs > Bs budget - re-engineer the test or add a
justified exception");
* a test still running at 2x its budget is hard-killed with its game subtree and
failed ("BUDGET HARD-KILL ... killed as a hung test"), so a hang can never
wedge a shard now that the wall is gone;
* real failures still retry once (flake tolerance); hangs are NOT retried, so the
worst case stays bounded.
run_coop_suite.ps1 previously had no per-test timeout at all - the 25-min step wall
was its only hang bound - so this adds a Start-Process/WaitForExit(ms)/taskkill-tree
ceiling there.
- .github/workflows/ci-main.yml and ci-validate.yml: removed the 25-min step timeout
on the "Coop test suite" step; raised the coop-shard job timeout 30 -> 60 as a
coarse runaway backstop only (per-test budgets are the primary bound). Leaving the
30-min job cap would have re-capped the shard and defeated the removal.
pull_request_target note (delayed effect): the PR merge gate is ci-validate.yml,
which runs on pull_request_target - GitHub always takes that workflow from the BASE
branch (main), so this branch's edit to ci-validate.yml does NOT change its own PR
gate; it takes effect only for PRs opened after this merges to main. ci-main.yml's
coop-shard runs only on push to main, so its edit likewise takes effect on the next
push to main (post-merge). The runner scripts + JSON, by contrast, are checked out
from the PR head and take effect immediately.
Validated with fake sleeping tests (no game launched, machine reserved for a timing
session): pass-under-budget, budget-exceeded-but-completed, hang-hard-killed-at-2x
(killed at 2s not 30s), real-failure-retried, and over-cap rejection - green in both
run_parallel.py (22/22 unit checks) and run_coop_suite.ps1 (end-to-end against a
scratch tree). No product C++ touched; Python/PowerShell/YAML only.
Cherry-picked from PR #166 (commit 835be08) so this CI change can land on
main independently while PR #166 is stabilized: per the pull_request_target note
above, the ci-validate.yml wall removal only takes effect for the merge gate once
it is on the base branch (main). The PR #166 hunk to tools/coop_test/run_parallel.py
(a branch-only local K-lane runner that does not exist on main and is not run by the
CI coop-shard gate) is intentionally omitted here; the four CI-effective files -- both
workflows, run_coop_suite.ps1, and the new slow_test_exceptions.json -- are applied
verbatim from 835be08.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
NonPolynomialTim
enabled auto-merge (squash)
August 17, 2026 19:22
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Replaces the per-shard 25-minute CI wall-clock timeout on the
coop-shardjob (both
ci-main.ymlandci-validate.yml) with per-test budgetenforcement:
tools/coop_test/slow_test_exceptions.json— single source of truth:default_budget_s180,hard_kill_multiplier2.0,max_budget_s900 hard cap(no unlimited entries; the runner errors on load if an exception exceeds the
cap), plus 7 seeded exceptions carrying their observed CI durations.
tools/ci/run_coop_suite.ps1(the CI shard runner) reads that JSON and enforcesit: a test that finishes over budget fails the suite even if it passed; a
test still running at 2× its budget is hard-killed with its game subtree and
failed, so a hang can never wedge a shard now that the wall is gone. Real
failures still retry once; hangs are never retried.
raised the
coop-shardjob cap 30 → 60 as a coarse runaway backstop only (theper-test budgets are the primary bound — leaving 30 would have re-capped the
shard and defeated the removal).
Why
The 25-minute shard wall is the wrong bound: the suite only grows, so a fixed
wall goes stale and starts killing healthy shards. In run
31999655291itspuriously killed 2 of 4 shards ("timed out after 25 minutes") that happened to
carry
test_coop_debrief_sync(529.8s) andtest_parallel_soak(305.8s). Aper-test budget keeps a single pathological test bounded while letting the suite
grow.
Why a separate PR (unblocks #166)
This is cherry-picked from PR #166 (commit
835be08a0) so the CI change canland on
mainon its own. The merge gate isci-validate.yml, which runs onpull_request_target— GitHub always takes that workflow from the basebranch, so the wall removal only takes effect for the gate once it is on
main.Landing it here activates the change for every subsequent PR while #166 is still
being stabilized.
The PR #166 hunk to
tools/coop_test/run_parallel.py(a branch-only localK-lane runner that does not exist on
mainand is not invoked by the CIcoop-shardgate) is intentionally omitted; the four CI-effective files areapplied verbatim from
835be08a0.Validation
835be08a0for all four (ci-main.yml,ci-validate.yml,run_coop_suite.ps1,slow_test_exceptions.json).run_coop_suite.ps1: PowerShell AST parse clean; both workflows parse as YAML;slow_test_exceptions.jsonparses as JSON; notimeout-minutes: 25remains;coop-shardjob cap is 60 in both.exit 0tests, no game launched): the loader toleratesthe 7 exception entries even though none match the present tests (the
main-suite condition — several of those tests live only on the Parallel battlescape turns + per-action desync detection, reporting, and crash bundling #166 branch), the
present tests PASS, exit 0.
sleeping tests: pass-under-budget, budget-exceeded-but-completed,
hang-hard-killed-at-2× (killed at 2s not 30s), real-failure-retried, and
over-cap rejection. No product C++ touched; Python/PowerShell/YAML only.
🤖 Generated with Claude Code