Skip to content

ci: replace coop-shard wall-clock timeout with per-test budget enforcement - #170

Merged
NonPolynomialTim merged 1 commit into
mainfrom
ci/per-test-budget-enforcement
Aug 17, 2026
Merged

ci: replace coop-shard wall-clock timeout with per-test budget enforcement#170
NonPolynomialTim merged 1 commit into
mainfrom
ci/per-test-budget-enforcement

Conversation

@NonPolynomialTim

Copy link
Copy Markdown
Collaborator

What

Replaces the per-shard 25-minute CI wall-clock timeout on the coop-shard
job (both ci-main.yml and ci-validate.yml) with per-test budget
enforcement
:

  • New tools/coop_test/slow_test_exceptions.json — single source of truth:
    default_budget_s 180, hard_kill_multiplier 2.0, max_budget_s 900 hard cap
    (no unlimited entries; the runner errors on load if an exception exceeds the
    cap), plus 7 seeded exceptions carrying their observed CI durations.
  • tools/ci/run_coop_suite.ps1 (the CI shard runner) reads that JSON and enforces
    it: a test that finishes over budget fails the suite even if it passed; a
    test still running at 2× its budget is hard-killed with its game subtree and
    failed, so a hang can never wedge a shard now that the wall is gone. Real
    failures still retry once; hangs are never retried.
  • Both workflows: removed the 25-min step timeout on the "Coop test suite" step;
    raised the coop-shard job cap 30 → 60 as a coarse runaway backstop only (the
    per-test budgets are the primary bound — leaving 30 would have re-capped the
    shard and defeated the removal).

Why

The 25-minute shard wall is the wrong bound: the suite only grows, so a fixed
wall goes stale and starts killing healthy shards. In run 31999655291 it
spuriously killed 2 of 4 shards ("timed out after 25 minutes") that happened to
carry test_coop_debrief_sync (529.8s) and test_parallel_soak (305.8s). A
per-test budget keeps a single pathological test bounded while letting the suite
grow.

Why a separate PR (unblocks #166)

This is cherry-picked from PR #166 (commit 835be08a0) so the CI change can
land on main on its own. The merge gate is ci-validate.yml, which runs on
pull_request_target — GitHub always takes that workflow from the base
branch, so the wall removal only takes effect for the gate once it is on main.
Landing it here activates the change for every subsequent PR while #166 is still
being stabilized.

The PR #166 hunk to tools/coop_test/run_parallel.py (a branch-only local
K-lane runner that does not exist on main and is not invoked by the CI
coop-shard gate) is intentionally omitted; the four CI-effective files are
applied verbatim from 835be08a0.

Validation

  • Staged files byte-identical to 835be08a0 for all four (ci-main.yml,
    ci-validate.yml, run_coop_suite.ps1, slow_test_exceptions.json).
  • run_coop_suite.ps1: PowerShell AST parse clean; both workflows parse as YAML;
    slow_test_exceptions.json parses as JSON; no timeout-minutes: 25 remains;
    coop-shard job cap is 60 in both.
  • Fake-tree dry run (stub exit 0 tests, no game launched): the loader tolerates
    the 7 exception entries even though none match the present tests (the
    main-suite condition — several of those tests live only on the Parallel battlescape turns + per-action desync detection, reporting, and crash bundling #166 branch), the
    present tests PASS, exit 0.
  • Original enforcement commit was fully validated on the feature branch with fake
    sleeping tests: pass-under-budget, budget-exceeded-but-completed,
    hang-hard-killed-at-2× (killed at 2s not 30s), real-failure-retried, and
    over-cap rejection. No product C++ touched; Python/PowerShell/YAML only.

🤖 Generated with Claude Code

…ement

The per-shard 25-minute CI wall-clock timeout is the wrong bound - the suite
only grows, and in run 31999655291 it spuriously killed 2 of 4 shards ("has
timed out after 25 minutes" on shards 2 and 4, which carried test_coop_debrief_sync
at 529.8s and test_parallel_soak at 305.8s). Replace the shard wall with per-TEST
budget enforcement so the suite can grow while a single pathological test is still
bounded.

What changed:
- New tools/coop_test/slow_test_exceptions.json: single source of truth for
  budgets. default 180s/test; hard_kill_multiplier 2.0; max_budget_s 900 (hard cap,
  no unlimited entries - both runners error on load if an exception exceeds it).
  7 seeded exceptions with observed CI durations in each "reason".
- tools/ci/run_coop_suite.ps1 (the CI shard runner) and tools/coop_test/run_parallel.py
  (the local K-lane runner) both read that JSON and enforce it:
    * a test that FINISHES over budget FAILS the suite even if it passed
      ("BUDGET EXCEEDED: <t> took Xs > Bs budget - re-engineer the test or add a
      justified exception");
    * a test still running at 2x its budget is hard-killed with its game subtree and
      failed ("BUDGET HARD-KILL ... killed as a hung test"), so a hang can never
      wedge a shard now that the wall is gone;
    * real failures still retry once (flake tolerance); hangs are NOT retried, so the
      worst case stays bounded.
  run_coop_suite.ps1 previously had no per-test timeout at all - the 25-min step wall
  was its only hang bound - so this adds a Start-Process/WaitForExit(ms)/taskkill-tree
  ceiling there.
- .github/workflows/ci-main.yml and ci-validate.yml: removed the 25-min step timeout
  on the "Coop test suite" step; raised the coop-shard job timeout 30 -> 60 as a
  coarse runaway backstop only (per-test budgets are the primary bound). Leaving the
  30-min job cap would have re-capped the shard and defeated the removal.

pull_request_target note (delayed effect): the PR merge gate is ci-validate.yml,
which runs on pull_request_target - GitHub always takes that workflow from the BASE
branch (main), so this branch's edit to ci-validate.yml does NOT change its own PR
gate; it takes effect only for PRs opened after this merges to main. ci-main.yml's
coop-shard runs only on push to main, so its edit likewise takes effect on the next
push to main (post-merge). The runner scripts + JSON, by contrast, are checked out
from the PR head and take effect immediately.

Validated with fake sleeping tests (no game launched, machine reserved for a timing
session): pass-under-budget, budget-exceeded-but-completed, hang-hard-killed-at-2x
(killed at 2s not 30s), real-failure-retried, and over-cap rejection - green in both
run_parallel.py (22/22 unit checks) and run_coop_suite.ps1 (end-to-end against a
scratch tree). No product C++ touched; Python/PowerShell/YAML only.

Cherry-picked from PR #166 (commit 835be08) so this CI change can land on
main independently while PR #166 is stabilized: per the pull_request_target note
above, the ci-validate.yml wall removal only takes effect for the merge gate once
it is on the base branch (main). The PR #166 hunk to tools/coop_test/run_parallel.py
(a branch-only local K-lane runner that does not exist on main and is not run by the
CI coop-shard gate) is intentionally omitted here; the four CI-effective files -- both
workflows, run_coop_suite.ps1, and the new slow_test_exceptions.json -- are applied
verbatim from 835be08.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@NonPolynomialTim
NonPolynomialTim enabled auto-merge (squash) August 17, 2026 19:22
@NonPolynomialTim
NonPolynomialTim merged commit 1e0f927 into main Aug 17, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant