Parallel test execution: per-suite processes with a zero-loss gate contract#1164
Merged
Conversation
The runner executed all 104 suites sequentially (~10 min locally, the bulk of every CI test leg). The suites are process-isolated already (per-process mkdtemp HOME sentinel; two full parallel runs produced zero cross-suite failures), so the serialization was pure convention. - test-runner --list-suites: prints every registered suite, one per line, emitted by the SAME macro table that executes suites — the list cannot drift from the run set by construction. - scripts/run-tests-parallel.sh: runs each suite as its own process (jobs = CPU count; CBM_TEST_PAR_JOBS overrides) under a ZERO-LOSS CONTRACT: a union guard fails the gate if the set of suites that produced a result differs from --list-suites (nothing can be silently dropped, a newly added suite is picked up automatically); per-suite pass/fail/skip are summed into the sequential runner's exact summary format; any suite crash, failure, or omission exits nonzero. Per-suite wall times are printed for balance tracking. - make test-par: the parallel target. make test (sequential) is unchanged and remains the escape hatch (CBM_TEST_SEQUENTIAL=1 in test.sh). - scripts/test.sh: builds, then routes through test-par — every CI test leg gets the speedup with zero workflow-topology change (same jobs, same gates, same billing; the legs just finish sooner). - The stack_overflow suite is split into a/b/c (7+6+7 of its 20 tests, pure re-registration): as one suite it was the wall-clock critical path of any parallel run — every other suite finished underneath its ~4 minutes. Measured locally (Apple Silicon, ASan runner): sequential ~10 min vs parallel 232 s, totals identical (6,361 passed / 0 failed / 1 skipped), union guard clean. The TSan leg keeps its dedicated subset runner (sequential) — out of scope here. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…e suites
First CI validation round of the parallel runner, three findings:
- Windows failed ALL 104 suites with "0 passed, 1 failed" each: the CRT
emits --list-suites lines as CRLF, so every dispatched name carried a
trailing CR ("arena\r" is an unknown suite). The union guard caught it
loudly, exactly as designed. The driver now strips CR when writing the
suite list.
- The ubuntu-gcc legs had 3 real failures in the cli suite: it spawns
subprocesses with fixed deadlines, and a fully saturated 4-core runner
starves those deadlines into flakes. Deadline-sensitive suites (cli,
subprocess, watcher, incremental, httpd, ui, index_resilience, and the
stack_overflow family) now run SEQUENTIALLY after the parallel wave on a
quiet machine — same suites, same tests, same gates, only the schedule
differs; the union guard checks the combined result set.
- Failing suites now print every FAIL site with context (the tail-30 of a
long suite log hid which tests actually failed on CI).
Local: totals unchanged (6,361 passed / 0 failed / 1 skipped), union guard
clean.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
perl_glr_deep_parse_recursion_capped wandered between 2s and 218s for the IDENTICAL input: past the merge cap the ambiguity-exploded GLR parse grinds at an environment-dependent rate (allocator/scheduler state in the forked child), and the test paid for however long that grind ran. It was the critical path of every parallel run and the dominant chunk of the sequential one. The guard's actual signal is the CRASH, not the grind: when CBM_TS_STACK_MERGE_MAX_DEPTH regresses, the recursive stack-merge overflows within the first second at this depth (same profile as the sibling pre-guard probe, rc=139 in under a second). The forked child now arms a 10s alarm whose handler exits cleanly — a real SIGSEGV/SIGBUS still terminates the child by signal first and stays visible to WIFSIGNALED, so the RED path is untouched; only the pointless remainder of the grind is skipped. Windows (in-process branch) is unchanged. Suite b: 13s across three consecutive runs (was 2-218s). Full parallel run: 97s wall, 6,374 passed / 0 failed / 1 skipped, union guard clean — vs 351s sequential on the same machine (3.6x). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
…alsification Follow-up correcting d8a78e8, whose message claimed a broken merge cap crashes "within the first second" so the 10s alarm could not mask it. Falsification disproved that claim AND found the guard was vacuous on POSIX from the start: - With CBM_TS_STACK_MERGE_MAX_DEPTH deleted outright, the ORIGINAL unbounded test stayed GREEN (220s full grind, no crash) on macOS ASan. - Small thread stacks (2 MB, then 512 KB) did not change that. - Instrumenting the vendored recursion showed why: the recursive merge path is NEVER ENTERED for this input in the observable window — max depth stays 0. There is nothing for a stack bound to catch here. So the cap-regression DETECTION lives in the Windows in-process branch (real ~1 MB native stack, where #913 actually manifested — exercised by the windows CI leg). The POSIX branch is, and always was, a bounded no-crash smoke of the pathological parse; the 10s alarm makes it deterministic (12-15s across three runs, was 2-218s machine-dependent) without any detection loss, because there was no POSIX detection to lose. The experimental pthread machinery is dropped; the test comment now carries the falsification data so no future reader mistakes the POSIX branch for a cap guard. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
… there) Flake audit: the whole corpus has exactly two wall-clock test deadlines — the EOF-shutdown alarm(5) and the issue-832 supervisor-spawn alarm(60), both in the mcp suite (the latter already flaked once under heavy local load). Serializing mcp costs ~6s of tail and closes the starvable-deadline class completely; every remaining parallel-wave suite is deadline-free. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
The parallel runner's isolated-process execution exposed three cli tests that were passing for the wrong reason on Linux: cbm_zed_config_dir (and the VS Code/vendor path resolvers) prefer $XDG_CONFIG_HOME over $HOME/.config, CI runners export it, and an isolated `test-runner cli` process therefore resolved config paths OUTSIDE the test fixture — the tests only stayed green in the all-suites single process because an earlier suite's env mutations leaked into them (hidden cross-suite coupling, deterministic failure in isolation, macOS unaffected since its resolvers do not consult XDG). - cli_detect_agents_finds_zed: save/set XDG_CONFIG_HOME to the fixture's .config for the detection call, restore after. - cli_vscode_profile_mcp_uninstall: same pin alongside its existing HOME/PATH/APPDATA handling. - cli_durable_profiles_follow_current_vendor_paths: XDG_CONFIG_HOME joins its unset-and-restore env list. All 222 cli tests pass locally with a hostile XDG_CONFIG_HOME exported. No production code changed; no assertion weakened. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The test runner executed all 104 suites sequentially — ~10 minutes locally and the bulk of every CI test leg. The suites are already process-isolated (per-process
mkdtempHOME sentinel), so this PR runs each suite as its own process and aggregates.Zero-loss gate contract (the design center)
Gate quality is bitwise-identical to the sequential run, enforced structurally:
--list-suitesis printed by the same macro table that executes suites — the enumeration cannot drift from the run set by construction.--list-suitesexactly; a suite that never ran (or ran twice) fails the gate loudly. A newly added suite is picked up automatically — omission is impossible, not just checked.make testis unchanged;CBM_TEST_SEQUENTIAL=1routestest.shback to it.CI scope (full disclosure)
scripts/test.sh(the single source of truth all test legs call) now builds, then runsmake test-parinstead ofmake test. Legs simply finish sooner.stack_overflowsplit into a/b/c (7+6+7 of its 20 tests, pure re-registration): as a single suite it was the critical path of any parallel run.Measured (local, Apple Silicon, ASan runner)
This PR's own CI run is the cross-platform validation: every test leg exercises the parallel path.