tests: a talking guest is never killed by the clock (smp:2 liveness flake) - #259
Draft
Japabu wants to merge 3 commits into
Draft
tests: a talking guest is never killed by the clock (smp:2 liveness flake)#259Japabu wants to merge 3 commits into
Japabu wants to merge 3 commits into
Conversation
…ss flake) #256 widened the run_test/wait ceilings by a guest's `vcpus/cores` oversubscription, but that factor clamps to 1 for the `smp:2` guests, so `launcher_refusals` and `screen_console_clear` still went red green-alone under a loaded runner: their wall-clock ceiling fired the instant it passed, whatever the guest was doing. `launcher_refusals` was killed at `192s "still talking 1s ago"` and `screen_console_clear` at `0 of 2073600 pixels, the paint never arriving`, both a guest making steady progress called wedged by a clock. This makes the wedge fire on silence, the way `await_guest` already does, without inventing a second timer: - `ceiling_verdict` keeps `elapsed > ceiling` as a *necessary* condition but the per-test budget now bites only a guest that has *also* gone quiet for `GUEST_QUIET`. A guest still talking past its budget is slow, not wedged, and runs to a `GUEST_WEDGED` backstop (`ceiling.max(GUEST_WEDGED)`, for the guest that is stuck *and* chatty). `run_test_paced` is unchanged — it still hands `budget_smp(timeout)` as the ceiling, so #256's oversubscription still composes. - `elapsed > ceiling` stays necessary on purpose. Silence alone is not a wedge on this suite's boots: a healthy but idle guest on a config with no live periodic speaker (no compositor, an idle soundd, the kernel's own ~10 s `sched:` line halting with the idle loop) was measured quiet for as long as 102 s. A guest's own budget is what says how long its silence is allowed. - `screendump_while_rendering` is the screen-channel form of the same rule, reusing `ceiling_verdict` with screen-freeze as the quiet signal. Only `screen_console_clear`'s three console-config waits move to it; the plain `screendump_while` stays for the panic-waiting and always-animating callers (the `screen_blocked_dump` retry loop's timeout is a deliberate re-send signal and must not extend). `ceiling_self_check` gains section 3b: the flake survives (192s/153s/1s -> no verdict), the backstop still bites a chatty-forever guest, the wedge still fires fast on a silent guest past its budget, and an idle-but-within-budget guest is not called wedged. No new registered test names, so no durations change. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Dpa781ehanBhUgrycmXcPW
The required checks are strict, so GitHub refuses the merge button until this branch contains main — which also makes the checks that run on this head checks on the merged result.
… quiet Placed by the orchestrator with the detector that enforces it. The sweep for this change measured a healthy idle guest quiet for 102s (no compositor, idle soundd, the kernel's idle-loop cadence halted), so silence alone cannot be the verdict; and the recorded CI failure killed a guest still talking 1s before the clock fired, so the clock alone cannot be either. ceiling_verdict now requires both, with the GUEST_WEDGED backstop behind a guest that keeps talking.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI flakiness part 2: the
smp:2liveness flake#256 widened the run_test/wait ceilings by a guest's
vcpus/coresoversubscription, but that factor clamps to 1 for the defaultsmp:2guests, solauncher_refusalsandscreen_console_clearstill went red green-alone under a loaded runner. Their wall-clock ceiling fired the instant it passed, whatever the guest was doing:launcher_refusalskilled at192s "still talking 1s ago",screen_console_clearat0 of 2073600 pixels, the paint never arriving in the window— a guest making steady progress, called wedged by a clock (recorded inissues/build/parallel-tests-red-under-other-suites.md).This makes the wedge fire on silence, the way
await_guestalready does, without inventing a second timer.The change
ceiling_verdictkeepselapsed > ceilingas a necessary condition, but the per-test budget now bites only a guest that has also gone quiet forGUEST_QUIET. A guest still talking past its budget is slow, not wedged, and runs to aGUEST_WEDGEDbackstop (ceiling.max(GUEST_WEDGED), for the guest that is stuck and chatty).run_test_pacedis unchanged — it still handsbudget_smp(timeout)as the ceiling, so tests: widen liveness ceilings by a guest's vcpus/cores oversubscription #256's oversubscription still composes into it.elapsed > ceilingstays necessary on purpose: silence alone is not a wedge on this suite's boots. A healthy but idle guest on a config with no live periodic speaker (no compositor, an idle soundd, the kernel's own ~10ssched:line halting with the idle loop) was measured quiet for as long as 102s. A guest's own budget is what says how long its silence is allowed.screendump_while_renderingis the screen-channel form of the same rule, reusingceiling_verdictwith screen-freeze as the quiet signal. Onlyscreen_console_clear's three console-config waits move to it. The plainscreendump_whilestays for the panic-waiting and always-animating callers — thescreen_blocked_dumpretry loop's timeout is a deliberate re-send signal and must not extend.No new registered test names, so no durations change.
Two checks (high-risk liveness-guard change)
1. Negative control — the guard still fires within a bounded, stated time
Mutation (whole change reverted onto #256, same session, same machine):
readdir_boundwith its timeout lowered to500ms(budget ≈ 0.8s), on an idle host.git stashofqemu.rs):FAIL readdir_bound: ... timed out after 0s, with the guest still talking 533ms ago (9 console line(s) while it ran) — it was working and did not finish— wide AND alone. The flake, reproduced.PASS readdir_bound (4s)— the guest ran 1.14s (spawn 0.294s → exit 1.439s,cpu=1132ms), far past the ~0.8s budget, kept working, and finished (all readdir bound tests passed).Safety still fires — induced hang.
readdir_bound's guest patched toloop { sleep(3600s) }(silent forever), timeout 500ms: caught at the backstop,FAIL ... timed out after 300s, with the guest still talking 9s ago (328 console line(s) while it ran) — it was working and did not finish. Ontest_configthe kernel emits asched:health line every ~10s, soquietnever reaches 15s and the guard correctly falls through to theGUEST_WEDGED=300s backstop (the same "settle predicate measured ending at 302s" that constant documents). A truly-silent guest (no periodic speaker) is caught faster, at the ceiling — pinned deterministically byceiling_self_checkcase (c).ceiling_self_check(runs inserial_vocabulary, no guest) gains section 3b, staged at the exact CI coordinates: (a)192s/153s/1s→ no verdict (flake survives), (b) chatty-forever past the backstop →did not finish, (c) silent past budget →STALLED(guard fires), (d) idle 90s within budget → no verdict (idle-safety). All pass.2. Independent oracle — the recorded CI failure + the full suite
The oracle is the recorded real failure in
issues/build/parallel-tests-red-under-other-suites.md:launcher_refusalskilled at192s "still talking 1s ago"andscreen_console_clearat0 of 2073600 pixelson loadedsmp:2runners, bothALONE: GREEN. The fix is aimed exactly at those coordinates;screen_console_clearnow passes locally on its 1920×1080 (2073600-pixel) panel (PASS screen_console_clear (6s)).The other half: full Fast suite green —
280 passed, 280 total (91.5s), exit 0, no truncation (includeslauncher_refusals,screen_console_clear,readdir_bound,serial_vocabulary).Verification commands
cargo test --test toyos-build -- serial_vocabulary(the pureceiling_self_check)cargo test --test toyos-build -- screen_console_clearcargo test --test toyos-build(full Fast suite)🤖 Generated with Claude Code
https://claude.ai/code/session_01Dpa781ehanBhUgrycmXcPW