Skip to content

tests: a talking guest is never killed by the clock (smp:2 liveness flake) - #259

Draft
Japabu wants to merge 3 commits into
mainfrom
wt/toyos-silence
Draft

tests: a talking guest is never killed by the clock (smp:2 liveness flake)#259
Japabu wants to merge 3 commits into
mainfrom
wt/toyos-silence

Conversation

@Japabu

@Japabu Japabu commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

CI flakiness part 2: the smp:2 liveness flake

#256 widened the run_test/wait ceilings by a guest's vcpus/cores oversubscription, but that factor clamps to 1 for the default smp:2 guests, so launcher_refusals and screen_console_clear still went red green-alone under a loaded runner. Their wall-clock ceiling fired the instant it passed, whatever the guest was doing: launcher_refusals killed at 192s "still talking 1s ago", screen_console_clear at 0 of 2073600 pixels, the paint never arriving in the window — a guest making steady progress, called wedged by a clock (recorded in issues/build/parallel-tests-red-under-other-suites.md).

This makes the wedge fire on silence, the way await_guest already does, without inventing a second timer.

The change

  • ceiling_verdict keeps elapsed > ceiling as a necessary condition, but the per-test budget now bites only a guest that has also gone quiet for GUEST_QUIET. A guest still talking past its budget is slow, not wedged, and runs to a GUEST_WEDGED backstop (ceiling.max(GUEST_WEDGED), for the guest that is stuck and chatty). run_test_paced is unchanged — it still hands budget_smp(timeout) as the ceiling, so tests: widen liveness ceilings by a guest's vcpus/cores oversubscription #256's oversubscription still composes into it.
  • Idle-safety. elapsed > ceiling stays necessary on purpose: silence alone is not a wedge on this suite's boots. A healthy but idle guest on a config with no live periodic speaker (no compositor, an idle soundd, the kernel's own ~10s sched: line halting with the idle loop) was measured quiet for as long as 102s. A guest's own budget is what says how long its silence is allowed.
  • screendump_while_rendering is the screen-channel form of the same rule, reusing ceiling_verdict with screen-freeze as the quiet signal. Only screen_console_clear's three console-config waits move to it. The plain screendump_while stays for the panic-waiting and always-animating callers — the screen_blocked_dump retry loop's timeout is a deliberate re-send signal and must not extend.

No new registered test names, so no durations change.

Two checks (high-risk liveness-guard change)

1. Negative control — the guard still fires within a bounded, stated time

Mutation (whole change reverted onto #256, same session, same machine): readdir_bound with its timeout lowered to 500ms (budget ≈ 0.8s), on an idle host.

  • OLD (tests: widen liveness ceilings by a guest's vcpus/cores oversubscription #256, git stash of qemu.rs): FAIL readdir_bound: ... timed out after 0s, with the guest still talking 533ms ago (9 console line(s) while it ran) — it was working and did not finish — wide AND alone. The flake, reproduced.
  • NEW: PASS readdir_bound (4s) — the guest ran 1.14s (spawn 0.294s → exit 1.439s, cpu=1132ms), far past the ~0.8s budget, kept working, and finished (all readdir bound tests passed).

Safety still fires — induced hang. readdir_bound's guest patched to loop { sleep(3600s) } (silent forever), timeout 500ms: caught at the backstop, FAIL ... timed out after 300s, with the guest still talking 9s ago (328 console line(s) while it ran) — it was working and did not finish. On test_config the kernel emits a sched: health line every ~10s, so quiet never reaches 15s and the guard correctly falls through to the GUEST_WEDGED=300s backstop (the same "settle predicate measured ending at 302s" that constant documents). A truly-silent guest (no periodic speaker) is caught faster, at the ceiling — pinned deterministically by ceiling_self_check case (c).

ceiling_self_check (runs in serial_vocabulary, no guest) gains section 3b, staged at the exact CI coordinates: (a) 192s/153s/1s → no verdict (flake survives), (b) chatty-forever past the backstop → did not finish, (c) silent past budget → STALLED (guard fires), (d) idle 90s within budget → no verdict (idle-safety). All pass.

2. Independent oracle — the recorded CI failure + the full suite

The oracle is the recorded real failure in issues/build/parallel-tests-red-under-other-suites.md: launcher_refusals killed at 192s "still talking 1s ago" and screen_console_clear at 0 of 2073600 pixels on loaded smp:2 runners, both ALONE: GREEN. The fix is aimed exactly at those coordinates; screen_console_clear now passes locally on its 1920×1080 (2073600-pixel) panel (PASS screen_console_clear (6s)).

The other half: full Fast suite green — 280 passed, 280 total (91.5s), exit 0, no truncation (includes launcher_refusals, screen_console_clear, readdir_bound, serial_vocabulary).

Verification commands

  • cargo test --test toyos-build -- serial_vocabulary (the pure ceiling_self_check)
  • cargo test --test toyos-build -- screen_console_clear
  • cargo test --test toyos-build (full Fast suite)

🤖 Generated with Claude Code

https://claude.ai/code/session_01Dpa781ehanBhUgrycmXcPW

Japabu and others added 3 commits August 23, 2026 22:42
…ss flake)

#256 widened the run_test/wait ceilings by a guest's `vcpus/cores`
oversubscription, but that factor clamps to 1 for the `smp:2` guests, so
`launcher_refusals` and `screen_console_clear` still went red green-alone under a
loaded runner: their wall-clock ceiling fired the instant it passed, whatever the
guest was doing. `launcher_refusals` was killed at `192s "still talking 1s ago"`
and `screen_console_clear` at `0 of 2073600 pixels, the paint never arriving`,
both a guest making steady progress called wedged by a clock.

This makes the wedge fire on silence, the way `await_guest` already does, without
inventing a second timer:

- `ceiling_verdict` keeps `elapsed > ceiling` as a *necessary* condition but the
  per-test budget now bites only a guest that has *also* gone quiet for
  `GUEST_QUIET`. A guest still talking past its budget is slow, not wedged, and
  runs to a `GUEST_WEDGED` backstop (`ceiling.max(GUEST_WEDGED)`, for the guest
  that is stuck *and* chatty). `run_test_paced` is unchanged — it still hands
  `budget_smp(timeout)` as the ceiling, so #256's oversubscription still composes.

- `elapsed > ceiling` stays necessary on purpose. Silence alone is not a wedge on
  this suite's boots: a healthy but idle guest on a config with no live periodic
  speaker (no compositor, an idle soundd, the kernel's own ~10 s `sched:` line
  halting with the idle loop) was measured quiet for as long as 102 s. A guest's
  own budget is what says how long its silence is allowed.

- `screendump_while_rendering` is the screen-channel form of the same rule,
  reusing `ceiling_verdict` with screen-freeze as the quiet signal. Only
  `screen_console_clear`'s three console-config waits move to it; the plain
  `screendump_while` stays for the panic-waiting and always-animating callers
  (the `screen_blocked_dump` retry loop's timeout is a deliberate re-send signal
  and must not extend).

`ceiling_self_check` gains section 3b: the flake survives (192s/153s/1s -> no
verdict), the backstop still bites a chatty-forever guest, the wedge still fires
fast on a silent guest past its budget, and an idle-but-within-budget guest is
not called wedged.

No new registered test names, so no durations change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Dpa781ehanBhUgrycmXcPW
The required checks are strict, so GitHub refuses the merge button until this branch contains main — which also makes the checks that run on this head checks on the merged result.
… quiet

Placed by the orchestrator with the detector that enforces it. The
sweep for this change measured a healthy idle guest quiet for 102s
(no compositor, idle soundd, the kernel's idle-loop cadence halted),
so silence alone cannot be the verdict; and the recorded CI failure
killed a guest still talking 1s before the clock fired, so the clock
alone cannot be either. ceiling_verdict now requires both, with the
GUEST_WEDGED backstop behind a guest that keeps talking.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant