stdiod: truthful state.json server states + token-based version-close matching - #38
stdiod: truthful state.json server states + token-based version-close matching#38Miyamura80 wants to merge 4 commits into
Conversation
|
ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing |
There was a problem hiding this comment.
1 issue found across 9 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="crates/stdiod/crates/edison-stdiod/src/daemon.rs">
<violation number="1" location="crates/stdiod/crates/edison-stdiod/src/daemon.rs:721">
P2: The tray can retain a stale `running` or `crashed` child entry after a crash or respawn. Passing the writer into child pumps makes these updates concurrent with `Supervisor::publish_state`, but the writer only serializes state mutation—not the subsequent file write—so the write should also be serialized (or protected by a dedicated I/O mutex).</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| &enriched, | ||
| self.tunnel_outgoing.clone(), | ||
| sensitive_arg_values, | ||
| Some(self.state.clone()), |
There was a problem hiding this comment.
P2: The tray can retain a stale running or crashed child entry after a crash or respawn. Passing the writer into child pumps makes these updates concurrent with Supervisor::publish_state, but the writer only serializes state mutation—not the subsequent file write—so the write should also be serialized (or protected by a dedicated I/O mutex).
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At crates/stdiod/crates/edison-stdiod/src/daemon.rs, line 721:
<comment>The tray can retain a stale `running` or `crashed` child entry after a crash or respawn. Passing the writer into child pumps makes these updates concurrent with `Supervisor::publish_state`, but the writer only serializes state mutation—not the subsequent file write—so the write should also be serialized (or protected by a dedicated I/O mutex).</comment>
<file context>
@@ -688,6 +718,7 @@ impl Supervisor {
&enriched,
self.tunnel_outgoing.clone(),
sensitive_arg_values,
+ Some(self.state.clone()),
) {
Ok(child) => {
</file context>
There was a problem hiding this comment.
Fixed in 223b93d: StateWriter::update now performs the file write while still holding the state mutex, so mutation + write are a single critical section and a stale snapshot can no longer be written last. (Cubic's re-review didn't self-mark this thread, hence the manual note.)
Generated by Claude Code
There was a problem hiding this comment.
All reported issues were addressed across 6 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
1 issue found across 7 files (changes from recent commits).
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="crates/stdiod/crates/edison-stdiod/src/proc.rs">
<violation number="1" location="crates/stdiod/crates/edison-stdiod/src/proc.rs:468">
P3: This supplementary crash-publication path in `ChildServer::take_terminal_error` uses a single `try_wait()` to detect the child's exit, whereas the primary pump path (`report_terminal`) polls via `child_exit_status`, which retries up to 10 times over ~100ms precisely because a child's pipes close a moment before it becomes reapable. In the dead-but-not-yet-reapable window the supervisor's `take_terminal_error` will see `status == None`, so `mark_observed_exit`/`publish_crashed` here are skipped. The pumps still cover this window (and `crash_published` makes the write one-shot), so it isn't a visible failure today, but the new path claims to publish crashed on the supervisor reap and won't reliably do so in exactly the timing gap the existing poll helper was added to close. Reusing `child_exit_status` here keeps the two observers consistent.</violation>
</file>
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
| // This reap may be the first (or only) observation of the exit, | ||
| // so state.json has to hear about it from here too, not just from | ||
| // the stdout pump's death path. | ||
| self.diagnostics.publish_crashed(&self.server_id).await; |
There was a problem hiding this comment.
P3: This supplementary crash-publication path in ChildServer::take_terminal_error uses a single try_wait() to detect the child's exit, whereas the primary pump path (report_terminal) polls via child_exit_status, which retries up to 10 times over ~100ms precisely because a child's pipes close a moment before it becomes reapable. In the dead-but-not-yet-reapable window the supervisor's take_terminal_error will see status == None, so mark_observed_exit/publish_crashed here are skipped. The pumps still cover this window (and crash_published makes the write one-shot), so it isn't a visible failure today, but the new path claims to publish crashed on the supervisor reap and won't reliably do so in exactly the timing gap the existing poll helper was added to close. Reusing child_exit_status here keeps the two observers consistent.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At crates/stdiod/crates/edison-stdiod/src/proc.rs, line 468:
<comment>This supplementary crash-publication path in `ChildServer::take_terminal_error` uses a single `try_wait()` to detect the child's exit, whereas the primary pump path (`report_terminal`) polls via `child_exit_status`, which retries up to 10 times over ~100ms precisely because a child's pipes close a moment before it becomes reapable. In the dead-but-not-yet-reapable window the supervisor's `take_terminal_error` will see `status == None`, so `mark_observed_exit`/`publish_crashed` here are skipped. The pumps still cover this window (and `crash_published` makes the write one-shot), so it isn't a visible failure today, but the new path claims to publish crashed on the supervisor reap and won't reliably do so in exactly the timing gap the existing poll helper was added to close. Reusing `child_exit_status` here keeps the two observers consistent.</comment>
<file context>
@@ -690,6 +462,10 @@ impl ChildServer {
+ // This reap may be the first (or only) observation of the exit,
+ // so state.json has to hear about it from here too, not just from
+ // the stdout pump's death path.
+ self.diagnostics.publish_crashed(&self.server_id).await;
}
self.diagnostics
</file context>
There was a problem hiding this comment.
Declining the swap to child_exit_status here — the single try_wait is deliberate, because this call site sits on the supervisor's dispatch loop, not on a dying child's pump:
take_terminal_erroris reached for every frame routed at a child whosehas_exited()latch is set, and that includes the broken-stdin child whose process is alive by design (the T-72 self-healing case).try_waitcan't distinguish "alive" from "dead but not yet reapable" — both returnNone— sochild_exit_statuswould burn its full ~100ms poll budget on the supervisor's single event loop for every frame routed at a live child until reconciliation replaces it. That trades a real, repeated dispatch-loop stall for a window the pumps already own.- This path is the late observer by construction: it runs in reaction to the pump's terminal report, i.e. after
report_terminalhas already done the bounded poll right at the moment of death. If that poll got the status,crashedis already published; if it timed out, the process has almost certainly become reapable by the time frames route here, so the singletry_waitsucceeds. The residual gap is exactly the one the pump path covers, andcrash_publishedmakes the overlap one-shot.
So: pumps stay the primary observer with the poll; this reap stays an opportunistic, zero-latency secondary.
Generated by Claude Code
…on-close matching - state.json now reports crashed for a child whose pump path latched an exit, instead of hardcoding running for every live child; the crash is published from the pump path itself (nothing else wakes the supervisor when a child dies alone), guarded by name+pid so a late report cannot mark a respawned replacement. starting stays reserved: a stdio child is silent until the backend opens a session, so no honest signal distinguishes starting from idle-running. - Version-rejection matching now keys on the protocol_version token in a 1008 close reason rather than a fixed prefix, so the backend can reword its reason (it now reports a supported range, not a single version) without silently turning needs_upgrade into a retry loop. server_hello version differences are informational by contract. - PROTOCOL.md: T-09/T-13/T-62 rewritten for the supported-version window landed backend-side, T-69/T-72 tightened to the truthful state contract, new error-code table (server_offline, spawn_failed, server_unresponsive, stdio_tunnel_disabled, device_offline). ARCHITECTURE.md handshake and state.json sections aligned. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NgN7YdfLT41kWnWqe1MPZa
Review follow-ups on the truthful-state work, all in the daemon's state.json path. - StateWriter::update now writes the file while still holding the mutex. It used to clone the state, release the lock, and write afterwards, so two callers could rename in the opposite order to the one they mutated in and leave an older generation on disk until the next write. That became reachable when child pumps started publishing deaths alongside the supervisor. The write is blocking under an async mutex, which is fine for a few hundred bytes written on transitions only. - A child is reported crashed only when its exit was observed. The pump path calls report_terminal on a stdin write or flush failure without confirming the process is gone, so a child that is unreachable but still running was published as crashed next to its own live PID. The one-shot latch still drives the terminal server_offline exactly as before (an unwritable child is terminal for MCP, T-42/T-47); a new observed-exit flag, set only by an exit status or a reap in shutdown, is what state.json and the supervisor snapshot key off. Such a child stays running until the supervisor kills and respawns it, which the unchanged has_exited latch makes it do on the next reconciliation. - publish_crashed is guarded by its own one-shot flag instead of firing from every path that observes a death, so one death produces one write. The reported latch cannot serve as that guard: a broken stdin consumes it while the process lives, and the later real exit still has to reach state.json. PROTOCOL.md T-69/T-72 and ARCHITECTURE.md now say crashed means an observed process exit and spell out the broken-stdin case. Tests: a daemon-level test that a live child with a broken stdin reports server_offline and still snapshots as running; proc_tests asserts the two flags separately on the same event. The state.rs concurrency test pins the invariant (highest generation and its matching servers array end up on disk) via a new test-only StateWriter::new_at path override; against the old release-then-write ordering it reproduces the stale write in roughly one run in five, and it holds deterministically with the write inside the critical section. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NgN7YdfLT41kWnWqe1MPZa
…lings CI's large-file check errors on any .rs file a PR touches that is over 800 lines (crates/stdiod/scripts/check_large_files.sh). daemon.rs was already at 1027 and proc.rs at 873 on this branch, so both had to come down. Everything here is a pure structural move: no behaviour changes, no signature changes, no test changes beyond the imports the moves require. - daemon.rs (1027 -> 681): the inline test module moves to a sibling daemon_tests.rs, wired as `#[cfg(all(test, unix))] #[path = ...] mod tests;` the way config.rs and proc.rs already do it. The desired-state application layer (apply_snapshot, apply_delta, apply_spec_update, apply_env_update, restart_unresponsive and the try_spawn they share) moves to daemon_reconcile.rs. It stays an `impl Supervisor` block in a child module of daemon, so the call sites are untouched and the methods keep private-field access; only the five entry points needed pub(super). - proc.rs (873 -> 645): ChildDiagnostics and its impl, mark_entry_crashed, sanitize_diagnostic_line and the three STDERR_* bounds move to child_diagnostics.rs. proc.rs owns the process and its pumps, that module owns what the daemon says about a child that stopped working. The type, its methods, the two free functions and the `exited` flag become pub(crate) because they are now used across a module boundary; every other field stays private. The stray "One running child stdio MCP server" doc comment sat above ChildDiagnostics rather than ChildServer, which it describes; it is reattached to ChildServer instead of following the struct it never belonged to. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NgN7YdfLT41kWnWqe1MPZa
take_terminal_error marks observed_exit when its try_wait reaps a dead child, but only the stdout pump's death path published the crashed state to state.json. A child whose exit is first observed by this reap (the supervisor polling it after an MCP-terminal report) kept its stale entry. publish_crashed is one-shot and no-ops without an observed exit, so calling it from both observation sites is safe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NgN7YdfLT41kWnWqe1MPZa
d04f313 to
cee91eb
Compare
Stack 2/3 (base: #37). First daemon behavior change in the stack.
state.jsoncan now actually reportcrashed:Supervisor::snapshot_entriespreviously hardcodedrunningfor every live child. Crash publication now happens from the pump path itself (nothing else wakes the supervisor when a child dies alone), guarded by name+pid so a late report can't mark a same-named respawned replacement.startingstays reserved with the rationale documented: a stdio child is silent until the backend lazily opens a session, so there is no honest signal separating it from idle-running.needs_upgradenow triggers on any 1008 close whose reason containsprotocol_version(was: exact-prefix match), so the backend can reword its close reason without silently turning upgrade-required into an infinite retry loop.server_helloversion differences are informational per T-13.server_offline,spawn_failed,server_unresponsive,stdio_tunnel_disabled,device_offline).Consumer check done: the desktop tray's
ServerRunStateunion already declares'starting' | 'running' | 'crashed'and both consumers bucket non-runningstates correctly (packages/desktop/src/main/stdiod/types.ts,trayMenu.ts,StdiodEnableCard.tsx) — this PR makes an already-designed state reachable.Verified: fmt/clippy/
cargo test --workspace(112 tests), including a load-bearing check that reverting the snapshot fix fails the newcrashedtest.🤖 Generated with Claude Code
https://claude.ai/code/session_01NgN7YdfLT41kWnWqe1MPZa
Generated by Claude Code
Summary by cubic
Make
state.jsontruthful and ordered: mark a servercrashedonly after an observed process exit, and write updates under the mutex to prevent out-of-order snapshots. Version handling now matches the backend’s supported window; any 1008 that mentionsprotocol_versiontriggersneeds_upgrade, and differingserver_helloversions are informational.Bug Fixes
crashedonce when an exit is observed (from pumps or a reap), addressed by name+pid; live-but-unreachable children still emitserver_offlineand stayrunninguntil respawn.StateWriter::updatewrites while holding the lock so generations stay monotonic under concurrent updates.protocol_versionas upgrade-required; logserver_hello.protocol_versiondifferences at info and continue.Refactors
daemon_reconcile.rsandchild_diagnostics.rs. Added tests for truthfulstate.json, broken-stdin behavior, and concurrent update ordering.Written for commit cee91eb. Summary will update on new commits.