Skip to content

stdiod: truthful state.json server states + token-based version-close matching - #38

Open
Miyamura80 wants to merge 4 commits into
mainfrom
claude/tunnel-2-daemon-state
Open

stdiod: truthful state.json server states + token-based version-close matching#38
Miyamura80 wants to merge 4 commits into
mainfrom
claude/tunnel-2-daemon-state

Conversation

@Miyamura80

@Miyamura80 Miyamura80 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Stack 2/3 (base: #37). First daemon behavior change in the stack.

  • state.json can now actually report crashed: Supervisor::snapshot_entries previously hardcoded running for every live child. Crash publication now happens from the pump path itself (nothing else wakes the supervisor when a child dies alone), guarded by name+pid so a late report can't mark a same-named respawned replacement. starting stays reserved with the rationale documented: a stdio child is silent until the backend lazily opens a session, so there is no honest signal separating it from idle-running.
  • Version-close matching widened from prefix to token: needs_upgrade now triggers on any 1008 close whose reason contains protocol_version (was: exact-prefix match), so the backend can reword its close reason without silently turning upgrade-required into an infinite retry loop. server_hello version differences are informational per T-13.
  • PROTOCOL.md T-09/T-13/T-62/T-69/T-72 updated to match; new error-code table (server_offline, spawn_failed, server_unresponsive, stdio_tunnel_disabled, device_offline).

Consumer check done: the desktop tray's ServerRunState union already declares 'starting' | 'running' | 'crashed' and both consumers bucket non-running states correctly (packages/desktop/src/main/stdiod/types.ts, trayMenu.ts, StdiodEnableCard.tsx) — this PR makes an already-designed state reachable.

Verified: fmt/clippy/cargo test --workspace (112 tests), including a load-bearing check that reverting the snapshot fix fails the new crashed test.

🤖 Generated with Claude Code

https://claude.ai/code/session_01NgN7YdfLT41kWnWqe1MPZa


Generated by Claude Code


Summary by cubic

Make state.json truthful and ordered: mark a server crashed only after an observed process exit, and write updates under the mutex to prevent out-of-order snapshots. Version handling now matches the backend’s supported window; any 1008 that mentions protocol_version triggers needs_upgrade, and differing server_hello versions are informational.

  • Bug Fixes

    • Publish crashed once when an exit is observed (from pumps or a reap), addressed by name+pid; live-but-unreachable children still emit server_offline and stay running until respawn.
    • StateWriter::update writes while holding the lock so generations stay monotonic under concurrent updates.
    • Treat any 1008 containing protocol_version as upgrade-required; log server_hello.protocol_version differences at info and continue.
  • Refactors

    • Split desired-state reconcile and per-child diagnostics into daemon_reconcile.rs and child_diagnostics.rs. Added tests for truthful state.json, broken-stdin behavior, and concurrent update ordering.

Written for commit cee91eb. Summary will update on new commits.

Review in cubic

@qodo-code-review

Copy link
Copy Markdown

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 9 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/stdiod/crates/edison-stdiod/src/daemon.rs">

<violation number="1" location="crates/stdiod/crates/edison-stdiod/src/daemon.rs:721">
P2: The tray can retain a stale `running` or `crashed` child entry after a crash or respawn. Passing the writer into child pumps makes these updates concurrent with `Supervisor::publish_state`, but the writer only serializes state mutation—not the subsequent file write—so the write should also be serialized (or protected by a dedicated I/O mutex).</violation>
</file>

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread crates/stdiod/crates/edison-stdiod/src/proc.rs
&enriched,
self.tunnel_outgoing.clone(),
sensitive_arg_values,
Some(self.state.clone()),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: The tray can retain a stale running or crashed child entry after a crash or respawn. Passing the writer into child pumps makes these updates concurrent with Supervisor::publish_state, but the writer only serializes state mutation—not the subsequent file write—so the write should also be serialized (or protected by a dedicated I/O mutex).

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At crates/stdiod/crates/edison-stdiod/src/daemon.rs, line 721:

<comment>The tray can retain a stale `running` or `crashed` child entry after a crash or respawn. Passing the writer into child pumps makes these updates concurrent with `Supervisor::publish_state`, but the writer only serializes state mutation—not the subsequent file write—so the write should also be serialized (or protected by a dedicated I/O mutex).</comment>

<file context>
@@ -688,6 +718,7 @@ impl Supervisor {
             &enriched,
             self.tunnel_outgoing.clone(),
             sensitive_arg_values,
+            Some(self.state.clone()),
         ) {
             Ok(child) => {
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 223b93d: StateWriter::update now performs the file write while still holding the state mutex, so mutation + write are a single critical section and a stale snapshot can no longer be written last. (Cubic's re-review didn't self-mark this thread, hence the manual note.)


Generated by Claude Code

Comment thread crates/stdiod/crates/edison-stdiod/src/proc.rs

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread crates/stdiod/crates/edison-stdiod/src/proc.rs

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 7 files (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="crates/stdiod/crates/edison-stdiod/src/proc.rs">

<violation number="1" location="crates/stdiod/crates/edison-stdiod/src/proc.rs:468">
P3: This supplementary crash-publication path in `ChildServer::take_terminal_error` uses a single `try_wait()` to detect the child's exit, whereas the primary pump path (`report_terminal`) polls via `child_exit_status`, which retries up to 10 times over ~100ms precisely because a child's pipes close a moment before it becomes reapable. In the dead-but-not-yet-reapable window the supervisor's `take_terminal_error` will see `status == None`, so `mark_observed_exit`/`publish_crashed` here are skipped. The pumps still cover this window (and `crash_published` makes the write one-shot), so it isn't a visible failure today, but the new path claims to publish crashed on the supervisor reap and won't reliably do so in exactly the timing gap the existing poll helper was added to close. Reusing `child_exit_status` here keeps the two observers consistent.</violation>
</file>

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

// This reap may be the first (or only) observation of the exit,
// so state.json has to hear about it from here too, not just from
// the stdout pump's death path.
self.diagnostics.publish_crashed(&self.server_id).await;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3: This supplementary crash-publication path in ChildServer::take_terminal_error uses a single try_wait() to detect the child's exit, whereas the primary pump path (report_terminal) polls via child_exit_status, which retries up to 10 times over ~100ms precisely because a child's pipes close a moment before it becomes reapable. In the dead-but-not-yet-reapable window the supervisor's take_terminal_error will see status == None, so mark_observed_exit/publish_crashed here are skipped. The pumps still cover this window (and crash_published makes the write one-shot), so it isn't a visible failure today, but the new path claims to publish crashed on the supervisor reap and won't reliably do so in exactly the timing gap the existing poll helper was added to close. Reusing child_exit_status here keeps the two observers consistent.

Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At crates/stdiod/crates/edison-stdiod/src/proc.rs, line 468:

<comment>This supplementary crash-publication path in `ChildServer::take_terminal_error` uses a single `try_wait()` to detect the child's exit, whereas the primary pump path (`report_terminal`) polls via `child_exit_status`, which retries up to 10 times over ~100ms precisely because a child's pipes close a moment before it becomes reapable. In the dead-but-not-yet-reapable window the supervisor's `take_terminal_error` will see `status == None`, so `mark_observed_exit`/`publish_crashed` here are skipped. The pumps still cover this window (and `crash_published` makes the write one-shot), so it isn't a visible failure today, but the new path claims to publish crashed on the supervisor reap and won't reliably do so in exactly the timing gap the existing poll helper was added to close. Reusing `child_exit_status` here keeps the two observers consistent.</comment>

<file context>
@@ -690,6 +462,10 @@ impl ChildServer {
+            // This reap may be the first (or only) observation of the exit,
+            // so state.json has to hear about it from here too, not just from
+            // the stdout pump's death path.
+            self.diagnostics.publish_crashed(&self.server_id).await;
         }
         self.diagnostics
</file context>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Declining the swap to child_exit_status here — the single try_wait is deliberate, because this call site sits on the supervisor's dispatch loop, not on a dying child's pump:

  • take_terminal_error is reached for every frame routed at a child whose has_exited() latch is set, and that includes the broken-stdin child whose process is alive by design (the T-72 self-healing case). try_wait can't distinguish "alive" from "dead but not yet reapable" — both return None — so child_exit_status would burn its full ~100ms poll budget on the supervisor's single event loop for every frame routed at a live child until reconciliation replaces it. That trades a real, repeated dispatch-loop stall for a window the pumps already own.
  • This path is the late observer by construction: it runs in reaction to the pump's terminal report, i.e. after report_terminal has already done the bounded poll right at the moment of death. If that poll got the status, crashed is already published; if it timed out, the process has almost certainly become reapable by the time frames route here, so the single try_wait succeeds. The residual gap is exactly the one the pump path covers, and crash_published makes the overlap one-shot.

So: pumps stay the primary observer with the poll; this reap stays an opportunistic, zero-latency secondary.


Generated by Claude Code

@Miyamura80
Miyamura80 changed the base branch from claude/tunnel-1-spec-goldens to main August 11, 2026 06:16
Claude and others added 4 commits August 11, 2026 06:17
…on-close matching

- state.json now reports crashed for a child whose pump path latched an
  exit, instead of hardcoding running for every live child; the crash is
  published from the pump path itself (nothing else wakes the supervisor
  when a child dies alone), guarded by name+pid so a late report cannot
  mark a respawned replacement. starting stays reserved: a stdio child
  is silent until the backend opens a session, so no honest signal
  distinguishes starting from idle-running.
- Version-rejection matching now keys on the protocol_version token in a
  1008 close reason rather than a fixed prefix, so the backend can
  reword its reason (it now reports a supported range, not a single
  version) without silently turning needs_upgrade into a retry loop.
  server_hello version differences are informational by contract.
- PROTOCOL.md: T-09/T-13/T-62 rewritten for the supported-version
  window landed backend-side, T-69/T-72 tightened to the truthful state
  contract, new error-code table (server_offline, spawn_failed,
  server_unresponsive, stdio_tunnel_disabled, device_offline).
  ARCHITECTURE.md handshake and state.json sections aligned.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NgN7YdfLT41kWnWqe1MPZa
Review follow-ups on the truthful-state work, all in the daemon's
state.json path.

- StateWriter::update now writes the file while still holding the mutex.
  It used to clone the state, release the lock, and write afterwards, so
  two callers could rename in the opposite order to the one they mutated
  in and leave an older generation on disk until the next write. That
  became reachable when child pumps started publishing deaths alongside
  the supervisor. The write is blocking under an async mutex, which is
  fine for a few hundred bytes written on transitions only.
- A child is reported crashed only when its exit was observed. The pump
  path calls report_terminal on a stdin write or flush failure without
  confirming the process is gone, so a child that is unreachable but
  still running was published as crashed next to its own live PID. The
  one-shot latch still drives the terminal server_offline exactly as
  before (an unwritable child is terminal for MCP, T-42/T-47); a new
  observed-exit flag, set only by an exit status or a reap in shutdown,
  is what state.json and the supervisor snapshot key off. Such a child
  stays running until the supervisor kills and respawns it, which the
  unchanged has_exited latch makes it do on the next reconciliation.
- publish_crashed is guarded by its own one-shot flag instead of firing
  from every path that observes a death, so one death produces one write.
  The reported latch cannot serve as that guard: a broken stdin consumes
  it while the process lives, and the later real exit still has to reach
  state.json.

PROTOCOL.md T-69/T-72 and ARCHITECTURE.md now say crashed means an
observed process exit and spell out the broken-stdin case.

Tests: a daemon-level test that a live child with a broken stdin reports
server_offline and still snapshots as running; proc_tests asserts the two
flags separately on the same event. The state.rs concurrency test pins
the invariant (highest generation and its matching servers array end up
on disk) via a new test-only StateWriter::new_at path override; against
the old release-then-write ordering it reproduces the stale write in
roughly one run in five, and it holds deterministically with the write
inside the critical section.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NgN7YdfLT41kWnWqe1MPZa
…lings

CI's large-file check errors on any .rs file a PR touches that is over
800 lines (crates/stdiod/scripts/check_large_files.sh). daemon.rs was
already at 1027 and proc.rs at 873 on this branch, so both had to come
down. Everything here is a pure structural move: no behaviour changes,
no signature changes, no test changes beyond the imports the moves
require.

- daemon.rs (1027 -> 681): the inline test module moves to a sibling
  daemon_tests.rs, wired as `#[cfg(all(test, unix))] #[path = ...] mod
  tests;` the way config.rs and proc.rs already do it. The desired-state
  application layer (apply_snapshot, apply_delta, apply_spec_update,
  apply_env_update, restart_unresponsive and the try_spawn they share)
  moves to daemon_reconcile.rs. It stays an `impl Supervisor` block in a
  child module of daemon, so the call sites are untouched and the
  methods keep private-field access; only the five entry points needed
  pub(super).
- proc.rs (873 -> 645): ChildDiagnostics and its impl, mark_entry_crashed,
  sanitize_diagnostic_line and the three STDERR_* bounds move to
  child_diagnostics.rs. proc.rs owns the process and its pumps, that
  module owns what the daemon says about a child that stopped working.
  The type, its methods, the two free functions and the `exited` flag
  become pub(crate) because they are now used across a module boundary;
  every other field stays private.

The stray "One running child stdio MCP server" doc comment sat above
ChildDiagnostics rather than ChildServer, which it describes; it is
reattached to ChildServer instead of following the struct it never
belonged to.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NgN7YdfLT41kWnWqe1MPZa
take_terminal_error marks observed_exit when its try_wait reaps a dead
child, but only the stdout pump's death path published the crashed state
to state.json. A child whose exit is first observed by this reap (the
supervisor polling it after an MCP-terminal report) kept its stale entry.
publish_crashed is one-shot and no-ops without an observed exit, so
calling it from both observation sites is safe.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NgN7YdfLT41kWnWqe1MPZa
@Miyamura80
Miyamura80 force-pushed the claude/tunnel-2-daemon-state branch from d04f313 to cee91eb Compare August 11, 2026 06:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants