Skip to content

Reactivate threads when a provider resumes root work on a settled turn - #2141

Open
SawyerHood wants to merge 3 commits into
mainfrom
bb/fix-1646-codex-idle-status
Open

Reactivate threads when a provider resumes root work on a settled turn#2141
SawyerHood wants to merge 3 commits into
mainfrom
bb/fix-1646-codex-idle-status

Conversation

@SawyerHood

@SawyerHood SawyerHood commented Aug 21, 2026

Copy link
Copy Markdown
Collaborator

What was wrong

Thread status was derived from root turn/started and turn/completed alone (applyEventEffects in apps/server/src/internal/events.ts). Codex can keep streaming root work (commandExecution, fileChange, agentMessage, reasoning) on a turn id it already reported complete, then complete it a second time. The store accepts that work because the turn had a stored turn/started, but nothing re-evaluated status from it. The thread read idle (no spinner, bb thread wait returned early, all activity counters 0) while commands ran.

Two more layers had the same blind spot:

  • Since Agent providers as a first-class plugin surface (provider bridge protocol) #1640 the host daemon's intake grammar (ThreadEventGrammar) silently dropped the provider's second turn/completed for that turn (turn/settles-once), so even a server that reactivated the thread had no settle signal.
  • RuntimeTurnState held no active turn after the first completion, so thread/stop became a session release (kills the Codex child) instead of turn/interrupt, and the idle-session reaper could target a busy session.

Issue: #1646. Report: https://get-bb.github.io/reports/issues/1646.html

Relation to #1697: that PR has the right server diagnosis but settles the reactivated thread only through the second turn/completed, which current main's daemon grammar drops. The report verified live that a reactivated thread under #1697 stays active forever. It also reactivates on any previously completed turn (a stale item on an old turn after newer turns ran) and is based on a pre-#1640 main with a conflicting test file. This PR supersedes it.

What changed

  • packages/domain/src/provider-event.ts: isRootTurnWorkItem names the one rule all three layers share: not echoed user input, not a background task, not context compaction, not delegated child work. Compaction is excluded because pi's threshold compaction runs after agent_end and attaches its contextCompaction item to the turn that just closed (attach: "currentOrLast", Keep post-turn compaction pending while idle #1542) with no second turn/completed; counting it as a reopen would leave every compacted pi thread active until a manual stop (the verifier's blocker on the first revision of this PR).
  • apps/server/src/internal/events.ts: a root item/started on an idle or error thread applies run.started when its turn has a stored root turn/started and either is still open (an unlinked auxiliary turn's completion settled the thread while the root turn kept running) or is the thread's latest root turn (the provider continued a turn it already settled and will settle it again). Work on an older settled turn is stale and does not reactivate. A system/thread/interrupted since that turn's request wins over any late work. The server logs a warning with provider item type and turn id when it reactivates. hasThreadStopBeforeTurnStarted now shares hasThreadStopSinceTurnRequested (same query, optional upper bound). Thread rows read by the reopen check are memoized per event batch and evicted whenever the loop applies a lifecycle change, so root items streaming on an active thread cost one primary-key lookup per thread per batch rather than one per item. The root turn/started lookup is inlined because main removed the unused @bb/db helper in Remove dead code and simplify module surfaces #2140.
  • packages/provider-bridge-protocol/src/thread-event-grammar.ts: root work on a completed turn moves it back to startedTurnIds, so the provider's re-completion passes turn/settles-once and settles the thread. A re-turn/started for the reopened turn is still a violation.
  • packages/agent-runtime/src/runtime-turn-state.ts, runtime.ts: root work with no active turn reopens that turn as the active one, so thread/stop sends turn/interrupt with the right turn id, and the provider session is marked busy again.
  • packages/host-daemon-contract/src/protocol.ts: HOST_DAEMON_PROTOCOL_VERSION 146 -> 147. No payload shapes change, but the daemon now forwards an event it used to drop, and the server depends on it. A new server paired with an old daemon would leave reactivated threads active until a manual stop; the bump forces enrolled daemons to update.

Deviation from the report's proposal and the verifier's suggestion to settle the thread when the reopened work's last open item completes: item/completed is not a settle signal. Between one root item's completion and the next item's start there is model latency (the report's real-Codex control run streamed six sequential item/started+item/completed pairs over ten seconds on one turn), so settling there would flip the thread idle in every such gap, make bb thread wait return early inside the continuation, and fire queued-message auto-send mid-work: the symptom this PR fixes, reintroduced for the reactivated window. Settlement stays tied to the provider's turn/completed, which the grammar now lets through; the case where a provider reopens a turn and never re-completes it is the same failure mode as a provider that hangs mid-turn and is handled by the user's stop. The only first-party flow that produces root-looking work on a closed turn without a re-completion is compaction, which is now excluded by type.

Verifier minor not changed: RuntimeTurnState does not need to remember interrupted turns. A bb stop goes through stopThread, which calls forgetThreadRuntimeStateForProviderState (runtime.ts): that clears the thread's turn state, grammar state, and identity registration, and late provider events for a forgotten thread are dropped at identity resolution (resolveProviderEventThreadId), so the daemon cannot hold a reopened turn after a stop. The live check below confirms the stop path (turn/interrupt sent, then the thread forgotten). Blocking reopen only on provider-originated interrupted completions would desynchronize the daemon from the server, whose stop guard keys on system/thread/interrupted, not on the provider's completion status.

Not changed: grammar drops are still only reported through the runtime's onStderr, which the host daemon does not wire to its logger. That observability gap is separate from this fix.

How you verified

Fail before / pass after (sources swapped with git checkout <rev> -- <src files>, tests from this branch):

  • Against origin/main: apps/server/test/internal/internal-events-turn-reopen.test.ts fails 3 of 6 with AssertionError: expected 'idle' to be 'active' (reactivation on a re-completed turn; completion and late work in one batch; reactivation when an auxiliary turn settled the thread). packages/provider-bridge-protocol/test/protocol.test.ts "lets a turn that root work reopened settle again" fails (the second completion is a turn/settles-once violation). packages/agent-runtime/src/runtime-turn-state.test.ts reopened-turn cases fail with expected null to be 'turn-1' / 'turn-a'.
  • Against the first revision of this PR (46d102b, before the compaction exclusion): the new compaction cases fail at all three layers: server stays idle through post-turn context compaction on the completed turn with AssertionError: expected 'active' to be 'idle'; runtime does not reopen a settled turn for post-turn context compaction with expected 'turn-1' to be null; grammar does not reopen a settled turn for user input, delegated child work, or post-turn compaction (the second completion passed instead of being a violation).
  • All pass on this branch.

Turbo, run from the committed tree (git status --porcelain empty): typecheck for @bb/server @bb/host-daemon @bb/agent-runtime @bb/provider-bridge-protocol @bb/domain @bb/host-daemon-contract ./plugins/provider-*: Tasks: 15 successful, 15 total. test for @bb/host-daemon @bb/agent-runtime @bb/provider-bridge-protocol @bb/domain @bb/host-daemon-contract ./plugins/provider-* @bb/integration-tests: Tasks: 16 successful, 16 total (host-daemon 46 files, agent-runtime 31, provider-bridge-protocol 10, domain 24, integration-tests 25). @bb/server test: 195/196 files pass; the one failure is internal-skill-trees.test.ts file-mode 0644 vs 0664, which fails on clean main under this machine's umask 0002 and passes in CI.

End to end on a dev instance with the report's scripted codex app-server (PATH shim), scenario: turn/started, message, turn/completed, 8 s pause, commandExecution on the same turn for 12 s, message, second turn/completed, 8 s pause, then item/started {contextCompaction} on the same completed turn, 6 s, item/completed + thread/compacted:

  • Status polled every 2 s: idle after the first completion; active from the late commandExecution item/started (server log: Provider resumed root work on a settled turn; reactivating thread {"itemType":"commandExecution",...}); idle within 2 s of the provider's second turn/completed (stored, not dropped); idle throughout the post-turn compaction and after it (no second reactivation warning).
  • bb thread stop on the idle, compacted thread released the session (no turn/interrupt in the fake's log; the child exited), so the daemon held no active turn after the compaction item.
  • bb thread stop on a reactivated thread (scenario M1, stopped 15 s into the late work): the fake received turn/interrupt {"turnId":"turn-X-1"}, the server recorded system/thread/interrupted, the thread settled idle, and it stayed idle.

Real-data check against a private corpus of 307 recorded bb threads (166 Codex, 141 Claude Code): zero normal-flow cases of root work after a completed turn, so the new branch does not misfire on ordinary sessions. The 14 cases of root work after a completion were all host-daemon-restarted interruptions (server-synthesized turn/completed interrupted + system/thread/interrupted), which the stop guard excludes.

Fixes #1646

AGENT GENERATED: by Claude Opus 5

Independent verification

Verified by a second agent on e7dfa09bc (rebased on origin/main f6fb434ab; git merge-base --is-ancestor origin/main HEAD holds) in a separate worktree with its own dev instance.

Fail before / pass after (sources swapped with git checkout origin/main -- apps/server/src/internal/events.ts packages/agent-runtime/src/runtime-turn-state.ts packages/agent-runtime/src/runtime.ts packages/domain/src/provider-event.ts packages/provider-bridge-protocol/src/thread-event-grammar.ts packages/host-daemon-contract/src/protocol.ts, tests from this branch):

  • apps/server: pnpm exec vitest run test/internal/internal-events-turn-reopen.test.ts -> 3 failed / 3 passed; all three reactivation cases fail with AssertionError: expected 'idle' to be 'active'.
  • packages/agent-runtime: pnpm exec vitest run src/runtime-turn-state.test.ts -> 2 failed / 11 passed: expected null to be 'turn-1', expected null to be 'turn-a'.
  • packages/provider-bridge-protocol: pnpm exec vitest run test/protocol.test.ts -> 1 failed / 33 passed: "lets a turn that root work reopened settle again" (second completion is a violation on main).
  • After git checkout HEAD -- <same files>: 6/6, 13/13, 34/34 pass.

Turbo on the committed tree: typecheck for @bb/server @bb/host-daemon @bb/agent-runtime @bb/provider-bridge-protocol @bb/domain @bb/host-daemon-contract ./plugins/provider-* -> Tasks: 15 successful, 15 total. test for @bb/host-daemon @bb/agent-runtime @bb/provider-bridge-protocol @bb/domain @bb/host-daemon-contract ./plugins/provider-* -> Tasks: 13 successful, 13 total. @bb/server test: 195/196 files, 1828/1829 tests; the only failure is the known umask-dependent internal-skill-trees.test.ts (fails on clean main locally, passes in CI). gh pr checks: all required checks green (server, packages, integration, app-1..3, package smoke x2, checks).

Repro on the fixed branch (dev instance on its own ports/data dir, real codex bridge driven by the report's scripted codex app-server via PATH shim plus model/list, scenario: complete turn X, 8 s pause, commandExecution on X for 20 s, message, second turn/completed, 8 s pause, post-turn item/started {contextCompaction} on X, item/completed + thread/compacted):

05:25:42 status=idle   -> 05:25:44 status=active   (late commandExecution item/started on the completed turn)
server: WARN Provider resumed root work on a settled turn; reactivating thread {"itemType":"commandExecution",...}
05:26:00 status=active -> 05:26:02 status=idle     (second turn/completed stored, not dropped by the daemon grammar)
05:26:10..05:26:50     status=idle                  (post-turn compaction item on the same turn; no second reactivation)

On main the report shows idle for the whole run. Stop during the reactivated window (report scenario M1, stopped 26 s into the late work): bb thread wait --timeout 10 timed out (thread correctly active), bb thread stop returned in 2 s, the fake received turn/interrupt {"turnId":"turn-X-1"}, the thread read idle and stayed idle for the next 30 s, and the child process emitted nothing further (session released).

Residual risks, none blocking: a provider that streams root work on its latest settled turn and never re-completes it stays active until a manual stop (only bridges that vouch provider turn ids, i.e. Codex, can place an item on a closed turn; the delta assembler falls back to "no turn" for everyone else, and pi's post-turn compaction is excluded by type). getActiveStoredTurnId is still null during the reactivated window, so an in-app send dispatches a new turn/start into the busy session instead of a steer, same as main. Grammar drops remain unlogged by the daemon.

AGENT GENERATED: by Claude Opus 5

SawyerHood and others added 3 commits August 21, 2026 04:48
Thread status was derived from root turn/started and turn/completed
alone. Codex can keep streaming root work (commands, file changes,
messages) on a turn id it already reported complete, and an unlinked
auxiliary turn's completion can settle the thread while the root turn is
still running. Either way the thread read idle (no spinner, bb thread
wait returned early) while commands ran.

- Domain: isRootTurnWorkItem names the one rule (not user input, not a
  background task, not delegated child work) shared by all three layers.
- Server: a root item/started on an idle/error thread reactivates it
  when its turn is still open or is the thread's latest root turn, and
  no stop was recorded since that turn's request. Stale work on an older
  settled turn does not reactivate.
- Host grammar: root work on a completed turn moves it back to started,
  so the provider's second turn/completed passes turn/settles-once and
  settles the thread instead of being dropped.
- Runtime: RuntimeTurnState reopens the active turn from that work so
  thread/stop interrupts it rather than releasing the session, and the
  provider session is marked busy again.

Fixes #1646

Co-Authored-By: Claude <noreply@anthropic.com>
The daemon's intake grammar now forwards a provider's second
turn/completed for a turn that root work reopened. The server relies on
that event to settle a thread it reactivated on such work; an older
daemon still drops it (turn/settles-once) and would leave the thread
active until a manual stop. The version mismatch forces enrolled daemons
to update.

Co-Authored-By: Claude <noreply@anthropic.com>
Pi's threshold compaction runs after agent_end and attaches its
contextCompaction item to the turn that just closed, with no second
turn/completed (#1542). isRootTurnWorkItem counted it as root work, so
every compacted pi thread flipped active and stayed there until a stop.
Compaction is provider maintenance, not agent work; exclude it at the
one shared rule so the server status, the daemon grammar, and the
runtime active-turn state all agree.

The server reopen check memoizes thread rows per event batch and evicts
a row whenever the loop applies a lifecycle change, so root items on an
active thread cost one lookup per thread rather than one per item. The
root turn/started lookup is inlined because main removed the unused
@bb/db helper in #2140.

Co-Authored-By: Claude <noreply@anthropic.com>
@SawyerHood
SawyerHood force-pushed the bb/fix-1646-codex-idle-status branch from 46d102b to e7dfa09 Compare August 21, 2026 05:12
@SawyerHood
SawyerHood marked this pull request as ready for review August 21, 2026 05:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Codex thread shows idle (no spinner) while actively running commands

1 participant