Reactivate threads when a provider resumes root work on a settled turn - #2141
Open
SawyerHood wants to merge 3 commits into
Open
Reactivate threads when a provider resumes root work on a settled turn#2141SawyerHood wants to merge 3 commits into
SawyerHood wants to merge 3 commits into
Conversation
Thread status was derived from root turn/started and turn/completed alone. Codex can keep streaming root work (commands, file changes, messages) on a turn id it already reported complete, and an unlinked auxiliary turn's completion can settle the thread while the root turn is still running. Either way the thread read idle (no spinner, bb thread wait returned early) while commands ran. - Domain: isRootTurnWorkItem names the one rule (not user input, not a background task, not delegated child work) shared by all three layers. - Server: a root item/started on an idle/error thread reactivates it when its turn is still open or is the thread's latest root turn, and no stop was recorded since that turn's request. Stale work on an older settled turn does not reactivate. - Host grammar: root work on a completed turn moves it back to started, so the provider's second turn/completed passes turn/settles-once and settles the thread instead of being dropped. - Runtime: RuntimeTurnState reopens the active turn from that work so thread/stop interrupts it rather than releasing the session, and the provider session is marked busy again. Fixes #1646 Co-Authored-By: Claude <noreply@anthropic.com>
The daemon's intake grammar now forwards a provider's second turn/completed for a turn that root work reopened. The server relies on that event to settle a thread it reactivated on such work; an older daemon still drops it (turn/settles-once) and would leave the thread active until a manual stop. The version mismatch forces enrolled daemons to update. Co-Authored-By: Claude <noreply@anthropic.com>
Pi's threshold compaction runs after agent_end and attaches its contextCompaction item to the turn that just closed, with no second turn/completed (#1542). isRootTurnWorkItem counted it as root work, so every compacted pi thread flipped active and stayed there until a stop. Compaction is provider maintenance, not agent work; exclude it at the one shared rule so the server status, the daemon grammar, and the runtime active-turn state all agree. The server reopen check memoizes thread rows per event batch and evicts a row whenever the loop applies a lifecycle change, so root items on an active thread cost one lookup per thread rather than one per item. The root turn/started lookup is inlined because main removed the unused @bb/db helper in #2140. Co-Authored-By: Claude <noreply@anthropic.com>
SawyerHood
force-pushed
the
bb/fix-1646-codex-idle-status
branch
from
August 21, 2026 05:12
46d102b to
e7dfa09
Compare
SawyerHood
marked this pull request as ready for review
August 21, 2026 05:33
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
Thread status was derived from root
turn/startedandturn/completedalone (applyEventEffectsinapps/server/src/internal/events.ts). Codex can keep streaming root work (commandExecution,fileChange,agentMessage,reasoning) on a turn id it already reported complete, then complete it a second time. The store accepts that work because the turn had a storedturn/started, but nothing re-evaluated status from it. The thread readidle(no spinner,bb thread waitreturned early, all activity counters 0) while commands ran.Two more layers had the same blind spot:
ThreadEventGrammar) silently dropped the provider's secondturn/completedfor that turn (turn/settles-once), so even a server that reactivated the thread had no settle signal.RuntimeTurnStateheld no active turn after the first completion, sothread/stopbecame a session release (kills the Codex child) instead ofturn/interrupt, and the idle-session reaper could target a busy session.Issue: #1646. Report: https://get-bb.github.io/reports/issues/1646.html
Relation to #1697: that PR has the right server diagnosis but settles the reactivated thread only through the second
turn/completed, which current main's daemon grammar drops. The report verified live that a reactivated thread under #1697 stays active forever. It also reactivates on any previously completed turn (a stale item on an old turn after newer turns ran) and is based on a pre-#1640 main with a conflicting test file. This PR supersedes it.What changed
packages/domain/src/provider-event.ts:isRootTurnWorkItemnames the one rule all three layers share: not echoed user input, not a background task, not context compaction, not delegated child work. Compaction is excluded because pi's threshold compaction runs afteragent_endand attaches itscontextCompactionitem to the turn that just closed (attach: "currentOrLast", Keep post-turn compaction pending while idle #1542) with no secondturn/completed; counting it as a reopen would leave every compacted pi thread active until a manual stop (the verifier's blocker on the first revision of this PR).apps/server/src/internal/events.ts: a rootitem/startedon an idle or error thread appliesrun.startedwhen its turn has a stored rootturn/startedand either is still open (an unlinked auxiliary turn's completion settled the thread while the root turn kept running) or is the thread's latest root turn (the provider continued a turn it already settled and will settle it again). Work on an older settled turn is stale and does not reactivate. Asystem/thread/interruptedsince that turn's request wins over any late work. The server logs a warning with provider item type and turn id when it reactivates.hasThreadStopBeforeTurnStartednow shareshasThreadStopSinceTurnRequested(same query, optional upper bound). Thread rows read by the reopen check are memoized per event batch and evicted whenever the loop applies a lifecycle change, so root items streaming on an active thread cost one primary-key lookup per thread per batch rather than one per item. The rootturn/startedlookup is inlined because main removed the unused@bb/dbhelper in Remove dead code and simplify module surfaces #2140.packages/provider-bridge-protocol/src/thread-event-grammar.ts: root work on a completed turn moves it back tostartedTurnIds, so the provider's re-completion passesturn/settles-onceand settles the thread. A re-turn/startedfor the reopened turn is still a violation.packages/agent-runtime/src/runtime-turn-state.ts,runtime.ts: root work with no active turn reopens that turn as the active one, sothread/stopsendsturn/interruptwith the right turn id, and the provider session is marked busy again.packages/host-daemon-contract/src/protocol.ts:HOST_DAEMON_PROTOCOL_VERSION146 -> 147. No payload shapes change, but the daemon now forwards an event it used to drop, and the server depends on it. A new server paired with an old daemon would leave reactivated threads active until a manual stop; the bump forces enrolled daemons to update.Deviation from the report's proposal and the verifier's suggestion to settle the thread when the reopened work's last open item completes:
item/completedis not a settle signal. Between one root item's completion and the next item's start there is model latency (the report's real-Codex control run streamed six sequentialitem/started+item/completedpairs over ten seconds on one turn), so settling there would flip the thread idle in every such gap, makebb thread waitreturn early inside the continuation, and fire queued-message auto-send mid-work: the symptom this PR fixes, reintroduced for the reactivated window. Settlement stays tied to the provider'sturn/completed, which the grammar now lets through; the case where a provider reopens a turn and never re-completes it is the same failure mode as a provider that hangs mid-turn and is handled by the user's stop. The only first-party flow that produces root-looking work on a closed turn without a re-completion is compaction, which is now excluded by type.Verifier minor not changed:
RuntimeTurnStatedoes not need to remember interrupted turns. A bb stop goes throughstopThread, which callsforgetThreadRuntimeStateForProviderState(runtime.ts): that clears the thread's turn state, grammar state, and identity registration, and late provider events for a forgotten thread are dropped at identity resolution (resolveProviderEventThreadId), so the daemon cannot hold a reopened turn after a stop. The live check below confirms the stop path (turn/interruptsent, then the thread forgotten). Blocking reopen only on provider-originatedinterruptedcompletions would desynchronize the daemon from the server, whose stop guard keys onsystem/thread/interrupted, not on the provider's completion status.Not changed: grammar drops are still only reported through the runtime's
onStderr, which the host daemon does not wire to its logger. That observability gap is separate from this fix.How you verified
Fail before / pass after (sources swapped with
git checkout <rev> -- <src files>, tests from this branch):origin/main:apps/server/test/internal/internal-events-turn-reopen.test.tsfails 3 of 6 withAssertionError: expected 'idle' to be 'active'(reactivation on a re-completed turn; completion and late work in one batch; reactivation when an auxiliary turn settled the thread).packages/provider-bridge-protocol/test/protocol.test.ts"lets a turn that root work reopened settle again" fails (the second completion is aturn/settles-onceviolation).packages/agent-runtime/src/runtime-turn-state.test.tsreopened-turn cases fail withexpected null to be 'turn-1'/'turn-a'.stays idle through post-turn context compaction on the completed turnwithAssertionError: expected 'active' to be 'idle'; runtimedoes not reopen a settled turn for post-turn context compactionwithexpected 'turn-1' to be null; grammardoes not reopen a settled turn for user input, delegated child work, or post-turn compaction(the second completion passed instead of being a violation).Turbo, run from the committed tree (
git status --porcelainempty):typecheckfor@bb/server @bb/host-daemon @bb/agent-runtime @bb/provider-bridge-protocol @bb/domain @bb/host-daemon-contract ./plugins/provider-*:Tasks: 15 successful, 15 total.testfor@bb/host-daemon @bb/agent-runtime @bb/provider-bridge-protocol @bb/domain @bb/host-daemon-contract ./plugins/provider-* @bb/integration-tests:Tasks: 16 successful, 16 total(host-daemon 46 files, agent-runtime 31, provider-bridge-protocol 10, domain 24, integration-tests 25).@bb/servertest: 195/196 files pass; the one failure isinternal-skill-trees.test.tsfile-mode 0644 vs 0664, which fails on clean main under this machine's umask 0002 and passes in CI.End to end on a dev instance with the report's scripted
codex app-server(PATH shim), scenario:turn/started, message,turn/completed, 8 s pause,commandExecutionon the same turn for 12 s, message, secondturn/completed, 8 s pause, thenitem/started {contextCompaction}on the same completed turn, 6 s,item/completed+thread/compacted:idleafter the first completion;activefrom the latecommandExecutionitem/started(server log:Provider resumed root work on a settled turn; reactivating thread {"itemType":"commandExecution",...});idlewithin 2 s of the provider's secondturn/completed(stored, not dropped);idlethroughout the post-turn compaction and after it (no second reactivation warning).bb thread stopon the idle, compacted thread released the session (noturn/interruptin the fake's log; the child exited), so the daemon held no active turn after the compaction item.bb thread stopon a reactivated thread (scenario M1, stopped 15 s into the late work): the fake receivedturn/interrupt {"turnId":"turn-X-1"}, the server recordedsystem/thread/interrupted, the thread settledidle, and it stayed idle.Real-data check against a private corpus of 307 recorded bb threads (166 Codex, 141 Claude Code): zero normal-flow cases of root work after a
completedturn, so the new branch does not misfire on ordinary sessions. The 14 cases of root work after a completion were allhost-daemon-restartedinterruptions (server-synthesizedturn/completed interrupted+system/thread/interrupted), which the stop guard excludes.Fixes #1646
Independent verification
Verified by a second agent on
e7dfa09bc(rebased onorigin/mainf6fb434ab;git merge-base --is-ancestor origin/main HEADholds) in a separate worktree with its own dev instance.Fail before / pass after (sources swapped with
git checkout origin/main -- apps/server/src/internal/events.ts packages/agent-runtime/src/runtime-turn-state.ts packages/agent-runtime/src/runtime.ts packages/domain/src/provider-event.ts packages/provider-bridge-protocol/src/thread-event-grammar.ts packages/host-daemon-contract/src/protocol.ts, tests from this branch):apps/server:pnpm exec vitest run test/internal/internal-events-turn-reopen.test.ts-> 3 failed / 3 passed; all three reactivation cases fail withAssertionError: expected 'idle' to be 'active'.packages/agent-runtime:pnpm exec vitest run src/runtime-turn-state.test.ts-> 2 failed / 11 passed:expected null to be 'turn-1',expected null to be 'turn-a'.packages/provider-bridge-protocol:pnpm exec vitest run test/protocol.test.ts-> 1 failed / 33 passed: "lets a turn that root work reopened settle again" (second completion is a violation on main).git checkout HEAD -- <same files>: 6/6, 13/13, 34/34 pass.Turbo on the committed tree:
typecheckfor@bb/server @bb/host-daemon @bb/agent-runtime @bb/provider-bridge-protocol @bb/domain @bb/host-daemon-contract ./plugins/provider-*->Tasks: 15 successful, 15 total.testfor@bb/host-daemon @bb/agent-runtime @bb/provider-bridge-protocol @bb/domain @bb/host-daemon-contract ./plugins/provider-*->Tasks: 13 successful, 13 total.@bb/servertest: 195/196 files, 1828/1829 tests; the only failure is the known umask-dependentinternal-skill-trees.test.ts(fails on clean main locally, passes in CI).gh pr checks: all required checks green (server, packages, integration, app-1..3, package smoke x2, checks).Repro on the fixed branch (dev instance on its own ports/data dir, real codex bridge driven by the report's scripted
codex app-servervia PATH shim plusmodel/list, scenario: complete turn X, 8 s pause,commandExecutionon X for 20 s, message, secondturn/completed, 8 s pause, post-turnitem/started {contextCompaction}on X,item/completed+thread/compacted):On main the report shows
idlefor the whole run. Stop during the reactivated window (report scenario M1, stopped 26 s into the late work):bb thread wait --timeout 10timed out (thread correctly active),bb thread stopreturned in 2 s, the fake receivedturn/interrupt {"turnId":"turn-X-1"}, the thread readidleand stayed idle for the next 30 s, and the child process emitted nothing further (session released).Residual risks, none blocking: a provider that streams root work on its latest settled turn and never re-completes it stays active until a manual stop (only bridges that vouch provider turn ids, i.e. Codex, can place an item on a closed turn; the delta assembler falls back to "no turn" for everyone else, and pi's post-turn compaction is excluded by type).
getActiveStoredTurnIdis still null during the reactivated window, so an in-app send dispatches a newturn/startinto the busy session instead of a steer, same as main. Grammar drops remain unlogged by the daemon.