feat(voice): background session real-time state + silent context-only insertion#6518
feat(voice): background session real-time state + silent context-only insertion#6518toubatbrian wants to merge 6 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
…sertion Adds two capabilities to background sessions: - BackgroundContext.set_state(): report JSON-serializable/raw-text real-time state (validated with the function-tool return-value rules). A generated lk_background_state tool lets the voice LLM query it on demand — e.g. when the user asks what is currently happening. Setting state never inserts context and never schedules a reply. - ctx.send(..., silent=True): silent context-only insertion — the rendered update enters the current Agent context and session history without scheduling a spoken reply. BackgroundMessageReceived events carry a new `silent` flag. Both generated tool names are reserved; conflicts fail at construction or activity tool setup. Examples updated: deep-research reports its phase/topic/stats per stage, and the Claude Code example tracks files changed via set_state and silent insertions. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
814730d to
b3161c9
Compare
3179c58 to
6767cde
Compare
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if state is None: | ||
| return "The background session has not reported any state yet." | ||
| return state |
There was a problem hiding this comment.
🟡 Background status of zero or empty values shows up blank to the voice agent
When a background session reports a status value that happens to be zero, false, or empty (returned by the status tool at livekit-agents/livekit/agents/voice/background_session.py:517), the value is turned into an empty string before it reaches the voice model, so the agent sees nothing instead of the reported status.
Impact: A background session that reports a legitimate but "falsy" status (0, False, empty list/dict) leaves the voice agent with no information, even though the feature explicitly allows numbers and booleans.
Falsy tool-output collapse in make_function_call_output
The generated tool returns the stored state verbatim (background_session.py:517). The tool-output path serializes it with output=str(output or "") at livekit-agents/livekit/agents/llm/utils.py:768. For any valid-but-falsy state — 0, 0.0, False, "", [], {} — output or "" evaluates to "", so the model receives an empty string rather than the reported value. set_state explicitly permits these via _is_valid_function_output (background_session.py:309), and the state is None guard (background_session.py:515) only special-cases None, so these falsy values are passed through and then lost. This is pre-existing behavior in the shared tool-output path but is newly reachable/impacted by user-controlled state values.
Was this helpful? React with 👍 or 👎 to provide feedback.
Moves the Claude Code example to examples/voice_agents/coding_agents/ and restructures it around explicit harness channels: - SPEAK: an in-process MCP send_to_user tool lets Claude Code push verbatim spoken messages (questions, confirmations, summaries) - CONTEXT-ONLY: narration text is inserted silently so the voice LLM sees the working narrative without voicing it - STATE: live status/task/recent-activity/files-changed via set_state, queried through lk_background_state - INTERRUPT: voice messages arriving mid-turn call client.interrupt() and become the next user turn Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A handoff during the awaited update_chat_ctx() left a silent insert on the previous Agent only — unlike enqueue(), insert_only() had no later delivery pass to copy it over. Reuse _retarget_until_stable() so the context-only note follows the current Agent, mirroring deferred-reply handoff semantics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both methods performed the same lock + eager context/history insertion + _PendingReply construction and diverged only at the end. Collapse into one private _insert(schedule_reply=...): the deferred path buffers and starts the delivery task; the silent path retargets inline. No behavior change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| def set_state(self, state: Any) -> None: | ||
| if self._closed: | ||
| raise RuntimeError(f"background session {self.definition.id!r} is closed") | ||
| if not _is_valid_function_output(state): | ||
| raise ValueError( | ||
| "background state must be a valid function-tool return value: a string, " | ||
| "number, bool, None, or a JSON-serializable list/dict/tuple/set of those" | ||
| ) | ||
| self._state = state |
There was a problem hiding this comment.
🟡 A background session that reports an empty status is shown as having reported nothing
When a background session reports a state of "none" (set_state(None) at livekit-agents/livekit/agents/voice/background_session.py:306-314), it is stored as the same value used to mean "nothing reported yet", so the status tool tells the voice agent no status has been reported even though one was.
Impact: A background session that deliberately reports an empty/none status is indistinguishable from one that never reported anything, so the voice agent gives the user a misleading "not reported yet" answer.
None sentinel collides with a valid reported value
The runtime initializes self._state = None and set_state accepts None because _is_valid_function_output(None) is True (the validation type set includes type(None)). The generated tool at livekit-agents/livekit/agents/voice/background_session.py:515-517 returns the default "The background session has not reported any state yet." whenever state is None, so an explicit set_state(None) is reported as unset. A distinct sentinel (e.g. an _unset object separate from the stored value) would disambiguate the two cases.
Prompt for agents
In _BackgroundRuntime (background_session.py), self._state is initialized to None and set_state(None) is accepted as a valid value (since _is_valid_function_output(None) is True). The generated lk_background_state tool treats state is None as "not reported yet". This makes an explicitly reported None state indistinguishable from an unset state. Consider introducing a distinct unset sentinel object (e.g. a module-level _UNSET = object()) used for the initial value and checked in _create_background_state_tool instead of `state is None`, so that a background that deliberately reports None is surfaced verbatim rather than as the default message.
Was this helpful? React with 👍 or 👎 to provide feedback.
…-code
Findings from a live cue-driven integrity test with debug-mode probes:
deep-research:
- the "done, listening" state was dead — the loop-top set_state
overwrote it in the same millisecond, so lk_background_state could
never report the last report path. Idle state is now a variable the
round outcome updates (done / no-sources / inconclusive) and persists
while waiting for the next question.
- voice agent instructions now require forwarding the user's request
with constraints intact ("just start, no clarifying questions" was
being trimmed, costing an extra clarify round-trip).
claude-code:
- send_to_user now uses session.say() so the SPEAK channel is truly
verbatim (ctx.send() schedules a generated reply that the voice LLM
paraphrased in every live sample); docstrings updated to match.
- interrupt guard: a burst of split messages interrupts once — later
pieces queue instead of killing the correction turn they belong to
(inbox-empty check + 2s grace after turn start).
- voice agent instructions forbid claiming completion on dispatch (the
LLM previously announced files created/deleted seconds before Claude
Code did the work).
- files_changed now drops deleted files at report time; REPO_ROOT
actually points at the repo root.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Revert the session.say() shortcut — background sessions are by design unable to speak to the end user directly; everything flows through the voice overlay. send_to_user goes back to ctx.send(), and the docstring, comment, and Claude Code system prompt now describe the relay honestly (the voice agent delivers the message and may lightly rephrase it) instead of claiming verbatim delivery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 participant
Overview
Stacked on #6517. Adds two capabilities to background sessions, both aimed at the "Hey, what's going on right now?" case and at keeping the voice agent informed without making it speak:
1. Real-time state —
ctx.set_state()+lk_background_stateBackgroundContext.set_state(state)stores the session's current state. The value follows the function-tool return-value rules (string, number, bool, None, or JSON-serializable list/dict/tuple/set — validated eagerly with the same_is_valid_function_outputcheck used for tool outputs, since the state is ultimately returned through a tool).lk_background_state(background_session_id), returns the reported state verbatim (serialized by the normal tool-output path). Its description deterministically lists every configured session; unknown IDs return aToolError; unset state returns a clear "not reported yet" message._RESERVED_BACKGROUND_TOOL_NAMES); conflicting session tools fail at construction, conflicting Agent tools fail during activity tool setup /update_tools()without mutating state.2. Silent context-only insertion —
ctx.send(..., silent=True)update_template) is eagerly inserted into the current Agent's chat context andsession.history, but is not buffered for a scheduled reply — the session never speaks it. The voice agent can draw on it when the user asks._ReplyScheduler._insert_items_locked(), shared byenqueue()and the newinsert_only(); close/race semantics are identical.BackgroundMessageReceivedevents carry a newsilent: boolflag (defaultFalse, backward compatible).Examples updated
phase/topic/angles/sources_read/claims_under_reviewat every pipeline stage, andlast_reportwhen idle — so mid-research "what are you researching right now?" gets a precise answer.status/task/files_changedviaset_state, and silently inserts a context note for each file Claude Code edits. Both voice agents' instructions point the LLM atlk_background_statefor status questions.Testing
tests/test_background_session.py: state tool round-trip (dict + raw text) and default message, unknown-IDToolError,set_statenever inserting context/scheduling replies, return-shape validation + post-close rejection, silent send inserting context exactly once with no reply and asilent=Trueevent, silent send post-close rejection, and reserved-name conflict forlk_background_state.Status: under active live testing 🧪
This PR is currently being exercised in live voice sessions against both example agents; treat it as functionally complete but still stabilizing.
Testing experience so far
send_to_useris now always mediated through the voice agent — the background session never speaks directly (7c5dc75).insert_only/enqueuewere unified on a shared_insertpath (0ed9041) after handoff testing exposed divergence.INTERRUPT_GRACEwindow queue instead of killing their own turn. The working tree carries temporary instrumentation around the interrupt guard,send_to_user, and state reporting while this is validated (not part of the PR).Notes on PR state
background_session.py,reply_scheduler.py, tool-executor refactor) are covered by the hermetic suite (1280 passed / 4 skipped, strict mypy + ruff clean) and considered stable.Meta-note: this status section was written and pushed to the PR by the very thing it describes — a LiveKit voice-driven Claude Code background session. The user talked, Claude Code typed,
send_to_userrelayed, and nobody got interrupted (outside the 2-second grace window). Dogfooding level: the dog wrote its own release notes. 🐕🎙️