Support AssemblyAI inference context carryover - #2057
Support AssemblyAI inference context carryover#2057rosetta-livekit-bot[bot] wants to merge 10 commits into
Conversation
🦋 Changeset detectedLatest commit: e2feaad The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Guard provider updates by live capabilities and propagate model capability changes so sessions and fallback adapters forward context only while supported. Co-authored-by: Cursor <cursoragent@cursor.com>
Cover the emitted session.update payload and the runtime previous_context_n_turns opt-out required by the Rosetta stack contract. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: rosetta-livekit-bot[bot] <282703043+rosetta-livekit-bot[bot]@users.noreply.github.com> Co-authored-by: Toubat <brian.yin@livekit.io>
Resolve conflicts by keeping both STT context-option and user-input transcription tests, and by combining fallback-adapter test imports. Co-authored-by: Cursor <cursoragent@cursor.com>
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
| cloudHostname, | ||
| roomId, | ||
| jobId, | ||
| metadata: options.metadata, | ||
| }); | ||
|
|
There was a problem hiding this comment.
🟡 Cloud log records are missing the agent identity labels that traces get
The agent name and cloud agent/deployment identifiers are collected but not handed to the log uploader (initPinoCloudExporter({ metadata: options.metadata }) at agents/src/telemetry/traces.ts:497-502), so uploaded logs cannot be attributed to the agent even though the code comment states they carry it.
Impact: Log records shipped to LiveKit Cloud lack the agent name and deployment labels, so agent-level log filtering/insights show nothing for them while traces do.
Why the identity attributes never reach the log pipeline
setupCloudTracer builds baseMetadata with lk.agent_name, lk.cloud_agent_id and lk.deployment_id (agents/src/telemetry/traces.ts:400-421) and merges it with the caller metadata into sessionMetadata (agents/src/telemetry/traces.ts:423). sessionMetadata is used for the trace resource and for the MetadataSpanProcessor on both provider paths, but the logging path forwards only options.metadata (the job's simulation/redaction flags) to initPinoCloudExporter, whose transport stamps room_id/job_id plus whatever config.metadata contains (agents/src/telemetry/pino_otel_transport.ts:134-137, :201-208). The inline comment at agents/src/telemetry/traces.ts:404-408 explicitly claims the agent name is "Included in both the resource (traces) and the session metadata (spans + logs)", which does not hold.
| cloudHostname, | |
| roomId, | |
| jobId, | |
| metadata: options.metadata, | |
| }); | |
| initPinoCloudExporter({ | |
| cloudHostname, | |
| roomId, | |
| jobId, | |
| metadata: sessionMetadata, |
Was this helpful? React with 👍 or 👎 to provide feedback.
Single commit on top of latest origin/main (db3263b). Squashes the PR stack (#2058 STT context options + #2057 AssemblyAI carryover) and the Devin-review fixes. - inference STT: chatContext capability + agent_context carryover for the U3 Pro family, 1750-char tail truncation, assistant-only forwarding; drop stale agent_context when switching to a model that can't accept it (Devin review). - session: sttContextOptions.forwardChatContext (default true), keytermsOptions deprecated/mapped; forwarding synced on capabilities_changed + model swap. - fallback_adapter: re-aggregate all dynamic child capabilities on capabilities_changed (alignedTranscript/keyterms/chatContext), not just chatContext. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
18e0d25 to
10f3002
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Integrate latest main (incl. #2152) into the verified AssemblyAI context-carryover port. Conflicts resolved in agents/src/inference/stt.ts and stt.test.ts by keeping both the port's chat-context carryover and main's word-alignment capability (alignedTranscriptForModel). No behavioral changes beyond the merge.
After merging main, inference STT recomputes alignedTranscript/keyterms/chatContext
on updateOptions({ model }) and emits capabilities_changed, but FallbackAdapter's
forwarder only re-read chatContext — so a child switching to a model without word
timings left the adapter advertising alignedTranscript it no longer had. Re-aggregate
every runtime-dynamic capability, mirroring the constructor. (Devin review)
…odel updateOptions merges modelOptions and never removes keys, and connectWs serializes the whole baseline as settings.extra, so after switching away from a U3 Pro model the stale agent_context would still be sent to a model that rejects it. Drop it on the transition and propagate the removal to live streams. (Devin review) Note: this diverges from Python livekit-agents, which retains agent_context across the transition.
10f3002 to
cbaa1fe
Compare
…lds on model switch The plugin STT computed chatContext + U3-Pro-only param validation only in the constructor, but updateOptions() accepts a new speechModel. With carryover defaulting on for the U3 Pro family, agentContext is auto-populated by _pushConversationItem; a later switch to a non-U3 model left the stale agentContext serialized into #connectWS / UpdateConfiguration and kept capabilities.chatContext advertising support, so the session kept forwarding assistant replies. updateOptions now, when speechModel changes: recomputes chatContext via updateCapabilities(), rejects U3-Pro-only fields passed alongside a non-U3 model (mirroring the constructor), and drops any stale U3-Pro-only fields (agentContext, previousContextNTurns, prompt, voiceFocus, voiceFocusThreshold, mode) the new model cannot accept, propagating the clearing to live streams. _pushConversationItem now guards on capabilities.chatContext. Mirrors agents/src/inference/stt.ts updateOptions.
cbaa1fe to
1507bb4
Compare
| chatContext: supportsChatContext(this.opts.model), | ||
| alignedTranscript: alignmentModels.every(alignedTranscriptForModel) ? 'word' : false, | ||
| }); | ||
|
|
There was a problem hiding this comment.
🟡 Switching speech models leaves provider-specific tuning fields behind, which the new model rejects
Only the carried-over reply text is removed when the speech model changes (delete cleared.agent_context at agents/src/inference/stt.ts:621), leaving the other model-specific settings in place, so the next connection sends settings the new model does not accept and can fail to start.
Impact: Users who switch models mid-session can end up with a speech-to-text connection that the provider rejects, breaking transcription.
Only `agent_context` is dropped, while the other U3-Pro-only extras persist
AssemblyAIOptions documents five fields as "Only supported with u3-rt-pro": agent_context, previous_context_n_turns, voice_focus, voice_focus_threshold and mode (agents/src/inference/stt.ts:136-146). updateOptions merges modelOptions and never removes keys, and connectWs serializes the whole baseline as settings.extra (agents/src/inference/stt.ts:669-675). The new clearing block only removes agent_context, so after e.g. updateOptions({ model: 'assemblyai/universal-streaming' }) a previously set voice_focus/mode/previous_context_n_turns is still shipped to a model that rejects U3-Pro-only parameters — the exact failure the block was added to prevent.
The AssemblyAI plugin does this correctly: it clears every entry of U3_PRO_ONLY_PARAMS when the model changes (plugins/assemblyai/src/stt.ts:274-287), so the inference path is inconsistent with it.
Was this helpful? React with 👍 or 👎 to provide feedback.
…_context The inference-STT clearing block added for agent_context only removed that one key, but AssemblyAIOptions documents five U3-Pro-only fields (agent_context, previous_context_n_turns, voice_focus, voice_focus_threshold, mode). connectWs serializes the whole modelOptions baseline as settings.extra, so a stale voice_focus/mode/ previous_context_n_turns would still be shipped to a non-U3 model and rejected. Clear all of them via ASSEMBLYAI_U3_PRO_ONLY_EXTRAS, matching U3_PRO_ONLY_PARAMS in plugins/assemblyai/src/stt.ts.
2 participants
Summary
agent_contextcarryover for supported U3 Pro family models.Source diff coverage
livekit-agents/livekit/agents/inference/stt.pyagents/src/inference/stt.tsports the linked PR's U3 Pro carryover model gate, option shape, capability wiring, and assistant-message forwarding. JS-specific safeguards prevent unsupported/disabled fallback children from being mutated and synchronize capability listeners across model changes.tests/test_inference_stt_context.pyagents/src/inference/stt.test.tscovers capability gating, assistant-only forwarding, 1,750-character tail truncation, ignored items, activesession.updatewire payloads, supported/unsupported/supported/disabled transitions, and listener cleanup.This PR is intentionally the lower stack. PR #2058 supersedes its temporary carryover-control surface with the current session-level Python architecture.
Validation
efb7c37e0f6a630f8761f6abe4a274391e42ab09.Cue voice E2E
sid_e3340fa16681.efb7c37e0f6a630f8761f6abe4a274391e42ab09.voicewith real LiveKit room transport, voice injection/recording, and Cartesiasonic-3through LiveKit inference TTS.session.create/session.updatepayloads and supplied deterministic transcripts; the LLM was deterministic.true/false/true/false; only the supported transition forwarded. Unsupported, disabled, and user items never appeared inagent_context.Bcharacters, SHA-256a35f084996bba9fddefe6fb9be79b29763dc6ecc810fada308db8601777abf58.1/1while active to0/0after close.CONTINUITY-2057and real TTS audio.~/.cue-cli/sessions/sid_e3340fa16681/events.jsonl, two commandresult.jsonfiles, sessionrecording.wav, and both command WAVs. All were flushed and inspected directly.Ported from livekit/agents#6466
Original PR description
already introduced in the assembly ai plugin
Post-rebase
origin/mainto clear conflicts infallback_adapter.test.tsimports andagent_session.test.tsdescribe blocks.efb7c37e0f6a630f8761f6abe4a274391e42ab09.