feat(voice): auto-disable realtime server-side turn detection#6495
Open
longcw wants to merge 11 commits into
Open
feat(voice): auto-disable realtime server-side turn detection#6495longcw wants to merge 11 commits into
longcw wants to merge 11 commits into
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
longcw
force-pushed
the
longc/realtime-auto-disable-td
branch
from
July 21, 2026 09:27
9c2cc63 to
9caf4ba
Compare
longcw
force-pushed
the
longc/realtime-barge-in
branch
from
July 22, 2026 01:49
a557e26 to
2926544
Compare
longcw
force-pushed
the
longc/realtime-auto-disable-td
branch
from
July 22, 2026 01:49
f9c7088 to
b525a7a
Compare
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
longcw
force-pushed
the
longc/realtime-auto-disable-td
branch
from
July 22, 2026 02:57
48359d0 to
9c988f3
Compare
A realtime model transcribes internally and commits turns manually, so barge-in can gatekeep the overlapping user turn by withholding the commit instead of holding and flushing STT transcripts. Relax _resolve_interruption_detection so a realtime model with server-side turn detection disabled no longer requires a separate STT; the STT pipeline path is unchanged.
When a realtime model runs client-side barge-in with server-side turn detection disabled, an overlapping utterance the adaptive detector classifies as a backchannel must not create a user turn. Track the backchannel verdict per turn and skip the end-of-turn commit; clear the model's audio buffer so the backchannel audio doesn't leak into the next turn.
Move the backchannel verdict from an AgentActivity flag reset on VAD start-of-speech to a turn-scoped signal owned by the audio recognition layer and surfaced on _EndOfTurnInfo. A second start-of-speech within a logical turn no longer resets the verdict, and a late verdict can no longer leak into the next turn (it is only recorded while the current turn has an unresolved overlap). Clear the buffered input audio at the drop point rather than at the verdict, so it also covers the case where the agent is still speaking and the verdict has not arrived, and no longer truncates a turn that escalates from a backchannel into an interruption.
Clear the turn's backchannel verdict when a new speech segment starts without overlapping agent speech, so a real turn following a backchannel in the same logical turn commits instead of being dropped. A stutter while the agent is still speaking keeps the verdict, so pure backchannels are still dropped.
…irst When the agent finishes speaking while the user is still overlapping, the overlap is force-ended and a synthesized is_interruption=False event is emitted. That isn't a real verdict — the user may still be mid-turn — but it was latched as a backchannel and carried to the end of the turn, dropping genuine speech. Tag the agent-driven overlap-end via agent_ended on the sentinel and OverlappingSpeechEvent, and skip the backchannel latch for it so the turn falls back to committing.
The backchannel verdict persisted a True from an earlier overlap into a later inconclusive overlap within the same agent turn, so a user turn that continued past the agent could be wrongly dropped. Reset it on every speech onset so each overlap re-derives its own verdict.
A confirmed backchannel over the agent left its audio in the realtime input buffer, so it prefixed the next committed turn. Clear the buffer when the backchannel is confirmed while the user is between segments; deferring when the user is already mid next-segment avoids clipping a real turn. This also applies when an STT is attached, since the realtime session buffers input regardless.
Disable a realtime model's server-side turn detection for the session when the user configured client-side turn-taking (a TurnDetector, "vad", "manual", or an explicit interruption mode) and the model supports it, so client-side EoT/barge-in works without also passing turn_detection=None to the RealtimeModel. - add RealtimeCapabilities.can_disable_turn_detection; OpenAI sets it from the ctor turn_detection given-ness - add session(turn_detection_disabled=...), applied on the initial session.update via the per-session opts copy so the model is never mutated (stays reusable) - AgentActivity resolves the effective state once (_rt_turn_detection_enabled) and routes every server-side turn-detection check through it Other realtime plugins accept-and-ignore the flag (can_disable_turn_detection stays False): xAI (client turn-taking not stable in testing) and Gemini/others (no commit_audio/clear_audio; turns are activity-signal based).
RealtimeModel.update_options force-synced turn_detection=self._opts.turn_detection to every session unconditionally, so changing an unrelated field (e.g. voice) re-enabled server VAD on a session that opted out of server-side turn detection. Only propagate turn_detection when the caller set it; an explicit value still propagates so callers can re-enable it.
…tection On an agent handoff the realtime session is reused as-is. If the new agent resolves server-side turn detection differently (it can override turn_detection, interruption, or vad), the reused session kept the previous agent's mode, causing duplicate or missing replies. Refuse reuse on a mismatch so a fresh session is created with the correct setting.
_resolve_interruption_detection and the on_end_of_turn backchannel-drop guard still read the model's static capabilities.turn_detection instead of the session-resolved _rt_turn_detection_enabled. When a realtime model with built-in turn detection has it auto-disabled for the session (client-side turn-taking configured), can_gatekeep stayed False so the adaptive detector was never created, and the backchannel-drop path was skipped — silently disabling the client-side barge-in this stack is meant to enable.
longcw
force-pushed
the
longc/realtime-barge-in
branch
from
July 23, 2026 06:01
82a241c to
2b70881
Compare
longcw
force-pushed
the
longc/realtime-auto-disable-td
branch
from
July 23, 2026 06:01
9c988f3 to
eca4e01
Compare
| capabilities=llm.RealtimeCapabilities( | ||
| message_truncation=True, | ||
| turn_detection=turn_detection is not None, | ||
| can_disable_turn_detection=not is_given(turn_detection), |
Member
There was a problem hiding this comment.
nit: we should update with_azure too
| # model to auto-generate a tool reply (auto_tool_reply_generation=True). | ||
| self._pending_auto_tool_reply_fut: asyncio.Future[None] | None = None | ||
|
|
||
| def _resolve_rt_turn_detection_enabled(self) -> bool: |
Member
There was a problem hiding this comment.
should we handle update_options with turn_detection for realtime too?
Member
There was a problem hiding this comment.
we should handle with_azure too since it assigns the default turn detection before init.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Auto-disables a realtime model's server-side turn detection for the session when the user configured client-side turn-taking (a TurnDetector,
vad,manual, or an explicit interruption mode) and the model supports it, so client-side EoT/barge-in works without also passingturn_detection=Noneto the RealtimeModel.Adds
RealtimeCapabilities.can_disable_turn_detection(OpenAI sets it from the ctorturn_detectiongiven-ness) and asession(turn_detection_disabled=...)seam applied on the initialsession.updatevia the per-session opts copy, so the model is never mutated and stays reusable.AgentActivityresolves the effective state once (_rt_turn_detection_enabled) and routes every server-side turn-detection check through it.Other realtime plugins accept-and-ignore the flag (
can_disable_turn_detectionstays False): xAI (client turn-taking not stable in testing) and Gemini/others (nocommit_audio/clear_audio; turns are activity-signal based).Stacked on
longc/realtime-barge-in; this PR targets that branch, not main.