fix: preserve adaptive interruption across tool calls - #6564
Conversation
|
This looks like the right fix — treating the thinking gap as a real speech boundary and restarting overlap inference when playout resumes addresses the root cause at the signal source, where my #6550 only made the stream tolerate the redundant start sentinel. The metrics and remote-session coverage go beyond what mine attempted, too. Closing #6550 in favor of this. If the hermetic stream-level regression tests from it would be useful here (a fake bargein gateway driving |
|
|
||
| def _on_pipeline_reply_done(self, _: asyncio.Task[None]) -> None: | ||
| if not self._speech_q and (not self._current_speech or self._current_speech.done()): | ||
| was_speaking = self._session.agent_state == "speaking" |
There was a problem hiding this comment.
question: what is the difference between session.agent_state and self._agent_speaking?
There was a problem hiding this comment.
(Chiming in from #6550, since this distinction was the heart of #6548 — hope it helps.)
They live at different layers and deliberately diverge mid-turn:
-
session.agent_stateis the session-level lifecycle (listening/thinking/speaking), updated viaAgentActivity._update_agent_state(). A reply that produces tool calls flips it to"thinking"between segments; it is what applications observe throughagent_state_changed. -
AudioRecognition._agent_speakingis a private recognition-side flag meaning "an agent playout window is open for interruption/endpointing purposes". It is set by_on_start_of_agent_speech()(first audio frame of playout) and cleared only inside_on_end_of_agent_speech()— which is intentionally not called when the reply produced tool calls, so it staysTruestraight through the thinking gap. It gates overlap tracking in_on_start_of_speech(), the transcript-ignore window, and_should_ignore_input_audio().
So during a tool-call gap the two disagree by design: agent_state == "thinking" while _agent_speaking == True. was_speaking as written captures "playout was audibly active at reply-done" (session view) rather than "the recognition-side speech window is still open" — which one is wanted here is of course the author's call.
Bug: Adaptive interruption could silently drop a user overlap when agent playout resumed after a tool call because the interruption stream received mismatched speech boundaries.
Treat thinking gaps as real speech boundaries and restart overlap inference when playout resumes mid-utterance. Agent-ended overlaps remain inconclusive and no longer count as backchannels; remote forwarding is noted until the protocol carries that marker.
Fixes #6548. Fixes #6548.