fix(interruption): keep the overlap armed when agent speech restarts mid-interrupt - #2131
fix(interruption): keep the overlap armed when agent speech restarts mid-interrupt#2131toubatbrian wants to merge 2 commits into
Conversation
🦋 Changeset detectedLatest commit: f058820 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 85cade1f97
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| expect(audioSendCount(h.ws)).toBeGreaterThan(0); | ||
|
|
||
| await h.close(); |
There was a problem hiding this comment.
If any preceding expect or waitFor rejects, execution never reaches this cleanup call, leaving the harness's infinite audio pump and WebSocket responder loops plus its interruption task running. Those leaked responders can then process sockets created by later tests through the global MockWebSocket.instances array, contaminating results or preventing the test worker from exiting; register each harness for afterEach cleanup or wrap its use in try/finally immediately after creation.
Useful? React with 👍 / 👎.
85cade1 to
b47d6a7
Compare
Adds the first coverage of the audio-to-verdict path as AudioRecognition wires it: room audio -> interruption stream channel -> audio transformer -> WS transport, with a mock gateway standing in for the service. Regression coverage for "every overlap is classified as backchannel": the classifier can only return `isInterruption: true` from the inference-response path, which requires audio to actually reach the transport while the overlap is open. Also exports `OverlappingSpeechEvent` by name from `voice/events.ts` so `overlapping_speech` handlers can be typed without reaching into `inference/interruption/types.js`. Co-authored-by: Cursor <cursoragent@cursor.com>
…mid-interrupt `overlapSpeechStarted` is the gate that lets a user's overlapping audio reach the interruption model, and the only thing that raises it is a VAD start-of-speech. Two events cleared it while the user was still talking. Because VAD never re-announces speech already under way, nothing could re-arm it for the rest of the agent turn: every remaining frame was dropped, no inference request was made, and the agent talked straight through the interruption. A transport failover rebuilds InterruptionStreamBase from scratch — all of its state lives in a setupTransform() closure, unlike Python where the equivalent flags are instance attributes a reconnect leaves alone. Replaying `agent-speech-started` restored only half of it. Hand the in-progress overlap to the replacement stream instead, via a distinct `agent-speech-resumed` sentinel so the real one keeps meaning "new turn, reset everything". A second speech segment in one turn (a queued SpeechHandle, or the reply after a tool call) raises `agent-speech-started` again with no `agent-speech-ended` in between, because onPipelineReplyDone only reports the end once the speech queue drains. Preserve an open overlap across that; a genuine new turn still resets the overlap, audio buffer, cache and counters. Also marks the forwarding task's rejection handled at creation: it rejects as soon as the stream is torn down, but the `finally` only attaches a handler after the retry backoff, so the rejection surfaced as an unhandled rejection. Co-authored-by: Cursor <cursoragent@cursor.com>
b47d6a7 to
f058820
Compare
toubatbrian
changed the title
toubatbrian
changed the title
toubatbrian
changed the title
|
Closing: this came out of a misdiagnosis on my side. The reported symptom — the agent's audio and its transcript disagreeing after a barge-in, with the lag compounding over a conversation — is caused by the inference gateway answering a single Measured on its own, #2146's branch contains only The defect it fixes is real and has its own red/green test, and the branch is not deleted. If it ever shows up in practice it is worth reproposing on its own merit rather than as a barge-in fix. |
1 participant
Bottom of a four-PR stack: #2131 → #2132 → #2136 → #2142. Base:
main.Problem
A barge-in can be silently swallowed, because two paths in
interruption_stream.tsdisarm a live overlap:agent-speech-started, wiping the overlap state the old stream was holding.agent-speech-startedfires once per TTS playout with no idempotence guard, while the matching end fires once per turn. A tool-calling turn emits two starts with no end between them, and the second resets an overlap that is still open.Either way the user is talking into a classifier that has stopped listening, and only a VAD start-of-speech re-arms it — which will not fire for speech already under way.
Fix
Skip the overlap reset when a start sentinel arrives while an overlap is already open, and carry overlap state across a failover.
Testing
Three new cases in
interruption_pipeline.test.tsfail onmainand pass here; the existing "resets overlap state on a new agent turn" case is the control and passes on both sides. Also fixes an unhandledAPIConnectionErrorrejectionmainemits during failover. Carried here too: the pipeline test harness and theOverlappingSpeechEventre-export, salvaged from the closed #2116.Python parity
Same bug —
InterruptionStreamBase._forward_data(inference/interruption.py:617-622) resets unconditionally, start and end sentinels sharing onecase; the port back splits that case. One metrics-only gap the other way: JS hardcodesoverlapCount = 1on the replacement stream and loses the audio prefix from its first post-failover window. Verdicts are unaffected.Scope — read this before merging
Written while investigating a live report of agent audio running behind the transcript after a barge-in. That report is now traced elsewhere: the gateway emits multiple
donemessages persession.flushon some TTS paths and the client returns on the first, truncating the reply and handing a still-streaming socket back to the pool. Fixes are in flight separately (#2144, #2146).This PR does not address it. A swallowed barge-in is a different observable, and the report still reproduces with the whole stack applied. Merge on the evidence above and nothing more.
Closes #2117.