Skip to content

fix(openai): discard stale pooled Responses WebSocket connections#6514

Open
JSap0914 wants to merge 1 commit into
livekit:mainfrom
JSap0914:fix/responses-ws-pool-stale-connection
Open

fix(openai): discard stale pooled Responses WebSocket connections#6514
JSap0914 wants to merge 1 commit into
livekit:mainfrom
JSap0914:fix/responses-ws-pool-stale-connection

Conversation

@JSap0914

Copy link
Copy Markdown

Summary

Fixes #6513.

The OpenAI Responses WebSocket transport (openai.responses.LLM(use_websocket=True)) pools connections through ConnectionPool. ConnectionPool.get() only validated max_session_duration before returning an available connection — it never checked whether the connection was still open.

After concurrent requests grow the pool (a single Responses socket handles one in-flight response, so parallel work creates multiple sockets) and those sockets sit idle, a peer / NAT gateway / VPN / proxy can close them. _ResponsesWebsocket.generate_response() then acquires a dead socket and fails at ws.send_str():

failed to send request over WebSocket, retrying in 0.1s
failed to send request over WebSocket, retrying in 2.0s

Each stale pooled connection burns one outer LLM retry attempt (and its backoff) before a healthy connection is reached, adding noticeable latency to a live voice turn.

Change

  • Add an optional validate_cb: Callable[[T], bool] | None to ConnectionPool. It is invoked before a pooled connection is handed out; connections that fail validation are removed/closed and get() keeps searching (or creates a fresh connection).
  • Wire the Responses pool with validate_cb=lambda ws: not ws.closed, so a socket that was closed while idle is discarded before it reaches the caller instead of surfacing a send failure and retry.
  • The existing retry behavior is preserved for connections that become unusable during the actual send (a live race), which validation cannot catch.

This follows the minimal approach suggested in the issue.

Tests

tests/test_connection_pool.py:

  • test_get_discards_invalid_connection — a healthy connection is returned even when stale (closed) connections are present in the pool.
  • test_get_creates_new_when_all_invalid — when every pooled connection fails validation, get() discards them and creates a fresh connection.
  • test_get_reuses_when_no_validate_cb — behavior is unchanged when no validate_cb is configured.
$ python -m pytest tests/test_connection_pool.py -q
7 passed

ruff format --check and ruff check pass on the changed files; mypy clean on connection_pool.py.

The OpenAI Responses WebSocket transport pools connections via
ConnectionPool. ConnectionPool.get() only checked max_session_duration
before returning an available connection, not whether the connection was
still open. After concurrent requests grow the pool and the sockets sit
idle, a peer/NAT/proxy can close them. generate_response() then acquired
a dead socket and failed at ws.send_str(), consuming one outer LLM retry
(and its backoff) per stale connection before reaching a healthy one,
adding latency to live voice turns.

Add an optional validate_cb to ConnectionPool that is invoked before a
pooled connection is handed out; connections that fail validation are
discarded and get() continues searching (or creates a fresh one). Wire
the Responses pool with validate_cb=lambda ws: not ws.closed so remotely
closed idle sockets are dropped without surfacing a retry. The existing
retry path still covers sockets that die during the send itself.

Adds regression tests covering discard-on-invalid, fallback-to-new when
all pooled connections are invalid, and unchanged reuse when no
validate_cb is configured.

Fixes livekit#6513
@JSap0914
JSap0914 requested a review from a team as a code owner July 22, 2026 18:34
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenAI Responses WebSocket pool may reuse stale connections after concurrent requests

2 participants