fix(openai): discard stale pooled Responses WebSocket connections#6514
Open
JSap0914 wants to merge 1 commit into
Open
fix(openai): discard stale pooled Responses WebSocket connections#6514JSap0914 wants to merge 1 commit into
JSap0914 wants to merge 1 commit into
Conversation
The OpenAI Responses WebSocket transport pools connections via ConnectionPool. ConnectionPool.get() only checked max_session_duration before returning an available connection, not whether the connection was still open. After concurrent requests grow the pool and the sockets sit idle, a peer/NAT/proxy can close them. generate_response() then acquired a dead socket and failed at ws.send_str(), consuming one outer LLM retry (and its backoff) per stale connection before reaching a healthy one, adding latency to live voice turns. Add an optional validate_cb to ConnectionPool that is invoked before a pooled connection is handed out; connections that fail validation are discarded and get() continues searching (or creates a fresh one). Wire the Responses pool with validate_cb=lambda ws: not ws.closed so remotely closed idle sockets are dropped without surfacing a retry. The existing retry path still covers sockets that die during the send itself. Adds regression tests covering discard-on-invalid, fallback-to-new when all pooled connections are invalid, and unchanged reuse when no validate_cb is configured. Fixes livekit#6513
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #6513.
The OpenAI Responses WebSocket transport (
openai.responses.LLM(use_websocket=True)) pools connections throughConnectionPool.ConnectionPool.get()only validatedmax_session_durationbefore returning an available connection — it never checked whether the connection was still open.After concurrent requests grow the pool (a single Responses socket handles one in-flight response, so parallel work creates multiple sockets) and those sockets sit idle, a peer / NAT gateway / VPN / proxy can close them.
_ResponsesWebsocket.generate_response()then acquires a dead socket and fails atws.send_str():Each stale pooled connection burns one outer LLM retry attempt (and its backoff) before a healthy connection is reached, adding noticeable latency to a live voice turn.
Change
validate_cb: Callable[[T], bool] | NonetoConnectionPool. It is invoked before a pooled connection is handed out; connections that fail validation are removed/closed andget()keeps searching (or creates a fresh connection).validate_cb=lambda ws: not ws.closed, so a socket that was closed while idle is discarded before it reaches the caller instead of surfacing a send failure and retry.This follows the minimal approach suggested in the issue.
Tests
tests/test_connection_pool.py:test_get_discards_invalid_connection— a healthy connection is returned even when stale (closed) connections are present in the pool.test_get_creates_new_when_all_invalid— when every pooled connection fails validation,get()discards them and creates a fresh connection.test_get_reuses_when_no_validate_cb— behavior is unchanged when novalidate_cbis configured.ruff format --checkandruff checkpass on the changed files;mypyclean onconnection_pool.py.