Add support for Deepgram's Flux TTS API (v2/speak) integration#6511
Add support for Deepgram's Flux TTS API (v2/speak) integration#6511dg-edcharbeneau wants to merge 3 commits into
Conversation
…m.com/docs/flux-tts/overview) API (the `/v2/speak` endpoint) to the Deepgram plugin, alongside the existing Aura TTS (`/v1/speak`). This is purely additive — **no breaking changes**. The implementation mirrors the non-breaking extension pattern the plugin already uses for Flux STT (`stt_v2.py` / `STTv2`): a new parallel module and class rather than modifying the existing `TTS`. - **`deepgram.TTSv2`** — a new class targeting `wss://api.deepgram.com/v2/speak` (streaming) and `POST https://api.deepgram.com/v2/speak` (batch). Named `TTSv2` for consistency with the existing `STTv2`. - **`deepgram.FluxTTSModels`** — a model-name literal (seeded with `flux-alexis-en`), typed as `FluxTTSModels | str` so any `flux-{voice}-{language}` string works without a code change, matching how v1 handles `TTSModels | str`. - New `tts_v2.py` with `TTSv2`, `ChunkedStreamv2` (batch/HTTP), and `SynthesizeStreamv2` (streaming/WebSocket). It reuses the existing infrastructure (`ConnectionPool`, `_to_deepgram_url`, `AudioEmitter`, word tokenizer) — no new dependencies. - The main protocol difference from v1 is the Flux server-message set: the streaming receive loop ends each turn's segment on **`SpeechMetadata`** (the documented "all audio sent" marker) and treats `Connected` / `SpeechStarted` / `Flushed` / `SessionMetadata` as lifecycle/telemetry messages. - `tts.py` (v1 `TTS`) is untouched; `__init__.py` exports the new symbols additively. - `ruff` and `mypy` (strict) pass on the new module. - Verified end-to-end against the live `/v2/speak` API: - **Batch** (`TTSv2().synthesize(...)`): ~104 KB of PCM audio returned. - **Streaming** (`TTSv2().stream(...)`): ~84 KB of audio across one cleanly-closed segment, confirming end-of-turn detection on `SpeechMetadata`. - Confirmed existing `deepgram.TTS()` continues to resolve and stream unchanged. ```python from livekit.plugins import deepgram tts = deepgram.TTSv2(model="flux-alexis-en")
Follow-up to the Flux TTS (v2/speak) plugin, resolving review findings. No breaking changes. - mip_opt_out: fix docstring that described the flag backwards — it's an opt-out, and default False means requests may be used to improve models (the code already sent the value correctly). - encoding/bit_rate: document that only linear16 works end-to-end on the streaming path; compressed encodings (mp3/opus/flac/aac) and bit_rate apply to the batch synthesize() path only. Validate encoding on stream() and fail fast with a clear error; drop the no-op bit_rate from the WS connect params. - batch mime type: derive the mime type from encoding via _encoding_to_mimetype (sibling pattern) instead of hardcoding audio/pcm, so compressed batch output is decoded correctly. Scoped to the encodings the framework can decode; mulaw/alaw stay blocked. - Warning log now includes the Flux error code (e.g. NO_ACTIVE_SPEECH). - Comment the intentional no-op absorption of the trailing Flushed frame on pooled connection reuse. - Add a hermetic, mocked-WebSocket test suite (no credentials required) and register TTSv2 in the cross-provider tts test list.
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| if mtype == "SpeechMetadata": | ||
| # Authoritative end-of-turn marker: all audio for the turn has | ||
| # been sent between SpeechStarted and this message. The server | ||
| # emits a trailing Flushed frame after this; on a pooled reuse it | ||
| # is harmlessly absorbed as a no-op at the top of the next turn. | ||
| output_emitter.end_segment() | ||
| break | ||
| elif mtype in ("Connected", "SpeechStarted", "Flushed", "SessionMetadata"): | ||
| # lifecycle / telemetry messages, no audio action needed | ||
| pass |
There was a problem hiding this comment.
🟡 A spoken segment that produces no audio stalls until it times out and errors
A synthesis segment that yields no audio waits for an end-of-turn marker (SpeechMetadata at livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/tts_v2.py:418) that never arrives, so instead of finishing quietly it keeps listening until the connection times out.
Impact: A turn containing only whitespace/punctuation (or otherwise producing no speech) hangs for the full connection timeout and then fails the synthesis with a timeout error instead of ending cleanly.
Divergence from v1 end-of-turn detection
In the v1 receive loop (tts.py:378-380) the segment ends on the Flushed message, which the server always sends in response to the Flush that send_task emits after each segment. This means even a segment that generates no audio terminates cleanly.
In v2, Flushed is treated as a no-op (tts_v2.py:425-427) and the segment only ends on SpeechMetadata (tts_v2.py:418-424). SpeechMetadata is documented as the marker that all audio for a turn has been sent; the tested "No active speech detected" warning path (tests/test_plugin_deepgram_tts.py:196-216) suggests the server can respond to a flush with a Warning (+Flushed) and no SpeechMetadata. In that case recv_task never breaks and blocks on ws.receive(timeout=self._conn_options.timeout) (tts_v2.py:399) until it raises asyncio.TimeoutError, which propagates as APITimeoutError.
Segments with no words can occur: _tokenize_input creates a word_stream as soon as any string is pushed (tts_v2.py:344-347), so whitespace-only input still produces a segment whose tokenizer may yield no words, sending only a Flush.
Prompt for agents
The v2 streaming receive loop in SynthesizeStreamv2._run_ws only ends a segment on the SpeechMetadata message and treats Flushed as a no-op. Unlike v1 (tts.py), which ends on Flushed, this means a segment that generates no audio (e.g. whitespace/punctuation-only input, or a flush that triggers a 'No active speech detected' warning) will never receive SpeechMetadata and recv_task will block on ws.receive until conn_options.timeout elapses, raising APITimeoutError and failing the turn. Consider also ending the segment (or otherwise breaking the loop cleanly) when a Flushed is observed after the flush has been sent and no SpeechMetadata is expected, or when a Warning indicates no speech will be produced, so empty/no-audio segments terminate cleanly like v1 does. Be careful to preserve the intended 'trailing Flushed absorbed as no-op on pooled reuse' behavior for the normal case where SpeechMetadata does arrive.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Summary
Adds support for Deepgram's new Flux TTS API (the
/v2/speakendpoint) to the Deepgram plugin, alongside the existing Aura TTS (/v1/speak). This is purely additive — no breaking changes.The implementation mirrors the non-breaking extension pattern the plugin already uses for Flux STT (
stt_v2.py/STTv2): a new parallel module and class rather than modifying the existingTTS.What's new
deepgram.TTSv2— a new class targetingwss://api.deepgram.com/v2/speak(streaming) andPOST https://api.deepgram.com/v2/speak(batch). NamedTTSv2for consistency with the existingSTTv2.deepgram.FluxTTSModels— a model-name literal (seeded withflux-alexis-en), typed asFluxTTSModels | strso anyflux-{voice}-{language}string works without a code change, matching how v1 handlesTTSModels | str.Implementation notes
tts_v2.pywithTTSv2,ChunkedStreamv2(batch/HTTP), andSynthesizeStreamv2(streaming/WebSocket). It reuses the existing infrastructure (ConnectionPool,_to_deepgram_url,AudioEmitter, word tokenizer) — no new dependencies.SpeechMetadata(the documented "all audio sent" marker) and treatsConnected/SpeechStarted/Flushed/SessionMetadataas lifecycle/telemetry messages.tts.py(v1TTS) is untouched;__init__.pyexports the new symbols additively.Testing
ruffandmypy(strict) pass on the new module./v2/speakAPI:TTSv2().synthesize(...)): ~104 KB of PCM audio returned.TTSv2().stream(...)): ~84 KB of audio across one cleanly-closed segment, confirming end-of-turn detection onSpeechMetadata.deepgram.TTS()continues to resolve and stream unchanged.Usage