feat: add wallet-backed speech generation - #243
Conversation
📝 WalkthroughWalkthroughThe PR adds ChangesSpeech generation
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant CLI as /cr-speech
participant SpeechEndpoint as /v1/speech/generations
participant BlockRunProxy as /v1/t2a_v2
participant LocalAudioStorage
CLI->>SpeechEndpoint: Submit text, model, and format
SpeechEndpoint->>BlockRunProxy: Forward speech request through x402
BlockRunProxy-->>SpeechEndpoint: Return encoded audio
SpeechEndpoint->>LocalAudioStorage: Save decoded audio
LocalAudioStorage-->>SpeechEndpoint: Return localhost audio URL
SpeechEndpoint-->>CLI: Return generated audio URL
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/proxy.speech.test.ts`:
- Around line 10-69: Add Vitest integration coverage around startProxy that
mocks the paid speech upstream, sends a request through the proxy, verifies the
generated /audio/ URL and returned media response, and confirms client
disconnects abort the upstream request. Keep the existing normalizeSpeechRequest
and parseSpeechResponse unit tests, and use the speech endpoint’s existing
lifecycle and mocking conventions.
In `@src/proxy.ts`:
- Around line 3030-3034: Normalize and validate the selected format across the
audio flow: in src/proxy.ts lines 3030-3034, use the requested format when
upstream metadata is absent, reject incompatible upstream bytes, and add MIME
mappings for every retained format in the /audio/ handler; update src/proxy.ts
lines 127-127 to retain only end-to-end servable formats, preserve an absent
upstream format at lines 197-198 so the endpoint can apply the requested
fallback, and align the supported-format help in src/index.ts lines 2046-2046.
- Around line 2997-3023: Update the speech-generation handler around payFetch to
create an AbortController and abort it from res.on("close") when
!res.writableEnded. Pass the controller’s signal to payFetch, and in the
upstream error path return without writing an error response when the signal is
aborted; preserve existing error handling for other failures.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: e2010d42-761b-418f-aa43-df600c39469c
📒 Files selected for processing (3)
src/index.tssrc/proxy.speech.test.tssrc/proxy.ts
| describe("speech request normalization", () => { | ||
| it("defaults the model and audio fields", () => { | ||
| expect(normalizeSpeechRequest({ text: "Hello" })).toEqual({ | ||
| model: DEFAULT_SPEECH_MODEL, | ||
| text: "Hello", | ||
| stream: false, | ||
| output_format: "hex", | ||
| audio_setting: { format: "mp3" }, | ||
| }); | ||
| }); | ||
|
|
||
| it("accepts every supported speech model and request field", () => { | ||
| for (const model of SPEECH_MODELS) { | ||
| expect( | ||
| normalizeSpeechRequest({ | ||
| model, | ||
| text: "Hello", | ||
| language_boost: "English", | ||
| voice_setting: { voice_id: "female-shaonv" }, | ||
| pronunciation_dict: { tone: ["hello/(he lou)"] }, | ||
| audio_setting: { format: "wav", sample_rate: 32000 }, | ||
| voice_modify: { pitch: 1 }, | ||
| subtitle_enable: true, | ||
| }).model, | ||
| ).toBe(model); | ||
| } | ||
| }); | ||
|
|
||
| it("rejects missing text, unknown models, and unsupported audio formats", () => { | ||
| expect(() => normalizeSpeechRequest({ model: DEFAULT_SPEECH_MODEL })).toThrow("requires text"); | ||
| expect(() => normalizeSpeechRequest({ model: "unknown", text: "Hello" })).toThrow( | ||
| "Unsupported speech model", | ||
| ); | ||
| expect(() => | ||
| normalizeSpeechRequest({ text: "Hello", audio_setting: { format: "ogg" } }), | ||
| ).toThrow("Unsupported speech audio format"); | ||
| }); | ||
| }); | ||
|
|
||
| describe("speech response parsing", () => { | ||
| it("decodes hex audio and returns response metadata", () => { | ||
| expect( | ||
| parseSpeechResponse({ | ||
| data: { audio: "48656c6c6f", status: 2 }, | ||
| base_resp: { status_code: 0 }, | ||
| extra_info: { audio_format: "wav" }, | ||
| }), | ||
| ).toEqual({ audio: Buffer.from("Hello"), format: "wav", status: 2 }); | ||
| }); | ||
|
|
||
| it("decodes base64 audio and rejects failed or empty responses", () => { | ||
| expect(parseSpeechResponse({ data: { audio: "SGVsbG8=" } }).audio).toEqual( | ||
| Buffer.from("Hello"), | ||
| ); | ||
| expect(() => | ||
| parseSpeechResponse({ base_resp: { status_code: 1001, status_msg: "Invalid request" } }), | ||
| ).toThrow("Invalid request"); | ||
| expect(() => parseSpeechResponse({ data: {} })).toThrow("did not contain audio"); | ||
| }); | ||
| }); |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift
Add speech endpoint lifecycle and integration tests.
These tests cover only pure request and response helpers. Add Vitest coverage that sends a request through startProxy, mocks the paid upstream response, verifies the generated /audio/ URL and media response, and verifies that a client disconnect aborts the upstream request.
As per coding guidelines: Use Vitest tests to cover error and lifecycle resilience, end-to-end tool ID sanitization, and Docker installation, edge-case, and integration behavior where applicable. Based on learnings: Use Vitest tests to cover error and lifecycle resilience, end-to-end tool ID sanitization, and Docker installation, edge-case, and integration behavior where applicable.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/proxy.speech.test.ts` around lines 10 - 69, Add Vitest integration
coverage around startProxy that mocks the paid speech upstream, sends a request
through the proxy, verifies the generated /audio/ URL and returned media
response, and confirms client disconnects abort the upstream request. Keep the
existing normalizeSpeechRequest and parseSpeechResponse unit tests, and use the
speech endpoint’s existing lifecycle and mocking conventions.
Sources: Coding guidelines, Learnings
| if (req.url === "/v1/speech/generations" && req.method === "POST") { | ||
| const speechStartTime = Date.now(); | ||
| const chunks: Buffer[] = []; | ||
| for await (const chunk of req) { | ||
| chunks.push(Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk)); | ||
| } | ||
|
|
||
| let request: ReturnType<typeof normalizeSpeechRequest>; | ||
| try { | ||
| request = normalizeSpeechRequest(JSON.parse(Buffer.concat(chunks).toString())); | ||
| } catch (err) { | ||
| res.writeHead(400, { "Content-Type": "application/json" }); | ||
| res.end( | ||
| JSON.stringify({ | ||
| error: err instanceof Error ? err.message : "Invalid speech request", | ||
| }), | ||
| ); | ||
| return; | ||
| } | ||
|
|
||
| try { | ||
| const upstream = await payFetch(`${apiBase}/v1/t2a_v2`, { | ||
| method: "POST", | ||
| headers: { "content-type": "application/json", "user-agent": USER_AGENT }, | ||
| body: JSON.stringify(request), | ||
| }); | ||
| const text = await upstream.text(); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Abort the upstream request when the client disconnects.
This route continues payFetch() after the client closes the response. The upstream speech request can generate audio and settle a wallet payment after the caller cannot receive the result.
Create an AbortController, abort it from res.on("close") when !res.writableEnded, and pass its signal to payFetch. Return without writing an error when that signal is aborted.
Proposed fix
+ const clientAbort = new AbortController();
+ res.on("close", () => {
+ if (!res.writableEnded) clientAbort.abort();
+ });
+
try {
const upstream = await payFetch(`${apiBase}/v1/t2a_v2`, {
method: "POST",
headers: { "content-type": "application/json", "user-agent": USER_AGENT },
body: JSON.stringify(request),
+ signal: clientAbort.signal,
});🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/proxy.ts` around lines 2997 - 3023, Update the speech-generation handler
around payFetch to create an AbortController and abort it from res.on("close")
when !res.writableEnded. Pass the controller’s signal to payFetch, and in the
upstream error path return without writing an error response when the signal is
aborted; preserve existing error handling for other failures.
| const speech = parseSpeechResponse(JSON.parse(text)); | ||
| await mkdir(AUDIO_DIR, { recursive: true }); | ||
| const format = SPEECH_AUDIO_FORMATS.has(speech.format) ? speech.format : "mp3"; | ||
| const filename = `${Date.now()}-${Math.random().toString(36).slice(2, 10)}.${format}`; | ||
| await writeFile(join(AUDIO_DIR, filename), speech.audio); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Preserve and serve the selected audio format correctly.
If the upstream omits extra_info.audio_format, Lines 197-198 set the format to mp3 even when the request selected wav, flac, or pcm. Lines 3030-3034 then save bytes with an MP3 extension. The /audio/ handler also has no FLAC or PCM MIME mapping and falls back to audio/mpeg.
src/proxy.ts#L3030-L3034: use the normalized request format when upstream metadata is absent. Reject an incompatible upstream format instead of relabeling its bytes. Add matching MIME handling for every retained format.src/proxy.ts#L127-L127: retain only formats that the cache and local audio endpoint can serve correctly.src/proxy.ts#L197-L198: preserve an absent upstream format so the endpoint can apply the requested-format fallback.src/index.ts#L2046-L2046: keep the command help aligned with the end-to-end supported format set.
📍 Affects 2 files
src/proxy.ts#L3030-L3034(this comment)src/proxy.ts#L127-L127src/proxy.ts#L197-L198src/index.ts#L2046-L2046
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/proxy.ts` around lines 3030 - 3034, Normalize and validate the selected
format across the audio flow: in src/proxy.ts lines 3030-3034, use the requested
format when upstream metadata is absent, reject incompatible upstream bytes, and
add MIME mappings for every retained format in the /audio/ handler; update
src/proxy.ts lines 127-127 to retain only end-to-end servable formats, preserve
an absent upstream format at lines 197-198 so the endpoint can apply the
requested fallback, and align the supported-format help in src/index.ts lines
2046-2046.
VickyXAI
left a comment
VickyXAI
left a comment
There was a problem hiding this comment.
Thanks for the PR — the plumbing (payFetch flow, /audio/ caching, /cr- command naming, logUsage wiring) follows this repo's patterns correctly. Unfortunately the API surface itself doesn't exist on the gateway this proxy fronts, so the feature cannot work. Closing for the same reason as #227.
The endpoint is not BlockRun's. POST ${apiBase}/v1/t2a_v2 is MiniMax's native TTS path. BlockRun has no /v1/t2a_v2 route. The real TTS endpoint is POST /v1/audio/speech.
All 8 model ids are MiniMax vendor ids. speech-2.8-hd, speech-02-turbo, etc. are MiniMax T2A model names. BlockRun's speech catalog is exactly 5 models: elevenlabs/flash-v2.5, elevenlabs/turbo-v2.5, elevenlabs/multilingual-v2, elevenlabs/v3, bytedance/seed-audio-1.0. As written, the SPEECH_MODELS allowlist only admits models the gateway does not serve — every request would fail even if the endpoint existed.
Request/response schema is MiniMax's too. audio_setting/voice_setting/pronunciation_dict, the base_resp.status_code envelope, hex-encoded audio, and extra_info.audio_format are all MiniMax API shapes. BlockRun's speech route accepts { model, input text, voice (alias or ElevenLabs voice_id), response_format } and supports formats mp3 | opus | pcm | wav (no flac). There's also a free discovery endpoint, GET /v1/audio/voices, for picking a voice.
The unit tests pass because they validate the wrong contract against itself — none of them touch the real gateway surface.
If you'd like to resubmit, the source of truth is the gateway's OpenAPI/route definitions (the /v1/audio/speech route publishes its schema), not the upstream vendor's docs. A resubmission should also abort the upstream request when the client disconnects (CodeRabbit flagged this) — otherwise a wallet payment settles for audio nobody receives.
|
Closing — the API surface (endpoint, model ids, request/response schema) is MiniMax's vendor API, none of which exists on the BlockRun gateway. Details in the review above. Happy to look at a resubmission built against |
2 participants
Reason: Add wallet-backed text-to-speech generation to the existing audio pipeline.
/cr-speechcommand and local speech-generation routeChecks:
node node_modules/vitest/vitest.mjs run src/proxy.speech.test.tsnode node_modules/typescript/bin/tsc --noEmitnode node_modules/eslint/bin/eslint.js src/index.ts src/proxy.ts src/proxy.speech.test.tsnode node_modules/prettier/bin/prettier.cjs --check src/index.ts src/proxy.ts src/proxy.speech.test.tsgit diff --check -- src/index.ts src/proxy.ts src/proxy.speech.test.tsSummary by CodeRabbit
New Features
/cr-speechcommand for generating speech audio.Tests