Chat input dictation (speech-to-text)#326044
Merged
Merged
Conversation
Adds a microphone button to the chat input toolbar that records audio
and transcribes it into the input using an Azure OpenAI transcription
deployment (e.g. gpt-4o-mini-transcribe). Client-only: captures mic
audio with MediaRecorder and POSTs it directly to the
audio/transcriptions endpoint; no backend required.
- IChatSpeechToTextService: getUserMedia + MediaRecorder capture,
webm upload to {endpoint}/openai/deployments/{deployment}/audio/transcriptions
- Toggle action in MenuId.ChatExecute; inserts transcript at the cursor
- chat.speechToText.azure.{endpoint,deployment,apiKey,apiVersion} settings;
button appears only once endpoint + apiKey are configured
- chatSpeechToTextRecording context key drives the record/stop icon
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
Ctrl/Cmd+Alt+V while focused in the chat input records for as long as the key is held (via keybinding hold mode) and transcribes on release. Releases under 500ms are discarded as accidental taps. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
Rewire the client from calling Azure OpenAI directly to POSTing audio to a transcription backend (chat.speechToText.serverUrl / product.chatSpeechToTextUrl) with the user's GitHub bearer token, matching the voiceClientService pattern. Add a Python FastAPI reference backend under chat-stt-backend/ that validates the GitHub token and forwards audio to Azure OpenAI, hiding the Azure key. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
Resolve the auth provider id from IDefaultAccountService (correct for GitHub Enterprise) instead of hardcoding 'github', and move the product default URL under IDefaultChatAgent.speechToTextUrl to group it with the other Copilot service URLs (entitlementUrl, tokenEntitlementUrl, ...), matching how chatEntitlementService and defaultAccount reach the backend with the user's GitHub bearer token. Raw fetch + FormData is retained since the audio upload is multipart (IRequestService is JSON-only). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
The VS Code renderer issues a cross-origin request with an Authorization header, triggering a CORS preflight the backend previously rejected (OPTIONS -> 405, surfacing as 'Failed to fetch'). Add CORSMiddleware and a .gitignore so local .env/venv are never committed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
- Add a speech-to-text mic button to the agents/sessions welcome composer, placed before the voice controls; inserts the transcript into the composer. - Strip filler words (um/uh/er/...) and collapse whitespace from transcripts. - Order the chat-input dictation button before the voice-mode button. - Untrack the committed __pycache__ .pyc build artifact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
Replace the record-then-upload flow with a realtime bridge so the transcript appears while the user speaks instead of only after they stop. - Backend: add a /transcribe/stream WebSocket endpoint that proxies the client to the Azure OpenAI realtime transcription API (server_vad segmentation); relays audio appends up and delta/segment transcripts down. - Client: stream PCM16 (16kHz mono) over a WebSocket; expose onDidUpdateTranscript with the growing cumulative transcript. - Add dictationSession helper that renders the live transcript into the target editor (replacing its own region on each update); shared by the toggle, hold-to-talk, and sessions composer entry points. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
- Move the Cmd+Shift+Space chord from hold-to-talk to the toggle action (press to start/stop, matching Voice Mode); gate it to only claim the chord when Voice Mode is not enabled so the two bindings are unambiguous. - Resume the AudioContext after the async mic/socket handshake so onaudioprocess actually fires and audio streams to the backend. - Backend: only commit the input buffer when audio is pending (server_vad may have already committed), avoiding input_audio_buffer_commit_empty; add light diagnostic logging. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
Contributor
|
Base:
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds live speech-to-text dictation to chat inputs, including microphone capture, streaming transcription, editor insertion, UI actions, and a reference backend.
Changes:
- Adds the dictation service, session lifecycle, commands, and configuration.
- Integrates dictation into workbench and Agents window chat inputs.
- Adds a local FastAPI/Azure reference backend.
Show a summary per file
| File | Description |
|---|---|
src/vs/workbench/contrib/chat/common/actions/chatContextKeys.ts |
Adds recording state context. |
src/vs/workbench/contrib/chat/browser/speechToText/dictationSession.ts |
Inserts live transcripts. |
src/vs/workbench/contrib/chat/browser/speechToText/chatSpeechToTextService.ts |
Captures and streams audio. |
src/vs/workbench/contrib/chat/browser/chat.shared.contribution.ts |
Registers configuration and service. |
src/vs/workbench/contrib/chat/browser/actions/chatSpeechToTextActions.ts |
Adds dictation actions and shortcuts. |
src/vs/sessions/contrib/chat/browser/newChatInput.ts |
Adds Agents composer control. |
src/vs/sessions/contrib/chat/browser/media/chatInput.css |
Styles the microphone control. |
src/vs/base/common/product.ts |
Adds hosted backend product URL. |
chat-stt-backend/requirements.txt |
Declares backend dependencies. |
chat-stt-backend/README.md |
Documents the backend. |
chat-stt-backend/main.py |
Implements the reference backend. |
chat-stt-backend/.gitignore |
Excludes local Python artifacts. |
chat-stt-backend/.env.example |
Documents backend environment values. |
Review details
- Files reviewed: 13/13 changed files
- Comments generated: 24
- Review effort level: Medium
- Add audio-reactive listening glow to chat inputs while dictating, across the main window, sessions ChatView, and new-session composer. Uses dictation-specific classes so the Voice Mode mic icon is not highlighted. - Expose an AnalyserNode from the STT service to drive the glow. - Prefetch the auth token so the auth frame is sent synchronously on socket open, fixing an intermittent "Expected auth frame" rejection. - Backend: ignore benign input_audio_buffer_commit_empty (server VAD already committed) instead of surfacing it as a failed transcription. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
…t endpoint - Backend now signals `done` and closes promptly after the final transcript instead of blocking on the always-open Azure realtime session, so the input no longer sits in the transcribing state (stop icon) for ~8s after stop. - Client resolves the stop wait on `done`; timeout reduced to a 5s safety net. - Introduce a bound `chatSpeechToTextConfigured` context key set from the `chat.speechToText.serverUrl` setting OR the product's `speechToTextUrl`, so the dictation button/keybinding appear in prod where the URL ships in product.json. Service is eager so gating is correct at startup. - Expose `isConfigured`; composer gates on it. Keybinding switched to Cmd+I in the chat input, outranking the legacy Start Voice Chat binding. - Clarify the setting as a developer override for the product default. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
Emit a chatSpeechToText.session telemetry event (outcome, duration, segments, transcript length, error code) on completion/cancel/error, and register the chat.speechToText.enabled kill-switch that gates the mic button alongside a configured backend. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
Add the transformers.js dependency (with onnxruntime-node's ABI-stable napi-v6 prebuilt binaries) to power local, downloaded-model Whisper transcription for chat dictation — no backend required, matching the GitHub app's on-device dictation. transformers.js hard-imports the native `sharp` image library, which is unused by the audio pipeline and fails to build against Electron; override it with a pure-JS stub so no libvips native binary is needed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
Remove the audio-reactive "listening" glow shown while dictating; it is unnecessary. Deletes dictationGlow.ts and its call sites, and drops the now-unused analyser tap and service injections. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
Repoint ChatSpeechToTextService from the external WebSocket backend to the new on-device ILocalTranscriptionService (transformers.js + onnxruntime-node in a utility process) when supported, keeping the cloud WebSocket as a web fallback. Adds the platform IPC contract, node service, utility entry, and renderer/web client singletons. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
Unpack onnxruntime-node/bin from the node_modules ASAR (its N-API addon dlopen's sibling shared libraries by on-disk path) and strip non-target platform/arch binaries so each build only ships the one it needs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
Surface ILocalTranscriptionService.onDidChangeModelStatus as a background progress notification while the on-device Whisper model downloads/loads on first use, and add the chat.speechToText.model setting (tiny/base/small) to choose the model. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
The Agents/Sessions window is a separate workbench instance with its own DI container; it registered services individually and lacked ILocalTranscriptionService, so the eager ChatSpeechToTextService failed to construct there and the dictation mic button never appeared. Register the electron-browser client (and web null fallback) in the sessions mains. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
…Escape + a11y - Force-instantiate the eager ChatSpeechToTextService at startup via a workbench contribution so it publishes the `chatSpeechToTextConfigured` context key that gates the mic button (registered singletons are created lazily, so the key was never set and the button never appeared). - Remove the legacy WebSocket/cloud transport entirely (serverUrl setting, product speechToTextUrl endpoint, auth-token fetching, base64 streaming) in favor of the on-device model path only. - Escape cancels an in-progress dictation. - Play voiceRecordingStarted/Stopped accessibility signals on start/stop. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
…tt-azure # Conflicts: # build/gulpfile.vscode.ts
…n cancel Show a spinning indicator (in both the standard chat input and the sessions composer) in place of the mic button while the on-device speech-to-text model is downloading or loading, driven by a new chatSpeechToTextPreparing context key and onDidChangePreparingModel event. Also flip the voiceRecordingStopped signal sound default to on so cancelling dictation is audible. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
- Drop the model-download notification; the toolbar spinner is the only progress affordance (removes the double-spinner). Default the chat.speechToText.enabled setting to off for now. - Fix voiceRecordingStopped signal default (was inherited 'auto' → never audible without a screen reader); now 'on' like voiceRecordingStarted. - Escape now reverts the live transcript so no dictated text is left behind. - Cancel dictation when the target editor is disposed; clear the active session when the service ends it (e.g. model load error). - Surface model-load and audio-stream (pushAudio) failures instead of silently returning an empty transcript; guard capture-setup errors. - Node service: await model preparation on stop() so first-use downloads aren't dropped; drop stale inference results across sessions (generation token); bound interim inference to avoid unbounded re-transcription. - Gate isSupported by packaged OS/arch (onnxruntime-node targets) so the mic isn't shown where inference can't run. - Sessions composer mic: constructor-inject the service, add Enter/Space and touch (Gesture) activation, and route Cmd/Ctrl+I to dictation. - Add chat accessibility-help entries for Dictate and Cancel Dictation. - Use the base codicon size token for the composer mic; refresh stale cloud-fallback docs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 02c7340-e851-429b-a94b-e5bae931ca4b
alexr00
reviewed
Jul 16, 2026
alexr00
left a comment
alexr00
left a comment
Member
There was a problem hiding this comment.
Just want to check: is the new .js file actually needed?
Collaborator
Author
|
@alexr00 Yes, it's needed. sharp is a hard dependency of @huggingface/transformers and gets imported eagerly (it's on the ASR path via utils/image.js ), even though we only use the speech pipeline. It needs the native libvips binary, which fails to build in CI/Electron. Transformers is loaded at runtime via a () in the utility process, and the bundled dist does a top-level require("sharp"), so a bundler external can't intercept it. So I point sharp at this tiny no-op stub via an npm override, which throws a clear error if any image API is ever called. |
alexdima
approved these changes
Jul 16, 2026
This was referenced Jul 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #317172
Fixes #315508
demo.mov
Chat input dictation (speech-to-text)
Adds a microphone button to the chat input that dictates speech into the prompt, with the transcript rendered live as you speak.
Approach: on-device, mirroring the GitHub Copilot app
This mirrors what the GitHub Copilot desktop app does for its dictation feature: speech is captured and transcribed fully on-device, with local transcription-model management — no backend, no API keys, no audio ever leaving the machine. (Cloud is reserved for conversational Voice mode, which is a separate feature.)
@huggingface/transformers(transformers.js) withonnxruntime-node, so the renderer stays responsive.User experience
Cmd/Ctrl+Iwhile the chat input is focused.Settings
chat.speechToText.enabled— kill-switch for the feature (defaulttrue). The mic button appears when enabled on a platform with on-device support.chat.speechToText.model— Whisper model size (tiny/base/small); larger is more accurate but slower and a larger download. Defaults tobase.Packaging
onnxruntime-node/binis unpacked fromnode_modules.asar(its N-API addondlopens sibling native libraries).Platform support
On-device inference requires a native onnxruntime binary, so dictation is available on
darwin/arm64,linux/{x64,arm64}, andwin32/{x64,arm64}. Where on-device inference is unavailable (e.g. web, or arches without a prebuilt binary), the mic button is hidden.Notes
chat.speechToText.serverUrlsetting / product endpoint) has been fully removed in favor of the on-device path.