feat(frontend): agent chat voice input — dictation + voice messages#5458
Draft
ardaerzin wants to merge 27 commits into
Draft
feat(frontend): agent chat voice input — dictation + voice messages#5458ardaerzin wants to merge 27 commits into
ardaerzin wants to merge 27 commits into
Conversation
Adds a mic button to the agent-chat composer that dictates speech to text via the browser Web Speech API. Words stream into the Lexical editor live as you speak (interim + finalized), and the result is fully editable before sending. Transcribes to plain text, so it is independent of any model/audio capability — the model receives words, not audio. The mic is hidden where SpeechRecognition is unavailable (e.g. Firefox), and shows a clear state while recording plus a mic-permission-denied message. New: hooks/useVoiceInput.ts (typed SpeechRecognition wrapper), components/VoiceInputButton.tsx (composer-prefix mic).
Adds a second voice mode alongside voice-to-text. 'Voice message' records mic audio via MediaRecorder and drops the clip into the composer attachment tray (addFiles), same path as a picked/dropped file; 'Voice to text' transcribes as before. A mode toggle (persisted) switches between them, and only browser-supported modes are offered. The recorded audio rides the normal attachment flow, so nothing here depends on the wire-contract or model work — it produces an audio File up to the send boundary. audio/* is now an accepted attachment type. New: hooks/useAudioRecorder.ts (MediaRecorder → File, elapsed timer, permission/error, cleanup).
Fleshes out the voice-message flow into a proper state machine instead of the tap/tap minimum:
- Recording takeover (RecordingBar) covers the composer while capturing: live timer, input-level meter (Web Audio), and distinct Discard (cancel) vs Stop & attach (keep) exits. Esc discards.
- Permission-pending ('Waiting for microphone…') and a persistent, dismissible denied/error strip above the composer — no more hover-only tooltip.
- Hard 5-min cap with a countdown near the limit; sub-0.7s / empty takes are discarded, not attached.
- Recorded clips play back in the tray (AudioChip: play/pause + elapsed) before sending.
The recorder is owned by AgentConversation so the takeover can cover the whole input; the mic just starts it. audio/* accepted; the clip still rides the normal attachment path.
Recording a voice message is the primary action; voice-to-text is the alternative. Also orders the mode menu primary-first. Falls back to dictation only where MediaRecorder is unavailable.
The waiting state read as an error and was mis-laid-out: - Actions now right-align (ml-auto). Previously the discard button sat jammed against the label, because the waiting state has no flex-1 meter to push it over. - Red border only once actually capturing. Red reads as 'live'; waiting on the mic is neutral, so it now uses the normal border with a softly pulsing dot instead of an inert grey one. - Copy is actionable rather than passive: 'Allow microphone access to start recording' (vs 'Waiting for microphone...'), and the buttons say 'Cancel' / 'Delete recording' / 'Attach to message'. - Overlay geometry: mb-3 moved from the input to the wrapper so the bar covers the composer box exactly (inset-0) — the input no longer peeks out below. - role=status + aria-live so the state change is announced.
…th motion Every voice surface popped in/out with no transition. Now they animate on the slice's existing idioms rather than bespoke ones: - Mic permission/error notice -> new MicPermissionNotice using RevealCollapse (the shared composer-chrome enter/leave, same as ConnectModelBanner and the HITL dock), with the message latched in a ref so the text survives the collapse instead of vanishing mid-animation. - Recording takeover -> AnimatePresence cross-fade + slight rise/fall on SESSION_SPRING, the same physics the session rail and tag bar use. - Within the bar, the waiting -> capturing swap is a wait-mode cross-fade (labels never overlap); the attach button springs in; the '30s left' warning fades; and the border, dot and timer colours ease via transition-colors instead of snapping.
…e it up Corner artefact on transition: the bar was rounded-xl over a rounded-lg composer, so the input's corners poked out from under it mid-cross-fade. The bar now mirrors the composer's box exactly — rounded-lg, --ag-composer-border, --ag-colorBgContainer, --ag-surface-chat-shadow — so the transition reads as the input changing state rather than a second card on top. Sizing: the content was set against 12px defaults and read as too small inside the taller editor. Label and timer to text-sm, status dot 10->12px, action icons 16->18px, level bars taller and wider, and roomier gap/padding.
…on, and drop the takeover flash Two problems behind the sluggish, flashy recording transitions. Perf: useAudioRecorder is owned by AgentConversation (so the takeover can cover the composer), but it held the input level in React state and wrote it every animation frame — re-rendering a ~2400-line component 60x a second. Level and elapsed now live in a ref that RecordingBar samples itself, so only that small component repaints. AgentConversation re-renders on status/error only. Flash: an instantly-resolved permission request (already granted, or already denied) faded the takeover in and straight back out before the denied banner expanded — three animations for one click. The takeover now appears immediately when recording, but only after a 350ms grace period while a request is still pending, so instant resolutions never render it.
The recorder was mixing an rAF meter loop, a 200ms polling interval, and a second rAF in the UI. Split on the property that actually matters — rAF pauses in a background tab, timers do not: - rAF drives everything painted (input level + the derived clock). Pausing while hidden is correct: nothing is on screen, and it stops burning battery on an invisible meter. - A single setTimeout enforces the max-duration cap, because that must still fire when the tab is backgrounded and rAF is parked. It replaces a 200ms poll: one deadline, not 1500 wakeups over a 5-minute take. - Elapsed is derived from a start timestamp instead of accumulated by a ticker, so it cannot drift and survives a paused loop. Also quantises the readout to what is visible — N discrete bars and a 1Hz clock — so the 60Hz loop only triggers a render when a bar lights or the second rolls over, rather than on every frame.
Cancel (and Esc) did nothing in the waiting state: it called stop() on recRef, but no MediaRecorder exists until getUserMedia resolves, so it was a no-op on null and the bar just sat there. Cancel now branches — stop the recorder when there is one, otherwise tear down and return to idle immediately. The browser's permission prompt can only be dismissed by the person, so the pending request is neutralised instead: if it later resolves, the stream's tracks are stopped at once rather than starting a recording that was backed out of, and if it rejects, no denied banner is shown for a request the person already abandoned.
A page cannot dismiss a permission prompt — getUserMedia takes no AbortSignal and there is no API to close it — so our waiting takeover offered a cancel that could not cancel, and covered the composer while the browser owned the interaction. The takeover is now capture-only. Awaiting permission is shown on the mic itself: it pulses, is disabled (a second press only queues another request), and its tooltip points at the browser's prompt. Backing out is the prompt's own job; if it later resolves the stream is still dropped rather than starting a recording nobody asked for. RecordingBar loses its requesting branch entirely. Also splits denied from dismissed: Chrome rejects identically for both, so the permission store is consulted and we no longer tell someone to change browser settings when they merely dismissed the prompt and could just try again.
The seven threshold bars were a level meter — they showed loudness, not the take. Replaced with a rolling waveform: each slice is the FFT magnitude of one moment (voice band only, the top of the spectrum is mostly hiss), appended on the right so the recording scrolls leftward as it is spoken, older slices receding. It shows the shape of what was actually said. Drawn on a canvas from the animation frame, which also removes it from React entirely: the previous meter had to quantise its level into discrete lit-bar counts purely to avoid repainting its parent at frame rate. Now the only React state left in the bar is the clock, at 1Hz. Splits the concerns too — the hook owns the audio graph and exposes the analyser; the view owns sampling and drawing.
A recording attaches like any other file, so the attachment guardrails could reject it after the fact — you record, hit attach, and the take is gone. - At the file limit the mic is now disabled for voice-message mode (as the paperclip already was), rather than letting someone record a take that can only be rejected. Dictation is unaffected: it writes into the editor, not the tray. - Pins the recorder to 64kbps. The default bitrate is unspecified, and a full-length take at a high one lands near the per-file size cap — the same silent-destruction path, just at the end of a five-minute recording. 64kbps is ample for voice (~2.4MB at the cap).
… tray A sent voice message rendered as an inert file card — you could record and send one but never hear it back. Audio parts now render an inline player in the transcript. Extracts the tray's bespoke player into a shared AudioPlayer used by both surfaces, so a voice message behaves the same before and after sending, and adds a progress bar. Also resolves the duration properly: a MediaRecorder webm reports an infinite duration until it has been seeked to the end, so the player nudges it there once and reads the real value back (guarded, so the seek never surfaces as a wild current-time reading). Previously every recording showed no duration at all.
…a take is in flight Two remaining voice gaps. Dictation errors (a denied mic, a recognition failure) only ever appeared in the mic's tooltip — the same hover-only trap the recording path already moved away from. They now feed the shared MicPermissionNotice, so both voice modes report failures the same visible way. Only the error is lifted; the transcript stays inside the button because it changes far too often to re-render the conversation with. A file dropped or pasted mid-recording could also take the last tray slot, so the take was rejected — and destroyed — on attach. Drops and pastes are ignored while capturing (the composer is covered by the recording bar anyway). Guarded at the entry points rather than in addFiles, because the recorder's own completion calls that directly and must always get through.
…light The bar was red four times over — border, dot, waveform, and the near-limit timer — so it read as an error state, and the one thing that should escalate had nothing left to escalate with. - Border moves to brand. An error border is the form-validation signal; nothing about recording is invalid. - Waveform moves to a brand tint: a single horizontal gradient across the strip, faint at the old end and near-full at the newest, so age reads as depth rather than needing a second colour. Replaces the per-bar globalAlpha ramp — one gradient built per resize, so it is also less work per frame. - Red now means exactly one thing: the record light (and the genuine near-limit warning).
…is visible Both modes drew the same microphone, so picking one from the menu changed nothing on screen — the only feedback was the menu closing, leaving you unsure whether the choice registered or what to do next. Each mode now carries its own icon (microphone for a voice message, text for dictation) and the button renders the active one, so switching visibly changes the control. The menu items carry the same icons, which is where the mapping gets taught. Tooltips are verbs too — 'Record a voice message' / 'Dictate into the message' — so the control says what pressing it will do rather than just naming the selected mode.
… attaching it Confirming a recording always parked it in the tray, so a voice message recorded into an empty composer needed a second, unexplained press to actually go. Confirm is now contextual. Recorded with nothing else composed, the take IS the message: the button becomes Send and the recording goes straight out. With text or other attachments staged, the person is mid-composition, so it still attaches and they send when ready — the icon and tooltip say which will happen before it does. Decided when recording starts rather than on confirm: the composer is covered by the recording bar and drops are blocked while capturing, so neither the text nor the tray can change in between.
…ing the document Dictation drove the composer with setMarkdown(base + transcript) on every interim result — a full document replace several times a second. That discarded the undo history, re-parsed markdown each tick, gave no way to distinguish settled words from provisional ones, and overwrote anything typed alongside it. Adds a dictation session to RichChatInput (plugins/dictation). It appends two text nodes once and only sets their text as speech arrives, leaving the rest of the document untouched: - committed words land as ordinary text, indistinguishable from typing; - the provisional tail is styled as unsettled (opacity + italic — theme-agnostic, no colour token) and loses that styling on settle, so nothing is dropped if the recogniser never finalises it; - updates are tagged history-merge, so a whole dictation is one undo entry rather than hundreds; - nodes are re-resolved per update, so editor garbage collection cannot orphan the session. A "dictating" prop locks the editor for the duration, so typing cannot interleave with the incoming transcript. useVoiceInput now reports committed and provisional text separately rather than flattening both into one string.
…e icons say "voice" Two things the recording bar got wrong. It grew its own send button — a paper-plane in a plain primary circle — while the composer's real one is an arrow-up in an accent circle with its own disabled treatment. Two similar-but-different send controls in the most-used flow in the product. The presentational half is now a shared ComposerSendButton in @agenta/ui, used by the editor's SendButton plugin and by the recording bar, so sending looks and behaves the same wherever it is triggered. Only the glyph changes when the take is attached to a message in progress rather than sent outright. The dictation mode also drew a text glyph, which says nothing about speech. Both modes now read as voice and differ by what comes out: a waveform for an audio clip, and the microphone every phone keyboard uses for dictation.
Interim speech was dimmed to 0.55 and italicised. Across a run that can be a whole sentence before the recogniser settles, the italic reads as emphasis and costs legibility — it was styling the majority of the message. A gentle dim (0.65, no italic) is enough: the words are visibly rewriting themselves as they settle, which carries most of the signal on its own.
Bold / Italic / Send / Newline stayed on screen during dictation, advertising shortcuts that cannot fire — editing is locked while speech comes in. They fade out the same way they already fade on blur, rather than unmounting: the hints are deliberately kept mounted so their space never reflows the toolbar, and dropping them would make the row jump at the start and end of every dictation.
The row mixed a two-line audio chip, 48px thumbnails and a dashed add-tile at three different heights, so it read as a ragged line rather than one band. Every tile is now the same height behind a shared chip shell, with one border treatment and one remove affordance across thumbnails, clips and file chips (on a thumbnail it sits over the image and appears on hover, so it stops covering the preview). Tiles also had no motion at all — they appeared and vanished instantly while the rest jumped to fill the gap. Each is now a layout-animated item inside AnimatePresence on the slice's shared spring: tiles scale in on add, scale out on remove, and the survivors slide into place. popLayout is deliberate — a removed tile leaves the flow immediately so the row closes the gap while it animates out, instead of waiting for it to finish. The add-tile and counter animate with them, and the rejection strip collapses instead of popping.
…rapping Five audio clips wrap onto five rows, growing the panel downward and pushing the composer up the screen. The tiles are now one horizontally scrolling band, mirroring the session tag bar in this slice: contained overscroll so it cannot chain to the page, motion-safe smooth scrolling, and no visible scrollbar under a 48px strip. Details that make it behave: - tiles are shrink-0, so a nowrap flex row cannot squash them to fit; - the scroller is min-w-0 flex-1, without which flex refuses to shrink below content width and the panel overflows instead of scrolling; - the counter sits outside the scroller, so it stays put rather than scrolling out of view; - adding a file scrolls the band to the end, so a tile added past the fold is actually visible.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (3)
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe chat composer now supports microphone recording, speech dictation, audio attachments, animated recording controls, and inline audio playback. Recorded clips can be sent immediately or added to attachments, while dictation updates the Lexical editor with committed and interim text. ChangesVoice composer and audio attachments
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant VoiceInputButton
participant useAudioRecorder
participant AgentConversation
User->>VoiceInputButton: Select audio recording
VoiceInputButton->>useAudioRecorder: Start microphone capture
useAudioRecorder->>AgentConversation: Report recording state
User->>AgentConversation: Stop or cancel recording
AgentConversation->>useAudioRecorder: Stop or cancel
useAudioRecorder->>AgentConversation: Return recorded File
AgentConversation->>AgentConversation: Submit file or add attachment
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Author
|
@coderabbitai review |
✅ Action performedReview finished.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (3)
web/oss/src/components/AgentChatSlice/components/AudioPlayer.tsx (1)
14-18: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueReduce this explanatory comment.
This describes component reuse rather than a surprising constraint; shorten it to one line or remove it.
As per coding guidelines, “Keep in-code comments to at most one short line; use longer comments only for genuinely surprising constraints such as bugs, races, or ordering requirements.”
Source: Coding guidelines
web/oss/src/components/AgentChatSlice/hooks/useAudioRecorder.ts (1)
117-133: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueAnalyser may read silence if the
AudioContextstarts suspended.
AudioContextcreated after an awaitedgetUserMediacan be in thesuspendedstate on some browsers, sogetByteFrequencyDatareturns zeros and the waveform stays flat. Since metering is best-effort this won't break capture, but aresume()makes the visualisation reliable.♻️ Optional resume
const ctx = new Ctor() audioCtxRef.current = ctx + if (ctx.state === "suspended") ctx.resume().catch(() => {}) const source = ctx.createMediaStreamSource(stream)web/oss/src/components/AgentChatSlice/components/ComposerAttachments.tsx (1)
105-118: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winObject URLs are revoked and recreated for every file on any list change. Adding or removing one attachment re-runs this effect for the whole list, handing
AudioPlayera freshsrcfor clips that didn't change — the<audio>element reloads and any in-progress playback resets (image previews also re-fetch/flicker). Consider reusing existing URLs byuidand only creating/revoking for the delta.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 74a5e8f2-a19f-464a-80d9-f97fbc3fa9c6
📒 Files selected for processing (16)
web/oss/src/components/AgentChatSlice/AgentConversation.tsxweb/oss/src/components/AgentChatSlice/assets/attachments.tsweb/oss/src/components/AgentChatSlice/components/AgentMessage.tsxweb/oss/src/components/AgentChatSlice/components/AudioPlayer.tsxweb/oss/src/components/AgentChatSlice/components/ComposerAttachments.tsxweb/oss/src/components/AgentChatSlice/components/MicPermissionNotice.tsxweb/oss/src/components/AgentChatSlice/components/RecordingBar.tsxweb/oss/src/components/AgentChatSlice/components/RecordingWaveform.tsxweb/oss/src/components/AgentChatSlice/components/VoiceInputButton.tsxweb/oss/src/components/AgentChatSlice/hooks/useAudioRecorder.tsweb/oss/src/components/AgentChatSlice/hooks/useVoiceInput.tsweb/packages/agenta-ui/src/RichChatInput/ComposerSendButton.tsxweb/packages/agenta-ui/src/RichChatInput/RichChatInput.tsxweb/packages/agenta-ui/src/RichChatInput/index.tsweb/packages/agenta-ui/src/RichChatInput/plugins/SendButton.tsxweb/packages/agenta-ui/src/RichChatInput/plugins/dictation.ts
Comment on lines
+34
to
+38
| className={ | ||
| disabled | ||
| ? "!border-[var(--ag-send-disabled-bg)] !bg-[var(--ag-send-disabled-bg)] !text-[var(--ag-send-disabled-fg)]" | ||
| : "!border-[var(--ag-surface-accent)] !bg-[var(--ag-surface-accent)] !text-[#191a0d] hover:!border-[#b8cb3f] hover:!bg-[#b8cb3f]" | ||
| } |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
Replace raw hex colors with theme tokens/variables.
#191a0d, #b8cb3f are hardcoded and won't adapt to light/dark. Route these through a supported var(--ag-color*)/--ag-* token (mirroring --ag-send-disabled-*), e.g. an accent-foreground and accent-hover variable, so both themes render correctly.
As per coding guidelines: "Consume theme colors through Ant Design semantic tokens, Tailwind color utilities, or supported var(--ag-color*) variables; do not use raw hex colors or --ag-c-* literals" and "Implement light and dark appearance and interaction states for every added or changed UI element, and verify both themes."
Source: Coding guidelines
A ref-tracked WebM duration probe left probingRef stuck true if the source changed mid-probe (ComposerAttachments recreates object URLs as attachments change), which gated off onTimeUpdate and froze the elapsed/progress readout. Reset it at the top of the [src] effect so each source starts a clean probe.
MediaRecorder fires `stop` after `error` (spec). teardown() in onerror zeroes startedAtRef, so the trailing onstop computed tooShort=false, flipped the status error→idle, and could emit a File from the partial chunks. Guard onstop with an erroredRef so an errored take stays errored and never emits.
During onboarding the composer commits the ephemeral via handleCreateAgent, but a voice MESSAGE routed through handleSubmit → submit, bypassing that commit. Gate audioSupported on !onboardingActive so the mic offers dictation only (which just fills the description text); voice-message returns once the agent exists.
Contributor
Author
|
@coderabbitai review |
✅ Action performedReview finished.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds voice to the agent-chat composer, in two modes:
Recording UX
Composer attachment tray
Evened out, motion-orchestrated, and horizontally scrollable instead of wrapping.
Notes
main. This is the lower half of a two-PR stack; attachments + drive uploads stack on top of this branch.