Windows dev - #13
Conversation
…ut fallback PTT Lifecycle Rewrite: - Merged stop+commit into atomic stopAndCommit() to prevent race conditions - Added isCommittingRef to block new starts during commit wait - All stop paths (Hotkey, MicButton, SpaceBar, Widget, Tray) now commit text - Settings save uses commit=false to avoid unwanted typing Real-Time Typing: - Text committed on raw ASR events during recording (immediate feedback) - Text committed on processed LLM events (final version) - User sees text appear as they speak, not just on release Typing Compatibility: - Added SendInput with KEYEVENTF_UNICODE as fallback for clipboard+Ctrl+V - Works in terminals, Electron apps, and UWP apps that ignore Ctrl+V - Character-by-character Unicode typing with surrogate pair support UI Fixes: - Fixed duplicate React keys (live-1 collision) with session-scoped IDs - Lines cleared on each new PTT session start - Added audio feedback beeps on PTT press (880Hz) and release (440Hz) Build Fixes: - Fixed pre-existing TS errors (unlistenShortcut, loadHistory, totalWords) - Created tsconfig.app.json to exclude test files from build - Fixed pnpm build to only check app code Hotkey: - Default: Ctrl+Shift+Space (valid Tauri format) - Added Alt+K option to Settings - Hotkey re-registration on change without restart
📝 WalkthroughWalkthroughThe PR adds a frontend PTT state machine, synchronized transcription completion, native text insertion, and an animated overlay. It also updates sidecar parsing, VAD orchestration, model cleanup, hotkey handling, build configuration, and repository settings. ChangesPTT Platform and Overlay Flow
Estimated code review effort: 5 (Critical) | ~120 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (1 warning, 1 inconclusive)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| # --- Wait for pending transcription threads before going idle --- | ||
| # This ensures pttTextRef has the latest transcription when the | ||
| # frontend reads it after receiving state:idle. | ||
| if _active_xcribe_threads: |
There was a problem hiding this comment.
🟡 Medium stt/orchestrator.py:788
run() joins each worker thread with a per-thread timeout derived from a single 15s deadline, then unconditionally calls _active_xcribe_threads.clear() regardless of whether any thread is still alive. When a _transcribe_and_print worker exceeds the deadline, it is dropped from the tracking list while still running, so its later raw/processed events can fire after state: idle is emitted or during a new recording session. Consider keeping unfinished threads in the list (e.g., only removing threads where t.is_alive() is False) so they remain tracked across the next PTT cycle.
🤖 Copy this AI Prompt to have your agent fix this:
In file @stt/orchestrator.py around line 788:
`run()` joins each worker thread with a per-thread timeout derived from a single 15s deadline, then unconditionally calls `_active_xcribe_threads.clear()` regardless of whether any thread is still alive. When a `_transcribe_and_print` worker exceeds the deadline, it is dropped from the tracking list while still running, so its later `raw`/`processed` events can fire after `state: idle` is emitted or during a new recording session. Consider keeping unfinished threads in the list (e.g., only removing threads where `t.is_alive()` is `False`) so they remain tracked across the next PTT cycle.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@stt-ui/src-tauri/src/lib.rs`:
- Around line 1063-1103: The Windows Unicode input path in send_text_unicode is
ignoring SendInput failures, which can make type_text report success even when
nothing was injected. Update send_text_unicode to return a status/result from
each SendInput call, including the surrogate-pair branch, and have type_text
check and propagate that failure instead of always returning Ok(true). Use the
existing send_text_unicode and type_text symbols to wire the error/status back
to the caller.
- Around line 1219-1226: The paste flow currently returns success immediately
after set_clipboard and send_ctrl_v, even though Ctrl+V is fire-and-forget and
may be ignored by some targets. Update the clipboard paste branch to verify
delivery before returning Ok(true), or route known incompatible targets directly
to send_text_unicode instead of exiting early. Use the existing
win32::set_clipboard, win32::send_ctrl_v, and win32::send_text_unicode paths to
keep the behavior aligned with the focused app’s capabilities.
In `@stt-ui/src/api-tauri.ts`:
- Around line 43-53: The shared lineBuffer in api-tauri.ts is mixing stdout and
stderr chunks, which can corrupt partial JSON events across streams. Update the
streaming parser around handleLine and the chunk processing to maintain separate
buffers per source (for example, keyed by source or separate stdout/stderr
state) so incomplete data is only joined with future chunks from the same
stream. Keep notifyError and the existing listener flow unchanged, but make sure
each stream’s framing is preserved before JSON parsing.
- Around line 109-111: The write failure in the command path is only being
logged inside the child.write catch, so callers like the
start_recording/stop_recording flow still assume delivery succeeded. Update the
command-sending logic in api-tauri.ts to surface this failure to listeners,
using the same notification mechanism the app relies on for recording state
changes, so the UI can exit the recording/committing path when a write to child
fails. Keep the fix centered around the child.write call and the surrounding
command dispatch helper so all command types propagate the error consistently.
In `@stt-ui/src/App.tsx`:
- Around line 1041-1050: The real-time PTT commit logic is sending the full
transcript snapshot on every ASR event, which causes duplicated text when
`type_text` is called again during commit. Update the handling in App.tsx around
the real-time commit path and the stop/processed commit paths so `type_text`
only receives newly finalized text (not the entire `event.text` snapshot) and
does not re-send text already inserted into the focused window. Use the existing
`pttTextRef`, `connectedRef`, `pttHwndRef`, and `stopAndCommit` flow to track
what has already been typed and only append the delta once.
- Around line 1166-1167: The stop handler wiring in App.tsx is stale after
renaming the implementation to stopAndCommit, so FeedView is still receiving the
wrong stop reference and may fall back to the global stop. Update the render
path that passes props into FeedView so its stop prop uses the same handler
stored in stopRef and points to stopAndCommit, keeping the PTT mic button
connected to the correct stop flow.
- Around line 1035-1040: The session-prefixed line ID used when inserting rows
in App.tsx is not being used consistently by later updates. Update the
processed, llm_partial, and error handlers to compute the same session-scoped ID
helper used for the live row before matching against line.id, so they continue
to find the existing entry instead of comparing against the bare utterance_id.
- Around line 837-840: The api.spawn() completion handler in App.tsx can still
write a stale api into runtimeRef.current after mode or settingsVersion changes
and cleanup has already invalidated that sidecar. Update the spawn flow to guard
the .then() callback against stale resolution, using the existing
runtimeRef/current lifecycle in App or the spawn/start effect, so only the
latest active api instance is stored and older resolved spawns are ignored.
In `@stt/orchestrator.py`:
- Around line 785-795: The idle transition in orchestrator.py still treats
pending transcription threads as finished even when a join times out, which can
let late output arrive after state:idle. Update the pending-thread handling
around _active_xcribe_threads so that after the join deadline you check each
thread with t.is_alive() before clearing the list, and if any are still running
either suppress their late raw/processed emissions or emit a timeout/error
instead of proceeding to idle. Use the existing _active_xcribe_threads wait
block and the idle/state publish path to ensure idle is only sent once all
worker output is truly complete.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: f20762e9-72fe-4430-95fc-4f6b41f4184c
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (12)
.gitattributesstt-ui/package.jsonstt-ui/src-tauri/src/lib.rsstt-ui/src/App.tsxstt-ui/src/api-tauri.tsstt-ui/src/components/HistoryPage.tsxstt-ui/src/components/InsightsPage.tsxstt-ui/src/components/SettingsPanel.tsxstt-ui/src/lib/audio.tsstt-ui/tsconfig.app.jsonstt/orchestrator.pytests/test_sidecar_e2e.py
💤 Files with no reviewable changes (1)
- stt-ui/src/components/InsightsPage.tsx
| pub fn send_text_unicode(text: &str) { | ||
| for ch in text.chars() { | ||
| let code = ch as u32; | ||
| let mut inputs = [ | ||
| Input { | ||
| r#type: INPUT_KEYBOARD, | ||
| ki: KeyBDInput { | ||
| wVk: 0, | ||
| wScan: code as u16, | ||
| dwFlags: KEYEVENTF_UNICODE, | ||
| time: 0, | ||
| dwExtraInfo: 0, | ||
| }, | ||
| _pad: [0; 8], | ||
| }, | ||
| Input { | ||
| r#type: INPUT_KEYBOARD, | ||
| ki: KeyBDInput { | ||
| wVk: 0, | ||
| wScan: code as u16, | ||
| dwFlags: KEYEVENTF_UNICODE | KEYEVENTF_KEYUP, | ||
| time: 0, | ||
| dwExtraInfo: 0, | ||
| }, | ||
| _pad: [0; 8], | ||
| }, | ||
| ]; | ||
| // Handle supplementary plane characters (surrogate pairs) | ||
| if code > 0xFFFF { | ||
| // Send as two UTF-16 surrogates | ||
| let hi = (0xD800 + ((code - 0x10000) >> 10)) as u16; | ||
| let lo = (0xDC00 + ((code - 0x10000) & 0x3FF)) as u16; | ||
| inputs[0].ki.wScan = hi; | ||
| inputs[1].ki.wScan = hi; | ||
| unsafe { SendInput(2, inputs.as_ptr(), std::mem::size_of::<Input>() as i32); } | ||
| inputs[0].ki.wScan = lo; | ||
| inputs[1].ki.wScan = lo; | ||
| unsafe { SendInput(2, inputs.as_ptr(), std::mem::size_of::<Input>() as i32); } | ||
| } else { | ||
| unsafe { SendInput(2, inputs.as_ptr(), std::mem::size_of::<Input>() as i32); } | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Win32 SendInput return value number of INPUT events successfully inserted zero failure UIPI
💡 Result:
The Win32 SendInput function returns the number of events that were successfully inserted into the keyboard or mouse input stream [1][2]. If the function returns zero, it indicates that the input was blocked by another thread [1][3]. When SendInput is blocked by User Interface Privilege Isolation (UIPI), the behavior is distinct in the following ways: 1. Return Value: If UIPI blocks the input, SendInput may return zero [4][5]. However, it is important to note that the return value is not a dedicated error code and does not always explicitly identify UIPI as the cause of failure [3][4]. 2. GetLastError: Calling GetLastError after a failure typically returns 5 (ERROR_ACCESS_DENIED) [4], but neither the return value nor GetLastError is guaranteed to indicate that UIPI blocking was the specific cause of the failure [3][4]. Because UIPI restricts input based on integrity levels—where an application can generally only send input to processes of equal or lower integrity—the discrepancy between the number of requested events and the successful return value is often used by developers as an indirect indicator that UIPI has blocked the input [4][6][5]. To bypass these restrictions, applications typically require the 'uiAccess="true"' attribute in their manifest, must be digitally signed, and often need to be installed in a protected location like Program Files [6][7].
Citations:
- 1: https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-sendinput
- 2: https://learn.microsoft.com/en-us/previous-versions/ms911051(v=msdn.10)
- 3: https://github.com/MicrosoftDocs/sdk-api/blob/docs/sdk-api-src/content/winuser/nf-winuser-sendinput.md
- 4: https://learn.microsoft.com/en-us/archive/msdn-technet-forums/b68a77e7-cd00-48d0-90a6-d6a4a46a95aa
- 5: https://stackoverflow.com/questions/62906186/how-to-run-windows-automation-application-in-aws-window-server
- 6: https://stackoverflow.com/questions/17645204/sendinput-fail-because-of-uipi
- 7: https://stackoverflow.com/questions/56595640/sendinput-fails-on-uac-prompt
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the relevant file and inspect the cited region plus the caller.
FILE="stt-ui/src-tauri/src/lib.rs"
echo "== file exists =="
git ls-files "$FILE"
echo
echo "== line count =="
wc -l "$FILE"
echo
echo "== outline around relevant symbols =="
ast-grep outline "$FILE" --view expanded || true
echo
echo "== cited region around send_text_unicode =="
sed -n '1040,1135p' "$FILE" | cat -n
echo
echo "== caller region around type_text =="
sed -n '1200,1255p' "$FILE" | cat -n
echo
echo "== search for send_text_unicode usages =="
rg -n "send_text_unicode|type_text\(" "$FILE"Repository: IntegerAlex/floure-core
Length of output: 11192
Propagate SendInput failures in the Windows Unicode fallback. send_text_unicode ignores SendInput’s return value, so type_text can still return Ok(true) even when no input was injected. Bubble the status back up at stt-ui/src-tauri/src/lib.rs:1063-1103 and 1226-1227.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt-ui/src-tauri/src/lib.rs` around lines 1063 - 1103, The Windows Unicode
input path in send_text_unicode is ignoring SendInput failures, which can make
type_text report success even when nothing was injected. Update
send_text_unicode to return a status/result from each SendInput call, including
the surrogate-pair branch, and have type_text check and propagate that failure
instead of always returning Ok(true). Use the existing send_text_unicode and
type_text symbols to wire the error/status back to the caller.
| // Try clipboard+Ctrl+V first (fast, preserves clipboard) | ||
| if win32::set_clipboard(&text) { | ||
| std::thread::sleep(std::time::Duration::from_millis(30)); | ||
| win32::send_ctrl_v(); | ||
| return Ok(true); | ||
| } | ||
| // Small delay for clipboard to propagate | ||
| std::thread::sleep(std::time::Duration::from_millis(30)); | ||
| // Send Ctrl+V via keybd_event (runs in Tauri's GUI thread — has active message loop) | ||
| win32::send_ctrl_v(); | ||
| // Fallback: SendInput Unicode (works in apps that ignore Ctrl+V) | ||
| win32::send_text_unicode(&text); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Can Windows keybd_event or SendInput Ctrl+V report whether a target application accepted paste
💡 Result:
No, the Windows keybd_event and SendInput functions cannot directly report whether a target application has accepted or processed a "paste" (Ctrl+V) command [1][2]. These functions operate by injecting events into the system input stream [3][4]. Once the input is injected, the operating system treats it as synthetic hardware input and dispatches it to the appropriate window's message queue [1]. Because this process is asynchronous, the calling application has no built-in mechanism to know if or when the target application retrieves that message from its queue, validates the clipboard content, or successfully performs the paste operation [2]. Technical limitations include: - Asynchronous Delivery: SendInput returns the number of events successfully inserted into the input stream, not the outcome of the target application's processing [4][2]. - Lack of Feedback: There is no return path from the target application to the source of the synthetic input, meaning the target application provides no signal to the sender regarding whether the paste was handled [2]. To achieve a reliable result, you must use alternative strategies to monitor the state of the target application: - Monitoring UI Changes: You can use Microsoft UI Automation to observe changes in the target application's UI (e.g., checking if text content has updated) [5]. - Input Synchronization: In some cases, you can use WaitForInputIdle to detect when a process is idle, which may indicate it has finished processing messages, though this is not a guaranteed confirmation of a specific paste operation [5]. - Direct Data Injection: If the target application supports it, rather than simulating keys, it is significantly more robust to interact with the application directly via its object model, accessibility APIs, or by sending specific application-defined messages (e.g., WM_PASTE) directly to the target window handle [2]. If you are attempting to detect when a paste happens in an application you control, you can monitor local events such as WM_PASTE, TextChanged, or use AddClipboardFormatListener to track clipboard interactions [6][7][8][9]. However, these are internal application hooks, not feedback provided by the SendInput API itself.
Citations:
- 1: https://devblogs.microsoft.com/oldnewthing/20250319-00/?p=110979
- 2: https://exchangetuts.com/when-using-sendinput-how-can-i-tell-when-the-receiving-window-has-received-and-processed-the-messages-1765058405099425
- 3: https://stackoverflow.com/questions/4553777/detecting-simulated-keyboard-mouse-input
- 4: https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-sendinput
- 5: https://stackoverflow.com/questions/21028326/is-it-possible-to-wait-until-other-thread-processes-input-messages-posted-to-it
- 6: https://forum.xojo.com/t/intercept-a-paste-action-ctrl-v-or-shift-ins-on-windows/21871
- 7: https://stackoverflow.com/questions/79306529/how-to-catch-copy-paste-events-inside-of-a-windows-application-using-windows-api
- 8: https://stackoverflow.com/questions/2226920/how-do-i-monitor-clipboard-content-changes-in-c
- 9: https://iditect.com/faq/csharp/detecting-if-paste-event-occurred-inside-a-rich-text-box-in-winforms.html
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the relevant area in the Tauri backend and locate the input helpers.
git ls-files stt-ui/src-tauri/src/lib.rs
rg -n "set_clipboard|send_ctrl_v|send_text_unicode|paste|Ctrl\\+V|SendInput|keybd_event" stt-ui/src-tauri/src/lib.rs stt-ui/src-tauri/src -S
# Read the nearby implementation with line numbers.
sed -n '1180,1265p' stt-ui/src-tauri/src/lib.rs
# Inspect the helper definitions if they live in the same crate.
rg -n "fn send_text_unicode|fn send_ctrl_v|fn set_clipboard" stt-ui/src-tauri/src -SRepository: IntegerAlex/floure-core
Length of output: 9651
Unicode fallback never runs after a successful clipboard paste
send_ctrl_v() is fire-and-forget, so this branch returns Ok(true) even when the focused app ignores Ctrl+V. Apps like terminals, Electron, or UWP never reach send_text_unicode(), which means input can be dropped while the command reports success. Route those targets through the Unicode path up front, or add a delivery check before returning success.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt-ui/src-tauri/src/lib.rs` around lines 1219 - 1226, The paste flow
currently returns success immediately after set_clipboard and send_ctrl_v, even
though Ctrl+V is fire-and-forget and may be ignored by some targets. Update the
clipboard paste branch to verify delivery before returning Ok(true), or route
known incompatible targets directly to send_text_unicode instead of exiting
early. Use the existing win32::set_clipboard, win32::send_ctrl_v, and
win32::send_text_unicode paths to keep the behavior aligned with the focused
app’s capabilities.
| child.write(JSON.stringify(cmd) + "\n").catch((e) => { | ||
| console.warn("[Engine] Failed to write command:", e); | ||
| } | ||
| }); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | ⚡ Quick win
Surface command write failures to the app.
The catch only logs, while callers update PTT state as if start_recording/stop_recording was delivered. Notify listeners so the UI can leave the recording/committing path instead of getting stuck.
🐛 Proposed fix to notify on write failure
- child.write(JSON.stringify(cmd) + "\n").catch((e) => {
- console.warn("[Engine] Failed to write command:", e);
+ child.write(JSON.stringify(cmd) + "\n").catch((e) => {
+ const message = e instanceof Error ? e.message : String(e);
+ console.warn("[Engine] Failed to write command:", e);
+ notifyError(`Failed to send command to engine: ${message}`);
});📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| child.write(JSON.stringify(cmd) + "\n").catch((e) => { | |
| console.warn("[Engine] Failed to write command:", e); | |
| } | |
| }); | |
| child.write(JSON.stringify(cmd) + "\n").catch((e) => { | |
| const message = e instanceof Error ? e.message : String(e); | |
| console.warn("[Engine] Failed to write command:", e); | |
| notifyError(`Failed to send command to engine: ${message}`); | |
| }); |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt-ui/src/api-tauri.ts` around lines 109 - 111, The write failure in the
command path is only being logged inside the child.write catch, so callers like
the start_recording/stop_recording flow still assume delivery succeeded. Update
the command-sending logic in api-tauri.ts to surface this failure to listeners,
using the same notification mechanism the app relies on for recording state
changes, so the UI can exit the recording/committing path when a write to child
fails. Keep the fix centered around the child.write call and the surrounding
command dispatch helper so all command types propagate the error consistently.
| const rawId = event.utterance_id ?? nextLocalId.current++; | ||
| const id = sessionCounter.current * 100000 + rawId; | ||
| setLines((prev) => [ | ||
| ...prev, | ||
| { id, raw: event.text, processed: "", status: "transcribing", createdAt: new Date().toISOString() }, | ||
| ].slice(-500)); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Keep the session-prefixed line ID consistent across later events.
Line 1036 stores the live row under a session-scoped ID, but the processed, llm_partial, and error handlers still look rows up with the bare backend utterance_id. After this change, those updates can no longer find the inserted row, so entries stay stuck in transcribing.
Suggested fix
+ const toSessionLineId = (utteranceId: number) =>
+ sessionCounter.current * 100000 + utteranceId;
if (event.type === "raw") {
const rawId = event.utterance_id ?? nextLocalId.current++;
- const id = sessionCounter.current * 100000 + rawId;
+ const id = toSessionLineId(rawId);
setLines((prev) => [
...prev,
{ id, raw: event.text, processed: "", status: "transcribing", createdAt: new Date().toISOString() },
].slice(-500));Apply the same helper in the processed, llm_partial, and error handlers before comparing against line.id.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt-ui/src/App.tsx` around lines 1035 - 1040, The session-prefixed line ID
used when inserting rows in App.tsx is not being used consistently by later
updates. Update the processed, llm_partial, and error handlers to compute the
same session-scoped ID helper used for the live row before matching against
line.id, so they continue to find the existing entry instead of comparing
against the bare utterance_id.
| startRef.current = start; | ||
| stopRef.current = stop; | ||
| stopRef.current = stopAndCommit; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Re-export the new stop handler to FeedView.
After renaming the implementation to stopAndCommit, the render path still passes stop={stop} into FeedView at Line 1352. In this module that can resolve to the global stop, so the mic button no longer calls the PTT stop flow.
Suggested fix
- stop={stop}
+ stop={stopAndCommit}📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| startRef.current = start; | |
| stopRef.current = stop; | |
| stopRef.current = stopAndCommit; | |
| stop={stopAndCommit} |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt-ui/src/App.tsx` around lines 1166 - 1167, The stop handler wiring in
App.tsx is stale after renaming the implementation to stopAndCommit, so FeedView
is still receiving the wrong stop reference and may fall back to the global
stop. Update the render path that passes props into FeedView so its stop prop
uses the same handler stored in stopRef and points to stopAndCommit, keeping the
PTT mic button connected to the correct stop flow.
| # --- Wait for pending transcription threads before going idle --- | ||
| # This ensures pttTextRef has the latest transcription when the | ||
| # frontend reads it after receiving state:idle. | ||
| if _active_xcribe_threads: | ||
| _echo(f"Waiting for {_active_xcribe_threads.__len__()} pending transcription(s)...") | ||
| _deadline = time.monotonic() + 15.0 # max 15 s wait | ||
| for t in _active_xcribe_threads: | ||
| remaining = max(0.1, _deadline - time.monotonic()) | ||
| t.join(timeout=remaining) | ||
| _active_xcribe_threads.clear() | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Don’t emit idle while timed-out workers may still publish text.
The join has a timeout, but the code clears _active_xcribe_threads and emits state: idle even if a worker is still alive. Late raw/processed events can arrive after the frontend has committed stale pttTextRef; this is especially reachable because the frontend commit timeout shown in context is 10s while this backend deadline is 15s. Check t.is_alive() after the deadline and emit a timeout/error or suppress late worker output instead of treating idle as complete.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt/orchestrator.py` around lines 785 - 795, The idle transition in
orchestrator.py still treats pending transcription threads as finished even when
a join times out, which can let late output arrive after state:idle. Update the
pending-thread handling around _active_xcribe_threads so that after the join
deadline you check each thread with t.is_alive() before clearing the list, and
if any are still running either suppress their late raw/processed emissions or
emit a timeout/error instead of proceeding to idle. Use the existing
_active_xcribe_threads wait block and the idle/state publish path to ensure idle
is only sent once all worker output is truly complete.
- Created commands.rs: begin_capture, end_capture, insert_text, show_overlay, hide_overlay - Removed Rust state machine (state.rs, typing.rs, overlay.rs) - Rewrote App.tsx with PttState: idle→listening→processing→inserting→success - Rewrote OverlayView.tsx to listen overlay:command events (pure rendering) - Added overlay window to tauri.conf.json (280x60, transparent, alwaysOnTop) - Added main.tsx routing for overlay window - Positioned overlay centered above taskbar via primary_monitor() - Backend now only executes commands, frontend owns all decisions
- Stop writing raw LLM tokens to stdout in json_mode (corrupted the Tauri JSON event channel, dropped processed events, and emitted non-UTF-8 bytes causing 'invalid utf-8' sidecar errors) - Wait for in-flight transcription threads before emitting idle so the UI commits transcript text instead of 'Nothing to commit' - Release ASR model on idle (STT_IDLE_RELEASE_SEC, default 20s) to free several GB of RAM/VRAM; re-warm on next session. Add release_backend() - Gate few-shot embeddings behind STT_EMBEDDINGS=1 (was loading sentence-transformers on every utterance) - Redirect whisper.cpp subprocess stderr to devnull instead of capture_output=True to avoid buffering child output in RAM - Add stt-win11/ (1.8GB generated venv copy) to .gitignore
There was a problem hiding this comment.
🟠 High
floure-core/stt-ui/src/App.tsx
Line 1 in 3bad50e
start sets the PTT state to listening before the awaited overlay/HWND operations complete, so if the user releases PTT during those awaits, stopAndCommit runs and sends stop_recording, then the suspended start resumes and sends start_recording. This leaves the backend recording while the frontend has already transitioned to processing (and will eventually time out), and because the PTT state is no longer listening, normal stop controls are blocked. Consider setting the state to listening immediately before runtimeRef.current.start() is called, or guarding the pending start against an intervening stop so that start_recording is not sent after stop_recording.
🤖 Copy this AI Prompt to have your agent fix this:
In file @stt-ui/src/App.tsx around line 1:
`start` sets the PTT state to `listening` before the awaited overlay/HWND operations complete, so if the user releases PTT during those awaits, `stopAndCommit` runs and sends `stop_recording`, then the suspended `start` resumes and sends `start_recording`. This leaves the backend recording while the frontend has already transitioned to `processing` (and will eventually time out), and because the PTT state is no longer `listening`, normal stop controls are blocked. Consider setting the state to `listening` immediately before `runtimeRef.current.start()` is called, or guarding the pending `start` against an intervening stop so that `start_recording` is not sent after `stop_recording`.
| if let Ok(Some(monitor)) = win.primary_monitor() { | ||
| let m_size = monitor.size(); | ||
| let m_pos = monitor.position(); | ||
| let pill_w = 280; |
There was a problem hiding this comment.
🟡 Medium src/commands.rs:39
show_overlay positions the overlay using hard-coded 280x60 dimensions, but these are logical-pixel values. On a scaled display (e.g. 150% or 200% DPI), the window's physical size is larger, so the computed PhysicalPosition is visibly off-center and the bottom margin is wrong. The position is derived from pill_w and pill_h constants instead of the window's actual physical size, so set_position receives coordinates that don't match the real window dimensions. Consider scaling the logical dimensions by monitor.scale_factor() (or using LogicalPosition so the framework converts for you).
🤖 Copy this AI Prompt to have your agent fix this:
In file @stt-ui/src-tauri/src/commands.rs around line 39:
`show_overlay` positions the overlay using hard-coded `280x60` dimensions, but these are logical-pixel values. On a scaled display (e.g. 150% or 200% DPI), the window's physical size is larger, so the computed `PhysicalPosition` is visibly off-center and the bottom margin is wrong. The position is derived from `pill_w` and `pill_h` constants instead of the window's actual physical size, so `set_position` receives coordinates that don't match the real window dimensions. Consider scaling the logical dimensions by `monitor.scale_factor()` (or using `LogicalPosition` so the framework converts for you).
|
|
||
| return { | ||
| /** Start capturing audio and emitting bar values. */ | ||
| async start() { |
There was a problem hiding this comment.
🟡 Medium overlay/waveform.ts:96
start() does not guard against concurrent calls while getUserMedia() is still resolving, so two calls before the first completes both pass the state.active check and each acquires its own microphone stream. They overwrite the shared state.stream, state.audioContext, and state.analyser fields, so stop() only tears down the last one — the earlier stream stays live, leaking microphone capture and audio resources. Repeated listening commands during a permission prompt can trigger this. Consider tracking a pending-start state so concurrent calls no-op while the first is in flight.
Also found in 1 other location(s)
stt-ui/src/overlay/OverlayView.tsx:65
Starting the waveform is asynchronous and is neither awaited nor cancelled. If a
processing/insertingcommand arrives whilestart()is waiting for microphone permission,isActive()is still false so line 69 does not stop it; once permission resolves, capture and its animation loop start even though the overlay is no longer listening. Anidlecommand during the wait can similarly makestart()fall into its catch path afterstop(), reactivating the fallback RAF while the overlay is idle.
🤖 Copy this AI Prompt to have your agent fix this:
In file @stt-ui/src/overlay/waveform.ts around line 96:
`start()` does not guard against concurrent calls while `getUserMedia()` is still resolving, so two calls before the first completes both pass the `state.active` check and each acquires its own microphone stream. They overwrite the shared `state.stream`, `state.audioContext`, and `state.analyser` fields, so `stop()` only tears down the last one — the earlier stream stays live, leaking microphone capture and audio resources. Repeated `listening` commands during a permission prompt can trigger this. Consider tracking a pending-start state so concurrent calls no-op while the first is in flight.
Also found in 1 other location(s):
- stt-ui/src/overlay/OverlayView.tsx:65 -- Starting the waveform is asynchronous and is neither awaited nor cancelled. If a `processing`/`inserting` command arrives while `start()` is waiting for microphone permission, `isActive()` is still false so line 69 does not stop it; once permission resolves, capture and its animation loop start even though the overlay is no longer listening. An `idle` command during the wait can similarly make `start()` fall into its catch path after `stop()`, reactivating the fallback RAF while the overlay is idle.
| _ws_clients: list = [] | ||
| _ws_loop = None # set by start_ws_server | ||
|
|
||
| # Pending idle model-release timer (see _schedule_idle_release / _cancel_idle_release). |
There was a problem hiding this comment.
🟠 High stt/orchestrator.py:82
_cancel_idle_release() sets _idle_release_timer to None unconditionally, so any code that later checks if _idle_release_timer is None to decide whether the model needs warm-up treats every session as cold-started. On the very first PTT session this races the startup warm-up, and because the model getters use an unsynchronized check-then-create, both threads load the same multi-GB model concurrently — risking RAM/VRAM exhaustion. Consider tracking model residency/warm-up state separately from the timer handle, or serializing warm-up and cache creation with a lock.
Also found in 1 other location(s)
stt/transcription.py:40
release_backend()clears the model caches without synchronizing with model warm-up/cache lookup.threading.Timer.cancel()does not stop a callback that has already begun, so a new PTT session can startwarm_up_backend()while the old idle callback executes these clears. The callback can then remove the newly loaded model (and clear framework allocator caches), causing the session's first transcription to reload it again and incur the cold-start/multi-GB allocation that the warm-up is intended to avoid. Guard release and all cache creation with the same lock, and make the timer callback verify that it still owns the current idle generation.
🤖 Copy this AI Prompt to have your agent fix this:
In file @stt/orchestrator.py around line 82:
`_cancel_idle_release()` sets `_idle_release_timer` to `None` unconditionally, so any code that later checks `if _idle_release_timer is None` to decide whether the model needs warm-up treats every session as cold-started. On the very first PTT session this races the startup warm-up, and because the model getters use an unsynchronized check-then-create, both threads load the same multi-GB model concurrently — risking RAM/VRAM exhaustion. Consider tracking model residency/warm-up state separately from the timer handle, or serializing warm-up and cache creation with a lock.
Also found in 1 other location(s):
- stt/transcription.py:40 -- `release_backend()` clears the model caches without synchronizing with model warm-up/cache lookup. `threading.Timer.cancel()` does not stop a callback that has already begun, so a new PTT session can start `warm_up_backend()` while the old idle callback executes these clears. The callback can then remove the newly loaded model (and clear framework allocator caches), causing the session's first transcription to reload it again and incur the cold-start/multi-GB allocation that the warm-up is intended to avoid. Guard release and all cache creation with the same lock, and make the timer callback verify that it still owns the current idle generation.
|
|
||
| // Tell overlay to clean up, wait for acknowledgement, then hide | ||
| overlayCleaningRef.current = true; | ||
| try { |
There was a problem hiding this comment.
🟡 Medium src/App.tsx:1217
Both overlay cleanup paths in stopAndCommit call emitTo("overlay", "overlay:command", "idle") and then install overlayIdleResolveRef afterwards. If the overlay processes the command and emits overlay:idle_ready before that assignment runs, the listener (set up in a separate effect) finds overlayIdleResolveRef.current === null and discards the acknowledgement. The code then blocks on the promise until the 2-second fallback timeout fires, delaying the overlay hide and the return to idle state by the full timeout on every race win. Install the resolver before emitting the command so the acknowledgement is always captured.
🤖 Copy this AI Prompt to have your agent fix this:
In file @stt-ui/src/App.tsx around line 1217:
Both overlay cleanup paths in `stopAndCommit` call `emitTo("overlay", "overlay:command", "idle")` and then install `overlayIdleResolveRef` afterwards. If the overlay processes the command and emits `overlay:idle_ready` before that assignment runs, the listener (set up in a separate effect) finds `overlayIdleResolveRef.current === null` and discards the acknowledgement. The code then blocks on the promise until the 2-second fallback timeout fires, delaying the overlay hide and the return to `idle` state by the full timeout on every race win. Install the resolver before emitting the command so the acknowledgement is always captured.
| # exists (pttTextRef still empty) → "Nothing to commit", nothing typed. | ||
| # Join with a bounded timeout so a hung ASR can't stall the loop | ||
| # forever, but a normal utterance finishes well within the window. | ||
| if _active_xcribe_threads: |
There was a problem hiding this comment.
🟠 High stt/orchestrator.py:1010
A start_recording command that arrives while the previous session's transcription threads are still being joined is silently lost. The join loop at line 1013 can block for up to 180 seconds; during that window _start_event is already set from the current session, so the stdin reader's _start_event.set() records no new state. Immediately after the join, lines 1018–1019 unconditionally clear both _start_event and _stop_event, discarding the pending start. The backend then emits idle and waits for another start_recording that the user already sent — the requested session never opens. Consider clearing _start_event (and _stop_event) before joining transcription threads, so a start_recording arriving during the join is preserved for the next loop iteration.
🤖 Copy this AI Prompt to have your agent fix this:
In file @stt/orchestrator.py around line 1010:
A `start_recording` command that arrives while the previous session's transcription threads are still being joined is silently lost. The join loop at line 1013 can block for up to 180 seconds; during that window `_start_event` is already set from the current session, so the stdin reader's `_start_event.set()` records no new state. Immediately after the join, lines 1018–1019 unconditionally clear both `_start_event` and `_stop_event`, discarding the pending start. The backend then emits `idle` and waits for another `start_recording` that the user already sent — the requested session never opens. Consider clearing `_start_event` (and `_stop_event`) *before* joining transcription threads, so a `start_recording` arriving during the join is preserved for the next loop iteration.
| const svgRef = useRef<SVGSVGElement>(null); | ||
| const waveformRef = useRef<ReturnType<typeof createWaveform> | null>(null); | ||
| const rendererInitRef = useRef(false); | ||
| const scaleSpring = createSpring(SPRING_SNAPPY); |
There was a problem hiding this comment.
🟡 Medium overlay/OverlayView.tsx:44
The pill's scale animation never updates the DOM. scaleSpring is created with createSpring during render, so every setState call produces a new spring instance. The mount-only useEffect registers the onUpdate callback that writes transform to #overlay-pill on the initial spring, but the [state] effect calls setTarget on whatever spring the latest render created — an instance that has no onUpdate callback registered. As a result, the spring animates internally but el.style.transform is never written. Wrap scaleSpring in useRef (or useMemo) so the same instance persists across renders and the callback stays attached.
| const scaleSpring = createSpring(SPRING_SNAPPY); | |
| const scaleSpringRef = useRef<ReturnType<typeof createSpring> | null>(null); | |
| if (!scaleSpringRef.current) { | |
| scaleSpringRef.current = createSpring(SPRING_SNAPPY); | |
| } | |
| const scaleSpring = scaleSpringRef.current; |
🤖 Copy this AI Prompt to have your agent fix this:
In file @stt-ui/src/overlay/OverlayView.tsx around line 44:
The pill's scale animation never updates the DOM. `scaleSpring` is created with `createSpring` during render, so every `setState` call produces a new spring instance. The mount-only `useEffect` registers the `onUpdate` callback that writes `transform` to `#overlay-pill` on the initial spring, but the `[state]` effect calls `setTarget` on whatever spring the latest render created — an instance that has no `onUpdate` callback registered. As a result, the spring animates internally but `el.style.transform` is never written. Wrap `scaleSpring` in `useRef` (or `useMemo`) so the same instance persists across renders and the callback stays attached.
| # The ASR model is the dominant memory user; we free it a short grace period | ||
| # after the user stops dictating, then re-warm on the next session. | ||
| _idle_release_timer: threading.Timer | None = None | ||
| _IDLE_RELEASE_GRACE_SEC = float(os.environ.get("STT_IDLE_RELEASE_SEC", "20")) |
There was a problem hiding this comment.
🟡 Medium stt/orchestrator.py:86
Parsing STT_IDLE_RELEASE_SEC with an unguarded float(...) at import time crashes module import when the env var is malformed (e.g. STT_IDLE_RELEASE_SEC=20s or empty), preventing the entire STT service from starting. Consider wrapping the parse in a try/except that falls back to the default on ValueError.
-_IDLE_RELEASE_GRACE_SEC = float(os.environ.get("STT_IDLE_RELEASE_SEC", "20"))
+_IDLE_RELEASE_GRACE_SEC = 20.0
+try:
+ _IDLE_RELEASE_GRACE_SEC = float(os.environ.get("STT_IDLE_RELEASE_SEC", "20"))
+except ValueError:
+ pass🤖 Copy this AI Prompt to have your agent fix this:
In file @stt/orchestrator.py around line 86:
Parsing `STT_IDLE_RELEASE_SEC` with an unguarded `float(...)` at import time crashes module import when the env var is malformed (e.g. `STT_IDLE_RELEASE_SEC=20s` or empty), preventing the entire STT service from starting. Consider wrapping the parse in a `try`/`except` that falls back to the default on `ValueError`.
| }, []); | ||
|
|
||
| // Initialize waveform and SVG renderer | ||
| useEffect(() => { |
There was a problem hiding this comment.
🟡 Medium overlay/OverlayView.tsx:124
The waveform initialization effect never initializes waveformRef.current, so microphone capture and bar animation never start. The effect runs only once on mount with empty deps, but at that point state is "idle" so the component returns null and svgRef.current is null — the guard if (svgRef.current && !rendererInitRef.current) fails. When a later listening command renders the <svg>, the effect does not re-run, so waveformRef.current stays null and the command handler's call to waveformRef.current.start() does nothing. Consider depending the effect on state (or on svgRef.current becoming available) so initialization runs after the SVG is actually rendered.
🤖 Copy this AI Prompt to have your agent fix this:
In file @stt-ui/src/overlay/OverlayView.tsx around line 124:
The waveform initialization effect never initializes `waveformRef.current`, so microphone capture and bar animation never start. The effect runs only once on mount with empty deps, but at that point `state` is `"idle"` so the component returns `null` and `svgRef.current` is `null` — the guard `if (svgRef.current && !rendererInitRef.current)` fails. When a later `listening` command renders the `<svg>`, the effect does not re-run, so `waveformRef.current` stays `null` and the command handler's call to `waveformRef.current.start()` does nothing. Consider depending the effect on `state` (or on `svgRef.current` becoming available) so initialization runs after the SVG is actually rendered.
| # Acquire the ASR semaphore (warmup holds it while loading the model). | ||
| # Use a blocking acquire with timeout so we wait for warmup instead of | ||
| # silently dropping the utterance. | ||
| if not _asr_semaphore.acquire(blocking=True, timeout=120): |
There was a problem hiding this comment.
🟠 High stt/orchestrator.py:1256
_asr_semaphore.acquire(blocking=True, timeout=120) queues every overlapping utterance, not just ones waiting for startup warm-up. The semaphore is held for the full decode, so when ASR is slower than real time (or speech is sustained), each VAD endpoint spawns a new thread that blocks up to 120s. Transcripts arrive stale and a growing backlog of blocked threads prevents prompt session shutdown. The old blocking=False call intentionally dropped overlap via asr_busy; the timeout-based acquire cannot distinguish warm-up from an in-progress decode. Consider gating the blocking wait on an explicit warm-up flag and dropping overlap otherwise.
🤖 Copy this AI Prompt to have your agent fix this:
In file @stt/orchestrator.py around line 1256:
`_asr_semaphore.acquire(blocking=True, timeout=120)` queues every overlapping utterance, not just ones waiting for startup warm-up. The semaphore is held for the full decode, so when ASR is slower than real time (or speech is sustained), each VAD endpoint spawns a new thread that blocks up to 120s. Transcripts arrive stale and a growing backlog of blocked threads prevents prompt session shutdown. The old `blocking=False` call intentionally dropped overlap via `asr_busy`; the timeout-based acquire cannot distinguish warm-up from an in-progress decode. Consider gating the blocking wait on an explicit warm-up flag and dropping overlap otherwise.
There was a problem hiding this comment.
Actionable comments posted: 13
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
stt-ui/src/App.tsx (1)
1077-1101: 🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy liftAccumulate every utterance before committing.
VAD creates a new
utterance_idfor each endpoint, but every event overwritespttTextRef. A PTT session containing pauses therefore inserts only whichever utterance produced the latest event.Track text by utterance ID and join the ordered finalized segments at commit time.
Also applies to: 1292-1294
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@stt-ui/src/App.tsx` around lines 1077 - 1101, Update the PTT text handling in the event-processing logic around the transcribing, processed, and llm_partial branches to accumulate text per utterance ID instead of overwriting pttTextRef.current on every event. Preserve segment order, update each segment as events arrive, and join all finalized utterance segments when the PTT session commits so pauses do not discard earlier speech.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@stt-ui/src-tauri/capabilities/default.json`:
- Line 5: Remove overlay from the windows list in the default capability and
create a dedicated overlay capability configuration for the OverlayView
component, granting only the required window and event permissions used by its
Tauri listen/emitTo calls. Preserve the existing permissions for main and widget
windows and ensure the overlay remains associated with its dedicated capability.
In `@stt-ui/src/App.tsx`:
- Around line 1215-1227: In both overlay cleanup sites in stt-ui/src/App.tsx
(anchor lines 1215-1227 and sibling lines 1341-1353), initialize the
acknowledgement promise and assign overlayIdleResolveRef before emitting the
"idle" command; retain the existing timeout fallback and cleanup behavior so
early overlay:idle_ready events resolve immediately.
- Around line 1145-1179: Update the PTT start flow around the listening-state
transition to create or advance the session ID, set the state, and invoke
runtimeRef.current.start() immediately without any await between them. Capture
that session ID and guard the subsequent overlay commands and foreground-HWND
updates so stale asynchronous work cannot affect a newer session; preserve the
existing setup and cleanup behavior.
- Around line 1296-1327: Update the PTT commit flow surrounding the insertion
logic so success is emitted only after text and hwnd are present, insert_text
completes without throwing, and returns ok === true. Track unsuccessful outcomes
from the ok === false, exception, missing-text, and missing-HWND branches, then
skip setPtt("success"), the success overlay event, and “Done” confirmation for
those cases while showing the existing error/no-result state and completing
normal cleanup.
- Around line 869-875: Update the catch handler for the engine spawn promise to
immediately return when spawnedRef.current !== api, before constructing the
message or calling setToast/addError. Preserve the existing cleanup for the
current API instance so stale spawn failures produce no user-facing errors.
In `@stt-ui/src/overlay/OverlayView.tsx`:
- Around line 123-162: Update the waveform initialization useEffect in
OverlayView so it depends on state and runs when the conditional SVG is mounted
in the "listening" state, rather than only on initial render. After creating the
waveform and registering onBars, start it for that state; when listening ends,
stop and dispose the existing waveform and reset rendererInitRef as needed so
future listening cycles reinitialize correctly.
- Line 44: Keep the spring instance in OverlayView stable across renders by
creating it through the component’s persistent state or memoization mechanism,
rather than invoking createSpring(SPRING_SNAPPY) on every render. Ensure the
mount effect’s onUpdate registration and subsequent setTarget calls both use
that same spring instance.
- Around line 106-120: Cancel asynchronous listener registration during cleanup
in OverlayView.tsx lines 106-120 and App.tsx lines 1397-1412. Add a cancellation
guard checked before retaining the listener, and immediately invoke the returned
unlisten function if registration completes after cleanup; preserve retry
behavior and existing cleanup for successfully registered listeners.
In `@stt-ui/src/overlay/spring.ts`:
- Around line 88-93: Update setValue() in the spring controller to cancel or
invalidate any active RAF animation before assigning the new value. Ensure the
current animation cannot continue updating state or invoke onUpdate after
setValue() resets the value and velocity.
In `@stt-ui/src/overlay/waveform.ts`:
- Around line 96-148: Update the waveform start/stop lifecycle around start()
and stop() to use a generation or cancellation token that invalidates pending
getUserMedia attempts. Acquire the AudioContext, analyser, stream, and related
resources in local variables, verify the start attempt is still current after
each await, and immediately stop tracks and close the context when invalidated;
only then assign shared state, set active, and schedule the RAF. Increment the
token in stop() before cleanup so a later getUserMedia resolution cannot restart
the waveform or leave the microphone open.
In `@stt/orchestrator.py`:
- Around line 739-750: The session-start warm-up logic currently relies on
_idle_release_timer, which is cleared by _cancel_idle_release and therefore
cannot indicate whether the backend was released. Add and maintain an explicit
_backend_released state, set it when idle release completes and clear it when
the backend is retained or successfully re-warmed; update the condition around
_warmup_target to start warm-up only when that state is true, preventing
duplicate concurrent warm-ups.
- Around line 111-124: Serialize ASR lifecycle operations so release, warm-up,
and model use cannot overlap. In stt/orchestrator.py lines 111-124, protect
timer cancellation and the _release callback with a shared lifecycle lock or
generation token, ensuring a canceled or stale timer cannot release backend
state after a new session begins. In stt/transcription.py lines 39-58, use the
same synchronization around cache clearing and allocator cleanup; update the
relevant release, warm-up, and model-use paths while preserving existing
behavior.
- Around line 949-958: Remove the logging-time call to
detector._compute_speech_score(chunk) in the VAD diagnostics block, since it
mutates spectral state before detector.update() processes the chunk. If the
score must remain logged, capture and expose it from detector.update() and log
that already-computed value without recomputing the chunk.
---
Outside diff comments:
In `@stt-ui/src/App.tsx`:
- Around line 1077-1101: Update the PTT text handling in the event-processing
logic around the transcribing, processed, and llm_partial branches to accumulate
text per utterance ID instead of overwriting pttTextRef.current on every event.
Preserve segment order, update each segment as events arrive, and join all
finalized utterance segments when the PTT session commits so pauses do not
discard earlier speech.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 045f2534-364a-427a-be9a-5f150732a5c3
⛔ Files ignored due to path filters (1)
stt-ui/src-tauri/Cargo.lockis excluded by!**/*.lock
📒 Files selected for processing (17)
.gitignorestt-ui/src-tauri/Cargo.tomlstt-ui/src-tauri/capabilities/default.jsonstt-ui/src-tauri/src/commands.rsstt-ui/src-tauri/src/lib.rsstt-ui/src-tauri/tauri.conf.jsonstt-ui/src/App.tsxstt-ui/src/api-tauri.tsstt-ui/src/components/PttOverlay.tsxstt-ui/src/components/WidgetView.tsxstt-ui/src/main.tsxstt-ui/src/overlay/OverlayView.tsxstt-ui/src/overlay/spring.tsstt-ui/src/overlay/waveform.tsstt/orchestrator.pystt/transcription.pystt/vad.py
💤 Files with no reviewable changes (1)
- stt-ui/src/components/PttOverlay.tsx
🚧 Files skipped from review as they are similar to previous changes (1)
- stt-ui/src/api-tauri.ts
| "identifier": "default", | ||
| "description": "Capability for the main window", | ||
| "windows": ["main", "widget"], | ||
| "windows": ["main", "widget", "overlay"], |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Verify which privileged APIs the overlay actually uses.
rg -n 'invoke|listen|emit|clipboard|shell|globalShortcut|updater' \
stt-ui/src/overlay/OverlayView.tsx stt-ui/src/main.tsx
# Inspect existing capability assignments.
rg -n '"windows"|shell:|clipboard-manager:|global-shortcut:|updater:' \
stt-ui/src-tauri/capabilitiesRepository: IntegerAlex/floure-core
Length of output: 2747
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map capability files and overlay references.
printf '\n## capabilities files\n'
git ls-files 'stt-ui/src-tauri/capabilities/*' | sort
printf '\n## overlay references in source\n'
rg -n 'overlay:|OverlayView|window.*overlay|label.*overlay|windows": \["main", "widget", "overlay"\]' stt-ui/src-tauri stt-ui/src/overlay
printf '\n## default capability context\n'
sed -n '1,120p' stt-ui/src-tauri/capabilities/default.jsonRepository: IntegerAlex/floure-core
Length of output: 2833
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '1,140p' stt-ui/src-tauri/src/commands.rsRepository: IntegerAlex/floure-core
Length of output: 4477
Split the overlay into a least-privilege capability. stt-ui/src-tauri/capabilities/default.json currently gives overlay shell, clipboard, global-shortcut, and updater permissions even though stt-ui/src/overlay/OverlayView.tsx only uses Tauri events (listen/emitTo). Move it to a dedicated overlay capability with just the window/event permissions it needs.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt-ui/src-tauri/capabilities/default.json` at line 5, Remove overlay from
the windows list in the default capability and create a dedicated overlay
capability configuration for the OverlayView component, granting only the
required window and event permissions used by its Tauri listen/emitTo calls.
Preserve the existing permissions for main and widget windows and ensure the
overlay remains associated with its dedicated capability.
| }).catch((err) => { | ||
| const msg = err instanceof Error ? err.message : "Failed to start engine"; | ||
| setToast(msg); | ||
| addError("connection", msg, true, "Check if stt-engine is installed"); | ||
| runtimeRef.current = null; | ||
| if (spawnedRef.current === api) { | ||
| runtimeRef.current = null; | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
file='stt-ui/src/App.tsx'
echo '=== file size ==='
wc -l "$file"
echo '=== outline ==='
ast-grep outline "$file" --view expanded | sed -n '1,220p'
echo '=== around reported lines ==='
sed -n '820,900p' "$file"Repository: IntegerAlex/floure-core
Length of output: 5306
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '840,890p' stt-ui/src/App.tsxRepository: IntegerAlex/floure-core
Length of output: 2156
Ignore failures from stale spawn attempts. Add the same spawnedRef.current !== api early return in the catch block so an outdated spawn failure doesn’t show a toast/error after a newer engine has already taken over.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt-ui/src/App.tsx` around lines 869 - 875, Update the catch handler for the
engine spawn promise to immediately return when spawnedRef.current !== api,
before constructing the message or calling setToast/addError. Preserve the
existing cleanup for the current API instance so stale spawn failures produce no
user-facing errors.
| // Idle → Listening | ||
| setPtt("listening"); | ||
|
|
||
| // Tell overlay to show in listening mode | ||
| try { | ||
| const { emitTo } = await import("@tauri-apps/api/event"); | ||
| await emitTo("overlay", "overlay:command", "listening" as const); | ||
| console.log("[Overlay] Command: listening"); | ||
| } catch { /* not in Tauri */ } | ||
|
|
||
| // Capture foreground window | ||
| try { | ||
| const { invoke } = await import("@tauri-apps/api/core"); | ||
| const hwnd = await invoke<number>("get_foreground_hwnd"); | ||
| pttHwndRef.current = hwnd; | ||
| console.log(`[PTT] Captured HWND: ${hwnd} (source=${source})`); | ||
| } catch { | ||
| pttHwndRef.current = null; | ||
| } | ||
|
|
||
| pttTextRef.current = ""; | ||
| console.log(`[PTT] Start requested — source=${source}`); | ||
| runtimeRef.current.start(); // Sends start_recording to backend | ||
| sessionCounter.current++; | ||
| nextLocalId.current = 1; | ||
| setLines([]); | ||
| playStartBeep(); | ||
|
|
||
| // Show overlay (positioned above taskbar) | ||
| try { | ||
| const { invoke } = await import("@tauri-apps/api/core"); | ||
| await invoke("show_overlay"); | ||
| console.log("[Overlay] Shown"); | ||
| } catch { /* not in Tauri */ } | ||
|
|
||
| // Tell backend to begin capturing | ||
| runtimeRef.current.start(); | ||
| setConnected(true); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
Send start_recording before yielding control.
The state becomes "listening" before several awaits, but Line 1178 sends the backend start afterward. A quick release can consequently send stop_recording first; the backend then clears that stale stop when the delayed start arrives and records without another matching stop.
Ensure there is no asynchronous gap between entering "listening" and sending start_recording, and guard the remaining overlay/HWND work with the session ID.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt-ui/src/App.tsx` around lines 1145 - 1179, Update the PTT start flow
around the listening-state transition to create or advance the session ID, set
the state, and invoke runtimeRef.current.start() immediately without any await
between them. Capture that session ID and guard the subsequent overlay commands
and foreground-HWND updates so stale asynchronous work cannot affect a newer
session; preserve the existing setup and cleanup behavior.
| // Tell overlay to clean up, wait for acknowledgement, then hide | ||
| overlayCleaningRef.current = true; | ||
| try { | ||
| const { emitTo } = await import("@tauri-apps/api/event"); | ||
| await emitTo("overlay", "overlay:command", "idle" as const); | ||
| } catch { /* not in Tauri */ } | ||
| await new Promise<void>((resolve) => { | ||
| overlayIdleResolveRef.current = resolve; | ||
| hideOverlayTimeoutRef.current = setTimeout(() => { | ||
| console.warn("[Overlay] idle_ready timeout — hiding anyway"); | ||
| resolve(); | ||
| }, 2000); | ||
| }); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Install the idle acknowledgement resolver before emitting "idle".
The overlay can return overlay:idle_ready before overlayIdleResolveRef is assigned, losing the acknowledgement and forcing the two-second fallback.
stt-ui/src/App.tsx#L1215-L1227: create the acknowledgement promise before the no-commit"idle"emission.stt-ui/src/App.tsx#L1341-L1353: do the same during success cleanup.
📍 Affects 1 file
stt-ui/src/App.tsx#L1215-L1227(this comment)stt-ui/src/App.tsx#L1341-L1353
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt-ui/src/App.tsx` around lines 1215 - 1227, In both overlay cleanup sites
in stt-ui/src/App.tsx (anchor lines 1215-1227 and sibling lines 1341-1353),
initialize the acknowledgement promise and assign overlayIdleResolveRef before
emitting the "idle" command; retain the existing timeout fallback and cleanup
behavior so early overlay:idle_ready events resolve immediately.
| if (text && hwnd) { | ||
| // Processing → Inserting | ||
| setPtt("inserting"); | ||
| console.log("[PTT] Inserting text"); | ||
| // Tell overlay to show inserting state | ||
| try { | ||
| const { emitTo } = await import("@tauri-apps/api/event"); | ||
| await emitTo("overlay", "overlay:command", "inserting" as const); | ||
| } catch { /* not in Tauri */ } | ||
| try { | ||
| const { invoke } = await import("@tauri-apps/api/core"); | ||
| const ok = await invoke<boolean>("insert_text", { text, restoreHwnd: hwnd }); | ||
| console.log(`[PTT] Inserted text to HWND ${hwnd} (source=${source}):`, ok); | ||
| if (!ok) setToast("Failed to insert text"); | ||
| } catch (e) { | ||
| console.error("[PTT] insert_text failed:", e); | ||
| setToast("Failed to insert text"); | ||
| } | ||
| } else { | ||
| console.log(`[PTT] Nothing to commit (source=${source})`); | ||
| } | ||
|
|
||
| // Inserting → Success (brief confirmation) | ||
| setPtt("success"); | ||
| setStatus("idle"); | ||
| console.log("[PTT] Success — showing confirmation"); | ||
|
|
||
| // Tell overlay to show success state | ||
| try { | ||
| const { emitTo } = await import("@tauri-apps/api/event"); | ||
| await emitTo("overlay", "overlay:command", "success" as const); | ||
| } catch { /* not in Tauri */ } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not report success when insertion fails or nothing is committed.
ok === false, invocation errors, missing text, and missing HWND all continue into the "success" state and show “Done”. Only emit the success state after a confirmed insertion; otherwise show an error/no-result state and perform normal cleanup.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt-ui/src/App.tsx` around lines 1296 - 1327, Update the PTT commit flow
surrounding the insertion logic so success is emitted only after text and hwnd
are present, insert_text completes without throwing, and returns ok === true.
Track unsuccessful outcomes from the ok === false, exception, missing-text, and
missing-HWND branches, then skip setPtt("success"), the success overlay event,
and “Done” confirmation for those cases while showing the existing
error/no-result state and completing normal cleanup.
| /** Set the current value directly (no animation). */ | ||
| setValue(value: number) { | ||
| state.value = value; | ||
| state.velocity = 0; | ||
| onUpdate?.(value); | ||
| }, |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Make setValue() stop the current animation.
An active RAF continues toward the old target after this “no animation” update, which can undo the overlay reset during a quick PTT session.
Proposed fix
setValue(value: number) {
+ if (rafId !== null) cancelAnimationFrame(rafId);
+ rafId = null;
+ lastTime = null;
state.value = value;
+ state.target = value;
state.velocity = 0;
onUpdate?.(value);
},📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| /** Set the current value directly (no animation). */ | |
| setValue(value: number) { | |
| state.value = value; | |
| state.velocity = 0; | |
| onUpdate?.(value); | |
| }, | |
| /** Set the current value directly (no animation). */ | |
| setValue(value: number) { | |
| if (rafId !== null) cancelAnimationFrame(rafId); | |
| rafId = null; | |
| lastTime = null; | |
| state.value = value; | |
| state.target = value; | |
| state.velocity = 0; | |
| onUpdate?.(value); | |
| }, |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt-ui/src/overlay/spring.ts` around lines 88 - 93, Update setValue() in the
spring controller to cancel or invalidate any active RAF animation before
assigning the new value. Ensure the current animation cannot continue updating
state or invoke onUpdate after setValue() resets the value and velocity.
| async start() { | ||
| if (state.active) return; | ||
|
|
||
| try { | ||
| // Create AudioContext | ||
| state.audioContext = new AudioContext(); | ||
| state.analyser = state.audioContext.createAnalyser(); | ||
| state.analyser.fftSize = fftSize; | ||
| state.analyser.minDecibels = minDecibels; | ||
| state.analyser.maxDecibels = maxDecibels; | ||
| state.analyser.smoothingTimeConstant = smoothingTimeConstant; | ||
|
|
||
| // Get microphone stream | ||
| state.stream = await navigator.mediaDevices.getUserMedia({ audio: true }); | ||
|
|
||
| // Connect: mic → analyser | ||
| const source = state.audioContext.createMediaStreamSource(state.stream); | ||
| source.connect(state.analyser); | ||
|
|
||
| // Allocate buffers | ||
| state.dataArray = new Uint8Array(state.analyser.frequencyBinCount); | ||
| state.smoothed = new Float32Array(barCount); | ||
|
|
||
| state.active = true; | ||
| state.rafId = requestAnimationFrame(tick); | ||
| } catch (err) { | ||
| console.error("[waveform] Failed to start:", err); | ||
| // Fallback: start fake waveform | ||
| state.active = true; | ||
| state.smoothed = new Float32Array(barCount); | ||
| state.rafId = requestAnimationFrame(tick); | ||
| } | ||
| }, | ||
|
|
||
| /** Stop capturing audio. */ | ||
| stop() { | ||
| state.active = false; | ||
| if (state.rafId !== null) { | ||
| cancelAnimationFrame(state.rafId); | ||
| state.rafId = null; | ||
| } | ||
| if (state.stream) { | ||
| state.stream.getTracks().forEach((t) => t.stop()); | ||
| state.stream = null; | ||
| } | ||
| if (state.audioContext) { | ||
| state.audioContext.close().catch(() => {}); | ||
| state.audioContext = null; | ||
| } | ||
| state.analyser = null; | ||
| state.dataArray = null; | ||
| state.smoothed = null; | ||
| }, |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Cancel pending microphone acquisition when stopping.
If stop() runs while getUserMedia() is pending, it sees no stream. A later resolution still sets active, starts the RAF, and can leave the microphone open after the overlay is hidden.
Use a generation/cancellation token, acquire resources into local variables, and immediately close them when the start attempt has been invalidated.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt-ui/src/overlay/waveform.ts` around lines 96 - 148, Update the waveform
start/stop lifecycle around start() and stop() to use a generation or
cancellation token that invalidates pending getUserMedia attempts. Acquire the
AudioContext, analyser, stream, and related resources in local variables, verify
the start attempt is still current after each await, and immediately stop tracks
and close the context when invalidated; only then assign shared state, set
active, and schedule the RAF. Increment the token in stop() before cleanup so a
later getUserMedia resolution cannot restart the waveform or leave the
microphone open.
| def _release(): | ||
| global _idle_release_timer | ||
| _idle_release_timer = None | ||
| try: | ||
| from stt.transcription import release_backend | ||
| release_backend() | ||
| logger.info("idle: released ASR model to free memory") | ||
| except Exception: | ||
| pass | ||
|
|
||
| try: | ||
| _idle_release_timer = threading.Timer(_IDLE_RELEASE_GRACE_SEC, _release) | ||
| _idle_release_timer.daemon = True | ||
| _idle_release_timer.start() |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Serialize ASR release, warm-up, and model use. Timer cancellation can race with cache clearing, allowing a new session to start while backend state is being released.
stt/orchestrator.py#L111-L124: make cancellation/release race-safe using a lifecycle lock or generation token.stt/transcription.py#L39-L58: use that lifecycle synchronization around cache and allocator cleanup.
📍 Affects 2 files
stt/orchestrator.py#L111-L124(this comment)stt/transcription.py#L39-L58
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt/orchestrator.py` around lines 111 - 124, Serialize ASR lifecycle
operations so release, warm-up, and model use cannot overlap. In
stt/orchestrator.py lines 111-124, protect timer cancellation and the _release
callback with a shared lifecycle lock or generation token, ensuring a canceled
or stale timer cannot release backend state after a new session begins. In
stt/transcription.py lines 39-58, use the same synchronization around cache
clearing and allocator cleanup; update the relevant release, warm-up, and
model-use paths while preserving existing behavior.
| # A new session is starting — keep the (possibly still warm) model. | ||
| _cancel_idle_release() | ||
|
|
||
| # If the model was released during the previous idle period, re-warm it | ||
| # in the background. The 1.5s calibration window covers most of the | ||
| # load time; if it isn't ready, the first transcription reloads cold. | ||
| try: | ||
| if _idle_release_timer is None: | ||
| _warmup_thread = threading.Thread( | ||
| target=_warmup_target, args=(config.transcription,), daemon=True | ||
| ) | ||
| _warmup_thread.start() |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
Track backend release state separately from the timer handle.
Line 740 always makes _idle_release_timer None, so Line 746 starts another warm-up on every PTT session—including concurrently with the initial warm-up. Use an explicit _backend_released state instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt/orchestrator.py` around lines 739 - 750, The session-start warm-up logic
currently relies on _idle_release_timer, which is cleared by
_cancel_idle_release and therefore cannot indicate whether the backend was
released. Add and maintain an explicit _backend_released state, set it when idle
release completes and clear it when the backend is retained or successfully
re-warmed; update the condition around _warmup_target to start warm-up only when
that state is true, preventing duplicate concurrent warm-ups.
| if chunk_count % 8 == 0: | ||
| st, et = detector.thresholds() | ||
| noise = detector.noise_floor | ||
| snr_db = 20 * np.log10(rms / max(noise, 1e-10)) if noise > 0 else 0 | ||
| state = detector.vad_state.name | ||
| _debug(config, f"rms={rms:.6f} noise={noise:.4f} snr={snr_db:.1f}dB state={state}") | ||
| in_speech = detector._in_speech | ||
| logger.info(f"VAD: rms={rms:.6f} noise={noise:.4f} snr={snr_db:.1f}dB state={state} in_speech={in_speech} onset_th={st:.4f}") | ||
| if config.vad.use_spectral_vad: | ||
| score = detector._compute_speech_score(chunk) | ||
| _debug(config, f" spectral: score={score:.4f}") | ||
| logger.info(f" spectral: score={score:.4f}") |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not mutate spectral VAD state while logging.
_compute_speech_score() updates _prev_spectrum; detector.update() then recomputes the same chunk, making spectral flux zero on every eighth frame and changing the actual VAD decision.
Minimal fix
- if config.vad.use_spectral_vad:
- score = detector._compute_speech_score(chunk)
- logger.info(f" spectral: score={score:.4f}")Log a score captured inside update() instead if needed.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if chunk_count % 8 == 0: | |
| st, et = detector.thresholds() | |
| noise = detector.noise_floor | |
| snr_db = 20 * np.log10(rms / max(noise, 1e-10)) if noise > 0 else 0 | |
| state = detector.vad_state.name | |
| _debug(config, f"rms={rms:.6f} noise={noise:.4f} snr={snr_db:.1f}dB state={state}") | |
| in_speech = detector._in_speech | |
| logger.info(f"VAD: rms={rms:.6f} noise={noise:.4f} snr={snr_db:.1f}dB state={state} in_speech={in_speech} onset_th={st:.4f}") | |
| if config.vad.use_spectral_vad: | |
| score = detector._compute_speech_score(chunk) | |
| _debug(config, f" spectral: score={score:.4f}") | |
| logger.info(f" spectral: score={score:.4f}") | |
| if chunk_count % 8 == 0: | |
| st, et = detector.thresholds() | |
| noise = detector.noise_floor | |
| snr_db = 20 * np.log10(rms / max(noise, 1e-10)) if noise > 0 else 0 | |
| state = detector.vad_state.name | |
| in_speech = detector._in_speech | |
| logger.info(f"VAD: rms={rms:.6f} noise={noise:.4f} snr={snr_db:.1f}dB state={state} in_speech={in_speech} onset_th={st:.4f}") |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@stt/orchestrator.py` around lines 949 - 958, Remove the logging-time call to
detector._compute_speech_score(chunk) in the VAD diagnostics block, since it
mutates spectral state before detector.update() processes the chunk. If the
score must remain logged, capture and expose it from detector.update() and log
that already-computed value without recomputing the chunk.
Long sentences split into multiple utterances by the VAD were only committing the last segment because pttTextRef was overwritten on each new utterance. Changed to a Map<utterance_id, text> that accumulates all segments and joins them at commit time. Also fixed processed/llm_partial line-matching: they used event.utterance_id directly as the line id, but raw events create lines with sessionCounter*100000+utterance_id, so processed events never matched and feed lines were never marked done.
| pttIdleResolveRef.current = null; | ||
|
|
||
| // Join all utterances' text in order for the PTT commit | ||
| const text = Array.from(pttPartsRef.current.values()).join(" ").trim(); |
There was a problem hiding this comment.
🟡 Medium src/App.tsx:1298
The commit text is joined from pttPartsRef.current.values(), which yields entries in Map insertion order — i.e. the order raw/processed events arrive — not in utterance_id order. Because the backend dispatches each utterance on a separate transcription thread, a later utterance can emit its first event before an earlier one and be inserted into the map first, so multi-utterance dictation can be pasted with phrases in the wrong order. Sort the map entries by numeric utterance_id before joining.
- const text = Array.from(pttPartsRef.current.values()).join(" ").trim();
+ const text = Array.from(pttPartsRef.current.entries())
+ .sort(([a], [b]) => a - b)
+ .map(([, v]) => v)
+ .join(" ").trim();🤖 Copy this AI Prompt to have your agent fix this:
In file @stt-ui/src/App.tsx around line 1298:
The commit text is joined from `pttPartsRef.current.values()`, which yields entries in `Map` insertion order — i.e. the order `raw`/`processed` events arrive — not in `utterance_id` order. Because the backend dispatches each utterance on a separate transcription thread, a later utterance can emit its first event before an earlier one and be inserted into the map first, so multi-utterance dictation can be pasted with phrases in the wrong order. Sort the map entries by numeric `utterance_id` before joining.
| // reads stale text before transcription completes. | ||
| // Backend emits idle immediately after recording loop ends (no thread join). | ||
| // Transcription runs in background and emits events through the pipeline. | ||
| const IDLE_TIMEOUT_MS = 20000; |
There was a problem hiding this comment.
🟡 Medium src/App.tsx:1266
stopAndCommit uses IDLE_TIMEOUT_MS = 20000 (20s), but the backend waits up to 180s for transcription threads to finish before emitting the idle state. When transcription takes longer than 20s, the frontend's idle timeout fires first, commits whatever partial text is in pttPartsRef, reports success, and returns to idle — while the backend is still processing. The committed text is empty or incomplete, and the user sees a false success indication. Either raise IDLE_TIMEOUT_MS above the backend's actual bound (180s), or on timeout, abort the commit rather than inserting stale data.
🤖 Copy this AI Prompt to have your agent fix this:
In file @stt-ui/src/App.tsx around line 1266:
`stopAndCommit` uses `IDLE_TIMEOUT_MS = 20000` (20s), but the backend waits up to 180s for transcription threads to finish before emitting the `idle` state. When transcription takes longer than 20s, the frontend's idle timeout fires first, commits whatever partial text is in `pttPartsRef`, reports success, and returns to idle — while the backend is still processing. The committed text is empty or incomplete, and the user sees a false success indication. Either raise `IDLE_TIMEOUT_MS` above the backend's actual bound (180s), or on timeout, abort the commit rather than inserting stale data.
1 participant
Stable version for window
Note
Add Windows-compatible overlay window and cross-platform text insertion for PTT workflow
overlay) that displays a pill UI driven byoverlay:commandevents, cycling through listening, processing, inserting, and success states with mic waveform animation and spring-based transitions.insert_textTauri command that pastes or types transcribed text into the previously focused window on Windows (clipboard + SendInput Unicode fallback), Linux (wtype/Ctrl+V), and macOS (AppleScript).PttOverlayReact component with the dedicated overlay window;App.tsxnow coordinates a full PTT state machine that waits for backend idle before committing, then drives overlay states and awaits cleanup acknowledgement.api-tauri.tsto prevent JSON event corruption from interleaved streams.force_endtoStreamingEndpointDetectorand updates the orchestrator to flush in-progress speech on PTT stop, wait for transcription threads, and schedule ASR model release after idle.Macroscope summarized 9963f29.