Surface + document the loopback Local API for on-device agents (Settings toggle + docs) - #715
Conversation
The loopback-only Local API (POST /v1/transcribe, /v1/postprocess) already exists but was enable-only via a hidden "LocalAPIEnabled" default and was undocumented. This surfaces and documents it: - SettingsStore.localAPIEnabled (reuses the existing "LocalAPIEnabled" key) - LocalAPIServer.refresh() to start/stop the server live when toggled - a "Local API (on-device agents)" toggle in Settings (loopback-only, off by default) - README section + docs/local-api.md documenting endpoints & security Lets other apps/agents on the same Mac reuse FluidVoice's transcription instead of loading their own ASR model. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
github-actions
Bot
added
needs PR template
|
The PR Policy check is blocking this PR because required template information is missing. Please update the PR description with:
Visual files detected:
Screenshots or video are required for UI, UX, settings, onboarding, overlay, menu bar, or visual behavior changes. If this PR has no visual changes, check the no-visual-change box in the template. If this remains incomplete for 48 hours after opening, the PR may be closed. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 66dbd621be
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
|
||
| self.optionToggleRow( | ||
| title: "Local API (on-device agents)", | ||
| description: "Expose a loopback-only HTTP API on 127.0.0.1 (default port 47733) so local agents/apps on this Mac can reuse FluidVoice's transcription (POST /v1/transcribe). Off by default; never leaves this machine.", |
There was a problem hiding this comment.
When the configured post-processing provider is OpenAI, Groq, Anthropic, or another remote provider, a caller of the enabled /v1/postprocess endpoint has its text sent externally: InferenceAPIController.postprocess delegates to DictationPostProcessingService.process, which calls LLMClient with the selected provider URL. Therefore the unqualified “never leaves this machine” promise is false for one of the documented endpoints and may cause users to expose agent-supplied text under an incorrect privacy assumption; clarify that only the HTTP listener is local and that post-processing follows the configured provider.
Useful? React with 👍 / 👎.
|
|
||
| FluidVoice can expose a **loopback-only** HTTP API so other apps and AI agents running **on the same Mac** can reuse its transcription and post-processing — instead of bundling and loading their own ASR model. One model stays warm in FluidVoice; agents get FluidVoice-quality transcription with no extra memory footprint. | ||
|
|
||
| **Enable it:** `Settings → Local API (on-device agents)`. It is **off by default** and only ever binds to `127.0.0.1` (non-loopback connections are rejected at the listener). |
There was a problem hiding this comment.
The server currently creates NWListener(using:on:) with only a port and never sets a loopback requiredLocalEndpoint, so it listens on the wildcard local address; the newConnectionHandler merely rejects non-loopback peers after their connections have reached the listener. Consequently the documented guarantee that it “only ever binds to 127.0.0.1” is inaccurate and the port remains externally visible/reachable at the transport layer. Either bind the listener explicitly to loopback or describe the accept-time filtering accurately.
Useful? React with 👍 / 👎.
| func refresh() { | ||
| if LocalAPI.Configuration.current.enabled { | ||
| self.start() | ||
| } else { | ||
| self.stop() | ||
| } |
There was a problem hiding this comment.
If the toggle is switched off and then back on before the cancelled listener's asynchronous state callback reaches the main actor, start() installs a new listener but the old listener's .cancelled callback subsequently executes handleState and clears self.listener. The restarted listener is then orphaned, so a later off toggle cannot cancel it and another on toggle may attempt to open a duplicate port. Associate state callbacks with their originating listener and only clear the property when they still match.
Useful? React with 👍 / 👎.
Greptile SummarySurfaces the existing loopback Local API and documents its behavior.
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
…o-cap docs) - LocalAPIServer: ignore state callbacks from a superseded listener (guard self.listener === listener, weak capture) so a rapid off->on toggle can no longer orphan/cancel the newly-installed listener (Codex P2 / Greptile P1). - Qualify the privacy claim: only the HTTP listener and /v1/transcribe are on-device; /v1/postprocess follows the configured provider (may be remote) (toggle text, README, docs). - Describe the accept-time loopback filtering accurately instead of claiming it binds 127.0.0.1. - Document the 300s (5 min) audio-duration cap alongside the 25 MB body cap. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
7d418cb to
0e68a5d
Compare
|
Thanks for the thorough review — all four points addressed in
Rebuilt clean ( |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0e68a5d532
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| } else { | ||
| self.stop() |
There was a problem hiding this comment.
When a connection has passed newConnectionHandler but its Task { @MainActor ... } has not executed yet, this call to stop() cannot see it in activeConnections. The queued task then creates and starts the handler without checking the enabled setting or its originating listener, allowing that connection to continue serving requests after the user switches the API off. Associate connection callbacks with the listener generation and cancel them when that listener is no longer current.
Useful? React with 👍 / 👎.
…ed section The toggle sat next to 'Copy to Clipboard', which lives inside the `if accessibilityEnabled` block, so it was hidden until the user granted Accessibility — unrelated to a loopback HTTP API. Moved it to the always-visible 'App Settings' section, and split the long description into description + footnote so it no longer truncates on one line. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Follow-up in |
settingsToggleRow renders description and footnote on single truncating lines; the previous footnote overflowed. Trimmed it to the essentials — the full contract (non-loopback filtering, limits, post-processing locality) lives in README + docs/local-api.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Adding a screenshot of the Toggle 
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e88f57419b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| You can also toggle it via `defaults`: | ||
|
|
||
| ```bash | ||
| defaults write com.FluidApp.app LocalAPIEnabled -bool true |
There was a problem hiding this comment.
When FluidVoice is already running, this command only changes the preference: LocalAPIServer.refresh() is invoked exclusively by the Settings toggle, and the lifecycle calls start() only at launch. Consequently, enabling via defaults does not start the listener, and disabling via defaults leaves the API serving until the app is relaunched. Either observe these preference changes or tell command-line users to restart FluidVoice.
Useful? React with 👍 / 👎.
| - Limits: request body **25 MB**; transcription audio **300 s (5 min)** per request (longer audio is | ||
| rejected). These caps are independent — compressed audio can satisfy the byte limit yet still hit the | ||
| duration limit. |
There was a problem hiding this comment.
For the documented audioBase64 input (and raw audio bodies), decodeAudioSamples reaches LocalAPIAudioDecoder.samples(from:), which reads min(file.length, maxFrames) and silently transcribes only the first five minutes. Only the path-based flow calls validateDurationWithinLimit and rejects overlong files. Clients relying on this statement can therefore receive a successful but incomplete transcript; either validate inline inputs too or document that they are truncated.
Useful? React with 👍 / 👎.
- LocalAPIServer: the deferred @mainactor task now also checks the server is still running (listener != nil) before creating a handler, so a connection accepted just before stop()/toggle-off is dropped instead of handled. - docs: note that enabling via `defaults` needs an app relaunch (the Settings toggle applies live via refresh()); clarify the 300s audio cap rejects file-path input but truncates inline audioBase64 input. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
34c108d to
cec22e7
Compare
|
Latest round (
Also filled the PR template + added the toggle screenshot to the description. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cec22e7eb0
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| title: "Local API (on-device agents)", | ||
| description: "Loopback-only HTTP API for on-device agents to reuse transcription (POST /v1/transcribe).", | ||
| footnote: "127.0.0.1:47733, off by default. /v1/postprocess may use a remote AI provider.", |
There was a problem hiding this comment.
When this toggle is enabled on a multi-user Mac, loopback does not identify the current login user, so any process in another local user session can call the unauthenticated routes. In particular, LocalAPIRouter exposes GET /v1/history, and HistoryAPIController returns raw and processed dictation text plus app and window metadata; the dictionary routes are also writable. Surfacing the API as a transcription toggle without authentication therefore exposes substantially more private and mutable state than the user is told they are enabling.
Useful? React with 👍 / 👎.
| self.settingsToggleRow( | ||
| title: "Local API (on-device agents)", | ||
| description: "Loopback-only HTTP API for on-device agents to reuse transcription (POST /v1/transcribe).", | ||
| footnote: "127.0.0.1:47733, off by default. /v1/postprocess may use a remote AI provider.", |
There was a problem hiding this comment.
When a user follows the newly documented LocalAPIPort configuration and chooses anything other than 47733, this footnote still presents 127.0.0.1:47733 as the active endpoint even though LocalAPIServer.start() reads and listens on the configured port. Agents configured from the Settings value will consequently connect to the wrong endpoint; label this as the default or render the effective port.
Useful? React with 👍 / 👎.
github-actions
Bot
removed
needs PR template
1 participant
Description
FluidVoice already ships a loopback-only Local API (
LocalAPIServer:POST /v1/transcribe,POST /v1/postprocess,GET /v1/health, plus history/dictionary), but it can only be enabled through the hiddenLocalAPIEnableddefault and is undocumented. This PR surfaces and documents it so other apps and AI agents running on the same Mac can reuse FluidVoice's transcription instead of bundling and loading their own ASR model.SettingsStore.localAPIEnabled— reuses the existing"LocalAPIEnabled"key (no change to how the server reads config).LocalAPIServer.refresh()— starts/stops the listener live when the setting changes (no app restart).docs/local-api.md(endpoints, request/response, security, limits, examples).Type of Change
Related Issue or Discussion
Related to #529 (Companion CLI for Local API — adoption discussion) and #414 (Extend the LocalAPI to read/set the speech model…). This PR surfaces the existing Local API via a Settings toggle and documents it (endpoints, security, limits), which directly supports the external/agent adoption those threads discuss. No dedicated issue was opened for the toggle+docs itself — happy to file one if you'd prefer.
Testing
swiftlint --strict --config .swiftlint.yml Sourcesswiftformat --config .swiftformat Sourcesxcodebuild -scheme Fluid -configuration Debug build→ BUILD SUCCEEDED. Manual checks: toggle on →GET /v1/health200 on127.0.0.1:47733;POST /v1/transcribe {"path": …}returns the transcript reusing the already-warm model; toggle off → listener stops; rapid off→on handled cleanly; non-loopback peer refused.Screenshots / Video
Notes
Addressed the automated review feedback:
stateUpdateHandlernow guardsself.listener === listener(weak capture) so a rapid off→on toggle can't orphan or cancel the freshly-installed listener; and connections accepted just beforestop()are now dropped instead of handled after disable./v1/transcribeare on-device;/v1/postprocessfollows the configured provider (may be remote). Documented the 300 s audio cap (file-pathinput rejected; inlineaudioBase64truncated) and that enabling viadefaultsneeds a relaunch, whereas the Settings toggle applies live.