docs(desktop): plan user-configurable inference providers

[mrlfarano]

Summary

This PR proposes the architecture and implementation plan for allowing Omi for Mac users to configure their own OpenAI-compatible inference providers and choose models independently for three workload levels:

• Fast
• Standard
• Reasoning & Agents

It includes first-class compatibility presets for Z.AI, Kimi/Moonshot, and DeepSeek, plus a generic OpenAI-compatible provider for other endpoints.

This is a planning-only PR. It does not change application behavior, provider defaults, credentials, backend traffic, or production dependencies.

Related issue: none. This planning PR is the direction-alignment artifact requested before implementation begins.

Why

Desktop inference choices are currently coupled to provider-owned model constants and agent-runtime selection. That prevents a user from keeping Omi AI as the runtime while selecting a different compatible inference endpoint or model for specific classes of work.

The proposed design separates:

1. Agent Runtime — Omi AI/Pi, Claude, Hermes, or OpenClaw.
2. Inference Route — managed Omi inference or an exact custom provider revision and model.
3. Workload Policy — Fast, Standard, Reasoning & Agents, with optional per-workload overrides.

Proposed behavior

• Omi-managed inference remains the default at every level.
• Users can add a provider connection with preset, base URL, model ID, API key, and declared capabilities.
• Provider metadata remains local; API keys are stored in revision-scoped Keychain items.
• Custom requests go directly from the Mac to the selected provider. URLs and keys are not relayed through the Omi backend.
• Custom-route failures are visible and terminal; they never silently fall back to managed Omi inference or another provider.
• Existing sessions retain their immutable inference route. Settings changes apply to future sessions unless the user explicitly migrates current Omi AI chats.
• Main, floating, task, onboarding, delegated, higher-model, and memory-export agent paths receive explicit workload identities.
• Embeddings, STT, realtime voice, TTS, image generation, offline backend jobs, and runtime-owned Claude/Hermes/OpenClaw inference remain outside this feature.
• The existing four-key BYOK/free-plan contract remains unchanged and is presented separately in Settings.

Architecture highlights

• A typed Swift configuration authority owns provider revisions, level assignments, workload overrides, capability admission, and Keychain references.
• A provider-neutral OpenAI-compatible transport normalizes streaming, tool calls, structured output, reasoning content, cancellation, and sanitized failures.
• Generic, Z.AI, Kimi, and DeepSeek compatibility logic stays in isolated preset adapters rather than workload call sites.
• The Swift/Node kernel contract gains an independent InferenceRouteRef in immutable execution profiles.
• The bundled Pi runtime receives bounded non-secret provider metadata and separately indexed secret environment values, then dynamically registers the exact provider/model.
• Endpoint policy allows HTTPS and HTTP loopback only, rejects embedded credentials and fragments, and blocks host-changing or TLS-downgrade redirects.
• A permanent hermetic loopback provider fixture covers Settings, streaming, tools, structured output, errors, cancellation, and secret-redaction behavior.

Implementation sequence

The plan divides delivery into nine reviewable commits/tasks:

1. Revisioned configuration, Keychain storage, and endpoint admission.
2. OpenAI-compatible transport and provider presets.
3. Exhaustive workload catalog and route resolver.
4. Migration of direct Swift inference workloads.
5. Immutable kernel execution-profile contract and SQLite migration 33.
6. Dynamic custom-provider registration in the bundled Pi runtime.
7. Main/floating/task/onboarding/delegated agent surface wiring and ModelQoS removal.
8. Settings UI, connection probes, automation, and hermetic E2E coverage.
9. Durable documentation, named-bundle verification, provider smoke testing, preflight, and independent review.

Codex worker allocation

The implementation plan intentionally avoids using the highest-cost model for routine work:

Work	Model	Reasoning
Configuration, routing policy, Swift consumer migration, Settings, docs	gpt-5.6-terra	medium
Transport normalization, kernel/SQLite contract, Pi provider registration, cross-surface wiring	gpt-5.6-sol	medium
Final security/privacy and cross-surface audit only	gpt-5.6-sol	high

This results in five Terra-medium implementation tasks, four Sol-medium implementation tasks, and one final Sol-high review.

Requested review

The Omi team should focus on these decisions before implementation begins:

1. Is direct Mac-to-provider traffic the correct privacy and ownership boundary, versus adding a backend proxy for arbitrary endpoints?
2. Are Fast, Standard, and Reasoning & Agents the right user-facing levels and workload mappings?
3. Should custom inference remain exclusive to Omi AI/Pi while Claude, Hermes, and OpenClaw retain runtime-owned configuration?
4. Is revision-scoped Keychain storage plus immutable session pinning the right rotation and continuity model?
5. Are the HTTPS/loopback and same-host redirect restrictions sufficiently strict without blocking expected compatible providers?
6. Does the proposed generic capability declaration/probe model cover providers beyond the initial Z.AI, Kimi, and DeepSeek presets?
7. Is the explicit no-fallback policy correct for cost, privacy, and behavioral predictability?
8. Is SQLite migration version 33 acceptable for the independent route field, or should route metadata live in a different kernel-owned shape?

Documents

• docs/superpowers/specs/2026-08-08-desktop-custom-inference-providers-design.md
• docs/superpowers/plans/2026-08-08-desktop-custom-inference-providers.md

The design includes the official Z.AI, Kimi, and DeepSeek protocol references used to define the initial compatibility presets.

Validation

• make preflight — PASS, 10 selected checks
• scripts/pr-preflight --suggest — no affected locked product invariants; no failure-class declaration required
• Plan structure audit — PASS: 9 tasks, 9 explicit implementation-worker assignments, balanced Markdown fences, and exactly one Sol-high review assignment
• git diff --check origin/main...HEAD — PASS

No desktop build or live provider test was run because this PR changes documentation only. The implementation plan defines the required hermetic, named-bundle, prompt-gauntlet, and credentialed-provider verification gates for the future implementation PR.

Backend Hermetic Merge Gate

PASS (out of scope): this diff changes two planning documents only and does not touch backend source, lockfiles, API contracts, runtime configuration, or backend gate workflows.

Impact

Developer impact is limited to reviewing and agreeing on the proposed contract and execution sequence. There is no user-visible or runtime impact from merging these documents.

[Git-on-my-level]

Thanks for this thorough planning artifact, @mrlfarano — this is a well-structured direction-alignment proposal.

Both changed files are documentation-only (no code, dependencies, or CI changes), so I'll focus on the design and plan quality.

docs/superpowers/specs/2026-08-08-desktop-custom-inference-providers-design.md — The design is thoughtful. Separating Agent Runtime, Inference Route, and Workload Policy as three distinct concepts (rather than overloading adapterId or providerBoundary) is the right call, and the rejected-alternative analysis (backend proxy, hardcoded providers) is well-reasoned. The scope boundary correctly excludes embeddings, STT, realtime, and TTS. The failure/fallback policy (terminal custom-route failures, no silent cross-back to managed inference, provider 401 never invalidating Firebase auth) addresses the key safety properties. The security properties described throughout (Keychain-scoped revision keys, HTTPS + loopback-only endpoints, redirect host/scheme revalidation, no raw response bodies in telemetry) are the right instincts for a feature handling user API keys.

docs/superpowers/plans/2026-08-08-desktop-custom-inference-providers.md — The 9-task plan follows a disciplined TDD approach (write tests, confirm red, implement, confirm green) with concrete type definitions and file paths. A few specific observations:

• The workload enum (Task 3) inventories 14 call sites mapped to Fast/Standard/Reasoning & Agents — the completeness guard that fails until every call site names a workload is a good structural invariant.
• The kernel execution-profile change (Task 5) adds inferenceRoute as a typed field with SQLite migration v33 and immutability enforcement — this matches the existing immutable-profile contract.
• The dynamic Pi provider registration (Task 6) correctly removes the hardcoded provider: "omi" selection and makes set_model provider-aware.

Agent-behavior note: This plan file is explicit agent-instruction documentation — it opens with "For agentic workers: REQUIRED SUB-SKILL..." and contains a full worker-allocation table assigning Codex models to each task. If merged, these become canonical guidance that AI coding agents in this repo may follow to implement a broad, security-sensitive feature across Swift production code, the Node agent runtime, SQLite migrations, and the Pi extension. That is worth human maintainer awareness before accepting.

What needs a human decision: This proposes a significant product direction — letting users configure arbitrary OpenAI-compatible inference providers and route models per workload level. That touches model/provider routing, API key handling, and the kernel execution profile, all of which need product and architecture sign-off from a maintainer. The design quality is strong, but whether Omi should build this and whether the proposed three-level abstraction matches the product vision is a judgment call I am deferring.

Leaving for human maintainer review on the product direction and architecture decision.

---

by AI on behalf of David — if you need David’s attention urgently, please @Git-on-my-level and escalate with need human response.