Scribe writing companion + Parakeet STT (core seams + engine)#32
Open
alichherawalla wants to merge 72 commits into
Open
Scribe writing companion + Parakeet STT (core seams + engine)#32alichherawalla wants to merge 72 commits into
alichherawalla wants to merge 72 commits into
Conversation
Generic extension points so pro can plug in Scribe without any pro logic in core. - sectionRegistry: SettingsSection gains title/summary; Settings.tsx now renders all registered pro sections (no hardcoded pro bodies) - slotRegistry: composerEditor slot + editorSlot.ts EditorSlotProps contract - MemoryChat: renders the registered composer editor (Scribe's AssistedTextarea) when present, else the plain textarea — free builds are unaffected Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Inert shell in core: a proCatalog entry (PenNib, outcome-first copy) + proItem('scribe')
nav item. Free builds show the locked upgrade screen; the pro build routes to the real
Scribe screen. Copy follows the brand voice — outcome first, privacy as mechanism.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drives the real built app (OFFGRID_PRO=1): navigates to Scribe, types a mix of a misspelling and a valid-but-uncommon word, and asserts via DOM enumeration that "recieve" is underlined while "photosynthesis"/"notes" are not. Screenshot attached. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
How the OSS engines work (reference-only, built our own), the two-engine architecture, the writing-companion vision, and the phased desktop plan. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends the Scribe E2E to run the review pass and assert the score + issue cards render (rule issues with no model). Screenshot attached. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ning loop, style guides
Phases 6-9:
- 6: import a file (txt/md/docx/pdf) -> rewrite/tone transform -> hand off to live edit
- 7: section-level free-text feedback from the rewrite toolbar
- 8: learning loop + a "What Scribe has learned" panel (split writing style into its
own store; the recordFeedback->distill->styleContext loop already runs)
- 9: style-guide .md files (e.g. brand_voice.md) as the authoritative baseline, above
learned prefs and RAG, with per-context scoping
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
E2E opens Settings, expands the Scribe section, and asserts the style-guide upload affordance + "What Scribe has learned" panel render. Screenshot attached. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nshot E2E opens the Import → transform menu (native picker can't be driven) and asserts its actions render. Squiggle screenshot refreshed after the alignment fix — underlines now sit exactly under "recieve" and "in order to". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MemoryChat listens for an og:chat-seed CustomEvent and prefills the composer, so Scribe's "Continue in chat" can hand a draft into a real chat thread. Generic seam; no Scribe/pro coupling in core. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The clipboard search placeholder became "Search content or tags…" (tag-search PR #19) but the E2E still expected the old copy. Not related to Scribe; just unbreaks the suite. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A second TranscriptionService for higher-accuracy on-device STT via NVIDIA Parakeet. Wraps a bundled sherpa-onnx offline-transducer CLI + ONNX model (encoder/decoder/ joiner/tokens), resolved from resources like whisper, decoding through the same ffmpeg 16 kHz path. Pure buildParakeetArgs + parseParakeetOutput (JSON or text: line, escape- safe) are unit-tested. isAvailable() gates on binary+model presence so callers can fall back to whisper until CI stages the runtime. Whisper stays the default; this is additive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
pickTranscription(engine, services) picks whisper (default) or Parakeet, honoring Parakeet only when installed and falling back to whisper otherwise, so a missing Parakeet runtime never breaks transcription. Pure + unit-tested; getTranscription() wires the real singletons. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
scripts/fetch-parakeet.sh stages the sherpa-onnx offline CLI + a Parakeet ONNX model into resources/bin/parakeet (bundled via extraResources, signed like the other engine binaries). Gated on SHERPA_ONNX_URL / PARAKEET_MODEL_URL repo vars — unset = no-op, so transcription stays on whisper and a release can't break. Includes the same foreign- dylib gate the llama engine uses. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds an `engine` field to ModelEntry and two Parakeet TDT 0.6B entries (v2 English, v3 multilingual) as kind:'transcription', engine:'parakeet'. Each is the 4-file sherpa-onnx transducer set (encoder/decoder/joiner/tokens) with slug-prefixed on-disk names so multiple Parakeet models coexist in the flat models dir. They appear in the Models screen and download through the existing multi-file download manager - no downloader change needed. Rebuilt dist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
parakeetModel() now resolves the active/downloaded Parakeet catalog model from the models dir (matchParakeetFiles maps encoder/decoder/joiner/tokens by name; pure + tested), falling back to the CI-bundled default. whisperModel()/smallWhisperModel() now only honor the shared active-transcription pick when it's a whisper ggml file, so a Parakeet ONNX selection can never be handed to whisper. +3 tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Opens Voice settings, switches the engine to Parakeet, and asserts the catalog models render with a Download action. Element screenshot of the picker attached. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…g copy whisperModel/smallWhisperModel now map the active-transcription slot (which stores a catalog id) to the entry's primary ggml filename, so picking a whisper model actually pins it (and Parakeet ids still resolve to null - never handed to whisper). Also drops em dashes from the two new Parakeet catalog descriptions per the brand rule. Rebuilt dist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…stid Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
parakeetModel() prefers the entry whose id is the active-transcription pick (the slot stores the id for transcription), not the primary filename. Screenshot refresh from the e2e run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fetch-parakeet.sh now preserves the sherpa-onnx prebuilt's bin/ + lib/ structure (so the binary's @rpath finds libonnxruntime + libsherpa-onnx dylibs), copies symlinks into real files (none in a signed .app), and gates like the llama engine: dependency-closure (every @rpath/<name> present under lib/), foreign-dep (/opt/homebrew,/usr/local) on the binary AND every dylib, and a minos log. parakeetBin() resolves bin/sherpa-onnx-offline. Still gated on SHERPA_ONNX_URL (no-op until pinned); exact layout/flags need on-Mac verification before it's called live. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a Scribe card to the onboarding Pro grid (PenNib, outcome-first copy, on-device mechanism). Also fixes fetch-parakeet.sh to create its staging dirs only on the real path, not on the skip no-op. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…n-Mac) Verified against the real sherpa-onnx-offline v1.13.3 --help on macOS arm64: - buildParakeetArgs drops the bogus --model-type (sherpa infers transducer from the encoder/decoder/joiner trio) and adds --decoding-method=greedy_search; wav stays last. - fetch-parakeet.sh: tar -xf (handles the .tar.bz2 release), model URL now optional (the in-app picker downloads models at runtime), and it stages ONLY sherpa-onnx-offline (the archive ships ~40 executables) - the static osx-arm64 build is self-contained (system libs only), so no dylibs to stage. Tests updated (13): assert --decoding-method + guard against --model-type. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Ran the staged sherpa-onnx-offline v1.13.3 on a real wav on macOS arm64 - it transcribed correctly (RTF ~0.06). Locks in parseParakeetOutput against the verbatim stdout (JSON with empty lang/emotion/event keys before "text"). Gitignores resources/bin/parakeet/ (CI-staged like mflux/coreml-sd, not committed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds Parakeet transcription support and engine routing, Scribe overlay tooling and renderer wiring, expanded Pro E2E coverage, and two Writing Assistant planning docs. It also updates release staging, model metadata, and settings formatting. ChangesParakeet STT Runtime and Transcription Engine
Scribe system-wide overlay tooling
Scribe composer editor and navigation wiring
Pro E2E coverage and local ignores
Writing Assistant design documents
Settings.tsx formatting cleanup
Estimated code review effort: 5 (Critical) | ~120 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant ParakeetCliTranscription
participant ffmpeg
participant sherpa-onnx-offline
Caller->>ParakeetCliTranscription: transcribe(path, opts)
ParakeetCliTranscription->>ffmpeg: convert input to 16kHz mono WAV
ParakeetCliTranscription->>sherpa-onnx-offline: execFile with CLI args
sherpa-onnx-offline-->>ParakeetCliTranscription: stdout text
ParakeetCliTranscription->>ParakeetCliTranscription: parseParakeetOutput
ParakeetCliTranscription-->>Caller: Transcript
sequenceDiagram
participant MemoryChat
participant getComposerEditor
participant slotRegistry
MemoryChat->>getComposerEditor: getComposerEditor()
getComposerEditor->>slotRegistry: lookup SLOTS.composerEditor
slotRegistry-->>getComposerEditor: component or undefined
getComposerEditor-->>MemoryChat: Editor or fallback
MemoryChat->>MemoryChat: render editor or textarea
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
PR Summary by QodoAdd Scribe extension seams and Parakeet STT engine with whisper fallback
AI Description
Diagram
High-Level Assessment
Files changed (55)
|
The card panel was at .floating while the overlay window is at .screenSaver, so the card could be occluded. Raise the panel above the overlay level. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replicates the TextWarden/Harper pattern: rather than trying to suppress the native spellchecker (not possible for third-party apps + browsers use their own), integrate with it — skip spelling squiggles for words the user taught macOS via Learn Spelling (NSSpellChecker.hasLearnedWord), so ours never contradict the OS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…+ OS guesses CRITICAL: the selection-trick (set caret to measure bounds) makes live typing in Electron/browser apps jumpy and corrupts input. Never move the caret while the user types — draw live squiggles only where bounds are non-invasive (native, direct AXBoundsForRange). Electron/browser are served by the on-demand hotkey flow instead (a non-invasive text-marker bounds method to restore live squiggles there is a follow-up). Also: correction card chips are now custom layer-drawn views (emerald fill + dark text) since NSButton ignores bezel/tint for rounded buttons; and spelling fixes use macOS NSSpellChecker guesses (e.g. gance -> glance) instead of the small bundled wordlist. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Card clicks did nothing: a non-activating panel swallows the first click, so add acceptsFirstMouse + a mouseDown override on the chip so mouseUp fires. And apply via select-then-paste (universal) when AXSelectedText doesn't take (Chromium returns success but doesn't replace). Recolor the card to @offgrid/design tokens (surface #141414 + border #2A2A2A + surface-light chips), not near-black. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Card now shows a category badge (SPELLING/GRAMMAR/CLARITY, emerald caps) above a wrapping message for more detail, plus a gear chip that emits an open-settings action. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hotkey over a selection shows a native rewrite menu at the selection (bounds read non-invasively from the live selection markers), emits the chosen action to the engine, shows a Rewriting card, then pastes the result over the selection. Works in every app (select+paste), so tone/rephrase/translate reach Slack/browser without touching the caret. Menu/busy cards are persistent (not hover-dismissed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every issue is now actionable: when there's no canned fix (passive voice, wordiness, clarity), the card shows an emerald primary action (Rephrase / Fix grammar / Make clearer) that selects the span, asks the model to rewrite it, shows a Rewriting card, then pastes the result. Brand the card with an emerald left accent stripe + design-token surface so it reads as Off Grid, not a system menu. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Card now mimics the app theme (light/dark palette from @offgrid/design tokens):
accepts {cmd:theme}, defaults from macOS AppleInterfaceStyle. No longer always dark.
- Fix Rephrase no-op: the result arrives seconds after the LLM call, by which time
the selection has collapsed, so paste inserted at the cursor instead of replacing.
Remember the target range (pendingRange) and re-select it before pasting (both the
hover AI-fix and the rewrite-menu paths).
- rewrite-cancel now shows a brief reason instead of vanishing silently.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Root cause of Rephrase doing nothing: style/clarity issues flag a fragment (e.g. 'was written'), and rewriting two words in isolation returns them unchanged, so the result was (correctly) rejected as no-change. Expand style/clarity AI-fixes to the enclosing sentence (sentenceRange), keeping grammar on the flagged span. Verified on-device: 'The report was written by the team.' -> 'The team wrote the report.' replaced in TextEdit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… nicer busy card - ROOT CAUSE of 'nothing in Slack': the overlay never woke Electron/Chromium AX trees, so their focused field read as nil. Set AXManualAccessibility (Electron) / AXEnhancedUserInterface (browser) once per app pid in focusedTextElement. - Re-enable live underlines in Slack/browser but ONLY after a real typing pause (>=1.2s idle) with a collapsed caret, measuring just the issue spans and restoring the caret - so squiggles appear shortly after you stop typing, never during it. - Busy card: compact emerald-dot + label instead of a big empty slab. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
npm run dev only bundles JS/TS - it never built the native Swift overlay, so main.swift edits silently didn't take effect without a manual build step. Add a predev hook that runs build-scribe-overlay.sh, now made incremental (skips when the binary is newer than the source), macOS/swiftc-gated, and non-fatal (a compile error warns but never blocks the app). So 'npm run dev' just works. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ation debt) Audit of the Scribe UI against ../brand/DESIGN_PHILOSOPHY.md plus open functional items from on-device testing. Tracks hardcoded-hex/light-mode breakage, the color-coding decision, a11y, Slack verification, and my own verification debt. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…merald suggestions) Match the renderer's G-2 decision - drop the rainbow for the two brand colors: error red for spelling/grammar/punctuation, emerald for style/clarity/tone/ word-choice. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…phrase pill Restores the select-to-rewrite entry point system-wide (lost when the hotkey was removed for the always-on/hover model). When you select a passage (>=12 chars) in any Scribe-active app, a small emerald 'Rephrase with Scribe' pill floats near the cursor; clicking it opens the full rewrite menu (Improve/Shorten/Tone/Translate/…). Own panel (doesn't fight the hover card); non-invasive selection read; gated to allowed+active apps via the existing underline/clear signal; waits for the drag to finish so it doesn't flicker. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The affordance floated at the cursor (far from the text). Anchor it to the selection's bounds (selectionBoundsCocoa) so it sits just below the selected line - same placement mechanism as the squiggle hover cards. Falls back to the cursor only when selection bounds aren't available (Chromium). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The existing reduced-motion block only disabled the shooting-star. Add the standard global rule collapsing all transitions/animations to ~instant so the scale/slide/fade flourishes don't play for users who opt out (design philosophy #7). App-wide, not just Scribe. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… status Write up the Scribe×provit run: NativeMacActor works, failures were foreground contention (not a Scribe bug), with ranked fixes for the provit agent (re-focus appTarget per capture, capture-by-windowID/serialize, fix journey red-vs-emerald copy) + repro steps. Mark G-3 (a11y) and G-4 (reduced-motion) resolved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Rephrase pill fired on the browser address bar (a URL). Now: skip search fields / omnibox (AXSearchField subrole) entirely, and the affordance only shows for real prose — isRephrasableProse() requires >=2 words and rejects URLs, emails, paths, and single tokens. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…element styles) The md preview WAS rendering (# heading → <h1>) but looked like body text because Tailwind's reset removes default heading/bold/list styles. Add scoped .scribe- preview prose CSS (headings, bold, em, lists, code, pre, blockquote, links, hr), theme-aware via --og-* tokens, monospace to match the brand. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds @tiptap/react + starter-kit + pm + extension-link for a true WYSIWYG rich editor (user chose a real library over the markdown-toolbar hack). Scoped CSS for the ProseMirror content (prose styles Tailwind Preflight strips) + the live squiggle decorations (wavy, brand-colored). Production build verified: TipTap bundles clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The AXSearchField subrole alone missed Chromium's omnibox (Chrome/Arc/Edge/ Brave expose it as a plain AXTextField with no subrole), so a multi-word URL or query selected in the address bar got a Rephrase pill. Add isAddressOrSearchField - subrole OR descriptive metadata (role-description/description/placeholder/title matching address/search/url markers) - and gate BOTH the squiggle path (focusedTextElement) and the pill path (updateAffordance) on it. Also reject any selection whose tokens are bare URLs/domains/emails in isRephrasableProse. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Remove posthog-js dependency. It was declared but never imported/initialized anywhere in source (zero usages) - a dangling analytics SDK that has no place in an app whose own onboarding promises 'no telemetry'. Gone from package.json + lockfile. (The PostHog *connector* in connectorCatalog is unrelated: an opt-in MCP integration to the user's OWN PostHog, like Slack/Figma.) - Onboarding step 2: the Pro grid grew to 9 cards (Scribe added), which overflowed behind the fixed Continue/nav footer (button covered the pricing line) and left an orphan card in a 4-col grid. Switch to a clean 3x3 (lg:grid-cols-3) and make the panel scroll with bottom padding that clears the fixed footer. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds jsdom + @testing-library/react/user-event so we can mount the real editor components and drive real DOM events (select text, click Improve) in vitest. Component-integration tests are named *.dom.test.tsx and opt into jsdom via a per-file docblock. This is the gap that let the 'card disappears on Improve' and 'no underline' regressions reach the user - node-only unit tests can't exercise the render/interaction path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scribe + Parakeet — core seams, shell, and transcription engine
Core half of the Scribe writing companion + Parakeet STT. The pro logic lives in off-grid-ai/desktop-pro
feat/scribe(#23); this PR is the open-core surface: generic seams, the inert shell, and the transcription engine layer (which sits in core beside whisper, since core RAG uses it too).What's here
og:chat-seedfor "continue in chat".proCatalogscribeentry +proItem('scribe')nav → UpgradeScreen in free builds; the real screen registers from pro.ParakeetCliTranscription(sherpa-onnx offline transducer) + a whisper-fallback selector.fetch-parakeet.sh+ a release.yml step stage the runtime (gated onSHERPA_ONNX_URL; no-op until pinned, with the 0.0.28-style closure/foreign-dep/minos gates).isCreatableEntityatrecordObservation,createEntity/linkObservationToEntity, "New" in Entities.Verified on-device (macOS 26, arm64)
Ran the staged static sherpa-onnx v1.13.3 binary on a real wav — transcribed correctly, RTF ~0.06 (~15× realtime). Real CLI flags confirmed via
--help(dropped a bogus--model-type);parseParakeetOutputlocked to the verbatim JSON output as a fixture test.Tests
tsc (node/web/pro) clean · eslint clean on new code · electron-vite build ok · 9/9 pro E2E green.
Screenshots
Video: flows are exercised by the Playwright Electron E2E (headless temp profile) with screenshots captured from those runs; Parakeet transcription verified against the real binary on-device.
To make Parakeet ship in release builds
Pin
SHERPA_ONNX_URL(repo var) to thesherpa-onnx v1.13.3 osx-arm64-staticasset — CI then stages it; signing is handled by electron-builder (static binary, no extra dylibs). Until then, selecting Parakeet transparently uses whisper.🤖 Generated with Claude Code
Summary by CodeRabbit