Accept Voice Memos drags and add batch file transcription - #716
Draft
shreeraman96 wants to merge 4 commits into
Draft
Accept Voice Memos drags and add batch file transcription#716shreeraman96 wants to merge 4 commits into
shreeraman96 wants to merge 4 commits into
Conversation
- AppKit promise-aware drop target (SwiftUI onDrop cannot receive file promises): concrete URLs + file promises, resolved off-main via modern receiver, legacy promise API, and raw pasteboard data fallbacks - BatchTranscriptionCoordinator: sequential per-item state machine with cancellation, staging cleanup, and batch/dictation arbitration - App-level FileTranscriptionSession so in-flight work survives navigation - Multi-select file picker; batch progress UI with per-item status
github-actions
Bot
added
the
needs screenshots
|
The PR Policy check is blocking this PR because required template information is missing. Please update the PR description with:
Visual files detected:
Screenshots or video are required for UI, UX, settings, onboarding, overlay, menu bar, or visual behavior changes. If this PR has no visual changes, check the no-visual-change box in the template. If this remains incomplete for 48 hours after opening, the PR may be closed. |
github-actions
Bot
removed
the
needs screenshots
NSPasteboard is not thread-safe: our worker's per-item reads raced the promise receivers' internal pasteboard access and crashed in the type cache (NSInternalInconsistencyException -> ggml terminate handler -> abort). All app-side reads now happen first on one thread, receivers start only afterwards on a serial queue, and every pasteboard call is wrapped in an ObjC exception catcher so a future race logs instead of killing the app.
- Single MainActor arbiter for the shared ASR model: dictation intent flag set synchronously before any await closes the TOCTOU gap; batch items wait on dictation intent, live capture, and in-flight single-file transcription - Batch guard moved to every hotkey entry point (dictate/prompt/command/ rewrite), with an audible cue when dictation is blocked mid-batch - Coordinator: pending-driven processing (completed items never re-run when enqueueing after an undismissed batch) and self-draining restart (items enqueued during the cancel window are picked up, not stranded) - CancellationError rethrown unwrapped from transcribeFile (no false failure analytics) - Drop resolver: empty-delivery path no longer deletes pending receiver dirs; receivers hold in-flight tokens so slow (iCloud) promises aren't abandoned at the 30s soft timeout; failed receivers count as resolved so partial failures deliver promptly; partial-failure accounting includes raw-data item counts - Single-file drops route into the batch when anything is in flight or a finished batch is still displayed; drop errors auto-dismiss - 7 new regression tests (182 total)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Dragging a recording from Apple Voice Memos into File Transcription did nothing, and only one file could be transcribed at a time. This PR makes both work:
PromiseAwareDropView): SwiftUI's.onDropcan never see Voice Memos drags — they are file promises with nopublic.file-url(open Apple gap, FB13583826). An AppKitNSViewRepresentableoverlay now accepts both concrete URLs and file promises. Resolution runs entirely off the main thread (verified live: any main-thread pasteboard read can deadlock intermittently and freezes the system-wide drag session) through three racing paths: modernNSFilePromiseReceiver(completion-callback-gated, so a stalled iCloud download is never delivered truncated) → raw pasteboard data (what Voice Memos actually serves, ~100 ms) → deprecated legacy promise API as a lazy last resort (an eager legacy request holds Voice Memos' promise machinery hostage for 35–80 s and blocks its next drag).BatchTranscriptionCoordinator): sequential per-item state machine — per-file status/progress/error, "no speech" detection, cancellation (pending items stop instantly; the in-flight native transcription runs out, shown as "Cancelling…"), and staging-dir cleanup only after each item finishes.FileTranscriptionSession): coordinator + service moved out of the view so in-flight work survives sidebar navigation.providers.first).Task.checkCancellation()points inMeetingTranscriptionService's chunk loop so a cancelled batch stops mid-file on the chunked path.Staging design: each promised file gets its own temp subdirectory (two memos titled "New Recording.m4a" never collide), duplicates from losing delivery paths are swept — except dirs whose promise is still pending, because deleting a promise's destination mid-write leaves the source app's drag session unresolved and Voice Memos then refuses all further drags until restarted (verified live; cleaned up on a deferred task instead).
Type of Change
Related Issue or Discussion
Closes #219
Testing
swiftlint --strictclean on all files touched by this PRswiftformatreflows ~3k unrelated pre-existing lines on main; new files follow the existing styleLive end-to-end verification on this machine (Voice Memos → FluidVoice):
Screenshots / Video
Batch card after a drop (per-item status, header summary, Done):
Notes
NSFilePromiseReceiverpath is unreliable for Voice Memos (a Catalyst app): called on the main thread it fails instantly with Cocoa 3072; on a background thread it completes only sometimes. The raw pasteboard data path is what reliably delivers Voice Memos audio; the receiver + legacy paths cover other promise sources (Photos, Mail).performDragOperationreturns is outside AppKit's documented lifetime contract — it is the only arrangement found that neither deadlocks nor loses drops; commented in code with the poll-timeout safety net.CloudRecordings.dbis an undocumented schema.