Skip to content

Add Transcripted Lab experiment workbench and CLI - #1705

Draft
r3dbars wants to merge 1 commit into
mainfrom
codex/transcripted-lab
Draft

Add Transcripted Lab experiment workbench and CLI#1705
r3dbars wants to merge 1 commit into
mainfrom
codex/transcripted-lab

Conversation

@r3dbars

@r3dbars r3dbars commented Aug 25, 2026

Copy link
Copy Markdown
Owner

What this adds

A separate local Transcripted Lab product for running repeatable experiments against Transcripted's real production and repository-owned test paths.

  • Native SwiftUI Transcripted Lab.app
  • Shared TranscriptedLabKit experiment/report engine
  • Headless transcripted-lab CLI backed by the same contract
  • Versioned JSON reports, run history, and baseline/candidate comparisons
  • Hard gates that fail before the numeric score can hide a serious regression

Benches

  • Runtime Snapshot — transcription elapsed/RTF, dictation start, first sample, stop-to-paste/done, fallback/retry events
  • Dictation Bench — production/native/pre-resampled/chunked variants, compute units, finalization order, repetitions, Auto Enter, silence guardrail, stable output hashes
  • Transcription Bench — existing local corpus validation and Transcripted-vs-truth comparison
  • Speaker Bench — labeled-audio threshold sweeps plus the existing frozen ASK/SUGGEST/AUTO auto-research path
  • QA Bench — quick/deep/full/UI/packaged/artifact/synthetic/live repository lanes

Safety rules

The score cannot average away:

  • cross-person speaker false merges
  • missing final dictation text
  • silence producing text
  • identical input producing unstable output hashes
  • blocking QA failures
  • timeouts or non-zero experiment exits

Local verification completed

  • swift test --package-path Tools/TranscriptedLab --jobs 4 — 5/5 focused tests passed
  • swift build --package-path Tools/TranscriptedLab --product transcripted-lab
  • CLI help and command parsing exercised
  • app-bundle script syntax validated

The new macOS workflow builds and tests the Foundation kit/CLI, compiles the SwiftUI app, and verifies the ad-hoc-signed app bundle.

Current intentional limits

  • Dictation Bench currently measures speed, delivery integrity, silence behavior, and repeatability. A dedicated labeled dictation WER/CER corpus is the next accuracy lane.
  • Transcription corpus accuracy and production runtime speed are separate lanes today.
  • Speaker auto-research keeps its native multi-gate holdout report rather than flattening false automatic names and contamination checks into a misleading single score.

Create a standalone SwiftUI app, shared Lab kit, and headless CLI that orchestrate Transcripted's existing runtime, dictation, transcription corpus, speaker identity, and QA lanes. Store versioned aggregate reports, surface hard gates before scores, compare baseline and candidate runs, and verify the package with focused tests plus a macOS CI lane.

r3dbars commented Aug 25, 2026

Copy link
Copy Markdown
Owner Author

Verification complete

All automated gates are green on 8f15c626:

  • Transcripted Lab focused tests: 5 passed
  • transcripted-lab CLI build: passed
  • SwiftUI TranscriptedLab build: passed
  • ad-hoc-signed Transcripted Lab.app bundle verification: passed
  • repo hygiene: passed
  • Transcripted app build + premium performance budget: passed
  • fast tests + E2E smoke: passed
  • Core package + integration + all Tools package tests: passed

First local launch after checking out this branch:

Tools/TranscriptedLab/script/build_and_run.sh

CLI entrypoint:

swift run --package-path Tools/TranscriptedLab transcripted-lab help

I am intentionally keeping the PR in draft while the next likely slice remains explicit: a labeled dictation/transcription WER/CER corpus, normalized speaker auto-research holdout metrics in the UI, and bounded multi-arm campaign recipes similar to Tilde Lab.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant