fix: isolate video codec runtime from capture

.github/workflows/test.yml

@@ -59,6 +59,9 @@ jobs:
       - name: Set up Python
         run: uv python install 3.12
 
+      - name: Install external video tools
+        run: choco install ffmpeg -y --no-progress
+
       - name: Install dependencies
         run: uv sync --extra dev
 
@@ -76,8 +79,8 @@ jobs:
         run: uv run pytest tests/test_performance.py -v -m slow --timeout=300
         timeout-minutes: 20
 
-  # macOS coverage: runs the unit suite (event processing, storage, video
-  # encode/decode via PyAV, headless-import invariant, Recorder API surface —
+  # macOS coverage: runs the unit suite (event processing, storage, external
+  # video boundary, headless-import invariant, Recorder API surface —
   # CGWindowList-backed screen capture works on GitHub macOS runners). The
   # 'slow' live-recording tests stay skipped here: synthetic input injection
   # requires Accessibility permissions a hosted runner cannot grant, and the
@@ -96,6 +99,9 @@ jobs:
       - name: Set up Python
         run: uv python install 3.12
 
+      - name: Install external video tools
+        run: brew install ffmpeg
+
       - name: Install dependencies
         run: uv sync --extra dev
 

README.md

@@ -53,7 +53,7 @@ Documentation for the whole stack lives at
 
 | Recording path | Current implementation |
 | --- | --- |
-| Windows and RDP demonstrations | `openadapt-capture` records native input and screen video; `openadapt-flow` converts the session into compiler input. |
+| Windows and RDP demonstrations | `openadapt-capture` records native input and action-gated screen video; `openadapt-flow` converts the session into compiler input. |
 | Browser demonstrations | `openadapt-flow` records its Playwright browser directly. It does not require this package. |
 | Chrome extension in this repository | Experimental DOM-capture code for development; it is not the supported web recorder or governed replay path. |
 
@@ -120,7 +120,17 @@ my-capture/
 └── profiling.json
 ```
 
-Audio and individual images are optional.
+Video remains the default evidence format. Capture stages exact-timestamp,
+lossless frames during the session, asks a separately provisioned FFmpeg
+executable to encode them, verifies the resulting MP4, and promotes it
+atomically. Capture never downloads, bundles, or links FFmpeg/PyAV. Set
+`OPENADAPT_FFMPEG_PATH`, pass `Recorder(ffmpeg_path=...)`, use Desktop's
+user-data `ffmpeg.json` provision manifest, or place `ffmpeg` and `ffprobe` on
+`PATH`. Recording performs a real encode-and-decode probe and refuses before
+input listeners start if the selected executable, codec, or PNG verification
+path is unavailable. A minimal managed runtime must provide the selected video
+encoder, MP4 demuxing/muxing, PNG decoding/encoding, the `image2pipe` muxer, and
+the `select` video filter; Desktop provisions and probes that exact closure.
 
 ## Window-scoped recording
 

docs/DESIGN.md

@@ -17,7 +17,9 @@ We need a platform-agnostic representation of GUI interactions that:
 ### Key Features
 
 1. **Production-ready** - Designed for continuous operation without degradation
-2. **No external dependencies** - Native video capture via PyAV (no OBS required)
+2. **Process-isolated video** - The MIT package stages frames and invokes a
+   separately provisioned FFmpeg executable; it does not bundle or link codec
+   libraries
 3. **Low resource footprint** - Non-blocking capture that doesn't slow down user's work
 4. **Chunked media** - Video/audio split into manageable segments for long captures
 5. **Audio capture** - Built-in audio recording with Whisper transcription
@@ -164,11 +166,21 @@ class Capture:
 
 ### Video Encoding
 
-Based on OpenAdapt's working implementation:
+Capture keeps its video-first behavior without importing PyAV. During a
+recording it stages numeric-name lossless PNG frames plus exact integer PTS.
+Finalization writes a safe `ffconcat` manifest, invokes an externally
+provisioned FFmpeg argument vector with bounded execution, verifies the output
+by decoding a PNG frame, and atomically promotes it. Successful finalization
+deletes the transient stage; failure retains it for recovery and never reports
+an MP4 as complete. A sibling or explicitly configured `ffprobe` preserves
+nearest-frame extraction and metadata inspection without linking codec
+libraries into Capture. The real preflight exercises the selected encoder,
+MP4, PNG through `image2pipe`, and the `select` filter before any input listener
+starts.
 
 ```python
-codec = "libx264"      # H.264 for compatibility
-pix_fmt = "yuv444p"    # Full color (vs yuv420p for smaller files)
+codec = None           # Probe platform encoders, then portable mpeg4 fallback
+pix_fmt = None         # Selected with the verified codec
 crf = 0                # Lossless (adjustable for size vs quality)
 preset = "veryslow"    # Maximum compression
 fps = 24               # Configurable
@@ -194,12 +206,12 @@ capture_abc123/
 
 ### Screenshots vs Video
 
-**Recommendation:** Default to video mode.
+**Decision:** Default to video mode.
 
-- More storage-efficient (H.264 compression)
-- Easier timestamp alignment
+- More storage-efficient than permanent individual screenshots
+- Exact timestamp alignment is preserved by staged-frame PTS and VFR durations
 - Screenshots can be extracted from video when needed
-- Individual screenshots useful for debugging frame alignment
+- Individual screenshots remain optional for debugging frame alignment
 
 ## Audio Handling
 
@@ -269,7 +281,7 @@ def scrub_capture(capture: Capture, scrubber: ScrubbingProvider) -> Capture:
 
 1. Define exact event schemas (Pydantic models)
 2. Implement SQLite storage for events
-3. Implement video capture with PyAV
+3. Maintain external-process video capture without bundling codec binaries
 4. Port event processing from OpenAdapt's `events.py`
 5. Add drag detection
 6. Integrate with `openadapt-privacy`

openadapt_capture/cli.py

@@ -27,9 +27,10 @@ def record(
     Args:
         output_dir: Directory to save capture.
         description: Optional task description.
-        video: Capture video (default: True).
+        video: Capture MP4 video (default: True). Requires an externally
+            provisioned FFmpeg executable.
         audio: Capture audio (default: False).
-        images: Save screenshots as PNGs (default: False).
+        images: Also save screenshots as PNGs (default: False).
         browser_events: Capture browser DOM events via Chrome extension (default: False).
             Requires the openadapt-capture Chrome extension to be installed and
             connects via WebSocket on localhost:8765.

openadapt_capture/config.py

@@ -48,8 +48,16 @@ class Settings(BaseSettings):
     RECORD_WINDOW_TITLE: str | None = None
     # useful for debugging but expensive computationally
     LOG_MEMORY: bool = False
-    VIDEO_ENCODING: str = "libx264"
-    VIDEO_PIXEL_FORMAT: str = "yuv444p"
+    # None means: use the Desktop provision manifest when present, otherwise
+    # probe a usable platform encoder with a portable mpeg4 fallback.
+    VIDEO_ENCODING: str | None = None
+    VIDEO_PIXEL_FORMAT: str | None = None
+    VIDEO_MUXER: str | None = None
+    # Optional exact path to an externally provisioned FFmpeg executable.
+    # Capture never downloads or bundles FFmpeg.
+    VIDEO_FFMPEG_PATH: str | None = None
+    VIDEO_FFPROBE_PATH: str | None = None
+    VIDEO_FFMPEG_TIMEOUT_SECONDS: float = 900.0
     # sequences that when typed, will stop the recording of ActionEvents
     STOP_SEQUENCES: list[list[str]] = [
         list(stop_str) for stop_str in STOP_STRS
@@ -97,6 +105,10 @@ class Settings(BaseSettings):
     "capture_full_video": "RECORD_FULL_VIDEO",
     "video_encoding": "VIDEO_ENCODING",
     "video_pixel_format": "VIDEO_PIXEL_FORMAT",
+    "video_muxer": "VIDEO_MUXER",
+    "ffmpeg_path": "VIDEO_FFMPEG_PATH",
+    "ffprobe_path": "VIDEO_FFPROBE_PATH",
+    "ffmpeg_timeout_seconds": "VIDEO_FFMPEG_TIMEOUT_SECONDS",
     "stop_sequences": "STOP_SEQUENCES",
     "log_memory": "LOG_MEMORY",
     "plot_performance": "PLOT_PERFORMANCE",
@@ -119,6 +131,10 @@ class RecordingConfig:
     capture_full_video: bool | None = None
     video_encoding: str | None = None
     video_pixel_format: str | None = None
+    video_muxer: str | None = None
+    ffmpeg_path: str | None = None
+    ffprobe_path: str | None = None
+    ffmpeg_timeout_seconds: float | None = None
     stop_sequences: list[list[str]] | None = None
     log_memory: bool | None = None
     plot_performance: bool | None = None

openadapt_capture/recorder.py

@@ -33,7 +33,6 @@
 from functools import partial
 from typing import Any, Callable
 
-import av
 import fire
 import numpy as np
 import psutil
@@ -272,6 +271,22 @@ def _join_tasks(
     return lingering
 
 
+def _raise_for_failed_processes(task_by_name: dict[str, Any]) -> None:
+    """Surface required child-process failures through the recording boundary."""
+    failures = {
+        name: task.exitcode
+        for name, task in task_by_name.items()
+        if isinstance(task, multiprocessing.process.BaseProcess)
+        and task.exitcode not in (None, 0)
+    }
+    if failures:
+        detail = ", ".join(
+            f"{name} (exit code {exitcode})"
+            for name, exitcode in sorted(failures.items())
+        )
+        raise RuntimeError(f"Recording child process failed: {detail}")
+
+
 def collect_stats(performance_snapshots: list[tracemalloc.Snapshot]) -> None:
     """Collects and appends performance snapshots using tracemalloc.
 
@@ -670,6 +685,8 @@ def video_pre_callback(
     recording: Recording,
     video_dir: str = None,
     frame_size: tuple[int, int] | None = None,
+    provision: video.FFmpegProvision | None = None,
+    timeout_seconds: float | None = None,
 ) -> dict[str, Any]:
     """Function to call before main loop.
 
@@ -680,6 +697,9 @@ def video_pre_callback(
         frame_size: Explicit (width, height) for the video stream. Used by
             window-scoped capture, whose frames are the target window's pixels
             rather than the monitor's. Defaults to the full-screen size.
+        provision: Parent-preflighted encoder contract. Passing this immutable
+            value keeps spawn-based writers on the exact executable and codec.
+        timeout_seconds: Bound for final encoding and verification.
 
     Returns:
         dict[str, Any]: The updated state.
@@ -691,7 +711,13 @@ def video_pre_callback(
         # TODO XXX replace with utils.get_monitor_dims() once fixed
         monitor_width, monitor_height = utils.take_screenshot().size
     video_container, video_stream, video_start_timestamp = (
-        video.initialize_video_writer(video_file_path, monitor_width, monitor_height)
+        video.initialize_video_writer(
+            video_file_path,
+            monitor_width,
+            monitor_height,
+            timeout_seconds=timeout_seconds,
+            preflight_provision=provision,
+        )
     )
     crud.update_video_start_time(db, recording, video_start_timestamp)
     return {
@@ -730,8 +756,8 @@ def write_video_event(
     recording_timestamp: float,
     event: Event,
     perf_q: sq.SynchronizedQueue,
-    video_container: av.container.OutputContainer,
-    video_stream: av.stream.Stream,
+    video_container: Any,
+    video_stream: Any,
     video_start_timestamp: float,
     last_pts: int = 0,
     num_copies: int = 2,
@@ -744,9 +770,8 @@ def write_video_event(
         recording_timestamp: The timestamp of the recording.
         event: A screen event to be written.
         perf_q: A queue for collecting performance data.
-        video_container (av.container.OutputContainer): The output container to which
-            the frame is written.
-        video_stream (av.stream.Stream): The video stream within the container.
+        video_container: The external-encoder staging container.
+        video_stream: The configured external-encoder stream.
         video_start_timestamp (float): The base timestamp from which the video
             recording started.
         last_pts: The last presentation timestamp.
@@ -1620,6 +1645,17 @@ def record(
         config.RECORD_VIDEO,
         config.RECORD_IMAGES,
     )
+    # Refuse before touching the display or starting any worker. Capture never
+    # downloads or bundles FFmpeg; Desktop/standalone provisioning owns it.
+    video_provision = None
+    if config.RECORD_VIDEO:
+        video_provision = video.require_video_encoder(
+            ffmpeg_path=config.VIDEO_FFMPEG_PATH,
+            ffprobe_path=config.VIDEO_FFPROBE_PATH,
+            codec=config.VIDEO_ENCODING,
+            pixel_format=config.VIDEO_PIXEL_FORMAT,
+            muxer=config.VIDEO_MUXER,
+        )
 
     # Configure loguru level for recording (without destroying global config)
     logger.configure(handlers=[{"sink": sys.stderr, "level": LOG_LEVEL}])
@@ -1884,6 +1920,8 @@ def record(
                         if initial_window_frame is not None
                         else None
                     ),
+                    provision=video_provision,
+                    timeout_seconds=config.VIDEO_FFMPEG_TIMEOUT_SECONDS,
                 ),
                 video_post_callback,
             ),
@@ -2013,6 +2051,7 @@ def record(
         task_name, task_error = task_errors.get_nowait()
         add_exception_note(task_error, f"recording task {task_name!r} failed")
         raise task_error
+    _raise_for_failed_processes(task_by_name)
 
     if config.PLOT_PERFORMANCE and startup_ready:
         from openadapt_capture import plotting
@@ -2141,6 +2180,10 @@ def __init__(
         capture_full_video: bool | None = None,
         video_encoding: str | None = None,
         video_pixel_format: str | None = None,
+        video_muxer: str | None = None,
+        ffmpeg_path: str | None = None,
+        ffprobe_path: str | None = None,
+        ffmpeg_timeout_seconds: float | None = None,
         stop_sequences: list[list[str]] | None = None,
         log_memory: bool | None = None,
         plot_performance: bool | None = None,
@@ -2170,6 +2213,10 @@ def __init__(
             capture_full_video=capture_full_video,
             video_encoding=video_encoding,
             video_pixel_format=video_pixel_format,
+            video_muxer=video_muxer,
+            ffmpeg_path=ffmpeg_path,
+            ffprobe_path=ffprobe_path,
+            ffmpeg_timeout_seconds=ffmpeg_timeout_seconds,
             stop_sequences=stop_sequences,
             log_memory=log_memory,
             plot_performance=plot_performance,