From bd67098f69ea8c732df5d948de325b171c22af3f Mon Sep 17 00:00:00 2001 From: Richard Abrich Date: Fri, 17 Jul 2026 18:04:01 -0400 Subject: [PATCH 1/2] chore: adopt openadapt-capture 0.5.4 and run the adapter tests in CI - Bump the `capture` extra floor to openadapt-capture>=0.5.4 (0.5.3/0.5.4 fixed the screenshot-at-import side effect and persist pixel_ratio on the recording model). uv.lock already resolved 0.5.4; specifier synced. - Replace the three stale "capture screenshots the display at import" comments (pyproject, tests/test_capture_adapter.py, the adapter's HiDPI caveat) with accurate 0.5.4 semantics. - CI: the fast unit `test` job now installs the `capture` extra and asserts `import openadapt_capture` works headless, so the 13 adapter tests in tests/test_capture_adapter.py execute for real instead of silently skipping -- making RECORDING.md's "CI-tested" claim true. The module now skips ONLY when the extra is absent; an import failure with the extra installed fails loudly. - docs/desktop/RECORDING.md + claims.yaml updated to state precisely what CI proves (real on-disk capture session -> convert -> compile, not a live recorder run). Verified locally with 0.5.4: 13/13 adapter tests pass; ruff check, ruff format, mypy, and scripts/check_consistency.py all clean. Co-Authored-By: Claude Fable 5 --- .github/workflows/ci.yml | 12 +++++++++++- claims.yaml | 8 +++++--- docs/desktop/RECORDING.md | 4 +++- openadapt_flow/adapters/capture.py | 13 +++++++------ pyproject.toml | 9 +++++---- tests/test_capture_adapter.py | 31 ++++++++++++++---------------- uv.lock | 2 +- 7 files changed, 46 insertions(+), 33 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6d11b6d6..52395912 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -239,8 +239,18 @@ jobs: cache: pip cache-dependency-path: pyproject.toml + # The `capture` extra rides along so the desktop on-ramp adapter tests + # (tests/test_capture_adapter.py) run for REAL in this job instead of + # silently skipping: openadapt-capture >=0.5.4 imports clean headless + # (no screenshot at import), so no display is needed. - name: Install - run: pip install -e .[dev] + run: pip install -e '.[dev,capture]' + + # Guard the invariant the adapter tests rely on: the capture extra must + # import headless. If this ever regresses upstream, fail here with a + # clear message rather than deep inside test collection. + - name: Verify openadapt-capture imports headless + run: python -c "import openadapt_capture" # Cache the Playwright browser binaries (chromium) so the lightweight # browser-backed unit tests don't re-download ~150MB every run. Keyed on diff --git a/claims.yaml b/claims.yaml index cc74dcdb..2c63314b 100644 --- a/claims.yaml +++ b/claims.yaml @@ -266,7 +266,8 @@ claims: proves: >- The capture->recording bridge over a REAL openadapt-capture session (its own event-processing pipeline + frame extraction) feeding the - UNMODIFIED compiler. + UNMODIFIED compiler (runs on default CI: the `test` job installs the + `capture` extra; openadapt-capture >=0.5.4 imports clean headless). - path: docs/desktop/RECORDING.md kind: doc proves: >- @@ -278,8 +279,9 @@ claims: Offline capture carries NO structural (UIA AutomationId) locator, so replay uses the visual ladder (template/ocr/geometry). The live capture needs a display (openadapt-capture is the optional `capture` extra); the - CI proof covers convert -> compile -> replay on the recording format, - not a live capture session. + CI proof covers a real on-disk capture session -> convert -> compile -> + replay, not a live recorder run (the live recorder has its own + integration suite in the openadapt-capture repo). - >- `--secret` is refused on the pixel/desktop substrate (no field geometry to redact), and `rdp` recording must be captured in the same pixel space diff --git a/docs/desktop/RECORDING.md b/docs/desktop/RECORDING.md index a3205610..7eb34c09 100644 --- a/docs/desktop/RECORDING.md +++ b/docs/desktop/RECORDING.md @@ -31,7 +31,9 @@ is a thin orchestration over two components that already exist and are tested: (`meta.json` + `events.jsonl` + `frames/{i:04d}_before.png` / `_after.png`), running openadapt-capture's own event-processing pipeline (raw streams → merged clicks / typed text). This adapter is unit-tested end to end against a - real capture session in `tests/test_capture_adapter.py`. + real capture session in `tests/test_capture_adapter.py`, and those tests run + on default CI: the fast unit `test` job installs the `capture` extra + (openadapt-capture >=0.5.4 imports clean headless, so no display is needed). The genuinely new piece (`openadapt_flow/desktop_record.py`, `record_desktop_capture`) is the **live orchestration**: start a capture diff --git a/openadapt_flow/adapters/capture.py b/openadapt_flow/adapters/capture.py index dcbedb3b..1d39e27e 100644 --- a/openadapt_flow/adapters/capture.py +++ b/openadapt_flow/adapters/capture.py @@ -57,12 +57,13 @@ Coordinate spaces: capture mouse coordinates are in *logical points* (pynput); video frames are *physical pixels*. openadapt-flow requires event coordinates in the same pixel space as the frames, so points are scaled by -``CaptureSession.pixel_ratio`` (physical / logical). NOTE: capture 0.5.x only -persists ``pixel_ratio`` when the recorder wrote it into the recording -``config`` JSON; absent that it defaults to 1.0 and coordinates pass through -unscaled. On a HiDPI capture whose ``pixel_ratio`` was not persisted, click -coordinates would be under-scaled — an honest limitation of the upstream -metadata, not something this adapter can recover from pixels alone. +``CaptureSession.pixel_ratio`` (physical / logical). NOTE: sessions recorded +with capture >=0.5.4 persist ``pixel_ratio`` on the recording model itself, so +scaling is always correct for them. Older 0.5.x sessions carry it only when +the recorder wrote it into the recording ``config`` JSON; absent that it +defaults to 1.0 and coordinates pass through unscaled — on such a legacy HiDPI +session click coordinates would be under-scaled, an honest limitation of the +old metadata that this adapter cannot recover from pixels alone. Frame selection: openadapt-capture records **action-gated** video (frames are encoded around user actions, not continuously). For an event at wall-clock time diff --git a/pyproject.toml b/pyproject.toml index d746c134..b088a057 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -108,10 +108,11 @@ service-mlx = [ ] # Cross-platform desktop recording on-ramp: convert an openadapt-capture # session into a flow recording (openadapt_flow.adapters.capture). Optional and -# imported lazily — the flow core never pulls it onto the replay hot path. Note: -# openadapt-capture screenshots the display when imported, so this on-ramp needs -# a real desktop session (not headless CI). -capture = ["openadapt-capture>=0.5.1"] +# imported lazily — the flow core never pulls it onto the replay hot path. +# Floor 0.5.4: imports clean headless (no screenshot at import, fixed in +# 0.5.3/0.5.4) and persists pixel_ratio on the recording model, so the adapter +# tests run for real in headless CI (the `test` job installs this extra). +capture = ["openadapt-capture>=0.5.4"] # Canonical action-vocabulary interop (openadapt_flow.interop.types). Boundary # only (emit / benchmark round-trip); never on the replay hot path. The shim # imports openadapt-types lazily, so the core stays importable without it. diff --git a/tests/test_capture_adapter.py b/tests/test_capture_adapter.py index b4ef44ea..fa681113 100644 --- a/tests/test_capture_adapter.py +++ b/tests/test_capture_adapter.py @@ -10,9 +10,10 @@ pass against a schema that no longer exists. The converted recording is fed to the UNMODIFIED compiler — the format bridge, proven end to end. -openadapt-capture screenshots the display at import time, so the whole module -is skipped when the package cannot be imported (headless CI / no display); it -runs for real on a developer desktop with the ``capture`` extra installed. +openadapt-capture >=0.5.4 imports clean headless (the historical +screenshot-at-import side effect was removed in 0.5.3/0.5.4), so this module +runs for real in headless CI — the `test` job installs the ``capture`` extra. +It is skipped only when that optional extra is not installed. """ from __future__ import annotations @@ -23,24 +24,19 @@ import numpy as np import pytest -# openadapt-capture is an optional extra AND screenshots the display when -# imported (a module-level side effect in its recorder). Skip the whole module -# if it is absent or cannot be imported in this environment. +# openadapt-capture is an optional extra: skip only when it is not installed. +# Since 0.5.4 the import is headless-clean (no screenshot at import), so when +# the extra IS installed — as in CI's `test` job — an import failure is a real +# regression and must fail loudly here instead of silently skipping. if importlib.util.find_spec("openadapt_capture") is None: pytest.skip( "openadapt-capture not installed (capture extra)", allow_module_level=True ) -try: # importing the package screenshots the display; skip if that fails - import openadapt_capture # noqa: F401 - from openadapt_capture.db import create_db - from openadapt_capture.db.models import ActionEvent, Recording - from openadapt_capture.video import VideoWriter -except Exception as exc: # pragma: no cover - environment dependent - pytest.skip( - f"openadapt-capture cannot be imported here ({exc})", - allow_module_level=True, - ) +import openadapt_capture # noqa: F401 +from openadapt_capture.db import create_db +from openadapt_capture.db.models import ActionEvent, Recording +from openadapt_capture.video import VideoWriter from PIL import Image, ImageDraw from openadapt_flow.adapters.capture import convert_capture @@ -137,7 +133,8 @@ def write_recording_db(path: Path, action_rows: list[dict]) -> None: video_start_time=VIDEO_T0, double_click_interval_seconds=0.5, double_click_distance_pixels=5.0, - # capture persists pixel_ratio only in the config JSON. + # config-JSON pixel_ratio: the legacy (pre-0.5.4) persistence path, + # which CaptureSession.pixel_ratio still honors as a fallback. config={"pixel_ratio": PIXEL_RATIO}, ) session.add(recording) diff --git a/uv.lock b/uv.lock index 8e51751c..b8ebab97 100644 --- a/uv.lock +++ b/uv.lock @@ -2291,7 +2291,7 @@ requires-dist = [ { name = "mlx-vlm", marker = "extra == 'service-mlx'", specifier = ">=0.6.4,<0.7" }, { name = "mypy", marker = "extra == 'dev'", specifier = ">=1.11" }, { name = "numpy", specifier = ">=1.26" }, - { name = "openadapt-capture", marker = "extra == 'capture'", specifier = ">=0.5.1" }, + { name = "openadapt-capture", marker = "extra == 'capture'", specifier = ">=0.5.4" }, { name = "openadapt-grounding", marker = "extra == 'grounding'", specifier = ">=0.1.0" }, { name = "openadapt-privacy", extras = ["presidio"], marker = "extra == 'privacy'", specifier = ">=1.0.0" }, { name = "openadapt-types", marker = "extra == 'interop'", specifier = ">=0.2.0,<0.4.0" }, From 142e12adcef62d2d8ff0f202baa489f1ef2914bb Mon Sep 17 00:00:00 2001 From: Richard Abrich Date: Fri, 17 Jul 2026 18:10:21 -0400 Subject: [PATCH 2/2] chore: regenerate verification report for updated capture-bridge evidence claims.yaml evidence/caveat wording changed in the previous commit; the gate requires docs/VERIFICATION.md + docs/verification.json to be regenerated in lockstep (scripts/validate_claims.py --report). Co-Authored-By: Claude Fable 5 --- docs/VERIFICATION.md | 4 ++-- docs/verification.json | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/VERIFICATION.md b/docs/VERIFICATION.md index df263ec6..060f02a2 100644 --- a/docs/VERIFICATION.md +++ b/docs/VERIFICATION.md @@ -147,12 +147,12 @@ | Backing evidence | Kind | Gating / CI stage | Strength | Proves | |---|---|---|---|---| | `tests/test_desktop_record.py` | test | ci (required PR gate (test)) | supported | The live-capture orchestration + CLI wiring (record --backend windows|rdp -> openadapt-capture -> convert_capture), and that a desktop-shaped recording COMPILES into a bundle and REPLAYS to completion through the desktop backend path, resolving each click to its recorded target (runs on default CI). | -| `tests/test_capture_adapter.py` | test | ci (required PR gate (test)) | supported | The capture->recording bridge over a REAL openadapt-capture session (its own event-processing pipeline + frame extraction) feeding the UNMODIFIED compiler. | +| `tests/test_capture_adapter.py` | test | ci (required PR gate (test)) | supported | The capture->recording bridge over a REAL openadapt-capture session (its own event-processing pipeline + frame extraction) feeding the UNMODIFIED compiler (runs on default CI: the `test` job installs the `capture` extra; openadapt-capture >=0.5.4 imports clean headless). | | `docs/desktop/RECORDING.md` | doc | artifact (doc/benchmark) | roadmap | The precise REAL-vs-deferred map (visual-ladder-only structural rung, secret redaction, rdp coordinate space) and the reuse of openadapt-capture + the capture adapter. | **Caveats (honest limits):** -- Offline capture carries NO structural (UIA AutomationId) locator, so replay uses the visual ladder (template/ocr/geometry). The live capture needs a display (openadapt-capture is the optional `capture` extra); the CI proof covers convert -> compile -> replay on the recording format, not a live capture session. +- Offline capture carries NO structural (UIA AutomationId) locator, so replay uses the visual ladder (template/ocr/geometry). The live capture needs a display (openadapt-capture is the optional `capture` extra); the CI proof covers a real on-disk capture session -> convert -> compile -> replay, not a live recorder run (the live recorder has its own integration suite in the openadapt-capture repo). - `--secret` is refused on the pixel/desktop substrate (no field geometry to redact), and `rdp` recording must be captured in the same pixel space the backend replays in (no cross-machine coordinate remap yet). ### `citrix-pixel-validating` — validating — opt-in / infra-gated or field test diff --git a/docs/verification.json b/docs/verification.json index aa96c0b0..2121b10b 100644 --- a/docs/verification.json +++ b/docs/verification.json @@ -381,7 +381,7 @@ ], "strongest_evidence": "supported", "caveats": [ - "Offline capture carries NO structural (UIA AutomationId) locator, so replay uses the visual ladder (template/ocr/geometry). The live capture needs a display (openadapt-capture is the optional `capture` extra); the CI proof covers convert -> compile -> replay on the recording format, not a live capture session.", + "Offline capture carries NO structural (UIA AutomationId) locator, so replay uses the visual ladder (template/ocr/geometry). The live capture needs a display (openadapt-capture is the optional `capture` extra); the CI proof covers a real on-disk capture session -> convert -> compile -> replay, not a live recorder run (the live recorder has its own integration suite in the openadapt-capture repo).", "`--secret` is refused on the pixel/desktop substrate (no field geometry to redact), and `rdp` recording must be captured in the same pixel space the backend replays in (no cross-machine coordinate remap yet)." ], "evidence": [ @@ -405,7 +405,7 @@ "node": null, "node_found": null, "junit_status": null, - "proves": "The capture->recording bridge over a REAL openadapt-capture session (its own event-processing pipeline + frame extraction) feeding the UNMODIFIED compiler." + "proves": "The capture->recording bridge over a REAL openadapt-capture session (its own event-processing pipeline + frame extraction) feeding the UNMODIFIED compiler (runs on default CI: the `test` job installs the `capture` extra; openadapt-capture >=0.5.4 imports clean headless)." }, { "path": "docs/desktop/RECORDING.md",