Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 34 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,15 +106,23 @@ openadapt-flow record --backend macos --macos-app TextEdit --out rec
openadapt-flow record --backend linux --linux-app gedit \
--linux-window-title "Untitled Document 1" --out rec

# Remote display / Citrix / VDI (pixel-only vision ladder over RDP).
# RDP remote display (pixel-only vision ladder over the network session).
openadapt-flow record --backend rdp --rdp-host 10.0.0.5 --out rec

# Citrix / VDI (one exact local Citrix Workspace window).
openadapt-flow record --backend citrix \
--window "Citrix Viewer" \
--rdp-window "Citrix Viewer" \
--rdp-window-title "Ward A" \
--rdp-readiness-text "Appointments" \
--out rec
```

`--backend web` is browser-first (the app is a `--url`). For
`windows`, `macos`, `linux`, and `rdp` the capture has no field identity, so the
`windows`, `macos`, `linux`, `rdp`, and `citrix` the capture has no field identity, so the
target is the app window or host: `--agent-url` for Windows, `--macos-app`,
`--linux-app` plus `--linux-window-title`, and `--rdp-host` (or a configured
`rdp_window` for a local Citrix / Parallels window). Pass the same `--backend`
exact `rdp_window` / `rdp_window_title` for Citrix Workspace). Pass the same `--backend`
plus target flags to `replay`, or drive a real deployment with
`openadapt-flow run bundle --config deploy.yaml`, which reads the backend,
effects, actuation, durable, and policy sections from one config. Recorded
Expand Down Expand Up @@ -208,14 +216,16 @@ runtime; none is a second-class add-on. Maturity is reported honestly per the
|---|---|---|---|
| Web / browser | `--backend web` | Validated | Full lifecycle on every CI build, plus third-party OpenEMR evidence |
| Native macOS (AX) | `--backend macos` | Validated | 3/3 fixed TextEdit trials with exact file-byte effects; refused a two-window ambiguity without changing either file |
| Native Windows (UIA) | `--backend windows` | Early | 3/3 fixed WinForms trials with independently confirmed SQLite effects; 3/3 refusal for both stale and ambiguous targets |
| Native Linux (AT-SPI) | `--backend linux` | Early | Required CI drives a real GTK3 workflow through an isolated X11 / session-D-Bus environment: three verified effects, plus three ambiguous-target and three stale-target refusals |
| RDP (remote display) | `--backend rdp` | Early | Real-network Aardwolf RDP into Windows 11 passed 3/3 fixed remote-input trials with independent guest-tools file verification |
| Citrix / VDI (pixel ladder) | `--backend rdp` + `rdp_window` | Exploratory | Pixel-only vision ladder; no validated ICA / HDX environment qualified yet. Enters design-partner qualification in the customer's exact published application |

These are accepted scoped qualifications. A customer application is qualified
against its own controls, session/display policy, identity evidence, and effect
oracle. Details:
| Native Windows (UIA) | `--backend windows` | Available | 3/3 fixed WinForms trials with independently confirmed SQLite effects; 3/3 refusal for both stale and ambiguous targets |
| Native Linux (AT-SPI) | `--backend linux` | Available | Required CI drives a real GTK3 workflow through an isolated X11 / session-D-Bus environment: three verified effects, plus three ambiguous-target and three stale-target refusals |
| RDP (remote display) | `--backend rdp` | Available | Real-network Aardwolf RDP into Windows 11 passed 3/3 fixed remote-input trials with independent guest-tools file verification; a separate real-FreeRDP batch covers record → compile → governed replay and refusal |
| Citrix / VDI (pixel ladder) | `--backend citrix` | Code-qualified | Dedicated exact-Workspace-window driver, readiness gate, durable resume, and 3 healthy + 3 drift-halt no-DOM trials; the retained artifact records `ica_hdx_accepted=false` until a counted ICA/HDX run is attached |

Every row is bounded to its stated evidence. Accepted application workflows are
qualified against their own controls, session/display policy, identity evidence,
and effect oracle; code-qualified Citrix deployments additionally attach the
counted ICA/HDX record for their exact Workspace/server/application matrix.
Details:
[`docs/backends/RDP.md`](docs/backends/RDP.md),
[`docs/desktop/LINUX_NATIVE.md`](docs/desktop/LINUX_NATIVE.md),
[`docs/desktop/CITRIX_PIXEL.md`](docs/desktop/CITRIX_PIXEL.md).
Expand Down Expand Up @@ -422,8 +432,11 @@ submission format: [`benchmark/effectbench/LEADERBOARD.md`](benchmark/effectbenc
The reference browser path runs record, compile, policy-check, deterministic
replay, refusal, and report generation in CI. Windows UIA, native macOS, native
Linux, and RDP each have retained 3/3 accepted task evidence with independent
effects or oracles. Citrix and each new third-party application are qualified
with a design partner in the exact deployment environment. The workflow-program
effects or oracles. Citrix has a dedicated exact-window backend and a retained
3+3 code-readiness record; an exact ICA/HDX deployment receives its own counted
qualification record rather than inheriting RDP or stand-in evidence. Each new
third-party application is similarly qualified against its controls and effect
oracle. The workflow-program
IR adds parameters, branches, loops, effect verification, and governed recovery
on the same runtime. `DESIGN.md` has the module contracts;
[`docs/design/WORKFLOW_PROGRAM_IR.md`](docs/design/WORKFLOW_PROGRAM_IR.md)
Expand Down Expand Up @@ -503,7 +516,7 @@ openadapt-flow push ./triage.sanitized --kind recording
openadapt-flow compile ./triage.sanitized --out ./triage.bundle --name triage
openadapt-flow lint ./triage.bundle --strict
openadapt-flow certify ./triage.bundle --policy permissive
openadapt-flow replay ./triage.bundle --url https://example.internal/login \
openadapt-flow replay ./triage.bundle --url https://app.example.com/login \
--run-dir ./triage.run --param patient_id=example

# Privacy-review the executable bytes. A changed executable is refused.
Expand All @@ -516,11 +529,17 @@ openadapt-flow approve-sanitized ./triage.bundle.sanitized \
openadapt-flow validate-hosted \
--recording ./triage.sanitized --bundle ./triage.bundle.sanitized \
--run-dir ./triage.run --policy permissive --risk-class low \
--environment staging-v1 --target-url https://example.internal/login \
--environment staging-v1 --target-kind web \
--target-url https://app.example.com/login \
--out triage.validation.json
openadapt-flow push ./triage.bundle.sanitized --kind bundle \
--validation-attestation triage.validation.json

# Desktop/RDP/Citrix use the same validation command, deriving the signed
# target kind from report.json. Their app/window/host details stay local:
# --target-kind citrix --environment clinic-citrix-qualified-v1
# (omit --target-url and --allowed-host outside the web substrate).

# To activate this as a new version of an existing workflow, add:
# --workflow-id 00000000-0000-0000-0000-000000000000
# To bind that replacement to the exact halted run it repairs, also add:
Expand Down
6 changes: 3 additions & 3 deletions benchmark/vision_hardening/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,9 +187,9 @@ pytest -q tests/test_hardening_flywheel.py

- These are **synthetic fixtures**, not a live RDP/Citrix/HDX frame. The sweep
measures the *resolution ladder's* silent-mis-resolution behaviour under a
controlled metamorphic perturbation space. It **does not** raise any
substrate's maturity claim; RDP/Citrix pixel-only stays **Research**
(`docs/LIMITS.md`).
controlled metamorphic perturbation space. It **does not independently
qualify** a substrate; the separate counted RDP and Citrix records and their
exact scope remain authoritative (`docs/LIMITS.md`).
- It **adds no capability and weakens no guard** — it only measures the existing
resolver and locks in that the silent-wrong rate can only fall.
- The over-halt vs safe-halt split uses a documented per-perturbation severity
Expand Down
104 changes: 89 additions & 15 deletions claims.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ claims:
- >-
"Supported" is scoped to the reference headless-browser backend in this
registry. Desktop and remote-display workflows use the separately scoped
acceptance and design-partner claims below.
acceptance and code-qualified claims below.
- >-
The full record->compile->replay e2e runs in the post-merge/nightly
full suite; the fast required PR gate covers the replayer + demo units.
Expand Down Expand Up @@ -443,11 +443,46 @@ claims:
environment, not arbitrary applications, identity policies, or Citrix.

# ------------------------------ Desktop recording via the CLI (validating)
- id: substrate-runtime-validation-supported
claim: >-
Runtime-validation v2 binds the successful run's resolved target kind
(`web|windows|macos|linux|rdp|citrix`) into the signed operator
attestation while preserving the same exact artifact, compiler,
parameter-schema, strict-lint, policy, risk, report, and opaque
environment bindings. Web retains its exact HTTPS/host boundary;
native and remote execution emits no app, window, host, readiness, or
backend-hint values to Cloud.
surfaces: [README.md, docs, website]
tier: supported
evidence:
- path: tests/test_runtime_validation.py
proves: >-
Required CI exercises all six target kinds, validates v2 against its
closed JSON Schema, refuses target-kind mismatch or missing report
provenance, preserves web URL/host checks, keeps native/remote
execution empty, verifies the canonical HMAC vector, and retains
signed v1 web compatibility.
- path: tests/test_cli_new_commands.py
proves: >-
Required CI carries the resolved web, Windows, and Citrix backend
token through durable resume into the new run report rather than
trusting an upload-time relabel.
- path: schemas/runtime-validation-attestation-v2.json
kind: doc
proves: >-
The public closed envelope requires target_kind and conditionally
permits the browser execution fields only for web.
caveats:
- >-
The HMAC proves operator-token possession and envelope integrity; it is
not independent observation of the local replay or certification of an
arbitrary target application.

- id: desktop-recording-validating
claim: >-
The product CLI records desktop workflows through openadapt-capture and
`record --backend windows|rdp` captures the operator's real demonstration
and emits the compile-ready recording format, so
`record --backend windows|macos|linux|rdp|citrix` captures the operator's
real demonstration and emits the compile-ready recording format, so
record -> compile -> replay closes on the desktop substrate, not just the
browser.
surfaces: [README.md, docs, website]
Expand All @@ -456,10 +491,10 @@ claims:
- path: tests/test_desktop_record.py
proves: >-
The live-capture orchestration + CLI wiring (record --backend
windows|rdp -> openadapt-capture -> convert_capture), and that a
desktop-shaped recording COMPILES into a bundle and REPLAYS to
completion through the desktop backend path, resolving each click to
its recorded target (runs on default CI).
windows|macos|linux|rdp|citrix -> openadapt-capture ->
convert_capture), and that a desktop-shaped recording COMPILES into a
bundle and REPLAYS to completion through the desktop backend path,
resolving each click to its recorded target (runs on default CI).
- path: tests/test_capture_adapter.py
proves: >-
The capture->recording bridge over a REAL openadapt-capture session
Expand Down Expand Up @@ -488,9 +523,12 @@ claims:
# ------------------------------- Citrix pixel-only (validating, opt-in gated)
- id: citrix-pixel-validating
claim: >-
Citrix uses the governed pixel-only remote-display contract and enters
design-partner qualification in the customer's exact ICA/HDX application,
with identity, ambiguity refusal, and independent effects above the driver.
Citrix uses a dedicated exact-Workspace-window backend beneath the governed
pixel-only remote-display contract. Required CI covers its readiness gate,
governed execution, durable resume, report binding, and refusal contracts;
the counted no-DOM qualification covers the driver, visual ladder,
independent effects, and drift halts. Live ICA/HDX acceptance remains
separately bound to the exact deployment.
surfaces: [README.md, docs, website]
tier: validating
evidence:
Expand All @@ -503,20 +541,56 @@ claims:
- path: tests/test_pixel_identity_probe.py
proves: >-
The pixel-substrate identity probe behind the on-pixels identity gate.
- path: tests/test_citrix_workspace_backend.py
proves: >-
Required CI covers the dedicated backend preset, exact owner/title
overrides, readiness probe construction, pixel-only capability
boundary, backend factory, and product CLI selectors.
- path: tests/test_run_gate.py
proves: >-
Required CI refuses governed Citrix execution before action when its
readiness binding is absent or blank, admits a sealed bundle carrying
a recorded readiness binding, and reports only the resolved backend
token rather than sensitive target strings.
- path: tests/test_cli_new_commands.py
proves: >-
Required CI restores the recorded Citrix owner, exact title, and
readiness binding through durable approve/resume, and refuses before
backend construction when resumed configuration is incomplete.
- path: tests/test_hosted.py
proves: >-
Required CI binds a completed local Citrix report to the closed
`citrix` execution token without copying target owner, title, or
readiness values into the hosted summary.
- path: tests/e2e/test_citrix_workspace_standin_e2e.py
proves: >-
The dedicated Citrix backend passes three healthy effect-confirmed
record->compile->replay trials and three severe-drift safe-halts over
a no-DOM surface, with zero model calls, false completion, or silent
incorrect success.
- path: benchmark/citrix_workspace/results.json
kind: artifact
proves: >-
The retained six-trial record reports code_readiness_accepted=true
and ica_hdx_accepted=false, keeping driver readiness distinct from a
counted live ICA/HDX qualification.
- path: docs/desktop/CITRIX_PIXEL.md
kind: doc
proves: >-
The Citrix driver model, reusable evidence, design-partner acceptance
contract, independent-effect boundary, and customer-controlled posture.
The Citrix driver model, reusable evidence, exact-deployment
acceptance contract, independent-effect boundary, and
customer-controlled posture.
caveats:
- >-
PIXEL-ONLY: no UIA/DOM crosses to the driver. On-screen OCR read-back is
SAME-SURFACE, not an independent system-of-record check.
- >-
A Parallels VM window is a Citrix ANALOG; real HDX/latency/DPI/lock-
screen drift is not simulated. OPT-IN + infra-gated
(`OAFLOW_CITRIX_PIXEL_E2E=1`, macOS screen-recording + accessibility);
never runs on default CI, and does not replace design-partner validation.
(`OAFLOW_CITRIX_PIXEL_E2E=1` or
`OAFLOW_CITRIX_STANDIN_E2E=1`); neither live-window proof runs on
default CI, and neither replaces a counted live ICA/HDX acceptance
record.

# ------------------------ Windows-host window replay (mock-tested, roadmap)
- id: win32-window-replay-roadmap
Expand All @@ -540,7 +614,7 @@ claims:
proves: >-
The remote-display contract the client implements and the evidence
ladder for qualifying it (Parallels/RDP client-window analog first,
then design-partner Citrix).
then a counted live Citrix host run).
caveats:
- >-
NOT qualified: no counted end-to-end batch on a real Windows host or
Expand Down
Loading