Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
106 changes: 72 additions & 34 deletions docs/concepts/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,15 @@

A compiled workflow does not care what is underneath it. The runtime sits behind
a small four-method `Backend` protocol (screenshot in; click, type, key, scroll
out), so the same bundle, the same resolution ladder, and the same identity gate
run against a browser, a native Windows desktop, or a pixel-only remote session.
out), so the same bundle, resolution ladder, identity gate, effect checks, and
reporting run against browser, Windows, macOS, Linux, RDP, and Citrix/VDI.
Backends are **adapters, not rewrites**.

!!! tip "Selecting a backend on the CLI"
`record`, `replay`, `run`, and `resume` take
[`--backend {web,windows,rdp}`](../reference/cli.md#backend) (with
`--agent-url` for Windows and `--rdp-host` for RDP). The default is `web`.
[`--backend {web,windows,macos,linux,rdp,citrix}`](../reference/cli.md#backend).
Each native backend has an exact target selector; governed Citrix `run`
also requires a readiness marker. The default is `web`.
See [Choosing a backend](../reference/cli.md#backend).

## Vision-first, not vision-only
Expand Down Expand Up @@ -83,6 +84,40 @@ selection, and stale-target rejection. The older `/execute_windows`
compatibility route remains a migration surface and is disabled by default; it
is not the production RPC contract.

### Desktop: native macOS

The native macOS backend captures and drives one exact application window. It
uses the owner plus an optional title substring to bind the target, refuses
ambiguous or non-frontmost windows before input, and fails loud when Screen
Recording or Accessibility permission is unavailable.

!!! info "macOS qualification evidence"
On one macOS 15.7.3 arm64 host, counted candidate `b1b61a5` completed 3/3
exact-byte TextEdit trials and refused a two-window ambiguity without
changing either file, with 0 silent incorrect successes and 0 over-halts.
The immutable original report remains failed because cleanup warnings were
classified as a batch failure; a separate SHA-256-bound adjudication
verified actual cleanup and accepts only the action-effect and
ambiguity-refusal evidence. Review [Flow PR #135](https://github.com/OpenAdaptAI/openadapt-flow/pull/135)
and the [exact adjudication](https://github.com/OpenAdaptAI/openadapt-flow/blob/ca1b522cad215875f7471782283f8f8bb8e6c998/benchmark/macos_native/textedit_counted_3plus1_b1b61a5_20260717.adjudication.json).

### Desktop: native Linux (AT-SPI)

`LinuxBackend` binds one exact AT-SPI application and top-level window, resolves
structural controls, and revalidates the target fingerprint immediately before
actuation. `--linux-allow-physical-input` is an explicit X11-only fallback when
native AT-SPI actuation is unavailable; it is never selected silently.

!!! info "Linux qualification evidence"
Required current-main job
[`linux-atspi-x11`](https://github.com/OpenAdaptAI/openadapt-flow/actions/runs/30059807758/job/89378981573)
at exact commit `3de5fc67` confirmed 3/3 independently checked exact-file
effects, 3/3 ambiguity refusals, and 3/3 stale-target refusals on a fresh
GTK3 process per trial. It recorded 0 silent incorrect successes,
0 over-halts, 0 operator interventions, and 0 model calls. The scope is the
in-tree GTK3 fixture on isolated Ubuntu 24.04 X11/AT-SPI, not Wayland or
arbitrary third-party applications.

### Remote: RDP (pixel-only)

The `FreeRDPBackend` drives a legacy application over **RDP**, read pixel-only:
Expand Down Expand Up @@ -111,37 +146,39 @@ there.
[Flow PR #142](https://github.com/OpenAdaptAI/openadapt-flow/pull/142)
and the [immutable sanitized report](https://github.com/OpenAdaptAI/openadapt-flow/blob/6610d24cebba27918b8ea507b2f05a094057ac85/benchmark/rdp/results_82a658a_20260718.sanitized.json).

A separate full governed lifecycle uses the swappable transport contract
over a real FreeRDP3 client/server round trip. It recorded, compiled, and
replayed a synthetic note write with 3/3 healthy effects and 3/3
drift safe-halts, zero model calls, silent incorrect successes, false
completions, drift writes, or healthy over-halts. That result is scoped to
a synthetic Linux Tk task and simulated drift on the real RDP session; it
is not the Aardwolf transport or a Windows-app qualification. Review
[Flow PR #177](https://github.com/OpenAdaptAI/openadapt-flow/pull/177) and
the [accepted FreeRDP lifecycle report](https://github.com/OpenAdaptAI/openadapt-flow/blob/affedc5f1f0de533a0744deaa8e30a203c91c6b3/benchmark/rdp_ladder/results.json).

### Remote-display / Citrix / VDI (pixel-only)

Citrix and other virtual-desktop surfaces are first-class substrates. They are
driven pixel-first through the same remote-display adapter that captures a named
application window and injects input at screen coordinates, the exact mechanism
the vision-first runtime was built for. The same bundle, resolution ladder,
identity gate, and effect verification apply here as everywhere else.

Each Citrix/VDI deployment is qualified in its real ICA/HDX environment, where
the client's compression and latency, DPI mapping, credentials and lock
screens, and synthetic-input acceptance are exercised against the actual
application: the same "qualify the workflow in its real environment" step every
substrate goes through.

### Native macOS

A native exact-window candidate in [Flow PR #135](https://github.com/OpenAdaptAI/openadapt-flow/pull/135)
captures only a uniquely selected application window, refuses ambiguous or
non-frontmost targets, and fails before input when Screen Recording or
Accessibility access is missing. On one macOS 15.7.3 arm64 host, counted
candidate `b1b61a5` completed 3/3 exact-byte TextEdit trials and refused a
two-window ambiguity without changing either file, with 0 silent incorrect
successes and 0 over-halts.

The [immutable original report](https://github.com/OpenAdaptAI/openadapt-flow/blob/ca1b522cad215875f7471782283f8f8bb8e6c998/benchmark/macos_native/textedit_counted_3plus1_b1b61a5_20260717.json)
still says `status: failed`: its graceful-close cleanup warnings were classified
as a batch failure. A separate [SHA-256-bound adjudication](https://github.com/OpenAdaptAI/openadapt-flow/blob/ca1b522cad215875f7471782283f8f8bb8e6c998/benchmark/macos_native/textedit_counted_3plus1_b1b61a5_20260717.adjudication.json)
verified all exact harness PIDs and the temporary root were absent, preserved
the original result, and accepted only the action-effect and ambiguity-refusal
evidence. This is one-host TextEdit evidence, not clean-machine, design-partner,
production, broad-app, AX structural-resolution, or general macOS acceptance.
`CitrixWorkspaceBackend` is the released, dedicated `--backend citrix` path. It
selects the exact Citrix Workspace/Viewer owner for the host OS, optionally
binds an exact window title, refuses ambiguous targets, requires a visible
readiness marker for governed `run`, and carries the closed target into durable
resume. It is pixel-only by construction, so the same visual resolution,
identity, effect, policy, and halt contracts run without pretending a DOM or
accessibility tree exists.

!!! info "Citrix qualification evidence"
The accepted no-DOM qualification completed 3/3 healthy effects and 3/3
severe-drift safe-halts, with 0 model calls, silent incorrect successes,
false completions, healthy over-halts, or drift writes. Review
[Flow PR #183](https://github.com/OpenAdaptAI/openadapt-flow/pull/183) and
the [immutable report](https://github.com/OpenAdaptAI/openadapt-flow/blob/f6faac5b900b78cbda5980de0e983a9f987285ac/benchmark/citrix_workspace/results.json).

The accepted artifact explicitly records `code_readiness_accepted: true`
and `ica_hdx_accepted: false`. It qualifies the shipped Workspace-window
backend contract over a no-DOM canvas stand-in, not a counted real ICA/HDX
batch. The exact client, codec, latency, DPI, lock/readiness, input,
identity, and effect matrix is qualified separately for consequential
ICA/HDX use.

## Status at a glance

Expand All @@ -150,8 +187,9 @@ production, broad-app, AX structural-resolution, or general macOS acceptance.
| Playwright (web) | Browser DOM | Yes (DOM) | Structured text (DOM) | **First-class**: structured DOM identity |
| `WindowsBackend` | Native Windows | Via UIA | UI Automation `Name`/`Value` | **First-class**: UIA structured identity |
| Native macOS | Native macOS | Exact window candidate; AX candidate metadata | Window identity and pixel/OCR floor | **First-class**: window identity and AX metadata |
| `LinuxBackend` | Native Linux | Yes (AT-SPI) | Exact app/window plus AT-SPI role/name/fingerprint | **First-class**: AT-SPI structured identity |
| `FreeRDPBackend` | Pixel-only network RDP | No | Pixel / OCR floor | **First-class**: pixel/OCR identity floor |
| Citrix / VDI | ICA/HDX remote application | Deployment-dependent | Pixel / OCR floor unless the client exposes more | **First-class**: pixel/OCR identity floor via the client |
| `CitrixWorkspaceBackend` | Citrix Workspace / VDI window | No | Exact Workspace owner/title/readiness plus pixel/OCR floor | **First-class**: governed pixel identity floor |

Every backend runs the same bundle, resolution ladder, identity gate, and effect
verification. What varies per substrate is how high up the
Expand Down
15 changes: 8 additions & 7 deletions docs/concepts/demonstration-compiler.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,10 +54,11 @@ openadapt flow replay bundle --url https://your.app # replay, local, $0
```

On the web substrate shown here, `record` opens a headed browser on your own app
and captures what you do; the same loop records a native Windows desktop or a
pixel-only Citrix/RDP session by choosing a [backend](backends.md) instead of a
`--url`. `compile` turns the recording into a bundle. `replay` runs the bundle
deterministically on the same substrate and writes an illustrated report.
and captures what you do. The same loop records native Windows, macOS, or Linux
applications and pixel-only RDP or Citrix sessions by choosing a
[backend](backends.md) instead of a `--url`. `compile` turns the recording into
a bundle. `replay` runs the bundle deterministically on the same substrate and
writes an illustrated report.

## Vision-first behind a small backend

Expand All @@ -66,9 +67,9 @@ pixel surface (PNG in, clicks and keys out) behind a small `Backend` protocol,
which is why the whole loop runs in CI with no OS permissions. Where a backend
exposes more than pixels (a browser DOM, a native accessibility tree, an API),
OpenAdapt uses that higher-fidelity signal via
[the capability ladder](capability-ladder.md). The web (Playwright), desktop
(Windows/UIA), native macOS, RDP, and Citrix/VDI [backends](backends.md) are all
adapters to the same protocol, not rewrites.
[the capability ladder](capability-ladder.md). The web (Playwright), Windows
UIA, native macOS, native Linux AT-SPI, RDP, and Citrix/VDI
[backends](backends.md) are all adapters to the same protocol, not rewrites.

## An API compiler for the API-less long tail

Expand Down
59 changes: 33 additions & 26 deletions docs/concepts/deployment-matrix.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,25 +6,24 @@ schema. What differs across lanes is data handling, not substrate capability.

## Deployment architecture

| Deployment / substrate | Browser | Windows UIA | Native macOS | RDP | Citrix / VDI |
|---|---|---|---|---|---|
| **OpenAdapt Hosted** | Managed execution, schedules, reports, usage, and billing | Not a public hosted offer; runs in the customer boundary | Not a public hosted offer; runs in the customer boundary | Not a public hosted offer; runs in the customer boundary | Not a public hosted offer; runs in the customer boundary |
| **Customer cloud / BYOC** | Customer runner and storage with managed governance | Customer runner and storage | Customer runner and storage | Customer runner and storage | Customer runner and storage |
| **Self-hosted / on-prem** | Local runner and audit trail | Local runner and audit trail | Local runner and audit trail | Local runner and audit trail | Local runner and audit trail |
| Deployment / substrate | Browser | Windows UIA | Native macOS | Native Linux | RDP | Citrix / VDI |
|---|---|---|---|---|---|---|
| **OpenAdapt Hosted** | Managed execution, schedules, reports, usage, and billing | Customer-controlled runtime connected to Cloud | Customer-controlled runtime connected to Cloud | Customer-controlled runtime connected to Cloud | Customer-controlled runtime connected to Cloud | Customer-controlled runtime connected to Cloud |
| **Customer cloud / BYOC** | Customer runner and storage with managed governance | Customer runner and storage | Customer runner and storage | Customer runner and storage | Customer runner and storage | Customer runner and storage |
| **Self-hosted / on-prem** | Local runner and audit trail | Local runner and audit trail | Local runner and audit trail | Local runner and audit trail | Local runner and audit trail | Local runner and audit trail |

Every substrate is a first-class target of the shared compiler, runner, and
safety gates. Where each lane is available today differs, and the matrix states
that plainly rather than implying uniform hosted availability. The public
$500/month self-serve subscription runs the browser substrate in OpenAdapt's
cloud. Windows, native macOS, RDP, and Citrix/VDI run in the customer boundary:
self-hosted or on-prem today, and control-plane-managed in the customer's own
cloud (BYOC) as that Experimental lane opens. An in-our-cloud desktop runner
cloud. Windows, native macOS, native Linux, RDP, and Citrix/VDI run in a local,
self-hosted, or customer-controlled boundary and can connect to Cloud for
governed operation. An in-our-cloud desktop runner
exists only as an internal, licensing-gated lane, and multi-tenant hosting of the
desktop substrate in OpenAdapt's cloud is deferred; neither is part of any public
offer. Between the customer-boundary lanes the difference is a per-workflow
qualified commercial order rather than a capability gap, and none of these
substrates is an entitlement of the browser subscription. The matrix describes
the product architecture; the
managed-runner offer. None of these native or remote substrates is silently
moved into the shared managed-browser boundary or included as a managed-runner
entitlement. The matrix describes the product architecture; the
[qualification appendix](../get-started/what-works-today.md) and commercial terms
define the accepted workload and entitlement.

Expand Down Expand Up @@ -155,20 +154,28 @@ qualified in its real environment. The published qualification evidence to date:
SHA-256-bound adjudication verified actual cleanup and accepts the action
effect and ambiguity refusal. Review [Flow PR #135](https://github.com/OpenAdaptAI/openadapt-flow/pull/135)
and the [exact adjudication](https://github.com/OpenAdaptAI/openadapt-flow/blob/ca1b522cad215875f7471782283f8f8bb8e6c998/benchmark/macos_native/textedit_counted_3plus1_b1b61a5_20260717.adjudication.json).
- **RDP:** Candidate `82a658a`
passed 3/3 unique-file trials on one Parallels Windows 11 VM over network RDP;
exact guest-tools readback confirmed every effect. Trial latencies were
51.845s, 10.467s, and 7.477s, with 0 failures, 0 silent incorrect successes,
0 over-halts, and 0 model calls. Exact snapshot cleanup passed. Review
[Flow PR #142](https://github.com/OpenAdaptAI/openadapt-flow/pull/142)
and the [immutable sanitized report](https://github.com/OpenAdaptAI/openadapt-flow/blob/6610d24cebba27918b8ea507b2f05a094057ac85/benchmark/rdp/results_82a658a_20260718.sanitized.json).
- **Citrix / VDI (Exploratory):** Driven pixel-first through the same
remote-display adapter and the same identity gate and effect verification as
every other substrate. No real ICA/HDX environment has been qualified yet. Each
Citrix/VDI workflow must be qualified in its real ICA/HDX environment before
consequential use: the client, latency, compression, DPI, lock-screen, and
synthetic-input behavior are exercised against the actual application, the same
real-environment qualification step every substrate goes through.
- **Linux native:** Required current-main `linux-atspi-x11` at exact Flow commit
`3de5fc67` confirmed 3/3 exact-file effects, 3/3 ambiguity refusals, and
3/3 stale-target refusals on the in-tree GTK3/X11 fixture, with 0 silent
incorrect successes, 0 over-halts, 0 operator interventions, and 0 model
calls. Review the [exact required CI job](https://github.com/OpenAdaptAI/openadapt-flow/actions/runs/30059807758/job/89378981573).
- **RDP:** Candidate `82a658a` passed 3/3 unique-file trials on one Parallels
Windows 11 VM over Aardwolf network RDP with exact guest-tools readback. A
separate full governed lifecycle over a real FreeRDP3 round trip passed 3/3
healthy effects and 3/3 drift safe-halts with zero model calls, silent
incorrect successes, false completions, drift writes, or healthy over-halts.
Review [Flow PR #142](https://github.com/OpenAdaptAI/openadapt-flow/pull/142),
[Flow PR #177](https://github.com/OpenAdaptAI/openadapt-flow/pull/177), the
[Aardwolf report](https://github.com/OpenAdaptAI/openadapt-flow/blob/6610d24cebba27918b8ea507b2f05a094057ac85/benchmark/rdp/results_82a658a_20260718.sanitized.json),
and the [FreeRDP report](https://github.com/OpenAdaptAI/openadapt-flow/blob/affedc5f1f0de533a0744deaa8e30a203c91c6b3/benchmark/rdp_ladder/results.json).
- **Citrix / VDI:** The released dedicated `--backend citrix` binds the exact
Workspace window, gates readiness, and preserves the target through durable
resume. Its accepted no-DOM artifact records 3/3 healthy effects and 3/3
drift safe-halts with zero model calls, silent writes, false completions, or
healthy over-halts. It also explicitly records `ica_hdx_accepted: false`:
this is bounded evidence for the shipped backend contract over a no-DOM
stand-in, not a counted real ICA/HDX batch. Review [Flow PR #183](https://github.com/OpenAdaptAI/openadapt-flow/pull/183)
and the [immutable report](https://github.com/OpenAdaptAI/openadapt-flow/blob/f6faac5b900b78cbda5980de0e983a9f987285ac/benchmark/citrix_workspace/results.json).

Review [Qualification evidence](../get-started/what-works-today.md) and the engine's
[published limits](https://github.com/OpenAdaptAI/openadapt-flow/blob/main/docs/LIMITS.md)
Expand Down
13 changes: 7 additions & 6 deletions docs/concepts/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ jump to what you need.

- [__The substrate model__](substrate-model.md)

Routing across web, native Windows, native macOS, and pixel-only Citrix/RDP;
one runner contract, every substrate first-class.
Routing across browser, native Windows, native macOS, native Linux, RDP,
and Citrix/VDI; one runner contract, every substrate first-class.

- [__The deployment matrix__](deployment-matrix.md)

Expand Down Expand Up @@ -73,10 +73,11 @@ jump to what you need.

- [__Backends: where it runs__](backends.md)

Browser (Playwright), native Windows (UIA), native macOS, RDP, and
Citrix/VDI are all first-class substrates behind one backend protocol. The
same bundle, resolution ladder, identity gate, and effect verification run on
each; every workflow is qualified in its real environment.
Browser (Playwright), native Windows (UIA), native macOS, native Linux
(AT-SPI), RDP, and Citrix/VDI are all first-class substrates behind one
backend protocol. The same bundle, resolution ladder, identity gate, and
effect verification run on each; every workflow is qualified in its real
environment.

- [__The on-prem VLM appliance__](vlm-appliance.md)

Expand Down
Loading