feat(benchmark): publish bounded paid-agent aggregates

README.md

@@ -402,8 +402,11 @@ containers, a lockfile, and independent REST, SQL, and exact table-delta
 verification of every write. In the model-free engineering matrix (compiled
 and direct-API arms, baseline plus cosmetic drift), it delivered **12/12
 correct rows with zero silent wrong writes, zero over-halts, and $0 model
-cost**. The paid agent arm and full per-cell trial counts are the next stage
-of the matrix.
+cost**. A separate paid-agent run completed 6/6 correct writes (5/6 clean;
+one post-write cost-cap over-halt) with zero silent incorrect successes. That
+small-N run used a separately provisioned baseline, so it is engineering
+evidence rather than a matched comparison or publication result. See the
+[aggregate agent-arm report](benchmark/agent_arm_verticals/README.md).
 
 The silent-wrong-effect result is also packaged as a standalone, versioned,
 independently runnable benchmark — **EffectBench** — that a third party can

benchmark/agent_arm_verticals/README.md

@@ -0,0 +1,55 @@
+# Paid computer-use agent arm across three reference verticals
+
+Status: **small-N local engineering evidence, not publication evidence**.
+
+On 2026-07-21, a paid `claude-sonnet-5` computer-use agent was evaluated on
+three pinned, synthetic reference environments: Frappe Lending, openIMIS, and
+the pinned-local OpenEMR subset. Each trial began from a reset baseline, drove
+the same task contract used by the compiled reference, and was classified by
+an arm-independent system-of-record oracle. Pixels and the agent's self-report
+did not establish success.
+
+## Aggregate results
+
+| Vertical | Environment | Trials | Primary outcome | Silent incorrect success | Mean list-price model cost | Mean wall time |
+|---|---|---:|---|---:|---:|---:|
+| Insurance | openIMIS | 3 | **3/3 correct** | 0/3 | $0.4793/run | 69.8 s |
+| Lending | Frappe Lending | 6 | **6/6 correct writes; 5/6 clean** (one post-write cost-cap over-halt) | 0/6 | $0.4240/run | 67.3 s |
+| Healthcare | OpenEMR (local) | 6 | **0/6 correct; 6/6 missing write** | 0/6 | $0.8901/run | 113.3 s |
+
+Across these 15 trials, the agent recorded zero silent incorrect successes. The
+only classified over-halt was the one post-write lending cost-cap halt; the
+insurance and OpenEMR arms recorded 0/3 and 0/6 over-halts, respectively. The
+OpenEMR result is a negative result: the bounded agent exhausted its action
+budget without completing the write in every trial.
+
+## Method and public/private boundary
+
+The public, generic agent-baseline mechanism remains in
+`openadapt_flow/benchmark/agent_baseline.py`. It provides explicit paid-run
+opt-in, action and cost caps, usage accounting, and an agent self-report that
+is never trusted for scoring. The public benchmark schemas and aggregate
+failure taxonomy also remain available for independently authored tasks.
+
+The application-specific driver adapters and oracle wiring, raw per-run
+JSON/JSONL, environment fingerprints, screenshots, and detailed cost ledger are
+private evidence. They are retained with file-level provenance and hashes in
+the private `OpenAdaptAI/openadapt-corpus` repository and are intentionally
+excluded from this public repository and its distributions.
+
+## Caveats
+
+- These agent trials used freshly provisioned baselines separate from the
+  earlier compiled/API model-free subsets. The columns must not be presented as
+  a matched three-arm comparison.
+- The sample is small: 3 trials for insurance and 6 each for lending and
+  healthcare.
+- The environments used synthetic data on one local host. This is not customer,
+  regulated-production, clean-machine, or broad reliability evidence.
+- A publication comparison still requires a preregistered, matched protocol
+  with at least 10 fresh trials per task and condition, a common baseline,
+  independent review, and complete reporting of incorrect success and
+  over-halt.
+
+This aggregate report does not claim publication readiness, certification,
+commercial availability, or superiority over computer-use agents generally.

benchmark/frappe_lending/.gitignore

@@ -4,3 +4,4 @@
 /recording-live*/
 /bundle-live*/
 /results-*/
+/agent-arm/

benchmark/frappe_lending/README.md

@@ -4,9 +4,34 @@ Status: **local model-free initial engineering matrix complete; not publication
 evidence**. On 2026-07-16, the compiled and direct-API arms each completed three
 trials under the pinned baseline and three under `ui_cosmetic_v1`: 12/12 rows
 were `correct`, with zero silent incorrect successes, zero over-halts, zero
-model calls, and $0 model cost. The agent arm was intentionally omitted, so the
-result records `full_matrix_complete: false` and `publication_ready: false`.
-Publication still requires the paid agent arm and 10 fresh trials per cell.
+model calls, and $0 model cost.
+
+On 2026-07-21 the **paid computer-use agent arm was run** (it is no longer
+omitted). On a separately provisioned synthetic baseline, the
+`claude-sonnet-5` agent completed 3 trials under the pinned condition and 3
+under `ui_cosmetic_v1` (6 total), driving the same Loan Application task and
+being scored by the same arm-independent effect contract:
+
+| metric | value |
+|---|---|
+| trials | 6 (2 conditions x 3) |
+| primary outcome | 6/6 `correct` (the target Loan Application persisted every time) |
+| clean success | 5/6; 1 run over-halted at the cost cap after the record had already persisted |
+| over-halt | 1/6 (the same post-write cost-cap halt) |
+| silent incorrect success | 0/6 |
+| model cost / run | $0.4240 (list price) |
+| total model cost | $2.5437 |
+| latency (wall) mean | 67.3 s |
+| model | `claude-sonnet-5`, `computer_20251124`, <=30 actions/run, <=$2.00/run cap |
+
+This agent arm was measured separately from the earlier compiled/API
+model-free subset; a fully matched single-baseline three-arm matrix at 10
+trials per cell remains the publication bar. Raw per-run rows, environment
+fingerprints, application-specific adapter/oracle wiring, and the detailed cost
+ledger are retained in the private `OpenAdaptAI/openadapt-corpus` repository.
+The public generic cost-capped agent mechanism remains in
+`openadapt_flow/benchmark/agent_baseline.py`. See the
+[aggregate report](../agent_arm_verticals/README.md).
 
 The accepted run used baseline SHA-256
 `7fd6c965f6b7a11f54e451cdc73fdf65f88d9883dc5f8eb5b2055b3cd4be8b83`.
@@ -126,7 +151,7 @@ conditions, with zero model calls and $0 model cost.
 | reset | exact hashed SQL snapshot before every trial | exact hashed SQL snapshot before every trial |
 | oracle | separate read-only REST client + direct SQL + exact table deltas | separate read-only REST user + direct SQL + exact table deltas |
 | completed subset | compiled/API, baseline + drift, 3/cell: 12/12 correct | compiled/API, baseline + drift, 3/cell: 12/12 correct |
-| agent arm | omitted | omitted |
+| paid-agent evidence outside the model-free subset | 0/6 correct, all missing-write, 0 silent incorrect (separate 2026-07-21 baseline) | 6/6 correct writes, 5/6 clean, 0 silent incorrect (separate 2026-07-21 baseline) |
 | publication ready | **No** | **No** |
 
 This is matched local engineering evidence, not a cross-application reliability

benchmark/openemr_local/.gitignore

@@ -5,3 +5,4 @@ recording-live*/
 bundle-live*/
 results-*/
 api-delta-probe-*/
+agent-arm/

benchmark/openemr_local/README.md

@@ -4,9 +4,37 @@ Status: **local model-free initial engineering matrix complete; not publication
 evidence**. On 2026-07-16, the compiled and direct-API arms each completed three
 trials under the pinned baseline and three under `ui_cosmetic_v1`: 12/12 rows
 were `correct`, with zero silent incorrect successes, zero over-halts, zero
-model calls, and $0 model cost. The agent arm was intentionally omitted, so the
-result records `full_matrix_complete: false` and `publication_ready: false`.
-Publication still requires the paid agent arm and 10 fresh trials per cell.
+model calls, and $0 model cost.
+
+On 2026-07-21 the **paid computer-use agent arm was run** (it is no longer
+omitted). On a separately provisioned synthetic baseline, the
+`claude-sonnet-5` agent drove the same patient-registration task and was scored
+by the same arm-independent effect contract:
+
+| metric | value |
+|---|---|
+| trials | 6 (2 conditions x 3) |
+| primary outcome | 0/6 `correct`; 6/6 `missing_write` |
+| behaviour | every run exhausted its 35-action budget without saving a patient; the agent never got past OpenEMR's duplicate-search / confirm-create step |
+| over-halt | 0/6 (all failures were classified `missing_write`) |
+| silent incorrect success | 0/6 (it wrote no wrong or duplicate patient) |
+| model cost / run | $0.8901 (list price) |
+| total model cost | $5.3408 |
+| latency (wall) mean | 113.3 s |
+| model | `claude-sonnet-5`, `computer_20251124`, <=35 actions/run, <=$1.50/run cap |
+
+This is an honest negative result: on this heavily framed local OpenEMR task,
+the bounded-budget computer-use agent failed to complete the write. The earlier
+model-free subset recorded deterministic compiled completion at $0, but the
+agent arm used a separately provisioned baseline, so this is not a matched
+comparison and no compiled result is claimed on the agent baseline. A fully
+matched single-baseline three-arm matrix at 10 trials per cell remains the
+publication bar. Raw per-run rows, environment fingerprints,
+application-specific adapter/oracle wiring, and the detailed cost ledger are
+retained in the private `OpenAdaptAI/openadapt-corpus` repository. The public
+generic cost-capped agent mechanism remains in
+`openadapt_flow/benchmark/agent_baseline.py`. See the
+[aggregate report](../agent_arm_verticals/README.md).
 
 The accepted run used baseline SHA-256
 `8d2901490a0a6a6044e94b6a8a1663436b7dacedda4f2fe1fb8c48405165011d`.

benchmark/openimis_claims/.gitignore

@@ -0,0 +1,4 @@
+state/
+out/
+__pycache__/
+agent-arm/

benchmark/openimis_claims/README.md

@@ -8,11 +8,34 @@ entered through the browser UI (claims intake), recorded once, compiled, and
 replayed — with success established only by a direct SQL read of the claim
 row, never by pixels or actor self-report.
 
-There is no timing matrix, no agent arm, and no publication protocol here. Any
-future reliability claim would need the full matched protocol the Frappe
-Lending and OpenEMR benchmarks define (paid agent arm, 10 fresh trials per
-cell, snapshot-reset baselines). Nothing in this directory is publication
-evidence.
+On 2026-07-21 the **paid computer-use agent arm was run** on this environment,
+so the insurance vertical is no longer agent-arm-free. On the pinned synthetic
+reference, the `claude-sonnet-5` agent drove the same claim-intake task and was
+scored by the same arm-independent effect contract:
+
+| metric | value |
+|---|---|
+| trials | 3 |
+| primary outcome | 3/3 `correct` (exactly one 'Entered' claim for the synthetic policyholder each time) |
+| over-halt | 0/3 |
+| silent incorrect success | 0/3 (no duplicate / wrong / collateral claim) |
+| model cost / run | $0.4793 (list price) |
+| total model cost | $1.4380 |
+| latency (wall) mean | 69.8 s |
+| model | `claude-sonnet-5`, `computer_20251124`, <=30 actions/run, <=$1.50/run cap |
+
+Raw per-run rows, environment fingerprints, application-specific
+adapter/oracle wiring, and the detailed cost ledger are retained in the private
+`OpenAdaptAI/openadapt-corpus` repository. The public generic cost-capped agent
+mechanism remains in
+`openadapt_flow/benchmark/agent_baseline.py`. See the
+[aggregate report](../agent_arm_verticals/README.md).
+
+There is still no full timing matrix and no publication protocol here: a
+publication reliability claim would need the full matched protocol the Frappe
+Lending and OpenEMR benchmarks define (compiled + agent + API arms, 10 fresh
+trials per cell, snapshot-reset baselines). Nothing in this directory is
+publication evidence.
 
 ## The application
 

benchmark/openimis_claims/fixture.py

@@ -9,7 +9,8 @@
 
 It is a DEMO/reference environment, not a benchmark:
 
-* no timing matrix, no agent arm, no publication protocol;
+* no matched timing matrix or publication protocol; a separate 3-trial paid
+  agent run is reported only as small-N aggregate engineering evidence;
 * the success oracle is a direct SQL read of the claim row (actor self-report
   and pixels never establish success), plus an exact one-row cardinality check;
 * every value is synthetic. The upstream openIMIS demo dataset is a fictional

scripts/check_release_consistency.py

@@ -31,13 +31,14 @@
 )
 # Private, deployment-derived hardening artifacts -- the GROWN failure corpus
 # from real deployments, the TUNED metamorphic-adversary parameters/weights,
-# deployment-derived THRESHOLDS, effect-verification oracle RECIPES, and
-# customer/deployment-derived real-EMR datasets -- live ONLY in the private
+# deployment-derived THRESHOLDS, effect-verification oracle RECIPES,
+# customer/deployment-derived real-EMR datasets, and raw paid-agent evidence or
+# per-system driver/oracle recipes -- live ONLY in the private
 # OpenAdaptAI/openadapt-corpus repo and must never ride inside an MIT wheel or
-# sdist. Reproducible synthetic fixtures and fake-patient public-demo evidence
-# remain public; they are mechanisms/samples, not customer-derived data. This
-# mirrors the AGPL boundary above: both a path-token check and a content-signature
-# check, so a rename cannot smuggle a private artifact in.
+# sdist. Reproducible synthetic fixtures, generic harnesses, fake-patient
+# public-demo evidence, and bounded aggregates remain public. This mirrors the
+# AGPL boundary above: both a path-token check and a content-signature check, so
+# a rename cannot smuggle a private artifact in.
 PRIVATE_DISTRIBUTION_PATH_TOKENS = (
     "openadapt-corpus",
     "adversary_corpus",
@@ -53,6 +54,13 @@
     "real_emr",
     "enterprise_productionized",
     "control_plane",
+    "paid_agent_evidence",
+    "agent-arm/",
+    "rows.jsonl",
+    "cost_ledger",
+    "frappe_agent_arm.py",
+    "openemr_agent_arm.py",
+    "openimis_agent_arm.py",
 )
 PRIVATE_DISTRIBUTION_PATH_SEGMENTS = frozenset({"private", ".private"})
 PRIVATE_DISTRIBUTION_EXACT_PATHS = frozenset(
@@ -301,7 +309,7 @@ def _has_private_path_segment(path: str) -> bool:
 
 
 def _private_distribution_hits(members: set[str], signature_hits: set[str]) -> set[str]:
-    """Members that are private deployment-derived hardening artifacts."""
+    """Members that cross the private source-availability boundary."""
     hits = {
         member
         for member in members
@@ -812,7 +820,51 @@ def validate_public_source_tree(root: Path = ROOT) -> None:
         "tests/test_openimis_claims_fixture.py",
     }
 )
-FORBIDDEN_SDIST_PREFIXES = ("benchmark/openimis_claims/",)
+FORBIDDEN_SDIST_PREFIXES = (
+    "benchmark/openimis_claims/",
+    "benchmark/frappe_lending/agent-arm/",
+    "benchmark/openemr_local/agent-arm/",
+)
+
+FORBIDDEN_PUBLIC_SOURCE_PATHS = frozenset(
+    {
+        "scripts/frappe_agent_arm.py",
+        "scripts/openemr_agent_arm.py",
+        "scripts/openimis_agent_arm.py",
+        "benchmark/agent_arm_verticals/COST_LEDGER.md",
+    }
+)
+
+
+def validate_public_source_policy(root: Path = ROOT) -> None:
+    """Refuse private paid-agent evidence and per-system recipes in source.
+
+    Archive inspection remains mandatory, but this earlier guard prevents a
+    sensitive path from being committed to the public repository even when a
+    packaging exclusion would otherwise hide it from the wheel or sdist.
+    """
+    hits = {
+        relative
+        for relative in FORBIDDEN_PUBLIC_SOURCE_PATHS
+        if (root / relative).is_file()
+    }
+    hits.update(
+        path.relative_to(root).as_posix()
+        for path in root.rglob("rows.jsonl")
+        if ".git" not in path.parts
+    )
+    hits.update(
+        path.relative_to(root).as_posix()
+        for path in root.glob("benchmark/**/agent-arm/**/*")
+        if path.is_file()
+    )
+    if hits:
+        raise ValueError(
+            "public source contains private paid-agent per-run evidence, "
+            "environment-linked result rows, detailed cost ledgers, or "
+            "per-system driver recipes that belong only in the private "
+            f"OpenAdaptAI/openadapt-corpus repo: {sorted(hits)}"
+        )
 
 
 def _expected_license_bytes(license_file: Path | None = None) -> bytes:
@@ -1005,9 +1057,10 @@ def validate_sdist_license_boundary(
     private = _private_distribution_hits(members, private_signature_hits)
     if private:
         raise ValueError(
-            "source distribution contains private, deployment-derived hardening "
+            "source distribution contains private source-policy "
             "material (grown corpus / tuned adversary / thresholds / oracle "
-            "recipes / real-EMR datasets) that belongs only in the private "
+            "recipes / real-EMR datasets / paid-agent raw evidence) that "
+            "belongs only in the private "
             f"OpenAdaptAI/openadapt-corpus repo: {sorted(private)}"
         )
     repository_only = _repository_only_evaluation_hits(members)
@@ -1111,9 +1164,10 @@ def validate_wheel_license_boundary(
     private = _private_distribution_hits(members, private_signature_hits)
     if private:
         raise ValueError(
-            "wheel contains private, deployment-derived hardening material "
+            "wheel contains private source-policy material "
             "(grown corpus / tuned adversary / thresholds / oracle recipes / "
-            "real-EMR datasets) that belongs only in the private "
+            "real-EMR datasets / paid-agent raw evidence) that belongs only "
+            "in the private "
             f"OpenAdaptAI/openadapt-corpus repo: {sorted(private)}"
         )
     repository_only = _repository_only_evaluation_hits(members)
@@ -1255,6 +1309,11 @@ def main() -> int:
     if args.license_file is not None:
         parser.error("--license-file requires --validate-dist-dir")
 
+    try:
+        validate_public_source_policy()
+    except ValueError as error:
+        parser.error(str(error))
+
     if args.sync:
         sync_lock_version()