diff --git a/README.md b/README.md index a73462a6..c2a196ca 100644 --- a/README.md +++ b/README.md @@ -402,8 +402,11 @@ containers, a lockfile, and independent REST, SQL, and exact table-delta verification of every write. In the model-free engineering matrix (compiled and direct-API arms, baseline plus cosmetic drift), it delivered **12/12 correct rows with zero silent wrong writes, zero over-halts, and $0 model -cost**. The paid agent arm and full per-cell trial counts are the next stage -of the matrix. +cost**. A separate paid-agent run completed 6/6 correct writes (5/6 clean; +one post-write cost-cap over-halt) with zero silent incorrect successes. That +small-N run used a separately provisioned baseline, so it is engineering +evidence rather than a matched comparison or publication result. See the +[aggregate agent-arm report](benchmark/agent_arm_verticals/README.md). The silent-wrong-effect result is also packaged as a standalone, versioned, independently runnable benchmark — **EffectBench** — that a third party can diff --git a/benchmark/agent_arm_verticals/README.md b/benchmark/agent_arm_verticals/README.md new file mode 100644 index 00000000..fbfe9544 --- /dev/null +++ b/benchmark/agent_arm_verticals/README.md @@ -0,0 +1,55 @@ +# Paid computer-use agent arm across three reference verticals + +Status: **small-N local engineering evidence, not publication evidence**. + +On 2026-07-21, a paid `claude-sonnet-5` computer-use agent was evaluated on +three pinned, synthetic reference environments: Frappe Lending, openIMIS, and +the pinned-local OpenEMR subset. Each trial began from a reset baseline, drove +the same task contract used by the compiled reference, and was classified by +an arm-independent system-of-record oracle. Pixels and the agent's self-report +did not establish success. + +## Aggregate results + +| Vertical | Environment | Trials | Primary outcome | Silent incorrect success | Mean list-price model cost | Mean wall time | +|---|---|---:|---|---:|---:|---:| +| Insurance | openIMIS | 3 | **3/3 correct** | 0/3 | $0.4793/run | 69.8 s | +| Lending | Frappe Lending | 6 | **6/6 correct writes; 5/6 clean** (one post-write cost-cap over-halt) | 0/6 | $0.4240/run | 67.3 s | +| Healthcare | OpenEMR (local) | 6 | **0/6 correct; 6/6 missing write** | 0/6 | $0.8901/run | 113.3 s | + +Across these 15 trials, the agent recorded zero silent incorrect successes. The +only classified over-halt was the one post-write lending cost-cap halt; the +insurance and OpenEMR arms recorded 0/3 and 0/6 over-halts, respectively. The +OpenEMR result is a negative result: the bounded agent exhausted its action +budget without completing the write in every trial. + +## Method and public/private boundary + +The public, generic agent-baseline mechanism remains in +`openadapt_flow/benchmark/agent_baseline.py`. It provides explicit paid-run +opt-in, action and cost caps, usage accounting, and an agent self-report that +is never trusted for scoring. The public benchmark schemas and aggregate +failure taxonomy also remain available for independently authored tasks. + +The application-specific driver adapters and oracle wiring, raw per-run +JSON/JSONL, environment fingerprints, screenshots, and detailed cost ledger are +private evidence. They are retained with file-level provenance and hashes in +the private `OpenAdaptAI/openadapt-corpus` repository and are intentionally +excluded from this public repository and its distributions. + +## Caveats + +- These agent trials used freshly provisioned baselines separate from the + earlier compiled/API model-free subsets. The columns must not be presented as + a matched three-arm comparison. +- The sample is small: 3 trials for insurance and 6 each for lending and + healthcare. +- The environments used synthetic data on one local host. This is not customer, + regulated-production, clean-machine, or broad reliability evidence. +- A publication comparison still requires a preregistered, matched protocol + with at least 10 fresh trials per task and condition, a common baseline, + independent review, and complete reporting of incorrect success and + over-halt. + +This aggregate report does not claim publication readiness, certification, +commercial availability, or superiority over computer-use agents generally. diff --git a/benchmark/frappe_lending/.gitignore b/benchmark/frappe_lending/.gitignore index 7fa1a5a4..46bc4684 100644 --- a/benchmark/frappe_lending/.gitignore +++ b/benchmark/frappe_lending/.gitignore @@ -4,3 +4,4 @@ /recording-live*/ /bundle-live*/ /results-*/ +/agent-arm/ diff --git a/benchmark/frappe_lending/README.md b/benchmark/frappe_lending/README.md index 77679651..1504ddf3 100644 --- a/benchmark/frappe_lending/README.md +++ b/benchmark/frappe_lending/README.md @@ -4,9 +4,34 @@ Status: **local model-free initial engineering matrix complete; not publication evidence**. On 2026-07-16, the compiled and direct-API arms each completed three trials under the pinned baseline and three under `ui_cosmetic_v1`: 12/12 rows were `correct`, with zero silent incorrect successes, zero over-halts, zero -model calls, and $0 model cost. The agent arm was intentionally omitted, so the -result records `full_matrix_complete: false` and `publication_ready: false`. -Publication still requires the paid agent arm and 10 fresh trials per cell. +model calls, and $0 model cost. + +On 2026-07-21 the **paid computer-use agent arm was run** (it is no longer +omitted). On a separately provisioned synthetic baseline, the +`claude-sonnet-5` agent completed 3 trials under the pinned condition and 3 +under `ui_cosmetic_v1` (6 total), driving the same Loan Application task and +being scored by the same arm-independent effect contract: + +| metric | value | +|---|---| +| trials | 6 (2 conditions x 3) | +| primary outcome | 6/6 `correct` (the target Loan Application persisted every time) | +| clean success | 5/6; 1 run over-halted at the cost cap after the record had already persisted | +| over-halt | 1/6 (the same post-write cost-cap halt) | +| silent incorrect success | 0/6 | +| model cost / run | $0.4240 (list price) | +| total model cost | $2.5437 | +| latency (wall) mean | 67.3 s | +| model | `claude-sonnet-5`, `computer_20251124`, <=30 actions/run, <=$2.00/run cap | + +This agent arm was measured separately from the earlier compiled/API +model-free subset; a fully matched single-baseline three-arm matrix at 10 +trials per cell remains the publication bar. Raw per-run rows, environment +fingerprints, application-specific adapter/oracle wiring, and the detailed cost +ledger are retained in the private `OpenAdaptAI/openadapt-corpus` repository. +The public generic cost-capped agent mechanism remains in +`openadapt_flow/benchmark/agent_baseline.py`. See the +[aggregate report](../agent_arm_verticals/README.md). The accepted run used baseline SHA-256 `7fd6c965f6b7a11f54e451cdc73fdf65f88d9883dc5f8eb5b2055b3cd4be8b83`. @@ -126,7 +151,7 @@ conditions, with zero model calls and $0 model cost. | reset | exact hashed SQL snapshot before every trial | exact hashed SQL snapshot before every trial | | oracle | separate read-only REST client + direct SQL + exact table deltas | separate read-only REST user + direct SQL + exact table deltas | | completed subset | compiled/API, baseline + drift, 3/cell: 12/12 correct | compiled/API, baseline + drift, 3/cell: 12/12 correct | -| agent arm | omitted | omitted | +| paid-agent evidence outside the model-free subset | 0/6 correct, all missing-write, 0 silent incorrect (separate 2026-07-21 baseline) | 6/6 correct writes, 5/6 clean, 0 silent incorrect (separate 2026-07-21 baseline) | | publication ready | **No** | **No** | This is matched local engineering evidence, not a cross-application reliability diff --git a/benchmark/openemr_local/.gitignore b/benchmark/openemr_local/.gitignore index f4942330..73b01aa8 100644 --- a/benchmark/openemr_local/.gitignore +++ b/benchmark/openemr_local/.gitignore @@ -5,3 +5,4 @@ recording-live*/ bundle-live*/ results-*/ api-delta-probe-*/ +agent-arm/ diff --git a/benchmark/openemr_local/README.md b/benchmark/openemr_local/README.md index 43885723..6a9441c4 100644 --- a/benchmark/openemr_local/README.md +++ b/benchmark/openemr_local/README.md @@ -4,9 +4,37 @@ Status: **local model-free initial engineering matrix complete; not publication evidence**. On 2026-07-16, the compiled and direct-API arms each completed three trials under the pinned baseline and three under `ui_cosmetic_v1`: 12/12 rows were `correct`, with zero silent incorrect successes, zero over-halts, zero -model calls, and $0 model cost. The agent arm was intentionally omitted, so the -result records `full_matrix_complete: false` and `publication_ready: false`. -Publication still requires the paid agent arm and 10 fresh trials per cell. +model calls, and $0 model cost. + +On 2026-07-21 the **paid computer-use agent arm was run** (it is no longer +omitted). On a separately provisioned synthetic baseline, the +`claude-sonnet-5` agent drove the same patient-registration task and was scored +by the same arm-independent effect contract: + +| metric | value | +|---|---| +| trials | 6 (2 conditions x 3) | +| primary outcome | 0/6 `correct`; 6/6 `missing_write` | +| behaviour | every run exhausted its 35-action budget without saving a patient; the agent never got past OpenEMR's duplicate-search / confirm-create step | +| over-halt | 0/6 (all failures were classified `missing_write`) | +| silent incorrect success | 0/6 (it wrote no wrong or duplicate patient) | +| model cost / run | $0.8901 (list price) | +| total model cost | $5.3408 | +| latency (wall) mean | 113.3 s | +| model | `claude-sonnet-5`, `computer_20251124`, <=35 actions/run, <=$1.50/run cap | + +This is an honest negative result: on this heavily framed local OpenEMR task, +the bounded-budget computer-use agent failed to complete the write. The earlier +model-free subset recorded deterministic compiled completion at $0, but the +agent arm used a separately provisioned baseline, so this is not a matched +comparison and no compiled result is claimed on the agent baseline. A fully +matched single-baseline three-arm matrix at 10 trials per cell remains the +publication bar. Raw per-run rows, environment fingerprints, +application-specific adapter/oracle wiring, and the detailed cost ledger are +retained in the private `OpenAdaptAI/openadapt-corpus` repository. The public +generic cost-capped agent mechanism remains in +`openadapt_flow/benchmark/agent_baseline.py`. See the +[aggregate report](../agent_arm_verticals/README.md). The accepted run used baseline SHA-256 `8d2901490a0a6a6044e94b6a8a1663436b7dacedda4f2fe1fb8c48405165011d`. diff --git a/benchmark/openimis_claims/.gitignore b/benchmark/openimis_claims/.gitignore new file mode 100644 index 00000000..eb1e06a6 --- /dev/null +++ b/benchmark/openimis_claims/.gitignore @@ -0,0 +1,4 @@ +state/ +out/ +__pycache__/ +agent-arm/ diff --git a/benchmark/openimis_claims/README.md b/benchmark/openimis_claims/README.md index c3f81c93..ceabb5b8 100644 --- a/benchmark/openimis_claims/README.md +++ b/benchmark/openimis_claims/README.md @@ -8,11 +8,34 @@ entered through the browser UI (claims intake), recorded once, compiled, and replayed — with success established only by a direct SQL read of the claim row, never by pixels or actor self-report. -There is no timing matrix, no agent arm, and no publication protocol here. Any -future reliability claim would need the full matched protocol the Frappe -Lending and OpenEMR benchmarks define (paid agent arm, 10 fresh trials per -cell, snapshot-reset baselines). Nothing in this directory is publication -evidence. +On 2026-07-21 the **paid computer-use agent arm was run** on this environment, +so the insurance vertical is no longer agent-arm-free. On the pinned synthetic +reference, the `claude-sonnet-5` agent drove the same claim-intake task and was +scored by the same arm-independent effect contract: + +| metric | value | +|---|---| +| trials | 3 | +| primary outcome | 3/3 `correct` (exactly one 'Entered' claim for the synthetic policyholder each time) | +| over-halt | 0/3 | +| silent incorrect success | 0/3 (no duplicate / wrong / collateral claim) | +| model cost / run | $0.4793 (list price) | +| total model cost | $1.4380 | +| latency (wall) mean | 69.8 s | +| model | `claude-sonnet-5`, `computer_20251124`, <=30 actions/run, <=$1.50/run cap | + +Raw per-run rows, environment fingerprints, application-specific +adapter/oracle wiring, and the detailed cost ledger are retained in the private +`OpenAdaptAI/openadapt-corpus` repository. The public generic cost-capped agent +mechanism remains in +`openadapt_flow/benchmark/agent_baseline.py`. See the +[aggregate report](../agent_arm_verticals/README.md). + +There is still no full timing matrix and no publication protocol here: a +publication reliability claim would need the full matched protocol the Frappe +Lending and OpenEMR benchmarks define (compiled + agent + API arms, 10 fresh +trials per cell, snapshot-reset baselines). Nothing in this directory is +publication evidence. ## The application diff --git a/benchmark/openimis_claims/fixture.py b/benchmark/openimis_claims/fixture.py index 93d04212..a5619578 100644 --- a/benchmark/openimis_claims/fixture.py +++ b/benchmark/openimis_claims/fixture.py @@ -9,7 +9,8 @@ It is a DEMO/reference environment, not a benchmark: -* no timing matrix, no agent arm, no publication protocol; +* no matched timing matrix or publication protocol; a separate 3-trial paid + agent run is reported only as small-N aggregate engineering evidence; * the success oracle is a direct SQL read of the claim row (actor self-report and pixels never establish success), plus an exact one-row cardinality check; * every value is synthetic. The upstream openIMIS demo dataset is a fictional diff --git a/scripts/check_release_consistency.py b/scripts/check_release_consistency.py index 0fb8ed2c..c87d3bea 100644 --- a/scripts/check_release_consistency.py +++ b/scripts/check_release_consistency.py @@ -31,13 +31,14 @@ ) # Private, deployment-derived hardening artifacts -- the GROWN failure corpus # from real deployments, the TUNED metamorphic-adversary parameters/weights, -# deployment-derived THRESHOLDS, effect-verification oracle RECIPES, and -# customer/deployment-derived real-EMR datasets -- live ONLY in the private +# deployment-derived THRESHOLDS, effect-verification oracle RECIPES, +# customer/deployment-derived real-EMR datasets, and raw paid-agent evidence or +# per-system driver/oracle recipes -- live ONLY in the private # OpenAdaptAI/openadapt-corpus repo and must never ride inside an MIT wheel or -# sdist. Reproducible synthetic fixtures and fake-patient public-demo evidence -# remain public; they are mechanisms/samples, not customer-derived data. This -# mirrors the AGPL boundary above: both a path-token check and a content-signature -# check, so a rename cannot smuggle a private artifact in. +# sdist. Reproducible synthetic fixtures, generic harnesses, fake-patient +# public-demo evidence, and bounded aggregates remain public. This mirrors the +# AGPL boundary above: both a path-token check and a content-signature check, so +# a rename cannot smuggle a private artifact in. PRIVATE_DISTRIBUTION_PATH_TOKENS = ( "openadapt-corpus", "adversary_corpus", @@ -53,6 +54,13 @@ "real_emr", "enterprise_productionized", "control_plane", + "paid_agent_evidence", + "agent-arm/", + "rows.jsonl", + "cost_ledger", + "frappe_agent_arm.py", + "openemr_agent_arm.py", + "openimis_agent_arm.py", ) PRIVATE_DISTRIBUTION_PATH_SEGMENTS = frozenset({"private", ".private"}) PRIVATE_DISTRIBUTION_EXACT_PATHS = frozenset( @@ -301,7 +309,7 @@ def _has_private_path_segment(path: str) -> bool: def _private_distribution_hits(members: set[str], signature_hits: set[str]) -> set[str]: - """Members that are private deployment-derived hardening artifacts.""" + """Members that cross the private source-availability boundary.""" hits = { member for member in members @@ -812,7 +820,51 @@ def validate_public_source_tree(root: Path = ROOT) -> None: "tests/test_openimis_claims_fixture.py", } ) -FORBIDDEN_SDIST_PREFIXES = ("benchmark/openimis_claims/",) +FORBIDDEN_SDIST_PREFIXES = ( + "benchmark/openimis_claims/", + "benchmark/frappe_lending/agent-arm/", + "benchmark/openemr_local/agent-arm/", +) + +FORBIDDEN_PUBLIC_SOURCE_PATHS = frozenset( + { + "scripts/frappe_agent_arm.py", + "scripts/openemr_agent_arm.py", + "scripts/openimis_agent_arm.py", + "benchmark/agent_arm_verticals/COST_LEDGER.md", + } +) + + +def validate_public_source_policy(root: Path = ROOT) -> None: + """Refuse private paid-agent evidence and per-system recipes in source. + + Archive inspection remains mandatory, but this earlier guard prevents a + sensitive path from being committed to the public repository even when a + packaging exclusion would otherwise hide it from the wheel or sdist. + """ + hits = { + relative + for relative in FORBIDDEN_PUBLIC_SOURCE_PATHS + if (root / relative).is_file() + } + hits.update( + path.relative_to(root).as_posix() + for path in root.rglob("rows.jsonl") + if ".git" not in path.parts + ) + hits.update( + path.relative_to(root).as_posix() + for path in root.glob("benchmark/**/agent-arm/**/*") + if path.is_file() + ) + if hits: + raise ValueError( + "public source contains private paid-agent per-run evidence, " + "environment-linked result rows, detailed cost ledgers, or " + "per-system driver recipes that belong only in the private " + f"OpenAdaptAI/openadapt-corpus repo: {sorted(hits)}" + ) def _expected_license_bytes(license_file: Path | None = None) -> bytes: @@ -1005,9 +1057,10 @@ def validate_sdist_license_boundary( private = _private_distribution_hits(members, private_signature_hits) if private: raise ValueError( - "source distribution contains private, deployment-derived hardening " + "source distribution contains private source-policy " "material (grown corpus / tuned adversary / thresholds / oracle " - "recipes / real-EMR datasets) that belongs only in the private " + "recipes / real-EMR datasets / paid-agent raw evidence) that " + "belongs only in the private " f"OpenAdaptAI/openadapt-corpus repo: {sorted(private)}" ) repository_only = _repository_only_evaluation_hits(members) @@ -1111,9 +1164,10 @@ def validate_wheel_license_boundary( private = _private_distribution_hits(members, private_signature_hits) if private: raise ValueError( - "wheel contains private, deployment-derived hardening material " + "wheel contains private source-policy material " "(grown corpus / tuned adversary / thresholds / oracle recipes / " - "real-EMR datasets) that belongs only in the private " + "real-EMR datasets / paid-agent raw evidence) that belongs only " + "in the private " f"OpenAdaptAI/openadapt-corpus repo: {sorted(private)}" ) repository_only = _repository_only_evaluation_hits(members) @@ -1255,6 +1309,11 @@ def main() -> int: if args.license_file is not None: parser.error("--license-file requires --validate-dist-dir") + try: + validate_public_source_policy() + except ValueError as error: + parser.error(str(error)) + if args.sync: sync_lock_version() diff --git a/tests/test_release_contract.py b/tests/test_release_contract.py index ce78ed22..1b4e6779 100644 --- a/tests/test_release_contract.py +++ b/tests/test_release_contract.py @@ -37,6 +37,7 @@ release_versions, sync_lock_version, validate_distribution_directory, + validate_public_source_policy, validate_public_source_tree, validate_sdist_license_boundary, validate_wheel_license_boundary, @@ -74,6 +75,27 @@ def test_release_versions_are_synchronized() -> None: assert len(set(versions.values())) == 1, versions +def test_public_source_refuses_paid_agent_raw_evidence_and_recipes( + tmp_path: Path, +) -> None: + validate_public_source_policy(tmp_path) + + forbidden = ( + "scripts/openemr_agent_arm.py", + "benchmark/openemr_local/agent-arm/official/results.json", + "benchmark/openemr_local/agent-arm/official/evidence/final.png", + "benchmark/frappe_lending/agent-arm/official/rows.jsonl", + "benchmark/agent_arm_verticals/COST_LEDGER.md", + ) + for relative in forbidden: + path = tmp_path / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text("private evidence\n") + with pytest.raises(ValueError, match="private paid-agent"): + validate_public_source_policy(tmp_path) + path.unlink() + + def test_semantic_release_stamps_lock_and_validates_artifacts() -> None: pyproject = tomllib.loads((ROOT / "pyproject.toml").read_text()) build_command = pyproject["tool"]["semantic_release"]["build_command"] @@ -524,7 +546,7 @@ def test_sdist_requires_mit_license_and_excludes_openimis_surface( def test_wheel_refuses_private_corpus_material(tmp_path: Path) -> None: - """The wheel must reject private, deployment-derived hardening artifacts. + """The wheel must reject private source-policy material. The public synthetic baseline (benchmark/vision_hardening/) is allowed; only the private grown-corpus / tuned-adversary / real-EMR class -- detected by @@ -556,6 +578,9 @@ def test_wheel_refuses_private_corpus_material(tmp_path: Path) -> None: "openadapt_flow/validation/identity_roc.py", "tests/test_identity_out_of_corpus.py", "private/identity/operating-point.json", + "openadapt_flow/benchmark/agent-arm/official/results.json", + "openadapt_flow/benchmark/run/rows.jsonl", + "openadapt_flow/benchmark/COST_LEDGER.md", "openadapt_flow/renamed-neutral.json", # caught by banner, not path ) ): @@ -569,12 +594,12 @@ def test_wheel_refuses_private_corpus_material(tmp_path: Path) -> None: else None ), ) - with pytest.raises(ValueError, match="private, deployment-derived"): + with pytest.raises(ValueError, match="private source-policy"): validate_wheel_license_boundary(mixed) def test_sdist_refuses_private_corpus_material(tmp_path: Path) -> None: - """The sdist must reject private, deployment-derived hardening artifacts.""" + """The sdist must reject private source-policy material.""" clean_members = {*REQUIRED_SDIST_PATHS, "PKG-INFO"} # The committed public synthetic corpus is allowed in the sdist. @@ -605,6 +630,9 @@ def test_sdist_refuses_private_corpus_material(tmp_path: Path) -> None: "docs/validation/identity_roc.json", "tests/test_identity_corpus_rates.py", "private/identity/operating-point.json", + "benchmark/openemr_local/agent-arm/official/results.json", + "benchmark/private-run/rows.jsonl", + "benchmark/agent_arm_verticals/COST_LEDGER.md", "benchmark/renamed-neutral.json", # caught by banner, not path ) ): @@ -618,7 +646,7 @@ def test_sdist_refuses_private_corpus_material(tmp_path: Path) -> None: else None ), ) - with pytest.raises(ValueError, match="private, deployment-derived"): + with pytest.raises(ValueError, match="private source-policy"): validate_sdist_license_boundary(mixed) @@ -728,12 +756,12 @@ def test_archives_refuse_nested_private_segments(tmp_path: Path) -> None: ): wheel = tmp_path / f"private-segment-{index}.whl" _write_wheel(wheel, {license_path, metadata_path, relative}) - with pytest.raises(ValueError, match="private, deployment-derived"): + with pytest.raises(ValueError, match="private source-policy"): validate_wheel_license_boundary(wheel) sdist = tmp_path / f"private-segment-{index}.tar.gz" _write_sdist(sdist, {*REQUIRED_SDIST_PATHS, "PKG-INFO", relative}) - with pytest.raises(ValueError, match="private, deployment-derived"): + with pytest.raises(ValueError, match="private source-policy"): validate_sdist_license_boundary(sdist)