feat(analyze): proof-carrying findings — analyst findings verified by executed replay, with receipts

README.md

@@ -260,9 +260,13 @@ traces upload   --since 1h --dry-run               # redact + dedup + preview, n
 traces upload   --since 24h                        # upload last day to the Intelligence Platform
 traces replay-verify --steps steps.json --image <docker-image> --at 37 --cwd /home --out ./replay-out \
   --fix-command "<corrected step>"                 # executed proof: replay prefix, reproduce failure, show fix
+traces verify-findings --findings findings.json --out ./receipts \
+  --steps steps.json --image <replay-ready-image> --cwd /app   # execute recorded analyst findings as proofs
+traces analyze --last 1 --llm --verify-findings --replay-corpus h=labels.json::prepared/  # proof-carrying analyze
 ```
 
 `replay-verify` replays a CodeTraceBench-style trajectory prefix in a real sandbox and executes step k twice — recorded (does the failure reproduce?) and corrected (does it vanish?).
+`verify-findings` runs that proof per analyst finding and annotates each with `reproduced | fix-flipped | divergent | not-replayable` plus a receipt directory; see [Verified findings](./docs/trace-analysts.md#verified-findings-executed-replay).
 See [Replay verification](./docs/replay-verify.md) for setup, semantics, and honest limits (SWE-style trajectories with a docker image only; commands run as the non-root sandbox user).
 
 | Flag | Meaning |
@@ -288,6 +292,9 @@ See [Replay verification](./docs/replay-verify.md) for setup, semantics, and hon
 | `--min-loop <n>` | Identical repeated calls before flagging a loop (default 3) |
 | `--mode <kind>` | `stream`: `visualizer` (spans + findings), `findings` (low-volume), or `agent` (findings + reports) |
 | `--supervisor-run-dir <dir>` | `analyze`: report one run tree; `watch`: tail it live |
+| `--verify-findings` | `analyze`: execute the findings as sandbox replay proofs; each is marked VERIFIED (receipt path) or UNVERIFIABLE (reason). Needs `--replay-corpus` and a running sandbox (`SANDBOX_API_KEY` / `SANDBOX_API_URL`) |
+| `--replay-corpus name=<labels>::<prepared>` | Trajectory source for `--verify-findings` (repeatable) |
+| `--verify-out <dir>` | Receipt root for `--verify-findings` (default: `<--out>.verify`) |
 | `--replay` | `stream`: scan once, then exit |
 | `--once` | `stream`: scan once; `watch <target>`: print ONE snapshot and exit |
 | `--no-spans` / `--no-findings` | `stream`: suppress raw span rows / finding rows |

docs/replay-verify.md

@@ -99,6 +99,12 @@ const { invocation, fixCommand, verdict } = await replayVerifyFinding(
 
 The subject grammar is the analyst benchmark's `incorrect-steps-<first>-<last>-<escaped|unescaped>-consequence-<step>`; `--at` is the finding's **first** incorrect step (the finding's claim, which may differ from the gold label). The wire resolves the trajectory across the given corpora, generates the arm-B fix through the same one-call generator (or accepts a pre-supplied `fixCommand`), and returns the full `ReplayVerdict`. It throws with a precise reason when the finding cannot be replayed (malformed subject, unknown trajectory, non-SWE case, step out of range) — the product surfaces that reason instead of a proof.
 
+## Product surface — verified findings
+
+`traces verify-findings` (and `traces analyze --verify-findings`) runs this proof per recorded analyst finding and annotates each with `reproduced | fix-flipped | divergent | not-replayable` plus a receipt directory.
+Unlike the wire above it accepts the shapes analysts actually emit (`incorrect-step-<n>` subjects, `metadata.block_first_step`, `trace://` evidence refs) and never throws on a finding-shaped dead end — the dead end becomes the finding's honest `not-replayable` receipt.
+See [Verified findings](./trace-analysts.md#verified-findings-executed-replay).
+
 ## Orchestrator prerequisites
 
 replay-verify talks to a sandbox API (`--base-url`); in local development that is the sandbox SDK adapter in front of an orchestrator running the docker driver.

docs/trace-analysts.md

@@ -210,6 +210,34 @@ console.log(renderAnalystBenchmarkMarkdown(result))
 Public labels test the measurement code, not the quality of every built-in analyst automatically.
 A real quality claim requires running the analyst over the corresponding trajectories, retaining all rows, and comparing it with named alternatives at equal model and request limits.
 
+## Verified findings (executed replay)
+
+An analyst finding is a cited claim until something executes it.
+`traces analyze --verify-findings` (and the standalone `traces verify-findings`) replays each finding's trajectory prefix in a real sandbox, re-runs the accused step, and annotates every finding with an executed verdict:
+
+| Verdict | Meaning |
+|---|---|
+| `reproduced` | the recorded failure signature (returncode + stable output substring) reproduced when the accused step re-ran |
+| `fix-flipped` | reproduced, and a supplied corrected command made the failure vanish in a fresh replay |
+| `divergent` | the step executed but the recorded failure did not reproduce — evidence against the finding, or against replay fidelity (the receipt carries prefix divergences so you can tell which) |
+| `not-replayable` | the finding could not be executed; the receipt names the precise reason (no step subject, unknown trajectory, no docker image, submit step, …) |
+
+```bash
+# Verify the findings an analyze run produced (marks each finding in the report):
+traces analyze --last 1 --llm --verify-findings \
+  --replay-corpus holdout=labels.json::prepared/ --verify-out ./receipts
+
+# Verify findings recorded earlier (e.g. extracted from an eval result.json):
+traces verify-findings --findings findings.json --out ./receipts \
+  --steps normalized/<traj>/steps.json --image <replay-ready-image> --cwd /app
+```
+
+Findings are matched by the shape analysts emit: subject `incorrect-step-<n>` (or the wire form `incorrect-steps-<f>-<l>-…`), `metadata.block_first_step`, and `trace://<trajectory>/…` evidence refs.
+Findings accusing the same step share one executed proof; each finding still gets its own receipt directory (`receipt.json` plus, when executed, `replay-verdict.json` and `report.md` with real stdout/stderr).
+Verification is execution, not generation: no LLM is involved unless you pass `--fix-command`.
+A missing sandbox is an error when any finding is replayable — verification never silently skips.
+Sandbox setup, execution semantics, and honest limits are in [Replay verification](./replay-verify.md).
+
 ## Turn findings into improvement
 
 Do not train or rewrite policy from an analyst's own prose alone.