feat(analyze): proof-carrying findings — analyst findings verified by executed replay, with receipts - #77
Merged
Conversation
…lyst findings as sandbox replays Every analyst finding can now ship with an executed verdict instead of a prose claim. New src/analyze-verify.ts maps a finding (incorrect-step-<n> subject, metadata.block_first_step, trace:// evidence) onto the replay wire, replays the trajectory prefix in a real sandbox, re-runs the accused step, and classifies the outcome: reproduced recorded failure signature reproduced (returncode + output substring) fix-flipped reproduced AND a supplied corrected command made it vanish divergent executed but the recorded failure did not reproduce not-replayable precise reason (no step subject, unknown trajectory, submit step, ...) Surfaces: - traces verify-findings --findings f.json (--steps/--image/--cwd | --corpus ...) --out DIR for findings recorded earlier (eval result.json extractions); corpus mode resolves trajectories by trace:// evidence and derives uid-1000 images via the batch preparer. - traces analyze --verify-findings --replay-corpus ... appends a Verified findings section marking each finding VERIFIED (receipt path) or UNVERIFIABLE (reason). One receipt directory per finding (receipt.json + replay-verdict.json + report.md when executed); findings accusing the same step share one executed proof. Sandbox absence while a finding is replayable is an error, never a silent skip. Verification is execution — no LLM unless --fix-command is given. Proven live against recorded cert findings on holdout-2: reproduced (ponyc-2532 step 12, exit 1 + 'No such file or directory' signature, 0/11 prefix divergences), fix-flipped (the claim's own -p1 correction, arm B exit 0), divergent (element-web step 17), not-replayable (submit step 22).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The product moat, wired:
traces verify-findings(standalone) andtraces analyze --verify-findingsexecute each analyst finding's accused step in a real sandbox and attach a verdict — reproduced / fix-flipped / divergent / not-replayable — with a receipt directory per finding (commands, exit codes, signature match, timings). A finding stops being a cited claim and becomes an executed proof. No shipped trace-analysis product does this.Executed proof (all four verdicts live, real sandbox, real recorded findings, $0 LLM)
Findings extracted from the certified holdout-2 run (
cert-g-h2/result.json):git applywrong strip level): arm A exit 1 == recorded rc, error signature matched, 11 prefix steps 0 divergences, 5.9s. Re-verified independently by the operator post-build: same verdict, 1 execution.--fix-command: arm B exit 0, failure vanished, 9.1s.Fail-loud:
analyze --verify-findingswithout a replay corpus exits 1 with the reason (harness sessions carry no docker image); with a corpus it appends a "Verified findings" section to the analyze report.Design
New module only (
src/analyze-verify.ts, 700 lines + 19 tests) — zero edits to the in-flight replay-batch/fix/loop files. Matches the analyst's real emit shape; findings sharing (case, step, fix) share one execution but keep individual receipts. Verification is execution, not generation — no LLM in the path.Battery: tsc clean, 606/606 tests, build + package checks green, merge-tree clean.