diff --git a/README.md b/README.md index c97071e..9bbda6c 100644 --- a/README.md +++ b/README.md @@ -260,9 +260,13 @@ traces upload --since 1h --dry-run # redact + dedup + preview, n traces upload --since 24h # upload last day to the Intelligence Platform traces replay-verify --steps steps.json --image --at 37 --cwd /home --out ./replay-out \ --fix-command "" # executed proof: replay prefix, reproduce failure, show fix +traces verify-findings --findings findings.json --out ./receipts \ + --steps steps.json --image --cwd /app # execute recorded analyst findings as proofs +traces analyze --last 1 --llm --verify-findings --replay-corpus h=labels.json::prepared/ # proof-carrying analyze ``` `replay-verify` replays a CodeTraceBench-style trajectory prefix in a real sandbox and executes step k twice — recorded (does the failure reproduce?) and corrected (does it vanish?). +`verify-findings` runs that proof per analyst finding and annotates each with `reproduced | fix-flipped | divergent | not-replayable` plus a receipt directory; see [Verified findings](./docs/trace-analysts.md#verified-findings-executed-replay). See [Replay verification](./docs/replay-verify.md) for setup, semantics, and honest limits (SWE-style trajectories with a docker image only; commands run as the non-root sandbox user). | Flag | Meaning | @@ -288,6 +292,9 @@ See [Replay verification](./docs/replay-verify.md) for setup, semantics, and hon | `--min-loop ` | Identical repeated calls before flagging a loop (default 3) | | `--mode ` | `stream`: `visualizer` (spans + findings), `findings` (low-volume), or `agent` (findings + reports) | | `--supervisor-run-dir ` | `analyze`: report one run tree; `watch`: tail it live | +| `--verify-findings` | `analyze`: execute the findings as sandbox replay proofs; each is marked VERIFIED (receipt path) or UNVERIFIABLE (reason). Needs `--replay-corpus` and a running sandbox (`SANDBOX_API_KEY` / `SANDBOX_API_URL`) | +| `--replay-corpus name=::` | Trajectory source for `--verify-findings` (repeatable) | +| `--verify-out ` | Receipt root for `--verify-findings` (default: `<--out>.verify`) | | `--replay` | `stream`: scan once, then exit | | `--once` | `stream`: scan once; `watch `: print ONE snapshot and exit | | `--no-spans` / `--no-findings` | `stream`: suppress raw span rows / finding rows | diff --git a/docs/replay-verify.md b/docs/replay-verify.md index 5738707..1c8744b 100644 --- a/docs/replay-verify.md +++ b/docs/replay-verify.md @@ -99,6 +99,12 @@ const { invocation, fixCommand, verdict } = await replayVerifyFinding( The subject grammar is the analyst benchmark's `incorrect-steps----consequence-`; `--at` is the finding's **first** incorrect step (the finding's claim, which may differ from the gold label). The wire resolves the trajectory across the given corpora, generates the arm-B fix through the same one-call generator (or accepts a pre-supplied `fixCommand`), and returns the full `ReplayVerdict`. It throws with a precise reason when the finding cannot be replayed (malformed subject, unknown trajectory, non-SWE case, step out of range) — the product surfaces that reason instead of a proof. +## Product surface — verified findings + +`traces verify-findings` (and `traces analyze --verify-findings`) runs this proof per recorded analyst finding and annotates each with `reproduced | fix-flipped | divergent | not-replayable` plus a receipt directory. +Unlike the wire above it accepts the shapes analysts actually emit (`incorrect-step-` subjects, `metadata.block_first_step`, `trace://` evidence refs) and never throws on a finding-shaped dead end — the dead end becomes the finding's honest `not-replayable` receipt. +See [Verified findings](./trace-analysts.md#verified-findings-executed-replay). + ## Orchestrator prerequisites replay-verify talks to a sandbox API (`--base-url`); in local development that is the sandbox SDK adapter in front of an orchestrator running the docker driver. diff --git a/docs/trace-analysts.md b/docs/trace-analysts.md index 72485f1..9d54c3b 100644 --- a/docs/trace-analysts.md +++ b/docs/trace-analysts.md @@ -210,6 +210,34 @@ console.log(renderAnalystBenchmarkMarkdown(result)) Public labels test the measurement code, not the quality of every built-in analyst automatically. A real quality claim requires running the analyst over the corresponding trajectories, retaining all rows, and comparing it with named alternatives at equal model and request limits. +## Verified findings (executed replay) + +An analyst finding is a cited claim until something executes it. +`traces analyze --verify-findings` (and the standalone `traces verify-findings`) replays each finding's trajectory prefix in a real sandbox, re-runs the accused step, and annotates every finding with an executed verdict: + +| Verdict | Meaning | +|---|---| +| `reproduced` | the recorded failure signature (returncode + stable output substring) reproduced when the accused step re-ran | +| `fix-flipped` | reproduced, and a supplied corrected command made the failure vanish in a fresh replay | +| `divergent` | the step executed but the recorded failure did not reproduce — evidence against the finding, or against replay fidelity (the receipt carries prefix divergences so you can tell which) | +| `not-replayable` | the finding could not be executed; the receipt names the precise reason (no step subject, unknown trajectory, no docker image, submit step, …) | + +```bash +# Verify the findings an analyze run produced (marks each finding in the report): +traces analyze --last 1 --llm --verify-findings \ + --replay-corpus holdout=labels.json::prepared/ --verify-out ./receipts + +# Verify findings recorded earlier (e.g. extracted from an eval result.json): +traces verify-findings --findings findings.json --out ./receipts \ + --steps normalized//steps.json --image --cwd /app +``` + +Findings are matched by the shape analysts emit: subject `incorrect-step-` (or the wire form `incorrect-steps---…`), `metadata.block_first_step`, and `trace:///…` evidence refs. +Findings accusing the same step share one executed proof; each finding still gets its own receipt directory (`receipt.json` plus, when executed, `replay-verdict.json` and `report.md` with real stdout/stderr). +Verification is execution, not generation: no LLM is involved unless you pass `--fix-command`. +A missing sandbox is an error when any finding is replayable — verification never silently skips. +Sandbox setup, execution semantics, and honest limits are in [Replay verification](./replay-verify.md). + ## Turn findings into improvement Do not train or rewrite policy from an analyst's own prose alone. diff --git a/src/analyze-verify.ts b/src/analyze-verify.ts new file mode 100644 index 0000000..334c800 --- /dev/null +++ b/src/analyze-verify.ts @@ -0,0 +1,756 @@ +/** + * Proof-carrying findings: execute analyst findings as sandbox replays. + * + * An analyst finding is a cited claim ("step 12 is where the run went + * wrong"). This module turns each finding into an executed verdict by + * replaying the trajectory prefix in a real sandbox and running the accused + * step (arm A), optionally followed by a corrected step (arm B): + * + * `reproduced` arm A re-produced the recorded failure signature + * (returncode + stable output substring). + * `fix-flipped` arm A reproduced AND arm B's corrected command made the + * failure vanish — the strongest per-finding proof. + * `divergent` arm A executed but the recorded failure did NOT + * reproduce; evidence against the finding (or against + * replay fidelity — the receipt carries prefix + * divergences so the reader can tell which). + * `not-replayable` the finding could not be executed at all; the receipt + * carries the precise reason (no step subject, unknown + * trajectory, no docker image, submit step, …). + * + * Verification is execution, not generation: no LLM is involved unless the + * caller supplies a corrected command for arm B. Sandbox infrastructure being + * absent while a finding needs execution is an error, never a silent skip. + * + * Findings are matched by the shape the analyst product emits + * (agent-eval `AnalystFinding`): `subject` (`incorrect-step-` or the wire's + * `incorrect-steps----consequence-`), + * `metadata.block_first_step`, and `trace:///…` evidence refs. + */ + +import { mkdirSync, readFileSync, writeFileSync } from 'node:fs' +import { join } from 'node:path' +import type { CodeTraceBenchStep } from './codetracebench.js' +import { dockerImagePreparer, type ImagePreparer } from './replay-batch.js' +import { + type CorpusSpec, + isSubmitAction, + parseCorpusFlag, + resolveCaseResources, +} from './replay-corpus.js' +import { + parseRecordedReturncode, + type ReplayExecBackend, + type ReplayVerdict, + replayVerify, +} from './replay-verify.js' +import { parseIncorrectStepsSubject } from './replay-wire.js' + +// ── Finding shape (structural subset of agent-eval's AnalystFinding) ─ + +export interface VerifiableFinding { + readonly finding_id?: string + readonly analyst_id?: string + readonly subject?: string + readonly area?: string + readonly claim?: string + readonly evidence_refs?: readonly { readonly kind?: string; readonly uri?: string; readonly excerpt?: string }[] + readonly metadata?: Readonly> +} + +const SINGLE_STEP_SUBJECT = /^incorrect-step-(\d+)$/ + +/** + * 1-based step the finding accuses, or null when the finding names none. + * `metadata.block_first_step` wins over the subject: the analyst records the + * block's first incorrect step there even when the subject names a later + * step of the same block. + */ +export function findingReplayStep(finding: VerifiableFinding): number | null { + const fromMetadata = finding.metadata?.block_first_step + if (typeof fromMetadata === 'number' && Number.isInteger(fromMetadata) && fromMetadata >= 1) { + return fromMetadata + } + const subject = finding.subject ?? '' + const wire = parseIncorrectStepsSubject(subject) + if (wire) return wire.firstStep + const single = SINGLE_STEP_SUBJECT.exec(subject) + if (single) return Number(single[1]) + return null +} + +const TRACE_EVIDENCE_URI = /^trace:\/\/([^/]+)\// + +/** Trajectory id from the finding's `trace:///…` evidence refs, or null. */ +export function findingTrajectoryId(finding: VerifiableFinding): string | null { + for (const ref of finding.evidence_refs ?? []) { + if (typeof ref.uri !== 'string') continue + const match = TRACE_EVIDENCE_URI.exec(ref.uri) + if (match) return match[1]! + } + return null +} + +// ── Replay source ──────────────────────────────────────────────────── + +/** + * Where the executable trajectory lives. + * `direct` — one trajectory's steps.json plus a replay-ready image (the + * caller owns image preparation, exactly like `traces replay-verify`). + * `corpus` — CodeTraceBench corpora; each finding's trajectory is resolved + * by its `trace://` evidence and the image is derived through the batch + * preparer (uid-1000 chown) unless a test backend is injected. + */ +export type FindingReplaySource = + | { + readonly kind: 'direct' + readonly stepsPath: string + readonly image: string + readonly cwd: string + readonly caseId?: string + } + | { + readonly kind: 'corpus' + readonly corpora: readonly CorpusSpec[] + readonly preparer?: ImagePreparer + } + +export interface ResolvedFindingReplay { + readonly caseId: string + readonly stepsPath: string + /** Raw image; corpus-mode execution derives the uid-1000 replay image from it. */ + readonly image: string + readonly cwd: string + /** 1-based step_id arm A executes. */ + readonly at: number + readonly recordedReturncode: number + readonly recordedStepTimeoutMs: number | null +} + +export type FindingReplayability = + | { readonly replayable: true; readonly resolved: ResolvedFindingReplay } + | { readonly replayable: false; readonly reason: string } + +function checkStep( + steps: readonly CodeTraceBenchStep[], + at: number, +): { readonly ok: true; readonly recordedReturncode: number } | { readonly ok: false; readonly reason: string } { + const step = steps.find((s) => s.step_id === at) + if (!step) { + return { ok: false, reason: `step ${at} is outside the trajectory (${steps.length} steps)` } + } + if (isSubmitAction(step.action)) { + return { + ok: false, + reason: `step ${at} is the submit action — a submit decision has no executable failure to replay`, + } + } + const recordedReturncode = parseRecordedReturncode(step.observation) + if (recordedReturncode === null) { + return { + ok: false, + reason: `step ${at} recorded no returncode — there is no executable failure signature to reproduce`, + } + } + return { ok: true, recordedReturncode } +} + +/** + * Decides whether one finding can be executed against the source, and with + * what invocation. Never throws for a finding-shaped problem — every dead end + * becomes a `not-replayable` reason the receipt can carry verbatim. + */ +export function resolveFindingReplayability( + finding: VerifiableFinding, + source: FindingReplaySource, +): FindingReplayability { + const at = findingReplayStep(finding) + if (at === null) { + return { + replayable: false, + reason: + `subject '${finding.subject ?? '(none)'}' names no trajectory step ` + + '(expected incorrect-step-, incorrect-steps---…, or metadata.block_first_step)', + } + } + if (source.kind === 'direct') { + const trajId = findingTrajectoryId(finding) + if (trajId && source.caseId && trajId !== source.caseId) { + return { + replayable: false, + reason: `finding cites trajectory '${trajId}' but the supplied steps are case '${source.caseId}'`, + } + } + let steps: CodeTraceBenchStep[] + try { + steps = JSON.parse(readFileSync(source.stepsPath, 'utf8')) as CodeTraceBenchStep[] + } catch (err) { + throw new Error( + `verify-findings: cannot read steps file ${source.stepsPath} — ${err instanceof Error ? err.message : String(err)}`, + ) + } + if (!Array.isArray(steps) || steps.length === 0) { + throw new Error(`verify-findings: ${source.stepsPath} is not a non-empty steps array`) + } + const step = checkStep(steps, at) + if (!step.ok) return { replayable: false, reason: step.reason } + return { + replayable: true, + resolved: { + caseId: source.caseId ?? trajId ?? source.stepsPath, + stepsPath: source.stepsPath, + image: source.image, + cwd: source.cwd, + at, + recordedReturncode: step.recordedReturncode, + recordedStepTimeoutMs: null, + }, + } + } + const trajId = findingTrajectoryId(finding) + if (!trajId) { + return { + replayable: false, + reason: 'finding carries no trace:/// evidence ref naming its trajectory', + } + } + const failures: string[] = [] + for (const corpus of source.corpora) { + const resolution = resolveCaseResources(corpus, trajId) + if (!resolution.resolved) { + failures.push(`${corpus.name}: ${resolution.reason}${resolution.detail ? ` (${resolution.detail})` : ''}`) + continue + } + const step = checkStep(resolution.resources.steps, at) + if (!step.ok) return { replayable: false, reason: step.reason } + return { + replayable: true, + resolved: { + caseId: trajId, + stepsPath: resolution.resources.stepsPath, + image: resolution.resources.image, + cwd: resolution.resources.cwd, + at, + recordedReturncode: step.recordedReturncode, + recordedStepTimeoutMs: resolution.resources.recordedStepTimeoutMs, + }, + } + } + return { + replayable: false, + reason: `trajectory ${trajId} is not replayable in any corpus — ${failures.join('; ')}`, + } +} + +// ── Verification run ───────────────────────────────────────────────── + +export type FindingVerificationStatus = 'reproduced' | 'fix-flipped' | 'not-replayable' | 'divergent' + +export interface FindingVerification { + readonly finding_id: string | null + readonly subject: string | null + readonly trajectory_id: string | null + /** 1-based step the proof executed at; null when not replayable. */ + readonly step: number | null + readonly verified: FindingVerificationStatus + /** Present exactly when `verified` is `not-replayable`. */ + readonly reason: string | null + /** Receipt directory: receipt.json plus, when executed, replay-verdict.json + report.md. */ + readonly receipt: string + readonly verdict_path: string | null + /** Receipt dir of the executed proof this finding shares (same case, step, and fix). */ + readonly deduplicated_with: string | null +} + +export interface VerifyFindingsRun { + readonly out: string + readonly verifications: readonly FindingVerification[] + readonly counts: Readonly> + /** Sandbox executions actually performed (deduplicated proofs count once). */ + readonly executions: number +} + +export interface VerifyFindingsOptions { + readonly source: FindingReplaySource + /** Receipt root; one subdirectory per finding plus verifications.json. */ + readonly out: string + /** Corrected command for arm B on every executed finding; omit for arm A only. */ + readonly fixCommand?: string + readonly stepTimeoutMs?: number + readonly prefixLimit?: number + /** Injectable for tests; when set, no sandbox reachability check and no image preparation run. */ + readonly backend?: ReplayExecBackend + readonly apiKey?: string + readonly baseUrl?: string + readonly maxLifetimeSeconds?: number + readonly onProgress?: (message: string) => void +} + +/** Arm A reproduced → the fix flipping it beats plain reproduction; anything else diverged. */ +export function classifyVerdict(verdict: ReplayVerdict): FindingVerificationStatus { + if (!verdict.armA.failureSignatureMatch) return 'divergent' + if (verdict.armB?.failureVanished) return 'fix-flipped' + return 'reproduced' +} + +export const DEFAULT_SANDBOX_BASE_URL = 'http://127.0.0.1:4097' + +/** + * Fails loud when the sandbox SDK adapter is not answering. Executed findings + * require real infrastructure; verification is never silently skipped. + */ +export async function assertSandboxReachable(baseUrl: string): Promise { + let response: Response + try { + response = await fetch(new URL('/health', baseUrl), { signal: AbortSignal.timeout(5000) }) + } catch (err) { + throw new Error( + `verify-findings: sandbox API unreachable at ${baseUrl} — ` + + `${err instanceof Error ? err.message : String(err)}. ` + + 'Executing findings requires a running sandbox orchestrator + SDK adapter ' + + '(see docs/replay-verify.md); start them or pass --base-url.', + ) + } + if (!response.ok) { + throw new Error( + `verify-findings: sandbox API at ${baseUrl} answered /health with HTTP ${response.status} — refusing to run proofs against degraded infrastructure`, + ) + } +} + +interface ReceiptExecution { + readonly image: string + readonly cwd: string + readonly recordedReturncode: number | null + readonly signature: string | null + readonly signatureBasis: string + readonly armA: { readonly command: string; readonly exitCode: number; readonly wallMs: number; readonly failureSignatureMatch: boolean } + readonly armB: { readonly command: string; readonly exitCode: number; readonly wallMs: number; readonly failureVanished: boolean } | null + readonly prefixExecuted: number + readonly prefixDivergences: number + readonly totalMs: number +} + +function receiptExecution(verdict: ReplayVerdict): ReceiptExecution { + return { + image: verdict.image, + cwd: verdict.cwd, + recordedReturncode: verdict.recordedReturncode, + signature: verdict.signature, + signatureBasis: verdict.signatureBasis, + armA: { + command: verdict.armA.command, + exitCode: verdict.armA.exitCode, + wallMs: verdict.armA.wallMs, + failureSignatureMatch: verdict.armA.failureSignatureMatch, + }, + armB: verdict.armB + ? { + command: verdict.armB.command, + exitCode: verdict.armB.exitCode, + wallMs: verdict.armB.wallMs, + failureVanished: verdict.armB.failureVanished, + } + : null, + prefixExecuted: verdict.prefixExecuted, + prefixDivergences: verdict.prefixDivergences.length, + totalMs: verdict.timings.totalMs, + } +} + +function writeReceipt( + receiptDir: string, + finding: VerifiableFinding, + verification: FindingVerification, + execution: ReceiptExecution | null, +): void { + const receipt = { + schema_version: '1.0.0', + produced_at: new Date().toISOString(), + finding_id: verification.finding_id, + analyst_id: finding.analyst_id ?? null, + subject: verification.subject, + claim: typeof finding.claim === 'string' ? finding.claim.slice(0, 600) : null, + trajectory_id: verification.trajectory_id, + step: verification.step, + verified: verification.verified, + reason: verification.reason, + execution, + verdict_path: verification.verdict_path, + deduplicated_with: verification.deduplicated_with, + } + writeFileSync(join(receiptDir, 'receipt.json'), `${JSON.stringify(receipt, null, 2)}\n`) +} + +function receiptDirName(index: number, finding: VerifiableFinding): string { + const id = typeof finding.finding_id === 'string' && finding.finding_id.length > 0 + ? finding.finding_id.replace(/[^A-Za-z0-9_-]/g, '_') + : 'finding' + return `${String(index + 1).padStart(3, '0')}-${id}` +} + +/** + * Verifies every finding against the source: resolves replayability, executes + * one sandbox proof per distinct (case, step, fix) — findings accusing the + * same step share the executed proof — and writes a receipt directory per + * finding plus a run-level verifications.json. + */ +export async function verifyFindings( + findings: readonly VerifiableFinding[], + options: VerifyFindingsOptions, +): Promise { + if (findings.length === 0) throw new Error('verify-findings: no findings to verify') + mkdirSync(options.out, { recursive: true }) + const resolutions = findings.map((finding) => resolveFindingReplayability(finding, options.source)) + const baseUrl = options.baseUrl ?? DEFAULT_SANDBOX_BASE_URL + if (resolutions.some((resolution) => resolution.replayable) && !options.backend) { + if (!options.apiKey) { + throw new Error( + 'verify-findings: replayable findings need a sandbox API key — export SANDBOX_API_KEY or pass --api-key-env', + ) + } + await assertSandboxReachable(baseUrl) + } + + const preparer = + options.source.kind === 'corpus' && !options.backend + ? options.source.preparer ?? dockerImagePreparer() + : null + const preparedImages = new Map() + const executedByKey = new Map() + const verifications: FindingVerification[] = [] + let executions = 0 + + for (let index = 0; index < findings.length; index++) { + const finding = findings[index]! + const resolution = resolutions[index]! + const receiptDir = join(options.out, receiptDirName(index, finding)) + mkdirSync(receiptDir, { recursive: true }) + const identity = { + finding_id: finding.finding_id ?? null, + subject: finding.subject ?? null, + trajectory_id: findingTrajectoryId(finding), + } + + if (!resolution.replayable) { + const verification: FindingVerification = { + ...identity, + step: findingReplayStep(finding), + verified: 'not-replayable', + reason: resolution.reason, + receipt: receiptDir, + verdict_path: null, + deduplicated_with: null, + } + writeReceipt(receiptDir, finding, verification, null) + verifications.push(verification) + options.onProgress?.(`${identity.finding_id ?? `finding ${index + 1}`}: not-replayable — ${resolution.reason}`) + continue + } + + const resolved = resolution.resolved + const dedupeKey = `${resolved.caseId}::${resolved.at}::${options.fixCommand ?? ''}` + const prior = executedByKey.get(dedupeKey) + if (prior) { + const verification: FindingVerification = { + ...identity, + step: resolved.at, + verified: prior.verified, + reason: prior.reason, + receipt: receiptDir, + verdict_path: prior.verdict_path, + deduplicated_with: prior.receipt, + } + const priorVerdict = prior.verdict_path + ? (JSON.parse(readFileSync(prior.verdict_path, 'utf8')) as ReplayVerdict) + : null + writeReceipt(receiptDir, finding, verification, priorVerdict ? receiptExecution(priorVerdict) : null) + verifications.push(verification) + options.onProgress?.( + `${identity.finding_id ?? `finding ${index + 1}`}: ${prior.verified} (shares proof with ${prior.finding_id ?? prior.receipt})`, + ) + continue + } + + let image = resolved.image + if (preparer) { + const preparationKey = `${resolved.image}::${resolved.cwd}` + let preparation = preparedImages.get(preparationKey) + if (!preparation) { + const ensured = await preparer.ensure(resolved.image, resolved.cwd) + preparation = ensured.succeeded + ? { succeeded: true, image: ensured.value.derivedImage } + : { succeeded: false, error: ensured.error } + preparedImages.set(preparationKey, preparation) + } + if (!preparation.succeeded) { + const verification: FindingVerification = { + ...identity, + step: resolved.at, + verified: 'not-replayable', + reason: `replay image could not be prepared — ${preparation.error}`, + receipt: receiptDir, + verdict_path: null, + deduplicated_with: null, + } + writeReceipt(receiptDir, finding, verification, null) + verifications.push(verification) + options.onProgress?.(`${identity.finding_id ?? `finding ${index + 1}`}: not-replayable — image preparation failed`) + continue + } + image = preparation.image + } + + options.onProgress?.( + `${identity.finding_id ?? `finding ${index + 1}`}: executing arm A at step ${resolved.at} of ${resolved.caseId} on ${image}`, + ) + const verdict = await replayVerify({ + stepsPath: resolved.stepsPath, + image, + at: resolved.at, + fixCommand: options.fixCommand, + cwd: resolved.cwd, + out: receiptDir, + caseId: resolved.caseId, + stepTimeoutMs: options.stepTimeoutMs ?? resolved.recordedStepTimeoutMs ?? undefined, + prefixLimit: options.prefixLimit, + backend: options.backend, + apiKey: options.apiKey, + baseUrl, + maxLifetimeSeconds: options.maxLifetimeSeconds, + onProgress: options.onProgress, + }) + executions += 1 + const verification: FindingVerification = { + ...identity, + step: resolved.at, + verified: classifyVerdict(verdict), + reason: null, + receipt: receiptDir, + verdict_path: join(receiptDir, 'replay-verdict.json'), + deduplicated_with: null, + } + executedByKey.set(dedupeKey, verification) + writeReceipt(receiptDir, finding, verification, receiptExecution(verdict)) + verifications.push(verification) + options.onProgress?.(`${identity.finding_id ?? `finding ${index + 1}`}: ${verification.verified}`) + } + + const counts: Record = { + reproduced: 0, + 'fix-flipped': 0, + divergent: 0, + 'not-replayable': 0, + } + for (const verification of verifications) counts[verification.verified] += 1 + const run: VerifyFindingsRun = { out: options.out, verifications, counts, executions } + writeFileSync( + join(options.out, 'verifications.json'), + `${JSON.stringify({ schema_version: '1.0.0', ...run }, null, 2)}\n`, + ) + return run +} + +// ── Report rendering ───────────────────────────────────────────────── + +function verdictCell(verification: FindingVerification): string { + switch (verification.verified) { + case 'reproduced': + return '**VERIFIED** — reproduced' + case 'fix-flipped': + return '**VERIFIED** — fix-flipped' + case 'divergent': + return 'DIVERGENT — recorded failure did not reproduce' + case 'not-replayable': + return `UNVERIFIABLE — ${verification.reason ?? 'no reason recorded'}` + } +} + +/** Markdown section the analyze report appends when --verify-findings ran. */ +export function renderVerifiedFindingsSection(run: VerifyFindingsRun): string { + const lines: string[] = ['## Verified findings (executed replay)', ''] + const total = run.verifications.length + lines.push( + `${total} finding(s) → ${run.counts.reproduced} reproduced, ${run.counts['fix-flipped']} fix-flipped, ` + + `${run.counts.divergent} divergent, ${run.counts['not-replayable']} not replayable ` + + `(${run.executions} sandbox execution(s); findings accusing the same step share one proof).`, + ) + lines.push('') + lines.push('| Finding | Subject | Step | Verdict | Receipt |') + lines.push('|---|---|---:|---|---|') + for (const verification of run.verifications) { + const shared = verification.deduplicated_with ? ' (shared proof)' : '' + lines.push( + `| \`${verification.finding_id ?? '—'}\` | \`${verification.subject ?? '—'}\` | ` + + `${verification.step ?? '—'} | ${verdictCell(verification)} | \`${verification.receipt}\`${shared} |`, + ) + } + lines.push('') + lines.push( + 'VERIFIED = the accused step was re-executed in a sandbox after replaying the trajectory prefix, and the recorded ' + + 'failure signature reproduced (fix-flipped: a corrected command additionally made it vanish). Each receipt ' + + 'directory carries receipt.json and, when executed, replay-verdict.json + report.md with real stdout/stderr.', + ) + lines.push('') + return lines.join('\n') +} + +// ── Findings file loading ──────────────────────────────────────────── + +/** + * Accepts the two shapes findings travel in: a bare JSON array of analyst + * findings, or an object with a `findings` array (e.g. an extracted + * `observations[n]` from an agent-eval result.json). + */ +export function readFindingsFile(path: string): VerifiableFinding[] { + const parsed: unknown = JSON.parse(readFileSync(path, 'utf8')) + const array = Array.isArray(parsed) + ? parsed + : parsed && typeof parsed === 'object' && Array.isArray((parsed as { findings?: unknown }).findings) + ? ((parsed as { findings: unknown[] }).findings) + : null + if (!array) { + throw new Error(`${path} is neither a findings array nor an object with a findings array`) + } + for (const entry of array) { + if (!entry || typeof entry !== 'object') { + throw new Error(`${path}: every finding must be an object, got ${JSON.stringify(entry)}`) + } + } + return array as VerifiableFinding[] +} + +// ── CLI ────────────────────────────────────────────────────────────── + +export interface VerifyFindingsCliArgs { + findingsPath: string + source: FindingReplaySource + out: string + fixCommand?: string + stepTimeoutMs?: number + prefixLimit?: number + baseUrl: string + apiKeyEnv: string + maxLifetimeSeconds?: number +} + +export function verifyFindingsUsage(): string { + return `traces verify-findings — execute analyst findings as sandbox replay proofs + +Usage: + traces verify-findings --findings FINDINGS.json --out DIR \\ + ( --steps STEPS.json --image IMG --cwd DIR [--case ID] + | --corpus name=:: [--corpus ...] ) \\ + [--fix-command CMD] [--step-timeout MS] [--prefix-limit N] \\ + [--base-url URL] [--api-key-env VAR] [--max-lifetime SECONDS] + + --findings JSON array of analyst findings (or an object with a findings + array) — the shape agent-eval analysts emit: subject + incorrect-step-, metadata.block_first_step, trace:// evidence + --out receipt root; one directory per finding + verifications.json + --steps/--image/--cwd + verify against one trajectory; the image must be replay-ready + (uid-1000 derived), exactly like traces replay-verify + --case trajectory id of --steps; findings citing other trajectories + are honestly marked not-replayable + --corpus resolve each finding's trajectory in CodeTraceBench corpora; + images are derived via the batch uid-1000 preparer (docker) + --fix-command corrected step for arm B; a reproduced finding whose fix + flips becomes fix-flipped + --base-url sandbox API url (default $SANDBOX_API_URL or ${DEFAULT_SANDBOX_BASE_URL}) + --api-key-env env var holding the sandbox API key (default SANDBOX_API_KEY) + +Every finding gets a verdict: reproduced | fix-flipped | divergent | not-replayable, +with a receipt directory carrying the executed evidence or the precise reason. +Requires a running sandbox orchestrator when any finding is replayable — infra +absence is an error, never a silent skip. See docs/replay-verify.md for setup. +` +} + +export function parseVerifyFindingsArgs(argv: readonly string[]): VerifyFindingsCliArgs | 'help' { + const values = new Map() + const corpora: CorpusSpec[] = [] + for (let i = 0; i < argv.length; i++) { + const arg = argv[i]! + if (arg === '--help' || arg === '-h') return 'help' + if (!arg.startsWith('--')) throw new Error(`unexpected positional argument: ${arg}`) + const value = argv[++i] + if (value === undefined) throw new Error(`missing value for ${arg}`) + if (arg === '--corpus') { + corpora.push(parseCorpusFlag(value)) + continue + } + values.set(arg, value) + } + const required = (flag: string): string => { + const v = values.get(flag) + if (v === undefined) throw new Error(`verify-findings: ${flag} is required`) + return v + } + const optionalNumber = (flag: string): number | undefined => { + const v = values.get(flag) + if (v === undefined) return undefined + const n = Number(v) + if (!Number.isFinite(n)) throw new Error(`verify-findings: ${flag} must be a number, got ${v}`) + return n + } + const findingsPath = required('--findings') + const out = required('--out') + const direct = values.has('--steps') || values.has('--image') || values.has('--cwd') + if (direct && corpora.length > 0) { + throw new Error('verify-findings: pass --steps/--image/--cwd or --corpus, not both') + } + let source: FindingReplaySource + if (direct) { + source = { + kind: 'direct', + stepsPath: required('--steps'), + image: required('--image'), + cwd: required('--cwd'), + ...(values.has('--case') ? { caseId: values.get('--case')! } : {}), + } + } else if (corpora.length > 0) { + source = { kind: 'corpus', corpora } + } else { + throw new Error('verify-findings: a replay source is required — --steps/--image/--cwd or --corpus') + } + return { + findingsPath, + source, + out, + fixCommand: values.get('--fix-command'), + stepTimeoutMs: optionalNumber('--step-timeout'), + prefixLimit: optionalNumber('--prefix-limit'), + baseUrl: values.get('--base-url') ?? process.env.SANDBOX_API_URL ?? DEFAULT_SANDBOX_BASE_URL, + apiKeyEnv: values.get('--api-key-env') ?? 'SANDBOX_API_KEY', + maxLifetimeSeconds: optionalNumber('--max-lifetime'), + } +} + +export async function cmdVerifyFindings(argv: readonly string[]): Promise { + const parsed = parseVerifyFindingsArgs(argv) + if (parsed === 'help') { + process.stdout.write(verifyFindingsUsage()) + return + } + const findings = readFindingsFile(parsed.findingsPath) + const apiKey = process.env[parsed.apiKeyEnv] + const run = await verifyFindings(findings, { + source: parsed.source, + out: parsed.out, + fixCommand: parsed.fixCommand, + stepTimeoutMs: parsed.stepTimeoutMs, + prefixLimit: parsed.prefixLimit, + apiKey, + baseUrl: parsed.baseUrl, + maxLifetimeSeconds: parsed.maxLifetimeSeconds, + onProgress: (message) => process.stderr.write(`${message}\n`), + }) + process.stdout.write(`${JSON.stringify(run, null, 2)}\n`) + process.stderr.write( + `verify-findings: ${run.verifications.length} finding(s) → ` + + `${run.counts.reproduced} reproduced, ${run.counts['fix-flipped']} fix-flipped, ` + + `${run.counts.divergent} divergent, ${run.counts['not-replayable']} not-replayable ` + + `(${run.executions} execution(s)) → ${run.out}\n`, + ) +} diff --git a/src/cli.ts b/src/cli.ts index da2145f..7b63902 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -48,6 +48,14 @@ import { importCodeTraceBench } from './codetracebench.js' import { buildPolicyEvidenceRecord, serializePolicyEvidence, writePolicyEvidenceFile } from './evidence.js' import { cmdReplayVerifyBatch } from './replay-batch.js' import { cmdReplayVerify } from './replay-verify.js' +import { + cmdVerifyFindings, + DEFAULT_SANDBOX_BASE_URL, + renderVerifiedFindingsSection, + verifyFindings, + type VerifyFindingsRun, +} from './analyze-verify.js' +import { parseCorpusFlag } from './replay-corpus.js' import { commandAnalyzer, commandRedactor, haloAnalyzer } from './external.js' import { hodoscopeAnalyzer } from './hodoscope.js' import { type TraceEvidenceFormatOption, exportTraceEvidenceFile, writeTraceEvidenceExportFile } from './file-export.js' @@ -158,6 +166,12 @@ interface Args { trajectoryDir?: string revision?: string concurrency: number + /** analyze --verify-findings: execute analyst findings as sandbox replay proofs. */ + verifyFindings: boolean + /** Repeatable `name=::` corpora resolving finding trajectories. */ + replayCorpora: string[] + /** Receipt root for --verify-findings; defaults next to --out. */ + verifyOut?: string } const DEFAULT_ANALYST_MODEL = 'gpt-5-mini' @@ -198,6 +212,8 @@ function parseArgs(argv: string[]): Args { noFindings: false, attrs: [], concurrency: 4, + verifyFindings: false, + replayCorpora: [], } for (let i = 1; i < argv.length; i++) { const arg = argv[i] @@ -240,6 +256,9 @@ function parseArgs(argv: string[]): Args { case '--no-spans': a.noSpans = true; break case '--no-findings': a.noFindings = true; break case '--analyzer': { const v = next(); if (v) a.analyzers.push(v); break } + case '--verify-findings': a.verifyFindings = true; break + case '--replay-corpus': { const v = next(); if (v) a.replayCorpora.push(v); break } + case '--verify-out': a.verifyOut = next(); break case '--analyzer-prompt': a.analyzerPrompt = next(); break case '--redactor': a.redactorCmd = next(); break case '--format': a.format = next(); break @@ -922,18 +941,55 @@ async function loadExportAttributes(args: Args): Promise async function cmdAnalyze(args: Args): Promise { if (args.supervisorRunDir) return cmdAnalyzeSupervisorRun(args.supervisorRunDir, args) const result = await investigate(args, { loadDefaultConfig: false }) + let report = result.report + let verifySummary = '' + if (args.verifyFindings) { + const run = await verifyAnalyzeFindings(args, result) + report = `${report}\n${renderVerifiedFindingsSection(run)}` + verifySummary = + `, verified: ${run.counts.reproduced} reproduced / ${run.counts['fix-flipped']} fix-flipped / ` + + `${run.counts.divergent} divergent / ${run.counts['not-replayable']} not-replayable → ${run.out}` + } if (args.out) { - await saveReport(args.out, result.report) + await saveReport(args.out, report) console.log( `report → ${args.out} (${result.findings.length} findings, ` + - `${result.pipelines.stuckLoops.findings.length} loops, OTLP: ${result.otlpPath})`, + `${result.pipelines.stuckLoops.findings.length} loops, OTLP: ${result.otlpPath}${verifySummary})`, ) } else { - console.log(result.report) + console.log(report) } assertAgenticAnalystsRan(args, result.agenticPerAnalyst) } +/** + * Executes the analyze run's findings as sandbox replay proofs. Findings name + * their trajectory through trace:// evidence; the executable steps and docker + * image come from --replay-corpus, because harness session stores carry no + * docker_config to replay — requiring the corpus up front fails louder than + * annotating every finding as not-replayable for the same missing reason. + */ +async function verifyAnalyzeFindings(args: Args, result: TraceInvestigationResult): Promise { + if (args.replayCorpora.length === 0) { + throw new Error( + 'analyze --verify-findings needs the executable trajectory source: pass ' + + '--replay-corpus name=:: (repeatable). Harness sessions ' + + 'carry no docker_config, so findings cannot be replayed from the session store alone.', + ) + } + if (result.findings.length === 0) { + throw new Error('analyze --verify-findings: the analysis produced no findings to verify') + } + const out = args.verifyOut ?? (args.out ? `${args.out}.verify` : 'traces-verify-findings') + return verifyFindings(result.findings, { + source: { kind: 'corpus', corpora: args.replayCorpora.map(parseCorpusFlag) }, + out, + apiKey: process.env.SANDBOX_API_KEY, + baseUrl: process.env.SANDBOX_API_URL ?? DEFAULT_SANDBOX_BASE_URL, + onProgress: (message) => process.stderr.write(`${message}\n`), + }) +} + /** * Supervision-tree view: what the TREE did (steers, spawn waves, concurrency, * idle wall, cost by role, accepted vs rejected), as opposed to the rest of @@ -1515,6 +1571,10 @@ Commands: replay-verify-batch Measure replayability + fix-flip rates across gold-labeled corpora: arm A per replayable case, LLM-generated arm-B fixes (--help for flags) + verify-findings + Execute recorded analyst findings as sandbox replay proofs: each + finding → reproduced | fix-flipped | divergent | not-replayable, + with a receipt directory per finding (--help for flags) evidence Emit compact session-evidence JSONL for downstream policy miners stream Emit JSONL trace stream events for live visualizers or replay watch Online observer: tail active sessions, notify on loops + semantic findings @@ -1569,6 +1629,12 @@ Options: --budget USD cap for agentic analysts --analyzer analyze: also run halo, hodoscope, or an installed command (repeatable) --analyzer-prompt

analyze: prompt passed to external analyzers (default: diagnose) + --verify-findings analyze: execute the findings as sandbox replay proofs and mark + each VERIFIED (receipt path) or UNVERIFIABLE (reason). Needs + --replay-corpus + a running sandbox (SANDBOX_API_KEY/_URL) + --replay-corpus name=:: + trajectory source for --verify-findings (repeatable) + --verify-out

receipt root for --verify-findings (default: <--out>.verify) --interval watch/stream: poll interval seconds (sessions 5, run tree 2) --window watch/stream: only sessions active in the last N minutes (default 30) --min-loop Min identical repeated calls to flag a loop (default 3) @@ -1605,6 +1671,10 @@ async function main(): Promise { await cmdReplayVerifyBatch(rawArgs.slice(1)) return } + if (rawArgs[0] === 'verify-findings') { + await cmdVerifyFindings(rawArgs.slice(1)) + return + } const parsedArgs = parseArgs(rawArgs) if (parsedArgs.help) { if ( diff --git a/src/index.ts b/src/index.ts index 625ad67..e3545e8 100644 --- a/src/index.ts +++ b/src/index.ts @@ -122,6 +122,7 @@ export * from './replay-corpus.js' // corpus enumeration: replayable cases + exc export * from './replay-batch.js' // batch runner: replayability + fix-flip rates export * from './replay-fix.js' // counterfactual patch synthesis (one LLM call per case) export * from './replay-wire.js' // analyst finding → replay-verify invocation +export * from './analyze-verify.js' // proof-carrying findings: verifyFindings() + receipts export * from './upload.js' // planUpload / executeUpload({ backend? }) export * from './upload-state.js' // dedup state diff --git a/tests/analyze-verify.test.ts b/tests/analyze-verify.test.ts new file mode 100644 index 0000000..151efed --- /dev/null +++ b/tests/analyze-verify.test.ts @@ -0,0 +1,414 @@ +import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, describe, expect, it } from 'vitest' +import { + classifyVerdict, + findingReplayStep, + findingTrajectoryId, + parseVerifyFindingsArgs, + readFindingsFile, + renderVerifiedFindingsSection, + resolveFindingReplayability, + type VerifiableFinding, + verifyFindings, +} from '../src/analyze-verify.js' +import { SUBMIT_ACTION_SIGNATURE } from '../src/replay-corpus.js' +import type { ReplayExecBackend, ReplayExecResult, ReplayVerdict } from '../src/replay-verify.js' +import { fixtureStep, writeFixtureCorpus } from './replay-corpus-fixture.js' + +let root: string +afterEach(() => { + if (root) rmSync(root, { recursive: true, force: true }) +}) + +function makeRoot(): string { + root = mkdtempSync(join(tmpdir(), 'analyze-verify-test-')) + return root +} + +function wrappedPayload(command: string): string { + const match = /printf %s ([A-Za-z0-9+/=]+) \| base64 -d \| sh$/.exec(command) + if (!match) throw new Error(`not a wrapped exec command: ${command}`) + return Buffer.from(match[1]!, 'base64').toString('utf8') +} + +function scriptedBackend( + script: (action: string) => ReplayExecResult, +): ReplayExecBackend & { executed: string[][] } { + const executed: string[][] = [] + return { + executed, + async open() { + const session: string[] = [] + executed.push(session) + return { + async exec(command: string): Promise { + const action = wrappedPayload(command) + session.push(action) + return script(action) + }, + async close() {}, + } + }, + } +} + +const steps = () => [ + fixtureStep(1, 'ls', 0, 'files'), + fixtureStep(2, 'sed -i broken file.c', 0), + fixtureStep(3, 'make target', 2, 'file.c:9:2: error: broken build\nstopped'), + fixtureStep(4, 'echo probe', 0, 'probe'), + fixtureStep(5, `echo ${SUBMIT_ACTION_SIGNATURE} && git diff`, null), +] + +function finding(overrides: Partial & { subject?: string }): VerifiableFinding { + return { + finding_id: 'f_test0001', + analyst_id: 'dspy-rlm', + area: 'incorrect', + claim: 'Step 3 is incorrect.', + subject: 'incorrect-step-3', + evidence_refs: [{ kind: 'span', uri: 'trace://traj-ok/span/step-3', excerpt: 'make target' }], + ...overrides, + } +} + +describe('findingReplayStep', () => { + it('prefers metadata.block_first_step over the subject', () => { + expect(findingReplayStep(finding({ subject: 'incorrect-step-9', metadata: { block_first_step: 3 } }))).toBe(3) + }) + + it('parses both subject forms and rejects everything else', () => { + expect(findingReplayStep(finding({ subject: 'incorrect-step-7' }))).toBe(7) + expect(findingReplayStep(finding({ subject: 'incorrect-steps-4-6-unescaped-consequence-8' }))).toBe(4) + expect(findingReplayStep(finding({ subject: 'knowledge-gap-tooling' }))).toBeNull() + expect(findingReplayStep({})).toBeNull() + }) +}) + +describe('findingTrajectoryId', () => { + it('reads the trajectory from trace:// evidence refs', () => { + expect(findingTrajectoryId(finding({}))).toBe('traj-ok') + expect(findingTrajectoryId(finding({ evidence_refs: [{ kind: 'artifact', uri: 'file:///x' }] }))).toBeNull() + expect(findingTrajectoryId(finding({ evidence_refs: [] }))).toBeNull() + }) +}) + +describe('resolveFindingReplayability', () => { + it('direct source: resolves a replayable finding with the recorded returncode', () => { + const dir = makeRoot() + const stepsPath = join(dir, 'steps.json') + writeFileSync(stepsPath, JSON.stringify(steps())) + const result = resolveFindingReplayability(finding({}), { + kind: 'direct', + stepsPath, + image: 'img:replay', + cwd: '/repo', + caseId: 'traj-ok', + }) + expect(result).toMatchObject({ + replayable: true, + resolved: { caseId: 'traj-ok', at: 3, recordedReturncode: 2, image: 'img:replay' }, + }) + }) + + it('direct source: names the precise not-replayable reason', () => { + const dir = makeRoot() + const stepsPath = join(dir, 'steps.json') + writeFileSync(stepsPath, JSON.stringify(steps())) + const source = { kind: 'direct', stepsPath, image: 'img:replay', cwd: '/repo', caseId: 'traj-ok' } as const + expect(resolveFindingReplayability(finding({ subject: 'no-step-here' }), source)).toMatchObject({ + replayable: false, + reason: expect.stringContaining('names no trajectory step'), + }) + expect(resolveFindingReplayability(finding({ subject: 'incorrect-step-99' }), source)).toMatchObject({ + replayable: false, + reason: expect.stringContaining('outside the trajectory'), + }) + expect(resolveFindingReplayability(finding({ subject: 'incorrect-step-5' }), source)).toMatchObject({ + replayable: false, + reason: expect.stringContaining('submit action'), + }) + const noReturncode = [...steps().slice(0, 3), fixtureStep(4, 'echo probe', null)] + writeFileSync(stepsPath, JSON.stringify(noReturncode)) + expect(resolveFindingReplayability(finding({ subject: 'incorrect-step-4' }), source)).toMatchObject({ + replayable: false, + reason: expect.stringContaining('recorded no returncode'), + }) + expect( + resolveFindingReplayability( + finding({ evidence_refs: [{ kind: 'span', uri: 'trace://other-traj/span/step-3' }] }), + source, + ), + ).toMatchObject({ + replayable: false, + reason: expect.stringContaining("cites trajectory 'other-traj'"), + }) + }) + + it('corpus source: resolves through the corpus and surfaces exclusion reasons', () => { + const dir = makeRoot() + const corpus = writeFixtureCorpus(dir, 'wire', [ + { + trajId: 'traj-ok', + steps: steps(), + goldIncorrectSteps: [3], + raw: { baseImage: 'example/img:1', runConfigCwd: '/repo', timeoutSeconds: 9 }, + }, + { trajId: 'traj-not-swe', steps: steps(), goldIncorrectSteps: [3] }, + ]) + expect(resolveFindingReplayability(finding({}), { kind: 'corpus', corpora: [corpus] })).toMatchObject({ + replayable: true, + resolved: { caseId: 'traj-ok', image: 'example/img:1', cwd: '/repo', at: 3, recordedStepTimeoutMs: 9000 }, + }) + expect( + resolveFindingReplayability( + finding({ evidence_refs: [{ kind: 'span', uri: 'trace://traj-not-swe/span/step-3' }] }), + { kind: 'corpus', corpora: [corpus] }, + ), + ).toMatchObject({ replayable: false, reason: expect.stringContaining('no-swe-raw-trajectory') }) + expect( + resolveFindingReplayability(finding({ evidence_refs: [] }), { kind: 'corpus', corpora: [corpus] }), + ).toMatchObject({ replayable: false, reason: expect.stringContaining('no trace://') }) + }) +}) + +describe('classifyVerdict', () => { + const verdict = (armAMatch: boolean, armB: { failureVanished: boolean } | null): ReplayVerdict => + ({ + armA: { failureSignatureMatch: armAMatch }, + armB, + }) as unknown as ReplayVerdict + + it('maps arm outcomes onto the enum', () => { + expect(classifyVerdict(verdict(true, null))).toBe('reproduced') + expect(classifyVerdict(verdict(true, { failureVanished: true }))).toBe('fix-flipped') + expect(classifyVerdict(verdict(true, { failureVanished: false }))).toBe('reproduced') + expect(classifyVerdict(verdict(false, null))).toBe('divergent') + expect(classifyVerdict(verdict(false, { failureVanished: true }))).toBe('divergent') + }) +}) + +describe('verifyFindings', () => { + function fixtureSource(dir: string) { + const corpus = writeFixtureCorpus(dir, 'wire', [ + { + trajId: 'traj-ok', + steps: steps(), + goldIncorrectSteps: [3], + raw: { baseImage: 'example/img:1', runConfigCwd: '/repo' }, + }, + ]) + return { kind: 'corpus', corpora: [corpus] } as const + } + + const reproducingBackend = () => + scriptedBackend((action) => + action === 'make target' + ? { exitCode: 2, stdout: 'stopped', stderr: 'file.c:9:2: error: broken build' } + : action === 'fix file.c && make target' + ? { exitCode: 0, stdout: 'built ok', stderr: '' } + : { exitCode: 0, stdout: '', stderr: '' }, + ) + + it('reproduces a finding and writes an executed receipt', async () => { + const dir = makeRoot() + const backend = reproducingBackend() + const run = await verifyFindings([finding({})], { + source: fixtureSource(dir), + out: join(dir, 'out'), + backend, + }) + expect(run.counts).toEqual({ reproduced: 1, 'fix-flipped': 0, divergent: 0, 'not-replayable': 0 }) + expect(run.executions).toBe(1) + const verification = run.verifications[0]! + expect(verification.verified).toBe('reproduced') + expect(verification.step).toBe(3) + const receipt = JSON.parse(readFileSync(join(verification.receipt, 'receipt.json'), 'utf8')) + expect(receipt).toMatchObject({ + finding_id: 'f_test0001', + verified: 'reproduced', + trajectory_id: 'traj-ok', + step: 3, + execution: { + armA: { command: 'make target', exitCode: 2, failureSignatureMatch: true }, + recordedReturncode: 2, + armB: null, + }, + }) + expect(existsSync(verification.verdict_path!)).toBe(true) + expect(existsSync(join(run.out, 'verifications.json'))).toBe(true) + expect(backend.executed).toEqual([['ls', 'sed -i broken file.c', 'make target']]) + }) + + it('fix-flips when the supplied corrected command makes the failure vanish', async () => { + const dir = makeRoot() + const run = await verifyFindings([finding({})], { + source: fixtureSource(dir), + out: join(dir, 'out'), + backend: reproducingBackend(), + fixCommand: 'fix file.c && make target', + }) + expect(run.verifications[0]!.verified).toBe('fix-flipped') + const receipt = JSON.parse(readFileSync(join(run.verifications[0]!.receipt, 'receipt.json'), 'utf8')) + expect(receipt.execution.armB).toMatchObject({ exitCode: 0, failureVanished: true }) + }) + + it('marks a non-reproducing execution divergent', async () => { + const dir = makeRoot() + const run = await verifyFindings([finding({})], { + source: fixtureSource(dir), + out: join(dir, 'out'), + backend: scriptedBackend(() => ({ exitCode: 0, stdout: 'all fine', stderr: '' })), + }) + expect(run.verifications[0]!.verified).toBe('divergent') + expect(run.counts.divergent).toBe(1) + }) + + it('writes an honest not-replayable receipt without touching the sandbox', async () => { + const dir = makeRoot() + const backend = scriptedBackend(() => ({ exitCode: 0, stdout: '', stderr: '' })) + const run = await verifyFindings( + [finding({ subject: 'incorrect-step-5', evidence_refs: [{ kind: 'span', uri: 'trace://traj-ok/span/step-5' }] })], + { source: fixtureSource(dir), out: join(dir, 'out'), backend }, + ) + const verification = run.verifications[0]! + expect(verification.verified).toBe('not-replayable') + expect(verification.reason).toContain('submit action') + expect(verification.verdict_path).toBeNull() + expect(backend.executed).toEqual([]) + const receipt = JSON.parse(readFileSync(join(verification.receipt, 'receipt.json'), 'utf8')) + expect(receipt.reason).toContain('submit action') + expect(receipt.execution).toBeNull() + }) + + it('shares one executed proof across findings accusing the same step', async () => { + const dir = makeRoot() + const backend = reproducingBackend() + const run = await verifyFindings( + [ + finding({}), + finding({ finding_id: 'f_test0002', subject: 'incorrect-step-4', metadata: { block_first_step: 3 } }), + ], + { source: fixtureSource(dir), out: join(dir, 'out'), backend }, + ) + expect(run.executions).toBe(1) + expect(backend.executed).toHaveLength(1) + expect(run.verifications[1]!).toMatchObject({ + verified: 'reproduced', + deduplicated_with: run.verifications[0]!.receipt, + verdict_path: run.verifications[0]!.verdict_path, + }) + const sharedReceipt = JSON.parse(readFileSync(join(run.verifications[1]!.receipt, 'receipt.json'), 'utf8')) + expect(sharedReceipt.execution).toMatchObject({ armA: { exitCode: 2 } }) + }) + + it('fails loud when replayable findings exist but the sandbox is unreachable', async () => { + const dir = makeRoot() + await expect( + verifyFindings([finding({})], { + source: fixtureSource(dir), + out: join(dir, 'out'), + apiKey: 'k', + baseUrl: 'http://127.0.0.1:1', + }), + ).rejects.toThrow(/sandbox API unreachable/) + await expect( + verifyFindings([finding({})], { source: fixtureSource(dir), out: join(dir, 'out') }), + ).rejects.toThrow(/SANDBOX_API_KEY/) + }) + + it('rejects an empty findings array', async () => { + const dir = makeRoot() + await expect( + verifyFindings([], { source: fixtureSource(dir), out: join(dir, 'out') }), + ).rejects.toThrow(/no findings/) + }) +}) + +describe('renderVerifiedFindingsSection', () => { + it('marks each finding VERIFIED with its receipt or UNVERIFIABLE with the reason', async () => { + const dir = makeRoot() + const corpus = writeFixtureCorpus(dir, 'wire', [ + { + trajId: 'traj-ok', + steps: steps(), + goldIncorrectSteps: [3], + raw: { baseImage: 'example/img:1', runConfigCwd: '/repo' }, + }, + ]) + const run = await verifyFindings( + [finding({}), finding({ finding_id: 'f_test0002', subject: 'not-a-step' })], + { + source: { kind: 'corpus', corpora: [corpus] }, + out: join(dir, 'out'), + backend: scriptedBackend((action) => + action === 'make target' + ? { exitCode: 2, stdout: '', stderr: 'file.c:9:2: error: broken build' } + : { exitCode: 0, stdout: '', stderr: '' }, + ), + }, + ) + const section = renderVerifiedFindingsSection(run) + expect(section).toContain('## Verified findings (executed replay)') + expect(section).toContain('2 finding(s) → 1 reproduced, 0 fix-flipped, 0 divergent, 1 not replayable') + expect(section).toContain('**VERIFIED** — reproduced') + expect(section).toContain(run.verifications[0]!.receipt) + expect(section).toContain('UNVERIFIABLE — subject') + }) +}) + +describe('readFindingsFile', () => { + it('accepts a bare array and an object with findings', () => { + const dir = makeRoot() + const arrayPath = join(dir, 'array.json') + writeFileSync(arrayPath, JSON.stringify([finding({})])) + expect(readFindingsFile(arrayPath)).toHaveLength(1) + const objectPath = join(dir, 'object.json') + writeFileSync(objectPath, JSON.stringify({ findings: [finding({}), finding({})] })) + expect(readFindingsFile(objectPath)).toHaveLength(2) + }) + + it('fails loud on anything else', () => { + const dir = makeRoot() + const badPath = join(dir, 'bad.json') + writeFileSync(badPath, JSON.stringify({ observations: [] })) + expect(() => readFindingsFile(badPath)).toThrow(/neither a findings array/) + writeFileSync(badPath, JSON.stringify([finding({}), 'not-an-object'])) + expect(() => readFindingsFile(badPath)).toThrow(/must be an object/) + }) +}) + +describe('parseVerifyFindingsArgs', () => { + it('parses the direct form', () => { + const parsed = parseVerifyFindingsArgs([ + '--findings', 'f.json', '--out', 'o', '--steps', 's.json', '--image', 'img:1', '--cwd', '/repo', + '--case', 'traj-ok', '--fix-command', 'true', '--prefix-limit', '2', + ]) + expect(parsed).toMatchObject({ + findingsPath: 'f.json', + out: 'o', + fixCommand: 'true', + prefixLimit: 2, + source: { kind: 'direct', stepsPath: 's.json', image: 'img:1', cwd: '/repo', caseId: 'traj-ok' }, + }) + }) + + it('parses the corpus form and rejects mixing or missing sources', () => { + const parsed = parseVerifyFindingsArgs([ + '--findings', 'f.json', '--out', 'o', '--corpus', 'h2=labels.json::prepared', + ]) + expect(parsed).toMatchObject({ + source: { kind: 'corpus', corpora: [{ name: 'h2', labelsPath: 'labels.json', preparedDir: 'prepared' }] }, + }) + expect(() => + parseVerifyFindingsArgs([ + '--findings', 'f.json', '--out', 'o', '--steps', 's.json', '--image', 'i', '--cwd', '/r', + '--corpus', 'h2=l::p', + ]), + ).toThrow(/not both/) + expect(() => parseVerifyFindingsArgs(['--findings', 'f.json', '--out', 'o'])).toThrow(/replay source/) + expect(parseVerifyFindingsArgs(['--help'])).toBe('help') + }) +})