feat(rl): verified-findings dataset — execution-verified labels as RL rows + the flywheel phase-2 spec - #532
Merged
Conversation
…as RL rows Join replay-verify batch verdicts (batch-report.json) with gold label corpora and normalized trajectories into RL-ready rows (agent-eval/verified-finding@0): trajectory prefix 1..k, gold step, execution verdict with exit codes and signatures, fix arm when present, and per-row provenance (label/steps/report sha256s, docker images, run ids). Every missing or inconsistent join throws — a partially joined dataset is never written. The batch report is authoritative for fix outcomes; per-case replay-verdict.json contributes divergence detail and run ids under loud cross-checks. Built run2-20260802 into ~/bench-cache/ctb-20260801/verified-dataset-v0 (22 rows: 16 reproduced, 13 signature-strict, 9 fix-flipped; rows.jsonl sha256 afcbfb7b21c14868f8654bb27d113f42ba66fd951bcdc64b5f32b1e8b33641e3, byte-identical across rebuilds). docs/verified-labels-flywheel.md maps which local session classes are replay-eligible and specifies the phase-2 capture changes (persist the sandbox image, envelope environment identity, exit codes in tool results).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Phase 1 of the verified-labels flywheel: a builder that joins replay-verification verdicts (arm-A reproduced / arm-B fix-flipped, exit codes, signatures) with gold labels and trajectory context into RL-ready rows — labels backed by execution, not annotation. Nobody can buy this data; it compounds with every future replay batch.
src/rl/verified-findings-dataset.ts(+19 tests) following the existingsrc/rl/grain;build-verified-dataset.mjsrunner; committed schema README sha-pinning the out-of-git artifact.rows.jsonlshaafcbfb7b…— deterministic (second build byte-identical), operator-re-verified.docs/verified-labels-flywheel.md: the own-traffic eligibility map — which session classes carry reconstructible environments, what capture is missing, ranked capture changes. This is the phase-2 spec for turning our production traffic into the flywheel's fuel.Proof
Row counts match the batch report exactly; invariant sweep over all 22 rows passed (prefix windows, action-at-k equality, sha pins); joins 22/22 with zero silent misses. Battery: lint clean, typecheck clean, 4704/4704 tests, build green, merge-tree clean. $0 LLM.