diff --git a/benchmarks/trace-analysis/LEADERBOARD.md b/benchmarks/trace-analysis/LEADERBOARD.md
new file mode 100644
index 00000000..9c9d58ab
--- /dev/null
+++ b/benchmarks/trace-analysis/LEADERBOARD.md
@@ -0,0 +1,277 @@
+# CodeTraceBench Official-Metric Leaderboard
+
+Every real-model trace-analysis configuration this repository has measured, ranked on CodeTraceBench's published metric, with the instrument-quality evidence needed to decide which numbers are trustworthy.
+Snapshot date: 2026-08-03.
+All numbers are extracted from run artifacts on disk; none are recomputed with different semantics (see [Reproduce](#reproduce) for the bit-match proof).
+
+## The metric
+
+**Official all-row F1** is CodeTraceBench's published score: per-trajectory F1 over predicted vs labeled incorrect step ids, averaged over every row of the split.
+The reference implementation is [`src/analyst/benchmark-public-calibration.ts`](../../src/analyst/benchmark-public-calibration.ts) (`officialCodeTraceF1`), which matches the scorer snippet in the upstream CodeTracer `README.md` ("Evaluation Metrics") symbol for symbol.
+Three properties matter when reading any number below:
+
+1. A row with an empty label set scores 0 regardless of the prediction, and a row where the analyst run failed scores 0.
+   The metric therefore mixes detection quality with split composition: a split with 16/32 label-empty rows has a hard ceiling of 0.50, and a split with 2/32 label-empty rows has a ceiling of 0.94.
+2. **Official all-row F1 is only comparable within a single split.**
+   Cross-split comparisons compare label composition, not analysts.
+3. Every artifact also records our diagnostic **scored micro F1** (micro precision/recall over labeled-positive rows plus solved label-empty controls, failures excluded from predictions but counted).
+   Micro F1 is reported as context, never as the ranking metric.
+
+## Citation rules
+
+1. Cite a number only together with its split's labels digest and status from the [instrument-quality ledger](#instrument-quality-ledger).
+   Numbers from SPENT splits are readable history and may not back new public claims.
+2. Same-rows only: a comparison row is honest only if both tools ran the identical trajectory set and labels (byte-verified by the `traceFiles[].sha256` and `labelsSha256` fields embedded in each artifact).
+   Numbers lifted from a paper on different rows are context, and must carry the not-same-rows caveat.
+3. Config identity is `protocolSha256` (and `implementationSha256`), not the narrative name.
+   Run identity for reproduction is `runIdentitySha256` plus `startedAt` (identical configs re-run share `runIdentitySha256`).
+4. A run with failed observations keeps its official score (failures score 0) but must disclose the failure count.
+   Runs from parallel measurement chains are flagged: parallel execution against this provider measurably destroys runs (see protocol notes).
+5. Smoke runs (fewer than the full split's rows) are never rankable.
+
+## Configurations
+
+All runs use analyst model `glm-5.2` (z.ai), dataset `NJU-LINK/CodeTraceBench` revision `aa213b84ffb6690fc37ca15766d6ca174ec36d4d`, runners `empty` + one analyst, `runnerOrderSeed 0`, census case selection unless noted.
+Narrative labels below come from the run naming and the `.evolve/progress.md` ledger; the digests are authoritative.
+
+| Label | `protocolSha256` (8) | `implementationSha256` (8) | Description |
+| --- | --- | --- | --- |
+| direct-v1 | `166e399c` | `4dba263b` | Retired one-shot direct runner, incorrect-step adapter; two arms: with final-test artifacts, and trajectory-only ("fair") |
+| direct-blocks | `2aa97505` | `b959a4c1` | One-shot direct runner, incorrect-block adapter |
+| rlm-stock-r0 | `bd0347aa` | `87e10f21` | Recursive DSPy RLM analyst, pre-GEPA stock prompt |
+| rlm-gepa (g) | `4006588e` | `027db213` | GEPA-round-1 optimized prompt; promoted to stock 2026-08-01 |
+| rlm-incumbent (inc) | `5811b534` | `027db213` | Incumbent stock prompt at certification time |
+| rlm-width-W (w) | `dc8e043a` | `1de4cccc` | Hand-designed width-adaptive arm; rejected by its pre-registered gate |
+| rlm-stock (cert2) | `747ac229` | `a3f5a820` | Shipped stock prompt at 2026-08-01 (post GEPA-r1 promotion), updated implementation |
+| rlm-gepa-r2 (g2) | `0629426b` | `a3f5a820` | GEPA-round-2 winner candidate; rejected (below stock on both cross-family splits) |
+| rlm-framing | `0bd09994` | `a3f5a820` | Family-framing prompt variant (dev smoke only) |
+| CodeTracer (pinned) | n/a | upstream `2d302191` | Upstream CodeTracer at revision `2d302191dd07e7c0c2da6f7a5e9451c7cbb62d34`, memory disabled, trajectory-only |
+| empty | n/a | n/a | Constant baseline embedded in every run; predicts nothing; official all-row F1 = 0 on every split |
+
+## Instrument-quality ledger
+
+Constant-rule calibration computed directly from each labels file under the official per-row semantics (`leaderboard/calibrate-splits.py`).
+`flag-last-step` predicts exactly the final step of every trajectory; `flag-all-steps` predicts every step.
+A split cannot certify a configuration whose score a constant rule approaches.
+
+| Split | Labels sha256 (8) | n | Positives | Label-empty | Gold steps | flag-last-step | last-step hit rows | flag-all-steps | Status |
+| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- |
+| mini-SWE cert32 ("dev32") | `5d8b4024` | 32 | 16 | 16 | 55 | 0.0000 | 0/32 | 0.0956 | SPENT (tuning split; published baseline) |
+| mini-SWE holdout-1 | `53af5ffe` | 32 | 16 | 16 | 76 | 0.0437 | 3/32 | 0.1125 | SPENT (used for GEPA train/selection) |
+| mini-SWE holdout-2 | `2db46579` | 32 | 30 | 2 | 188 | 0.0879 | 9/32 | 0.3326 | Burned once (2026-08-01 certification); mini-SWE claims closed |
+| mini-SWE split3 remainder-37 | `d0347ec7` | 37 | 37 | 0 | 58 | 0.1099 | 5/37 | 0.1125 | DEGENERATE for config selection (see finding below) |
+| mini-SWE thin-blind-28 (restored) | `bf573bec` | 28 | 28 | 0 | 40 | 0.1310 | 4/28 | 0.1061 | Smoke only; input-blindness-fixed restoration of split3 rows |
+| OpenHands cert32 | `2cf55793` | 32 | 16 | 16 | 87 | 0.0125 | 1/32 | 0.1021 | SPENT (burned 2026-08-01 cert2) |
+| Terminus2 cert32 | `24e47110` | 32 | 16 | 16 | 100 | 0.0000 | 0/32 | 0.0770 | SPENT (burned 2026-08-01 cert2) |
+| OpenHands dev pool | `2bd62a9d` | 464 | 146 | 318 | 494 | 0.0313 | 22/464 | 0.0455 | Tuning-legal |
+| Terminus2 dev pool | `8388ffde` | 188 | 50 | 138 | 338 | 0.0000 | 0/188 | 0.0395 | Tuning-legal |
+| SWE-agent 106 | `399cfed3` | 106 | 57 | 49 | 200 | 0.1895 | 38/106 | 0.0941 | CLEAN — imported 2026-08-02, no analyst run yet |
+
+### Finding: split3 cannot select between configurations
+
+On split3 (37 rows, all positive, only 58 gold steps — 1.6 per row), the three certified configurations land within 0.54 points of each other on the official metric (g 0.1844, inc 0.1830, w 0.1791) while the flag-last-step constant rule scores 0.1099.
+The metric spread between real configs is smaller than the gap to a trivial rule, and the two metrics disagree on the ranking (micro F1 orders w 0.1805 > g 0.1799 > inc 0.1693).
+Any leaderboard position established on this split is noise; the split is retained only as evidence about the thin-gold narrow regime.
+
+### Finding: SWE-agent labels carry a last-step artifact
+
+On the freshly imported SWE-agent 106 rows, flagging only each trajectory's final step scores official all-row F1 0.1895 by hitting 38/106 rows — higher than every measured cross-family configuration score on the OpenHands and Terminus2 cert32 splits (0.118–0.150).
+(A prior session note quoted this as 39/106; the executed recomputation in `leaderboard/calibrate-splits.py` gives 38/106.)
+Any future SWE-agent leaderboard entry must publish the constant-rule row beside the model row, or the number is not interpretable.
+
+### Finding: parallel measurement chains destroy runs
+
+`mp-tw-full-h1` (parallel chain) failed all 96/96 observations — 66 scored plus 30 unlabeled — in 4.3 minutes (96 cost-ledger records with unknown cost), while the serially executed `mp-tw-serial-h1` on the same split completed 94/96 observations.
+All rankable numbers below come from serial chains; parallel-chain runs are listed as non-rankable.
+
+### Note: holdout-2 composition
+
+Holdout-2 is cascade-skewed by construction: 30/32 rows positive, 188 gold steps.
+Its official numbers (0.48–0.54) are not comparable to any 16/32-positive split (ceiling differences dominate); they are comparable only within the holdout-2 table.
+
+## Leaderboards
+
+One table per split, ranked on official all-row F1.
+"Failed" counts scored observations (positives + trusted negatives) that errored; they score 0 and stay in the mean.
+Costs are settled totals from each run's `cost-ledger.jsonl`; the 20260730 trio costs are the same-rate estimates from that run's README.
+
+### mini-SWE cert32 — labels `5d8b4024`, 32 rows, SPENT
+
+| # | Config | Reps | Obs | Official all-row F1 | Micro F1 | Failed | Cost USD | Date | Artifact |
+| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- | --- |
+| 1 | rlm-width-W (serial) | 3 | 96 | 0.1526 | 0.3231 | 2/72 | 9.96 | 2026-08-01 | `salvaged-runs/devshm-20260803/mp-tw-serial-dev` |
+| 2 | direct-v1 (trajectory-only) | 2 | 64 | 0.1502 | 0.3099 | 2/48 | 1.16 | 2026-07-30 | repo `codetracebench-glm52-20260730/fair-result.json` |
+| 3 | direct-v1 (+ test artifacts) | 2 | 64 | 0.1347 | 0.3273 | 1/48 | 1.21 | 2026-07-30 | repo `codetracebench-glm52-20260730/result.json` |
+| 4 | direct-blocks | 2 | 64 | 0.1296 | 0.3007 | 7/48 | n/a | 2026-07-31 | repo `codetracebench-phasea-blocks-20260731` (= `salvaged-runs/ctb-phasea3`) |
+| 5 | CodeTracer (pinned, trajectory-only) | 2 | 64 | 0.1161 | 0.2754 | 2/48 | 7.27 | 2026-07-30 | repo `codetracebench-glm52-20260730/codetracer-result.json` |
+| 6 | rlm-stock-r0 | 2 | 64 | 0.1021 | 0.3282 | 3/48 | n/a | 2026-07-31 | repo `codetracebench-rlm-glm52-20260731` (= `salvaged-runs/rlm-full7`) |
+
+Non-rankable on this split: `mp-tw-full-dev` (parallel chain, 8/72 failed, official 0.1486, $9.79); `rlm-full5`/`rlm-full6` (46/48 and 34/48 failed, official 0.0179 each — broken runs); smokes `rlm-smoke9`, `rlm-chk`, `ctb-smoke`, `gepa-run/cli-proof`, `mp-tw-smoke-narrow` (1–4 obs).
+
+The direct-v1 rows and the CodeTracer row are the same-rows head-to-head published 2026-07-30: identical 32 trajectories, identical labels, same model, and CodeTracer's own normalizer preparing the inputs.
+On that run CodeTracer used 933 model calls, 10.44M uncached input tokens, and $7.27 (same-rate repricing; its own price config reported $63.64) against Agent Eval's 64 calls and $1.16–$1.21.
+
+### mini-SWE holdout-1 — labels `53af5ffe`, 32 rows, SPENT
+
+| # | Config | Reps | Obs | Official all-row F1 | Micro F1 | Failed | Cost USD | Date | Artifact |
+| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- | --- |
+| 1 | rlm-width-W (serial) | 3 | 96 | 0.1737 | 0.3542 | 2/66 | 11.33 | 2026-08-01 | `salvaged-runs/devshm-20260803/mp-tw-serial-h1` |
+
+Non-rankable: `mp-tw-full-h1` (parallel wipeout, 66/66 failed, official 0.0000); smokes `mp-tw-h1-probe`, `mp-tw-smoke-wide`, `mp-tw-smoke-wide-rep` (1–2 obs).
+
+### mini-SWE holdout-2 — labels `2db46579`, 32 rows (30 positive), burned once 2026-08-01
+
+| # | Config | Reps | Obs | Official all-row F1 | Micro F1 | Failed | Cost USD | Artifact |
+| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
+| 1 | rlm-gepa (g) | 2 | 64 | 0.5427 | 0.6288 | 0/60 | 7.96 | `certification/cert-g-h2` |
+| 2 | rlm-incumbent (inc) | 2 | 64 | 0.5246 | 0.5641 | 0/60 | 7.97 | `certification/cert-inc-h2` |
+| 3 | rlm-width-W (w) | 2 | 64 | 0.4805 | 0.5224 | 1/60 | 7.67 | `certification/cert-w-h2` |
+
+Constant flag-last-step on this split: 0.0879 — the ranked configs clear it by 5.5–6.2×, so holdout-2 discriminates.
+This is the certification that promoted the GEPA prompt to stock (evolve ledger, 2026-08-01: pooled-sealed micro 0.4809 vs incumbent 0.4285 over both burned splits).
+
+### mini-SWE split3 remainder-37 — labels `d0347ec7`, 37 rows, DEGENERATE
+
+| # | Config | Reps | Obs | Official all-row F1 | Micro F1 | Failed | Cost USD | Artifact |
+| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
+| 1 | rlm-gepa (g) | 2 | 74 | 0.1844 | 0.1799 | 0/74 | 9.07 | `certification/cert-g-s3` |
+| 2 | rlm-incumbent (inc) | 2 | 74 | 0.1830 | 0.1693 | 1/74 | 9.24 | `certification/cert-inc-s3` |
+| 3 | rlm-width-W (w) | 2 | 74 | 0.1791 | 0.1805 | 1/74 | 8.72 | `certification/cert-w-s3` |
+
+Ranked for completeness only; see the degeneracy finding above.
+
+### OpenHands cert32 — labels `2cf55793`, 32 rows, SPENT
+
+| # | Config | Reps | Obs | Official all-row F1 | Micro F1 | Failed | Cost USD | Artifact |
+| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
+| 1 | rlm-stock | 2 | 64 | 0.1381 | 0.2436 | 0/60 | 7.33 | `cert2/stock-oh` |
+| 2 | rlm-gepa-r2 (g2) | 2 | 64 | 0.1220 | 0.1828 | 1/60 | 8.15 | `cert2/g2-oh` |
+
+### Terminus2 cert32 — labels `24e47110`, 32 rows, SPENT
+
+| # | Config | Reps | Obs | Official all-row F1 | Micro F1 | Failed | Cost USD | Artifact |
+| ---: | --- | ---: | ---: | ---: | ---: | ---: | ---: | --- |
+| 1 | rlm-stock | 2 | 64 | 0.1497 | 0.1896 | 0/52 | 7.56 | `cert2/stock-t2` |
+| 2 | rlm-gepa-r2 (g2) | 2 | 64 | 0.1186 | 0.1633 | 2/52 | 8.03 | `cert2/g2-t2` |
+
+The cross-family result is the sharpest fact in this document: the prompt GEPA-optimized on mini-SWE traces transfers WORSE than stock to both other agent families (−1.6pp OpenHands, −3.1pp Terminus2 official).
+For scale, a flag-last-step constant rule reaches 0.1895 on the (different-split) SWE-agent labels — absolute scores in the 0.12–0.15 range must always be read against their own split's constant-rule calibration.
+`cert2/status.txt` records exit code 2 for both g2 arms; their artifacts contain complete 64-observation sets (1–2 failed observations).
+
+### Dev smokes (tuning-legal pools, non-rankable)
+
+| Run | Split (labels) | Obs | Official | Micro | Failed | Cost USD | Date |
+| --- | --- | ---: | ---: | ---: | ---: | ---: | --- |
+| `family-framing-smoke/stock-openhands` | OH dev 6-row smoke (`1b687899`) | 12 | 0.4611 | 0.3023 | 0/12 | 1.51 | 2026-08-03 |
+| `family-framing-smoke/framing-openhands` | OH dev 6-row smoke (`1b687899`) | 12 | 0.2149 | 0.2151 | 1/12 | 1.38 | 2026-08-03 |
+| `split3-restored/smoke-run` | thin-blind-28 subset (`bf573bec`) | 8 | 0.1958 | 0.2000 | 0/8 | n/a | 2026-08-01 |
+
+A `family-framing-smoke/stock-terminus2` run was in flight (holding the measurement mutex) at this snapshot's extraction time, 2026-08-03T06:32Z; it is not included.
+
+## Published-tool context (NOT same rows)
+
+The CodeTracer paper reports headline incorrect-step localization scores of roughly 48% and 50.9%.
+Those numbers were produced on the full curated dataset with different rows, different label composition, its own harness, and its cross-trajectory memory enabled — none of which hold for any row in this document, so they are not comparable to any number here.
+We additionally could not verify those figures against any artifact on disk at snapshot time (they do not appear in the pinned CodeTracer repository at `2d302191`); they are recorded here only as the upstream claim to be tested by a future same-rows run.
+The only honest CodeTracer comparison we hold is the pinned same-rows run above: 0.1161 official all-row F1 on the mini-SWE cert32 rows with memory disabled, against our 0.1347/0.1502 on identical inputs.
+
+## Full run ledger
+
+Every analyst-benchmark artifact holding an embedded `codeTraceCalibration`, including duplicates and broken runs.
+`Run id` is `runIdentitySha256` (8); identical configs re-run share it, so `startedAt` disambiguates.
+Paths are relative to `~/bench-cache/ctb-20260801/` unless prefixed `repo:` (this repository, `benchmarks/trace-analysis/`).
+
+| Artifact | Run id | Labels | Protocol | Reps | Official | Micro | Failed | Started |
+| --- | --- | --- | --- | ---: | ---: | ---: | ---: | --- |
+| repo:`codetracebench-glm52-20260730/result.json` | `044393f7` | `5d8b4024` | `166e399c` | 2 | 0.1347 | 0.3273 | 1 | 2026-07-30T15:31Z |
+| repo:`codetracebench-glm52-20260730/fair-result.json` | `34159faa` | `5d8b4024` | `166e399c` | 2 | 0.1502 | 0.3099 | 2 | 2026-07-30T15:35Z |
+| repo:`codetracebench-glm52-20260730/codetracer-result.json` | n/a | `5d8b4024` | upstream `2d302191` | 2 | 0.1161 | 0.2754 | 2 | 2026-07-30 |
+| repo:`codetracebench-phasea-blocks-20260731/result.json` | `ddc3e665` | `5d8b4024` | `2aa97505` | 2 | 0.1296 | 0.3007 | 7 | 2026-07-31T06:51Z |
+| repo:`codetracebench-rlm-glm52-20260731/result.json` | `e6304c8a` | `5d8b4024` | `bd0347aa` | 2 | 0.1021 | 0.3282 | 3 | 2026-07-31T09:42Z |
+| `certification/cert-g-h2` | `24883695` | `2db46579` | `4006588e` | 2 | 0.5427 | 0.6288 | 0 | 2026-08-01T12:53Z |
+| `certification/cert-w-h2` | `74ece843` | `2db46579` | `dc8e043a` | 2 | 0.4805 | 0.5224 | 1 | 2026-08-01T12:28Z |
+| `certification/cert-inc-h2` | `229d11c0` | `2db46579` | `5811b534` | 2 | 0.5246 | 0.5641 | 0 | 2026-08-01T12:04Z |
+| `certification/cert-g-s3` | `417161b6` | `d0347ec7` | `4006588e` | 2 | 0.1844 | 0.1799 | 0 | 2026-08-01T14:21Z |
+| `certification/cert-w-s3` | `df4ce98c` | `d0347ec7` | `dc8e043a` | 2 | 0.1791 | 0.1805 | 1 | 2026-08-01T13:51Z |
+| `certification/cert-inc-s3` | `dfff2ee1` | `d0347ec7` | `5811b534` | 2 | 0.1830 | 0.1693 | 1 | 2026-08-01T13:20Z |
+| `cert2/stock-oh` | `30d80958` | `2cf55793` | `747ac229` | 2 | 0.1381 | 0.2436 | 0 | 2026-08-01T22:51Z |
+| `cert2/stock-t2` | `abda3035` | `24e47110` | `747ac229` | 2 | 0.1497 | 0.1896 | 0 | 2026-08-01T23:37Z |
+| `cert2/g2-oh` | `0e688fd3` | `2cf55793` | `0629426b` | 2 | 0.1220 | 0.1828 | 1 | 2026-08-01T23:11Z |
+| `cert2/g2-t2` | `0f1d655c` | `24e47110` | `0629426b` | 2 | 0.1186 | 0.1633 | 2 | 2026-08-02T00:00Z |
+| `salvaged-runs/rlm-smoke9` | `b4b0f57e` | `5d8b4024` | `80f3ca7f` | 1 | 0.1250 | 0.4000 | 0 | smoke, 4 obs |
+| `salvaged-runs/rlm-chk` | `a9d72390` | `5d8b4024` | `80f3ca7f` | 1 | 0.0000 | 0.0000 | 0 | smoke, 2 obs |
+| `salvaged-runs/rlm-full5` | `1856e5b1` | `5d8b4024` | `bd0347aa` | 2 | 0.0179 | 0.0678 | 46 | broken, 2026-07-31T09:12Z |
+| `salvaged-runs/rlm-full6` | `1be7492a` | `5d8b4024` | `bd0347aa` | 2 | 0.0179 | 0.0593 | 34 | broken, 2026-07-31T09:23Z |
+| `salvaged-runs/rlm-full7` | `e6304c8a` | `5d8b4024` | `bd0347aa` | 2 | 0.1021 | 0.3282 | 3 | duplicate of repo rlm run (same `startedAt`) |
+| `salvaged-runs/ctb-smoke` | `5f60033c` | `5d8b4024` | `1e31880c` | 1 | 0.0000 | 0.0000 | 1 | smoke, 2 obs |
+| `salvaged-runs/ctb-phasea3` | `ddc3e665` | `5d8b4024` | `2aa97505` | 2 | 0.1296 | 0.3007 | 7 | duplicate of repo phasea run (same `startedAt`) |
+| `family-framing-smoke/stock-openhands` | `a5446d3f` | `1b687899` | `747ac229` | 2 | 0.4611 | 0.3023 | 0 | 2026-08-03T05:08Z |
+| `family-framing-smoke/framing-openhands` | `c83c5424` | `1b687899` | `0bd09994` | 2 | 0.2149 | 0.2151 | 1 | 2026-08-03T05:39Z |
+| `split3-restored/smoke-run` | `e8ee878d` | `bf573bec` | `6a9d8784` | 2 | 0.1958 | 0.2000 | 0 | smoke, 8 obs |
+| `gepa-run/cli-proof` | `8d4f6631` | `5d8b4024` | `cb2a5beb` | 1 | 0.5000 | 0.5000 | 0 | smoke, 1 obs |
+| `salvaged-runs/devshm-20260803/mp-tw-full-dev` | `526dc2ff` | `5d8b4024` | `dc8e043a` | 3 | 0.1486 | 0.3660 | 8 | parallel chain, 2026-08-01T09:04Z |
+| `salvaged-runs/devshm-20260803/mp-tw-full-h1` | `3664d62f` | `53af5ffe` | `bd0347aa` | 3 | 0.0000 | 0.0000 | 66 | parallel wipeout, 2026-08-01T09:04Z |
+| `salvaged-runs/devshm-20260803/mp-tw-serial-dev` | `526dc2ff` | `5d8b4024` | `dc8e043a` | 3 | 0.1526 | 0.3231 | 2 | 2026-08-01T10:42Z |
+| `salvaged-runs/devshm-20260803/mp-tw-serial-h1` | `5cdb66a4` | `53af5ffe` | `dc8e043a` | 3 | 0.1737 | 0.3542 | 2 | 2026-08-01T10:05Z |
+| `salvaged-runs/devshm-20260803/mp-tw-h1-probe` | `b6a4eebc` | `53af5ffe` | `dc8e043a` | 1 | 0.0000 | 0.0000 | 0 | smoke, 1 obs |
+| `salvaged-runs/devshm-20260803/mp-tw-smoke-narrow` | `ec347ceb` | `5d8b4024` | `bd0347aa` | 1 | 0.6667 | 0.6667 | 0 | smoke, 1 obs |
+| `salvaged-runs/devshm-20260803/mp-tw-smoke-wide` | `b6a4eebc` | `53af5ffe` | `dc8e043a` | 1 | 0.0000 | 0.0000 | 0 | smoke, 1 obs |
+| `salvaged-runs/devshm-20260803/mp-tw-smoke-wide-rep` | `550c5da5` | `53af5ffe` | `dc8e043a` | 2 | 0.7636 | 0.7619 | 0 | smoke, 2 obs |
+
+## Full digests
+
+Labels files (sha256 → local path at snapshot time):
+
+```
+5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412  repo benchmarks/trace-analysis/codetracebench-glm52-20260730/input-labels.json
+53af5ffe3962f3378f2d65419b92b8a56fe7d6c8efc619a0bc2b8f0872bc4f83  ~/bench-cache/ctb-20260801/ctb-holdout-labels.json
+2db46579b7993edc376acbbcacf67a1d0ddfcdb94e28930c2bb8dfcf1dc32fb2  ~/bench-cache/ctb-20260801/ctb-holdout2-labels.json
+d0347ec7a5ec9a07bd3fcd16aa06b07bcb33ffabca39cb0b0f7a564fb500ae08  ~/bench-cache/ctb-20260801/split3/ctb-split3-labels.json
+bf573beca3bd58c2f2937671dfaa54095f6438523d07caa22608e1e87b114c1d  ~/bench-cache/ctb-20260801/split3-restored/thin-blind-labels.json
+2cf557938f0c1d8d239ede37de91e0f5cb58a9f2854e2e74ab1baa764cf91fde  ~/bench-cache/ctb-20260801/oht2/ctb-openhands-cert32-labels.json
+24e471101c652aa0a7d9edc6dc9081d1a67589b1481f21c71552d47106cae3e6  ~/bench-cache/ctb-20260801/oht2/ctb-terminus2-cert32-labels.json
+2bd62a9d5ec785482611e1b02a1a829cf98c0777b6ae1ac9b1867b8fc9e4b8c6  ~/bench-cache/ctb-20260801/oht2/ctb-openhands-dev-labels.json
+8388ffde9c1b3b3f8f04afb4356a3b10abefda6db9ecd0216f65003bf6251958  ~/bench-cache/ctb-20260801/oht2/ctb-terminus2-dev-labels.json
+399cfed3b9b53dc47e61735ad0dde94acfeb3fcd63cd9879fe96e90df093d6d1  ~/bench-cache/ctb-20260801/sweagent/ctb-sweagent-labels.json
+```
+
+Only the first labels file is published in this repository today; publishing the remaining label manifests is gap item 2 below.
+Every run artifact additionally embeds per-trajectory `sha256` digests for its exact input archives (`inputs.traceFiles[]`), so any row can be byte-verified against re-prepared inputs.
+
+## Reproduce
+
+Extraction and calibration scripts live in [`leaderboard/`](./leaderboard/); paths inside them reference the operator bench cache described above.
+
+```bash
+# 1. Extract every embedded official-metric number (writes JSON to stdout)
+python3 benchmarks/trace-analysis/leaderboard/extract-official.py
+
+# 2. Prove extraction reads the exact field the scorer wrote (bit-match, exit 0 on success)
+python3 benchmarks/trace-analysis/leaderboard/extract-official.py --bitmatch \
+  ~/bench-cache/ctb-20260801/certification/cert-g-h2/result.json
+
+# 3. Constant-rule calibration table from raw labels
+python3 benchmarks/trace-analysis/leaderboard/calibrate-splits.py
+```
+
+Bit-match note: the recomputation reproduces the embedded doubles exactly (`cert-g-h2` → `0.5427281892181798`, `stock-oh` → `0.13809445762570763`, 20260730 baseline → `0.13470062923187923`).
+One trap is documented in the script: Python ≥3.12 `sum()` is Neumaier-compensated and differs from the TypeScript `reduce` fold by 1 ulp on 64-row sums, so the bit-match uses an explicit left fold.
+The extractor itself only ever *reads* embedded values; the bit-match exists to prove it reads the same field the scorer wrote.
+The extraction snapshot used for this document is committed as [`leaderboard/official-extract-20260803.json`](./leaderboard/official-extract-20260803.json).
+
+## Appendix: what a submission-ready public claim still needs
+
+Ranked by effort, smallest first.
+
+1. **File the upstream CodeTraceBench issue for the degenerate wave** (hours).
+   Report the split3 finding: 37-row all-positive remainder with 58 gold steps where three materially different analysts land within 0.54pp of the official metric and within 7.5pp of a flag-last-step constant rule, plus the last-step label artifact measured on the SWE-agent rows (0.1895 / 38 of 106 rows), plus the input-blindness normalizer defect that thin-blind-28 restores.
+   The constant-rule calibration script in this directory is the reproducible evidence.
+2. **Publish the label manifests** (hours).
+   Nine of ten label files above exist only in the operator bench cache; a public claim needs them (or their derivation scripts plus digests) in-repo.
+3. **Official-metric run of the shipping config on a clean split with ≥3 reps** (one mutex session, ~$40 at the measured ~$0.125/observation — 64-observation runs above cost $7.3–8.2 — for 106 rows × 3 reps).
+   Every existing shipping-config number sits on a SPENT or burned split; the SWE-agent 106 (imported, sealed, spot-proofed 3/3) is the only clean candidate, and its table must carry the constant-rule row (0.1895) beside the model row.
+   2-rep confidence intervals of ±0.15–0.28 (evolve ledger) cannot certify small deltas; 3+ reps minimum.
+4. **Same-rows CodeTracer head-to-head on that clean split** (one mutex session, ~$25–35 extrapolating $7.27/64 obs to 106 rows).
+   Pinned revision, memory disabled, byte-identical inputs via its own normalizer — the same protocol as the 20260730 trio.
+5. **Full official verified-1000 run** (days, ~$125 per repetition for our analyst at the measured ~$0.125/observation — ~$375 at 3 reps — plus a materially larger CodeTracer bill).
+   Only after items 1–4: the verified-1000 split mixes agent families whose labels carry the artifacts documented above, so publishing without the instrument ledger would reproduce the same interpretability failure this leaderboard exists to prevent.
diff --git a/benchmarks/trace-analysis/leaderboard/calibrate-splits.py b/benchmarks/trace-analysis/leaderboard/calibrate-splits.py
new file mode 100644
index 00000000..4a0d30a1
--- /dev/null
+++ b/benchmarks/trace-analysis/leaderboard/calibrate-splits.py
@@ -0,0 +1,55 @@
+#!/usr/bin/env python3
+"""Constant-rule calibration per split, under the official CodeTraceBench per-row F1.
+
+Semantics match agent-eval src/analyst/benchmark-public-calibration.ts:officialCodeTraceF1
+and the scorer snippet in CodeTracer README.md: per-row F1 over incorrect step ids,
+0 whenever either predicted or expected set is empty; mean over every row.
+"""
+import hashlib
+import json
+import statistics
+
+SPLITS = [
+    ('mini-SWE cert32 (dev32)', '/home/drew/code/agent-eval/benchmarks/trace-analysis/codetracebench-glm52-20260730/input-labels.json'),
+    ('mini-SWE holdout-1', '/home/drew/bench-cache/ctb-20260801/ctb-holdout-labels.json'),
+    ('mini-SWE holdout-2', '/home/drew/bench-cache/ctb-20260801/ctb-holdout2-labels.json'),
+    ('mini-SWE split3 remainder-37', '/home/drew/bench-cache/ctb-20260801/split3/ctb-split3-labels.json'),
+    ('mini-SWE thin-blind-28 (restored)', '/home/drew/bench-cache/ctb-20260801/split3-restored/thin-blind-labels.json'),
+    ('OpenHands cert32', '/home/drew/bench-cache/ctb-20260801/oht2/ctb-openhands-cert32-labels.json'),
+    ('Terminus2 cert32', '/home/drew/bench-cache/ctb-20260801/oht2/ctb-terminus2-cert32-labels.json'),
+    ('OpenHands dev pool', '/home/drew/bench-cache/ctb-20260801/oht2/ctb-openhands-dev-labels.json'),
+    ('Terminus2 dev pool', '/home/drew/bench-cache/ctb-20260801/oht2/ctb-terminus2-dev-labels.json'),
+    ('SWE-agent 106', '/home/drew/bench-cache/ctb-20260801/sweagent/ctb-sweagent-labels.json'),
+]
+
+
+def expected_steps(row):
+    steps = set()
+    for stage in row.get('incorrect_stages') or []:
+        for step in stage.get('incorrect_step_ids') or []:
+            steps.add(int(step))
+    return steps
+
+
+def row_f1(pred, expected):
+    matched = len(pred & expected)
+    precision = 0 if not pred else matched / len(pred)
+    recall = 0 if not expected else matched / len(expected)
+    return 0 if precision + recall == 0 else 2 * precision * recall / (precision + recall)
+
+
+print('| Split | labels sha256 (8) | n | positives | label-empty | gold steps | empty rule | flag-last-step | last-step hit rows | flag-all-steps |')
+print('| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |')
+for name, path in SPLITS:
+    raw = open(path, 'rb').read()
+    sha = hashlib.sha256(raw).hexdigest()[:8]
+    data = json.loads(raw)
+    rows = data if isinstance(data, list) else data['rows']
+    exp = [(expected_steps(r), int(r['step_count'])) for r in rows]
+    positives = sum(1 for e, _ in exp if e)
+    gold = sum(len(e) for e, _ in exp)
+    last = [row_f1({sc}, e) for e, sc in exp]
+    allsteps = [row_f1(set(range(1, sc + 1)), e) for e, sc in exp]
+    print(f'| {name} | `{sha}` | {len(rows)} | {positives} | {len(rows) - positives} | {gold} '
+          f'| 0.000 | {statistics.mean(last):.4f} | {sum(1 for v in last if v > 0)}/{len(rows)} '
+          f'| {statistics.mean(allsteps):.4f} |')
diff --git a/benchmarks/trace-analysis/leaderboard/extract-official.py b/benchmarks/trace-analysis/leaderboard/extract-official.py
new file mode 100644
index 00000000..11b16cb2
--- /dev/null
+++ b/benchmarks/trace-analysis/leaderboard/extract-official.py
@@ -0,0 +1,152 @@
+#!/usr/bin/env python3
+"""Extract embedded codeTraceCalibration official-metric numbers from every analyst-benchmark artifact.
+
+Never recomputes silently: the embedded officialAllRowF1 is the cited value.
+A separate --bitmatch mode recomputes one artifact's value from raw observations
+with the exact semantics of benchmark-public-calibration.ts:officialCodeTraceF1
+and asserts bit equality, proving the extraction reads the same field the scorer wrote.
+"""
+import json
+import re
+import sys
+import urllib.parse
+from pathlib import Path
+
+TRACE_DIR = Path(__file__).resolve().parents[1]
+BENCH_CACHE = Path.home() / 'bench-cache/ctb-20260801'
+
+ARTIFACTS = [
+    # in-repo published runs
+    str(TRACE_DIR / 'codetracebench-glm52-20260730/result.json'),
+    str(TRACE_DIR / 'codetracebench-glm52-20260730/fair-result.json'),
+    str(TRACE_DIR / 'codetracebench-glm52-20260730/codetracer-result.json'),
+    str(TRACE_DIR / 'codetracebench-phasea-blocks-20260731/result.json'),
+    str(TRACE_DIR / 'codetracebench-rlm-glm52-20260731/result.json'),
+    # certification (shipped-config runs)
+    str(BENCH_CACHE / 'certification/cert-g-h2/result.json'),
+    str(BENCH_CACHE / 'certification/cert-w-h2/result.json'),
+    str(BENCH_CACHE / 'certification/cert-inc-h2/result.json'),
+    str(BENCH_CACHE / 'certification/cert-g-s3/result.json'),
+    str(BENCH_CACHE / 'certification/cert-w-s3/result.json'),
+    str(BENCH_CACHE / 'certification/cert-inc-s3/result.json'),
+    # cert2 (OH/T2 stock + g2)
+    str(BENCH_CACHE / 'cert2/stock-oh/result.json'),
+    str(BENCH_CACHE / 'cert2/stock-t2/result.json'),
+    str(BENCH_CACHE / 'cert2/g2-oh/result.json'),
+    str(BENCH_CACHE / 'cert2/g2-t2/result.json'),
+    # salvaged
+    str(BENCH_CACHE / 'salvaged-runs/rlm-smoke9/result.json'),
+    str(BENCH_CACHE / 'salvaged-runs/rlm-chk/result.json'),
+    str(BENCH_CACHE / 'salvaged-runs/rlm-full5/result.json'),
+    str(BENCH_CACHE / 'salvaged-runs/rlm-full6/result.json'),
+    str(BENCH_CACHE / 'salvaged-runs/rlm-full7/result.json'),
+    str(BENCH_CACHE / 'salvaged-runs/ctb-smoke/result.json'),
+    str(BENCH_CACHE / 'salvaged-runs/ctb-phasea3/result.json'),
+    # smokes / probes
+    str(BENCH_CACHE / 'family-framing-smoke/stock-openhands/result.json'),
+    str(BENCH_CACHE / 'family-framing-smoke/framing-openhands/result.json'),
+    str(BENCH_CACHE / 'split3-restored/smoke-run/result.json'),
+    str(BENCH_CACHE / 'gepa-run/cli-proof/result.json'),
+    # selection-experiment runs salvaged from /dev/shm 2026-08-03
+    str(BENCH_CACHE / 'salvaged-runs/devshm-20260803/mp-tw-full-dev/result.json'),
+    str(BENCH_CACHE / 'salvaged-runs/devshm-20260803/mp-tw-full-h1/result.json'),
+    str(BENCH_CACHE / 'salvaged-runs/devshm-20260803/mp-tw-serial-dev/result.json'),
+    str(BENCH_CACHE / 'salvaged-runs/devshm-20260803/mp-tw-serial-h1/result.json'),
+    str(BENCH_CACHE / 'salvaged-runs/devshm-20260803/mp-tw-h1-probe/result.json'),
+    str(BENCH_CACHE / 'salvaged-runs/devshm-20260803/mp-tw-smoke-narrow/result.json'),
+    str(BENCH_CACHE / 'salvaged-runs/devshm-20260803/mp-tw-smoke-wide/result.json'),
+    str(BENCH_CACHE / 'salvaged-runs/devshm-20260803/mp-tw-smoke-wide-rep/result.json'),
+]
+
+STEP_URI = re.compile(r'^trace://([^/]+)/span/step-(\d+)$')
+
+
+def official_f1(observation):
+    expected = set()
+    score = observation['score']
+    for issue_id in list(score['matchedIssueIds']) + list(score['missedIssueIds']):
+        m = re.fullmatch(r'incorrect:(\d+)', issue_id)
+        if not m:
+            raise TypeError(f"{observation['caseId']}: invalid label {issue_id}")
+        expected.add(int(m.group(1)))
+    trajectory_id = (observation.get('caseMetadata') or {}).get('trajectoryId')
+    predicted = set()
+    if not observation.get('error'):
+        for finding in observation['findings']:
+            if finding['area'] != 'incorrect':
+                continue
+            for evidence in finding['evidence_refs']:
+                m = STEP_URI.match(evidence['uri'])
+                if not m or urllib.parse.unquote(m.group(1)) != trajectory_id:
+                    raise TypeError(f"{observation['caseId']}: bad evidence {evidence['uri']}")
+                predicted.add(int(m.group(2)))
+    matched = len(predicted & expected)
+    precision = 0 if not predicted else matched / len(predicted)
+    recall = 0 if not expected else matched / len(expected)
+    return 0 if precision + recall == 0 else (2 * precision * recall) / (precision + recall)
+
+
+def bitmatch(path):
+    d = json.loads(Path(path).read_text())
+    ok = True
+    for runner in d['codeTraceCalibration']['runners']:
+        rid = runner['runnerId']
+        obs = [o for o in d['result']['observations'] if o['runnerId'] == rid]
+        rows = [official_f1(o) for o in obs]
+        # explicit left fold: TS uses reduce((a,b)=>a+b,0); Python>=3.12 sum() is
+        # Neumaier-compensated and differs by 1 ulp on 64-row sums
+        total = 0.0
+        for value in rows:
+            total += value
+        recomputed = total / len(rows) if rows else None
+        embedded = runner['officialAllRowF1']
+        match = recomputed == embedded
+        ok = ok and match
+        print(f'{path} runner={rid} embedded={embedded!r} recomputed={recomputed!r} '
+              f'rows={len(rows)} BITMATCH={match}')
+    return ok
+
+
+def extract():
+    out = []
+    for path in ARTIFACTS:
+        p = Path(path)
+        if not p.exists():
+            out.append({'path': path, 'missing': True})
+            continue
+        d = json.loads(p.read_text())
+        cal = d.get('codeTraceCalibration')
+        inputs = d.get('inputs', {})
+        prov = d.get('result', {}).get('provenance', {})
+        meta = prov.get('metadata', {})
+        row = {
+            'path': path,
+            'runIdentitySha256': d.get('runIdentitySha256'),
+            'datasetSplit': inputs.get('datasetSplit'),
+            'datasetRevision': inputs.get('datasetRevision'),
+            'labelsSha256': inputs.get('labelsSha256'),
+            'sourceRowCount': inputs.get('sourceRowCount'),
+            'model': meta.get('model'),
+            'rlmSamples': meta.get('rlmSamples'),
+            'outputAdapter': meta.get('outputAdapter'),
+            'protocolSha256': meta.get('protocolSha256'),
+            'implementationSha256': meta.get('implementationSha256'),
+            'promptVariant': meta.get('promptVariant') or meta.get('instructionsVariant'),
+            'metadataExtra': {k: v for k, v in meta.items()
+                              if k not in ('model', 'rlmSamples', 'outputAdapter', 'protocolSha256',
+                                           'implementationSha256')
+                              and isinstance(v, (str, int, float, bool))},
+            'repetitions': prov.get('repetitions'),
+            'caseCount': prov.get('caseCount'),
+            'startedAt': prov.get('startedAt'),
+            'endedAt': prov.get('endedAt'),
+            'runners': cal['runners'] if cal else None,
+        }
+        out.append(row)
+    print(json.dumps(out, indent=1))
+
+
+if __name__ == '__main__':
+    if len(sys.argv) > 1 and sys.argv[1] == '--bitmatch':
+        sys.exit(0 if bitmatch(sys.argv[2]) else 1)
+    extract()
diff --git a/benchmarks/trace-analysis/leaderboard/official-extract-20260803.json b/benchmarks/trace-analysis/leaderboard/official-extract-20260803.json
new file mode 100644
index 00000000..ca2dbab0
--- /dev/null
+++ b/benchmarks/trace-analysis/leaderboard/official-extract-20260803.json
@@ -0,0 +1,2459 @@
+[
+ {
+  "path": "/home/drew/code/agent-eval/benchmarks/trace-analysis/codetracebench-glm52-20260730/result.json",
+  "runIdentitySha256": "044393f7d937c621b568fd4d4ec92a9c9d0a4040d55db9b3faf72647e0270cdb",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": "codetracebench-incorrect-step",
+  "protocolSha256": "166e399c9a93c9806b007273bf0b54078709389c52c6a33e3cbce0f554dab302",
+  "implementationSha256": "4dba263b6256a30d56c7fdb2d992d3a953c0035d731f359b704db806f68f75ac",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "1e03f2daed356d60316aabefb407ec1e437ac94d408d61eea4ae096e9c6fbb5b",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-07-30T15:31:48.463Z",
+  "endedAt": "2026-07-30T15:34:30.624Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 48,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "model",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 47,
+    "failedRuns": 1,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 165,
+    "matchedIncorrectSteps": 45,
+    "officialAllRowF1": 0.13470062923187923,
+    "officialAllRowRuns": 64,
+    "precision": 0.2727272727272727,
+    "recall": 0.4090909090909091,
+    "f1": 0.32727272727272727,
+    "trustedNegativeFalsePositiveRate": 0.9333333333333333,
+    "trustedNegativeFailureRate": 0.0625,
+    "unlabeledPredictionRate": 1,
+    "unlabeledFailureRate": 0
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/code/agent-eval/benchmarks/trace-analysis/codetracebench-glm52-20260730/fair-result.json",
+  "runIdentitySha256": "34159faaf98bcbe9ceb5ffa061004555276619b6f74938010eebd523fa9953d8",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": "codetracebench-incorrect-step",
+  "protocolSha256": "166e399c9a93c9806b007273bf0b54078709389c52c6a33e3cbce0f554dab302",
+  "implementationSha256": "4dba263b6256a30d56c7fdb2d992d3a953c0035d731f359b704db806f68f75ac",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "1e03f2daed356d60316aabefb407ec1e437ac94d408d61eea4ae096e9c6fbb5b",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-07-30T15:35:45.811Z",
+  "endedAt": "2026-07-30T15:38:26.585Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 48,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "model",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 46,
+    "failedRuns": 2,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 174,
+    "matchedIncorrectSteps": 44,
+    "officialAllRowF1": 0.15021525367393335,
+    "officialAllRowRuns": 64,
+    "precision": 0.25287356321839083,
+    "recall": 0.4,
+    "f1": 0.3098591549295775,
+    "trustedNegativeFalsePositiveRate": 0.9375,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0.875,
+    "unlabeledFailureRate": 0
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/code/agent-eval/benchmarks/trace-analysis/codetracebench-glm52-20260730/codetracer-result.json",
+  "runIdentitySha256": null,
+  "datasetSplit": null,
+  "datasetRevision": null,
+  "labelsSha256": null,
+  "sourceRowCount": null,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": null,
+  "protocolSha256": null,
+  "implementationSha256": null,
+  "promptVariant": null,
+  "metadataExtra": {
+   "upstream": "NJU-LINK/CodeTracer",
+   "upstreamRevision": "2d302191dd07e7c0c2da6f7a5e9451c7cbb62d34",
+   "memoryEnabled": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-07-30T00:00:00.000Z",
+  "endedAt": "2026-07-30T00:00:00.000Z",
+  "runners": [
+   {
+    "runnerId": "codetracer",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 46,
+    "failedRuns": 2,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 166,
+    "matchedIncorrectSteps": 38,
+    "officialAllRowF1": 0.11609110809178744,
+    "officialAllRowRuns": 64,
+    "precision": 0.2289156626506024,
+    "recall": 0.34545454545454546,
+    "f1": 0.2753623188405797,
+    "trustedNegativeFalsePositiveRate": 0.8571428571428571,
+    "trustedNegativeFailureRate": 0.125,
+    "unlabeledPredictionRate": 0.8666666666666667,
+    "unlabeledFailureRate": 0.0625
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/code/agent-eval/benchmarks/trace-analysis/codetracebench-phasea-blocks-20260731/result.json",
+  "runIdentitySha256": "ddc3e6654179c0803bf5e234e3a3ff9b6a97f7d3d53cf2ce5368e078c573ea1f",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "2aa975055cbb8c8bc7295dcd4fbcb453b60847791776ac25a0b772f39ac68d0f",
+  "implementationSha256": "b959a4c19de0ea12bbdbb2bcc883e5c2a9920c96087b0dd9b931446c656f0463",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "779ff0146f56c14a059aac9276f4b673ee70515afe95959a62a9f2d376c688dc",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-07-31T06:51:12.760Z",
+  "endedAt": "2026-07-31T06:54:43.504Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 48,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "direct",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 41,
+    "failedRuns": 7,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 196,
+    "matchedIncorrectSteps": 46,
+    "officialAllRowF1": 0.12956512871373552,
+    "officialAllRowRuns": 64,
+    "precision": 0.23469387755102042,
+    "recall": 0.41818181818181815,
+    "f1": 0.30065359477124187,
+    "trustedNegativeFalsePositiveRate": 0.9285714285714286,
+    "trustedNegativeFailureRate": 0.125,
+    "unlabeledPredictionRate": 1,
+    "unlabeledFailureRate": 0.125
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/code/agent-eval/benchmarks/trace-analysis/codetracebench-rlm-glm52-20260731/result.json",
+  "runIdentitySha256": "e6304c8a719cb7793d259c265fbebd6f485e0c72fe43a8afeae1bb7c1b8adf9e",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "bd0347aa7e7e2879832631741035ccbbe7cdef0af6bab2c6e0cf901937aff128",
+  "implementationSha256": "87e10f211b028ccb897969f7a8759179b7d75d7f06f3ea511dc6be71a7c69b5c",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "1ca280998a9f3a416b40cadb9c14789ada490a727a4ec78491c92bb2022e6c95",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-07-31T09:42:24.063Z",
+  "endedAt": "2026-07-31T10:09:44.276Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 48,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 45,
+    "failedRuns": 3,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 152,
+    "matchedIncorrectSteps": 43,
+    "officialAllRowF1": 0.10208276098901098,
+    "officialAllRowRuns": 64,
+    "precision": 0.28289473684210525,
+    "recall": 0.39090909090909093,
+    "f1": 0.3282442748091603,
+    "trustedNegativeFalsePositiveRate": 0.8,
+    "trustedNegativeFailureRate": 0.0625,
+    "unlabeledPredictionRate": 0.7333333333333333,
+    "unlabeledFailureRate": 0.0625
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/certification/cert-g-h2/result.json",
+  "runIdentitySha256": "24883695f29e0b928f3a55d000e985682d18f810eb63c2c70b00e95418a34fac",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "2db46579b7993edc376acbbcacf67a1d0ddfcdb94e28930c2bb8dfcf1dc32fb2",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "4006588e33c5b49fcdecc0cff424a1ebed20f42589233d42965d0b9256a273eb",
+  "implementationSha256": "027db213af78776e25e68914321cd599fe7a175a9f791b730067c2f943c2c0db",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "a3ea8abf439c0907da6c685b3d00becca30baa6953753bb755442a3ddb306dae",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-08-01T12:53:01.011Z",
+  "endedAt": "2026-08-01T13:20:16.918Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 60,
+    "positiveRuns": 60,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 4,
+    "failedLabelEmptyRuns": 4,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 60,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 376,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 60,
+    "positiveRuns": 60,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 4,
+    "failedLabelEmptyRuns": 4,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 60,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 376,
+    "predictedIncorrectSteps": 416,
+    "matchedIncorrectSteps": 249,
+    "officialAllRowF1": 0.5427281892181798,
+    "officialAllRowRuns": 64,
+    "precision": 0.5985576923076923,
+    "recall": 0.6622340425531915,
+    "f1": 0.6287878787878787,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": 1,
+    "unlabeledFailureRate": 0
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/certification/cert-w-h2/result.json",
+  "runIdentitySha256": "74ece8433fbb27610929328e1f5e8775d3a6a7732baca7c3c26fa277badd2ce0",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "2db46579b7993edc376acbbcacf67a1d0ddfcdb94e28930c2bb8dfcf1dc32fb2",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "dc8e043acacb26a5a4400881d83caee35add1983b0fe2705c9aaa119aafa1add",
+  "implementationSha256": "1de4cccc40c09acdd96c98fdc3fe62eec42a05c1d43a2337f83d7d2797f40054",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "701ea59184f3d227cf590ea3f16d24d65dbed2ad5644f8294801ceb66a30d4ef",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-08-01T12:28:36.480Z",
+  "endedAt": "2026-08-01T12:53:00.353Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 60,
+    "positiveRuns": 60,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 4,
+    "failedLabelEmptyRuns": 4,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 60,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 376,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 60,
+    "positiveRuns": 60,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 4,
+    "failedLabelEmptyRuns": 4,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 59,
+    "failedRuns": 1,
+    "expectedIncorrectSteps": 376,
+    "predictedIncorrectSteps": 405,
+    "matchedIncorrectSteps": 204,
+    "officialAllRowF1": 0.48051112661978174,
+    "officialAllRowRuns": 64,
+    "precision": 0.5037037037037037,
+    "recall": 0.5425531914893617,
+    "f1": 0.5224071702944942,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": 1,
+    "unlabeledFailureRate": 0
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/certification/cert-inc-h2/result.json",
+  "runIdentitySha256": "229d11c08ecb6ad786564d084f5e4afbbed47f92cb0c46a502a9c3566b0194fe",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "2db46579b7993edc376acbbcacf67a1d0ddfcdb94e28930c2bb8dfcf1dc32fb2",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "5811b534f7e6d9153403361b5663dce3b3558a67eff43d9cc2ec1c62ae9d3eff",
+  "implementationSha256": "027db213af78776e25e68914321cd599fe7a175a9f791b730067c2f943c2c0db",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "a3ea8abf439c0907da6c685b3d00becca30baa6953753bb755442a3ddb306dae",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-08-01T12:04:03.758Z",
+  "endedAt": "2026-08-01T12:28:35.869Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 60,
+    "positiveRuns": 60,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 4,
+    "failedLabelEmptyRuns": 4,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 60,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 376,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 60,
+    "positiveRuns": 60,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 4,
+    "failedLabelEmptyRuns": 4,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 60,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 376,
+    "predictedIncorrectSteps": 482,
+    "matchedIncorrectSteps": 242,
+    "officialAllRowF1": 0.5246150972163016,
+    "officialAllRowRuns": 64,
+    "precision": 0.5020746887966805,
+    "recall": 0.6436170212765957,
+    "f1": 0.5641025641025641,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": 0.75,
+    "unlabeledFailureRate": 0
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/certification/cert-g-s3/result.json",
+  "runIdentitySha256": "417161b661138fb03977e475ae76be22e54d2153b32bb1842c0a6dde49ecc200",
+  "datasetSplit": "verified-miniswe-normalizer-remainder-37",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "d0347ec7a5ec9a07bd3fcd16aa06b07bcb33ffabca39cb0b0f7a564fb500ae08",
+  "sourceRowCount": 37,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "4006588e33c5b49fcdecc0cff424a1ebed20f42589233d42965d0b9256a273eb",
+  "implementationSha256": "027db213af78776e25e68914321cd599fe7a175a9f791b730067c2f943c2c0db",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "a3ea8abf439c0907da6c685b3d00becca30baa6953753bb755442a3ddb306dae",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 37,
+  "startedAt": "2026-08-01T14:21:52.449Z",
+  "endedAt": "2026-08-01T14:53:25.347Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 74,
+    "positiveRuns": 74,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 74,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 116,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 74,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 74,
+    "positiveRuns": 74,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 74,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 116,
+    "predictedIncorrectSteps": 273,
+    "matchedIncorrectSteps": 35,
+    "officialAllRowF1": 0.18444190944190944,
+    "officialAllRowRuns": 74,
+    "precision": 0.1282051282051282,
+    "recall": 0.3017241379310345,
+    "f1": 0.1799485861182519,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/certification/cert-w-s3/result.json",
+  "runIdentitySha256": "df4ce98c1455f35b99cc8d05f9a2d77be0b2ae915b18e0c59c65339002b3c2d3",
+  "datasetSplit": "verified-miniswe-normalizer-remainder-37",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "d0347ec7a5ec9a07bd3fcd16aa06b07bcb33ffabca39cb0b0f7a564fb500ae08",
+  "sourceRowCount": 37,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "dc8e043acacb26a5a4400881d83caee35add1983b0fe2705c9aaa119aafa1add",
+  "implementationSha256": "1de4cccc40c09acdd96c98fdc3fe62eec42a05c1d43a2337f83d7d2797f40054",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "701ea59184f3d227cf590ea3f16d24d65dbed2ad5644f8294801ceb66a30d4ef",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 37,
+  "startedAt": "2026-08-01T13:51:55.675Z",
+  "endedAt": "2026-08-01T14:21:51.679Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 74,
+    "positiveRuns": 74,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 74,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 116,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 74,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 74,
+    "positiveRuns": 74,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 73,
+    "failedRuns": 1,
+    "expectedIncorrectSteps": 116,
+    "predictedIncorrectSteps": 294,
+    "matchedIncorrectSteps": 37,
+    "officialAllRowF1": 0.1790651915651916,
+    "officialAllRowRuns": 74,
+    "precision": 0.12585034013605442,
+    "recall": 0.31896551724137934,
+    "f1": 0.18048780487804877,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/certification/cert-inc-s3/result.json",
+  "runIdentitySha256": "dfff2ee1bfeaa02073917d6ff32e461c3526974cd7f45e058a40d0509ed573df",
+  "datasetSplit": "verified-miniswe-normalizer-remainder-37",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "d0347ec7a5ec9a07bd3fcd16aa06b07bcb33ffabca39cb0b0f7a564fb500ae08",
+  "sourceRowCount": 37,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "5811b534f7e6d9153403361b5663dce3b3558a67eff43d9cc2ec1c62ae9d3eff",
+  "implementationSha256": "027db213af78776e25e68914321cd599fe7a175a9f791b730067c2f943c2c0db",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "a3ea8abf439c0907da6c685b3d00becca30baa6953753bb755442a3ddb306dae",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 37,
+  "startedAt": "2026-08-01T13:20:18.235Z",
+  "endedAt": "2026-08-01T13:51:54.914Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 74,
+    "positiveRuns": 74,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 74,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 116,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 74,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 74,
+    "positiveRuns": 74,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 73,
+    "failedRuns": 1,
+    "expectedIncorrectSteps": 116,
+    "predictedIncorrectSteps": 333,
+    "matchedIncorrectSteps": 38,
+    "officialAllRowF1": 0.1829867204867205,
+    "officialAllRowRuns": 74,
+    "precision": 0.11411411411411411,
+    "recall": 0.3275862068965517,
+    "f1": 0.16926503340757237,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/cert2/stock-oh/result.json",
+  "runIdentitySha256": "30d80958be3d71f56a780850c7791cb4d078a8d98bc696140181611f94f7be95",
+  "datasetSplit": "sealed-openhands-cert32-20260801",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "2cf557938f0c1d8d239ede37de91e0f5cb58a9f2854e2e74ab1baa764cf91fde",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "747ac229be6729bd4bbc24e8be4f2a510663bd062dd748c430d05c9d580944df",
+  "implementationSha256": "a3f5a8203f0a7c74736170c53100c50c4db1cf84381533002f0539ef1e7a4c6d",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "a3ea8abf439c0907da6c685b3d00becca30baa6953753bb755442a3ddb306dae",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-08-01T22:51:09.854Z",
+  "endedAt": "2026-08-01T23:11:49.636Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 60,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 28,
+    "unlabeledRuns": 4,
+    "failedLabelEmptyRuns": 4,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 60,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 174,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 60,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 28,
+    "unlabeledRuns": 4,
+    "failedLabelEmptyRuns": 4,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 60,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 174,
+    "predictedIncorrectSteps": 179,
+    "matchedIncorrectSteps": 43,
+    "officialAllRowF1": 0.13809445762570763,
+    "officialAllRowRuns": 64,
+    "precision": 0.24022346368715083,
+    "recall": 0.2471264367816092,
+    "f1": 0.24362606232294617,
+    "trustedNegativeFalsePositiveRate": 0.5,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 1,
+    "unlabeledFailureRate": 0
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/cert2/stock-t2/result.json",
+  "runIdentitySha256": "abda3035ded04b3625981b2073d6b24a152fb4223836471481d4858f8a9589a3",
+  "datasetSplit": "sealed-terminus2-cert32-20260801",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "24e471101c652aa0a7d9edc6dc9081d1a67589b1481f21c71552d47106cae3e6",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "747ac229be6729bd4bbc24e8be4f2a510663bd062dd748c430d05c9d580944df",
+  "implementationSha256": "a3f5a8203f0a7c74736170c53100c50c4db1cf84381533002f0539ef1e7a4c6d",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "a3ea8abf439c0907da6c685b3d00becca30baa6953753bb755442a3ddb306dae",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-08-01T23:37:49.522Z",
+  "endedAt": "2026-08-02T00:00:09.383Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 52,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 20,
+    "unlabeledRuns": 12,
+    "failedLabelEmptyRuns": 12,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 52,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 200,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 52,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 20,
+    "unlabeledRuns": 12,
+    "failedLabelEmptyRuns": 12,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 52,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 200,
+    "predictedIncorrectSteps": 222,
+    "matchedIncorrectSteps": 40,
+    "officialAllRowF1": 0.14974177716776402,
+    "officialAllRowRuns": 64,
+    "precision": 0.18018018018018017,
+    "recall": 0.2,
+    "f1": 0.1895734597156398,
+    "trustedNegativeFalsePositiveRate": 0.6,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 1,
+    "unlabeledFailureRate": 0
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/cert2/g2-oh/result.json",
+  "runIdentitySha256": "0e688fd3cba3aeb8f9326811960ab9f6e9fd4e9191a40fe022f5b47fc1148612",
+  "datasetSplit": "sealed-openhands-cert32-20260801",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "2cf557938f0c1d8d239ede37de91e0f5cb58a9f2854e2e74ab1baa764cf91fde",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "0629426b2cdb70b7ac9c47c6a373db1679999a2c01cc09e3561540c3ac8644dc",
+  "implementationSha256": "a3f5a8203f0a7c74736170c53100c50c4db1cf84381533002f0539ef1e7a4c6d",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "a3ea8abf439c0907da6c685b3d00becca30baa6953753bb755442a3ddb306dae",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-08-01T23:11:51.588Z",
+  "endedAt": "2026-08-01T23:37:33.702Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 60,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 28,
+    "unlabeledRuns": 4,
+    "failedLabelEmptyRuns": 4,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 60,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 174,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 60,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 28,
+    "unlabeledRuns": 4,
+    "failedLabelEmptyRuns": 4,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 59,
+    "failedRuns": 1,
+    "expectedIncorrectSteps": 174,
+    "predictedIncorrectSteps": 198,
+    "matchedIncorrectSteps": 34,
+    "officialAllRowF1": 0.12195734957659299,
+    "officialAllRowRuns": 64,
+    "precision": 0.1717171717171717,
+    "recall": 0.19540229885057472,
+    "f1": 0.1827956989247312,
+    "trustedNegativeFalsePositiveRate": 0.5,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 1,
+    "unlabeledFailureRate": 0
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/cert2/g2-t2/result.json",
+  "runIdentitySha256": "0f1d655c99d56c6c88c3f13101cd97d844eaa740be4a0c17214943f2cb5d9406",
+  "datasetSplit": "sealed-terminus2-cert32-20260801",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "24e471101c652aa0a7d9edc6dc9081d1a67589b1481f21c71552d47106cae3e6",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "0629426b2cdb70b7ac9c47c6a373db1679999a2c01cc09e3561540c3ac8644dc",
+  "implementationSha256": "a3f5a8203f0a7c74736170c53100c50c4db1cf84381533002f0539ef1e7a4c6d",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "a3ea8abf439c0907da6c685b3d00becca30baa6953753bb755442a3ddb306dae",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-08-02T00:00:11.875Z",
+  "endedAt": "2026-08-02T00:24:25.980Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 52,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 20,
+    "unlabeledRuns": 12,
+    "failedLabelEmptyRuns": 12,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 52,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 200,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 52,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 20,
+    "unlabeledRuns": 12,
+    "failedLabelEmptyRuns": 12,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 50,
+    "failedRuns": 2,
+    "expectedIncorrectSteps": 200,
+    "predictedIncorrectSteps": 339,
+    "matchedIncorrectSteps": 44,
+    "officialAllRowF1": 0.1185983509784695,
+    "officialAllRowRuns": 64,
+    "precision": 0.12979351032448377,
+    "recall": 0.22,
+    "f1": 0.16326530612244897,
+    "trustedNegativeFalsePositiveRate": 0.75,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0.9166666666666666,
+    "unlabeledFailureRate": 0
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/rlm-smoke9/result.json",
+  "runIdentitySha256": "b4b0f57ec934050e69e3fd3e67587f6073f8e0c7775c7b236850586f9cbb7a62",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "80f3ca7fb351f37f456b60ec4448942184adfbc753b1bc7de56139902827a35c",
+  "implementationSha256": "7f14b20834dab1402a222980ad66515f81cb6b508f89ea2081c867dc43c3ec46",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "deterministic-hash",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "1ca280998a9f3a416b40cadb9c14789ada490a727a4ec78491c92bb2022e6c95",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 1,
+  "caseCount": 4,
+  "startedAt": "2026-07-31T08:16:18.900Z",
+  "endedAt": "2026-07-31T08:17:52.198Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 2,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 1,
+    "unlabeledRuns": 2,
+    "failedLabelEmptyRuns": 2,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 2,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 3,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 4,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 2,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 1,
+    "unlabeledRuns": 2,
+    "failedLabelEmptyRuns": 2,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 2,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 3,
+    "predictedIncorrectSteps": 7,
+    "matchedIncorrectSteps": 2,
+    "officialAllRowF1": 0.125,
+    "officialAllRowRuns": 4,
+    "precision": 0.2857142857142857,
+    "recall": 0.6666666666666666,
+    "f1": 0.4,
+    "trustedNegativeFalsePositiveRate": 1,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 1,
+    "unlabeledFailureRate": 0
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/rlm-chk/result.json",
+  "runIdentitySha256": "a9d72390cb3f5aef44e4949a0888e8fff379cac8bfb66b42ca6c66044289a4d6",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "80f3ca7fb351f37f456b60ec4448942184adfbc753b1bc7de56139902827a35c",
+  "implementationSha256": "23f1e9fb82238df5c9931bf797dc47ac3c5e2b4c0c0ebfb861eff72cf452e865",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "deterministic-hash",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "1ca280998a9f3a416b40cadb9c14789ada490a727a4ec78491c92bb2022e6c95",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 1,
+  "caseCount": 2,
+  "startedAt": "2026-07-31T08:34:21.375Z",
+  "endedAt": "2026-07-31T08:35:20.758Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 1,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 1,
+    "failedLabelEmptyRuns": 1,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 1,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 3,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 2,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 1,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 1,
+    "failedLabelEmptyRuns": 1,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 1,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 3,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 2,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/rlm-full5/result.json",
+  "runIdentitySha256": "1856e5b1c90142dc77141734d05c65ee12f47f3220f083f8a9262f0b58402f09",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "bd0347aa7e7e2879832631741035ccbbe7cdef0af6bab2c6e0cf901937aff128",
+  "implementationSha256": "f76f75d1b659a0734d00104e856022fc3cbe0a734bb22e2dc4777d78f7bd9324",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "1ca280998a9f3a416b40cadb9c14789ada490a727a4ec78491c92bb2022e6c95",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-07-31T09:12:55.928Z",
+  "endedAt": "2026-07-31T09:15:35.982Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 48,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 2,
+    "failedRuns": 46,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 8,
+    "matchedIncorrectSteps": 4,
+    "officialAllRowF1": 0.01785714285714286,
+    "officialAllRowRuns": 64,
+    "precision": 0.5,
+    "recall": 0.03636363636363636,
+    "f1": 0.06779661016949153,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": 1,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": 1
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/rlm-full6/result.json",
+  "runIdentitySha256": "1be7492ac9f25d55e253d445869ff5fc274501d75d682a1af5d579504b7aa522",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "bd0347aa7e7e2879832631741035ccbbe7cdef0af6bab2c6e0cf901937aff128",
+  "implementationSha256": "948cfcfba58ff576787cbf6acbe7a04f3d323b3ab4eca0202793a80097df4c73",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "1ca280998a9f3a416b40cadb9c14789ada490a727a4ec78491c92bb2022e6c95",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-07-31T09:23:58.141Z",
+  "endedAt": "2026-07-31T09:37:02.154Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 48,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 14,
+    "failedRuns": 34,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 25,
+    "matchedIncorrectSteps": 4,
+    "officialAllRowF1": 0.017857142857142856,
+    "officialAllRowRuns": 64,
+    "precision": 0.16,
+    "recall": 0.03636363636363636,
+    "f1": 0.05925925925925926,
+    "trustedNegativeFalsePositiveRate": 0.42857142857142855,
+    "trustedNegativeFailureRate": 0.5625,
+    "unlabeledPredictionRate": 0.5,
+    "unlabeledFailureRate": 0.5
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/rlm-full7/result.json",
+  "runIdentitySha256": "e6304c8a719cb7793d259c265fbebd6f485e0c72fe43a8afeae1bb7c1b8adf9e",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "bd0347aa7e7e2879832631741035ccbbe7cdef0af6bab2c6e0cf901937aff128",
+  "implementationSha256": "87e10f211b028ccb897969f7a8759179b7d75d7f06f3ea511dc6be71a7c69b5c",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "1ca280998a9f3a416b40cadb9c14789ada490a727a4ec78491c92bb2022e6c95",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-07-31T09:42:24.063Z",
+  "endedAt": "2026-07-31T10:09:44.276Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 48,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 45,
+    "failedRuns": 3,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 152,
+    "matchedIncorrectSteps": 43,
+    "officialAllRowF1": 0.10208276098901098,
+    "officialAllRowRuns": 64,
+    "precision": 0.28289473684210525,
+    "recall": 0.39090909090909093,
+    "f1": 0.3282442748091603,
+    "trustedNegativeFalsePositiveRate": 0.8,
+    "trustedNegativeFailureRate": 0.0625,
+    "unlabeledPredictionRate": 0.7333333333333333,
+    "unlabeledFailureRate": 0.0625
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/ctb-smoke/result.json",
+  "runIdentitySha256": "5f60033ce05a8531c8c152cc373bd6b2d3aca7353b216b7925c08f8880b41b49",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "1e31880c9feed84df7a821b181d69b23ae55204fca14f52b9628b0dff29c02f7",
+  "implementationSha256": "36b430660a5cf2f441e344017d27b44a66c72b53789f552c67dd5514d4a56d38",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "deterministic-hash",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "779ff0146f56c14a059aac9276f4b673ee70515afe95959a62a9f2d376c688dc",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 1,
+  "caseCount": 2,
+  "startedAt": "2026-07-31T06:22:42.607Z",
+  "endedAt": "2026-07-31T06:22:42.655Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 1,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 1,
+    "failedLabelEmptyRuns": 1,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 1,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 3,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 2,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 1,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 1,
+    "failedLabelEmptyRuns": 1,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 0,
+    "failedRuns": 1,
+    "expectedIncorrectSteps": 3,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 2,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": 1
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/ctb-phasea3/result.json",
+  "runIdentitySha256": "ddc3e6654179c0803bf5e234e3a3ff9b6a97f7d3d53cf2ce5368e078c573ea1f",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "2aa975055cbb8c8bc7295dcd4fbcb453b60847791776ac25a0b772f39ac68d0f",
+  "implementationSha256": "b959a4c19de0ea12bbdbb2bcc883e5c2a9920c96087b0dd9b931446c656f0463",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "779ff0146f56c14a059aac9276f4b673ee70515afe95959a62a9f2d376c688dc",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 32,
+  "startedAt": "2026-07-31T06:51:12.760Z",
+  "endedAt": "2026-07-31T06:54:43.504Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 48,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 64,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "direct",
+    "selectedRuns": 48,
+    "positiveRuns": 32,
+    "trustedNegativeRuns": 16,
+    "unlabeledRuns": 16,
+    "failedLabelEmptyRuns": 16,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 41,
+    "failedRuns": 7,
+    "expectedIncorrectSteps": 110,
+    "predictedIncorrectSteps": 196,
+    "matchedIncorrectSteps": 46,
+    "officialAllRowF1": 0.12956512871373552,
+    "officialAllRowRuns": 64,
+    "precision": 0.23469387755102042,
+    "recall": 0.41818181818181815,
+    "f1": 0.30065359477124187,
+    "trustedNegativeFalsePositiveRate": 0.9285714285714286,
+    "trustedNegativeFailureRate": 0.125,
+    "unlabeledPredictionRate": 1,
+    "unlabeledFailureRate": 0.125
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/family-framing-smoke/stock-openhands/result.json",
+  "runIdentitySha256": "a5446d3f3fb4be02efe4921e843f7b84b1a0511f75bd9650b7aa5516168fd2a4",
+  "datasetSplit": "devsmoke-openhands-stock-20260802",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "1b6878992df858b2bb4685de0b4951d4e080308db8621de4a87217d75f6fec8e",
+  "sourceRowCount": 6,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "747ac229be6729bd4bbc24e8be4f2a510663bd062dd748c430d05c9d580944df",
+  "implementationSha256": "a3f5a8203f0a7c74736170c53100c50c4db1cf84381533002f0539ef1e7a4c6d",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "55744d067f70ffcb0223eae6121739021a9e2567208cfd14098582bb73cdbf97",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 6,
+  "startedAt": "2026-08-03T05:08:56.226Z",
+  "endedAt": "2026-08-03T05:39:26.071Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 12,
+    "positiveRuns": 12,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 12,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 52,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 12,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 12,
+    "positiveRuns": 12,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 12,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 52,
+    "predictedIncorrectSteps": 34,
+    "matchedIncorrectSteps": 13,
+    "officialAllRowF1": 0.461111111111111,
+    "officialAllRowRuns": 12,
+    "precision": 0.38235294117647056,
+    "recall": 0.25,
+    "f1": 0.3023255813953488,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/family-framing-smoke/framing-openhands/result.json",
+  "runIdentitySha256": "c83c54247dae1403a4d28cee50156f42352963404c04629c87449443cda8bfcd",
+  "datasetSplit": "devsmoke-openhands-framing-20260802",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "1b6878992df858b2bb4685de0b4951d4e080308db8621de4a87217d75f6fec8e",
+  "sourceRowCount": 6,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "0bd099940d309e4e956a37209f3a0e0adf323de07186e124e2667edee55ed16a",
+  "implementationSha256": "a3f5a8203f0a7c74736170c53100c50c4db1cf84381533002f0539ef1e7a4c6d",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "55744d067f70ffcb0223eae6121739021a9e2567208cfd14098582bb73cdbf97",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 6,
+  "startedAt": "2026-08-03T05:39:31.137Z",
+  "endedAt": "2026-08-03T06:05:50.499Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 12,
+    "positiveRuns": 12,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 12,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 52,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 12,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 12,
+    "positiveRuns": 12,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 11,
+    "failedRuns": 1,
+    "expectedIncorrectSteps": 52,
+    "predictedIncorrectSteps": 41,
+    "matchedIncorrectSteps": 10,
+    "officialAllRowF1": 0.21490384615384617,
+    "officialAllRowRuns": 12,
+    "precision": 0.24390243902439024,
+    "recall": 0.19230769230769232,
+    "f1": 0.21505376344086025,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/split3-restored/smoke-run/result.json",
+  "runIdentitySha256": "e8ee878db70d4ce7cf39603be546af3bed7ab968d385ba728f479bf955c4a851",
+  "datasetSplit": "verified-miniswe-normalizer-remainder-thin-blind-28",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "bf573beca3bd58c2f2937671dfaa54095f6438523d07caa22608e1e87b114c1d",
+  "sourceRowCount": 28,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "6a9d87845f36c6f7dfda1d7e5d7ab2fa2fb010e3316b02eb83c4da07506523f7",
+  "implementationSha256": "a3f5a8203f0a7c74736170c53100c50c4db1cf84381533002f0539ef1e7a4c6d",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "deterministic-hash",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "a3ea8abf439c0907da6c685b3d00becca30baa6953753bb755442a3ddb306dae",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 4,
+  "startedAt": "2026-08-01T20:06:53.361Z",
+  "endedAt": "2026-08-01T20:14:38.278Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 8,
+    "positiveRuns": 8,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 8,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 10,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 8,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 8,
+    "positiveRuns": 8,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 8,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 10,
+    "predictedIncorrectSteps": 20,
+    "matchedIncorrectSteps": 3,
+    "officialAllRowF1": 0.19583333333333333,
+    "officialAllRowRuns": 8,
+    "precision": 0.15,
+    "recall": 0.3,
+    "f1": 0.2,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/gepa-run/cli-proof/result.json",
+  "runIdentitySha256": "8d4f663151fd47236fc1b7c1906641592eb5c3ce0082d6b81d8910a97ab3017b",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "cb2a5bebd422bebc4c5788c63a5646daa04a5cdda45be86f55c86c3403ea53da",
+  "implementationSha256": "027db213af78776e25e68914321cd599fe7a175a9f791b730067c2f943c2c0db",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "deterministic-hash",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "a3ea8abf439c0907da6c685b3d00becca30baa6953753bb755442a3ddb306dae",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 1,
+  "caseCount": 1,
+  "startedAt": "2026-08-01T07:23:00.195Z",
+  "endedAt": "2026-08-01T07:26:29.873Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 1,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 1,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 3,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 1,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 1,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 1,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 3,
+    "predictedIncorrectSteps": 1,
+    "matchedIncorrectSteps": 1,
+    "officialAllRowF1": 0.5,
+    "officialAllRowRuns": 1,
+    "precision": 1,
+    "recall": 0.3333333333333333,
+    "f1": 0.5,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/devshm-20260803/mp-tw-full-dev/result.json",
+  "runIdentitySha256": "526dc2ffa9fcee85249020e7bc2b23802a371d0ec1e60c9d4a570e8afdd2cbd8",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "dc8e043acacb26a5a4400881d83caee35add1983b0fe2705c9aaa119aafa1add",
+  "implementationSha256": "1de4cccc40c09acdd96c98fdc3fe62eec42a05c1d43a2337f83d7d2797f40054",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "701ea59184f3d227cf590ea3f16d24d65dbed2ad5644f8294801ceb66a30d4ef",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 3,
+  "caseCount": 32,
+  "startedAt": "2026-08-01T09:04:37.841Z",
+  "endedAt": "2026-08-01T09:41:31.314Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 72,
+    "positiveRuns": 48,
+    "trustedNegativeRuns": 24,
+    "unlabeledRuns": 24,
+    "failedLabelEmptyRuns": 24,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 72,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 165,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 96,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 72,
+    "positiveRuns": 48,
+    "trustedNegativeRuns": 24,
+    "unlabeledRuns": 24,
+    "failedLabelEmptyRuns": 24,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 64,
+    "failedRuns": 8,
+    "expectedIncorrectSteps": 165,
+    "predictedIncorrectSteps": 305,
+    "matchedIncorrectSteps": 86,
+    "officialAllRowF1": 0.14864104259937594,
+    "officialAllRowRuns": 96,
+    "precision": 0.2819672131147541,
+    "recall": 0.5212121212121212,
+    "f1": 0.36595744680851056,
+    "trustedNegativeFalsePositiveRate": 0.75,
+    "trustedNegativeFailureRate": 0.16666666666666666,
+    "unlabeledPredictionRate": 1,
+    "unlabeledFailureRate": 0.16666666666666666
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/devshm-20260803/mp-tw-full-h1/result.json",
+  "runIdentitySha256": "3664d62f755f089769f4e4a9d6752b423bbf1a193add00823a0a51bec34932a6",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "53af5ffe3962f3378f2d65419b92b8a56fe7d6c8efc619a0bc2b8f0872bc4f83",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "bd0347aa7e7e2879832631741035ccbbe7cdef0af6bab2c6e0cf901937aff128",
+  "implementationSha256": "56dd4c7ed19fc5855f99ad238464ad95cb1bea155ad58c49dab3eb0f6cbe7d6a",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "701ea59184f3d227cf590ea3f16d24d65dbed2ad5644f8294801ceb66a30d4ef",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 3,
+  "caseCount": 32,
+  "startedAt": "2026-08-01T09:04:37.891Z",
+  "endedAt": "2026-08-01T09:08:53.724Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 66,
+    "positiveRuns": 48,
+    "trustedNegativeRuns": 18,
+    "unlabeledRuns": 30,
+    "failedLabelEmptyRuns": 30,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 66,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 228,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 96,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 66,
+    "positiveRuns": 48,
+    "trustedNegativeRuns": 18,
+    "unlabeledRuns": 30,
+    "failedLabelEmptyRuns": 30,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 0,
+    "failedRuns": 66,
+    "expectedIncorrectSteps": 228,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 96,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": 1,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": 1
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/devshm-20260803/mp-tw-serial-dev/result.json",
+  "runIdentitySha256": "526dc2ffa9fcee85249020e7bc2b23802a371d0ec1e60c9d4a570e8afdd2cbd8",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "dc8e043acacb26a5a4400881d83caee35add1983b0fe2705c9aaa119aafa1add",
+  "implementationSha256": "1de4cccc40c09acdd96c98fdc3fe62eec42a05c1d43a2337f83d7d2797f40054",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "701ea59184f3d227cf590ea3f16d24d65dbed2ad5644f8294801ceb66a30d4ef",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 3,
+  "caseCount": 32,
+  "startedAt": "2026-08-01T10:42:41.423Z",
+  "endedAt": "2026-08-01T11:18:58.142Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 72,
+    "positiveRuns": 48,
+    "trustedNegativeRuns": 24,
+    "unlabeledRuns": 24,
+    "failedLabelEmptyRuns": 24,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 72,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 165,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 96,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 72,
+    "positiveRuns": 48,
+    "trustedNegativeRuns": 24,
+    "unlabeledRuns": 24,
+    "failedLabelEmptyRuns": 24,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 70,
+    "failedRuns": 2,
+    "expectedIncorrectSteps": 165,
+    "predictedIncorrectSteps": 293,
+    "matchedIncorrectSteps": 74,
+    "officialAllRowF1": 0.15258174683540124,
+    "officialAllRowRuns": 96,
+    "precision": 0.2525597269624573,
+    "recall": 0.4484848484848485,
+    "f1": 0.3231441048034934,
+    "trustedNegativeFalsePositiveRate": 0.875,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 1,
+    "unlabeledFailureRate": 0
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/devshm-20260803/mp-tw-serial-h1/result.json",
+  "runIdentitySha256": "5cdb66a489d02f9bb3ae125b3a54a18c02d5b2977e5886236ed1beba50fd6386",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "53af5ffe3962f3378f2d65419b92b8a56fe7d6c8efc619a0bc2b8f0872bc4f83",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "dc8e043acacb26a5a4400881d83caee35add1983b0fe2705c9aaa119aafa1add",
+  "implementationSha256": "1de4cccc40c09acdd96c98fdc3fe62eec42a05c1d43a2337f83d7d2797f40054",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "census",
+   "caseSelectionSeed": 0,
+   "selectionStratified": false,
+   "dependencyLockSha256": "701ea59184f3d227cf590ea3f16d24d65dbed2ad5644f8294801ceb66a30d4ef",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 3,
+  "caseCount": 32,
+  "startedAt": "2026-08-01T10:05:35.881Z",
+  "endedAt": "2026-08-01T10:42:40.648Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 66,
+    "positiveRuns": 48,
+    "trustedNegativeRuns": 18,
+    "unlabeledRuns": 30,
+    "failedLabelEmptyRuns": 30,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 66,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 228,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 96,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": 0,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0,
+    "unlabeledFailureRate": 0
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 66,
+    "positiveRuns": 48,
+    "trustedNegativeRuns": 18,
+    "unlabeledRuns": 30,
+    "failedLabelEmptyRuns": 30,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 64,
+    "failedRuns": 2,
+    "expectedIncorrectSteps": 228,
+    "predictedIncorrectSteps": 252,
+    "matchedIncorrectSteps": 85,
+    "officialAllRowF1": 0.17367424242424243,
+    "officialAllRowRuns": 96,
+    "precision": 0.3373015873015873,
+    "recall": 0.37280701754385964,
+    "f1": 0.3541666666666667,
+    "trustedNegativeFalsePositiveRate": 0.6666666666666666,
+    "trustedNegativeFailureRate": 0,
+    "unlabeledPredictionRate": 0.9666666666666667,
+    "unlabeledFailureRate": 0
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/devshm-20260803/mp-tw-h1-probe/result.json",
+  "runIdentitySha256": "b6a4eebc13c1c5567f6292c8678f18e6ae4bda5b554e32e10a9731059ae4850d",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "53af5ffe3962f3378f2d65419b92b8a56fe7d6c8efc619a0bc2b8f0872bc4f83",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "dc8e043acacb26a5a4400881d83caee35add1983b0fe2705c9aaa119aafa1add",
+  "implementationSha256": "1de4cccc40c09acdd96c98fdc3fe62eec42a05c1d43a2337f83d7d2797f40054",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "deterministic-hash",
+   "caseSelectionSeed": 48,
+   "selectionStratified": false,
+   "dependencyLockSha256": "701ea59184f3d227cf590ea3f16d24d65dbed2ad5644f8294801ceb66a30d4ef",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 1,
+  "caseCount": 1,
+  "startedAt": "2026-08-01T10:02:45.839Z",
+  "endedAt": "2026-08-01T10:04:23.108Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 1,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 1,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 10,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 1,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 1,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 1,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 10,
+    "predictedIncorrectSteps": 2,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 1,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/devshm-20260803/mp-tw-smoke-narrow/result.json",
+  "runIdentitySha256": "ec347ceba91b9b63fadb2efe03c82957f848e1e16837676f1dd1604f57702a4d",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "5d8b4024c3e2114965cbf2f2fa0124bbf59b3fb134824fa06dd6a38ee07e8412",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": null,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "bd0347aa7e7e2879832631741035ccbbe7cdef0af6bab2c6e0cf901937aff128",
+  "implementationSha256": "56dd4c7ed19fc5855f99ad238464ad95cb1bea155ad58c49dab3eb0f6cbe7d6a",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "deterministic-hash",
+   "caseSelectionSeed": 12,
+   "selectionStratified": false,
+   "dependencyLockSha256": "701ea59184f3d227cf590ea3f16d24d65dbed2ad5644f8294801ceb66a30d4ef",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 1,
+  "caseCount": 1,
+  "startedAt": "2026-08-01T07:20:06.740Z",
+  "endedAt": "2026-08-01T07:22:03.851Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 1,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 1,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 2,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 1,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 1,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 1,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 2,
+    "predictedIncorrectSteps": 4,
+    "matchedIncorrectSteps": 2,
+    "officialAllRowF1": 0.6666666666666666,
+    "officialAllRowRuns": 1,
+    "precision": 0.5,
+    "recall": 1,
+    "f1": 0.6666666666666666,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/devshm-20260803/mp-tw-smoke-wide/result.json",
+  "runIdentitySha256": "b6a4eebc13c1c5567f6292c8678f18e6ae4bda5b554e32e10a9731059ae4850d",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "53af5ffe3962f3378f2d65419b92b8a56fe7d6c8efc619a0bc2b8f0872bc4f83",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "dc8e043acacb26a5a4400881d83caee35add1983b0fe2705c9aaa119aafa1add",
+  "implementationSha256": "1de4cccc40c09acdd96c98fdc3fe62eec42a05c1d43a2337f83d7d2797f40054",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "deterministic-hash",
+   "caseSelectionSeed": 48,
+   "selectionStratified": false,
+   "dependencyLockSha256": "701ea59184f3d227cf590ea3f16d24d65dbed2ad5644f8294801ceb66a30d4ef",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 1,
+  "caseCount": 1,
+  "startedAt": "2026-08-01T07:20:06.732Z",
+  "endedAt": "2026-08-01T07:21:59.803Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 1,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 1,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 10,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 1,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 1,
+    "positiveRuns": 1,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 1,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 10,
+    "predictedIncorrectSteps": 2,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 1,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   }
+  ]
+ },
+ {
+  "path": "/home/drew/bench-cache/ctb-20260801/salvaged-runs/devshm-20260803/mp-tw-smoke-wide-rep/result.json",
+  "runIdentitySha256": "550c5da5114e56644b2a8f40f119450a967cdd8e49f42292880ba0ad600c534b",
+  "datasetSplit": "verified-miniswe-normalizer-compatible-32",
+  "datasetRevision": "aa213b84ffb6690fc37ca15766d6ca174ec36d4d",
+  "labelsSha256": "53af5ffe3962f3378f2d65419b92b8a56fe7d6c8efc619a0bc2b8f0872bc4f83",
+  "sourceRowCount": 32,
+  "model": "glm-5.2",
+  "rlmSamples": 1,
+  "outputAdapter": "codetracebench-incorrect-block",
+  "protocolSha256": "dc8e043acacb26a5a4400881d83caee35add1983b0fe2705c9aaa119aafa1add",
+  "implementationSha256": "1de4cccc40c09acdd96c98fdc3fe62eec42a05c1d43a2337f83d7d2797f40054",
+  "promptVariant": null,
+  "metadataExtra": {
+   "caseSelection": "deterministic-hash",
+   "caseSelectionSeed": 48,
+   "selectionStratified": false,
+   "dependencyLockSha256": "701ea59184f3d227cf590ea3f16d24d65dbed2ad5644f8294801ceb66a30d4ef",
+   "populationRepresentativenessProven": false
+  },
+  "repetitions": 2,
+  "caseCount": 1,
+  "startedAt": "2026-08-01T07:23:29.229Z",
+  "endedAt": "2026-08-01T07:26:53.097Z",
+  "runners": [
+   {
+    "runnerId": "empty",
+    "selectedRuns": 2,
+    "positiveRuns": 2,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 2,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 20,
+    "predictedIncorrectSteps": 0,
+    "matchedIncorrectSteps": 0,
+    "officialAllRowF1": 0,
+    "officialAllRowRuns": 2,
+    "precision": 0,
+    "recall": 0,
+    "f1": 0,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   },
+   {
+    "runnerId": "dspy-rlm",
+    "selectedRuns": 2,
+    "positiveRuns": 2,
+    "trustedNegativeRuns": 0,
+    "unlabeledRuns": 0,
+    "failedLabelEmptyRuns": 0,
+    "unknownLabelEmptyRuns": 0,
+    "completedRuns": 2,
+    "failedRuns": 0,
+    "expectedIncorrectSteps": 20,
+    "predictedIncorrectSteps": 22,
+    "matchedIncorrectSteps": 16,
+    "officialAllRowF1": 0.7636363636363637,
+    "officialAllRowRuns": 2,
+    "precision": 0.7272727272727273,
+    "recall": 0.8,
+    "f1": 0.761904761904762,
+    "trustedNegativeFalsePositiveRate": null,
+    "trustedNegativeFailureRate": null,
+    "unlabeledPredictionRate": null,
+    "unlabeledFailureRate": null
+   }
+  ]
+ }
+]