feat(benchmarks): official-metric leaderboard — extraction-verified head-to-head + the instrument-quality ledger

[drewstone]

What this is

The field's metric, our discipline: benchmarks/trace-analysis/leaderboard/ + LEADERBOARD.md — every configuration we have ever measured, ranked on the official CodeTraceBench all-row F1, extracted (never recomputed) from the 34 result artifacts we hold, each value bit-matched against the embedded scorer output.

Headlines the table surfaces for the first time

• Same-rows head-to-head: on identical dev-32 rows, our W-serial run scores 0.1526 vs pinned CodeTracer's 0.1161 (+31% relative); every measured config beats CodeTracer on same rows except the retired rlm-stock-r0.
• Shipping config on clean sealed gold: 0.5427 official all-row F1 on holdout-2 (constant positional rule: 0.0879).
• The instrument-quality ledger: constant-rule calibration for all 10 label sets — split3's arm spread (0.54pp) is smaller than its gap to a trivial rule; the SWE-agent family carries a 0.1895 flag-last-step artifact that exceeds every measured cross-family score (sealing guidance recorded). Split identity keyed on label digests, not split names.
• Paper numbers (48%/50.9%) recorded as unverifiable-on-disk, not-same-rows context — stated, not compared.

Rigor notes

Bit-match discipline caught a real trap: Python ≥3.12's compensated sum() differs from TS's fold by 1 ulp — fixed with an explicit fold; three artifact generations verified to exact double equality (operator re-ran the bitmatch independently: TRUE). A brief-vs-recomputation discrepancy (39 vs 38 positional rows on SWE-agent) resolved by execution and documented. 8 volatile /dev/shm runs salvaged to the bench cache before citation.

Gap analysis (appendix) ranks the path to a submission-ready public claim: upstream degeneracy issue → label manifests → shipping config @3 reps on clean SWE-agent-106 ($40) → same-rows CodeTracer rerun ($25-35).

$0 LLM. Battery: lint, typecheck, full suite green; merge-tree clean.

[tangletools]

✅ Auto-approved drewstone PR — 009daff4

This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.

_{tangletools · auto-approval · reason: drewstone_author · 2026-08-03T06:48:28Z}

[tangletools]

🟡 Value Audit — sound-with-nits

Verdict	sound-with-nits
Concerns	3 (3 weak-concern)
Heuristic	0.0s
Duplication	0.0s
Interrogation	171.8s (2 bridge agents)
Total	171.8s

💰 Value — sound-with-nits

Adds a leaderboard + instrument-quality ledger (constant-rule calibration per split) backed by an extraction script that cites embedded scorer values verbatim — coherent, in-grain, with two minor reproducibility nits.

• What it does: Adds benchmarks/trace-analysis/leaderboard/ with two Python tools and a 2459-line extracted snapshot, plus LEADERBOARD.md (277 lines). (1) extract-official.py reads 34 run artifacts and dumps each embedded officialAllRowF1 plus provenance (digests, reps, splits) to JSON; its --bitmatch mode recomputes one artifact's value from raw observations in Python and asserts double equality. (2) calibra
• Goals it achieves: Make the field's official CodeTraceBench metric the single ranking axis for this repo's trace-analysis configs, and — more importantly — attach an 'instrument-quality ledger' that says whether each split's number is even interpretable (does it separate a real analyst from a constant rule?). The ledger catches three real findings: split3 is degenerate (three configs within 0.54pp, inside the gap to
• Assessment: Sound and in the grain. Python already lives in benchmarks/trace-analysis (prepare.py, prepare-bulk.py, parser.py); committing run artifacts and snapshots is established (codetracebench-glm52-20260730/result.json etc. are in-repo); the canonical scorer is correctly cited at src/analyst/benchmark-public-calibration.ts:129 (officialCodeTraceF1) and the Python reimplements that formula faithfully, in
• Better / existing approach: Searched benchmarks/trace-analysis/tools/ and src/analyst/. Two partial overlaps, neither rises to 'use this instead': (a) compare-analyst-runs.mjs:426 (--verify-embedded) already recomputes the WHOLE codeTraceCalibration block including officialAllRowF1 and asserts per-field equality — but it shares the TS scorer's code path, so the Python --bitmatch is a deliberate cross-language independent che
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 2
• Bridge warning: opencode/kimi-for-coding/k2p7: opencode: opencode error

🎯 Usefulness — sound-with-nits

A disciplined, reproducible benchmark leaderboard with an instrument-quality ledger, backed by extraction/bit-match scripts and a committed JSON snapshot — fits the operator-tool pattern exactly and does real work.

• Integration: Operator-reached, not CI-wired — identical to the existing benchmarks/trace-analysis/tools/*.mjs scripts, which no automation invokes either (grep'd .github/, scripts/, package.json: zero callers for compare-analyst-runs/decompose-analyst). The human-facing surface is LEADERBOARD.md; the two .py scripts reproduce it; official-extract-20260803.json is the committed output so any reader can audit th
• Fit with existing patterns: Fits the established grain. benchmarks/trace-analysis/ already holds per-run dirs + a tools/ dir, and Python scripts already live in benchmark subdirs (prepare.py, prepare-bulk.py, parser.py) — so .py here is precedent, not novelty. The .mjs tools import dist/ because they need the package's stats; these .py scripts do independent recomputation, which is precisely why Python is the correct choice
• Real-world viability: Mostly robust. extract-official.py is portable (Path.home() + script-relative TRACE_DIR) and degrades cleanly on missing artifacts ({'missing': True}). calibrate-splits.py hardcodes absolute /home/drew/... paths for all 10 label files — including one at /home/drew/code/agent-eval/... that is outside this repo's root and not even the checkout path — so it runs only on the operator's machine. The do
• Model: opencode/zai-coding-plan/glm-5.2
• Bridge attempts: 1

🎯 Usefulness Audit

🟡 calibrate-splits.py hardcodes /home/drew/... paths while its sibling already shows the portable pattern [ergonomics] ``

calibrate-splits.py:13-22 lists 10 absolute paths under /home/drew/code/agent-eval/ and /home/drew/bench-cache/...; the first points outside this repo entirely. extract-official.py:15-16 in the same directory already resolves paths portably via TRACE_DIR = Path(file).resolve().parents[1] and Path.home(). The in-repo labels file it needs (benchmarks/trace-analysis/codetracebench-glm52-20260730/input-labels.json, present on disk) could be referenced the same way. Not a blocker — documented as

💰 Value Audit

🟡 --bitmatch overlaps compare-analyst-runs.mjs --verify-embedded without naming it [duplication] ``

compare-analyst-runs.mjs:426-495 already recomputes a persisted artifact's codeTraceCalibration (incl. officialAllRowF1) from observations and exits 1 on mismatch. extract-official.py:96-117 --bitmatch does the same assertion for one runner. The cross-language independence (Python vs the TS scorer path) is a real methodological gain, but LEADERBOARD.md:249-259 presents --bitmatch as THE verification rather than as an independent complement to the existing one. One line in the doc naming --verify

🟡 Hardcoded /home/drew and ~/bench-cache paths regress the existing parameterized CLI convention [against-grain] ``

calibrate-splits.py:10-21 hardcodes 10 /home/drew/... and ~/bench-cache/... label paths as module constants; extract-official.py:11-46 hardcodes 34 artifact paths the same way. The established pattern in the same tree — codetracebench-glm52-20260730/prepare.py:26-30 — uses argparse --labels/--out Path args, fully parameterized. The new scripts are reproducible only on one operator machine; LEADERBOARD.md:243 acknowledges 'paths reference the operator bench cache' but that documents the regressio

---

What this audit checks

It judges the change on its merits — not whether it was tasked out in an issue. Unticketed, fast-moving work is fine; the question is whether the change is good and whether a better or existing approach should be used instead.

Pass	What it asks
Heuristic	Vague title? Whitespace-only or cruft-bearing diff? (content signals only)
Duplication	Do added function/class names already exist elsewhere in the repo?
Value Audit	What does it do? What goal does it achieve? Is it good? Better architecture or already-exists?
Usefulness Audit	Does it integrate and fit? Will it hold up in real use and actually get used?

Findings are concerns, not blocks — the human reviewer decides what to do with them.

_{value-audit · 20260803T065440Z}

[tangletools]

Premise check withheld merge — 009daff4

Classifier flagged this PR as a premise claim (numeric pp/% delta + eval terminology). Confidence: medium.

Recommend re-running the underlying eval with pairedEvalueSequence before merging.

• Cited claim: 0.54pp
• PR body excerpt: feat(benchmarks): official-metric leaderboard — extraction-verified head-to-head + the instrument-quality ledger

Run:

pnpm eval:evolve --reps 5 --skip-mutation

Classifier rationale: Body cites 1 numeric claim(s) (0.54pp) and eval-related terms appear in pr_body, review_findings. PR is asserting a measurable result that repair-pr cannot polish away — re-run the underlying evaluation before merging.

---

_{tangletools premise check · #533}