feat(benchmarks): family-gap decomposition — the gap is localization; family-framing arm killed by pre-registered gate - #534
Merged
Merged
Conversation
Extend decompose-analyst-loss.mjs with the official positive-runs micro view, gold classes by tool_type, blind reasons (blank vs duplicate), far-miss direction, gold/predicted block shape distributions, a snapFar counterfactual, and a labels-level constant-rule calibration sweep. Decomposition of the cert2 stock runs (committed artifact): the 2x family gap is localization + under-enumeration on 2x-longer trajectories — far holds 47-49% of gold mass vs 14% on mini-SWE, snapFar recovers +32.3pp (OH) / +25.4pp (T2), predicted blocks halve (1.6 vs 3.0 gold blocks/case).
One arm from the family-gap decomposition: stock RLM instructions (d3829fb8, byte-identical) plus a coverage-and-enumeration section and two family grammar notes (ecb8a207). Smoke on 6 seeded long-trace cases per family from the tuning-legal dev pools, paired against a fresh stock arm on the same cases; gates fixed before any model run.
…s pre-registered no-harm gate
tangletools
approved these changes
Aug 3, 2026
tangletools
left a comment
tangletools
left a comment
Contributor
There was a problem hiding this comment.
✅ Auto-approved drewstone PR — e963d9e7
This PR was opened by the trusted drewstone account.
The full PR reviewer audit still runs separately and will publish findings if it detects issues.
tangletools · auto-approval · reason: drewstone_author · 2026-08-03T07:28:18Z
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
The /evolve round on the cross-family gap (mini-SWE 0.63 vs OpenHands 0.29 / Terminus2 0.22 micro): a free decomposition that reframes the problem, plus one pre-registered arm that died honestly.
Finding 1 — the family gap is a LOCALIZATION problem ($0, recomputed from cert2 observations)
The "far" class (predictions >2 steps from any gold) holds 48.9% (OH) / 46.5% (T2) of gold mass vs 14.4% on mini-SWE; snapping far predictions to gold is worth +32.3pp / +25.4pp counterfactually — essentially the whole gap. The analyst investigates the wrong regions of unfamiliar session formats; it does not fail to detect. Full per-case tables:
.evolve/family-gap-decomposition-2026-08-02.md;decompose-analyst-loss.mjsextended with family-aware taxonomy.Finding 2 — the arm died by its own pre-registered gate
Family-framing coverage instructions (stock + enumeration/coverage section + family grammar notes), gates committed before any paid run: paired seeded OpenHands smoke (clean instrument, 0/12 + 1/12 failures) scored 0.2151 vs stock 0.3023 — no-harm gate violated (confound-checked), pad-ratio kill tripped (2.1×), paired recall CI excludes zero. Fourth consecutive hand-authored prompt arm to lose this campaign. The Terminus2 pair is recorded as an invalid instrument day (>10% failed runs from seat degradation; carries no evidential weight; directionally consistent).
Provenance
Override runs carry
instructionsOverrideSha256(re-hashed, matches the committed instructions file). Battery: lint/typecheck/4685 tests green; merge-tree clean. Spend $6.0 total.Next arm, evidence-ranked and pre-seeded for the following round: family-GEPA on the tuning-legal OH/T2 dev pools — the only intervention class that has survived certification here.