feat(benchmarks): family-gap decomposition — the gap is localization; family-framing arm killed by pre-registered gate

benchmarks/trace-analysis/codetracebench-oht2-20260801/family-framing/oht2-coverage-instructions.txt

@@ -0,0 +1,66 @@
+Analyze exactly one coding-agent trajectory and its attached final verification.
+Your task is the CodeTraceBench incorrect-step task: identify every incorrect step, defined as a wrong state-changing intervention given the evidence — a mislocalized edit, a wrong hypothesis that drives an action, a regression, an irrelevant change, or an incorrect dependency or configuration choice.
+If the final verification failed, the trajectory MUST contain at least one incorrect step. Never return an empty findings array on a failing trajectory; trace backward until you find the root cause.
+Work backward, the way this benchmark was annotated, never by scanning forward for suspicious steps: start from the final verification outcome or the latest observed failure evidence, identify the immediately preceding step whose action or output produced that observed error, then recursively ask which earlier decision led to each intermediate failure, until the preceding steps contain no error or the cause is unrelated to the trajectory's own decisions.
+Each backward chain terminates at an error-critical step — the earliest decision that triggered the downstream cascade — and that step is the block's first_step: the step that committed the mistake, not the step that planned it and not a later step that repeats it.
+A block is a maximal contiguous sequence of strictly incorrect steps. A step belongs in the block ONLY if it introduces, propagates, or compounds the error. 
+Do NOT include steps that merely "act on", diagnose, or react to the error. A diagnostic command, a test run exposing the bug, or a correct exploratory read is a CORRECT step. 
+If an incorrect step is followed by a correct diagnostic step and then another incorrect step, you MUST emit two separate blocks. NEVER bridge correct steps by grouping them into a single block with incorrect steps. Over-blocking drastically hurts your precision.
+After identifying first_step, extend last_step forward ONLY through consecutive steps that independently introduce, propagate, or compound the mistake. A cascade of repeated failed attempts at the same wrong approach is one maximal block, provided EVERY step is independently incorrect.
+Do not end a block merely because the agent tried a variation of the same wrong approach; a variation that still carries the error stays inside the block.
+A partially correct or ambiguous fix still counts as incorrect; the block ends only at the first step free of the error — a clean diagnostic read, the corrective action that closes the issue and needs no further rework, or a genuine abandonment of the wrong approach.
+Block extent follows the traced chain and this forward extension, nothing else.
+Report each failure block as exactly one finding whose first_step is the block's first incorrect step and whose last_step is its last, covering every consecutive step between them.
+Every step inside a block is scored on its own: naming a correct step costs exactly as much as missing an incorrect one, and naming only the first step of a longer block forfeits every unnamed step. Because of this, carefully verify every step between first_step and last_step. Only include steps that introduce, propagate, or compound the error.
+Report blocks separated by at least one correct step as separate findings, and never let two blocks overlap. If there are multiple separate failure cascades, emit a separate finding for each one.
+Prefer anchored blocks: a block whose chain traces back from observed failure evidence — a failing command or verification, an error observation, a regression, or, on a solved trajectory, a later step that reverts or supersedes it — outranks one without.
+When an action is clearly wrong on its own evidence but you cannot trace such an anchor, report the block anyway with proportionally lower confidence.
+A solved trajectory still carries every mistake made along the way: inspect its final patching and verification stages for a state-changing action that a later step reverted, superseded, or corrected — a wrong edit just before the final fix is incorrect even when every test ends green.
+Before emitting a candidate block, check its boundaries.
+Neighbor check: ask whether the accusation fits one step earlier (the decision rather than its consequence) or one step later (the next step still acts on or reworks the same error) better than where you placed it, and move the boundary when it does; a boundary off by one step scores zero at that step.
+Completeness check: a block must cover the maximal contiguous sequence of incorrect steps. If an agent fails at step 10, tries to fix it at 11, fails, and tries again at 12 and 13, all four steps are incorrect and must be included in the block. Never truncate a cascade. If you miss the later steps of a cascade, your recall drops to zero for them.
+Counterfactual check: ask which step's correct execution would have made the downstream failure or rework disappear, and move first_step onto that step; use this check only to move a boundary, never to delete a block.
+Width check: inspect the step immediately before first_step and the step immediately after last_step; when either neighbor commits to, compounds, or propagates the same mistake, it belongs inside the block — move that boundary outward and repeat until both neighbors are free of the error.
+For each block you keep, name as consequence_step the step number whose action or observation shows the damage — a failing command, a wrong file state, a repeated failure, or rework the agent had to do because of this block; that step is the block's own last step when its observation already shows the damage, and a later step otherwise.
+When you cannot name that step number from the trace you were given, drop the block; a plausible story about why a step looks wrong is not evidence that it was.
+A passing final verification is not evidence that a block caused nothing, and a failing final verification is not evidence that any particular block caused it.
+For every block, decide whether the agent escaped the failure.
+Mark escape_status "escaped" only when you can name the single later step that fully reversed the block, the agent needed no other step to recover, and nothing after that step revisits the same file, command, or hypothesis; write that step number in the rationale. If the agent required multiple steps to fix the issue, or if you are unsure, it is not escaped.
+Mark escape_status "unescaped" in every other case, including whenever you are unsure.
+A passing final verification never makes a block escaped.
+Do not label a diagnostic probe or test run merely because its output exposes an earlier defect.
+Do not label a redundant but correct read or search; CodeTraceBench scores unuseful steps separately, and this run scores incorrect steps only. Never include a correct step in a block just to bridge two incorrect steps; instead, emit two separate blocks.
+Do not label a step solely because final verification failed.
+When final verification is unavailable, trace backward from the latest failure evidence inside the trajectory itself.
+Every step in a reported block MUST be the positive integer n from an existing assistant LLM span named step-<n>.
+Never select an EVALUATOR, TOOL, CHAIN, final-verification, benchmark-verification, or message-<n> span.
+Before emitting a finding, inspect every covered span's attributes.content and describe only the actions shown there.
+Report at most 16 blocks and at most 12 steps in one block; when more candidates than that exist, report the ones whose chains carry the clearest downstream evidence.
+When the trajectory has no incorrect steps — its final verification passed and the final-stage sweep found no reverted, superseded, or corrected action — return an empty findings array.
+Trajectory families and coverage.
+This trajectory may come from a long agent session (OpenHands tool-call sessions, Terminus2 terminal sessions) that contains several independent failure incidents spread across the whole session, not one late cascade.
+Coverage rule: your compact table of assistant steps must span the ENTIRE trajectory, from step 1 through the final step, before you rank any candidate.
+After tracing the incident with the clearest downstream evidence, do not submit: resume scanning every remaining segment of the trajectory — early, middle, and late thirds — for further independent incidents, and only submit once each segment has been examined against its own failure evidence.
+Finding one incident is not evidence that other segments are clean.
+Approach transitions are where labeled mistakes concentrate: whenever the agent abandons an approach, adopts a new plan, or reworks after a failed check, trace that segment's own error-critical step and report its block even when the session eventually ended fine.
+Emit one block per independent incident you can support with that incident's own evidence; never drop a supportable incident because you already reported another, and never merge distinct incidents separated by correct steps.
+Action grammar.
+OpenHands steps are rendered tool calls — read({...}), edit({...}), run({...}), think({...}) — and the reasoning text before the call is part of the step's content.
+A run step's command can be buried inside many blank lines; the non-blank text is the action, and a step is never clean merely because its rendering is mostly whitespace.
+A wrong hypothesis committed through an edit or a wrong tool invocation is exactly as incorrect as a wrong shell command.
+Terminus2 steps are raw terminal keystrokes; sessions repeat identical commands and send bare Enter keystrokes.
+When the same command text occurs at several steps, anchor the step number on the surrounding observations and cite the occurrence inside the failing segment, never the first text match.
+Use the trace tools rather than asking for the whole trajectory in the prompt.
+Keep retrieved trace objects in Python variables.
+Never print an entire trace, full source file, or more than 12000 characters in one iteration.
+Read the final verification and the latest failure evidence first, then build a compact table of assistant step ids, actions, and following observations.
+Trace backward from that evidence with viewSpans or searchSpan, confirming each candidate step's own action content, instead of repeatedly printing the table.
+This runner emits no JSON fields, so the block is encoded in the finding's subject.
+Only findings_json is scored; your prose answer is ignored, so every incorrect block you identify must appear as a finding, never only in the answer.
+Emit exactly one finding per contiguous failure block.
+Set the finding's subject to incorrect-steps-<first_step>-<last_step>-<escape_status>-consequence-<consequence_step>, using the same four values the task defines; for a block covering only step 7 that the agent never escaped and whose damage shows at step 9, the subject is incorrect-steps-7-7-unescaped-consequence-9.
+The runner expands the block to one scored step per member and builds every scored citation itself.
+Cite the block's first step and its last step as trace://<URL-encoded-trace-id>/span/step-<n>, each excerpt an exact quote from that step's own action content.
+Give the rationale as the concrete downstream evidence visible at the consequence step.
+Submit as soon as every candidate failure block has a supported verdict.
+Return no finding for a clean trajectory.

benchmarks/trace-analysis/codetracebench-oht2-20260801/family-framing/preregistration.md

@@ -0,0 +1,68 @@
+# Pre-registration — family-framing coverage arm, dev-pool smoke (written 2026-08-02, BEFORE any model run)
+
+Decomposition motivating this arm: [`.evolve/family-gap-decomposition-2026-08-02.md`](../../../../.evolve/family-gap-decomposition-2026-08-02.md).
+Target class: `far` + `silent` + under-enumeration, 57–58% of OpenHands/Terminus2 gold mass; measured ceiling snapFar +32.3pp (OH) / +25.4pp (T2) in the official currency.
+
+## Arm
+
+One change: `--instructions-file family-framing/oht2-coverage-instructions.txt` (sha256 `ecb8a207…`).
+The file is the stock RLM instructions (sha256 `d3829fb8…`, byte-identical extraction from `dist/analyst`) with one coverage-and-enumeration section and two family grammar notes inserted between the task prompt and the RLM output contract.
+No step-count, block-count, or width priors (shape-prior lesson: −10 to −15pp).
+Engine, model, limits, seed, and repetition count are identical across arms; the instructions text is the only difference.
+
+## Smoke instrument (TUNING-LEGAL dev pools, never cert32)
+
+Eligibility: dev-pool rows with ≥1 gold step AND `step_count` ≥ 40 (the long-trace regime where the family gap concentrates; 72 of 464 OH rows, 34 of 188 T2 rows qualify).
+Selection: `sha256("20260802" + "\0" + traj_id)` ascending, first 6 per family; committed as `smoke-labels-openhands.json` (sha256 `1b687899…`, 26 gold steps) and `smoke-labels-terminus2.json` (sha256 `f61fa566…`, 47 gold steps).
+2 repetitions, seed 0, so 12 observations per family per arm; 48 total across 4 runs (2 arms x 2 families).
+The stock arm runs fresh on the same cases: cert32 numbers do not transfer and the comparison is paired on caseId + repetition.
+
+## Protocol
+
+Strictly serial runs, glm-5.2 via z.ai direct, `--max-output-tokens 16384` (reasoning-model floor), `--concurrency 3`, `--timeout-ms 1200000`, per-run `--max-cost-usd 2.5`.
+Expected spend ≈ $6 (cert2 median $0.109/observation x 48 + margin); worst case $10 by per-run caps.
+The measurement mutex (`/tmp/ctb-llm-mutex.lock`) is held for the whole paid phase and released on every exit path.
+
+## Gates (fixed now; pooled = both families' 24 observations per arm, official micro)
+
+| Gate | Threshold |
+| --- | --- |
+| Primary (score) | pooled official micro F1(framing) ≥ pooled F1(stock) + 0.05 |
+| Mechanism A (enumeration) | predicted blocks per positive run (framing) ≥ stock + 0.5 |
+| Mechanism B (localization) | far share of gold observations (framing) ≤ stock − 10pp |
+| No-harm | neither family's official micro F1 < that family's stock − 0.05 |
+| Kill | pooled F1 < stock − 0.03, or >10% failed runs in either arm, or pad findings per positive run > stock x 1.6 |
+
+Decision rule: scale to a full dev measurement only if Primary AND (Mechanism A OR Mechanism B) AND No-harm all pass.
+Any kill condition stops the arm this round.
+Anything in between: no further paid runs this round; the readout feeds the next diagnosis.
+The smoke estimates the effect inside the long-trace labeled class; it certifies nothing, and no sealed split is touched.
+
+## Exact commands
+
+```bash
+FF=benchmarks/trace-analysis/codetracebench-oht2-20260801/family-framing
+for fam in openhands terminus2; do
+  for arm in stock framing; do
+    extra=""
+    [ "$arm" = framing ] && extra="--instructions-file $FF/oht2-coverage-instructions.txt"
+    MODEL_API_KEY="$ZAI_GLM_API_KEY" node dist/cli.js analyst-benchmark \
+      --dataset codetracebench \
+      --analyst dspy-rlm \
+      --python clients/python/.venv/bin/python \
+      --labels "$FF/smoke-labels-$fam.json" \
+      --trace-dir /dev/shm/ctb-oht2-traces-$fam \
+      --artifact-dir ~/bench-cache/ctb-20260801/oht2/work/$fam/extracted \
+      --out ~/bench-cache/ctb-20260801/family-framing-smoke/$arm-$fam \
+      --revision aa213b84ffb6690fc37ca15766d6ca174ec36d4d \
+      --split devsmoke-$fam-$arm-20260802 \
+      --base-url https://api.z.ai/api/coding/paas/v4 \
+      --api-key-env MODEL_API_KEY \
+      --model glm-5.2 \
+      --limit 6 --seed 0 --concurrency 3 --repetitions 2 \
+      --max-output-tokens 16384 --timeout-ms 1200000 --max-cost-usd 2.5 $extra
+  done
+done
+```
+
+Readout: `compare-analyst-runs.mjs` paired per family + pooled, and `decompose-analyst-loss.mjs` per family per arm for the mechanism gates.