feat(benchmarks): SWE-agent normalizer skill — corpus complete, all four families importable - #531
Merged
Merged
Conversation
Add the sweagent_traj CodeTracer user skill (one step per trajectory[] entry, blanks included; 106/108 verified rows reproduce the annotated step_count), route the SWE-agent family through prepare-bulk.py, and record the import results. The 2 mismatched rows publish function-call style trajectories whose annotated view condensed the run; they fail the step-count gate instead of importing misaligned labels.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The last CodeTraceBench family: a
sweagent_trajCodeTracer user-skill +prepare-bulk.pyrouting, following the OHT2 pattern exactly (pinned CodeTracer2d302191, dataset revisionaa213b84, fail-loud per row).Upstream has no SWE-agent parser (verified at pin and HEAD); the convention was derived empirically over all 108 archives — one-step-per-
trajectory[]-entry matched 106/108 (vs 21/108 for both alternatives; derivation table in receipts). The 2 failures are function-call-style trajectories whose published views cannot reproduce the annotated count (45/48 and 92/94 empty response fields) — excluded loudly, never imported.Live result
106/108 imported: 4,370 steps, 8,952 spans, 200 gold incorrect steps across 57 labeled rows. Receipts + labels + traces durable in
~/bench-cache/ctb-20260801/sweagent/.Proof
Calibration note for future sealing: a constant last-or-penultimate rule touches 39/106 rows (crude F1 ≈ 0.26) — positionally milder than the retired split3 (0.568) but any sealed SWE-agent split requires the full constant-rule calibration first (recorded in the README).
Corpus status after this: mini-SWE 133, OpenHands 496, Terminus2 220, SWE-agent 106 — 955 labeled sessions importable, zero families remaining.