The gap
All build variants land on src/eigenscript (release, make asan, make http…). CLAUDE.md warns: "never rebuild one while a suite run against another is in flight." But that guard is prose only, and it protects invisible state — you can't see that a background suite is running. So a rebuild mid-run silently swaps the binary under the suite, and the run keeps going, grading later sections against the wrong binary. The result looks like a pass/fail but is meaningless.
Observed live: an ASan+leaks suite was running in the background; a make (release) was issued to verify an unrelated rebuild; the ASan run continued against the clobbered binary. Caught only by noticing, not by any check.
Proposed fix (loud, cheap)
Make the corruption an error instead of a silent-wrong result: run_all_tests.sh records the binary's hash (or mtime+size) at start, and re-checks it — at the end, or per-section — aborting with a clear message if it changed:
ERROR: src/eigenscript changed during the run (rebuilt mid-suite) — results are invalid.
This turns "silently graded against the wrong binary" into a loud failure, matching the repo's silent-tolerance discipline. No path or workflow changes.
Heavier alternative (root fix, bigger blast radius)
Give each variant its own output path (src/eigenscript, src/eigenscript-asan, src/eigenscript-http) so builds never collide. Eliminates this whole footgun class — including the related "make asan overwrites src/eigenscript, rebuild with make before timing" gotcha — but touches the Makefile, install.sh, CI workflows, and the test harness, all of which assume src/eigenscript. Filed as the option, not the recommendation.
Why it's worth it
The rule keeps being a rule you can violate invisibly; this project's pattern is that such rules should become mechanical checks. A context-rich agent tripped on it here — a lower bound on how often a human running two terminals will.
The gap
All build variants land on
src/eigenscript(release,make asan,make http…). CLAUDE.md warns: "never rebuild one while a suite run against another is in flight." But that guard is prose only, and it protects invisible state — you can't see that a background suite is running. So a rebuild mid-run silently swaps the binary under the suite, and the run keeps going, grading later sections against the wrong binary. The result looks like a pass/fail but is meaningless.Observed live: an ASan+leaks suite was running in the background; a
make(release) was issued to verify an unrelated rebuild; the ASan run continued against the clobbered binary. Caught only by noticing, not by any check.Proposed fix (loud, cheap)
Make the corruption an error instead of a silent-wrong result:
run_all_tests.shrecords the binary's hash (or mtime+size) at start, and re-checks it — at the end, or per-section — aborting with a clear message if it changed:This turns "silently graded against the wrong binary" into a loud failure, matching the repo's silent-tolerance discipline. No path or workflow changes.
Heavier alternative (root fix, bigger blast radius)
Give each variant its own output path (
src/eigenscript,src/eigenscript-asan,src/eigenscript-http) so builds never collide. Eliminates this whole footgun class — including the related "make asanoverwritessrc/eigenscript, rebuild withmakebefore timing" gotcha — but touches the Makefile, install.sh, CI workflows, and the test harness, all of which assumesrc/eigenscript. Filed as the option, not the recommendation.Why it's worth it
The rule keeps being a rule you can violate invisibly; this project's pattern is that such rules should become mechanical checks. A context-rich agent tripped on it here — a lower bound on how often a human running two terminals will.