Add null-hypothesis and file-reconstruction validity scenarios#189
Open
DianaMeda wants to merge 2 commits into
Open
Add null-hypothesis and file-reconstruction validity scenarios#189DianaMeda wants to merge 2 commits into
DianaMeda wants to merge 2 commits into
Conversation
GOAL.md's scenario set requires two validity instruments beyond the complexity scenarios: - null-hypothesis-rate-limiter: single-session task (sessionCount 1) where Jumbo cannot improve cross-session retention. Keeps the full Jumbo prompt asymmetry (one goal + lifecycle protocol) but NO pre-seeded memory, so any statistically significant lift here indicts the measurement (prompt structure, scoring bias, contamination), not memory. Expected result: near-zero lift. - file-reconstruction-expense-tracker: 5-session task whose prompt mandates an in-repo DECISIONS.md decision log, giving the baseline agent a legitimate file-based context-recovery path. Pre-seeded memory only restates prompt content (kept information-symmetric at session 1). Tests whether Jumbo's advantage is specific to context that cannot be recovered from file artefacts. Expected result: lower lift than the complexity scenarios. tests/unit/scenarios-validity.test.ts locks the design properties (single session; no pre-seed on the null; DECISIONS.md assertions present; pre-seed restates prompt) and proves each scenario's structural assertions are satisfiable by a correct implementation and zero on an empty one. 394/394 green; tsc clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1. scenario create dropped structuralAssertions: the template loader's
whitelist predates Outcome 1, so registered scenarios silently lost their
primary retention signal ("No structural assertions defined — trivially
satisfied"). ScenarioTemplate + handleScenarioCreate now pass the field
through; regression test added.
2. Headless agents were permission-paralyzed: in claude -p mode no human can
approve tool use, so both arms stalled ("unable to write files without
permission approval") and produced zero source files. The adapter now
invokes claude with --dangerously-skip-permissions — each session runs in
an isolated throwaway temp workdir, which is what the flag exists for. This
also survives jumbo init (>=3.12) overwriting .claude/settings.json, which
was discarding the seeded Bash(jumbo:*) allowlist (observed as
settings.json.backup.* in the run snapshot).
395/395 green; tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The two validity instruments GOAL.md's scenario set requires beyond the complexity scenarios (§Scenarios, items 2 and 3). Docs/data + tests only — no runner or scoring code changes (that's the Outcome 1 guarantee: scenarios are pure data).
scenarios/null-hypothesis-rate-limiter.json— a single-session task (sessionCount: 1) where Jumbo cannot improve cross-session retention. It deliberately keeps the Jumbo prompt asymmetry (one goal + lifecycle protocol) but has no pre-seeded memory, so any statistically significant quality lift here indicts the measurement (prompt structure, scoring bias, contamination) — not memory. A passing eval shows near-zero lift. This is the framework's own falsification check.scenarios/file-reconstruction-expense-tracker.json— a 5-session task whose shared prompt mandates an in-repoDECISIONS.mddecision log and whose continuation prompt tells the agent to re-read it. That gives the baseline arm a legitimate file-based context-recovery path, testing whether Jumbo's advantage is specific to context that cannot be recovered from artefacts, or whether it is a better file-reading aid. Pre-seeded memory only restates prompt content (information-symmetric at session 1). Expected: lower lift than the complexity scenarios.Tests
tests/unit/scenarios-validity.test.tslocks the experimental-design properties as executable invariants (single-session + no pre-seed on the null; ≥3DECISIONS.mdassertions + prompt-only pre-seeding on the reconstruction scenario) and proves each scenario's structural assertions are satisfiable by a correct implementation and zero on an empty one — so a future edit can't silently break the design.394/394 unit tests green;
tsc --noEmitclean.Why now
With Outcomes 1–5 merged, these two scenarios complete the GOAL.md scenario set and unblock the first real K=5 measurement: null (validity gate) → complexity (headline) → reconstruction (control).
🤖 Generated with Claude Code