SLM-421: powered Wilson-interval rerun of the SLM-282 recurrence-health screen - #861
Conversation
…health screen LAR3 stays gate-closed (PR #853-#856) on two blocking prerequisites, one of which is recursive_core_positive from the SLM-282 recurrence-health screen. That screen's n=2 evidence base (a fixed passed_seeds>=2/2 rule) is too thin to distinguish a real contraction violation from fixture-scale noise: the seed-1 failure was a single ~2% CE regression at one specific (depth, example) cell. Add an additive, default-off power_rule to _evaluate_recurrence_preregistration/_run_recurrence_health: when given {"min_pass_rate": <rate>}, the primary as_is arm's per-seed pass rate gets a Wilson score interval (slm_training.evals.power_protocol.wilson_interval, the repo's canonical binomial-evidence utility) and the disposition becomes recursive_core_positive/recursive_core_negative/inconclusive_underpowered based on where the interval sits relative to the threshold, instead of a raw count. power_rule=None (the default) reproduces the original 2/2 rule byte-for-byte, so historical SLM-282 evidence stays reproducible. New CLI flags --seed-count/--min-pass-rate/--issue expose this from `--mode recurrence-health`; --seed-count != 2 without --min-pass-rate fails closed (a raw pass count is not a meaningful threshold at arbitrary n). version_stamp: bump model.recursive_denoiser to v19. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6ZbkQY3QDmXVti2yom4Bv
…ve_core_positive Preregistered before running: primary arm as_is, unchanged CE(final) <= CE(previous) <= CE(r=1) condition, 20 fresh seeds (2..21, disjoint from SLM-282's 0/1), min_pass_rate=0.5 (Wilson 95% CI), locked independent of any outcome data from this seed range. Result: 18/20 seeds passed (rate 0.90, Wilson 95% CI [0.699, 0.972]) -> lower bound clears 0.5 -> recursive_core_positive. The SLM-282 seed-1 regression reads as fixture-scale noise at n=2, not a systematic contraction violation. This satisfies only one of PR #853-#856's two LAR3 reopening conditions (recursive_core_positive AND a passing repair advancement screen). LAR3 stays closed: SLM-317's value gate is the remaining blocker, and its harness is not yet merged to main, so it isn't a landable follow-up from this branch. Updated docs/brains/repo/recursive-recurrence-health.md (status dead-end -> active, both the SLM-282 and SLM-421 records preserved) and MOC.md open questions with the new evidence and the concrete next step (land SLM-317's harness so its value gate gets the same powered-rerun treatment). Verification: 154 passed / 1 skipped (tests/test_models/test_recursive_denoiser.py + tests/test_scripts/test_verify_version_stamps.py); ruff + format clean; repo_policy ok; verify_version_stamps --check ok; git diff --check clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6ZbkQY3QDmXVti2yom4Bv
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 32 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (7)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
2 participants
Summary
SLM-421 / LAR0-04: a powered rerun of the SLM-282 recurrence-health screen, one of the two blocking prerequisites PR #853/#854/#855/#856 just cited to gate-close LAR3-01..04 as
not_authorized.recursive_core_negativedisposition rested on a fixedpassed_seeds >= 2rule over only 2 seeds — seed 1 regressed by ~2% CE at one specific (depth-4, example-b) cell. That's too thin to tell a real contraction violation from fixture-scale noise, and SLM-286 (LAR0-03) already established Wilson-interval evidence classification as this repo's doctrine for exactly this situation; the recurrence-health screen predates that doctrine._evaluate_recurrence_preregistration/_run_recurrence_healthgain an optionalpower_rule(--seed-count/--min-pass-rate/--issueCLI flags on--mode recurrence-health). When set, the primaryas_isarm's per-seed pass rate gets a Wilson score interval (slm_training.evals.power_protocol.wilson_interval) and the disposition becomesrecursive_core_positive/recursive_core_negative/inconclusive_underpoweredbased on where the interval sits relative to the threshold.power_rule=None(the default) reproduces the original 2-seed rule byte-for-byte — historical SLM-282 evidence stays reproducible and all 4 pre-existing SLM-282 tests pass unchanged.nchanges), 20 fresh seeds (2..21, disjoint from SLM-282's 0/1),min_pass_rate = 0.5, locked in code before any of those seeds were observed.[0.699, 0.972]) → lower bound clears 0.5 →recursive_core_positive. The SLM-282 seed-1 regression reads as fixture-scale noise at n=2, not a systematic property of the core.recursive_core_positive). LAR3 stays closed — the second condition (a passing SLM-317-style valid-state repair advancement screen) is untouched by this issue; SLM-317's own harness lives on an unmerged branch, so a powered rerun of that screen isn't buildable against currentmainyet. Filed as the concrete next open question in the brain.Fold-back
docs/brains/repo/recursive-recurrence-health.md: statusdead-end→active; both the SLM-282 (n=2, negative) and SLM-421 (n=20, positive) records preserved and clearly attributed.docs/brains/repo/MOC.md: open-questions entry updated with the new evidence and next step.Verification
tests/test_models/test_recursive_denoiser.py+tests/test_scripts/test_verify_version_stamps.py), including 12 new/updated tests for the Wilson-interval power rule, CLI validation, and legacy-path preservation.ruff check+ruff format --checkclean.python -m scripts.repo_policyok.python -m scripts.verify_version_stamps --checkok (model.recursive_denoiserbumped v18 → v19, plus ano-bump:entry registering the new design-doc artifact).git diff --checkclean.Note
The first commit's message says "SLM-321" — that's a typo; it should read SLM-421 (this PR's actual issue). SLM-321 is an unrelated, already-closed issue (LAR3-02, see PR #854). Leaving it uncorrected per this session's "don't amend commits" instruction; flagging it here instead.
Closes SLM-421.
Generated by Claude Code