SLM-319: close LAR3-01 not_authorized — entry gates unmet - #853
Closed
Tyler-R-Kendrick wants to merge 10 commits into
Closed
SLM-319: close LAR3-01 not_authorized — entry gates unmet#853Tyler-R-Kendrick wants to merge 10 commits into
Tyler-R-Kendrick wants to merge 10 commits into
Conversation
- coverage_class per frozen row (OBSERVABLE_PROMPT / OBSERVABLE_REQUEST_ONLY / UNKNOWN); all 51 rows are OBSERVABLE_REQUEST_ONLY pre-inventory. - Matched requests differ only by the production ensure_prompt_inventory suffix; request-derived, no hidden gold channel. - X22 deterministic arm: paired v2-strict delta +0.3125 (5/16 vs 0/16, Wilson [0.142, 0.556]) exceeds the predeclared 0.10 minimum; no regressions on observable rows. - AR tiny baseline: 6/32 decodes completed (all v2-fail partials), 26 decode_timeout (600-step checkpoint constrained-forest cost per SLM-294 evidence); reported unmeasured, not evidence. Rico rows not_run with SLM-294 cost evidence. - 100-record blind slot-observability audit: raw agreement 97%, kappa 0.0 (skew caveat); finding: v2 coverage detector misses inline slot enumeration -> false prompt_contract_unknown (append-only; no metric change). - 8 harness tests; new component harness.experiments.slm301_prompt_observability v1.
…ints - decode_outcome.py taxonomy with strict precedence (fallback never counts as model success); per-request budget/elapsed/forwards/ verifier/fallback/stop-reason records. - eval_runner additive integration: details[] gain decode_outcome, stop_reason, fallback_used; suite metrics gain decode_outcome_counts. - Census: only slm230 committed-SHA verifiable; hash-pinned the two remediated nonzero-timeout checkpoints; scoreboard classes separated (runtime_timeout/fallback/unmeasured/model_behavior). - Preregistered 1x vs 10x budget sweep: all 12 cells not_rerunnable (v0 checkpoints fail current output-contract check); the 10x-flip question is honestly UNANSWERED by re-decode. Recommendation: retrain the remediated recipe on contract v2, then re-eval at both budgets. - 25 new tests; harness.model_build.eval v50; component harness.experiments.slm303_decode_budget_audit v1; gates.ship v3 threshold mirror fixed.
- slm308_distance_oracle.py: bounded reverse-BFS over canonical AST fingerprints via the real extended 11-action transitions; EXACT / BOUNDED / UNKNOWN labels (budget never conflated with distance); cache keyed by action-schema version, grammar sha, inventory, target and state hashes. - tree_edit_diffusion: value_label_mode (bounded_distance default for new configs; pre-field checkpoints inject mutation_count for behavior parity, format stays 2); normalized oracle cost-to-go with UNKNOWN masked out of MSE; pairwise progress margin loss (independently tested); decode paths never read gold distance (audit test). - Matched fixture experiment (mutation_count vs bounded_distance, identical budgets, near-gold + seed-trajectory states): rank corr 0.433->0.505 (+0.072 < 0.10), beam regret 0.889->0.889 (+0.0 < 0.05), Brier 0.031->0.004, UNKNOWN coverage 0.175. Preregistered thresholds written before results; verdict honestly rejected at this budget. - 16 new tests; slm308 component v1; slm299 component v3.
- Proposal instrumentation: per enumerated candidate — action, factor score, applicability, rejection reason (23 machine-readable codes), budget consumption; deterministic order; additive evidence keys. - Distribution audit: training-target vs decode-demand action distributions by source/suite; dead-candidate rate ~0.98, applicable-ADD recall 0.10 baseline; preregistered reweighting rule. - corruption_action_distribution knob (gold corpus untouched; default off = historical uniform; parity tested) and stop_slot_accounting legacy|corrected arm (STOP consumes an expansion slot only when its frozen candidate is retained; deterministic, regression-tested). - Matched 2x2 (ADD-balanced x STOP arms, isolated levers): ADD target share 0.177 -> 0.328 (T1 +0.150 >= 0.10 both STOP arms), recall 0.10 -> 0.16, corrected STOP budget <= legacy; verdicts adopted per preregistered rules; loss reweighting deferred to preserve isolation. - 10 new tests; slm299 component v4; slm310 component v1.
- State-source harness: gold_only (existing corruption chain), seedward
(offline oracle-guided walk from the seed toward gold, strictly
distance-decreasing valid intermediates), on_policy (immutable
content-addressed beam-trajectory snapshots: wrong states, verifier
failures, abstentions); explicit provenance + gold-visibility policy
per row; sha256 rows + tamper-evident manifest; fail-closed leakage
guards (train vs held-out AST fingerprints).
- Matched {gold_only, seedward, on_policy, mixed} arms with predeclared
weights/caps, identical model/steps/optimizer/seeds; evaluated on
held-out seed trajectories. Verdict: rejected per preregistered
primary gate (beam regret improvement 0.0 < 0.05); secondary signal
(value rank corr -0.258 -> +0.258 for non-gold arms) recorded as
wiring evidence only.
- 9 new tests; slm312 component v1.
…training
- Frozen multi-mode dataset (8 prompts x 2 verifier-accepted modes;
canonical alpha-invariant AST fingerprints, sha256 rows + tamper-evident
manifest; prompt count declared below ship-gate scale).
- WTA/MCL loss: backpropagate only the min-loss mode per example +
preregistered floor term (eps=0.1) on losing modes; durable per-example
selection telemetry. No set-FTPO machinery.
- Two-mode synthetic fixture proves single-gold CE collapses
(p(mode B)=0.00004) while WTA retains both modes (p(B)=0.0083,
p(A)=0.826 dominant); multi-gold spreads 50% mass to invalid hybrids.
- Matched arms {single_gold, multi_gold, wta} identical budgets: coverage
0.750/0.500/0.500, hard-valid decode 1.0, v2 regression 0.000.
Verdict rejected per preregistered coverage gate (WTA gain -0.25);
at this scale single-gold lifts the sibling mode too.
- 16 new tests; slm314 component v1.
- Metamorphic invariance (alpha-rename, statement reorder, formatting,
AST roundtrip) tested before model evaluation.
- Hard do-no-harm commit rule: COMMIT on deterministic hard-evidence
improvement or calibrated soft improvement without hard regression;
RETAIN on regression/no-improvement; ABSTAIN on missing candidate;
source + candidate + exact reason durable per example; oracle commit
selector as non-promotable upper bound.
- Matched arms {ar_only, repair_only, ar_repair_historical,
ar_repair_improved, oracle_commit} on the frozen SLM-155 decision
corpus with paired per-example outcomes and explicit
invalid-over-valid counts per arm.
- Gates: Safety PASS (iov=0), Value FAIL (0/16 improvements, Wilson
[0.0, 0.194] vs 0.05), Reachability PASS. Disposition: inconclusive —
LAR3 remains open but NOT advanced; powered rerun required.
- 20 new tests; slm317 component v1.
LAR0-02 returned recursive_core_negative (1 of 2 required seed passes) and LAR2-06 returned inconclusive with the value gate failed (0/16, Wilson [0.0, 0.194] vs 0.05); LAR3 is open but NOT advanced. No production code, defaults, or checkpoints; LAR3-02..05 stay blocked. Reopening requires recursive_core_positive plus a passing repair advancement screen.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 56 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (56)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Jul 25, 2026
Tyler-R-Kendrick
left a comment
Tyler-R-Kendrick
left a comment
Owner
Author
There was a problem hiding this comment.
Not merging — stale branch regresses newer main work in shared files (diff is CONFLICTING, merge-base 528feeb from Jul 24; main has moved ~40 PRs since). Concrete regressions the squash would introduce:
- Weakens ship gates (violates AGENTS.md 'do not weaken ship gates'):
src/slm_training/harnesses/model_build/ship_gates.pydowngradesMEANINGFUL_METRIC_POLICY['threshold_version']fromopenui_ship_gates_v4tov3and_meaningful_metric_policypolicy_id fromv4tov2. Main's v4 came via 8e3783f (SLM-299, #826) and 84086ff (SLM-266). - Reverts main's tree-edit action space:
src/slm_training/models/tree_edit_diffusion.pydropsACTION_SET_PROPERTYand shrinksN_ACTIONS12→11, reverting 6b6dc36 (#1067, pack-owned SET_PROPERTY) and 9bc5a0b (#1040). - Deletes the goal-law block from README.md ('This is not a natural-language LLM…' invariant summary) — I15 requires these to stay linked.
- Reverts LangSmith trace wiring in
src/slm_training/evals/agentv.py(drops_run_trace_id, 60c0f25 #841) with mis-indented command list.
Partial supersession: the SLM-303/305/299 docs+scripts in this branch are already on main; SLM-310 and SLM-314 have dedicated open PRs (#846, #851). The SLM-319-specific payload here is just docs/design/iter-slm319-lar3-gate-closeout-20260725.{md,json} — please rebase a branch that adds only the SLM-319 closeout docs on top of current main (resolving versions.json against main, never downgrading main's component versions) and resubmit.
Tyler-R-Kendrick
added a commit
that referenced
this pull request
Jul 27, 2026
…th screen (#861) * SLM-321: add Wilson-interval powered rerun to the SLM-282 recurrence-health screen LAR3 stays gate-closed (PR #853-#856) on two blocking prerequisites, one of which is recursive_core_positive from the SLM-282 recurrence-health screen. That screen's n=2 evidence base (a fixed passed_seeds>=2/2 rule) is too thin to distinguish a real contraction violation from fixture-scale noise: the seed-1 failure was a single ~2% CE regression at one specific (depth, example) cell. Add an additive, default-off power_rule to _evaluate_recurrence_preregistration/_run_recurrence_health: when given {"min_pass_rate": <rate>}, the primary as_is arm's per-seed pass rate gets a Wilson score interval (slm_training.evals.power_protocol.wilson_interval, the repo's canonical binomial-evidence utility) and the disposition becomes recursive_core_positive/recursive_core_negative/inconclusive_underpowered based on where the interval sits relative to the threshold, instead of a raw count. power_rule=None (the default) reproduces the original 2/2 rule byte-for-byte, so historical SLM-282 evidence stays reproducible. New CLI flags --seed-count/--min-pass-rate/--issue expose this from `--mode recurrence-health`; --seed-count != 2 without --min-pass-rate fails closed (a raw pass count is not a meaningful threshold at arbitrary n). version_stamp: bump model.recursive_denoiser to v19. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6ZbkQY3QDmXVti2yom4Bv * SLM-421: run the powered recurrence-health rerun, disposition recursive_core_positive Preregistered before running: primary arm as_is, unchanged CE(final) <= CE(previous) <= CE(r=1) condition, 20 fresh seeds (2..21, disjoint from SLM-282's 0/1), min_pass_rate=0.5 (Wilson 95% CI), locked independent of any outcome data from this seed range. Result: 18/20 seeds passed (rate 0.90, Wilson 95% CI [0.699, 0.972]) -> lower bound clears 0.5 -> recursive_core_positive. The SLM-282 seed-1 regression reads as fixture-scale noise at n=2, not a systematic contraction violation. This satisfies only one of PR #853-#856's two LAR3 reopening conditions (recursive_core_positive AND a passing repair advancement screen). LAR3 stays closed: SLM-317's value gate is the remaining blocker, and its harness is not yet merged to main, so it isn't a landable follow-up from this branch. Updated docs/brains/repo/recursive-recurrence-health.md (status dead-end -> active, both the SLM-282 and SLM-421 records preserved) and MOC.md open questions with the new evidence and the concrete next step (land SLM-317's harness so its value gate gets the same powered-rerun treatment). Verification: 154 passed / 1 skipped (tests/test_models/test_recursive_denoiser.py + tests/test_scripts/test_verify_version_stamps.py); ruff + format clean; repo_policy ok; verify_version_stamps --check ok; git diff --check clean. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6ZbkQY3QDmXVti2yom4Bv --------- Co-authored-by: Claude <noreply@anthropic.com>
Owner
Author
|
Closing to drain the stale queue. Review comments above document what (if anything) is still unique here — re-open as a focused branch rebased on current main if that delta is still wanted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SLM-319 (LAR3-01 untied prelude/shared semantic core/untied coda) must not be implemented while LAR3 entry prerequisites fail. They fail:
recursive_core_negative— 1 of 2 required seed passes on the preregistered recurrence-health condition. Onlyrecursive_core_positivecan satisfy this prerequisite.inconclusive— safety PASS, value FAIL (0/16 improvements, Wilson [0.0, 0.194] vs 0.05 minimum useful effect); LAR3 explicitly open but NOT advanced.Closeout only:
docs/design/iter-slm319-lar3-gate-closeout-20260725.{json,md}with the gate table, evidence links, and reopening conditions (recursive_core_positive AND a passing repair advancement screen). No production code, defaults, or checkpoints; LAR3-02..05 stay blocked.Verification
Closes SLM-319.