SLM-324: close LAR3-03 not_authorized - #855
Closed
Tyler-R-Kendrick wants to merge 12 commits into
Closed
Conversation
- coverage_class per frozen row (OBSERVABLE_PROMPT / OBSERVABLE_REQUEST_ONLY / UNKNOWN); all 51 rows are OBSERVABLE_REQUEST_ONLY pre-inventory. - Matched requests differ only by the production ensure_prompt_inventory suffix; request-derived, no hidden gold channel. - X22 deterministic arm: paired v2-strict delta +0.3125 (5/16 vs 0/16, Wilson [0.142, 0.556]) exceeds the predeclared 0.10 minimum; no regressions on observable rows. - AR tiny baseline: 6/32 decodes completed (all v2-fail partials), 26 decode_timeout (600-step checkpoint constrained-forest cost per SLM-294 evidence); reported unmeasured, not evidence. Rico rows not_run with SLM-294 cost evidence. - 100-record blind slot-observability audit: raw agreement 97%, kappa 0.0 (skew caveat); finding: v2 coverage detector misses inline slot enumeration -> false prompt_contract_unknown (append-only; no metric change). - 8 harness tests; new component harness.experiments.slm301_prompt_observability v1.
…ints - decode_outcome.py taxonomy with strict precedence (fallback never counts as model success); per-request budget/elapsed/forwards/ verifier/fallback/stop-reason records. - eval_runner additive integration: details[] gain decode_outcome, stop_reason, fallback_used; suite metrics gain decode_outcome_counts. - Census: only slm230 committed-SHA verifiable; hash-pinned the two remediated nonzero-timeout checkpoints; scoreboard classes separated (runtime_timeout/fallback/unmeasured/model_behavior). - Preregistered 1x vs 10x budget sweep: all 12 cells not_rerunnable (v0 checkpoints fail current output-contract check); the 10x-flip question is honestly UNANSWERED by re-decode. Recommendation: retrain the remediated recipe on contract v2, then re-eval at both budgets. - 25 new tests; harness.model_build.eval v50; component harness.experiments.slm303_decode_budget_audit v1; gates.ship v3 threshold mirror fixed.
- slm308_distance_oracle.py: bounded reverse-BFS over canonical AST fingerprints via the real extended 11-action transitions; EXACT / BOUNDED / UNKNOWN labels (budget never conflated with distance); cache keyed by action-schema version, grammar sha, inventory, target and state hashes. - tree_edit_diffusion: value_label_mode (bounded_distance default for new configs; pre-field checkpoints inject mutation_count for behavior parity, format stays 2); normalized oracle cost-to-go with UNKNOWN masked out of MSE; pairwise progress margin loss (independently tested); decode paths never read gold distance (audit test). - Matched fixture experiment (mutation_count vs bounded_distance, identical budgets, near-gold + seed-trajectory states): rank corr 0.433->0.505 (+0.072 < 0.10), beam regret 0.889->0.889 (+0.0 < 0.05), Brier 0.031->0.004, UNKNOWN coverage 0.175. Preregistered thresholds written before results; verdict honestly rejected at this budget. - 16 new tests; slm308 component v1; slm299 component v3.
- Proposal instrumentation: per enumerated candidate — action, factor score, applicability, rejection reason (23 machine-readable codes), budget consumption; deterministic order; additive evidence keys. - Distribution audit: training-target vs decode-demand action distributions by source/suite; dead-candidate rate ~0.98, applicable-ADD recall 0.10 baseline; preregistered reweighting rule. - corruption_action_distribution knob (gold corpus untouched; default off = historical uniform; parity tested) and stop_slot_accounting legacy|corrected arm (STOP consumes an expansion slot only when its frozen candidate is retained; deterministic, regression-tested). - Matched 2x2 (ADD-balanced x STOP arms, isolated levers): ADD target share 0.177 -> 0.328 (T1 +0.150 >= 0.10 both STOP arms), recall 0.10 -> 0.16, corrected STOP budget <= legacy; verdicts adopted per preregistered rules; loss reweighting deferred to preserve isolation. - 10 new tests; slm299 component v4; slm310 component v1.
- State-source harness: gold_only (existing corruption chain), seedward
(offline oracle-guided walk from the seed toward gold, strictly
distance-decreasing valid intermediates), on_policy (immutable
content-addressed beam-trajectory snapshots: wrong states, verifier
failures, abstentions); explicit provenance + gold-visibility policy
per row; sha256 rows + tamper-evident manifest; fail-closed leakage
guards (train vs held-out AST fingerprints).
- Matched {gold_only, seedward, on_policy, mixed} arms with predeclared
weights/caps, identical model/steps/optimizer/seeds; evaluated on
held-out seed trajectories. Verdict: rejected per preregistered
primary gate (beam regret improvement 0.0 < 0.05); secondary signal
(value rank corr -0.258 -> +0.258 for non-gold arms) recorded as
wiring evidence only.
- 9 new tests; slm312 component v1.
…training
- Frozen multi-mode dataset (8 prompts x 2 verifier-accepted modes;
canonical alpha-invariant AST fingerprints, sha256 rows + tamper-evident
manifest; prompt count declared below ship-gate scale).
- WTA/MCL loss: backpropagate only the min-loss mode per example +
preregistered floor term (eps=0.1) on losing modes; durable per-example
selection telemetry. No set-FTPO machinery.
- Two-mode synthetic fixture proves single-gold CE collapses
(p(mode B)=0.00004) while WTA retains both modes (p(B)=0.0083,
p(A)=0.826 dominant); multi-gold spreads 50% mass to invalid hybrids.
- Matched arms {single_gold, multi_gold, wta} identical budgets: coverage
0.750/0.500/0.500, hard-valid decode 1.0, v2 regression 0.000.
Verdict rejected per preregistered coverage gate (WTA gain -0.25);
at this scale single-gold lifts the sibling mode too.
- 16 new tests; slm314 component v1.
- Metamorphic invariance (alpha-rename, statement reorder, formatting,
AST roundtrip) tested before model evaluation.
- Hard do-no-harm commit rule: COMMIT on deterministic hard-evidence
improvement or calibrated soft improvement without hard regression;
RETAIN on regression/no-improvement; ABSTAIN on missing candidate;
source + candidate + exact reason durable per example; oracle commit
selector as non-promotable upper bound.
- Matched arms {ar_only, repair_only, ar_repair_historical,
ar_repair_improved, oracle_commit} on the frozen SLM-155 decision
corpus with paired per-example outcomes and explicit
invalid-over-valid counts per arm.
- Gates: Safety PASS (iov=0), Value FAIL (0/16 improvements, Wilson
[0.0, 0.194] vs 0.05), Reachability PASS. Disposition: inconclusive —
LAR3 remains open but NOT advanced; powered rerun required.
- 20 new tests; slm317 component v1.
LAR0-02 returned recursive_core_negative (1 of 2 required seed passes) and LAR2-06 returned inconclusive with the value gate failed (0/16, Wilson [0.0, 0.194] vs 0.05); LAR3 is open but NOT advanced. No production code, defaults, or checkpoints; LAR3-02..05 stay blocked. Reopening requires recursive_core_positive plus a passing repair advancement screen.
…ise negative SLM-319 closed not_authorized (LAR3 entry gates unmet), so the recurrent-core abstraction this issue anchors on does not exist; independently SLM-282's matched audit found the as_is arm negative (1/2 seeds) and residual_delta_can_promote=false. No production code.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 52 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (60)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Tyler-R-Kendrick
left a comment
Tyler-R-Kendrick
left a comment
Owner
Author
There was a problem hiding this comment.
Reviewed, not merging — stale-branch regression with semantic conflicts beyond versions.json.
Findings:
- Branch is based on 528feeb (main @ PR #835, 2026-07-24); main has since advanced ~3 days (2531 files changed, now at 8c3702a). Trial merge against origin/main conflicts in 14 files, including code semantics: src/slm_training/harnesses/model_build/ship_gates.py, eval_runner.py, evals/agentv.py, runtime/telemetry/trace.py, models/tree_edit_diffusion.py (25 conflict hunks), models/checkpoint_migrate.py, plus docs/scripts/tests.
- Concrete regression: the PR's ship_gates.py downgrades MEANINGFUL_METRIC_POLICY threshold_version from main's openui_ship_gates_v4 back to openui_ship_gates_v3 (and policy_id v4 -> v2) — it would silently revert newer gate-versioning work on main, exactly the ship-gate weakening AGENTS.md forbids.
- This is a cumulative branch (SLM-303/305/308/310/312/314/317/319/321/324); main already contains the SLM-299/301/303 portions, and open PRs #851/#846/#853/#854 overlap the earlier rungs — this PR duplicates them while carrying the stale shared-file versions.
To land SLM-324: rebase a branch containing ONLY the SLM-324 closeout (docs/design/iter-slm324-* + its versions.json bump) onto current main, leave the already-merged/earlier-rung files alone, and reopen.
Owner
Author
|
Closing to drain the stale queue. Review comments above document what (if anything) is still unique here — re-open as a focused branch rebased on current main if that delta is still wanted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gate closeout: both prerequisites are closed not_authorized (SLM-319 LAR3-01 shared core, SLM-321 LAR3-02 update modes — same LAR3 entry-gate evidence). Loop-specific LoRA/FiLM has no shared core to attach to. No production code. Evidence: docs/design/iter-slm324-lar3-gate-closeout-20260725.{json,md}. Closes SLM-324.