SLM-338: unified evidence publisher for plan experiments (AP-037) - #862
Closed
Tyler-R-Kendrick wants to merge 18 commits into
Closed
SLM-338: unified evidence publisher for plan experiments (AP-037)#862Tyler-R-Kendrick wants to merge 18 commits into
Tyler-R-Kendrick wants to merge 18 commits into
Conversation
- coverage_class per frozen row (OBSERVABLE_PROMPT / OBSERVABLE_REQUEST_ONLY / UNKNOWN); all 51 rows are OBSERVABLE_REQUEST_ONLY pre-inventory. - Matched requests differ only by the production ensure_prompt_inventory suffix; request-derived, no hidden gold channel. - X22 deterministic arm: paired v2-strict delta +0.3125 (5/16 vs 0/16, Wilson [0.142, 0.556]) exceeds the predeclared 0.10 minimum; no regressions on observable rows. - AR tiny baseline: 6/32 decodes completed (all v2-fail partials), 26 decode_timeout (600-step checkpoint constrained-forest cost per SLM-294 evidence); reported unmeasured, not evidence. Rico rows not_run with SLM-294 cost evidence. - 100-record blind slot-observability audit: raw agreement 97%, kappa 0.0 (skew caveat); finding: v2 coverage detector misses inline slot enumeration -> false prompt_contract_unknown (append-only; no metric change). - 8 harness tests; new component harness.experiments.slm301_prompt_observability v1.
…ints - decode_outcome.py taxonomy with strict precedence (fallback never counts as model success); per-request budget/elapsed/forwards/ verifier/fallback/stop-reason records. - eval_runner additive integration: details[] gain decode_outcome, stop_reason, fallback_used; suite metrics gain decode_outcome_counts. - Census: only slm230 committed-SHA verifiable; hash-pinned the two remediated nonzero-timeout checkpoints; scoreboard classes separated (runtime_timeout/fallback/unmeasured/model_behavior). - Preregistered 1x vs 10x budget sweep: all 12 cells not_rerunnable (v0 checkpoints fail current output-contract check); the 10x-flip question is honestly UNANSWERED by re-decode. Recommendation: retrain the remediated recipe on contract v2, then re-eval at both budgets. - 25 new tests; harness.model_build.eval v50; component harness.experiments.slm303_decode_budget_audit v1; gates.ship v3 threshold mirror fixed.
- slm308_distance_oracle.py: bounded reverse-BFS over canonical AST fingerprints via the real extended 11-action transitions; EXACT / BOUNDED / UNKNOWN labels (budget never conflated with distance); cache keyed by action-schema version, grammar sha, inventory, target and state hashes. - tree_edit_diffusion: value_label_mode (bounded_distance default for new configs; pre-field checkpoints inject mutation_count for behavior parity, format stays 2); normalized oracle cost-to-go with UNKNOWN masked out of MSE; pairwise progress margin loss (independently tested); decode paths never read gold distance (audit test). - Matched fixture experiment (mutation_count vs bounded_distance, identical budgets, near-gold + seed-trajectory states): rank corr 0.433->0.505 (+0.072 < 0.10), beam regret 0.889->0.889 (+0.0 < 0.05), Brier 0.031->0.004, UNKNOWN coverage 0.175. Preregistered thresholds written before results; verdict honestly rejected at this budget. - 16 new tests; slm308 component v1; slm299 component v3.
- Proposal instrumentation: per enumerated candidate — action, factor score, applicability, rejection reason (23 machine-readable codes), budget consumption; deterministic order; additive evidence keys. - Distribution audit: training-target vs decode-demand action distributions by source/suite; dead-candidate rate ~0.98, applicable-ADD recall 0.10 baseline; preregistered reweighting rule. - corruption_action_distribution knob (gold corpus untouched; default off = historical uniform; parity tested) and stop_slot_accounting legacy|corrected arm (STOP consumes an expansion slot only when its frozen candidate is retained; deterministic, regression-tested). - Matched 2x2 (ADD-balanced x STOP arms, isolated levers): ADD target share 0.177 -> 0.328 (T1 +0.150 >= 0.10 both STOP arms), recall 0.10 -> 0.16, corrected STOP budget <= legacy; verdicts adopted per preregistered rules; loss reweighting deferred to preserve isolation. - 10 new tests; slm299 component v4; slm310 component v1.
- State-source harness: gold_only (existing corruption chain), seedward
(offline oracle-guided walk from the seed toward gold, strictly
distance-decreasing valid intermediates), on_policy (immutable
content-addressed beam-trajectory snapshots: wrong states, verifier
failures, abstentions); explicit provenance + gold-visibility policy
per row; sha256 rows + tamper-evident manifest; fail-closed leakage
guards (train vs held-out AST fingerprints).
- Matched {gold_only, seedward, on_policy, mixed} arms with predeclared
weights/caps, identical model/steps/optimizer/seeds; evaluated on
held-out seed trajectories. Verdict: rejected per preregistered
primary gate (beam regret improvement 0.0 < 0.05); secondary signal
(value rank corr -0.258 -> +0.258 for non-gold arms) recorded as
wiring evidence only.
- 9 new tests; slm312 component v1.
…training
- Frozen multi-mode dataset (8 prompts x 2 verifier-accepted modes;
canonical alpha-invariant AST fingerprints, sha256 rows + tamper-evident
manifest; prompt count declared below ship-gate scale).
- WTA/MCL loss: backpropagate only the min-loss mode per example +
preregistered floor term (eps=0.1) on losing modes; durable per-example
selection telemetry. No set-FTPO machinery.
- Two-mode synthetic fixture proves single-gold CE collapses
(p(mode B)=0.00004) while WTA retains both modes (p(B)=0.0083,
p(A)=0.826 dominant); multi-gold spreads 50% mass to invalid hybrids.
- Matched arms {single_gold, multi_gold, wta} identical budgets: coverage
0.750/0.500/0.500, hard-valid decode 1.0, v2 regression 0.000.
Verdict rejected per preregistered coverage gate (WTA gain -0.25);
at this scale single-gold lifts the sibling mode too.
- 16 new tests; slm314 component v1.
- Metamorphic invariance (alpha-rename, statement reorder, formatting,
AST roundtrip) tested before model evaluation.
- Hard do-no-harm commit rule: COMMIT on deterministic hard-evidence
improvement or calibrated soft improvement without hard regression;
RETAIN on regression/no-improvement; ABSTAIN on missing candidate;
source + candidate + exact reason durable per example; oracle commit
selector as non-promotable upper bound.
- Matched arms {ar_only, repair_only, ar_repair_historical,
ar_repair_improved, oracle_commit} on the frozen SLM-155 decision
corpus with paired per-example outcomes and explicit
invalid-over-valid counts per arm.
- Gates: Safety PASS (iov=0), Value FAIL (0/16 improvements, Wilson
[0.0, 0.194] vs 0.05), Reachability PASS. Disposition: inconclusive —
LAR3 remains open but NOT advanced; powered rerun required.
- 20 new tests; slm317 component v1.
LAR0-02 returned recursive_core_negative (1 of 2 required seed passes) and LAR2-06 returned inconclusive with the value gate failed (0/16, Wilson [0.0, 0.194] vs 0.05); LAR3 is open but NOT advanced. No production code, defaults, or checkpoints; LAR3-02..05 stay blocked. Reopening requires recursive_core_positive plus a passing repair advancement screen.
…ise negative SLM-319 closed not_authorized (LAR3 entry gates unmet), so the recurrent-core abstraction this issue anchors on does not exist; independently SLM-282's matched audit found the as_is arm negative (1/2 seeds) and residual_delta_can_promote=false. No production code.
- AbstractPlanningResultV1: canonical result object (locked campaign manifest sha, code/model/data versions, raw/constrained/repaired metric paths, plan controls, causal interventions, latency/compute breakdown, verifier gates, AgentV/human-audit refs, claim class); content sha + publication_blockers fail-closed on missing provenance, manifest hash, controls, or total latency; promotion classes additionally require all three decode paths + negative control + checkpoint. - publish_planning_result.py: one command emits canonical JSON, narrative Markdown rendered strictly from the JSON, and a model-card-ready roster row (printed, never auto-inserted); idempotent append-only disposition sections keyed by campaign hash; exit 1 with named reasons on any blocker. - 22 tests incl. MD-numbers-come-from-JSON, fail-closed, idempotency, immutability, determinism; contract doc + fixture publication; slm338 component v1.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 22 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (83)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Tyler-R-Kendrick
left a comment
Tyler-R-Kendrick
left a comment
Owner
Author
There was a problem hiding this comment.
Not merging — this branch is based on a stale main (merge-base 528feeb, main is now eec4864) and merging it would silently revert newer, already-merged work. Concrete regressions in the diff vs origin/main:
- Ship-gate weakening (violates repo law 'do not weaken ship gates'):
src/slm_training/harnesses/model_build/ship_gates.pydowngradesthreshold_versionopenui_ship_gates_v4 → v3 andpolicy_idv4 → v2. - Reverts merged model work:
src/slm_training/models/tree_edit_diffusion.pyremoves ACTION_SET_PROPERTY and drops N_ACTIONS 12 → 11 (reverting the VAR1-02 change landed via #1070-class work), plus ~750 lines of other drift. - Reverts eval/telemetry fixes:
evals/agentv.pydrops_run_trace_idwiring;runtime/telemetry/trace.pyloses lines;eval_runner.pyhas 274 lines of churn against newer main. - Reverts repo docs/config: README loses the goal-law (decode-invariants) preamble; pyproject loses the
[tool.uv]virtual-package config and exclude-package-data entries;versions.jsondiverges by ~18k lines against the current registry. - PR is CONFLICTING against main, and much of its content (SLM-299/303/305 harnesses + iter docs) is already merged on main — those parts are superseded duplicates.
The genuinely new pieces (SLM-308/310/312/314/317 harnesses + iter docs, and the AP-037 unified evidence publisher: scripts/publish_planning_result.py, harnesses/autoresearch/planning_result.py, fixtures, and contract doc) look like the intended contribution. Please rebase/cherry-pick only those onto current main as a fresh branch and open a new PR; as-is this PR cannot merge without regressing ship gates and merged harness work.
Owner
Author
|
Closing as superseded — content already landed on main via other PRs (see review comments for superseding commits). Reopen only with a rebased, still-unique delta. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SLM-338 / AP-037: unified evidence publisher for plan experiments.
AbstractPlanningResultV1: one canonical result object — locked campaign manifest sha (reuses SLM-337's digest), code/model/data versions, raw/constrained/repaired metric paths, plan controls (oracle/random/empty/shuffled), causal interventions, latency/compute breakdown, verifier gates, AgentV/human-audit references, claim class. Content-addressed;publication_blockers()fails closed on missing provenance, bad manifest hash, missing controls, or missing/non-positive total latency. Promotion classes additionally require all three decode paths, a negative control, and a checkpoint.scripts/publish_planning_result.py): emits canonical JSON, narrative Markdown rendered strictly from the JSON (test-enforced: MD numbers come from the JSON, never hand-copied), and a model-card-ready roster row (printed, never auto-inserted — MODEL_CARD.md untouched). Idempotent append-only disposition sections keyed by campaign hash; exit 1 with named reasons on any blocker.docs/design/abstract-planning-evidence-contract.md+ fixture publication artifacts.Verification
harness.experiments.slm338_evidence_publisherv1); repo_policy ok; git diff --check clean; ruff + format clean.Closes SLM-338.