Conversation
Add temporary config and eval plumbing, preserve hunk-first atoms with compatible packing and profile floors, emit Stage 6 provenance, and support base or atom-scaled tool budgets.
Implement Plan 102 steps 5-6 with fail-closed replay, evaluation economics, raw policy reconstruction, deterministic reconciliation, and focused regression coverage.
Plan 102 failed its treatment gate because it partitioned packing candidates by the planner's per-hunk coverage grade. That predicate is self-defeating for recall validation: a hunk interesting enough to carry an expectation is graded differently from its safe siblings, so the target could never pack. Both paid cohorts died on treatment validity, never on recall, while the free deterministic replay passed every invariant across four production runs. Plan 103 changes the predicate. Atoms combine when the existing hunk relationship graph relates them - same_symbol, strong symbol_mention, uniquely-resolved planner_hint, plus a new tree-sitter same_container source keyed on structural owner identity. Packets grow by connectivity-preserving greedy expansion, so they stay connected without requiring source contiguity. Coverage promotion is permitted and recorded per member rather than banned, leaving Plan 92's packet-level escalation field untouched. The container signal lives in a separate packing-affinity view so flag-off context, notes, profiles, and the relationship-graph artifact stay byte-identical. Symbol context follows membership: multi-symbol packets read every member's primary symbol under a shared budget instead of silently keeping the top-ranked one. Each paid pair consumes one hash-pinned Stage-5 plan, so packing is the only within-pair difference and cross-arm reconciliation becomes exact. A pre-registered decision table with explicit Void conditions replaces the inherited non-inferiority bound, which could have passed on two zeroes. Free replay gates are numeric and counted on distinct diffs. Single flag, no budget-mode arm, teardown to one path under either outcome. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ship ordering Artifact-only diagnostics over the retained runs disproved the first draft's premise. Relationship-required packing yields 4 combinable atom pairs and a 4.2% packet reduction on the one run with usable planner coverage, against Plan 102's measured 21.9%. Of 13 candidate atom pairs, the 5-hunk cap blocks 7 and coverage inequality blocks only 2 - and zero structural pairs cross a coverage grade. Relationships between distinct atoms are sparse because today's grouper already absorbs most same-symbol adjacency into atoms. So the plan now adopts Plan 102's compatibility predicate and caps wholesale - its deterministic gate already passed on four runs - and demotes relatedness from a membership predicate to a composition-ordering preference. Related atoms are admitted first when filling a packet, which changes which hunks sit together without changing how many packets exist. Selection order and render order are separated explicitly; hunks always display in source order. A cap sweep against a simulator validated to within one packet of Plan 102's real implementation shows 5 hunks captures ~80% of achievable compression, with 10 adding ~5 points before saturating. The cap stays. Deleted on evidence: the relationship-required predicate, all coverage-promotion machinery and its CoverageEscalation conflict with Plan 92, and the proposed stage reordering (the planner already grades related siblings consistently in 132 of 134 measured pairs). The paid phase is now one question - recall versus packet size, measured at caps 1/3/5 over six repeats sharing one hash-pinned Stage-5 plan. It is a non-inferiority test because the benefit, roughly 19 more reviewed hunks per run, is already measured deterministically. Relationship ordering gets no paid arm; it is adopted or dropped on a free deterministic gate. Adds an introduction and TL;DR in plain language covering the problem, the outcome, the seven pieces of machinery, the dilution risk, and the delete-or-ship teardown. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Consolidates the three throwaway analyses that drove Plan 103's revision
into one reusable tool. Reads recorded telemetry runs only - no review,
no model calls - and answers three questions:
pairs which atoms are related but currently split, and whether
coverage, the caps, or a whole-file bypass is what blocks them
sweep how packet count responds to the hunk and patch caps
simulate packet count under source, compatibility, or related admission
Calibration: sweep at 5h/12K yields 75/66/83 packets against Plan 102's
measured 75/68/85, within the documented patch-proxy residual.
Known approximations are recorded in the file header: patch size uses the
recorded contentWithLineNumbers proxy rather than combinedPatchChars(),
container identity is recovered by splitting the owner prefix out of the
enclosingSymbol display string rather than a structural ownerKey, and
undeclared hunks share a synthetic DEFAULT lens signature. This is a
planning aid; Plan 103 step 8's real-builder replay is what gates the work.
Note for whoever lands Plan 103's report script: tsconfig.json includes
only src and tests, so scripts/*.ts is never typechecked by pnpm run
check. Plan 102 shipped a 5,318-line TypeScript script that no gate ever
looked at.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… experiment Measured relationship-preferred admission against plain source order with scripts/packing-diagnostics.mjs: identical packing on all three distinct retained diffs - same packet counts, same composition. Related atoms are usually already source-adjacent, so preferring them picks the same atom. Combined with the earlier finding that relationship-required packing yields 4.2% against compatibility's 21.9%, the whole relationship subsystem is removed: no affinity view, no ownerKey, no same_container, no edge-cap rework. Diagnostic outputs are preserved under packet-dilution/reports with their own manifest, leaving Plan 102's manifest exactly as its reconciliation note describes. Review of the prior draft found two blocking defects in the paid experiment, both fixed: The dilution fixture could repeat Plan 102's zero-treatment failure. Pinning a drawn plan makes arms comparable but not compatible - if the planner grades the buggy hunk deep and its siblings normal they never share a partition. The plan is now hand-authored rather than drawn, which guarantees one partition, removes planner variance, removes any temptation to redraw unfavourable samples, and costs nothing. Treatment is proven model-free at 15/5/3 packets before any paid call, and a reference draw is recorded alongside so the authored plan's realism is documented. The decision table could approve a net loss. With integer counts the old rule had no reachable Inconclusive branch, R3 was inert, and R1=5/R5=4 passed at 80% against a break-even of 89/108 = 82.4%. The bar is now derived from the fixed-slot hunk yield measured in phase A, requires a reliable R1, and gives R3 a real role distinguishing a cliff from a gradient. Three independent bugs at three packet positions raise resolution from 6 to 18 observations per arm at the same spend. Also: fixed-slot hunk yield at 57 dispatch slots becomes the primary free gate, replacing packet count as a proxy for the coverage claim; multi-member symbol context gets explicit budget arithmetic with quality measured after final rendering; the fixture moves to evals/packet-dilution since it is synthetic and belongs in the standing suite; cross-language shape cases are added because skill-semantics files are 10-15 lines and yield one atom each; both settings become eval/internal-only rather than repo-configurable; and the TL;DR drops the false claim that compatibility prevents more expensive review. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Five defects, all resolved before implementation. No-symbol atoms were undefined. The context contract required every member's primary-symbol source to survive 600 characters, but config, docs, and generic-adapter hunks have no resolvable primary symbol - readEnclosingSymbolSource returns empty and their standalone quality is outline_only or path_only. As written, every packet containing one would have been rejected, which is exactly the file shape where compatible atoms are most abundant. Only symbol-bearing members now participate in the allocation; symbol-less members are exempt from the floor and must instead retain their standalone quality. The reference planner draw had no gate. Recording a divergence between the authored plan and a real draw is not enough - if production would rarely create a 15-atom compatible partition, the experiment measures a shape that does not occur. Three draws are now taken and at least two must place every target in a partition of three or more compatible atoms, or the fixture is invalid before any reviewer call. Two bugs shared a cap-5 packet. Positions 1, 5, 8 put the first two in packet [1-5], so two opportunities collapsed into one conversation and that arm reviewed an unnaturally bug-dense packet. Positions are now 1, 10, 13, which keeps all three in separate packets at both cap 3 and cap 5 while retaining first/last/middle placement. The 18 opportunities are also described as clustered rather than independent, reported per bug, and a per-bug collapse rule stops an aggregate from hiding one target falling from reliable detection to zero. Void and extension were discretionary and mutually contradictory - the decision table said fix the fixture, teardown said remove packing immediately. Both are now bounded and preregistered: one fixture redesign after a first Void, one six-repeat extension across all three arms after a first monotone decline, the same rule re-applied to combined totals, and a second occurrence of either is a Fail. Both contingencies are budgeted, raising the reservation to ~$119. The fixed-slot estimator was unreconciled. It is now validated: applying the worker-runner sort to the flag-off packets and taking the first N reproduces exactly the 89 hunks the run actually reviewed, with 55 of 56 packets in common. Reconciliation is a gate before the estimator may define B. The measurement also corrected the slot count - artifacts record 40 not-dispatched of 96, so the baseline is 56 dispatched packets, not the 57 in Plan 102's note. Also: production capacity confirmation is required rather than optional; pinned plans validate schema and diff identity, not just hash and hunk IDs; preservation commands copy the frozen plan, all reference draws, capacity reports, and a not_run ledger before regenerating the manifest; and diagnostics reproduction covers all three distinct diffs. scripts/packing-diagnostics.mjs now applies lens-signature compatibility in its pair classifier, which it previously omitted. On dca8d870 that reclassifies 13 pairs as 2 coverage / 2 lenses / 6 caps / 3 eligible, strengthening rather than weakening the case against relationship packing. Preserved evidence regenerated; Plan 102's manifest untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ify pinned artifact Five contract defects found in review, all verified against telemetry or code before fixing. The scheduler description was factually wrong. I claimed the undispatched set is a suffix of dispatch order. Worker IDs encode that order, and the telemetry shows w7-056 recorded not_dispatched while w7-057 completed - a prefix with a hole, which is expected at concurrency 4 where a later worker can clear its budget checkpoint before an earlier one fails. That hole, not missing input-order telemetry, is also the real explanation for the 55/56 packet overlap. The estimator is now described as a calibrated counterfactual capacity proxy validated against a known 89-hunk outcome, not as scheduler reproduction, and the stale --dispatch-slots 57 is 56. Extension thresholds were absolute and would silently halve on extension. Applying R1 < 15 to a 36-opportunity cohort halves the baseline bar and redefines collapse. All thresholds are now proportions - Void at R1/N < 5/6, collapse at R1[i]/K >= 5/6 with R5[i]/K <= 1/6 - yielding 15/18 and 5-of-6 initially, 30/36 and 10-of-12 after the one authorized extension. The realism gate proved the wrong thing, and the fix differs from the one proposed. Plan 102's replay shows five-HUNK packets are ordinary production output (7-8 per run, ~10% of packed packets) while five-ATOM packets never occur (max 4, once per run). Since the fixture uses fifteen single-hunk atoms, its cap-5 arm produces a five-hunk packet - the production-real shape. So cap 5 is a genuine production case rather than a stress bound, but only via hunks, and the plan now claims nothing about five-atom packets. The gate is expressed in hunks and enforced by a new realism report mode that rebuilds each draw's packet shapes and exits non-zero below threshold, with draws taken cache-off for independence. The pinned-plan artifact was underspecified. ReviewPlan carries no base/head identity and pinnedPlanPath alone defined no trusted expected hash, leaving the executor to invent a contract. Specified as a versioned PinnedPlanArtifact with baseSha, headSha, and planSha256 over a canonical sorted-key serialization of the plan alone, with per-failure-mode tests. The draw mode becomes scripts/draw-pinned-plan.ts rather than the pnpm dev plan CLI verb I invented, which does not exist and would have contradicted the eval-only scope; src/cli/ stays out of scope. The capacity pair had no machine-checkable rule. It now requires both sides to consume the same pinned plan and identical provider settings, and gates on three numbers: at least 10% more reviewed hunks, a reviewed set that is a superset of baseline minus at most 2 hunks, and measured yield within 10% of the phase-A prediction. Also: step 8 says Extend-once rather than the retired Stop-for- investigation; evidence preservation copies the realism report and the reference diffs it emits; and the diagnostics header no longer claims Plan 103 uses ownerKey or relationship-preferred ordering - those modes now state they exist to reproduce a rejected design. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…se 1) Implements Plan 103 steps 1-3: the two eval-only settings, semantic atoms wrapped around today's hunkFirstGroups() output, and Plan 102's compatibility partition with source-order greedy fill behind a dark flag. Config surface is deliberately narrow. review.packCompatibleAtoms (default false) and review.packMaxHunks (default 5) live in CodegenieConfig, the resolved schema, defaults, the strict eval-case schema, and applyCaseReviewConfig. They are absent from rawConfigSchema, DEFAULT_SOURCE_PATHS, and REPO_SAFE_REVIEW_KEYS, so no codegenie.toml or user config.toml can reach them - tests assert those throw rather than being silently filtered. Config-source attribution is deliberately not claimed for a value no config file can supply. MAX_PACK_HUNKS is duplicated in schema.ts rather than imported from the pipeline to keep the config schema dependency-free, with a test pinning the two together. PacketGroup gained a required origin field. The direct whole-file and content-probed file-diff returns are now structurally excluded from packing rather than inferred from kind, which is ambiguous because packetGroup() can also produce file-diff. packCompatibleAtoms() partitions each file's atoms by effective coverage and normalized planner lens signature, fills sequentially in source order under packMaxHunks and MAX_PATCH_CHARS, orders packets by earliest member hunk, and rebuilds combined groups through the existing packetGroup and packetKind rules with degradation reasons merged by sorted deduplicated join. Fill is sequential-flush, matching Plan 102's described algorithm rather than the best-fit search used in the planning simulation: sequential preserves strict source order where best-fit would let a later small atom jump ahead of an earlier larger one. The two differ by about one packet on the retained runs, inside the documented residual. Atoms are indivisible. An atom that alone exceeds a cap still becomes its own packet rather than being split. Twelve new tests, no existing test modified: 761 to 773. A one-hunk cap is asserted byte-identical to flag-off, which is the strongest parity check available without a pre-change baseline. Coverage and lens boundaries, hunk bijection, in-packet source ordering, atom indivisibility, whole-file bypass, dispatch-rank recomputation, and the 15/5/3 packet shapes Phase 4's recall curve depends on are all covered. The phase plan records one scope correction: the plan's atom description also lists standalone review profile and per-member context quality, but both need the dry-build machinery Phase 2 owns and neither is needed to partition or fill, so they move to Phase 2 rather than being half-built here. Packing stays off by default and must not be enabled before Phase 2 - without multi-member symbol context a packed packet would carry only the top-ranked member's source and leave the rest as bare diffs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…rofile floor (Plan 103 phase 2) Implements Plan 103 steps 4-5 plus the atom metadata deferred from phase 1. Multi-member symbol context. readMemberSymbolSources() reads every distinct primary symbol in a packed packet under a shared 5,000-character budget with equal shares, a floor, and one redistribution pass handing surplus from members needing less to members needing more. At least 3,000 of MAX_CONTEXT_CHARS stays reserved for outline, tests, and hints. Symbol source renders at the head of the packet context and truncateTail keeps the head, so the reserve is what guarantees survival - but survival is verified against the final rendered text by locating each member's header rather than assumed. A single-symbol packet delegates unchanged to readEnclosingSymbolSource, so flag-off behaviour is untouched. Transactional evaluation. Candidates dry-build against a scratch relationship accumulator and a quiet telemetry sink with throwaway metrics; only a candidate passing every check is rebuilt against the real recorder. A rejected candidate emits its atoms separately with a recorded reason and leaves no telemetry, graph, or artifact residue. Single-atom packets build once, unchanged. Profile floor, applied before lens routing rather than after. routedPacketLenses prunes core/code-review at a simple profile, so raising the profile afterwards would leave a lens dropped that the floor exists to preserve. Verified with the case Plan 102 described: two hunks sharing an enclosing symbol separated by a third, where packing absorbs the same_symbol edge target. Two findings changed the design. A short symbol is not a collapsed member. The first cut rejected any member emitting under MIN_SLICED_MEMBER_CHARS, which fails a packet whose symbol is simply small - a 500-character function is fully represented. Members now carry a complete flag and only an incomplete member below the floor counts as collapsed. Without this, packing would have been rejected for most real packets. Both budget rejections are unreachable at the shipped cap, and that is now pinned by test. A packet holds at most 5 hunks so at most 5 distinct symbols; 5 x 800 = 4,000 <= 5,000 means oversubscription cannot fire, and the smallest share of 1,000 exceeds the 600 sliced minimum so truncation-collapse cannot fire. Both are kept as defensive invariants that become live only if the cap rises past six, with a test pinning the arithmetic so a cap change surfaces there. Two drafted tests asserting unreachable behaviour were removed rather than contrived into passing. 773 to 778 tests, no existing test modified. The flag stays false; phase 3 owns the replay gate that authorizes any paid work. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ase 3) Implements Plan 103 step 7. Step 6 (the pinned-plan seam) is deliberately deferred to phase 4: it is paid-phase infrastructure, and this replay is the gate that decides whether a paid phase happens at all. scripts/packet-packing-report.ts rebuilds Stage 6 from recorded run artifacts with the real builder - resolved input, diff parse, file filter, classification, repository index, buildReviewPackets off and on - observing zero model calls. Failure records are structured and templated: a closed-set code plus typed fields, no raw exception text and no repository source. The gate passes: dca8d870 96 -> 75 21.9% yield@56 89 -> 109 (+20) 142 hunks 81f806a6 93 -> 68 26.9% yield@56 89 -> 116 (+27) 136 hunks 740d73f2 93 -> 69 25.8% yield@56 90 -> 116 (+26) 137 hunks fe1548ae duplicate diff of 740d73f2 Zero failures, three distinct diffs. Estimator reconciles at exactly 89; flag-on yield 109 clears the 102 bar; reduction exceeds 20% on every distinct diff; deviation from Plan 102's frozen counts is 0 on both post-Plan-100 runs. Break-even ratio for phase 4 is B = 89/109 = 0.8165. Report preserved with its hash in the private evidence manifest. The reconciliation gate paid for itself on first use. The initial replay produced 114 off-packets against the recorded 96 and 161 reviewable hunks against 142. The diff was byte-correct at 88 files and 217 hunks, so the divergence was downstream: the reviewed repository carries its own codegenie.toml whose classification path rules skip generated webrpc clients and docs, schema dumps, and e2e snapshots, and raise workers, protocol, and migrations to critical. The real run loaded it - 88 files, 61 kept, 27 skipped - while the replay used bare defaults and reviewed files the run never saw. applyRepoConfigLayer fixed it, after which dca8d870 reproduces Plan 102's frozen 96 -> 75 and its 142 reviewable hunks exactly. Without the gate this would have reported a plausible-looking 20.2% on the wrong workload. Two older runs legitimately diverge: 740d73f2 and fe1548ae replay at 93 packets against a recorded 109. Both predate Plan 100's hunk-identity change and both had their planner output dropped entirely, which Plan 102's reconciliation note already flagged as needing a compatibility view. Their off/on comparison stays internally valid, so they contribute reduction evidence but not historical-count evidence. comparePackets originally tested coverage in one direction, catching promotion but not demotion - the more dangerous case, since a demoted hunk is reviewed more shallowly than planned. The partition key forces identical coverage across a packet's members, so any change is a violation; the check is now an inequality named coverage_changed. Importing the script no longer runs main, so its pure helpers are unit tested - and being reachable from tests means this script is typechecked by pnpm run check, unlike Plan 102's 5,318-line report which never was. 778 to 782 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…reconditions) Records the owner-approved ceiling and completes the free preconditions that must pass before Plan 103 spends anything. approvedValidationCostUSD: $300, recorded in the plan. The owner approved the $119 reservation, raised the ceiling to $300, and authorized running the production capacity pair inside phase 4. Spend to date: $0.00. Fixture: one Go file, fifteen independent guard functions separated by ~62 lines so today's grouper yields exactly fifteen atoms. Three bugs at atom positions 1, 10 and 13 - an inclusive bound turned exclusive, a retry guard off by one, and a shard index mapped outside its range - plus a negative control at atom 7 that looks like a boundary edit and is provably equivalent. Adds a treatment mode to the report script: rebuilds Stage 6 at caps 1/3/5 with zero model calls and asserts 15/5/3 packets, each target in a packet of exactly that size, and no two targets sharing a packet. That check caught a fixture defect on first run. I had targeted the hunk at line 679, which is atom 11 rather than atom 10, placing two targets inside the same cap-5 packet and collapsing two of three recall opportunities into one model conversation - the exact clustering defect raised in review. targets_share_packet failed closed and the target was corrected to line 617. Without the proof the curve would have run at $14 with two of three observations correlated. Still required before the first paid call: the pinned-plan seam, the three eval case YAMLs, fake-provider expectation validation, and the authored plan plus three reference draws with the realism gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e 4) Completes the remaining preconditions before the recall curve. Pinned-plan seam (step 6, carried from phase 3). ReviewPlan has no base/head identity and no self-describing hash, so a bare plan file cannot prove it belongs to the diff under review. PinnedPlanArtifact is an explicit versioned wrapper carrying baseSha, headSha, and planSha256 over a canonical serialization - keys sorted recursively, no whitespace, over the plan alone - so the hash is independent of field order and of the wrapper's own fields. loadPinnedPlan fails closed on unreadable JSON, wrapper schema mismatch, plan schema mismatch, hash mismatch, base mismatch, head mismatch, and any hunk id absent from the current diff. A hash that matches a mutated plan is still rejected against its diff. Fixture repos are materialized fresh per eval run, so their commit SHAs vary while content-derived hunk ids do not. A "*" sentinel records that an artifact is content-anchored rather than ref-anchored; hash, schema, and hunk-membership validation still apply in full. Ref pinning stays available for the production pair, where SHAs are fixed. review.pinnedPlanPath is eval-only: resolved schema and eval-case schema only, never rawConfigSchema or repo-safe filtering. The review runner substitutes the pinned plan for the Stage-5 draw and records a planner_plan_pinned event. scripts/draw-pinned-plan.ts writes artifacts as an internal script rather than a CLI verb, keeping src/cli out of scope as the plan requires. Three case YAMLs at repeat 6, cache off, lang/go only, identical except packMaxHunks 1/3/5, all pinned to one authored plan of 15 uniform- coverage hunks (planSha256 52e62561...). Expectations constrain path and failure mode with permissive title patterns and no category constraint - Plan 102 lost three executions to an expectation that demanded correctness when the reviewer said security. A should_not_find control covers the provably-equivalent AtCapacity rewrite. 785 tests. Spend so far is one in-flight smoke execution; the full curve has not run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three unpinned cap-5 draws all placed every target in a three-hunk packet rather than five, and put all three targets in the same packet: draw 1 packet deep 3h starts: 59, 617, 806 draw 2 packet deep 3h starts: 59, 617, 806 draw 3 packet deep 3h starts: 59, 617, 806 The real planner grades exactly the three bug hunks deep and everything else normal or light, splitting the fifteen atoms across three coverage partitions. The authored plan's uniform normal coverage yields one partition and five-hunk packets; production yields a three-atom deep packet. The curve would have measured dilution at a shape production does not build for these hunks, and with three recall opportunities collapsed into one model conversation. Draws 1 and 3 are byte-identical, so this is deterministic planner behaviour rather than sampling variance. Cause is a fixture defect against this plan's own rule 3: each bug must be one the planner grades ordinarily, because a bug obvious enough to draw a deep grade lands in its own partition and never packs. Comparator-boundary bugs are exactly that obvious. This is the same structural finding the artifact-only diagnostics produced before implementation, now confirmed with real model calls. Per the pre-registered stop condition the fixture must be redesigned before any reviewer call. Gate cost $2.85 and prevented $27.36 of measurement at the wrong shape. Total spend $4.3634 of the $300 ceiling. Also records measured cost at $1.5154 per execution against the $0.78 projected, and three redesign options for the owner to choose between. Side evidence, not a gate result: across four paid executions nine of ten target opportunities were hit, with one loss at verification and none before candidate generation. Nothing so far suggests packing is harmful. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…roduction A/B
The money-path fixture tested a real hypothesis - that planner coverage
tracks risk surface rather than detected defects, so a file where every
function handles money would be graded uniformly - and falsified it. The
planner located precisely the three defective functions and packed them
into one deep packet, identically to the comparator-flip fixture:
draw 1 packet deep 3h starts: 63, 462, 861
packet light 2h starts: 130, 995
packet normal 5h starts: 197,264,331,398,529
packet normal 5h starts: 596,663,730,797,928
Four fixture designs across two plans have now failed the same way: 102's
cross-atom-consistency and dilution-control, and 103's limits and
settlement. In every case the seeded defects were graded deep and their
safe siblings normal or light, partitioning targets away from the hunks
they were meant to pack with.
The obstacle is structural. The compatibility predicate separates atoms by
coverage; the planner assigns coverage by detected defect. A synthetic
fixture therefore cannot manufacture a defect packed among safe siblings
at equal coverage - any bug findable enough to measure recall against is
visible enough to be graded deep. This is the harness working correctly
and it is not solvable by better fixture craft. Step 8's curve is not_run
by design; the four-fixture record is the finding.
Owner decision: ship behind the flag and A/B on real pull requests
instead. Recorded as a deliberate exception to this plan's no-dark-path
teardown rule, with conditions - the flag stays false by default and
eval-only, no codegenie.toml or user config can reach it, it is not a
supported product surface, and it is either promoted to unconditional or
deleted once the production A/B reports. It does not persist unreviewed.
The rationale is that real PRs contain what fixtures cannot: hunks the
planner grades ordinarily that nonetheless contain defects. That is where
the benefit comes from - the planner issued coverage for 32 of 142 hunks
on the motivating run, so the 21.9% reduction is overwhelmingly the
packing of default-normal atoms it never singled out.
What passed stands: phases 1-3 complete, free replay gate clearing every
pre-registered threshold with zero model calls - 96 to 75 packets, fixed
slot yield 89 to 109 against a 102 bar, estimator reconciling at exactly
89, zero deviation from Plan 102's frozen counts, and no coverage, profile,
budget, lens, cap or hunk violations across four runs.
Paid validation $5.3850 of a $300 ceiling; measured cost $1.5154 per
execution against $0.78 projected. Steps 9-11 never authorized. Both
fixture repositories retained as the record of why synthetic measurement
was abandoned. 785 tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Renames review.packCompatibleAtoms to review.packRelatedHunks across source, tests, eval cases, the report script, and the plan documents - 37 occurrences, no behaviour change. Adds --pack-related-hunks and --no-pack-related-hunks to the review command, following the repo's kebab-case convention, wired through CliConfigOverrides and applyCliOverrides with "cli" config-source attribution. The setting is now reachable from exactly two places: an eval case, and this per-run CLI flag. It stays absent from rawConfigSchema and REPO_SAFE_REVIEW_KEYS, so no codegenie.toml and no user config.toml can set it - a repository cannot enable experimental packing for everyone who reviews it. A test asserts a codegenie.toml setting it still throws invalid config file while the CLI flag succeeds and is attributed to cli. This brings src/cli into scope, which Plan 103 originally excluded. That exclusion was written when the setting had no product consumer; the owner-approved production A/B is one, and toggling per run is the whole point of it. Recorded in the plan's reconciliation note. 786 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Off is already the default, so the negative form carried no meaning that omitting the flag did not. Omitting it now leaves the setting at its default with "defaults" config-source attribution, which a test pins. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Without this a multi-hunk packet is indistinguishable from one today's grouper produced on its own - both look identical in the packet artifact - so a packing A/B cannot tell treated packets from untreated ones. Plan 102's design called for this provenance; phase 1 trimmed atom metadata from scope and phase 2 added the context work but not the record, and the omission only surfaced when the first real A/B run started. Emitted once per committed packed packet, carrying packet id, ordered source atom ids and their hunk counts, resulting hunk ids and count, coverage, lens signature, routed lenses, each member's standalone profile, derived versus effective profile with a floor-applied flag, and cap usage against packMaxHunks and MAX_PATCH_CHARS. Artifact-only: it stays out of the reviewer prompt and out of the packet id, so it cannot change review behaviour or packet identity. Purely additive - packet construction is untouched. Two tests: a six-atom file at a five-hunk cap emits exactly one event naming five source atoms and matching the committed packet id, and nothing is emitted with packing off. 788 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…om-run The script only authored uniform plans - every hunk the same coverage and lenses. That was right for the dilution fixture, where making packet size the only variable was the point, but wrong for a production A/B: it discards the planner's per-hunk deep/normal/light grading and reviews a workload production never produces. --from-run <runDir> wraps the real Stage-5 draw recorded by an existing run, so both arms of an A/B replay one planner output and packing is the only difference between them. It validates before writing: every hunk id in the recorded plan must be present in the diff for the given base and head, and a mismatch exits non-zero naming the stale ids. That caught a real trap immediately - the review pipeline diffs from the merge-base, not from the base branch tip, so drawing against "master" produces different hunk ids than the run used. The error now says so instead of silently emitting an artifact that would fail at load time. Output reports whether the plan was recorded or authored, its coverage histogram, and how many diff hunks it covers, so an unlisted-hunk gap is visible before the artifact is used. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With a pinned plan Stage 5 never executes, so nothing wrote review-plan.json and the run could not describe which plan produced its packets. Runs 50 and 51 both have an empty 05-planner stage directory - the plan they actually used lives only in /tmp, outside the run, and the artifacts stop being self-contained the moment that file moves or is overwritten by the next A/B. The seam now writes review-plan.json with the pinned plan, matching what the planner would have written, plus pinned-plan-source.json recording the source path, the canonical plan hash, the coverage entry count, and the base and head the plan was validated against. The planner_plan_pinned event carries the hash too, so two arms of an A/B can be proven to have consumed the same plan from their artifacts alone rather than by trusting that the file did not change between them. Both names are added to the artifact path map so they land in stages/05-planner/ rather than outside it, with a test pinning the seam and the map together - the failure mode is silent, since an unmapped artifact still writes, just somewhere useless. 789 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
plans 102 and 103 are experiments to group more related hunks into fewer packets. The idea would be that we have fewer tokens used, lower cost and faster review time, while maintaining the same level of quality. However, in practice and experiments, it seems the cost and time was the same, and quality in fact went down. So far, this experiment isn't to be merged, but could be useful for future thoughts/work.