From 5551547b0a49d8b081a004937a78bfbea3c83258 Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Fri, 24 Jul 2026 21:24:12 -0400 Subject: [PATCH 01/24] feat: add same-file packet packing experiment Add temporary config and eval plumbing, preserve hunk-first atoms with compatible packing and profile floors, emit Stage 6 provenance, and support base or atom-scaled tool budgets. --- specs/plans/phase-plans/102/phase_1.md | 28 ++ src/config/config-loader.ts | 26 +- src/config/schema.ts | 7 + src/evals/eval-runner.ts | 11 + src/pipeline/packet-builder.ts | 375 ++++++++++++++- src/types.ts | 7 + tests/config-loader.test.ts | 36 ++ tests/evals.test.ts | 36 +- tests/pipeline-phase5.test.ts | 629 +++++++++++++++++++++++++ 9 files changed, 1137 insertions(+), 18 deletions(-) create mode 100644 specs/plans/phase-plans/102/phase_1.md diff --git a/specs/plans/phase-plans/102/phase_1.md b/specs/plans/phase-plans/102/phase_1.md new file mode 100644 index 0000000..a941963 --- /dev/null +++ b/specs/plans/phase-plans/102/phase_1.md @@ -0,0 +1,28 @@ +--- +status: complete +--- + +# Phase 1: Dark Same-File Packet Packing + +## Overview + +Implement Plan 102 steps 1–4 only. This phase adds temporary experiment configuration, preserves the current hunk-first groups as explicit indivisible atoms, packs only compatible same-file atoms behind the dark flag, preserves each atom's exact standalone review profile and routed-lens/focus safeguards, emits Stage-6 treatment provenance, and keeps base versus atom-scaled tool budgets experimentally separable. The report script, retained-run replay, paid evals, rollout decision, and teardown remain out of scope. + +## Steps + +1. Extend `src/types.ts`, `src/config/schema.ts`, `src/config/config-loader.ts`, and `src/evals/eval-runner.ts` with temporary `review.packSameFileHunks: boolean` and `review.packedToolBudgetMode: "base" | "atom-scaled"` fields. Default them to `false` and `base`, apply user/repository/eval overrides through strict schemas, mark both repository-safe, record config-source winners, and expose the effective values in eval run metadata. +2. Refactor only `hunkFirstGroups()` results in `src/pipeline/packet-builder.ts` into internal hunk-first groups and explicit packet atoms. Give atoms stable ordered-hunk IDs, exact rendered patch size, source position, effective coverage, normalized sorted/deduplicated planner-lens signature, and the exact standalone packet profile derived by the existing Stage-6 build path. Preserve the direct whole-file and content-probed file-diff returns, and bypass all atom work when packing is disabled so packet artifacts, IDs, order, profiles, context, lenses, and budgets remain unchanged. +3. Add a stable compatible-atom pass for hunk-first atoms only. Partition by effective coverage and normalized planner-lens signature, greedily pack in partition source order under five-hunk/12,000-character caps, restore packet order by first source position, preserve atom and hunk ordering, merge degradation reasons deterministically, and rebuild packet kind/context/routing/dispatch rank through existing helpers. Dry-build candidates against isolated relationship telemetry so a newly lost standalone routed lens or high/critical focus note leaves atoms separate. Apply an explicit `simple < standard < investigate` profile floor from exact standalone profiles and emit `same_file_atoms_packed` Stage-6 provenance without changing prompts or packet IDs. +4. Add the isolated budget policy in `src/pipeline/packet-builder.ts`. `base` uses the existing budget from the effective profile. `atom-scaled` applies only to non-simple packets that combine multiple pre-existing atoms, adds at most one call and 2,000 result characters per additional atom subject to 1.75x ceilings, leaves investigation rounds and source-extension policy unchanged, and applies `budgetBoost` last. +5. Add focused coverage in `tests/config-loader.test.ts`, `tests/evals.test.ts`, and `tests/pipeline-phase5.test.ts` for strict/default/source config behavior, eval propagation, flag-off parity, atom identity/profile capture, whole-file bypass, compatible packing, atom/cap/source-order/coverage/lens/focus/degradation/dispatch invariants, relationship-driven profile floors (including `symbol_mention`, `planner_hint`, ordinary context, `primarySymbols`, and `same_symbol` controls), Stage-6 provenance, and base/atom-scaled budgets. +6. Run the focused Plan 102 commands, then the full repository workflow: `pnpm run check`, `pnpm test`, and `pnpm build`. Fix all failures without implementing Plan 102 step 5 or later. + +## Tests + +- `config loader defaults and precedence for packet-packing experiment fields`: proves `false`/`base` defaults, repository-safe overrides, source attribution, and strict enum rejection. +- `eval packet-packing review overrides`: proves strict YAML parsing, both overrides reaching effective run config, and invalid budget modes failing closed. +- `packet atom flag-off parity and whole-file bypass`: proves dark mode preserves packet/artifact shape and direct whole-file/file-diff construction never enters the packer. +- `same-file packing compatibility and invariants`: proves atoms remain indivisible, coverage and requested-lens boundaries hold, interleaved partitions restore stable order, all hunks appear once in source order, caps split correctly, degradation reasons merge deterministically, and dispatch rank uses the existing combined changed-line formula. +- `same-file packing focus and lens safeguards`: proves a candidate that would newly omit a standalone routed lens or high/critical focus is left unpacked. +- `same-file packing profile floor`: proves absorbed strong `symbol_mention`/`planner_hint`, ordinary related context, and the same-name `primarySymbols` case cannot reduce effective profile or base budget, while `same_symbol` alone does not invent an investigate profile. +- `packed tool budget policies`: proves base parity, simple zero budget, one-atom/five-hunk parity, per-additional-atom scaling and 1.75x caps across standard/investigate/deep packets, unchanged rounds/source extension, and boost-last behavior. diff --git a/src/config/config-loader.ts b/src/config/config-loader.ts index bb8d2de..0b7490b 100644 --- a/src/config/config-loader.ts +++ b/src/config/config-loader.ts @@ -64,6 +64,8 @@ const DEFAULT_SOURCE_PATHS = [ "review.maxTime", "review.perPassTimeoutMs", "review.budgetBoost", + "review.packSameFileHunks", + "review.packedToolBudgetMode", "review.maxBudgetTokens", "github.summaryWhenNoFindings", "classification.pathRules", @@ -79,7 +81,15 @@ const DEFAULT_SOURCE_PATHS = [ "eval.logsDir" ]; -const REPO_SAFE_REVIEW_KEYS = new Set(["depth", "maxFindings", "softCommentCap", "budgetBoost", "maxTime"]); +const REPO_SAFE_REVIEW_KEYS = new Set([ + "depth", + "maxFindings", + "softCommentCap", + "budgetBoost", + "maxTime", + "packSameFileHunks", + "packedToolBudgetMode" +]); const CREDENTIAL_KEY_PATTERN = /(?:api[_-]?key|apikey|secret|token|password|passwd|authorization|credentials|auth)/i; export function loadConfig(opts: LoadConfigOptions): LoadedConfig { @@ -254,6 +264,14 @@ function applyRawConfig( config.review.budgetBoost = raw.review.budgetBoost; sources["review.budgetBoost"] = source; } + if (raw.review?.packSameFileHunks !== undefined) { + config.review.packSameFileHunks = raw.review.packSameFileHunks; + sources["review.packSameFileHunks"] = source; + } + if (raw.review?.packedToolBudgetMode !== undefined) { + config.review.packedToolBudgetMode = raw.review.packedToolBudgetMode; + sources["review.packedToolBudgetMode"] = source; + } if (raw.review?.maxBudgetTokens !== undefined) { config.review.maxBudgetTokens = raw.review.maxBudgetTokens; sources["review.maxBudgetTokens"] = source; @@ -365,6 +383,12 @@ function filterRepoConfig(raw: RawCodegenieConfig, warnings: ConfigWarning[]): R if (raw.review.budgetBoost !== undefined) { safe.review.budgetBoost = raw.review.budgetBoost; } + if (raw.review.packSameFileHunks !== undefined) { + safe.review.packSameFileHunks = raw.review.packSameFileHunks; + } + if (raw.review.packedToolBudgetMode !== undefined) { + safe.review.packedToolBudgetMode = raw.review.packedToolBudgetMode; + } if (raw.review.maxTime !== undefined) { safe.review.maxTime = raw.review.maxTime; } diff --git a/src/config/schema.ts b/src/config/schema.ts index d000099..7b639c7 100644 --- a/src/config/schema.ts +++ b/src/config/schema.ts @@ -6,6 +6,7 @@ export const reasoningLevelSchema = z.enum(["low", "medium", "high", "xhigh"]); export const severitySchema = z.enum(["critical", "high", "medium", "low"]); export const confidenceSchema = z.enum(["high", "medium", "low"]); export const logLevelSchema = z.enum(["debug", "info", "warn", "error"]); +export const packedToolBudgetModeSchema = z.enum(["base", "atom-scaled"]); const positiveIntSchema = z.number().int().positive(); const nonNegativeIntSchema = z.number().int().nonnegative(); @@ -51,6 +52,8 @@ export const rawConfigSchema = z maxTime: reviewMaxTimeMinutesSchema.optional(), perPassTimeoutMs: positiveIntSchema.optional(), budgetBoost: positiveFiniteNumberSchema.optional(), + packSameFileHunks: z.boolean().optional(), + packedToolBudgetMode: packedToolBudgetModeSchema.optional(), maxBudgetTokens: positiveIntSchema.optional(), maxModelCalls: positiveIntSchema.optional(), deepEnsemblePasses: positiveIntSchema.max(MAX_DEEP_ENSEMBLE_PASSES).optional(), @@ -138,6 +141,8 @@ export const codegenieConfigSchema = z maxTimeMs: positiveFiniteNumberSchema.max(MAX_REVIEW_TIME_MS), perPassTimeoutMs: positiveIntSchema, budgetBoost: positiveFiniteNumberSchema, + packSameFileHunks: z.boolean(), + packedToolBudgetMode: packedToolBudgetModeSchema, maxBudgetTokens: positiveIntSchema.optional(), maxModelCalls: positiveIntSchema.optional(), deepEnsemblePasses: positiveIntSchema.max(MAX_DEEP_ENSEMBLE_PASSES).optional(), @@ -209,6 +214,8 @@ export const defaultConfig: CodegenieConfig = { maxTimeMs: 30 * 60 * 1000, perPassTimeoutMs: 8 * 60 * 1000, budgetBoost: 1, + packSameFileHunks: false, + packedToolBudgetMode: "base", // Primary coverage budget (plan 90): work-denominated so provider latency // can never shrink a review. Re-derived 2026-07-04 after run 0c4d5213/53 // (5,921,791 tokens of legitimate work — planner-deep + escalator diff --git a/src/evals/eval-runner.ts b/src/evals/eval-runner.ts index 8470145..01b974c 100644 --- a/src/evals/eval-runner.ts +++ b/src/evals/eval-runner.ts @@ -6,6 +6,7 @@ import { z } from "zod"; import { applyRepoConfigLayer } from "../config/config-loader.js"; import { MAX_DEEP_ENSEMBLE_PASSES, + packedToolBudgetModeSchema, reasoningLevelSchema, reviewDepthSchema, reviewMaxTimeMinutesSchema, @@ -134,6 +135,8 @@ const caseSchema = z maxFindings: positiveIntSchema.optional(), concurrency: positiveIntSchema.optional(), budgetBoost: positiveNumberSchema.optional(), + packSameFileHunks: z.boolean().optional(), + packedToolBudgetMode: packedToolBudgetModeSchema.optional(), maxTimeMinutes: reviewMaxTimeMinutesSchema.optional(), maxBudgetTokens: positiveIntSchema.optional(), deepEnsemblePasses: positiveIntSchema.max(MAX_DEEP_ENSEMBLE_PASSES).optional(), @@ -813,6 +816,12 @@ function applyCaseReviewConfig( if (review?.budgetBoost !== undefined) { config.review.budgetBoost = review.budgetBoost; } + if (review?.packSameFileHunks !== undefined) { + config.review.packSameFileHunks = review.packSameFileHunks; + } + if (review?.packedToolBudgetMode !== undefined) { + config.review.packedToolBudgetMode = review.packedToolBudgetMode; + } if (review?.maxTimeMinutes !== undefined) { config.review.maxTimeMs = Math.round(review.maxTimeMinutes * 60_000); } @@ -866,6 +875,8 @@ function evalEffectiveConfig(config: CodegenieConfig): NonNullable; + requestedLensSignature: string; + standalonePacket: ReviewPacket; + protectedFocusNotes: string[]; +}; + +type PackedAtomGroup = { + group: HunkFirstGroup; + atoms: PacketAtom[]; + firstSourcePosition: number; + stableOrder: number; +}; + +type PacketBuildOverrides = { + profileFloor?: ReviewProfile; + atomCount?: number; +}; + +type BuiltPacket = { + packet: ReviewPacket; + derivedReviewProfile: ReviewProfile; + baseToolBudget: ToolBudget; + patchChars: number; +}; + type HunkRelationshipSource = "same_symbol" | "symbol_mention" | "planner_hint"; type HunkRelationshipStrength = "strong" | "medium" | "weak"; @@ -190,15 +226,108 @@ export async function buildReviewPackets( } const allowWholeFileContext = includedPlanned.length === planned.length; - for (const group of await groupHunks(includedPlanned, repoIndex, telemetry, { allowWholeFileContext })) { + const groups = await groupHunks(includedPlanned, repoIndex, telemetry, { allowWholeFileContext }); + if (!opts.config.review.packSameFileHunks || !groups.every(isHunkFirstGroup)) { + for (const group of groups) { + const first = group.hunks[0]; + if (!first) { + continue; + } + const includedDecisions = decisionsForGroup(group, effectiveByHunk); + const built = await buildPacket(group.hunks, includedDecisions, group, relationshipGraph, repoIndex, opts.config, telemetry, opts.reviewContext, symbolContextMetrics, packetBuildMetrics); + packets.push(built.packet); + } + continue; + } + + const sourcePositionByHunk = new Map(includedPlanned.map((entry, index) => [entry.hunk.id, index])); + const atoms: PacketAtom[] = []; + for (const group of groups) { const first = group.hunks[0]; if (!first) { continue; } - const groupDecisions = group.hunks.map((entry) => effectiveByHunk.get(entry.hunk.id)).filter((decision): decision is EffectiveDecision => decision !== undefined); - const includedDecisions = groupDecisions.filter(isNonSkipDecision); - const packet = await buildPacket(group.hunks, includedDecisions, group, relationshipGraph, repoIndex, opts.config, telemetry, opts.reviewContext, symbolContextMetrics, packetBuildMetrics); - packets.push(packet); + const includedDecisions = decisionsForGroup(group, effectiveByHunk); + const standalone = await buildPacket( + group.hunks, + includedDecisions, + group, + isolatedRelationshipGraph(relationshipGraph), + repoIndex, + opts.config, + probeTelemetry(telemetry), + opts.reviewContext, + emptySymbolContextMetrics(), + emptyPacketBuildMetrics() + ); + atoms.push(packetAtom(group, includedDecisions, standalone.packet, sourcePositionByHunk.get(first.hunk.id) ?? atoms.length)); + } + + const candidates = packCompatibleAtoms(atoms); + const accepted: PackedAtomGroup[] = []; + for (const candidate of candidates) { + if (candidate.atoms.length === 1) { + accepted.push(candidate); + continue; + } + const profileFloor = maxReviewProfile(candidate.atoms.map((atom) => atom.standalonePacket.reviewProfile)); + const includedDecisions = decisionsForGroup(candidate.group, effectiveByHunk); + const probe = await buildPacket( + candidate.group.hunks, + includedDecisions, + candidate.group, + isolatedRelationshipGraph(relationshipGraph), + repoIndex, + opts.config, + probeTelemetry(telemetry), + opts.reviewContext, + emptySymbolContextMetrics(), + emptyPacketBuildMetrics(), + { profileFloor, atomCount: candidate.atoms.length } + ); + const rejection = packingCandidateRejection(candidate.atoms, probe.packet); + if (rejection === undefined) { + accepted.push(candidate); + continue; + } + telemetry.event({ + stage: 6, + level: "info", + message: "same_file_atom_pack_rejected", + ...(candidate.group.hunks[0]?.file.path !== undefined ? { file: candidate.group.hunks[0].file.path } : {}), + data: { reason: rejection, atomIds: candidate.atoms.map((atom) => atom.atomId) } + }); + accepted.push(...candidate.atoms.map((atom, index) => ({ + group: atom.group, + atoms: [atom], + firstSourcePosition: atom.firstSourcePosition, + stableOrder: candidate.stableOrder + index / Math.max(1, candidate.atoms.length) + }))); + } + + accepted.sort((a, b) => a.firstSourcePosition - b.firstSourcePosition || a.stableOrder - b.stableOrder); + for (const packed of accepted) { + const profileFloor = maxReviewProfile(packed.atoms.map((atom) => atom.standalonePacket.reviewProfile)); + const includedDecisions = decisionsForGroup(packed.group, effectiveByHunk); + const built = await buildPacket( + packed.group.hunks, + includedDecisions, + packed.group, + relationshipGraph, + repoIndex, + opts.config, + telemetry, + opts.reviewContext, + symbolContextMetrics, + packetBuildMetrics, + { profileFloor, atomCount: packed.atoms.length } + ); + const rejection = packingCandidateRejection(packed.atoms, built.packet); + if (rejection !== undefined) { + throw new Error(`same-file packet packing invariant failed: ${rejection}`); + } + emitSameFileAtomsPacked(telemetry, packed.atoms, built, profileFloor, opts.config.review.packedToolBudgetMode); + packets.push(built.packet); } } @@ -271,6 +400,143 @@ async function groupHunks( return hunkFirstGroups(planned); } +function isHunkFirstGroup(group: PacketGroup): group is HunkFirstGroup { + return "origin" in group && group.origin === "hunk-first"; +} + +function decisionsForGroup( + group: PacketGroup, + effectiveByHunk: Map +): NonSkipDecision[] { + return group.hunks + .map((entry) => effectiveByHunk.get(entry.hunk.id)) + .filter((decision): decision is NonSkipDecision => decision !== undefined && isNonSkipDecision(decision)); +} + +function packetAtom( + group: HunkFirstGroup, + decisions: NonSkipDecision[], + standalonePacket: ReviewPacket, + firstSourcePosition: number +): PacketAtom { + const hunkIds = group.hunks.map((entry) => entry.hunk.id); + const protectedFocusNotes = dedupe(group.hunks.flatMap((entry, index) => { + if (entry.facts.reviewPriority !== "critical" && entry.facts.reviewPriority !== "high") { + return []; + } + return (decisions[index]?.focusNotes ?? []).map(normalizeNote); + })).filter((note) => standalonePacket.attentionNotes.includes(note)); + return { + origin: "hunk-first", + atomId: sha256Hex(`hunk-first\n${hunkIds.join("\n")}`), + group, + hunkCount: group.hunks.length, + patchChars: combinedPatchChars(group.hunks), + firstSourcePosition, + effectiveCoverage: maxCoverage(decisions.map((decision) => decision.coverage)), + requestedLensSignature: normalizedLensSignature(decisions), + standalonePacket, + protectedFocusNotes + }; +} + +function normalizedLensSignature(decisions: NonSkipDecision[]): string { + return JSON.stringify(cleanStrings(decisions.flatMap((decision) => decision.lenses))); +} + +function packCompatibleAtoms(atoms: PacketAtom[]): PackedAtomGroup[] { + const partitions = new Map(); + for (const atom of atoms) { + const key = `${atom.effectiveCoverage}\0${atom.requestedLensSignature}`; + const groups = partitions.get(key) ?? []; + const current = groups.at(-1); + if ( + current !== undefined && + current.reduce((sum, member) => sum + member.hunkCount, 0) + atom.hunkCount <= MAX_HUNKS_PER_PACKET && + current.reduce((sum, member) => sum + member.patchChars, 0) + atom.patchChars <= MAX_PATCH_CHARS + ) { + current.push(atom); + } else { + groups.push([atom]); + } + partitions.set(key, groups); + } + + let stableOrder = 0; + return [...partitions.values()] + .flatMap((groups) => groups.map((members): PackedAtomGroup => { + const degradationReason = cleanStrings(members.flatMap((atom) => atom.group.degradationReason ?? [])); + const hunks = members.flatMap((atom) => atom.group.hunks); + return { + group: packetGroup(hunks, degradationReason.length > 0 ? degradationReason.join("; ") : undefined), + atoms: members, + firstSourcePosition: members[0]?.firstSourcePosition ?? Number.MAX_SAFE_INTEGER, + stableOrder: stableOrder++ + }; + })) + .sort((a, b) => a.firstSourcePosition - b.firstSourcePosition || a.stableOrder - b.stableOrder); +} + +function packingCandidateRejection(atoms: PacketAtom[], packet: ReviewPacket): string | undefined { + const expectedHunkIds = atoms.flatMap((atom) => atom.group.hunks.map((entry) => entry.hunk.id)); + const actualHunkIds = packet.hunks.map((hunk) => hunk.hunkId); + if (JSON.stringify(actualHunkIds) !== JSON.stringify(expectedHunkIds)) { + return "source atom hunks were split, duplicated, or reordered"; + } + if ( + packet.hunks.length > MAX_HUNKS_PER_PACKET || + (atoms.length > 1 && atoms.reduce((sum, atom) => sum + atom.patchChars, 0) > MAX_PATCH_CHARS) + ) { + return "packet cap exceeded"; + } + if (atoms.some((atom) => atom.effectiveCoverage !== packet.coverage)) { + return "effective coverage promotion"; + } + const missingLenses = cleanStrings(atoms.flatMap((atom) => atom.standalonePacket.lenses)) + .filter((lens) => !packet.lenses.includes(lens)); + if (missingLenses.length > 0) { + return `standalone routed lenses omitted: ${missingLenses.join(", ")}`; + } + const missingFocus = dedupe(atoms.flatMap((atom) => atom.protectedFocusNotes)) + .filter((note) => !packet.attentionNotes.includes(note)); + if (missingFocus.length > 0) { + return `high-priority planner focus omitted: ${missingFocus.join(" | ")}`; + } + const profileFloor = maxReviewProfile(atoms.map((atom) => atom.standalonePacket.reviewProfile)); + if (reviewProfileRank(packet.reviewProfile) < reviewProfileRank(profileFloor)) { + return `effective profile ${packet.reviewProfile} is below ${profileFloor}`; + } + return undefined; +} + +function isolatedRelationshipGraph(graph: HunkRelationshipGraph): HunkRelationshipGraph { + return { + ...graph, + relatedContextAttached: [], + relatedContextOmitted: [] + }; +} + +function emptyPacketBuildMetrics(): PacketBuildMetrics { + return { + relatedContextBudgetNudges: 0, + relatedContextBudgetNudgeSources: new Set() + }; +} + +function probeTelemetry(telemetry: TelemetryRecorder): TelemetryRecorder { + return { + runId: telemetry.runId, + runDir: telemetry.runDir, + event: () => undefined, + recordModelCall: () => undefined, + recordToolCall: () => "stage6-packet-probe", + writeArtifact: async () => undefined, + writeDebug: async () => undefined, + flush: async () => undefined + }; +} + async function buildPacket( planned: PlannedHunk[], decisions: NonSkipDecision[], @@ -281,8 +547,9 @@ async function buildPacket( telemetry: TelemetryRecorder, reviewContext: PacketReviewContext | undefined, symbolContextMetrics: SymbolContextMetrics, - packetBuildMetrics: PacketBuildMetrics -): Promise { + packetBuildMetrics: PacketBuildMetrics, + overrides: PacketBuildOverrides = {} +): Promise { const first = planned[0]; if (!first) { throw new Error("cannot build empty packet"); @@ -353,7 +620,7 @@ async function buildPacket( hasRelatedChangedContext, strongRelatedContext: false }); - const reviewProfile = packetReviewProfile({ + const derivedReviewProfile = packetReviewProfile({ coverage, reviewPriority, planned, @@ -362,7 +629,8 @@ async function buildPacket( hasRelatedChangedContext, strongRelatedContext }); - if (strongRelatedContext && baseReviewProfile !== "investigate" && reviewProfile === "investigate") { + const reviewProfile = maxReviewProfile([derivedReviewProfile, overrides.profileFloor]); + if (strongRelatedContext && baseReviewProfile !== "investigate" && derivedReviewProfile === "investigate") { const relationshipSources = cleanStrings(relatedChangedContext.flatMap((context) => context.relationshipSource ?? [])); packetBuildMetrics.relatedContextBudgetNudges += 1; for (const source of relationshipSources) { @@ -376,7 +644,7 @@ async function buildPacket( data: { coverage, baseReviewProfile, - reviewProfile, + reviewProfile: derivedReviewProfile, relatedContextCount: relatedChangedContext.length, relationshipSources } @@ -396,6 +664,7 @@ async function buildPacket( telemetry }); emitPacketContextQuality(telemetry, first.file.path, coverage, reviewPriority, contextQuality, contextDegradationReasons); + const baseToolBudget = toolBudget(coverage, config.review.depth, reviewProfile); const packet: ReviewPacket = { id: sha256Hex(`${first.file.path}\n${[...hunkIds].sort().join("\n")}\n${kind}`), dispatchRank: packetDispatchRank(first.file.path, first.facts, packetChangedLines), @@ -423,7 +692,10 @@ async function buildPacket( labels: first.facts.labels, attentionNotes, relatedChangedContext, - toolBudget: scaleToolBudget(toolBudget(coverage, config.review.depth, reviewProfile), config.review.budgetBoost), + toolBudget: scaleToolBudget( + packedToolBudget(baseToolBudget, reviewProfile, overrides.atomCount ?? 1, config.review.packedToolBudgetMode), + config.review.budgetBoost + ), ...(reviewContext?.intentText !== undefined ? { intentText: reviewContext.intentText } : {}), ...(reviewContext?.intentSignals !== undefined ? { intentSignals: reviewContext.intentSignals } : {}), ...(context.degradation !== undefined || truncationReason.length > 0 || contextDropReason !== undefined || contextTruncationReason !== undefined || group.degradationReason !== undefined @@ -435,7 +707,45 @@ async function buildPacket( ? { fileContext: { mode: "file-diff", reason: "grouped file hunks" } } : {}) }; - return packet; + return { packet, derivedReviewProfile, baseToolBudget, patchChars }; +} + +function emitSameFileAtomsPacked( + telemetry: TelemetryRecorder, + atoms: PacketAtom[], + built: BuiltPacket, + profileFloor: ReviewProfile, + toolBudgetMode: CodegenieConfig["review"]["packedToolBudgetMode"] +): void { + telemetry.event({ + stage: 6, + level: "info", + message: "same_file_atoms_packed", + file: built.packet.path, + data: { + packetId: built.packet.id, + atomIds: atoms.map((atom) => atom.atomId), + standaloneProfiles: atoms.map((atom) => atom.standalonePacket.reviewProfile), + sourceAtomCount: atoms.length, + hunkCount: built.packet.hunks.length, + effectiveCoverage: built.packet.coverage, + requestedLensSignature: atoms[0]?.requestedLensSignature ?? "[]", + capUsage: { + hunks: built.packet.hunks.length, + maxHunks: MAX_HUNKS_PER_PACKET, + patchChars: built.patchChars, + maxPatchChars: MAX_PATCH_CHARS + }, + derivedPackedProfile: built.derivedReviewProfile, + profileFloor, + effectiveProfile: built.packet.reviewProfile, + profileFloorApplied: reviewProfileRank(built.derivedReviewProfile) < reviewProfileRank(profileFloor), + plannerLensesPreserved: true, + toolBudgetMode, + baseToolBudget: built.baseToolBudget, + effectiveToolBudget: built.packet.toolBudget + } + }); } const DOCS_CONFIG_EXTENSIONS = new Set([".md", ".yml", ".yaml", ".toml", ".conf", ".sample", ".txt"]); @@ -1179,8 +1489,8 @@ function dedupe(values: T[]): T[] { return [...new Set(values)]; } -function hunkFirstGroups(planned: PlannedHunk[], degradationReason?: string): PacketGroup[] { - const groups: PacketGroup[] = []; +function hunkFirstGroups(planned: PlannedHunk[], degradationReason?: string): HunkFirstGroup[] { + const groups: HunkFirstGroup[] = []; let current: PlannedHunk[] = []; for (const entry of planned) { @@ -1266,7 +1576,7 @@ function lineInRange(line: number, start: number, lines: number): boolean { return line >= start && line <= start + lines - 1; } -function packetGroup(hunks: PlannedHunk[], degradationReason?: string): PacketGroup { +function packetGroup(hunks: PlannedHunk[], degradationReason?: string): HunkFirstGroup { const first = hunks[0]; const kind: ReviewPacket["kind"] = first && hunks.length > 1 && hunks.length === first.file.hunks.length @@ -1274,7 +1584,7 @@ function packetGroup(hunks: PlannedHunk[], degradationReason?: string): PacketGr : hunks.length > 1 ? "coalesced-hunks" : "hunk"; - return { hunks, kind, ...(degradationReason !== undefined ? { degradationReason } : {}) }; + return { origin: "hunk-first", hunks, kind, ...(degradationReason !== undefined ? { degradationReason } : {}) }; } function packetKind(group: PacketGroup, planned: PlannedHunk[], file: DiffFile): ReviewPacket["kind"] { @@ -2948,6 +3258,39 @@ export function toolBudget(coverage: Exclude, depth: Code }; } +function packedToolBudget( + base: ToolBudget, + profile: ReviewProfile, + atomCount: number, + mode: CodegenieConfig["review"]["packedToolBudgetMode"] +): ToolBudget { + if (mode === "base" || atomCount <= 1 || profile === "simple") { + return base; + } + const additionalAtoms = atomCount - 1; + return { + ...base, + maxToolCalls: Math.min(base.maxToolCalls + additionalAtoms, Math.ceil(1.75 * base.maxToolCalls)), + maxResultChars: Math.min(base.maxResultChars + additionalAtoms * 2_000, Math.ceil(1.75 * base.maxResultChars)) + }; +} + +const REVIEW_PROFILE_RANK: Record = { + simple: 0, + standard: 1, + investigate: 2 +}; + +function reviewProfileRank(profile: ReviewProfile): number { + return REVIEW_PROFILE_RANK[profile]; +} + +function maxReviewProfile(profiles: Array): ReviewProfile { + return profiles + .filter((profile): profile is ReviewProfile => profile !== undefined) + .reduce((highest, profile) => reviewProfileRank(profile) > reviewProfileRank(highest) ? profile : highest, "simple"); +} + function packetReviewProfile(input: { coverage: Exclude; reviewPriority: ReviewPriority; diff --git a/src/types.ts b/src/types.ts index 2666377..a21679d 100644 --- a/src/types.ts +++ b/src/types.ts @@ -8,6 +8,7 @@ export type ProcessingMode = "per-hunk" | "whole-file" | "skip"; export type ReviewPriority = "critical" | "high" | "normal" | "low"; export type OutputFormat = "markdown" | "json"; export type LogLevel = "debug" | "info" | "warn" | "error"; +export type PackedToolBudgetMode = "base" | "atom-scaled"; export type ConfigSource = | "defaults" @@ -44,6 +45,8 @@ export type CodegenieConfig = { maxTimeMs: number; perPassTimeoutMs: number; budgetBoost: number; + packSameFileHunks: boolean; + packedToolBudgetMode: PackedToolBudgetMode; maxBudgetTokens?: number; maxModelCalls?: number; // Plan 84: K independent Stage-7 passes for deep-coverage packets, union @@ -1008,6 +1011,8 @@ export type EvalCase = { maxFindings?: number; concurrency?: number; budgetBoost?: number; + packSameFileHunks?: boolean; + packedToolBudgetMode?: PackedToolBudgetMode; maxTimeMinutes?: number; maxBudgetTokens?: number; deepEnsemblePasses?: number; @@ -1261,6 +1266,8 @@ export type EvalRunInfo = { review: { concurrency: number; timeoutMs: number; + packSameFileHunks: boolean; + packedToolBudgetMode: PackedToolBudgetMode; maxBudgetTokens?: number; }; llm: { diff --git a/tests/config-loader.test.ts b/tests/config-loader.test.ts index 89e20b6..3895121 100644 --- a/tests/config-loader.test.ts +++ b/tests/config-loader.test.ts @@ -14,6 +14,42 @@ import { loadProviderSettings, saveProviderSettings } from "../src/provider/prov import { CodegenieError } from "../src/util/errors.js"; describe("config loader", () => { + it("loads dark packet-packing defaults and tracks safe user and repository overrides", () => { + const repoRoot = tempDir(); + const home = tempDir(); + + const defaults = loadConfig({ repoRoot, homeOverride: home }); + expect(defaults.config.review.packSameFileHunks).toBe(false); + expect(defaults.config.review.packedToolBudgetMode).toBe("base"); + expect(defaults.sources["review.packSameFileHunks"]).toBe("defaults"); + expect(defaults.sources["review.packedToolBudgetMode"]).toBe("defaults"); + + writeFileSync(path.join(home, "config.toml"), [ + "[review]", + "packSameFileHunks = true", + 'packedToolBudgetMode = "atom-scaled"' + ].join("\n")); + const userConfigured = loadConfig({ repoRoot, homeOverride: home }); + expect(userConfigured.config.review.packSameFileHunks).toBe(true); + expect(userConfigured.config.review.packedToolBudgetMode).toBe("atom-scaled"); + expect(userConfigured.sources["review.packSameFileHunks"]).toBe("user-config"); + expect(userConfigured.sources["review.packedToolBudgetMode"]).toBe("user-config"); + + writeFileSync(path.join(repoRoot, "codegenie.toml"), [ + "[review]", + "packSameFileHunks = false", + 'packedToolBudgetMode = "base"' + ].join("\n")); + const repoConfigured = loadConfig({ repoRoot, homeOverride: home }); + expect(repoConfigured.config.review.packSameFileHunks).toBe(false); + expect(repoConfigured.config.review.packedToolBudgetMode).toBe("base"); + expect(repoConfigured.sources["review.packSameFileHunks"]).toBe("repo-config"); + expect(repoConfigured.sources["review.packedToolBudgetMode"]).toBe("repo-config"); + expect(repoConfigured.warnings).toEqual([]); + + expect(rawConfigSchema.safeParse({ review: { packedToolBudgetMode: "linear" } }).success).toBe(false); + }); + it("resolves maxTime minutes from defaults, user config, repo config, and CLI in precedence order", () => { const repoRoot = tempDir(); const home = tempDir(); diff --git a/tests/evals.test.ts b/tests/evals.test.ts index 64cdd97..a5e7c33 100644 --- a/tests/evals.test.ts +++ b/tests/evals.test.ts @@ -126,6 +126,34 @@ describe("eval suite validation", () => { }); }); + it("strictly parses packet-packing eval overrides", async () => { + const suiteDir = mkdtempSync(path.join(tmpdir(), "codegenie-eval-packet-packing-")); + writeFileSync(path.join(suiteDir, "packing.yml"), [ + "name: packet-packing", + "artifacts:", + " path: logs/1", + "review:", + " packSameFileHunks: true", + " packedToolBudgetMode: atom-scaled" + ].join("\n")); + + const suite = await loadEvalSuite(suiteDir); + expect(suite.cases[0]?.evalCase.review).toMatchObject({ + packSameFileHunks: true, + packedToolBudgetMode: "atom-scaled" + }); + + const invalidDir = mkdtempSync(path.join(tmpdir(), "codegenie-eval-packet-packing-invalid-")); + writeFileSync(path.join(invalidDir, "packing.yml"), [ + "name: packet-packing-invalid", + "artifacts:", + " path: logs/1", + "review:", + " packedToolBudgetMode: linear" + ].join("\n")); + await expect(loadEvalSuite(invalidDir)).rejects.toMatchObject({ code: "config_error" }); + }); + it("accepts pinned head/base eval commands", async () => { const suiteDir = mkdtempSync(path.join(tmpdir(), "codegenie-eval-head-base-")); writeFileSync(path.join(suiteDir, "head.yml"), [ @@ -1725,6 +1753,8 @@ describe("eval command fixture suite", () => { " model: not-real-model", " reasoning: low", " concurrency: 3", + " packSameFileHunks: true", + " packedToolBudgetMode: atom-scaled", " lenses:", " - core/code-review", "llm:", @@ -1748,7 +1778,11 @@ describe("eval command fixture suite", () => { expect(result.status).toBe("pass"); expect(result.info.effectiveConfig).toMatchObject({ - review: { concurrency: 3 }, + review: { + concurrency: 3, + packSameFileHunks: true, + packedToolBudgetMode: "atom-scaled" + }, llm: { provider: "fake", model: "fake-model", reasoning: "high", maxConcurrentCalls: 2 } }); expect(result.info.codegenieRuntime).toMatchObject({ diff --git a/tests/pipeline-phase5.test.ts b/tests/pipeline-phase5.test.ts index 51e457e..df06f39 100644 --- a/tests/pipeline-phase5.test.ts +++ b/tests/pipeline-phase5.test.ts @@ -3980,6 +3980,617 @@ describe("phase 5 pipeline regressions", () => { }); }); + it("keeps packet atom flag-off parity and bypasses whole-file groups", async () => { + const file = fakeMultiHunkFile([ + { id: "h1", newStart: 1, content: "export const one = 1;" }, + { id: "h2", newStart: 100, content: "export const two = 2;" } + ]); + const facts = { ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }; + const baselineArtifacts = new Map(); + const baseline = await buildReviewPackets( + fakePlanForHunks(["h1", "h2"]), + [file], + [facts], + fakeRepositoryIndex(), + { + ...nullTelemetry(), + writeArtifact: async (name, data) => { + baselineArtifacts.set(name, data); + } + }, + { config: config(), enabledLenses: ["core/code-review"] } + ); + const darkArtifacts = new Map(); + const dark = await buildReviewPackets( + fakePlanForHunks(["h1", "h2"]), + [file], + [facts], + fakeRepositoryIndex(), + { + ...nullTelemetry(), + writeArtifact: async (name, data) => { + darkArtifacts.set(name, data); + } + }, + { + config: { + ...config(), + review: { ...config().review, packSameFileHunks: false, packedToolBudgetMode: "atom-scaled" } + }, + enabledLenses: ["core/code-review"] + } + ); + + expect(dark).toEqual(baseline); + expect(darkArtifacts).toEqual(baselineArtifacts); + + const events: Array> = []; + const wholeFilePackets = await buildReviewPackets( + fakePlan(), + [fakeDiffFile("app.ts")], + [fakeFacts("app.ts", "whole-file")], + fakeRepositoryIndex(fakeTools("export const value = 1;\n")), + { ...nullTelemetry(), event: (event) => events.push(event) }, + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review"] + } + ); + expect(wholeFilePackets[0]?.kind).toBe("whole-file"); + expect(events.some((event) => event.message === "same_file_atoms_packed")).toBe(false); + + const probedFileDiff = await buildReviewPackets( + fakePlan(), + [fakeDiffFile("app.ts")], + [fakeFacts("app.ts", "whole-file")], + fakeRepositoryIndex(fakeTools("x".repeat(9_000))), + { ...nullTelemetry(), event: (event) => events.push(event) }, + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review"] + } + ); + expect(probedFileDiff[0]?.kind).toBe("file-diff"); + expect(events.some((event) => event.message === "same_file_atoms_packed")).toBe(false); + }); + + it("applies stable same-file packing with source-order, cap, telemetry, and dispatch-rank invariants", async () => { + const file = fakeMultiHunkFile(Array.from({ length: 6 }, (_, index) => ({ + id: `h${String(index + 1)}`, + newStart: index * 100 + 1, + content: `export const value${String(index + 1)} = ${String(index + 1)};` + }))); + const facts = { ...fakeFacts("app.ts", "per-hunk"), hunkCount: 6, changedLines: 6 }; + const events: Array> = []; + const packets = await buildReviewPackets( + fakePlanForHunks(["h1", "h2", "h3", "h4", "h5", "h6"]), + [file], + [facts], + fakeRepositoryIndex(), + { ...nullTelemetry(), event: (event) => events.push(event) }, + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review"] + } + ); + + expect(packets.map((packet) => packet.hunks.map((hunk) => hunk.hunkId))).toEqual([ + ["h1", "h2", "h3", "h4", "h5"], + ["h6"] + ]); + expect(packets.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId))).toEqual(["h1", "h2", "h3", "h4", "h5", "h6"]); + expect(packets[0]?.dispatchRank).toEqual(packetDispatchRank("app.ts", facts, 5)); + const packedEvents = events.filter((event) => event.message === "same_file_atoms_packed"); + expect(packedEvents).toHaveLength(2); + expect(packedEvents.map((event) => event.data)).toEqual([ + expect.objectContaining({ + atomIds: ["h1", "h2", "h3", "h4", "h5"].map((hunkId) => sha256Hex(`hunk-first\n${hunkId}`)), + sourceAtomCount: 5, + hunkCount: 5, + effectiveCoverage: "normal", + requestedLensSignature: '["core/code-review"]', + standaloneProfiles: ["standard", "standard", "standard", "standard", "standard"], + plannerLensesPreserved: true, + capUsage: expect.objectContaining({ hunks: 5, maxHunks: 5, maxPatchChars: 12_000 }) + }), + expect.objectContaining({ + atomIds: [sha256Hex("hunk-first\nh6")], + sourceAtomCount: 1, + hunkCount: 1 + }) + ]); + + const nearbyEvents: Array> = []; + await buildReviewPackets( + fakePlanForHunks(["h1", "h2"]), + [fakeMultiHunkFile([ + { id: "h1", newStart: 1, content: "export const one = 1;" }, + { id: "h2", newStart: 10, content: "export const two = 2;" } + ])], + [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }], + fakeRepositoryIndex(), + { ...nullTelemetry(), event: (event) => nearbyEvents.push(event) }, + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review"] + } + ); + const nearbyAtomIds = nearbyEvents.find((event) => event.message === "same_file_atoms_packed")?.data?.atomIds; + expect(nearbyAtomIds).toEqual([sha256Hex("hunk-first\nh1\nh2")]); + expect(nearbyAtomIds).not.toEqual([sha256Hex("hunk-first\nh1"), sha256Hex("hunk-first\nh2")]); + expect(nearbyAtomIds).not.toEqual([sha256Hex("hunk-first\nh2\nh1")]); + }); + + it("partitions same-file packing by coverage and lens signature before restoring packet order", async () => { + const ids = ["h1", "h2", "h3", "h4", "h5", "h6"]; + const file = fakeMultiHunkFile(ids.map((id, index) => ({ + id, + newStart: index * 100 + 1, + content: `export function value${String(index)}() { return ${String(index)}; }` + }))); + const plan = fakePlanForHunks(ids); + plan.coverage = plan.coverage.map((decision, index) => ({ + ...decision, + coverage: index === 1 || index === 3 ? "light" : "normal", + lenses: index >= 4 ? ["core/security"] : ["core/code-review"] + })); + const packets = await buildReviewPackets( + plan, + [file], + [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 6, changedLines: 6 }], + fakeRepositoryIndex(), + nullTelemetry(), + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review", "core/security"] + } + ); + + expect(packets.map((packet) => ({ + coverage: packet.coverage, + lenses: packet.lenses, + hunks: packet.hunks.map((hunk) => hunk.hunkId) + }))).toEqual([ + { coverage: "normal", lenses: ["core/code-review"], hunks: ["h1", "h3"] }, + { coverage: "light", lenses: ["core/code-review"], hunks: ["h2", "h4"] }, + { coverage: "normal", lenses: ["core/security"], hunks: ["h5", "h6"] } + ]); + }); + + it("leaves same-file atoms separate when packing would omit a standalone routed lens", async () => { + const ids = ["h1", "h2", "h3", "h4"]; + const file = fakeMultiHunkFile(ids.map((id, index) => ({ + id, + newStart: index * 100 + 1, + content: `export function value${String(index)}() { return ${String(index)}; }` + }))); + const plan = fakePlanForHunks(ids); + plan.coverage = plan.coverage.map((decision, index) => ({ + ...decision, + lenses: ["core/code-review", "core/tests"], + reason: "default_coverage", + focusNotes: [index === 3 ? "Check regression tests for this branch." : `Inspect behavior ${String(index + 1)}.`] + })); + const events: Array> = []; + const packets = await buildReviewPackets( + plan, + [file], + [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 4, changedLines: 4 }], + fakeRepositoryIndex(), + { ...nullTelemetry(), event: (event) => events.push(event) }, + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review", "core/tests"] + } + ); + + expect(packets.map((packet) => packet.hunks.map((hunk) => hunk.hunkId))).toEqual([ + ["h1"], + ["h2"], + ["h3"], + ["h4"] + ]); + expect(packets.map((packet) => packet.lenses)).toEqual([ + ["core/code-review"], + ["core/code-review"], + ["core/code-review"], + ["core/code-review", "core/tests"] + ]); + expect(events).toContainEqual(expect.objectContaining({ + stage: 6, + message: "same_file_atom_pack_rejected", + data: expect.objectContaining({ reason: "standalone routed lenses omitted: core/tests" }) + })); + }); + + it("preserves mixed-coverage baseline atoms and scales by newly combined atom count", async () => { + const file = fakeMultiHunkFile([ + { id: "h1", newStart: 1, content: "export function one() { return 1; }" }, + { id: "h2", newStart: 10, content: "export function two() { return 2; }" }, + { id: "h3", newStart: 100, content: "export function three() { return 3; }" } + ]); + const plan = fakePlanForHunks(["h1", "h2", "h3"]); + plan.coverage[1] = { ...plan.coverage[1]!, coverage: "deep" }; + plan.coverage[2] = { ...plan.coverage[2]!, coverage: "deep" }; + const facts = { ...fakeFacts("app.ts", "per-hunk"), hunkCount: 3, changedLines: 3 }; + const off = await buildReviewPackets( + plan, + [file], + [facts], + fakeRepositoryIndex(), + nullTelemetry(), + { config: config(), enabledLenses: ["core/code-review"] } + ); + expect(off.map((packet) => packet.hunks.map((hunk) => hunk.hunkId))).toEqual([["h1", "h2"], ["h3"]]); + expect(off.map((packet) => packet.coverage)).toEqual(["deep", "deep"]); + + const events: Array> = []; + const packed = await buildReviewPackets( + plan, + [file], + [facts], + fakeRepositoryIndex(), + { ...nullTelemetry(), event: (event) => events.push(event) }, + { + config: { + ...config(), + review: { ...config().review, packSameFileHunks: true, packedToolBudgetMode: "atom-scaled" } + }, + enabledLenses: ["core/code-review"] + } + ); + expect(packed).toHaveLength(1); + expect(packed[0]?.hunks.map((hunk) => hunk.hunkId)).toEqual(["h1", "h2", "h3"]); + expect(packed[0]?.coverage).toBe("deep"); + expect(packed[0]?.toolBudget.maxToolCalls).toBe(16); + expect(events).toContainEqual(expect.objectContaining({ + message: "same_file_atoms_packed", + data: expect.objectContaining({ sourceAtomCount: 2, hunkCount: 3, effectiveCoverage: "deep" }) + })); + }); + + it("leaves same-file atoms separate when packing would omit high-priority planner attention", async () => { + const ids = ["h1", "h2", "h3", "h4"]; + const file = fakeMultiHunkFile(ids.map((id, index) => ({ + id, + newStart: index * 100 + 1, + content: `export function value${String(index)}() { return ${String(index)}; }` + }))); + const plan = fakePlanForHunks(ids); + plan.coverage = plan.coverage.map((decision, index) => ({ + ...decision, + focusNotes: [`Protect focus ${String(index + 1)}.`] + })); + const events: Array> = []; + const packets = await buildReviewPackets( + plan, + [file], + [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 4, changedLines: 4, reviewPriority: "high" }], + fakeRepositoryIndex(), + { ...nullTelemetry(), event: (event) => events.push(event) }, + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review"] + } + ); + + expect(packets).toHaveLength(4); + expect(packets.map((packet) => packet.attentionNotes)).toEqual([ + expect.arrayContaining(["Protect focus 1."]), + expect.arrayContaining(["Protect focus 2."]), + expect.arrayContaining(["Protect focus 3."]), + expect.arrayContaining(["Protect focus 4."]) + ]); + expect(events).toContainEqual(expect.objectContaining({ + stage: 6, + message: "same_file_atom_pack_rejected", + data: expect.objectContaining({ reason: expect.stringContaining("high-priority planner focus omitted") }) + })); + }); + + it("merges degradation reasons deterministically while respecting the real patch cap", async () => { + const file = fakeMultiHunkFile([ + { id: "h1", newStart: 1, content: "a".repeat(5_000) }, + { id: "h2", newStart: 100, content: "b".repeat(5_000) }, + { id: "h3", newStart: 200, content: "c".repeat(5_000) } + ]); + const packets = await buildReviewPackets( + fakePlanForHunks(["h1", "h2", "h3"]), + [file], + [{ ...fakeFacts("app.ts", "whole-file"), hunkCount: 3, changedLines: 3 }], + fakeRepositoryIndex(), + nullTelemetry(), + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review"] + } + ); + + expect(packets.map((packet) => packet.hunks.length)).toEqual([2, 1]); + expect(packets[0]?.degraded?.reason.match(/whole-file downgraded/gu)).toHaveLength(1); + expect(packets[0]?.hunks.reduce((sum, hunk) => sum + hunk.contentWithLineNumbers.length, 0)).toBeLessThanOrEqual(12_000); + + const oversizedAtom = await buildReviewPackets( + fakePlan(), + [fakeDiffFile("app.ts", "x".repeat(20_000))], + [fakeFacts("app.ts", "per-hunk")], + fakeRepositoryIndex(), + nullTelemetry(), + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review"] + } + ); + expect(oversizedAtom).toHaveLength(1); + expect(oversizedAtom[0]?.hunks[0]).toMatchObject({ hunkId: "h1", truncated: true }); + }); + + it("applies a planner-hint profile floor when same-file packing absorbs strong related context", async () => { + const file = fakeMultiHunkFile([ + { id: "h1", newStart: 1, content: "export function alpha() { return beta(); }" }, + { id: "h2", newStart: 100, content: "export function beta() { return 2; }" } + ]); + const symbolFacts: HunkSymbolFacts[] = [ + packetPackingSymbolFact("h1", "alpha", [1, 3]), + packetPackingSymbolFact("h2", "beta", [100, 102]) + ]; + const plan = fakePlanForHunks(["h1", "h2"]); + plan.coverage[0] = { ...plan.coverage[0]!, reason: "default_coverage", relatedSymbols: ["beta"] }; + const off = await buildReviewPackets( + plan, + [file], + [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }], + { ...fakeRepositoryIndex(), symbolFacts }, + nullTelemetry(), + { config: config(), enabledLenses: ["core/code-review"] } + ); + expect(off.map((packet) => packet.reviewProfile)).toEqual(["investigate", "standard"]); + + const events: Array> = []; + const packed = await buildReviewPackets( + plan, + [file], + [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }], + { ...fakeRepositoryIndex(), symbolFacts }, + { ...nullTelemetry(), event: (event) => events.push(event) }, + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review"] + } + ); + expect(packed).toHaveLength(1); + expect(packed[0]).toMatchObject({ + reviewProfile: "investigate", + relatedChangedContext: [], + toolBudget: { maxToolCalls: 6, maxInvestigationRounds: 2, maxResultChars: 12_000 } + }); + expect(events).toContainEqual(expect.objectContaining({ + message: "same_file_atoms_packed", + data: expect.objectContaining({ + standaloneProfiles: ["investigate", "standard"], + derivedPackedProfile: "standard", + profileFloor: "investigate", + effectiveProfile: "investigate", + profileFloorApplied: true + }) + })); + }); + + it("applies an ordinary-context profile floor instead of demoting a packed light packet to simple", async () => { + const file = fakeMultiHunkFile([ + { id: "h1", newStart: 1, content: "import { one } from './one';" }, + { id: "h2", newStart: 100, content: "import { two } from './two';" } + ]); + const plan = fakePlanForHunks(["h1", "h2"]); + plan.coverage = plan.coverage.map((decision) => ({ ...decision, coverage: "light" })); + plan.coverage[0] = { ...plan.coverage[0]!, reason: "default_coverage", relatedFiles: ["app.ts"] }; + const events: Array> = []; + const packets = await buildReviewPackets( + plan, + [file], + [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }], + fakeRepositoryIndex(), + { ...nullTelemetry(), event: (event) => events.push(event) }, + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review"] + } + ); + + expect(packets).toHaveLength(1); + expect(packets[0]).toMatchObject({ reviewProfile: "standard", toolBudget: { maxToolCalls: 1 } }); + expect(events).toContainEqual(expect.objectContaining({ + message: "same_file_atoms_packed", + data: expect.objectContaining({ + standaloneProfiles: ["standard", "simple"], + derivedPackedProfile: "simple", + profileFloor: "standard", + effectiveProfile: "standard", + profileFloorApplied: true + }) + })); + }); + + it("applies a symbol-mention profile floor and does not invent one for same-symbol atoms", async () => { + const file = fakeMultiHunkFile([ + { id: "h-helper", newStart: 1, content: "export function scaleAmount(value) { return value / 10; }" }, + { id: "h-caller", newStart: 100, content: "export function quote(value) { return scaleAmount(value); }" } + ]); + const symbolFacts: HunkSymbolFacts[] = [ + packetPackingSymbolFact("h-helper", "scaleAmount", [1, 3]), + packetPackingSymbolFact("h-caller", "quote", [100, 102]) + ]; + const meta = { backend: "tree-sitter" as const, precision: "syntactic" as const, degraded: false }; + const tools = { + ...fakeTools(), + findSymbolMentions: async (symbolName: string, options: SymbolMentionOptions = {}) => ({ + results: symbolName === "scaleAmount" && options.contextMode === "symbols" + ? [{ + path: "app.ts", + line: 101, + matchText: "return scaleAmount(value);", + enclosingSymbol: { path: "app.ts", name: "quote", kind: "function" as const, lineRange: [100, 102] as [number, number] } + }] + : [], + meta + }) + }; + const events: Array> = []; + const packets = await buildReviewPackets( + fakePlanForHunks(["h-helper", "h-caller"]), + [file], + [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }], + { ...fakeRepositoryIndex(tools), symbolFacts }, + { ...nullTelemetry(), event: (event) => events.push(event) }, + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review"] + } + ); + expect(packets).toHaveLength(1); + expect(packets[0]?.reviewProfile).toBe("investigate"); + expect(events).toContainEqual(expect.objectContaining({ + message: "same_file_atoms_packed", + data: expect.objectContaining({ derivedPackedProfile: "standard", profileFloor: "investigate", profileFloorApplied: true }) + })); + + const sameSymbolEvents: Array> = []; + const sameSymbolFacts = [ + packetPackingSymbolFact("h-helper", "shared", [1, 200]), + packetPackingSymbolFact("h-caller", "shared", [1, 200]) + ]; + const sameSymbolPackets = await buildReviewPackets( + fakePlanForHunks(["h-helper", "h-caller"]), + [file], + [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }], + { ...fakeRepositoryIndex(), symbolFacts: sameSymbolFacts }, + { ...nullTelemetry(), event: (event) => sameSymbolEvents.push(event) }, + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review"] + } + ); + expect(sameSymbolPackets).toHaveLength(1); + expect(sameSymbolPackets[0]?.reviewProfile).toBe("standard"); + expect(sameSymbolEvents).toContainEqual(expect.objectContaining({ + message: "same_file_atoms_packed", + data: expect.objectContaining({ sourceAtomCount: 1, standaloneProfiles: ["standard"], profileFloorApplied: false }) + })); + }); + + it("preserves the standalone profile in the same-name primarySymbols packing edge case", async () => { + const file = fakeMultiHunkFile([ + { id: "h-x", newStart: 1, content: "export function source() { return shared(); }" }, + { id: "h-y", newStart: 100, content: "export function shared() { return 1; }" }, + { id: "h-z", newStart: 200, content: "export function shared() { return 2; }" } + ]); + const symbolFacts = [ + packetPackingSymbolFact("h-x", "source", [1, 3]), + packetPackingSymbolFact("h-y", "shared", [100, 102]), + packetPackingSymbolFact("h-z", "shared", [200, 202]) + ]; + const plan = fakePlanForHunks(["h-x", "h-y", "h-z"]); + plan.coverage[0] = { ...plan.coverage[0]!, reason: "default_coverage", relatedSymbols: ["shared"] }; + plan.coverage[2] = { ...plan.coverage[2]!, coverage: "deep" }; + const events: Array> = []; + const packets = await buildReviewPackets( + plan, + [file], + [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 3, changedLines: 3 }], + { ...fakeRepositoryIndex(), symbolFacts }, + { ...nullTelemetry(), event: (event) => events.push(event) }, + { + config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, + enabledLenses: ["core/code-review"] + } + ); + const packed = packets.find((packet) => packet.hunks.some((hunk) => hunk.hunkId === "h-x")); + expect(packed?.hunks.map((hunk) => hunk.hunkId)).toEqual(["h-x", "h-y"]); + expect(packed?.reviewProfile).toBe("investigate"); + expect(events).toContainEqual(expect.objectContaining({ + message: "same_file_atoms_packed", + data: expect.objectContaining({ + packetId: packed?.id, + standaloneProfiles: ["investigate", "standard"], + derivedPackedProfile: "standard", + profileFloorApplied: true + }) + })); + }); + + it("scales packed tool budgets by additional atoms while preserving base, simple, rounds, and boost order", async () => { + const ids = ["h1", "h2", "h3", "h4", "h5"]; + const distantFile = fakeMultiHunkFile(ids.map((id, index) => ({ + id, + newStart: index * 100 + 1, + content: `export function value${String(index)}() { return ${String(index)}; }` + }))); + const facts = { ...fakeFacts("app.ts", "per-hunk"), hunkCount: 5, changedLines: 5 }; + const build = (review: Partial, plan = fakePlanForHunks(ids), file = distantFile, fileFacts = facts) => + buildReviewPackets( + plan, + [file], + [fileFacts], + fakeRepositoryIndex(), + nullTelemetry(), + { + config: { + ...config(), + review: { ...config().review, packSameFileHunks: true, packedToolBudgetMode: "atom-scaled", ...review } + }, + enabledLenses: ["core/code-review"] + } + ); + + const base = await build({ packedToolBudgetMode: "base" }); + expect(base[0]?.toolBudget).toEqual({ maxToolCalls: 4, maxInvestigationRounds: 2, maxResultChars: 10_000 }); + + const scaled = await build({}); + expect(scaled[0]?.toolBudget).toEqual({ maxToolCalls: 7, maxInvestigationRounds: 2, maxResultChars: 17_500 }); + + const investigate = await build({}, fakePlanForHunks(ids), distantFile, { ...facts, reviewPriority: "high" }); + expect(investigate[0]?.toolBudget).toEqual({ + maxToolCalls: 10, + maxInvestigationRounds: 2, + maxResultChars: 20_000, + sourceExtension: { maxToolCalls: 1, maxResultChars: 4_000 } + }); + + const boosted = await build({ budgetBoost: 1.5 }, fakePlanForHunks(ids), distantFile, { ...facts, reviewPriority: "high" }); + expect(boosted[0]?.toolBudget).toEqual({ + maxToolCalls: 15, + maxInvestigationRounds: 3, + maxResultChars: 30_000, + sourceExtension: { maxToolCalls: 2, maxResultChars: 6_000 } + }); + + const deepPlan = fakePlanForHunks(ids); + deepPlan.coverage = deepPlan.coverage.map((decision) => ({ ...decision, coverage: "deep" })); + const deep = await build({}, deepPlan); + expect(deep[0]?.toolBudget).toEqual({ + maxToolCalls: 19, + maxInvestigationRounds: 5, + maxResultChars: 40_000, + sourceExtension: { maxToolCalls: 1, maxResultChars: 4_000 } + }); + + const nearbyFile = fakeMultiHunkFile(ids.map((id, index) => ({ + id, + newStart: index * 10 + 1, + content: `export function nearby${String(index)}() { return ${String(index)}; }` + }))); + const oneAtom = await build({}, fakePlanForHunks(ids), nearbyFile); + expect(oneAtom[0]?.toolBudget).toEqual({ maxToolCalls: 4, maxInvestigationRounds: 2, maxResultChars: 10_000 }); + + const simpleFile = fakeMultiHunkFile([ + { id: "h1", newStart: 1, content: "import { one } from './one';" }, + { id: "h2", newStart: 100, content: "import { two } from './two';" } + ]); + const simple = await build({}, fakePlanForHunks(["h1", "h2"]), simpleFile, { ...facts, hunkCount: 2, changedLines: 2 }); + expect(simple[0]?.reviewProfile).toBe("simple"); + expect(simple[0]?.toolBudget).toEqual({ maxToolCalls: 0, maxInvestigationRounds: 0, maxResultChars: 0 }); + }); + it("does not coalesce add and deletion-only hunks by comparing new and old coordinates", async () => { const file: DiffFile = { path: "app.ts", @@ -13381,6 +13992,24 @@ function fakeMultiHunkFile(hunks: Array<{ id: string; newStart: number; content: }; } +function packetPackingSymbolFact( + hunkId: string, + enclosingSymbol: string, + symbolRange: [number, number] +): HunkSymbolFacts { + return { + path: "app.ts", + hunkId, + enclosingSymbol, + symbolKind: "function", + symbolRange, + changedLines: [symbolRange[0]], + changedLinesSide: "new", + source: "tree-sitter", + confidence: "syntactic" + }; +} + function fakeFacts(path: string, processingMode: FileFacts["processingMode"]): FileFacts { return { path, From 8fceba9474cd0d1c1222526d8ef6d32582ac55b9 Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 02:44:51 -0400 Subject: [PATCH 02/24] feat(evals): add Plan 102 packet-packing validation Implement Plan 102 steps 5-6 with fail-closed replay, evaluation economics, raw policy reconstruction, deterministic reconciliation, and focused regression coverage. --- scripts/packet-packing-report.ts | 5318 +++++++++++++++++ .../102-issue-102-same-file-packet-packing.md | 29 +- specs/plans/phase-plans/102/phase_2.md | 33 + src/evals/eval-artifacts.ts | 11 + src/evals/eval-command.ts | 45 +- src/evals/eval-runner.ts | 38 +- src/pipeline/composer.ts | 136 +- src/pipeline/review-runner.ts | 8 +- src/pipeline/verifier.ts | 102 + src/telemetry/run-artifacts.ts | 181 +- src/types.ts | 35 + tests/evals.test.ts | 112 + tests/packet-packing-report.test.ts | 2236 +++++++ 13 files changed, 8191 insertions(+), 93 deletions(-) create mode 100644 scripts/packet-packing-report.ts create mode 100644 specs/plans/phase-plans/102/phase_2.md create mode 100644 tests/packet-packing-report.test.ts diff --git a/scripts/packet-packing-report.ts b/scripts/packet-packing-report.ts new file mode 100644 index 0000000..3e82fb2 --- /dev/null +++ b/scripts/packet-packing-report.ts @@ -0,0 +1,5318 @@ +import { mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises"; +import { existsSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { pathToFileURL } from "node:url"; +import { z } from "zod"; +import { defaultConfig } from "../src/config/schema.js"; +import { createGitClient } from "../src/git/git-client.js"; +import { parseDiff } from "../src/git/diff-parser.js"; +import { runGit } from "../src/git/subprocess.js"; +import { buildReviewPackets, packetDispatchRank, packetReviewContextFromDossier, toolBudget } from "../src/pipeline/packet-builder.js"; +import { reconstructComposerGroupsFromArtifacts, reconstructComposerPolicyFromArtifacts } from "../src/pipeline/composer.js"; +import { + reconstructGatedVerifierCandidatesFromArtifacts, + reconstructDuplicateVerificationVerdict, + reconstructVerifiedFindingsFromArtifacts +} from "../src/pipeline/verifier.js"; +import { buildRepositoryIndex } from "../src/repo/repository-index.js"; +import { stripCredentials } from "../src/telemetry/redaction.js"; +import { reconstructRunTelemetryDerivedEvidence } from "../src/telemetry/run-artifacts.js"; +import type { LlmCallRecord, TelemetryRecorder } from "../src/telemetry/telemetry-recorder.js"; +import type { + CodegenieConfig, + DiffFile, + DiffLine, + EvalBudgetResult, + EvalCase, + EvalArtifacts, + EvalExpectationList, + EvalExpectationResult, + EvalInvocationManifest, + EvalRunInfo, + EvalSelectionRecord, + EvalScore, + EvalVerificationRecord, + FileFacts, + FileFilterDecision, + PacketContextQuality, + PlannerDossier, + ResolvedReviewInput, + ReviewPacket, + ReviewPlan, + ReviewProfile, + RunCoverageStatus, + CandidateFinding, + FinalFinding, + TelemetryEvent, + ToolBudget, + ToolCallRecord, + UnifiedDiff +} from "../src/types.js"; +import { sha256Hex } from "../src/util/hashing.js"; +import { scaleToolBudget } from "../src/util/budget.js"; +import { isLocalToolBudgetRejectionReason } from "../src/util/context-pressure.js"; +import { loadEvalCaseDeclaration } from "../src/evals/eval-runner.js"; +import { aggregateRepeatScores, scoreEvalRun } from "../src/evals/eval-scoring.js"; + +const MAX_HUNKS_PER_PACKET = 5; +const MAX_PATCH_CHARS = 12_000; +const EQUIVALENT_TARGET_HUNKS = 142; +const PRODUCTION_BASE_SHA = "d1c49bdf6a8002ec2ec27faac94a932d736532b2"; +const PRODUCTION_HEAD_SHA = "fbb5f8761c2c296e115af17e919a7c35d9de8373"; +const PRODUCTION_REPO_ROOT = "/home/peter/Dev/0xsequence/trails-api"; +const PRODUCTION_TIMEOUT_MS = 60 * 60 * 1000; +const PRODUCTION_CONCURRENCY = 6; +const LEGACY_DIFF_SCHEMA_RUN_IDS = new Set([ + "20260724-135818-740d73f2", + "20260724-150405-fe1548ae", + "20260724-162739-81f806a6" +]); +const PROFILE_RANK: Record = { simple: 0, standard: 1, investigate: 2 }; +const CONTEXT_QUALITY_RANK: Record = { + path_only: 0, + outline_only: 1, + sliced: 2, + full: 3 +}; +const DOCS_CONFIG_EXTENSIONS = new Set([".md", ".yml", ".yaml", ".toml", ".conf", ".sample", ".txt"]); + +const commitShaSchema = z.string().regex(/^[a-f0-9]{40}$/iu); +const jsonObjectSchema = z.record(z.string(), z.unknown()); +const nonnegativeIntSchema = z.number().int().nonnegative(); +const positiveIntSchema = z.number().int().positive(); + +const pullRequestMetadataSchema = z.object({ + owner: z.string(), + repo: z.string(), + number: positiveIntSchema, + title: z.string(), + body: z.string(), + url: z.string(), + baseRefName: z.string(), + baseSha: commitShaSchema, + headRefName: z.string(), + headSha: commitShaSchema +}).strict(); + +const resolvedInputSchema = z.object({ + mode: z.enum(["github_pr", "branch", "head", "commit_range"]), + repoRoot: z.string(), + baseRef: commitShaSchema.optional(), + baseRefName: z.string().optional(), + headRef: commitShaSchema.optional(), + headRefName: z.string().optional(), + startCommit: commitShaSchema.optional(), + endCommit: commitShaSchema.optional(), + mergeBase: commitShaSchema.optional(), + headSha: commitShaSchema.optional(), + pr: pullRequestMetadataSchema.optional(), + commits: z.array(z.object({ + sha: commitShaSchema, + title: z.string(), + body: z.string(), + authorName: z.string().optional(), + authoredAt: z.string().optional() + }).strict()), + rawDiffChars: nonnegativeIntSchema +}).strict(); + +const diffLineSchema = z.object({ + kind: z.enum(["context", "add", "delete"]), + content: z.string(), + oldLineNumber: positiveIntSchema.optional(), + newLineNumber: positiveIntSchema.optional() +}).strict(); + +const diffHunkSchema = z.object({ + id: z.string().min(1), + // The pre-Plan-100 retained artifacts predate persisted hunkHash. Their + // versioned migration is checked against a freshly parsed diff below. + hunkHash: z.string().min(1).optional(), + path: z.string().min(1), + oldStart: nonnegativeIntSchema, + oldLines: nonnegativeIntSchema, + newStart: nonnegativeIntSchema, + newLines: nonnegativeIntSchema, + header: z.string(), + lines: z.array(diffLineSchema) +}).strict(); + +const diffFileSchema = z.object({ + path: z.string().min(1), + oldPath: z.string().min(1).optional(), + status: z.enum(["added", "modified", "deleted", "renamed", "copied"]), + isBinary: z.boolean().optional(), + modeOnly: z.boolean().optional(), + isSymlink: z.boolean().optional(), + isSubmodule: z.boolean().optional(), + language: z.string(), + hunks: z.array(diffHunkSchema) +}).strict(); + +const unifiedDiffSchema = z.object({ files: z.array(diffFileSchema) }).strict(); + +const factProvenanceSchema = z.object({ + fact: z.string(), + source: z.enum(["path", "filename", "extension", "parser", "git", "diff", "config", "generated_detector"]), + confidence: z.enum(["high", "medium", "low"]), + reason: z.string() +}).strict(); + +const fileFilterDecisionSchema = z.object({ + path: z.string().min(1), + action: z.enum(["skip", "keep"]), + reason: z.string(), + provenance: z.array(factProvenanceSchema) +}).strict(); + +const fileFactsSchema = z.object({ + path: z.string().min(1), + language: z.string(), + packageRoot: z.string().optional(), + processingMode: z.enum(["per-hunk", "whole-file", "skip"]), + testStatus: z.enum(["test", "source", "unknown"]), + isGenerated: z.boolean(), + isVendored: z.boolean(), + isLockfile: z.boolean(), + isBinary: z.boolean(), + changedLines: nonnegativeIntSchema, + hunkCount: nonnegativeIntSchema, + labels: z.array(z.string()), + reviewPriority: z.enum(["critical", "high", "normal", "low"]), + reasons: z.array(z.string()), + provenance: z.array(factProvenanceSchema), + degraded: z.object({ reason: z.string() }).strict().optional() +}).strict(); + +const surroundingContextHintSchema = z.object({ + kind: z.enum(["enclosing_symbol", "call_site", "test", "line_range", "other"]), + path: z.string().optional(), + symbol: z.string().optional(), + lineRange: z.tuple([positiveIntSchema, positiveIntSchema]).optional(), + reason: z.string(), + expectedUse: z.enum(["packet_context", "tool_lookup"]) +}).strict(); + +const hunkCoverageDecisionSchema = z.object({ + hunkId: z.string().min(1), + path: z.string().min(1), + coverage: z.enum(["deep", "normal", "light", "skip"]), + lenses: z.array(z.string()), + surroundingContextHints: z.array(surroundingContextHintSchema), + reason: z.string(), + focusNotes: z.array(z.string()).optional(), + relatedSymbols: z.array(z.string()).optional(), + relatedFiles: z.array(z.string()).optional() +}).strict(); + +const reviewPlanSchema = z.object({ + diffUnderstanding: z.object({ declaredIntent: z.string(), inferredBehavior: z.string() }).strict(), + intentSignals: jsonObjectSchema.optional(), + coverage: z.array(hunkCoverageDecisionSchema), + plannerRecovery: jsonObjectSchema.optional(), + partialReview: z.object({ + isPartial: z.boolean(), + reason: z.string(), + reviewedHunks: nonnegativeIntSchema, + totalHunks: nonnegativeIntSchema + }).strict().optional() +}).strict(); + +const plannerDossierSchema = z.object({ + commits: z.array(jsonObjectSchema), + compaction: jsonObjectSchema, + depth: z.enum(["light", "normal", "deep"]), + directories: z.array(z.unknown()), + files: z.array(z.unknown()), + filterSummary: jsonObjectSchema, + hunkIndex: z.array(z.unknown()).optional(), + intentSignals: jsonObjectSchema, + lenses: z.array(z.object({ id: z.string().min(1), summary: z.string() }).strict()), + mode: z.enum(["github_pr", "branch", "head", "commit_range"]), + policyFilesChanged: z.array(z.string()), + pr: z.object({ + title: z.string(), + body: z.string(), + url: z.string(), + baseRefName: z.string(), + headRefName: z.string() + }).strict().optional(), + runId: z.string().min(1), + target: z.object({ + baseRef: commitShaSchema.optional(), + headRef: commitShaSchema.optional(), + headSha: commitShaSchema.optional(), + mergeBase: commitShaSchema.optional() + }).strict(), + totals: z.object({ + addedLines: nonnegativeIntSchema, + deletedLines: nonnegativeIntSchema, + files: nonnegativeIntSchema, + hunks: nonnegativeIntSchema, + keptFiles: nonnegativeIntSchema + }).strict() +}).strict(); + +const runMetadataSchema = z.object({ + argv: z.array(z.string()), + budgetStop: z.unknown().nullable(), + codegenieRuntime: jsonObjectSchema, + codegenieVersion: z.string(), + completedAt: z.string().optional(), + durationMs: nonnegativeIntSchema, + finishedAt: z.string(), + nodeVersion: z.string(), + outcome: jsonObjectSchema, + repoRoot: z.string(), + review: jsonObjectSchema, + runId: z.string().min(1), + schemaVersion: positiveIntSchema, + startedAt: z.string(), + totals: jsonObjectSchema +}).strict(); + +const toolBudgetSchema = z.object({ + maxToolCalls: z.number().int().nonnegative(), + maxInvestigationRounds: z.number().int().nonnegative(), + maxResultChars: z.number().int().nonnegative(), + maxSingleToolResultChars: z.number().int().nonnegative().optional(), + reservedSourceResultChars: z.number().int().nonnegative().optional(), + sourceExtension: z.object({ + maxToolCalls: z.number().int().nonnegative(), + maxResultChars: z.number().int().nonnegative() + }).strict().optional() +}).strict(); + +const packetLineSchema = z.object({ + kind: z.enum(["context", "add", "delete"]), + oldLine: positiveIntSchema.optional(), + newLine: positiveIntSchema.optional(), + content: z.string() +}).strict(); + +const packetHunkSchema = z.object({ + hunkId: z.string().min(1), + oldStart: nonnegativeIntSchema, + oldLines: nonnegativeIntSchema, + newStart: nonnegativeIntSchema, + newLines: nonnegativeIntSchema, + header: z.string().optional(), + contentWithLineNumbers: z.string(), + lines: z.array(packetLineSchema), + changedNewLineNumbers: z.array(positiveIntSchema), + changedOldLineNumbers: z.array(positiveIntSchema), + staticSignals: z.array(jsonObjectSchema).optional(), + omittedSignalCount: nonnegativeIntSchema.optional(), + truncated: z.boolean().optional(), + omittedLineCount: nonnegativeIntSchema.optional(), + plannerFallbackReason: z.string().optional() +}).strict(); + +const reviewPacketSchema = z.object({ + id: z.string().min(1), + dispatchRank: z.tuple([nonnegativeIntSchema, z.number().int().nonpositive()]).optional(), + kind: z.enum(["hunk", "coalesced-hunks", "file-diff", "whole-file"]), + coverageEscalation: z.object({ rule: z.literal("test_coverage_delta"), reason: z.string() }).strict().optional(), + prSummary: z.string(), + intentText: z.string().optional(), + intentSignals: jsonObjectSchema.optional(), + path: z.string().min(1), + oldPath: z.string().min(1).optional(), + fileStatus: z.enum(["added", "modified", "deleted", "renamed", "copied"]), + isDeletedContent: z.boolean(), + language: z.string(), + reviewPriority: z.enum(["critical", "high", "normal", "low"]), + coverage: z.enum(["deep", "normal", "light"]), + reviewProfile: z.enum(["simple", "standard", "investigate"]), + lenses: z.array(z.string()), + hunks: z.array(packetHunkSchema).min(1), + symbolFacts: z.array(jsonObjectSchema), + context: jsonObjectSchema, + contextText: z.string(), + contextQuality: z.enum(["path_only", "outline_only", "sliced", "full"]).optional(), + contextDegradationReasons: z.array(z.string()).optional(), + testCoverageDelta: jsonObjectSchema.optional(), + packetSymbols: z.array(jsonObjectSchema).optional(), + relevantTests: z.array(jsonObjectSchema), + surroundingContextHints: z.array(surroundingContextHintSchema), + labels: z.array(z.string()), + attentionNotes: z.array(z.string()), + relatedChangedContext: z.array(jsonObjectSchema), + toolBudget: toolBudgetSchema, + degraded: z.object({ reason: z.string() }).strict().optional(), + fileContext: z.object({ mode: z.enum(["file-diff", "whole-file"]), reason: z.string() }).strict().optional() +}).strict(); + +const diffAnchorSchema = z.object({ + path: z.string().min(1), line: positiveIntSchema, side: z.enum(["RIGHT", "LEFT"]), hunkId: z.string().min(1), + startLine: positiveIntSchema.optional(), startSide: z.enum(["RIGHT", "LEFT"]).optional(), commitSha: commitShaSchema.optional() +}).strict(); + +const candidateFindingSchema = z.object({ + id: z.string().min(1), title: z.string(), severity: z.enum(["critical", "high", "medium", "low"]), + severityBeforeCap: z.enum(["critical", "high", "medium", "low"]).optional(), confidence: z.enum(["high", "medium", "low"]), + path: z.string().min(1), anchor: diffAnchorSchema.optional(), + anchorSource: z.enum(["model", "backfill_changed_code", "backfill_packet_representative", "verifier_revised"]).optional(), + modelAnchorSubmitted: z.boolean().optional(), changedLine: z.boolean(), + category: z.enum(["logic_bug", "correctness", "security", "performance", "architecture", "testing", "maintainability"]), + evidence: z.object({ + changedCode: z.string(), relatedCode: z.array(z.object({ path: z.string(), lines: z.string(), whyRelevant: z.string() }).strict()).optional() + }).strict(), + failureMode: z.string(), whyThisMatters: z.string(), suggestedFix: z.string().optional(), suggestedTest: z.string().optional(), + verification: z.string(), behaviorChange: z.enum(["accidental_regression", "intentional_needs_confirmation", "specified_change", "unknown"]).optional(), + intentEvidence: z.array(z.string()).optional(), + producedBy: z.object({ + kind: z.literal("packet"), stage: z.number().int().min(1).max(11), packetId: z.string().min(1), lensId: z.string().min(1), + skillIds: z.array(z.string()), workerId: z.string().optional(), ensemblePass: positiveIntSchema.optional() + }).strict(), + provenance: z.object({ + source: z.literal("uncertainty_promotion"), sourceKind: z.enum(["uncertainty", "follow_up_hint"]), sourcePacketId: z.string(), + question: z.string(), files: z.array(z.string()), symbols: z.array(z.string()), reason: z.string() + }).strict().optional(), + clusterId: z.string().optional(), duplicateOf: z.string().optional() +}).strict(); + +const finalFindingSchema = candidateFindingSchema.extend({ + fingerprint: z.string().min(1), finalBody: z.string(), publication: z.enum(["inline", "summary-only", "suppressed"]), + mergedCandidateIds: z.array(z.string()), + mergedCategories: z.array(z.enum(["logic_bug", "correctness", "security", "performance", "architecture", "testing", "maintainability"])).optional(), + mergedSeverities: z.array(z.enum(["critical", "high", "medium", "low"])).optional(), mergedPaths: z.array(z.string()).optional(), + mergedTitles: z.array(z.string()).optional(), + // Persisted artifacts pass through the shared redactor, which emits this + // controlled sentinel when mergedAnchors reuses the primary anchor object. + mergedAnchors: z.array(z.union([diffAnchorSchema, z.literal("[redacted:circular]")])).optional() +}).strict(); + +const verificationGateFactsSchema = z.object({ + anchorSource: z.enum(["model", "backfill_changed_code", "backfill_packet_representative", "verifier_revised"]).optional(), + category: z.enum(["logic_bug", "correctness", "security", "performance", "architecture", "testing", "maintainability"]), + changedLine: z.boolean(), confidence: z.enum(["high", "medium", "low"]), failureModeConcrete: z.boolean(), + hasChangedCode: z.boolean(), hasFailureMode: z.boolean(), modelAnchorSubmitted: z.boolean(), modelAnchorValid: z.boolean(), + relatedEvidenceCount: nonnegativeIntSchema, severity: z.enum(["critical", "high", "medium", "low"]), validAnchorPresent: z.boolean() +}).strict(); + +const candidateProvenanceSchema = z.object({ + source: z.literal("uncertainty_promotion"), sourceKind: z.enum(["uncertainty", "follow_up_hint"]), sourcePacketId: z.string().min(1), + question: z.string(), files: z.array(z.string()), symbols: z.array(z.string()), reason: z.string() +}).strict(); + +const verificationVerdictSchema = z.object({ + candidateId: z.string().min(1), verdict: z.enum(["keep", "reject", "revise", "incomplete"]), reason: z.string(), + requiredEvidencePresent: z.boolean(), falsePositiveRisk: z.enum(["low", "medium", "high"]), + finalFinding: candidateFindingSchema.optional(), revisedAnchor: diffAnchorSchema.optional(), verificationIncomplete: z.boolean().optional(), + behaviorChange: z.enum(["accidental_regression", "intentional_needs_confirmation", "specified_change", "unknown"]).optional(), + intentEvidence: z.array(z.string()).optional() +}).strict(); + +const verificationRecordBase = { + candidateId: z.string().min(1), gate: z.enum(["suppressed", "passed", "gate_anchor_stripped"]), + gateDecision: z.enum(["suppressed", "scheduled", "scheduled_for_evidence_resolution"]).optional(), gateReason: z.string().optional(), + verificationLane: z.enum(["standard", "evidence_resolution"]).optional(), gateFacts: verificationGateFactsSchema.optional(), + candidateProvenance: candidateProvenanceSchema.optional(), duplicateOf: z.string().optional(), clusterId: z.string().optional(), + verificationStatus: z.enum(["completed", "incomplete"]).optional(), incompleteReason: z.string().optional(), errorCode: z.string().optional() +}; + +const verificationRecordSchema = z.object({ ...verificationRecordBase, verdict: verificationVerdictSchema.optional() }).strict().superRefine((record, ctx) => { + if (record.gate === "suppressed" && record.gateReason === undefined) { + ctx.addIssue({ code: "custom", path: ["gateReason"], message: "suppressed verification records require gateReason" }); + } + if (record.gate === "passed" && record.verdict === undefined) { + ctx.addIssue({ code: "custom", path: ["verdict"], message: "passed verification records require a verdict" }); + } +}); + +const selectionRecordSchema = z.object({ + findingId: z.string().min(1), decision: z.enum(["published", "merged", "suppressed"]), reason: z.string(), + mergedIntoFingerprint: z.string().optional() +}).strict(); + +const finalSelectionArtifactSchema = z.object({ + composition: z.object({ + mode: z.enum(["llm", "llm_degraded", "deterministic_fallback", "schema_repair_fallback"]), fallbackReason: z.string().optional() + }).strict(), + records: z.array(selectionRecordSchema), + publicationAnchors: z.array(z.object({ + findingId: z.string().min(1), fingerprint: z.string().min(1), publication: z.enum(["inline", "summary-only", "suppressed"]), + source: z.enum(["selected", "merged", "none"]), reason: z.string(), sourceFindingId: z.string().optional(), anchor: diffAnchorSchema.optional() + }).strict()), + confidenceSelections: z.array(z.object({ + findingId: z.string().min(1), confidence: z.enum(["high", "medium", "low"]), representativeConfidence: z.enum(["high", "medium", "low"]), + sourceFindingId: z.string().optional(), reason: z.enum(["representative", "same_severity", "compatible_lower_severity"]) + }).strict()), + groups: z.array(z.object({ fingerprint: z.string().min(1), findingIds: z.array(z.string().min(1)).min(1) }).strict()) +}).strict(); + +type FinalSelectionArtifact = z.infer; + +const telemetryEventSchema = z.object({ + runId: z.string().min(1), + eventId: z.string().min(1), + timestamp: z.string().min(1), + stage: z.number().int().min(0).max(11), + level: z.enum(["debug", "info", "warn", "error"]), + message: z.string().min(1), + file: z.string().optional(), + lineRange: z.tuple([positiveIntSchema, positiveIntSchema]).optional(), + packetId: z.string().optional(), + lensId: z.string().optional(), + workerId: z.string().optional(), + durationMs: nonnegativeIntSchema.optional(), + cacheStatus: z.enum(["hit", "miss", "disabled", "write"]).optional(), + data: jsonObjectSchema.optional() +}).strict().superRefine((event, ctx) => { + if (event.message !== "same_file_atoms_packed") { + return; + } + const parsed = sameFilePackingEventDataSchema.safeParse(event.data); + if (!parsed.success) { + ctx.addIssue({ code: "custom", path: ["data"], message: "same_file_atoms_packed data must match the strict packing schema" }); + } +}); + +const llmCallRecordSchema = z.object({ + callId: z.string().min(1), runId: z.string().min(1), stage: z.number().int().min(0).max(11), + role: z.enum(["planner", "packetReview", "systemReview", "verifier", "composer"]), + model: z.string(), provider: z.string(), workerId: z.string().optional(), packetId: z.string().optional(), candidateId: z.string().optional(), + kind: z.enum(["initial", "tool-continuation", "repair", "finalize"]), + finalizeMode: z.enum(["compact", "full"]).optional(), finalizeTarget: z.enum(["no_findings", "candidate_or_unknown"]).optional(), + toolChoiceRequested: z.string().optional(), toolChoiceEffective: z.string().optional(), toolChoiceDowngraded: z.boolean().optional(), + reasoningRequested: z.string().optional(), reasoningMechanism: z.string().optional(), reasoningLevelEffective: z.string().optional(), + ttfbMs: nonnegativeIntSchema.optional(), providerHttpStatus: nonnegativeIntSchema.optional(), providerRequestId: z.string().optional(), + rateLimit: z.record(z.string(), z.string()).optional(), attempt: positiveIntSchema, promptChars: nonnegativeIntSchema, promptHash: z.string(), + outputChars: nonnegativeIntSchema, outputHash: z.string(), inputTokens: nonnegativeIntSchema.optional(), uncachedInputTokens: nonnegativeIntSchema.optional(), + cacheReadTokens: nonnegativeIntSchema.optional(), cacheWriteTokens: nonnegativeIntSchema.optional(), billableInputTokens: nonnegativeIntSchema.optional(), + outputTokens: nonnegativeIntSchema.optional(), reasoningTokens: nonnegativeIntSchema.optional(), totalTokens: nonnegativeIntSchema.optional(), + costUSD: z.number().nonnegative().optional(), inputCostUSD: z.number().nonnegative().optional(), outputCostUSD: z.number().nonnegative().optional(), + cacheReadCostUSD: z.number().nonnegative().optional(), cacheWriteCostUSD: z.number().nonnegative().optional(), durationMs: nonnegativeIntSchema, + cacheStatus: z.enum(["hit", "miss", "disabled", "write"]), schemaValid: z.boolean().optional(), + stopReason: z.enum(["submit", "tool_calls", "text", "error"]), + status: z.enum(["ok", "schema_invalid", "transient_error", "auth_error", "timeout", "aborted"]), + errorCode: z.string().optional(), errorMessage: z.string().optional(), retryable: z.boolean().optional(), retryReason: z.string().optional(), + maxAttempts: positiveIntSchema.optional(), retryExhausted: z.boolean().optional() +}).strict(); + +const toolCallRecordSchema = z.object({ + runId: z.string().min(1), toolCallId: z.string().min(1), timestamp: z.string().min(1), stage: z.number().int().min(1).max(11), + initiator: z.enum(["model", "harness"]), workerId: z.string().optional(), packetId: z.string().optional(), taskId: z.string().optional(), + candidateId: z.string().optional(), modelCallId: z.string().optional(), tool: z.string().min(1), args: z.object({ + path: z.string().optional(), symbolName: z.string().optional(), line: positiveIntSchema.optional(), startLine: positiveIntSchema.optional(), + endLine: positiveIntSchema.optional(), query: z.string().optional(), glob: z.string().optional(), source: z.string().optional(), + contextMode: z.string().optional(), maxResults: positiveIntSchema.optional() + }).strict(), + backend: z.enum(["tree-sitter", "text", "language-analyzer"]), precision: z.enum(["exact", "semantic", "syntactic", "heuristic", "text"]), + engine: z.literal("git-grep").optional(), degraded: z.boolean(), degradationReason: z.string().optional(), truncated: z.boolean().optional(), + omittedCount: nonnegativeIntSchema.optional(), lookupStatus: z.enum(["found", "not_found", "ambiguous", "file_missing", "unavailable"]).optional(), + deliveryStatus: z.enum(["full", "truncated", "budget_rejected", "empty"]).optional(), recovery: z.object({ + tool: z.literal("read_range"), path: z.string(), startLine: positiveIntSchema, endLine: positiveIntSchema, + source: z.enum(["head", "base"]), reason: z.string() + }).strict().optional(), + budgetState: z.object({ + toolCallsUsed: nonnegativeIntSchema, maxToolCalls: nonnegativeIntSchema, investigationRoundsUsed: nonnegativeIntSchema, + maxInvestigationRounds: nonnegativeIntSchema, resultCharsUsed: nonnegativeIntSchema, maxResultChars: nonnegativeIntSchema, + remainingResultChars: nonnegativeIntSchema, maxSingleToolResultChars: nonnegativeIntSchema.optional(), reservedSourceResultChars: nonnegativeIntSchema.optional(), + toolResultCharLimit: nonnegativeIntSchema.optional(), sourceExtensionCallsUsed: nonnegativeIntSchema.optional(), sourceExtensionMaxCalls: nonnegativeIntSchema.optional(), + sourceExtensionResultCharsUsed: nonnegativeIntSchema.optional(), sourceExtensionMaxResultChars: nonnegativeIntSchema.optional(), + sourceExtensionRemainingResultChars: nonnegativeIntSchema.optional(), sourceExtensionActive: z.boolean().optional() + }).strict().optional(), cacheStatus: z.enum(["hit", "miss", "disabled", "write"]).optional(), backendExecuted: z.boolean().optional(), + cacheHitKind: z.enum(["stored", "inflight"]).optional(), cacheEvictedEntries: nonnegativeIntSchema.optional(), resultCount: nonnegativeIntSchema.optional(), + resultChars: nonnegativeIntSchema, durationMs: nonnegativeIntSchema, status: z.enum(["ok", "error", "rejected", "skipped"]), errorCode: z.string().optional() +}).strict(); + +const cacheCountsSchema = z.object({ + hit: nonnegativeIntSchema, miss: nonnegativeIntSchema, disabled: nonnegativeIntSchema, write: nonnegativeIntSchema +}).strict(); +const providerPromptCacheSchema = z.object({ + readTokens: nonnegativeIntSchema, writeTokens: nonnegativeIntSchema, + readCostUSD: z.number().nonnegative(), writeCostUSD: z.number().nonnegative() +}).strict(); +const modelFinalizeSummarySchema = z.object({ + compactCalls: nonnegativeIntSchema, fullCalls: nonnegativeIntSchema, noFindingCalls: nonnegativeIntSchema, + candidateOrUnknownCalls: nonnegativeIntSchema, promptChars: nonnegativeIntSchema, noFindingPromptChars: nonnegativeIntSchema, + candidateOrUnknownPromptChars: nonnegativeIntSchema, costUSD: z.number().nonnegative(), noFindingCostUSD: z.number().nonnegative(), + candidateOrUnknownCostUSD: z.number().nonnegative(), unknownCostCalls: nonnegativeIntSchema +}).strict(); +const schemaRecoveryCountersSchema = z.object({ + schemaInvalidCalls: nonnegativeIntSchema, schemaInvalidRecovered: nonnegativeIntSchema, schemaInvalidUnrecovered: nonnegativeIntSchema, + schemaRepairAttempts: nonnegativeIntSchema, schemaRepairRecovered: nonnegativeIntSchema, + deterministicSchemaRecovered: nonnegativeIntSchema, schemaRecoveryFailed: nonnegativeIntSchema +}).strict(); +const schemaRecoverySummarySchema = schemaRecoveryCountersSchema.extend({ + byStage: z.record(z.string().regex(/^\d+$/u), schemaRecoveryCountersSchema) +}).strict(); +const modelSummaryBaseShape = { + totalRecords: nonnegativeIntSchema, totalCalls: nonnegativeIntSchema, providerCalls: nonnegativeIntSchema, + inputTokens: nonnegativeIntSchema, uncachedInputTokens: nonnegativeIntSchema, cacheReadTokens: nonnegativeIntSchema, + cacheWriteTokens: nonnegativeIntSchema, billableInputTokens: nonnegativeIntSchema, outputTokens: nonnegativeIntSchema, + reasoningTokens: nonnegativeIntSchema, totalTokens: nonnegativeIntSchema, costUSD: z.number().nonnegative(), + inputCostUSD: z.number().nonnegative(), outputCostUSD: z.number().nonnegative(), cacheReadCostUSD: z.number().nonnegative(), + cacheWriteCostUSD: z.number().nonnegative(), unknownCostCalls: nonnegativeIntSchema, cache: cacheCountsSchema, + retryAttempts: nonnegativeIntSchema, repairCalls: nonnegativeIntSchema, schemaInvalidCalls: nonnegativeIntSchema, + toolChoiceDowngradedCalls: nonnegativeIntSchema, finalize: modelFinalizeSummarySchema +}; +const modelStageSummarySchema = z.object({ + recordCount: nonnegativeIntSchema, count: nonnegativeIntSchema, providerCalls: nonnegativeIntSchema, + inputTokens: nonnegativeIntSchema, uncachedInputTokens: nonnegativeIntSchema, cacheReadTokens: nonnegativeIntSchema, + cacheWriteTokens: nonnegativeIntSchema, billableInputTokens: nonnegativeIntSchema, outputTokens: nonnegativeIntSchema, + reasoningTokens: nonnegativeIntSchema, totalTokens: nonnegativeIntSchema, costUSD: z.number().nonnegative(), + inputCostUSD: z.number().nonnegative(), outputCostUSD: z.number().nonnegative(), cacheReadCostUSD: z.number().nonnegative(), + cacheWriteCostUSD: z.number().nonnegative(), unknownCostCalls: nonnegativeIntSchema, cache: cacheCountsSchema, + retryAttempts: nonnegativeIntSchema, repairCalls: nonnegativeIntSchema, schemaInvalidCalls: nonnegativeIntSchema, + statuses: z.object({ + ok: nonnegativeIntSchema, schema_invalid: nonnegativeIntSchema, transient_error: nonnegativeIntSchema, + auth_error: nonnegativeIntSchema, timeout: nonnegativeIntSchema, aborted: nonnegativeIntSchema + }).strict(), + finalize: modelFinalizeSummarySchema, localModelCallCache: cacheCountsSchema, + providerPromptCache: providerPromptCacheSchema, schemaRecovery: schemaRecoveryCountersSchema +}).strict(); +const modelCallsSummarySchema = z.object({ + ...modelSummaryBaseShape, + localModelCallCache: cacheCountsSchema, + providerPromptCache: providerPromptCacheSchema, + schemaRecovery: schemaRecoverySummarySchema, + byStage: z.record(z.string().regex(/^\d+$/u), modelStageSummarySchema) +}).strict(); +const costBreakdownSchema = z.object({ + uncachedInput: z.object({ tokens: nonnegativeIntSchema, costUSD: z.number().nonnegative() }).strict(), + providerPromptCacheRead: z.object({ tokens: nonnegativeIntSchema, costUSD: z.number().nonnegative() }).strict(), + providerPromptCacheWrite: z.object({ tokens: nonnegativeIntSchema, costUSD: z.number().nonnegative() }).strict(), + output: z.object({ tokens: nonnegativeIntSchema, costUSD: z.number().nonnegative() }).strict(), + total: z.object({ tokens: nonnegativeIntSchema, costUSD: z.number().nonnegative() }).strict() +}).strict(); +const costProfileStageSchema = modelStageSummarySchema.omit({ schemaRecovery: true }).extend({ costBreakdown: costBreakdownSchema }).strict(); +const costProfileSchema = z.object({ + totalCostUSD: z.number().nonnegative(), unknownCostCalls: nonnegativeIntSchema, + localModelCallCache: cacheCountsSchema, providerPromptCache: providerPromptCacheSchema, costBreakdown: costBreakdownSchema, + tokens: z.object({ + inputTokens: nonnegativeIntSchema, uncachedInputTokens: nonnegativeIntSchema, cacheReadTokens: nonnegativeIntSchema, + cacheWriteTokens: nonnegativeIntSchema, billableInputTokens: nonnegativeIntSchema, outputTokens: nonnegativeIntSchema, + reasoningTokens: nonnegativeIntSchema, totalTokens: nonnegativeIntSchema + }).strict(), + cost: z.object({ + inputCostUSD: z.number().nonnegative(), outputCostUSD: z.number().nonnegative(), cacheReadCostUSD: z.number().nonnegative(), + cacheWriteCostUSD: z.number().nonnegative(), totalCostUSD: z.number().nonnegative() + }).strict(), + byStage: z.record(z.string().regex(/^\d+$/u), costProfileStageSchema) +}).strict(); +const toolResultCacheSummarySchema = z.object({ + hits: nonnegativeIntSchema, misses: nonnegativeIntSchema, writes: nonnegativeIntSchema, disabled: nonnegativeIntSchema, + inflightHits: nonnegativeIntSchema, evictions: nonnegativeIntSchema, backendExecutions: nonnegativeIntSchema, + savedBackendCalls: nonnegativeIntSchema +}).strict(); +const toolBucketSummarySchema = z.object({ + count: nonnegativeIntSchema, errors: nonnegativeIntSchema, rejections: nonnegativeIntSchema, degraded: nonnegativeIntSchema, + backendExecutions: nonnegativeIntSchema, savedBackendCalls: nonnegativeIntSchema, totalDurationMs: nonnegativeIntSchema, + totalResultChars: nonnegativeIntSchema, resultCache: toolResultCacheSummarySchema, + averageDurationMs: z.number().nonnegative(), averageResultChars: z.number().nonnegative() +}).strict(); +const toolCallsSummarySchema = z.object({ + totalCalls: nonnegativeIntSchema, resultCache: toolResultCacheSummarySchema, + byTool: z.record(z.string().min(1), toolBucketSummarySchema), byStage: z.record(z.string().regex(/^\d+$/u), toolBucketSummarySchema) +}).strict(); + +const evalDeclaredExpectationSchema = z.object({ + id: z.string().min(1), tier: z.enum(["required", "optional"]).optional(), path: z.string().min(1).optional(), + lineRange: z.tuple([positiveIntSchema, positiveIntSchema]).optional(), + category: z.enum(["logic_bug", "correctness", "security", "performance", "architecture", "testing", "maintainability"]).optional(), + severityAtLeast: z.enum(["critical", "high", "medium", "low"]).optional(), titlePattern: z.string().min(1).optional(), + failureModePattern: z.string().min(1).optional(), minRecallRate: z.number().min(0).max(1).optional(), minCandidateRate: z.number().min(0).max(1).optional() +}).strict(); + +const evalCaseArtifactSchema = z.object({ + name: z.string().min(1), repeat: positiveIntSchema.optional(), + repo: z.object({ external: z.string().min(1).optional(), fixture: z.string().min(1).optional() }).strict().optional(), + command: z.object({ + pr: positiveIntSchema.optional(), branch: z.string().min(1).optional(), head: z.string().min(1).optional(), + base: z.string().min(1).optional(), target: z.string().min(1).optional() + }).strict().optional(), + review: z.object({ + depth: z.enum(["light", "normal", "deep"]).optional(), lenses: z.array(z.string().min(1)).optional(), maxFindings: positiveIntSchema.optional(), + concurrency: positiveIntSchema.optional(), budgetBoost: z.number().positive().optional(), packSameFileHunks: z.boolean().optional(), + packedToolBudgetMode: z.enum(["base", "atom-scaled"]).optional(), maxTimeMinutes: z.number().positive().optional(), + maxBudgetTokens: positiveIntSchema.optional(), deepEnsemblePasses: positiveIntSchema.optional(), adaptiveSecondPass: z.boolean().optional(), + verify: z.boolean().optional(), cache: z.boolean().optional(), cacheDir: z.string().min(1).optional(), debug: z.boolean().optional(), + provider: z.string().min(1).optional(), model: z.string().min(1).optional(), reasoning: z.string().min(1).optional() + }).strict().optional(), + llm: z.object({ + provider: z.string().min(1).optional(), model: z.string().min(1).optional(), reasoning: z.string().min(1).optional(), + maxConcurrentCalls: positiveIntSchema.optional() + }).strict().optional(), + logs: z.object({ dir: z.string().min(1).optional() }).strict().optional(), + artifacts: z.object({ path: z.string().min(1) }).strict().optional(), + expect: z.object({ + minFindings: z.number().nonnegative().optional(), maxFindings: z.number().positive().optional(), maxDuplicateGroups: z.number().positive().optional(), + maxCostUSD: z.number().positive().optional(), maxElapsedSeconds: z.number().positive().optional(), maxModelCalls: z.number().positive().optional(), + maxToolCalls: z.number().positive().optional(), maxPromptCharsByStage: z.record(z.string(), positiveIntSchema).optional(), + reviewCompleteness: z.enum(["complete", "partial"]).optional(), maxBudgetOverruns: nonnegativeIntSchema.optional(), + maxToolBudgetRejections: nonnegativeIntSchema.optional(), maxDegradedHunks: nonnegativeIntSchema.optional(), + maxUnresolvedNotesSuppressed: nonnegativeIntSchema.optional() + }).strict().optional(), + should_find: z.array(evalDeclaredExpectationSchema).optional(), + should_find_candidate: z.array(evalDeclaredExpectationSchema).optional(), + should_not_find: z.array(evalDeclaredExpectationSchema).optional() +}).strict(); + +const evalExpectationResultSchema = z.object({ + expectationId: z.string().min(1), list: z.enum(["should_find", "should_find_candidate", "should_not_find"]), + tier: z.enum(["required", "optional"]).optional(), status: z.enum(["pass", "fail", "skipped"]), skipReason: z.string().optional(), + fromReplayedArtifacts: z.boolean().optional(), + matched: z.array(z.object({ findingId: z.string().min(1), artifact: z.enum(["final-findings", "candidate-findings"]) }).strict()), + loss: z.object({ + label: z.enum(["missed-before-candidate-generation", "lost-at-verification", "lost-at-composition", "partial-match"]), + subReason: z.string().optional(), nearestInstances: z.array(z.object({ + findingId: z.string().optional(), artifact: z.enum(["final-findings", "final-selection", "verification", "candidate-findings", "events"]), + outcome: z.string(), fieldMismatches: z.array(z.object({ + field: z.enum(["path", "lineRange", "category", "severityAtLeast", "titlePattern", "failureModePattern"]), + present: z.boolean(), matched: z.boolean(), expected: z.string().optional(), actual: z.string().optional(), via: z.string().optional() + }).strict()).optional() + }).strict()), + matchingHints: z.array(z.object({ + packetId: z.string().optional(), question: z.string(), files: z.array(z.string()), symbols: z.array(z.string()), + confidence: z.enum(["high", "medium", "low"]) + }).strict()).optional(), coveringPacketIds: z.array(z.string()).optional(), coveringPacketLenses: z.array(z.string()).optional(), + plannerCoverage: z.string().optional(), surfacedAsNote: z.boolean().optional() + }).strict().optional(), note: z.string().optional() +}).strict(); + +const evalBudgetResultSchema = z.object({ + check: z.string().min(1), stage: z.number().int().min(1).max(11).optional(), status: z.enum(["pass", "fail", "skipped"]), + skipReason: z.string().optional(), limit: z.number().optional(), actual: z.number().optional(), expected: z.string().optional(), + actualText: z.string().optional(), direction: z.enum(["minimum", "maximum", "equals"]), fromReplayedArtifacts: z.boolean().optional() +}).strict(); + +const evalMetricsSchema = z.object({ + reportedFindings: nonnegativeIntSchema, inlineFindings: nonnegativeIntSchema, summaryOnlyFindings: nonnegativeIntSchema, + suppressedFindings: nonnegativeIntSchema, candidateFindings: nonnegativeIntSchema, duplicateGroups: nonnegativeIntSchema, + costUSD: z.number().nonnegative().optional(), elapsedSeconds: z.number().nonnegative().optional(), modelCalls: nonnegativeIntSchema.optional(), + verificationCalls: nonnegativeIntSchema.optional(), toolCalls: nonnegativeIntSchema.optional(), toolChoiceDowngradedCalls: nonnegativeIntSchema.optional(), + attentionEfficiency: jsonObjectSchema.optional(), missingArtifacts: z.array(z.string()).optional(), maxPromptCharsByStage: jsonObjectSchema.optional(), + reviewCompleteness: z.enum(["complete", "partial"]).optional(), budgetOverruns: nonnegativeIntSchema.optional(), + toolBudgetRejections: nonnegativeIntSchema.optional(), toolBudgetExtensions: nonnegativeIntSchema.optional(), toolBudgetExtensionDenials: nonnegativeIntSchema.optional(), + degradedHunks: nonnegativeIntSchema.optional(), unresolvedNotesSuppressed: nonnegativeIntSchema.optional(), localModelCallCacheHits: nonnegativeIntSchema.optional(), + localModelCallCacheMisses: nonnegativeIntSchema.optional(), localModelCallCacheWrites: nonnegativeIntSchema.optional(), + providerPromptCacheReadTokens: nonnegativeIntSchema.optional(), providerPromptCacheWriteTokens: nonnegativeIntSchema.optional(), + providerPromptCacheReadCostUSD: z.number().nonnegative().optional(), providerPromptCacheWriteCostUSD: z.number().nonnegative().optional(), + reasoningTokens: nonnegativeIntSchema.optional(), schemaInvalidCalls: nonnegativeIntSchema.optional(), schemaInvalidRecovered: nonnegativeIntSchema.optional(), + schemaInvalidUnrecovered: nonnegativeIntSchema.optional(), schemaRepairAttempts: nonnegativeIntSchema.optional(), schemaRepairRecovered: nonnegativeIntSchema.optional(), + deterministicSchemaRecovered: nonnegativeIntSchema.optional(), schemaRecoveryFailed: nonnegativeIntSchema.optional(), cacheHits: nonnegativeIntSchema.optional(), + cacheMisses: nonnegativeIntSchema.optional(), + stageLossCounts: z.object({ + "missed-before-candidate-generation": nonnegativeIntSchema, + "lost-at-verification": nonnegativeIntSchema, + "lost-at-composition": nonnegativeIntSchema, + "partial-match": nonnegativeIntSchema + }).strict() +}).strict(); + +const evalScoreSchema = z.object({ + status: z.enum(["pass", "fail", "error"]), expectationResults: z.array(evalExpectationResultSchema), budgetResults: z.array(evalBudgetResultSchema), + violations: z.array(z.object({ expectationId: z.string(), findingId: z.string(), publication: z.enum(["inline", "summary-only"]) }).strict()), + nearViolations: z.array(z.object({ expectationId: z.string(), findingId: z.string(), artifact: z.string() }).strict()), + metrics: evalMetricsSchema, error: z.object({ code: z.string(), message: z.string() }).strict().optional() +}).strict(); + +const evalRepeatExpectationSchema = z.object({ + expectationId: z.string().min(1), list: z.enum(["should_find", "should_find_candidate", "should_not_find"]), + tier: z.enum(["required", "optional"]), finalMatched: nonnegativeIntSchema, candidateMatched: nonnegativeIntSchema, + noteSurfaced: nonnegativeIntSchema, finalRecallRate: z.number().min(0).max(1), candidateRecallRate: z.number().min(0).max(1), + noteRate: z.number().min(0).max(1), lossHistogram: z.record(z.string(), nonnegativeIntSchema), fingerprintsStable: z.boolean().optional(), + distinctFingerprints: nonnegativeIntSchema.optional(), gate: z.object({ + minRecallRate: z.number().min(0).max(1).optional(), minCandidateRate: z.number().min(0).max(1).optional(), passed: z.boolean() + }).strict().optional() +}).strict(); + +const evalRepeatAggregateSchema = z.object({ + repeat: positiveIntSchema, + executions: z.array(z.object({ runDir: z.string().min(1), status: z.enum(["pass", "fail", "error"]) }).strict()), + expectations: z.array(evalRepeatExpectationSchema), + totals: z.object({ costUSD: z.number().nonnegative(), elapsedSeconds: z.number().nonnegative(), errors: nonnegativeIntSchema }).strict() +}).strict(); + +const evalRepoProvenanceSchema = z.object({ + root: z.string().min(1), baseSha: commitShaSchema.optional(), headSha: commitShaSchema.optional(), mergeBase: commitShaSchema.optional() +}).strict(); + +const codegenieRuntimeSchema = z.object({ + packageVersion: z.string().min(1), commit: commitShaSchema.optional(), shortCommit: z.string().min(1).optional(), + branch: z.string().min(1).optional(), dirty: z.boolean().optional(), source: z.enum(["build_env", "git", "package", "unknown"]) +}).strict(); + +const attentionRecordSchema = z.object({ + packetId: z.string().min(1), path: z.string().min(1), coverage: z.enum(["deep", "normal", "light"]), + coverageSource: z.union([z.enum(["planner", "deterministic_default"]), z.string().regex(/^escalated:.+/u)]), + ensemblePasses: nonnegativeIntSchema, directCandidates: nonnegativeIntSchema, promotedCandidates: nonnegativeIntSchema, + hintsEmitted: nonnegativeIntSchema, uncertaintiesEmitted: nonnegativeIntSchema, keptVerified: nonnegativeIntSchema, + published: nonnegativeIntSchema +}).strict(); +const attentionDroppedPathSchema = z.object({ path: z.string(), reason: z.string() }).strict(); +const attentionNoteSchema = z.object({ + question: z.string(), files: z.array(z.string()), symbols: z.array(z.string()), reason: z.string(), + confidence: z.enum(["high", "medium"]), sourcePacketIds: z.array(z.string()).optional() +}).strict(); +const attentionRawHintSchema = z.object({ + id: z.string(), source: z.enum(["follow_up_hint", "uncertainty"]), packetId: z.string(), question: z.string(), reason: z.string(), + confidence: z.enum(["high", "medium", "low"]), files: z.array(z.string()), originalFiles: z.array(z.string()), + droppedPaths: z.array(attentionDroppedPathSchema), symbols: z.array(z.string()), suggestedLenses: z.array(z.string()) +}).strict(); +const attentionGroupSchema = z.object({ + key: z.string(), noteIds: z.array(z.string()), question: z.string(), reason: z.string(), confidence: z.enum(["high", "medium"]), + files: z.array(z.string()), droppedPaths: z.array(attentionDroppedPathSchema), invalidPathCount: nonnegativeIntSchema, + symbols: z.array(z.string()), reasons: z.array(z.string()), packetIds: z.array(z.string()), + sources: z.array(z.enum(["follow_up_hint", "uncertainty"])), count: positiveIntSchema +}).strict(); +const humanAttentionArtifactSchema = z.union([ + z.array(z.object({ question: z.string(), files: z.array(z.string()), reasons: z.array(z.string()).optional(), reason: z.string().optional() }).strict()), + z.object({ + schemaVersion: z.literal(2), notes: z.array(attentionRawHintSchema), groups: z.array(attentionGroupSchema), + mergeStats: z.object({ exactDuplicateHints: nonnegativeIntSchema, nearDuplicateHints: nonnegativeIntSchema, nearDuplicateGroupsMerged: nonnegativeIntSchema }).strict(), + composerPromptGroupIds: z.array(z.string()), outputGroupIds: z.array(z.string()), outputNotes: z.array(attentionNoteSchema), + omittedCount: nonnegativeIntSchema, + suppressedByFindings: z.array(z.object({ groupKey: z.string(), noteIds: z.array(z.string()) }).strict()), + suppressedByVerification: z.array(z.object({ + groupKey: z.string(), noteIds: z.array(z.string()), candidateId: z.string(), verdict: z.enum(["keep", "reject", "revise", "incomplete"]), + reason: z.string(), verdictReason: z.string(), match: z.object({ + sharedFiles: z.array(z.string()), sharedSymbols: z.array(z.string()), sharedTerms: nonnegativeIntSchema, + similarity: z.number().min(0).max(1), questionMatched: z.boolean(), provenanceMatched: z.boolean() + }).strict() + }).strict()), + keptForOutputGroupIds: z.array(z.string()) + }).strict() +]); +const budgetLimitEventSchema = z.object({ + stage: z.number().int().min(0).max(11), reason: z.enum(["runtime_reserved_tail", "max_model_calls", "max_budget_tokens", "hard_timeout"]), + elapsedMs: nonnegativeIntSchema, kind: z.enum(["runtime", "model_calls", "tokens"]), actual: z.number().nonnegative(), + limit: z.number().nonnegative(), totalTokens: nonnegativeIntSchema, modelCalls: nonnegativeIntSchema, afterDispatchedCall: z.boolean() +}).strict(); +const contextPressureSummarySchema = z.object({ + toolBudgetRejections: nonnegativeIntSchema, toolBudgetRejectionsByStage: z.record(z.string().regex(/^\d+$/u), nonnegativeIntSchema), + toolBudgetExtensions: z.object({ + granted: nonnegativeIntSchema, denied: nonnegativeIntSchema, resultChars: nonnegativeIntSchema, + grantedByStage: z.record(z.string().regex(/^\d+$/u), nonnegativeIntSchema), + deniedByStage: z.record(z.string().regex(/^\d+$/u), nonnegativeIntSchema) + }).strict().optional(), + degradedToolResults: nonnegativeIntSchema, degradedToolResultsByStage: z.record(z.string().regex(/^\d+$/u), nonnegativeIntSchema), + degradedHunks: nonnegativeIntSchema, rejectionReasons: z.array(z.object({ reason: z.string(), count: positiveIntSchema }).strict()), + unresolvedNotes: z.object({ emitted: nonnegativeIntSchema, omitted: nonnegativeIntSchema }).strict() +}).strict(); +const budgetSummarySchema = z.object({ + completeness: z.enum(["complete", "partial"]), partialReasons: z.array(z.string()), multiplier: z.number().positive(), + configured: z.object({ timeoutMs: positiveIntSchema, maxModelCalls: positiveIntSchema.optional(), maxBudgetTokens: positiveIntSchema.optional() }).strict(), + effective: z.object({ timeoutMs: positiveIntSchema, maxModelCalls: positiveIntSchema.optional(), maxBudgetTokens: positiveIntSchema.optional() }).strict(), + usage: z.object({ + modelCalls: nonnegativeIntSchema, totalTokens: nonnegativeIntSchema, costUSD: z.number().nonnegative().optional(), + byStage: z.array(z.object({ stage: z.number().int().min(1).max(11), modelCalls: nonnegativeIntSchema, totalTokens: nonnegativeIntSchema }).strict()) + }).strict(), + overruns: z.array(budgetLimitEventSchema), dispatchBlocks: z.array(budgetLimitEventSchema), contextPressure: contextPressureSummarySchema.optional() +}).strict(); +const budgetStopSchema = z.object({ + reason: z.enum(["runtime_reserved_tail", "max_model_calls", "max_budget_tokens", "hard_timeout"]), stage: z.number().int().min(0).max(11), + elapsedMs: nonnegativeIntSchema, timeoutMs: positiveIntSchema, hardTimeoutMs: positiveIntSchema, remainingRuntimeMs: nonnegativeIntSchema, + reservedTailRuntimeMs: nonnegativeIntSchema, modelCalls: nonnegativeIntSchema, inFlightModelCalls: nonnegativeIntSchema, + projectedModelCalls: nonnegativeIntSchema, maxModelCalls: positiveIntSchema.optional(), remainingModelCalls: nonnegativeIntSchema.optional(), + reservedModelCalls: nonnegativeIntSchema.optional(), totalTokens: nonnegativeIntSchema, inFlightTokens: nonnegativeIntSchema, + projectedTokens: nonnegativeIntSchema, maxBudgetTokens: positiveIntSchema.optional(), remainingTokens: nonnegativeIntSchema.optional(), + reservedTokens: nonnegativeIntSchema.optional() +}).strict(); +const runReviewSummarySchema = z.object({ + mode: z.string(), target: z.unknown().nullable(), prNumber: positiveIntSchema.nullable(), baseRef: z.string().nullable(), headRef: z.string().nullable(), + baseSha: z.string().nullable(), headSha: z.string().nullable(), depth: z.string().nullable(), concurrency: positiveIntSchema.nullable(), + budgetBoost: z.number().positive().nullable(), llmMaxConcurrentCalls: positiveIntSchema.nullable(), lenses: z.array(z.string()), + format: z.string().nullable(), postGithubComments: z.boolean() +}).strict(); +const modelTotalsSchema = z.object({ + events: nonnegativeIntSchema, modelCallRecords: nonnegativeIntSchema, modelCalls: nonnegativeIntSchema, providerCalls: nonnegativeIntSchema, + toolCalls: nonnegativeIntSchema, toolResultCache: toolResultCacheSummarySchema, + inputTokens: nonnegativeIntSchema, uncachedInputTokens: nonnegativeIntSchema, cacheReadTokens: nonnegativeIntSchema, + cacheWriteTokens: nonnegativeIntSchema, billableInputTokens: nonnegativeIntSchema, outputTokens: nonnegativeIntSchema, + reasoningTokens: nonnegativeIntSchema, totalTokens: nonnegativeIntSchema, totalCostUSD: z.number().nonnegative(), + inputCostUSD: z.number().nonnegative(), outputCostUSD: z.number().nonnegative(), cacheReadCostUSD: z.number().nonnegative(), + cacheWriteCostUSD: z.number().nonnegative(), costBreakdown: costBreakdownSchema, unknownCostCalls: nonnegativeIntSchema, + cache: cacheCountsSchema, localModelCallCache: cacheCountsSchema, providerPromptCache: providerPromptCacheSchema, + retryAttempts: nonnegativeIntSchema, repairCalls: nonnegativeIntSchema, schemaInvalidCalls: nonnegativeIntSchema, + schemaRecovery: schemaRecoverySummarySchema, + stage7SchemaRepair: z.object({ + candidateInvalidSubmits: nonnegativeIntSchema, noFindingInvalidSubmits: nonnegativeIntSchema, cleanupAttempted: nonnegativeIntSchema, + cleanupRecovered: nonnegativeIntSchema, cleanupRejected: nonnegativeIntSchema, compactRepairScheduled: nonnegativeIntSchema, + appendRepairScheduled: nonnegativeIntSchema, repairRecovered: nonnegativeIntSchema, repairFailed: nonnegativeIntSchema, + repairPromptChars: nonnegativeIntSchema, compactRepairPromptChars: nonnegativeIntSchema, appendRepairPromptChars: nonnegativeIntSchema, + actualRepairCalls: nonnegativeIntSchema, actualRepairPromptChars: nonnegativeIntSchema + }).strict(), + logOverflow: z.object({ droppedDebugInfo: nonnegativeIntSchema, droppedWarnError: nonnegativeIntSchema }).strict(), + filesChanged: nonnegativeIntSchema, hunks: nonnegativeIntSchema, packets: nonnegativeIntSchema, packetReviews: nonnegativeIntSchema, + candidates: nonnegativeIntSchema, verified: nonnegativeIntSchema, finalFindings: nonnegativeIntSchema, postedComments: nonnegativeIntSchema +}).strict(); +const evalRunTelemetrySchema = z.object({ + schemaVersion: z.literal(1), runId: z.string().min(1), codegenieVersion: z.string(), codegenieRuntime: codegenieRuntimeSchema, + nodeVersion: z.string(), argv: z.array(z.string()), repoRoot: z.string().nullable(), review: runReviewSummarySchema, + startedAt: z.string(), finishedAt: z.string(), completedAt: z.string(), durationMs: nonnegativeIntSchema, + outcome: z.object({ status: z.enum(["completed_full", "completed_partial", "failed"]), errorCode: z.string().nullable(), exitCode: z.number().int(), budgetStop: budgetStopSchema.nullable() }).strict(), + budgetStop: budgetStopSchema.optional(), totals: modelTotalsSchema +}).strict(); +const telemetryStageSummarySchema = z.object({ + events: nonnegativeIntSchema, + levels: z.object({ debug: nonnegativeIntSchema, info: nonnegativeIntSchema, warn: nonnegativeIntSchema, error: nonnegativeIntSchema }).strict(), + cache: cacheCountsSchema, startedAt: z.string().optional(), completedAt: z.string().optional(), runtimeMs: nonnegativeIntSchema, + schemaRecovery: schemaRecoveryCountersSchema +}).strict(); +const pipelineTelemetrySummaryShape = { + workers: z.object({ started: nonnegativeIntSchema, completed: nonnegativeIntSchema, failed: nonnegativeIntSchema, retried: nonnegativeIntSchema, timedOut: nonnegativeIntSchema }).strict(), + packets: z.object({ generated: nonnegativeIntSchema, reviewed: nonnegativeIntSchema, failed: nonnegativeIntSchema, degraded: nonnegativeIntSchema }).strict(), + lenses: z.object({ selected: nonnegativeIntSchema, byLens: z.record(z.string(), nonnegativeIntSchema) }).strict(), + coverage: z.object({ + byLevel: z.object({ deep: nonnegativeIntSchema, normal: nonnegativeIntSchema, light: nonnegativeIntSchema, skip: nonnegativeIntSchema }).strict(), + hunks: z.object({ total: nonnegativeIntSchema, reviewed: nonnegativeIntSchema, skipped: nonnegativeIntSchema, failed: nonnegativeIntSchema, degraded: nonnegativeIntSchema }).strict() + }).strict(), + candidates: z.object({ + generated: nonnegativeIntSchema, gateRejected: nonnegativeIntSchema, verificationScheduled: nonnegativeIntSchema, + verificationBudgetLimited: nonnegativeIntSchema, clusteredDuplicates: nonnegativeIntSchema, verificationRepresentatives: nonnegativeIntSchema, + lowConfidenceSuppressed: nonnegativeIntSchema, lowConfidenceEvidenceEligible: nonnegativeIntSchema, + lowConfidenceEvidenceScheduled: nonnegativeIntSchema, lowConfidenceEvidenceLaneLimited: nonnegativeIntSchema, + lowConfidenceEvidenceKept: nonnegativeIntSchema, lowConfidenceEvidenceRejected: nonnegativeIntSchema, + lowConfidenceEvidenceIncomplete: nonnegativeIntSchema + }).strict(), + verdicts: z.object({ accept: nonnegativeIntSchema, revise: nonnegativeIntSchema, reject: nonnegativeIntSchema, incomplete: nonnegativeIntSchema }).strict(), + dedup: z.object({ clusters: nonnegativeIntSchema, duplicates: nonnegativeIntSchema, suppressed: nonnegativeIntSchema }).strict(), + finalSelection: z.object({ published: nonnegativeIntSchema, merged: nonnegativeIntSchema, suppressed: nonnegativeIntSchema, finalFindings: nonnegativeIntSchema, compositionMode: z.string().nullable(), fallbackReason: z.string().nullable() }).strict(), + posting: z.object({ attempted: nonnegativeIntSchema, postedComments: nonnegativeIntSchema, skippedDuplicates: nonnegativeIntSchema, failed: nonnegativeIntSchema }).strict() +}; +const telemetrySummarySchema = z.object({ + schemaVersion: z.literal(1), runId: z.string().min(1), codegenieRuntime: codegenieRuntimeSchema, startedAt: z.string(), finishedAt: z.string(), + completedAt: z.string(), durationMs: nonnegativeIntSchema, logLevel: z.enum(["debug", "info", "warn", "error"]), debugTrace: z.boolean(), + events: nonnegativeIntSchema, logs: z.object({ bufferedOverflow: z.object({ droppedDebugInfo: nonnegativeIntSchema, droppedWarnError: nonnegativeIntSchema }).strict() }).strict(), + budgetStop: budgetStopSchema.optional(), totals: modelTotalsSchema, stages: z.record(z.string().regex(/^\d+$/u), telemetryStageSummarySchema), + ...pipelineTelemetrySummaryShape, + schemaRecovery: schemaRecoverySummarySchema, + schemaRepair: z.object({ stage7: modelTotalsSchema.shape.stage7SchemaRepair }).strict(), + modelCalls: modelCallsSummarySchema, toolCalls: toolCallsSummarySchema +}).strict(); + +const invocationIdSchema = z.string().regex(/^[a-zA-Z0-9_-]+$/u); +const evalInvocationReferenceSchema = z.object({ + id: invocationIdSchema, caseIndex: nonnegativeIntSchema, manifest: z.string().regex(/^invocations\/[a-zA-Z0-9_-]+\.json$/u) +}).strict(); + +const evalInvocationCaseSchema = z.object({ + caseIndex: nonnegativeIntSchema, caseName: z.string().min(1), caseHash: z.string().min(1), caseFile: z.string().min(1) +}).strict(); + +const evalInvocationRunSchema = z.object({ + caseIndex: nonnegativeIntSchema, caseName: z.string().min(1), caseHash: z.string().min(1), runNumber: positiveIntSchema, + logsRoot: z.string().min(1).refine((value) => path.isAbsolute(value), "logsRoot must be absolute"), + runPath: z.string().regex(/^\d+$/u) +}).strict(); + +const evalInvocationManifestSchema = z.object({ + schemaVersion: z.literal(1), invocationId: invocationIdSchema, + suiteDir: z.string().min(1).refine((value) => path.isAbsolute(value), "suiteDir must be absolute"), + status: z.enum(["running", "complete"]), + startedAt: z.string().min(1), completedAt: z.string().min(1).optional(), cases: z.array(evalInvocationCaseSchema).min(1), runs: z.array(evalInvocationRunSchema) +}).strict(); + +const evalRunInfoSchema = z.object({ + runNumber: positiveIntSchema, caseName: z.string().min(1), caseFile: z.string().optional(), caseHash: z.string().min(1), + caseSnapshot: evalCaseArtifactSchema, mode: z.enum(["live", "replay"]), repeats: evalRepeatAggregateSchema.optional(), + replay: z.object({ sourceArtifacts: z.string().min(1), caseSource: z.enum(["yaml", "snapshot"]) }).strict().optional(), + repo: evalRepoProvenanceSchema.optional(), reviewRunId: z.string().optional(), codegenieRuntime: codegenieRuntimeSchema.optional(), + invocation: evalInvocationReferenceSchema.optional(), + cache: z.object({ enabled: z.boolean(), source: z.enum(["cli", "case", "config"]), dir: z.string().optional() }).strict(), + effectiveConfig: z.object({ + review: z.object({ + concurrency: positiveIntSchema, timeoutMs: positiveIntSchema, verify: z.boolean(), + minSeverity: z.enum(["critical", "high", "medium", "low"]).optional(), maxFindings: positiveIntSchema, + softCommentCap: positiveIntSchema, minConfidence: z.enum(["high", "medium", "low"]), + minInlineConfidence: z.enum(["high", "medium", "low"]), packSameFileHunks: z.boolean(), + packedToolBudgetMode: z.enum(["base", "atom-scaled"]), maxBudgetTokens: positiveIntSchema.optional() + }).strict(), + llm: z.object({ + provider: z.string().optional(), model: z.string().optional(), reasoning: z.string().optional(), maxConcurrentCalls: positiveIntSchema + }).strict() + }).strict().optional(), + startedAt: z.string().min(1), finishedAt: z.string().min(1), score: evalScoreSchema +}).strict(); + +const coverageArtifactSchema = z.object({ + status: z.object({ + totalHunks: nonnegativeIntSchema, reviewedHunks: nonnegativeIntSchema, skippedHunks: nonnegativeIntSchema, failedHunks: nonnegativeIntSchema, + coverageByLevel: z.object({ deep: nonnegativeIntSchema, normal: nonnegativeIntSchema, light: nonnegativeIntSchema, skip: nonnegativeIntSchema }).strict(), + degradedPlanning: z.boolean(), budgetStopped: z.boolean(), budgetStop: jsonObjectSchema.optional(), + unreviewedHunksByPath: z.array(jsonObjectSchema).optional(), verificationIncompleteCount: nonnegativeIntSchema, + verificationSkipped: z.boolean().optional(), partial: z.boolean(), reasons: z.array(z.string()) + }).strict(), + records: z.array(z.object({ + hunkId: z.string().min(1), path: z.string().min(1), coverage: z.enum(["deep", "normal", "light", "skip"]), + source: z.enum(["planner", "deterministic_default", "config"]), status: z.enum(["reviewed", "skipped", "review_failed", "degraded"]), + reason: z.string().optional() + }).strict()) +}).strict(); + +export type PacketPackingArtifactKind = + | "packet" | "event" | "model-call" | "tool-call" | "coverage" | "eval-info" + | "attention" | "human-attention" | "budget-summary" | "cost-profile" | "model-summary" | "tool-summary" | "run-summary" | "telemetry-summary"; + +export function validatePacketPackingArtifact(kind: PacketPackingArtifactKind, value: unknown): void { + const schemas: Record = { + packet: reviewPacketSchema, + event: telemetryEventSchema, + "model-call": llmCallRecordSchema, + "tool-call": toolCallRecordSchema, + coverage: coverageArtifactSchema, + "eval-info": evalRunInfoSchema, + attention: z.array(attentionRecordSchema), + "human-attention": humanAttentionArtifactSchema, + "budget-summary": budgetSummarySchema, + "cost-profile": costProfileSchema, + "model-summary": modelCallsSummarySchema, + "tool-summary": toolCallsSummarySchema, + "run-summary": evalRunTelemetrySchema, + "telemetry-summary": telemetrySummarySchema + }; + const parsed = schemas[kind].safeParse(value); + assertReport(parsed.success, "corrupt_artifact_schema", `invalid ${kind} artifact`, { + issues: parsed.success ? [] : parsed.error.issues.map((issue) => ({ path: issue.path.join("."), code: issue.code, message: issue.message })) + }); +} + +const sameFilePackingEventDataSchema = z.object({ + packetId: z.string().min(1), + atomIds: z.array(z.string().min(1)).min(1), + standaloneProfiles: z.array(z.enum(["simple", "standard", "investigate"])).min(1), + sourceAtomCount: z.number().int().positive(), + hunkCount: z.number().int().positive(), + effectiveCoverage: z.enum(["deep", "normal", "light"]), + requestedLensSignature: z.string(), + capUsage: z.object({ + hunks: z.number().int().nonnegative(), + maxHunks: z.number().int().positive(), + patchChars: z.number().int().nonnegative(), + maxPatchChars: z.number().int().positive() + }), + derivedPackedProfile: z.enum(["simple", "standard", "investigate"]), + profileFloor: z.enum(["simple", "standard", "investigate"]), + effectiveProfile: z.enum(["simple", "standard", "investigate"]), + profileFloorApplied: z.boolean(), + plannerLensesPreserved: z.boolean(), + toolBudgetMode: z.enum(["base", "atom-scaled"]), + baseToolBudget: toolBudgetSchema, + effectiveToolBudget: toolBudgetSchema +}).strict(); + +type PackingEventData = z.infer & { + baseToolBudget: ToolBudget; + effectiveToolBudget: ToolBudget; +}; + +export type ReportFailure = { + code: string; + message: string; + context?: Record; +}; + +export type CapturedTelemetry = { + events: Array>; + modelCalls: Array>; + toolCalls: Array>; +}; + +export type ReplayAnalysisInput = { + runId: string; + recordedPackets: ReviewPacket[]; + offPackets: ReviewPacket[]; + onPackets: ReviewPacket[]; + onEvents: Array | TelemetryEvent>; + fileFacts: FileFacts[]; + diff: UnifiedDiff; + plan: ReviewPlan; + expectedRefs?: { base: string; head: string }; + actualRefs?: { base: string; head: string }; + flagOffParityPackets?: ReviewPacket[]; + flagOffParityMigrations?: Array<{ code: string; packets: number; explanation: string }>; + modelCallsObserved?: number; +}; + +export type ReplayRow = { + runId: string; + offPackets: number; + onPackets: number; + reductionPercent: number; + sourceAtoms: number; + packedMultiAtomPackets: number; + reviewableHunks: number; + newCoveragePromotions: number; + hunkBijectionFailures: number; + atomInvariantFailures: number; + capViolations: number; + lensDrops: number; + highPriorityNoteOmissions: number; + deepContextDowngrades: number; + relatedContextOmissions: number; + derivedProfileDowngrades: number; + profileFloorApplications: number; + effectiveProfileDowngrades: number; + effectiveBudgetDowngrades: number; + invalidDispatchRanks: number; + modelCallsObserved: number; + schedulingMoves: number; + proxyPackedPackets: number; + countReconciliation: "historical-target" | "real-patch-74" | "real-patch-76" | "not-motivating-run"; + flagOffParityDifferences: { + missingPacketIds: string[]; + extraPacketIds: string[]; + changedPackets: Array<{ + packetId: string; + paths: string[]; + samples: Array<{ path: string; recorded: ValueFingerprint; rebuilt: ValueFingerprint }>; + }>; + historicalMigrations: Array<{ code: string; packets: number; explanation: string }>; + }; + contextComparison: { + contextTruncationsOff: number; + contextTruncationsOn: number; + relatedContextOmissions: number; + attentionNoteOmissions: number; + lensDrops: number; + }; + distribution: { + offHunksPerPacket: Record; + onHunksPerPacket: Record; + sourceAtomsPerOnPacket: Record; + onCoverage: Record; + onProfiles: Record; + onBudgetModes: Record; + maxOnHunks: number; + eligiblePackingPackets: number; + bypassPackets: number; + maxEligiblePackingPatchChars: number; + }; + packetMembership: Array<{ + packetId: string; + path: string; + hunkIds: string[]; + coverage: string; + lenses: string[]; + profile: string; + budget: ToolBudget; + dispatchRank: [number, number]; + atomIds: string[]; + }>; + failures: ReportFailure[]; +}; + +type ValueFingerprint = { + kind: "string" | "json"; + length: number; + sha256: string; +}; + +export type EvalExecutionInput = { + repeat: number; + score: EvalScore; + telemetryDir: string; + packets: ReviewPacket[]; + events: TelemetryEvent[]; + modelCalls: LlmCallRecord[]; + toolCalls: ToolCallRecord[]; + fileFacts: FileFacts[]; + diff: UnifiedDiff; + plan: ReviewPlan; + candidateFindings: CandidateFinding[]; + verification: EvalVerificationRecord[]; + finalSelection: EvalSelectionRecord[]; + finalSelectionArtifact: FinalSelectionArtifact; + finalFindings: FinalFinding[]; + scoringArtifacts: EvalArtifacts; + summaryArtifacts?: { + attention: unknown; + humanAttention: unknown; + budget: unknown; + cost: unknown; + model: unknown; + tool: unknown; + run: unknown; + telemetry: unknown; + }; + reviewedHunkIds: string[]; + wallTimeSeconds: number; +}; + +export type EvalCaseRunInput = { + runNumber: number; + runDir: string; + info: EvalRunInfo; + declaredCase: EvalCase; + invocationManifest?: EvalInvocationManifest; + executions: EvalExecutionInput[]; +}; + +export type CohortSelection = { + id: string; + runs: EvalCaseRunInput[]; +}; + +export class PacketPackingReportError extends Error { + readonly failures: ReportFailure[]; + + constructor(message: string, failures: ReportFailure[]) { + super(message); + this.name = "PacketPackingReportError"; + this.failures = failures; + } +} + +function failure(code: string, message: string, context?: Record): ReportFailure { + return { code, message, ...(context === undefined ? {} : { context }) }; +} + +function fail(code: string, message: string, context?: Record): never { + throw new PacketPackingReportError(message, [failure(code, message, context)]); +} + +function assertReport(condition: unknown, code: string, message: string, context?: Record): asserts condition { + if (!condition) { + fail(code, message, context); + } +} + +function isRecord(value: unknown): value is Record { + return value !== null && typeof value === "object" && !Array.isArray(value); +} + +function asFiniteNumber(value: unknown): number | undefined { + return typeof value === "number" && Number.isFinite(value) ? value : undefined; +} + +function stableValue(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map(stableValue); + } + if (!isRecord(value)) { + return value; + } + return Object.fromEntries(Object.keys(value).sort().map((key) => [key, stableValue(value[key])])); +} + +function stableJson(value: unknown): string { + return JSON.stringify(stableValue(value)); +} + +function unique(values: T[]): T[] { + return [...new Set(values)]; +} + +function sortedUnique(values: string[]): string[] { + return unique(values.filter((value) => value.length > 0)).sort(); +} + +function sum(values: number[]): number { + return values.reduce((total, value) => total + value, 0); +} + +function countDistribution(values: Array): Record { + const entries = new Map(); + for (const value of values) { + const key = String(value); + entries.set(key, (entries.get(key) ?? 0) + 1); + } + return Object.fromEntries([...entries].sort(([left], [right]) => left.localeCompare(right, undefined, { numeric: true }))); +} + +function atomIdForPacket(packet: ReviewPacket): string { + return sha256Hex(`hunk-first\n${packet.hunks.map((hunk) => hunk.hunkId).join("\n")}`); +} + +function changedLines(packet: ReviewPacket): number { + return packet.hunks.reduce((total, hunk) => total + hunk.changedNewLineNumbers.length + hunk.changedOldLineNumbers.length, 0); +} + +function budgetDowngraded(actual: ToolBudget, floor: ToolBudget): boolean { + if ( + actual.maxToolCalls < floor.maxToolCalls || + actual.maxInvestigationRounds < floor.maxInvestigationRounds || + actual.maxResultChars < floor.maxResultChars + ) { + return true; + } + const actualExtension = actual.sourceExtension; + const floorExtension = floor.sourceExtension; + return floorExtension !== undefined && ( + actualExtension === undefined || + actualExtension.maxToolCalls < floorExtension.maxToolCalls || + actualExtension.maxResultChars < floorExtension.maxResultChars + ); +} + +function packingEvents( + events: Array | TelemetryEvent>, + failures: ReportFailure[] +): Map { + const byPacket = new Map(); + for (const event of events) { + if (event.stage !== 6 || event.message !== "same_file_atoms_packed") { + continue; + } + const result = sameFilePackingEventDataSchema.safeParse(event.data); + if (!result.success) { + failures.push(failure("missing_or_corrupt_treatment_telemetry", "same_file_atoms_packed telemetry is invalid", { + issues: result.error.issues + })); + continue; + } + const data = result.data as PackingEventData; + if (byPacket.has(data.packetId)) { + failures.push(failure("duplicate_treatment_telemetry", `duplicate same_file_atoms_packed event for ${data.packetId}`)); + continue; + } + byPacket.set(data.packetId, data); + } + return byPacket; +} + +function hunkPacketMap(packets: ReviewPacket[], failures: ReportFailure[], side: string): Map { + const result = new Map(); + for (const packet of packets) { + for (const hunk of packet.hunks) { + if (result.has(hunk.hunkId)) { + failures.push(failure("hunk_bijection", `${side} duplicates hunk ${hunk.hunkId}`, { packetId: packet.id })); + } + result.set(hunk.hunkId, packet); + } + } + return result; +} + +function contextTruncated(packet: ReviewPacket): boolean { + return packet.contextDegradationReasons?.some((reason) => reason.includes("context truncated")) === true || + packet.contextText.includes("[... content truncated to fit packet context budget ...]"); +} + +function packetSourcePositions(diff: UnifiedDiff): Map { + const positions = new Map(); + let position = 0; + for (const file of diff.files) { + for (const hunk of file.hunks) { + positions.set(hunk.id, position); + position += 1; + } + } + return positions; +} + +function actualPatchCharsForHunk(diff: UnifiedDiff): Map { + const values = new Map(); + for (const file of diff.files) { + for (const hunk of file.hunks) { + values.set(hunk.id, renderDiffLines(hunk.lines).length); + } + } + return values; +} + +function renderDiffLines(lines: DiffLine[]): string { + return lines.map((line) => { + const oldLine = line.oldLineNumber === undefined ? " " : String(line.oldLineNumber); + const newLine = line.newLineNumber === undefined ? " " : String(line.newLineNumber); + const prefix = line.kind === "add" ? "+" : line.kind === "delete" ? "-" : " "; + return `${oldLine.padStart(4)} ${newLine.padStart(4)} ${prefix}${line.content}`; + }).join("\n"); +} + +function legacyDiffParityView(diff: UnifiedDiff): unknown { + return { + files: diff.files.map((file) => ({ + ...file, + hunks: file.hunks.map(({ hunkHash: _hunkHash, id: _id, ...hunk }) => hunk) + })) + }; +} + +export function validateRecordedDiffParity(runId: string, recorded: UnifiedDiff, rebuilt: UnifiedDiff): void { + const recordedHunks = recorded.files.flatMap((file) => file.hunks); + const missingHunkHashes = recordedHunks.filter((hunk) => hunk.hunkHash === undefined).length; + assertReport(missingHunkHashes === 0 || missingHunkHashes === recordedHunks.length, "recorded_diff_mismatch", `recorded diff partially persists hunk hashes for ${runId}`); + if (missingHunkHashes > 0) { + assertReport(LEGACY_DIFF_SCHEMA_RUN_IDS.has(runId), "unpermitted_legacy_diff_schema", `run ${runId} is not allowlisted for the legacy hunk identity schema`); + } + const rebuiltParity = missingHunkHashes === 0 ? rebuilt : legacyDiffParityView(rebuilt); + const recordedParity = missingHunkHashes === 0 ? recorded : legacyDiffParityView(recorded); + assertReport(stableJson(rebuiltParity) === stableJson(recordedParity), "recorded_diff_mismatch", `rebuilt diff content differs for ${runId}`, { + recorded: valueFingerprint(recorded), + actual: valueFingerprint(rebuilt) + }); +} + +function normalizedRequestedLenses(plan: ReviewPlan, hunkIds: string[]): string { + const ids = new Set(hunkIds); + return JSON.stringify(sortedUnique(plan.coverage.filter((decision) => ids.has(decision.hunkId)).flatMap((decision) => decision.lenses ?? []))); +} + +function proxyPackedPacketCount( + offPackets: ReviewPacket[], + events: Map, + plan: ReviewPlan, + diff: UnifiedDiff +): { proxy: number; actualMeasurement: number; changedSplits: number } { + const referencedAtomIds = new Set([...events.values()].flatMap((event) => event.atomIds)); + const eligible = offPackets.filter((packet) => referencedAtomIds.has(atomIdForPacket(packet))); + const ineligible = offPackets.length - eligible.length; + const realPatchChars = actualPatchCharsForHunk(diff); + type Group = { hunks: number; proxyChars: number; realChars: number }; + const proxyGroups = new Map(); + const realGroups = new Map(); + for (const packet of eligible) { + const key = `${packet.path}\0${packet.coverage}\0${normalizedRequestedLenses(plan, packet.hunks.map((hunk) => hunk.hunkId))}`; + const proxyChars = sum(packet.hunks.map((hunk) => hunk.contentWithLineNumbers.length)); + const realChars = sum(packet.hunks.map((hunk) => realPatchChars.get(hunk.hunkId) ?? Number.MAX_SAFE_INTEGER)); + appendGreedy(proxyGroups, key, packet.hunks.length, proxyChars, realChars, "proxyChars"); + appendGreedy(realGroups, key, packet.hunks.length, proxyChars, realChars, "realChars"); + } + const proxy = ineligible + sum([...proxyGroups.values()].map((groups) => groups.length)); + const actualMeasurement = ineligible + sum([...realGroups.values()].map((groups) => groups.length)); + return { proxy, actualMeasurement, changedSplits: Math.abs(proxy - actualMeasurement) }; +} + +function appendGreedy( + groupsByKey: Map>, + key: string, + hunks: number, + proxyChars: number, + realChars: number, + measurement: "proxyChars" | "realChars" +): void { + const groups = groupsByKey.get(key) ?? []; + const current = groups.at(-1); + if (current !== undefined && current.hunks + hunks <= MAX_HUNKS_PER_PACKET && current[measurement] + (measurement === "proxyChars" ? proxyChars : realChars) <= MAX_PATCH_CHARS) { + current.hunks += hunks; + current.proxyChars += proxyChars; + current.realChars += realChars; + } else { + groups.push({ hunks, proxyChars, realChars }); + } + groupsByKey.set(key, groups); +} + +export function analyzeReplayComparison(input: ReplayAnalysisInput): ReplayRow { + const failures: ReportFailure[] = []; + const modelCallsObserved = input.modelCallsObserved ?? 0; + if (modelCallsObserved !== 0) { + failures.push(failure("replay_model_call", "deterministic replay recorded model calls", { modelCallsObserved })); + } + if (input.expectedRefs !== undefined && input.actualRefs !== undefined) { + if (input.expectedRefs.base !== input.actualRefs.base || input.expectedRefs.head !== input.actualRefs.head) { + failures.push(failure("stale_replay_refs", "recorded replay refs do not match resolved refs", { + expected: input.expectedRefs, + actual: input.actualRefs + })); + } + } + + const recordedById = new Map(input.recordedPackets.map((packet) => [packet.id, packet])); + const parityOffById = new Map((input.flagOffParityPackets ?? input.offPackets).map((packet) => [packet.id, packet])); + const flagOffParityDifferences = { + missingPacketIds: [...recordedById.keys()].filter((id) => !parityOffById.has(id)).sort(), + extraPacketIds: [...parityOffById.keys()].filter((id) => !recordedById.has(id)).sort(), + changedPackets: [...recordedById].flatMap(([id, packet]) => { + const current = parityOffById.get(id); + return current === undefined || stableJson(packet) === stableJson(current) + ? [] + : [{ + packetId: id, + paths: changedValuePaths(packet, current).slice(0, 50), + samples: changedValueSamples(packet, current).slice(0, 10) + }]; + }), + historicalMigrations: input.flagOffParityMigrations ?? [] + }; + if (recordedById.size !== parityOffById.size || [...recordedById].some(([id, packet]) => stableJson(packet) !== stableJson(parityOffById.get(id)))) { + failures.push(failure("flag_off_parity", "packing-off packets are not artifact-identical to the recorded Stage-6 packets", flagOffParityDifferences)); + } + + const offById = new Map(input.offPackets.map((packet) => [packet.id, packet])); + const offByHunk = hunkPacketMap(input.offPackets, failures, "packing-off"); + const onByHunk = hunkPacketMap(input.onPackets, failures, "packing-on"); + const expectedHunks = [...offByHunk.keys()].sort(); + const actualHunks = [...onByHunk.keys()].sort(); + if (stableJson(expectedHunks) !== stableJson(actualHunks)) { + failures.push(failure("hunk_bijection", "packing-on hunk set differs from packing-off", { + missing: expectedHunks.filter((id) => !onByHunk.has(id)), + extra: actualHunks.filter((id) => !offByHunk.has(id)) + })); + } + + const positions = packetSourcePositions(input.diff); + const patchCharsByHunk = actualPatchCharsForHunk(input.diff); + const factsByPath = new Map(input.fileFacts.map((facts) => [facts.path, facts])); + const eventByPacket = packingEvents(input.onEvents, failures); + const offAtomById = new Map(input.offPackets.map((packet) => [atomIdForPacket(packet), packet])); + const observedAtomIds: string[] = []; + let atomInvariantFailures = 0; + let capViolations = 0; + let newCoveragePromotions = 0; + let lensDrops = 0; + let highPriorityNoteOmissions = 0; + let deepContextDowngrades = 0; + let relatedContextOmissions = 0; + let derivedProfileDowngrades = 0; + let profileFloorApplications = 0; + let effectiveProfileDowngrades = 0; + let effectiveBudgetDowngrades = 0; + let invalidDispatchRanks = 0; + let previousPacketSourcePosition = -1; + const eligiblePackingPatchChars: number[] = []; + + for (const packet of input.offPackets) { + const facts = factsByPath.get(packet.path); + if (facts === undefined) { + invalidDispatchRanks += 1; + failures.push(failure("missing_file_facts", `missing file facts for packing-off packet ${packet.path}`)); + continue; + } + const expectedRank = packetDispatchRank(packet.path, facts, changedLines(packet)); + if (stableJson(packet.dispatchRank) !== stableJson(expectedRank)) { + invalidDispatchRanks += 1; + failures.push(failure("invalid_dispatch_rank", `packing-off packet ${packet.id} has invalid dispatch rank`, { + expected: expectedRank, + actual: packet.dispatchRank + })); + } + } + + for (const packet of input.onPackets) { + const sourcePositions = packet.hunks.map((hunk) => positions.get(hunk.hunkId) ?? Number.MAX_SAFE_INTEGER); + if (sourcePositions.some((position, index) => index > 0 && position < (sourcePositions[index - 1] ?? -1))) { + atomInvariantFailures += 1; + failures.push(failure("source_order", `packet ${packet.id} reorders hunks`)); + } + const firstSourcePosition = Math.min(...sourcePositions); + if (firstSourcePosition < previousPacketSourcePosition) { + atomInvariantFailures += 1; + failures.push(failure("source_packet_order", `packet ${packet.id} appears before an earlier source packet`, { + previousPacketSourcePosition, + firstSourcePosition + })); + } + previousPacketSourcePosition = firstSourcePosition; + const facts = factsByPath.get(packet.path); + if (facts === undefined) { + invalidDispatchRanks += 1; + failures.push(failure("missing_file_facts", `missing file facts for ${packet.path}`)); + } else { + const expectedRank = packetDispatchRank(packet.path, facts, changedLines(packet)); + if (stableJson(packet.dispatchRank) !== stableJson(expectedRank)) { + invalidDispatchRanks += 1; + failures.push(failure("invalid_dispatch_rank", `packet ${packet.id} has invalid dispatch rank`, { + expected: expectedRank, + actual: packet.dispatchRank + })); + } + } + + const event = eventByPacket.get(packet.id); + const atoms = event?.atomIds.map((id) => offAtomById.get(id)).filter((atom): atom is ReviewPacket => atom !== undefined) ?? []; + if (event !== undefined) { + observedAtomIds.push(...event.atomIds); + const actualPatchChars = sum(packet.hunks.map((hunk) => patchCharsByHunk.get(hunk.hunkId) ?? Number.MAX_SAFE_INTEGER)); + const combinesMultipleAtoms = event.sourceAtomCount > 1; + if (combinesMultipleAtoms) { + eligiblePackingPatchChars.push(actualPatchChars); + } + if (event.atomIds.length !== event.sourceAtomCount || event.standaloneProfiles.length !== event.sourceAtomCount || atoms.length !== event.sourceAtomCount) { + atomInvariantFailures += 1; + failures.push(failure("atom_bijection", `packet ${packet.id} has missing or inconsistent atom provenance`, { + atomIds: event.atomIds, + sourceAtomCount: event.sourceAtomCount, + resolvedAtoms: atoms.length + })); + } + const flattened = atoms.flatMap((atom) => atom.hunks.map((hunk) => hunk.hunkId)); + if (stableJson(flattened) !== stableJson(packet.hunks.map((hunk) => hunk.hunkId))) { + atomInvariantFailures += 1; + failures.push(failure("atom_split_or_reorder", `packet ${packet.id} does not preserve source atom membership/order`)); + } + if ( + event.hunkCount !== packet.hunks.length || + event.capUsage.hunks !== packet.hunks.length || + event.capUsage.maxHunks !== MAX_HUNKS_PER_PACKET || + event.capUsage.maxPatchChars !== MAX_PATCH_CHARS || + event.capUsage.hunks > MAX_HUNKS_PER_PACKET || + event.capUsage.patchChars > MAX_PATCH_CHARS || + (combinesMultipleAtoms && (event.capUsage.patchChars !== actualPatchChars || actualPatchChars > MAX_PATCH_CHARS)) + ) { + capViolations += 1; + failures.push(failure("packet_cap", `packet ${packet.id} violates packing caps`, { + capUsage: event.capUsage, + actualPatchChars + })); + } + if (atoms.some((atom) => atom.path !== packet.path || atom.language !== packet.language)) { + atomInvariantFailures += 1; + failures.push(failure("file_language_boundary", `packet ${packet.id} crosses a file or language boundary`)); + } + if (atoms.some((atom) => atom.coverage !== packet.coverage) || event.effectiveCoverage !== packet.coverage) { + newCoveragePromotions += 1; + failures.push(failure("coverage_promotion", `packet ${packet.id} promotes or mixes atom coverage`)); + } + const missingLenses = sortedUnique(atoms.flatMap((atom) => atom.lenses)).filter((lens) => !packet.lenses.includes(lens)); + if (!event.plannerLensesPreserved || missingLenses.length > 0) { + lensDrops += missingLenses.length > 0 ? missingLenses.length : 1; + failures.push(failure("planner_lens_drop", `packet ${packet.id} drops a standalone routed lens`, { missingLenses })); + } + const floor = event.standaloneProfiles.reduce((highest, profile) => PROFILE_RANK[profile] > PROFILE_RANK[highest] ? profile : highest, "simple" as ReviewProfile); + if (event.profileFloor !== floor) { + failures.push(failure("profile_floor_mismatch", `packet ${packet.id} reports the wrong profile floor`, { expected: floor, actual: event.profileFloor })); + } + if (PROFILE_RANK[event.derivedPackedProfile] < PROFILE_RANK[floor]) { + derivedProfileDowngrades += 1; + } + if (event.profileFloorApplied) { + profileFloorApplications += 1; + } + if (PROFILE_RANK[event.effectiveProfile] < PROFILE_RANK[floor] || packet.reviewProfile !== event.effectiveProfile) { + effectiveProfileDowngrades += 1; + failures.push(failure("effective_profile_downgrade", `packet ${packet.id} falls below its standalone profile floor`, { + floor, + effective: event.effectiveProfile + })); + } + if (atoms.some((atom) => budgetDowngraded(event.effectiveToolBudget, atom.toolBudget)) || stableJson(event.effectiveToolBudget) !== stableJson(packet.toolBudget)) { + effectiveBudgetDowngrades += 1; + failures.push(failure("effective_budget_downgrade", `packet ${packet.id} has a downgraded or inconsistent effective tool budget`)); + } + } else { + observedAtomIds.push(atomIdForPacket(packet)); + const off = offById.get(packet.id); + if (off === undefined || stableJson(off) !== stableJson(packet)) { + atomInvariantFailures += 1; + failures.push(failure("missing_treatment_telemetry", `packing-on packet ${packet.id} has no provenance and is not an unchanged bypass packet`)); + } + } + + for (const hunk of packet.hunks) { + const off = offByHunk.get(hunk.hunkId); + if (off === undefined) { + continue; + } + const omittedLenses = off.lenses.filter((lens) => !packet.lenses.includes(lens)); + lensDrops += omittedLenses.length; + const omittedNotes = off.attentionNotes.filter((note) => !packet.attentionNotes.includes(note)); + if ((off.reviewPriority === "critical" || off.reviewPriority === "high") && omittedNotes.length > 0) { + highPriorityNoteOmissions += omittedNotes.length; + failures.push(failure("high_priority_focus_omitted", `packing omits a high/critical attention note for hunk ${hunk.hunkId}`, { + omittedNoteCount: omittedNotes.length, + omittedNoteFingerprints: omittedNotes.map(valueFingerprint) + })); + } + if (off.coverage === "deep" && contextQualityRank(packet.contextQuality) < contextQualityRank(off.contextQuality)) { + deepContextDowngrades += 1; + failures.push(failure("deep_context_downgrade", `deep hunk ${hunk.hunkId} loses context quality`, { + from: off.contextQuality, + to: packet.contextQuality + })); + } + const offRelated = off.relatedChangedContext.map(relatedContextKey); + const onRelated = new Set(packet.relatedChangedContext.map(relatedContextKey)); + relatedContextOmissions += offRelated.filter((key) => !onRelated.has(key)).length; + } + } + + const expectedAtomIds = [...offAtomById.keys()].sort(); + const actualAtomIds = observedAtomIds.sort(); + if (stableJson(expectedAtomIds) !== stableJson(actualAtomIds)) { + atomInvariantFailures += 1; + failures.push(failure("atom_bijection", "packing-on atom set differs from packing-off atoms", { + missing: expectedAtomIds.filter((id) => !actualAtomIds.includes(id)), + extra: actualAtomIds.filter((id) => !expectedAtomIds.includes(id)) + })); + } + + const offOrder = schedulingOrder(input.offPackets); + const onOrder = schedulingOrder(input.onPackets); + const offPositionByHunk = new Map(offOrder.flatMap((packet, index) => packet.hunks.map((hunk) => [hunk.hunkId, index] as const))); + const onPositionByHunk = new Map(onOrder.flatMap((packet, index) => packet.hunks.map((hunk) => [hunk.hunkId, index] as const))); + const schedulingMoves = expectedHunks.filter((hunkId) => offPositionByHunk.get(hunkId) !== onPositionByHunk.get(hunkId)).length; + const proxy = proxyPackedPacketCount(input.offPackets, eventByPacket, input.plan, input.diff); + const motivating = input.runId.includes("dca8d870"); + let countReconciliation: ReplayRow["countReconciliation"] = "not-motivating-run"; + if (input.onPackets.length > input.offPackets.length) { + failures.push(failure("packet_count_increase", `${input.runId} gains packets with packing enabled`)); + } + if (motivating) { + if (input.offPackets.length !== 96) { + failures.push(failure("motivating_off_count", `motivating run must rebuild 96 packing-off packets, got ${input.offPackets.length}`)); + } + if (input.onPackets.length === 75) { + countReconciliation = "historical-target"; + } else if (input.onPackets.length === 74 || input.onPackets.length === 76) { + countReconciliation = input.onPackets.length === 74 ? "real-patch-74" : "real-patch-76"; + if (proxy.proxy !== 75 || proxy.actualMeasurement !== input.onPackets.length || proxy.changedSplits !== 1) { + failures.push(failure("unexplained_packet_count", "74/76 motivating count is not attributable solely to proxy-versus-real patch measurement", { proxy })); + } + } else { + failures.push(failure("unpermitted_packet_count", `motivating run packed count must be 74, 75, or 76; got ${input.onPackets.length}`)); + } + if ((1 - input.onPackets.length / Math.max(1, input.offPackets.length)) < 0.2) { + failures.push(failure("insufficient_packet_reduction", "motivating run packet reduction is below 20%")); + } + } + + const attentionNoteOmissions = sum(expectedHunks.map((hunkId) => { + const off = offByHunk.get(hunkId); + const on = onByHunk.get(hunkId); + return off === undefined || on === undefined ? 0 : off.attentionNotes.filter((note) => !on.attentionNotes.includes(note)).length; + })); + const contextComparison = { + contextTruncationsOff: input.offPackets.filter(contextTruncated).length, + contextTruncationsOn: input.onPackets.filter(contextTruncated).length, + relatedContextOmissions, + attentionNoteOmissions, + lensDrops + }; + const packingEventValues = [...eventByPacket.values()]; + const distribution = { + offHunksPerPacket: countDistribution(input.offPackets.map((packet) => packet.hunks.length)), + onHunksPerPacket: countDistribution(input.onPackets.map((packet) => packet.hunks.length)), + sourceAtomsPerOnPacket: countDistribution(input.onPackets.map((packet) => eventByPacket.get(packet.id)?.sourceAtomCount ?? 1)), + onCoverage: countDistribution(input.onPackets.map((packet) => packet.coverage)), + onProfiles: countDistribution(input.onPackets.map((packet) => packet.reviewProfile)), + onBudgetModes: countDistribution(input.onPackets.map((packet) => eventByPacket.get(packet.id)?.toolBudgetMode ?? "bypass-base")), + maxOnHunks: Math.max(0, ...input.onPackets.map((packet) => packet.hunks.length)), + eligiblePackingPackets: packingEventValues.length, + bypassPackets: input.onPackets.length - packingEventValues.length, + maxEligiblePackingPatchChars: Math.max(0, ...eligiblePackingPatchChars) + }; + + return { + runId: input.runId, + offPackets: input.offPackets.length, + onPackets: input.onPackets.length, + reductionPercent: round((1 - input.onPackets.length / Math.max(1, input.offPackets.length)) * 100, 3), + sourceAtoms: input.offPackets.length, + packedMultiAtomPackets: [...eventByPacket.values()].filter((event) => event.sourceAtomCount > 1).length, + reviewableHunks: expectedHunks.length, + newCoveragePromotions, + hunkBijectionFailures: failures.filter((entry) => entry.code === "hunk_bijection").length, + atomInvariantFailures, + capViolations, + lensDrops, + highPriorityNoteOmissions, + deepContextDowngrades, + relatedContextOmissions, + derivedProfileDowngrades, + profileFloorApplications, + effectiveProfileDowngrades, + effectiveBudgetDowngrades, + invalidDispatchRanks, + modelCallsObserved, + schedulingMoves, + proxyPackedPackets: proxy.proxy, + countReconciliation, + flagOffParityDifferences, + contextComparison, + distribution, + packetMembership: input.onPackets.map((packet) => ({ + packetId: packet.id, + path: packet.path, + hunkIds: packet.hunks.map((hunk) => hunk.hunkId), + coverage: packet.coverage, + lenses: [...packet.lenses], + profile: packet.reviewProfile, + budget: packet.toolBudget, + dispatchRank: packet.dispatchRank, + atomIds: eventByPacket.get(packet.id)?.atomIds ?? [atomIdForPacket(packet)] + })), + failures + }; +} + +function changedValuePaths(before: unknown, after: unknown, prefix = ""): string[] { + if (Object.is(before, after)) { + return []; + } + if (Array.isArray(before) && Array.isArray(after)) { + if (before.length !== after.length) { + return [`${prefix}.length`]; + } + return before.flatMap((value, index) => changedValuePaths(value, after[index], `${prefix}[${index}]`)); + } + if (isRecord(before) && isRecord(after)) { + return sortedUnique([...Object.keys(before), ...Object.keys(after)]).flatMap((key) => + changedValuePaths(before[key], after[key], prefix.length === 0 ? key : `${prefix}.${key}`) + ); + } + return [prefix.length === 0 ? "" : prefix]; +} + +function changedValueSamples( + before: unknown, + after: unknown, + prefix = "" +): Array<{ path: string; recorded: ValueFingerprint; rebuilt: ValueFingerprint }> { + if (Object.is(before, after)) { + return []; + } + if (Array.isArray(before) && Array.isArray(after)) { + if (before.length !== after.length) { + return [{ path: `${prefix}.length`, recorded: valueFingerprint(before.length), rebuilt: valueFingerprint(after.length) }]; + } + return before.flatMap((value, index) => changedValueSamples(value, after[index], `${prefix}[${index}]`)); + } + if (isRecord(before) && isRecord(after)) { + return sortedUnique([...Object.keys(before), ...Object.keys(after)]).flatMap((key) => + changedValueSamples(before[key], after[key], prefix.length === 0 ? key : `${prefix}.${key}`) + ); + } + return [{ + path: prefix.length === 0 ? "" : prefix, + recorded: valueFingerprint(before), + rebuilt: valueFingerprint(after) + }]; +} + +function valueFingerprint(value: unknown): ValueFingerprint { + const serialized = typeof value === "string" ? value : JSON.stringify(stableValue(value)) ?? String(value); + return { + kind: typeof value === "string" ? "string" : "json", + length: serialized.length, + sha256: sha256Hex(serialized) + }; +} + +export function historicalFlagOffParityView( + runId: string, + recordedPackets: ReviewPacket[], + rebuiltPackets: ReviewPacket[] +): { + packets: ReviewPacket[]; + migrations: Array<{ code: string; packets: number; explanation: string }>; +} { + const legacyDispatchRuns = new Set([ + "20260724-135818-740d73f2", + "20260724-150405-fe1548ae", + "20260724-162739-81f806a6" + ]); + const packets = structuredClone(rebuiltPackets); + const migrations: Array<{ code: string; packets: number; explanation: string }> = []; + const recordedById = new Map(recordedPackets.map((packet) => [packet.id, packet])); + + if ( + legacyDispatchRuns.has(runId) && + recordedPackets.length > 0 && + recordedPackets.every((packet) => !Object.prototype.hasOwnProperty.call(packet, "dispatchRank")) + ) { + for (const packet of packets) { + delete (packet as Partial).dispatchRank; + } + migrations.push({ + code: "pre_plan100_dispatch_rank_schema", + packets: packets.length, + explanation: "recorded artifacts predate dispatchRank; parity omits only that absent field after validating the rebuilt current rank formula" + }); + } + + if (runId === "20260724-135818-740d73f2") { + let migratedPackets = 0; + for (const packet of packets) { + const recorded = recordedById.get(packet.id); + if (recorded === undefined || stableJson(recorded) === stableJson(packet)) { + continue; + } + const onlyKnownPaths = changedValuePaths(recorded, packet).every((entry) => entry === "contextText" || entry === "relevantTests.length"); + const selfTest = packet.relevantTests.length === 1 ? packet.relevantTests[0] : undefined; + const expectedContext = selfTest === undefined + ? undefined + : `${recorded.contextText}\nLikely tests: ${selfTest.path}:${selfTest.name}`; + if ( + onlyKnownPaths && + recorded.relevantTests.length === 0 && + selfTest?.path === packet.path && + selfTest.name === packet.path && + packet.contextText === expectedContext + ) { + packet.relevantTests = []; + packet.contextText = recorded.contextText; + migratedPackets += 1; + } + } + if (migratedPackets > 0) { + migrations.push({ + code: "pre_self_test_context_filter", + packets: migratedPackets, + explanation: "the oldest clean artifact predates the self-test context filter; parity removes only the exact redundant path:path test line" + }); + } + } + + return { packets, migrations }; +} + +function relatedContextKey(context: ReviewPacket["relatedChangedContext"][number]): string { + return stableJson({ + path: context.path, + hunkId: context.hunkId, + relatedHunkIds: context.relatedHunkIds, + symbol: context.symbol, + source: context.relationshipSource, + strength: context.relationshipStrength + }); +} + +function contextQualityRank(quality: PacketContextQuality | undefined): number { + return quality === undefined ? CONTEXT_QUALITY_RANK.path_only : CONTEXT_QUALITY_RANK[quality]; +} + +function schedulingOrder(packets: ReviewPacket[]): ReviewPacket[] { + const priorityRank = { critical: 0, high: 1, normal: 2, low: 3 } as const; + const coverageRank = { deep: 0, normal: 1, light: 2 } as const; + return packets.map((packet, index) => ({ packet, index })).sort((a, b) => + priorityRank[a.packet.reviewPriority] - priorityRank[b.packet.reviewPriority] || + coverageRank[a.packet.coverage] - coverageRank[b.packet.coverage] || + a.packet.dispatchRank[0] - b.packet.dispatchRank[0] || + a.packet.dispatchRank[1] - b.packet.dispatchRank[1] || + a.index - b.index + ).map((entry) => entry.packet); +} + +function round(value: number, digits = 6): number { + const factor = 10 ** digits; + return Math.round(value * factor) / factor; +} + +function createCaptureTelemetry(runId: string): { telemetry: TelemetryRecorder; captured: CapturedTelemetry } { + const captured: CapturedTelemetry = { events: [], modelCalls: [], toolCalls: [] }; + let toolCall = 0; + return { + captured, + telemetry: { + runId, + runDir: undefined, + event: (event) => captured.events.push(structuredClone(event)), + recordModelCall: (record) => captured.modelCalls.push(structuredClone(record)), + recordToolCall: (record) => { + captured.toolCalls.push(structuredClone(record)); + toolCall += 1; + return `replay-tool-${String(toolCall).padStart(6, "0")}`; + }, + writeArtifact: async () => undefined, + writeDebug: async () => undefined, + flush: async () => undefined + } + }; +} + +type ExperimentArm = "A" | "B" | "C"; + +type ExecutionPressure = { + reviewedPackets: number; + reviewedAtoms: number; + requestedToolCalls: number; + usedToolCalls: number; + rejectedToolCalls: number; + rejectionRatePerAtom: number; + rejectionReasons: Record; + resultChars: number; + continuations: number; + modelServiceSeconds: number; + inputTokens: number; + outputTokens: number; + stage7CostUSD: number; + costPerReviewedAtomUSD: number | null; +}; + +type TreatmentExecution = { + repeat: number; + treated: boolean; + atomIds: string[]; + hunkIds: string[]; + targetPacketId?: string; + targetPath?: string; + targetFilePackets?: number; + targetAtomCount?: number; + targetCoverage?: string; + targetRequestedLensSignature?: string; + pressure: ExecutionPressure; + failures: ReportFailure[]; +}; + +type ExpectationRate = { + expectationId: string; + list: EvalExpectationList; + candidateHits: number; + finalHits: number; + denominator: number; + candidateRate: number; + finalRate: number; + lossHistogram: Record; + lossByAtomCount: Record; +}; + +type EvalArmReport = { + runNumber: number; + treatment: { treated: number; total: number; minimum: number; valid: boolean }; + intentToTreat: ExpectationRate[]; + treatedOnly: { label: "secondary-treated-only"; denominatorByExpectation: Record; rates: ExpectationRate[] }; + pressure: ExecutionPressure; + actualCostUSD: number; +}; + +type EvalCaseReport = { + case: string; + selectedArm?: "B" | "C"; + arms: Record; +}; + +export type EvalReport = { + schemaVersion: 1; + mode: "eval"; + cohort: { id: string; runNumbers: number[]; actualCostUSD: number }; + evidence: "repeated-packing-sensitive" | "production-capacity"; + expectedRepeats: number; + selectedArm?: "B" | "C"; + cases?: EvalCaseReport[]; + productionEconomics?: ProductionEconomics; + productionThroughput?: ProductionThroughput; + failures: ReportFailure[]; +}; + +export type ProductionThroughput = { + baseline: ProductionArmThroughput; + selected: ProductionArmThroughput; +}; + +type ProductionArmThroughput = { + reviewedHunkIds: string[]; + reviewedHunks: number; + reviewedAtoms: number; + reviewedPackets: number; + wallTimeSeconds: number; + reviewedHunksPerWallSecond: number; + modelServiceSeconds: number; + modelServiceSecondsPerReviewedHunk: number; + totalTokens: number; + tokensPerReviewedHunk: number; + reasoningTokens: number; + reasoningTokensPerReviewedHunk: number; + continuations: number; + continuationsPerReviewedAtom: number; + pressure: ExecutionPressure; +}; + +export type ProductionEconomics = { + equivalentTargetHunks: 142; + baseline: ProductionArmEconomics; + selected: ProductionArmEconomics; + equivalentReviewSavingsUSD: number; + validationCostInputUSD: number; + validationCostInputLabel: "cohort_actual_cost_minimum" | "explicit_cumulative_validation_cost"; + breakEvenReviewCount: number; +}; + +type ProductionArmEconomics = { + actualCostUSD: number; + reviewedHunks: number; + costPerReviewedHunkUSD: number; + equivalentReviewCostUSD: number; + equivalentCostExtrapolated: boolean; +}; + +export type RegressionReport = { + schemaVersion: 1; + mode: "regression"; + evidence: "one-repeat-collateral-only"; + baselineCohort: { id: string; runNumbers: number[]; actualCostUSD: number }; + selectedCohort: { id: string; runNumbers: number[]; actualCostUSD: number }; + expectedRepeats: number; + cases: Array<{ + caseName: string; + baselineRun: number; + selectedRun: number; + expectationTransitions: Array<{ expectationId: string; list: string; from: string; to: string }>; + selectedTreatmentExecutions: number; + dispatchOrderChanges: number; + baselinePressure: ExecutionPressure; + selectedPressure: ExecutionPressure; + }>; + failures: ReportFailure[]; +}; + +function armForRun(run: EvalCaseRunInput): ExperimentArm { + const effective = run.info.effectiveConfig?.review; + assertReport(effective !== undefined, "missing_effective_config", `run ${run.runNumber} has no effective packet-packing config`); + assertReport( + typeof effective.packSameFileHunks === "boolean" && + (effective.packedToolBudgetMode === "base" || effective.packedToolBudgetMode === "atom-scaled"), + "invalid_arm_config", + `run ${run.runNumber} has invalid effective packet-packing settings` + ); + if (!effective.packSameFileHunks) { + assertReport(effective.packedToolBudgetMode === "base", "invalid_arm_config", `baseline run ${run.runNumber} must use base tool budget`); + return "A"; + } + return effective.packedToolBudgetMode === "base" ? "B" : "C"; +} + +function caseFamily(run: EvalCaseRunInput): string { + const source = run.info.caseFile ?? run.info.caseName; + return source + .replace(/\.ya?ml$/u, "") + .replace(/-(?:a|b|c|baseline|selected)$/iu, ""); +} + +export function selectExplicitCohort(runs: EvalCaseRunInput[], selector: string): CohortSelection { + assertReport(runs.length > 0, "empty_cohort", "no eval run info files were found"); + const byInvocation = new Map(); + for (const run of runs.filter((candidate) => candidate.info.invocation !== undefined && candidate.invocationManifest !== undefined)) { + const id = run.info.invocation?.id; + assertReport(id !== undefined, "missing_invocation_manifest", `run ${run.runNumber} lacks invocation identity`); + byInvocation.set(id, [...(byInvocation.get(id) ?? []), run]); + } + for (const [id, invocation] of byInvocation) { + byInvocation.set(id, [...invocation].sort((left, right) => + (left.info.invocation?.caseIndex ?? Number.MAX_SAFE_INTEGER) - (right.info.invocation?.caseIndex ?? Number.MAX_SAFE_INTEGER) + )); + } + assertReport(byInvocation.size > 0, "missing_invocation_manifest", "no cohort-eligible eval invocation manifests were found"); + const invocations = [...byInvocation.values()].sort((left, right) => + (left[0]?.invocationManifest?.startedAt ?? "").localeCompare(right[0]?.invocationManifest?.startedAt ?? "") || + Math.max(...left.map((run) => run.runNumber)) - Math.max(...right.map((run) => run.runNumber)) || + (left[0]?.info.invocation?.id ?? "").localeCompare(right[0]?.info.invocation?.id ?? "") + ); + let selected: EvalCaseRunInput[] | undefined; + if (byInvocation.has(selector)) { + selected = byInvocation.get(selector); + } else if (selector === "latest") { + selected = invocations.at(-1); + } else if (/^\d+$/u.test(selector)) { + const endingRun = Number(selector); + const matches = invocations.filter((invocation) => invocation.at(-1)?.runNumber === endingRun); + assertReport(matches.length <= 1, "ambiguous_cohort_selector", `run ${selector} matches multiple persisted invocations; use the invocation UUID`); + selected = matches[0]; + assertReport(selected !== undefined, "unknown_cohort", `run ${selector} is not an exact persisted invocation boundary`); + } else { + const match = /^(\d+)-(\d+)$/u.exec(selector); + assertReport(match !== null, "invalid_cohort_selector", `cohort must be latest, an invocation UUID, an ending run number, or a min-max range: ${selector}`); + const start = Number(match[1]); + const end = Number(match[2]); + const matches = invocations.filter((invocation) => invocation[0]?.runNumber === start && invocation.at(-1)?.runNumber === end); + assertReport(matches.length <= 1, "ambiguous_cohort_selector", `cohort ${selector} matches multiple persisted invocations; use the invocation UUID`); + selected = matches[0]; + assertReport(selected !== undefined, "unknown_cohort", `cohort ${selector} is not an exact persisted invocation boundary`); + } + assertReport(selected !== undefined && selected.length > 0, "unknown_cohort", `cohort ${selector} is empty`); + validateCompleteInvocation(selected); + assertNoMixedRuns(selected); + return { id: selected[0]?.info.invocation?.id ?? "unknown", runs: selected }; +} + +function validateCompleteInvocation(runs: EvalCaseRunInput[]): void { + const manifest = runs[0]?.invocationManifest; + assertReport(manifest !== undefined, "missing_invocation_manifest", "cohort invocation manifest is missing"); + const expectedIndexes = Array.from({ length: manifest.cases.length }, (_, index) => index); + const caseIndexes = manifest.cases.map((entry) => entry.caseIndex); + const runIndexes = manifest.runs.map((entry) => entry.caseIndex); + const loadedIndexes = runs.map((run) => run.info.invocation?.caseIndex).filter((index): index is number => index !== undefined); + const loadedRunDirs = runs.map((run) => path.resolve(run.runDir)); + assertReport( + manifest.status === "complete" && manifest.completedAt !== undefined && + stableJson(caseIndexes) === stableJson(expectedIndexes) && + stableJson(runIndexes) === stableJson(expectedIndexes) && + stableJson(loadedIndexes) === stableJson(expectedIndexes) && + manifest.runs.length === manifest.cases.length && runs.length === manifest.cases.length && + unique(loadedRunDirs).length === loadedRunDirs.length, + "partial_latest_cohort", + "persisted eval invocation is incomplete or its exact ordered case set does not match loaded runs", + { invocationId: manifest.invocationId, expectedCount: manifest.cases.length, recordedCount: manifest.runs.length, loadedCount: runs.length } + ); + for (const [index, run] of runs.entries()) { + validateInvocationManifestRun(manifest, run.info, run.runDir); + assertReport(run.info.invocation?.caseIndex === index, "invocation_manifest_join", `run ${run.runNumber} is out of invocation order`); + } +} + +function assertNoMixedRuns(runs: EvalCaseRunInput[]): void { + const caseNames = runs.map((run) => run.info.caseName); + assertReport(unique(caseNames).length === caseNames.length, "mixed_cohort", "selected cohort contains multiple generations of the same case", { caseNames }); + const runtimeCommits = sortedUnique(runs.flatMap((run) => run.info.codegenieRuntime?.commit ?? [])); + if (runtimeCommits.length > 1) { + fail("mixed_cohort", "selected cohort mixes Codegenie commits", { runtimeCommits }); + } +} + +export function analyzeEvalCohort( + cohort: CohortSelection, + expectedRepeats: number, + options: { actualValidationCostUSD?: number } = {} +): EvalReport { + const failures: ReportFailure[] = []; + assertReport(Number.isInteger(expectedRepeats) && expectedRepeats > 0, "invalid_expected_repeats", "expected repeats must be a positive integer"); + const arms = cohort.runs.map((run) => ({ run, arm: armForRun(run), family: caseFamily(run) })); + const armSet = new Set(arms.map((entry) => entry.arm)); + const production = cohort.runs.length === 2 && armSet.has("A") && (armSet.has("B") || armSet.has("C")); + const actualCostUSD = aggregateCohortCost(cohort.runs, failures); + + if (production) { + const baseline = arms.find((entry) => entry.arm === "A"); + const selected = arms.find((entry) => entry.arm !== "A"); + assertReport(baseline !== undefined && selected !== undefined, "production_pair", "production comparison requires one baseline and one selected run"); + if (stableJson(normalizeExperimentSnapshot(baseline.run.declaredCase)) !== stableJson(normalizeExperimentSnapshot(selected.run.declaredCase))) { + failures.push(failure("production_case_mismatch", "production baseline and selected cases differ outside packet-packing review settings")); + } + validateRunRepeatCount(baseline.run, expectedRepeats, failures); + validateRunRepeatCount(selected.run, expectedRepeats, failures); + validateRunEvidence(baseline.run, "production/baseline", failures); + validateRunEvidence(selected.run, "production/selected", failures); + const baselineTreatment = baseline.run.executions.map((execution) => analyzeTreatmentExecution(baseline.run, execution, "A")); + const selectedTreatment = selected.run.executions.map((execution) => analyzeTreatmentExecution( + selected.run, + execution, + selected.arm, + baseline.run.executions.find((candidate) => candidate.repeat === execution.repeat) + )); + failures.push(...baselineTreatment.flatMap((entry) => entry.failures), ...selectedTreatment.flatMap((entry) => entry.failures)); + validateProductionProvenance(baseline.run, selected.run, failures); + validateRunScoreExpectations("production/baseline", baseline.run, failures); + validateRunScoreExpectations("production/selected", selected.run, failures); + validateProductionExpectations(baseline.run, selected.run, failures); + validateProductionModelAccounting(baseline.run, baselineTreatment, failures, "baseline"); + validateProductionModelAccounting(selected.run, selectedTreatment, failures, "selected"); + if (selectedTreatment.every((entry) => !entry.treated)) { + failures.push(failure("missing_treatment", "selected production run never packed multiple atoms")); + } + const productionThroughput = analyzeProductionThroughput( + baseline.run, + baselineTreatment, + selected.run, + selectedTreatment, + failures + ); + const economics = computeProductionEconomics( + { + actualCostUSD: costForRun(baseline.run, failures), + reviewedHunks: reviewedHunksForRun(baseline.run) + }, + { + actualCostUSD: costForRun(selected.run, failures), + reviewedHunks: reviewedHunksForRun(selected.run) + }, + options.actualValidationCostUSD ?? actualCostUSD, + options.actualValidationCostUSD === undefined ? "cohort_actual_cost_minimum" : "explicit_cumulative_validation_cost", + failures + ); + if (selected.run.info.score.status === "error" || baseline.run.info.score.status === "error") { + failures.push(failure("eval_error", "production cohort contains an errored eval run")); + } + return { + schemaVersion: 1, + mode: "eval", + cohort: { id: cohort.id, runNumbers: cohort.runs.map((run) => run.runNumber), actualCostUSD }, + evidence: "production-capacity", + expectedRepeats, + productionEconomics: economics, + productionThroughput, + failures + }; + } + + const grouped = new Map>>(); + for (const entry of arms) { + const group = grouped.get(entry.family) ?? {}; + if (group[entry.arm] !== undefined) { + failures.push(failure("mixed_cohort", `cohort has duplicate ${entry.arm} arm for ${entry.family}`)); + } + group[entry.arm] = entry.run; + grouped.set(entry.family, group); + } + + const cases: NonNullable = []; + for (const [family, group] of [...grouped].sort(([a], [b]) => a.localeCompare(b))) { + if (group.A === undefined || group.B === undefined || group.C === undefined) { + failures.push(failure("incomplete_abc_cohort", `case ${family} does not have exactly one A/B/C run`)); + continue; + } + const runByArm = group as Record; + const evidenceScores = {} as Record>; + for (const arm of ["A", "B", "C"] as const) { + validateRunRepeatCount(runByArm[arm], expectedRepeats, failures); + evidenceScores[arm] = new Map(validateRunEvidence(runByArm[arm], `${family}/${arm}`, failures) + .map((score, index) => [runByArm[arm].executions[index]!.repeat, score])); + if (runByArm[arm].info.score.status === "error") { + failures.push(failure("eval_error", `${family}/${arm} is an errored eval run`)); + } + } + assertMatchingCaseDefinitions(runByArm, failures); + validateExpectationJoins(family, runByArm, failures); + const treatmentByArm = {} as Record; + treatmentByArm.A = runByArm.A.executions.map((execution) => analyzeTreatmentExecution(runByArm.A, execution, "A")); + for (const arm of ["B", "C"] as const) { + treatmentByArm[arm] = runByArm[arm].executions.map((execution) => analyzeTreatmentExecution( + runByArm[arm], + execution, + arm, + runByArm.A.executions.find((candidate) => candidate.repeat === execution.repeat) + )); + } + failures.push(...Object.values(treatmentByArm).flatMap((entries) => entries.flatMap((entry) => entry.failures))); + compareTargetTreatment(family, treatmentByArm, failures); + + const minimumTreatment = expectedRepeats >= 10 ? 8 : expectedRepeats; + const armReports = {} as EvalCaseReport["arms"]; + for (const arm of ["A", "B", "C"] as const) { + const executions = runByArm[arm].executions; + const treatedCount = arm === "A" ? executions.length : treatmentByArm[arm].filter((entry) => entry.treated).length; + if (arm !== "A" && treatedCount < minimumTreatment) { + failures.push(failure("insufficient_treatment", `${family}/${arm} treated ${treatedCount}/${executions.length}; minimum is ${minimumTreatment}`)); + } + const atomCounts = new Map(treatmentByArm[arm].map((entry) => [entry.repeat, entry.targetAtomCount ?? 1])); + const intentToTreat = expectationRates(executions, () => true, evidenceScores[arm], atomCounts); + const treatedOnly = expectationRates(executions, (execution) => { + if (arm === "A") { + return true; + } + return treatmentByArm[arm].find((entry) => entry.repeat === execution.repeat)?.treated === true; + }, evidenceScores[arm], atomCounts); + armReports[arm] = { + runNumber: runByArm[arm].runNumber, + treatment: { + treated: treatedCount, + total: executions.length, + minimum: arm === "A" ? executions.length : minimumTreatment, + valid: arm === "A" || treatedCount >= minimumTreatment + }, + intentToTreat, + treatedOnly: { + label: "secondary-treated-only", + denominatorByExpectation: Object.fromEntries(treatedOnly.map((rate) => [`${rate.list}:${rate.expectationId}`, rate.denominator])), + rates: treatedOnly + }, + pressure: combinePressure(treatmentByArm[arm].map((entry) => entry.pressure)), + actualCostUSD: costForRun(runByArm[arm], failures) + }; + } + if (expectedRepeats >= 10) { + enforceRequiredRecallZero(family, armReports, failures); + enforcePressureGate(family, armReports, failures); + } + cases.push({ case: family, arms: armReports }); + } + + if (expectedRepeats >= 10) { + enforceCohortRecallGates(cases, failures); + } + const selectedArm = expectedRepeats >= 10 ? selectCohortPackedArm(cases, expectedRepeats, failures) : undefined; + if (selectedArm !== undefined) { + for (const entry of cases) { + entry.selectedArm = selectedArm; + } + } + + return { + schemaVersion: 1, + mode: "eval", + cohort: { id: cohort.id, runNumbers: cohort.runs.map((run) => run.runNumber), actualCostUSD }, + evidence: "repeated-packing-sensitive", + expectedRepeats, + ...(selectedArm === undefined ? {} : { selectedArm }), + cases, + failures + }; +} + +function validateRunRepeatCount(run: EvalCaseRunInput, expectedRepeats: number, failures: ReportFailure[]): void { + const declared = run.declaredCase.repeat ?? 1; + if (declared !== expectedRepeats || run.executions.length !== expectedRepeats) { + failures.push(failure("repeat_count_mismatch", `run ${run.runNumber} has ${run.executions.length} executions, expected ${expectedRepeats}`, { + declared + })); + } + const indexes = run.executions.map((execution) => execution.repeat).sort((a, b) => a - b); + if (stableJson(indexes) !== stableJson(Array.from({ length: expectedRepeats }, (_, index) => index + 1))) { + failures.push(failure("repeat_join_mismatch", `run ${run.runNumber} has missing or duplicate repeat indexes`, { indexes })); + } +} + +function assertMatchingCaseDefinitions( + runs: Record, + failures: ReportFailure[] +): void { + const normalized = (["A", "B", "C"] as const).map((arm) => stableJson(normalizeExperimentSnapshot(runs[arm].declaredCase))); + if (unique(normalized).length !== 1) { + failures.push(failure("arm_case_mismatch", "A/B/C case definitions differ outside packet-packing review fields")); + } +} + +function validateExpectationJoins( + family: string, + runs: Record, + failures: ReportFailure[] +): void { + const canonical = declaredExpectationKeys(runs.A.declaredCase); + if (canonical.duplicates.length > 0) { + failures.push(failure("declared_expectation_join", `${family}/A declares duplicate expectation keys`, { duplicateKeys: canonical.duplicates })); + } + for (const arm of ["B", "C"] as const) { + const declared = declaredExpectationKeys(runs[arm].declaredCase); + if (declared.duplicates.length > 0 || stableJson(declared.values) !== stableJson(canonical.values)) { + failures.push(failure("declared_expectation_join", `${family}/${arm} declared expectations differ from A`, { + expected: canonical.values, + actual: declared.values, + duplicateKeys: declared.duplicates + })); + } + } + for (const arm of ["A", "B", "C"] as const) { + const scores = [runs[arm].info.score, ...runs[arm].executions.map((execution) => execution.score)]; + for (const [scoreIndex, score] of scores.entries()) { + const keys = expectationKeys(score); + if (keys.duplicates.length > 0 || stableJson(keys.values) !== stableJson(canonical.values)) { + failures.push(failure("expectation_join", `${family}/${arm}/score ${scoreIndex} does not have the exact bidirectional declared expectation join`, { + expected: canonical.values, + actual: keys.values, + duplicateKeys: keys.duplicates + })); + } + } + } +} + +function expectationKeys(score: EvalScore | undefined): { values: string[]; duplicates: string[] } { + const raw = score?.expectationResults.map((result) => `${result.list}\0${result.expectationId}`) ?? []; + return { values: [...raw].sort(), duplicates: duplicateValues(raw) }; +} + +function declaredExpectationKeys(evalCase: EvalCase): { values: string[]; duplicates: string[] } { + const raw = (["should_find", "should_find_candidate", "should_not_find"] as const).flatMap((list) => + (evalCase[list] ?? []).map((expectation) => `${list}\0${expectation.id}`) + ); + return { values: [...raw].sort(), duplicates: duplicateValues(raw) }; +} + +function normalizeExperimentSnapshot(snapshot: EvalCase): unknown { + const copy = structuredClone(snapshot); + copy.name = copy.name.replace(/-(?:a|b|c|baseline|selected)$/iu, ""); + delete copy.repeat; + if (copy.review !== undefined) { + delete copy.review.packSameFileHunks; + delete copy.review.packedToolBudgetMode; + } + return copy; +} + +function analyzeTreatmentExecution( + run: EvalCaseRunInput, + execution: EvalExecutionInput, + arm: ExperimentArm, + baselineExecution?: EvalExecutionInput, + requireTreatment = true +): TreatmentExecution { + const failures: ReportFailure[] = []; + const eventByPacket = packingEvents(execution.events, failures); + const packetById = new Map(execution.packets.map((packet) => [packet.id, packet])); + const atoms = arm === "A" ? execution.packets : baselineExecution?.packets ?? []; + const atomById = new Map(atoms.map((packet) => [atomIdForPacket(packet), packet])); + if (atomById.size !== atoms.length) { + failures.push(failure("atom_bijection", `run ${run.runNumber}/repeat ${execution.repeat} has ambiguous A atom IDs`)); + } + if (arm !== "A" && baselineExecution === undefined) { + failures.push(failure("missing_baseline_atoms", `run ${run.runNumber}/repeat ${execution.repeat} cannot join treatment to A atoms`)); + } + const duplicateHunks = duplicateValues(execution.packets.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId))); + if (duplicateHunks.length > 0) { + failures.push(failure("hunk_bijection", `run ${run.runNumber}/repeat ${execution.repeat} duplicates hunks`, { duplicateHunks })); + } + const factsByPath = new Map(execution.fileFacts.map((facts) => [facts.path, facts])); + if (factsByPath.size !== execution.fileFacts.length) { + failures.push(failure("duplicate_file_facts", `run ${run.runNumber}/repeat ${execution.repeat} has duplicate file facts`)); + } + for (const packet of execution.packets) { + if (packet.hunks.length > MAX_HUNKS_PER_PACKET) { + failures.push(failure("packet_cap", `packet ${packet.id} exceeds the five-hunk cap`)); + } + const facts = factsByPath.get(packet.path); + if (facts === undefined || stableJson(packet.dispatchRank) !== stableJson(packetDispatchRank(packet.path, facts, changedLines(packet)))) { + failures.push(failure("invalid_dispatch_rank", `packet ${packet.id} has an invalid dispatch rank`)); + } + if (requestedLensSignatureForAtom(execution, packet) === undefined) { + failures.push(failure("stage5_lens_join", `packet ${packet.id} lacks a non-empty Stage-5 lens decision for every hunk`)); + } + } + if (arm !== "A") { + const baselineHunks = atoms.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId)); + const selectedHunks = execution.packets.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId)); + if (stableJson([...baselineHunks].sort()) !== stableJson([...selectedHunks].sort())) { + failures.push(failure("hunk_bijection", `run ${run.runNumber}/repeat ${execution.repeat} does not preserve the exact A hunk set`)); + } + } + const usedAtomIds: string[] = []; + for (const [packetId, event] of eventByPacket) { + const packet = packetById.get(packetId); + if (packet === undefined) { + failures.push(failure("missing_packet", `treatment telemetry references missing packet ${packetId}`)); + continue; + } + const sourceAtoms = event.atomIds.map((atomId) => atomById.get(atomId)); + const allAtomsResolved = sourceAtoms.every((atom): atom is ReviewPacket => atom !== undefined); + if (!allAtomsResolved) { + failures.push(failure("unknown_source_atom", `packet ${packetId} references an atom absent from the matching A repeat`)); + } + const resolvedAtoms = sourceAtoms.filter((atom): atom is ReviewPacket => atom !== undefined); + const expectedHunks = resolvedAtoms.flatMap((atom) => atom.hunks.map((hunk) => hunk.hunkId)); + const packetHunks = packet.hunks.map((hunk) => hunk.hunkId); + const standaloneProfiles = resolvedAtoms.map((atom) => atom.reviewProfile); + const profileFloor = standaloneProfiles.reduce((highest, profile) => PROFILE_RANK[profile] > PROFILE_RANK[highest] ? profile : highest, "simple" as ReviewProfile); + const rawPatchChars = actualPatchCharsForHunk(execution.diff); + const patchChars = sum(packet.hunks.map((hunk) => rawPatchChars.get(hunk.hunkId) ?? Number.MAX_SAFE_INTEGER)); + const depth = run.declaredCase.review?.depth ?? "normal"; + const baseBudget = toolBudget(packet.coverage, depth, packet.reviewProfile); + const additionalAtoms = Math.max(0, resolvedAtoms.length - 1); + const packedBudget = arm === "C" && resolvedAtoms.length > 1 && packet.reviewProfile !== "simple" + ? { + ...baseBudget, + maxToolCalls: Math.min(baseBudget.maxToolCalls + additionalAtoms, Math.ceil(1.75 * baseBudget.maxToolCalls)), + maxResultChars: Math.min(baseBudget.maxResultChars + additionalAtoms * 2_000, Math.ceil(1.75 * baseBudget.maxResultChars)) + } + : baseBudget; + const expectedEffectiveBudget = scaleToolBudget(packedBudget, run.declaredCase.review?.budgetBoost ?? 1); + if ( + event.sourceAtomCount !== event.atomIds.length || + unique(event.atomIds).length !== event.atomIds.length || + stableJson(event.standaloneProfiles) !== stableJson(standaloneProfiles) || + stableJson(expectedHunks) !== stableJson(packetHunks) || + event.hunkCount !== packet.hunks.length || + event.capUsage.hunks !== packet.hunks.length || + event.capUsage.maxHunks !== MAX_HUNKS_PER_PACKET || + event.capUsage.maxPatchChars !== MAX_PATCH_CHARS || + event.capUsage.patchChars !== patchChars || + event.capUsage.hunks > MAX_HUNKS_PER_PACKET || + event.capUsage.patchChars > MAX_PATCH_CHARS || + resolvedAtoms.some((atom) => atom.path !== packet.path || atom.language !== packet.language || atom.coverage !== packet.coverage) || + event.effectiveCoverage !== packet.coverage || + event.effectiveProfile !== packet.reviewProfile || + event.profileFloor !== profileFloor || + stableJson(event.baseToolBudget) !== stableJson(baseBudget) || + stableJson(event.effectiveToolBudget) !== stableJson(expectedEffectiveBudget) || + stableJson(event.effectiveToolBudget) !== stableJson(packet.toolBudget) || + event.toolBudgetMode !== (arm === "C" ? "atom-scaled" : "base") + ) { + failures.push(failure("treatment_invariant", `packet ${packetId} has invalid atom/cap telemetry`)); + } + if (PROFILE_RANK[event.effectiveProfile] < PROFILE_RANK[profileFloor]) { + failures.push(failure("effective_profile_downgrade", `packet ${packetId} falls below its standalone profile floor`)); + } + if (budgetDowngraded(event.effectiveToolBudget, event.baseToolBudget)) { + failures.push(failure("effective_budget_downgrade", `packet ${packetId} effective budget falls below its base budget`)); + } + if (!event.plannerLensesPreserved) { + failures.push(failure("planner_lens_drop", `packet ${packetId} reports a planner-selected lens drop`)); + } + if (baselineExecution !== undefined) { + const requestedSignatures = resolvedAtoms.map((atom) => requestedLensSignatureForAtom(baselineExecution, atom)); + const selectedRequestedSignature = requestedLensSignatureForAtom(execution, packet); + if ( + requestedSignatures.some((signature) => signature === undefined) || selectedRequestedSignature === undefined || + unique(requestedSignatures).length !== 1 || requestedSignatures[0] !== event.requestedLensSignature || + selectedRequestedSignature !== event.requestedLensSignature + ) { + failures.push(failure("requested_lens_join", `packet ${packetId} requested-lens telemetry does not match Stage-5 decisions for its A atoms`)); + } + const aRequestedLenses = sortedUnique(resolvedAtoms.flatMap((atom) => requestedLensesForAtom(baselineExecution, atom) ?? [])); + const aRoutedLenses = sortedUnique(resolvedAtoms.flatMap((atom) => atom.lenses)); + const selectedRoutedLenses = sortedUnique(packet.lenses); + const missingFromA = aRequestedLenses.filter((lens) => !aRoutedLenses.includes(lens)); + if (missingFromA.length > 0) { + failures.push(failure("a_lens_route_join", `A atoms omit one or more of their explicit Stage-5 lenses`, { + missingLensHashes: missingFromA.map((lens) => sha256Hex(lens)) + })); + } + if (stableJson(selectedRoutedLenses) !== stableJson(aRoutedLenses)) { + const missingRoutedLenses = aRoutedLenses.filter((lens) => !selectedRoutedLenses.includes(lens)); + const extraRoutedLenses = selectedRoutedLenses.filter((lens) => !aRoutedLenses.includes(lens)); + failures.push(failure("routed_lens_join", `packet ${packetId} does not exactly preserve the allowed routed-lens set from its A atoms`, { + missingLensHashes: missingRoutedLenses.map((lens) => sha256Hex(lens)), + extraLensHashes: extraRoutedLenses.map((lens) => sha256Hex(lens)) + })); + } + } + usedAtomIds.push(...event.atomIds); + } + if (arm !== "A") { + for (const packet of execution.packets.filter((candidate) => !eventByPacket.has(candidate.id))) { + const matchingAtom = atomById.get(atomIdForPacket(packet)); + if (matchingAtom === undefined || stableJson(matchingAtom) !== stableJson(packet)) { + failures.push(failure("untreated_packet_mismatch", `packet ${packet.id} has no treatment event and is not an exact A atom`)); + } else { + usedAtomIds.push(atomIdForPacket(matchingAtom)); + } + } + if (stableJson([...usedAtomIds].sort()) !== stableJson([...atomById.keys()].sort())) { + failures.push(failure("atom_bijection", `run ${run.runNumber}/repeat ${execution.repeat} does not consume each A atom exactly once`)); + } + } + if (arm === "A" && eventByPacket.size > 0) { + failures.push(failure("baseline_treatment", `baseline run ${run.runNumber} emitted packing treatment telemetry`)); + } + if (arm !== "A" && requireTreatment && eventByPacket.size === 0) { + failures.push(failure("missing_treatment_telemetry", `packed run ${run.runNumber}/repeat ${execution.repeat} has no same_file_atoms_packed telemetry`)); + } + + const target = targetPacket(run.declaredCase, execution.packets); + const targetEvent = target === undefined ? undefined : eventByPacket.get(target.id); + const treated = arm !== "A" && targetEvent !== undefined && targetEvent.sourceAtomCount >= 2; + if (arm !== "A" && target === undefined) { + failures.push(failure("missing_target_packet", `run ${run.runNumber}/repeat ${execution.repeat} has no packet for its positive expectation`)); + } + const pressure = executionPressure(execution, eventByPacket, failures); + const targetAtomCount = target === undefined + ? undefined + : arm === "A" + ? 1 + : targetEvent?.atomIds.length ?? atoms.filter((atom) => atom.hunks.some((hunk) => target.hunks.some((targetHunk) => targetHunk.hunkId === hunk.hunkId))).length; + const targetRequestedLensSignature = arm === "A" && target !== undefined + ? requestedLensSignatureForAtom(execution, target) + : targetEvent?.requestedLensSignature; + return { + repeat: execution.repeat, + treated, + atomIds: [...atomById.keys()].sort(), + hunkIds: execution.packets.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId)).sort(), + ...(target === undefined ? {} : { + targetPacketId: target.id, + targetPath: target.path, + targetFilePackets: execution.packets.filter((packet) => packet.path === target.path).length, + ...(targetAtomCount === undefined ? {} : { targetAtomCount }), + targetCoverage: target.coverage, + ...(targetRequestedLensSignature === undefined ? {} : { targetRequestedLensSignature }) + }), + pressure, + failures + }; +} + +function targetPacket(evalCase: EvalCase, packets: ReviewPacket[]): ReviewPacket | undefined { + const expectation = evalCase.should_find_candidate?.[0] ?? evalCase.should_find?.[0]; + if (expectation === undefined) { + return packets[0]; + } + const pathMatches = packets.filter((packet) => expectation.path === undefined || globMatches(expectation.path, packet.path)); + if (expectation.lineRange === undefined) { + return pathMatches[0]; + } + return pathMatches.find((packet) => packet.hunks.some((hunk) => + [...hunk.changedNewLineNumbers, ...hunk.changedOldLineNumbers].some((line) => line >= expectation.lineRange![0] && line <= expectation.lineRange![1]) + )); +} + +function globMatches(pattern: string, value: string): boolean { + const escaped = pattern.replace(/[.+?^${}()|[\]\\]/gu, "\\$&").replace(/\*\*/gu, "\u0000").replace(/\*/gu, "[^/]*").replace(/\u0000/gu, ".*"); + return new RegExp(`^${escaped}$`, "u").test(value); +} + +function requestedLensSignatureForAtom(execution: EvalExecutionInput, atom: ReviewPacket): string | undefined { + const lenses = requestedLensesForAtom(execution, atom); + return lenses === undefined ? undefined : JSON.stringify(lenses); +} + +function requestedLensesForAtom(execution: EvalExecutionInput, atom: ReviewPacket): string[] | undefined { + const decisions = new Map(execution.plan.coverage.map((decision) => [decision.hunkId, decision])); + const lenses = atom.hunks.flatMap((hunk) => { + const decision = decisions.get(hunk.hunkId); + return decision?.lenses ?? []; + }); + if (atom.hunks.some((hunk) => { + const decision = decisions.get(hunk.hunkId); + return decision === undefined || decision.lenses.length === 0 || decision.lenses.some((lens) => lens.trim().length === 0); + })) { + return undefined; + } + return sortedUnique(lenses.map((lens) => lens.trim())); +} + +function compareTargetTreatment( + family: string, + treatments: Record, + failures: ReportFailure[] +): void { + for (const repeat of treatments.A.map((entry) => entry.repeat)) { + const a = treatments.A.find((entry) => entry.repeat === repeat); + const b = treatments.B.find((entry) => entry.repeat === repeat); + const c = treatments.C.find((entry) => entry.repeat === repeat); + if (a === undefined || b === undefined || c === undefined) { + continue; + } + for (const [arm, packed] of [["B", b], ["C", c]] as const) { + if (packed.targetPath !== a.targetPath || packed.targetCoverage !== a.targetCoverage) { + failures.push(failure("target_semantics_mismatch", `${family}/${arm}/repeat ${repeat} changes target path or coverage`)); + } + if (packed.targetFilePackets !== undefined && a.targetFilePackets !== undefined && packed.treated && packed.targetFilePackets >= a.targetFilePackets) { + failures.push(failure("target_packet_count", `${family}/${arm}/repeat ${repeat} treatment does not reduce target-file packet count`)); + } + } + if (b.targetRequestedLensSignature !== a.targetRequestedLensSignature || c.targetRequestedLensSignature !== a.targetRequestedLensSignature) { + failures.push(failure("target_lens_signature", `${family}/repeat ${repeat} packed requested-lens signatures differ from A`)); + } + } +} + +function executionPressure( + execution: EvalExecutionInput, + eventByPacket: Map, + failures: ReportFailure[] +): ExecutionPressure { + const reviewedPacketIds = new Set(execution.modelCalls.filter((call) => call.stage === 7 && call.role === "packetReview" && call.packetId !== undefined).map((call) => call.packetId as string)); + const reviewedAtoms = sum([...reviewedPacketIds].map((packetId) => eventByPacket.get(packetId)?.sourceAtomCount ?? 1)); + const tools = execution.toolCalls.filter((call) => call.stage === 7 && call.initiator === "model" && (call.packetId === undefined || reviewedPacketIds.has(call.packetId))); + const modelCalls = execution.modelCalls.filter((call) => call.stage === 7 && call.role === "packetReview" && call.cacheStatus !== "hit"); + const rejected = tools.filter((call) => call.status === "rejected"); + const reasons: Record = {}; + for (const call of rejected) { + const reason = call.degradationReason ?? call.errorCode ?? "unknown"; + reasons[reason] = (reasons[reason] ?? 0) + 1; + } + if (modelCalls.length > 0 && reviewedAtoms === 0) { + failures.push(failure("missing_reviewed_atom_telemetry", "Stage-7 model calls cannot be joined to reviewed packets/atoms")); + } + const stage7CostUSD = sum(modelCalls.map((call) => call.costUSD ?? 0)); + return { + reviewedPackets: reviewedPacketIds.size, + reviewedAtoms, + requestedToolCalls: tools.length, + usedToolCalls: tools.filter((call) => call.status !== "rejected" && call.status !== "skipped").length, + rejectedToolCalls: rejected.length, + rejectionRatePerAtom: reviewedAtoms === 0 ? 0 : round(rejected.length / reviewedAtoms), + rejectionReasons: reasons, + resultChars: sum(tools.map((call) => call.resultChars)), + continuations: modelCalls.filter((call) => call.kind === "tool-continuation").length, + modelServiceSeconds: round(sum(modelCalls.map((call) => call.durationMs)) / 1000), + inputTokens: sum(modelCalls.map((call) => call.inputTokens ?? 0)), + outputTokens: sum(modelCalls.map((call) => call.outputTokens ?? 0)), + stage7CostUSD: round(stage7CostUSD), + costPerReviewedAtomUSD: reviewedAtoms === 0 ? null : round(stage7CostUSD / reviewedAtoms) + }; +} + +function combinePressure(entries: ExecutionPressure[]): ExecutionPressure { + const reviewedAtoms = sum(entries.map((entry) => entry.reviewedAtoms)); + const rejected = sum(entries.map((entry) => entry.rejectedToolCalls)); + const stage7CostUSD = sum(entries.map((entry) => entry.stage7CostUSD)); + const reasons: Record = {}; + for (const entry of entries) { + for (const [reason, count] of Object.entries(entry.rejectionReasons)) { + reasons[reason] = (reasons[reason] ?? 0) + count; + } + } + return { + reviewedPackets: sum(entries.map((entry) => entry.reviewedPackets)), + reviewedAtoms, + requestedToolCalls: sum(entries.map((entry) => entry.requestedToolCalls)), + usedToolCalls: sum(entries.map((entry) => entry.usedToolCalls)), + rejectedToolCalls: rejected, + rejectionRatePerAtom: reviewedAtoms === 0 ? 0 : round(rejected / reviewedAtoms), + rejectionReasons: reasons, + resultChars: sum(entries.map((entry) => entry.resultChars)), + continuations: sum(entries.map((entry) => entry.continuations)), + modelServiceSeconds: round(sum(entries.map((entry) => entry.modelServiceSeconds)), 3), + inputTokens: sum(entries.map((entry) => entry.inputTokens)), + outputTokens: sum(entries.map((entry) => entry.outputTokens)), + stage7CostUSD: round(stage7CostUSD), + costPerReviewedAtomUSD: reviewedAtoms === 0 ? null : round(stage7CostUSD / reviewedAtoms) + }; +} + +function expectationRates( + executions: EvalExecutionInput[], + include: (execution: EvalExecutionInput) => boolean, + evidenceScores: Map, + atomCounts: Map +): ExpectationRate[] { + const selected = executions.filter(include); + const keys = sortedUnique(executions.flatMap((execution) => (evidenceScores.get(execution.repeat) ?? execution.score).expectationResults + .filter((result) => result.list !== "should_not_find") + .map((result) => `${result.list}\0${result.expectationId}`))); + return keys.map((key) => { + const [list, expectationId] = key.split("\0") as [EvalExpectationList, string]; + let candidateHits = 0; + let finalHits = 0; + const lossHistogram: Record = {}; + const lossByAtomCount: Record = {}; + for (const execution of selected) { + const result = (evidenceScores.get(execution.repeat) ?? execution.score).expectationResults.find((entry) => entry.list === list && entry.expectationId === expectationId); + const atomBucket = String(atomCounts.get(execution.repeat) ?? 1); + const bucket = lossByAtomCount[atomBucket] ?? { denominator: 0, missedBeforeCandidate: 0 }; + bucket.denominator += 1; + lossByAtomCount[atomBucket] = bucket; + if (result === undefined) { + lossHistogram.error = (lossHistogram.error ?? 0) + 1; + continue; + } + if (candidateMatched(result)) { + candidateHits += 1; + } + if (finalMatched(result)) { + finalHits += 1; + } + if (result.loss !== undefined) { + lossHistogram[result.loss.label] = (lossHistogram[result.loss.label] ?? 0) + 1; + if (result.loss.label === "missed-before-candidate-generation") { + bucket.missedBeforeCandidate += 1; + } + } + } + return { + expectationId, + list, + candidateHits, + finalHits, + denominator: selected.length, + candidateRate: selected.length === 0 ? 0 : round(candidateHits / selected.length), + finalRate: selected.length === 0 ? 0 : round(finalHits / selected.length), + lossHistogram, + lossByAtomCount + }; + }); +} + +function candidateMatched(result: EvalExpectationResult): boolean { + if (result.status === "pass") { + return true; + } + return result.list === "should_find" && + (result.loss?.label === "lost-at-verification" || result.loss?.label === "lost-at-composition"); +} + +function finalMatched(result: EvalExpectationResult): boolean { + return result.list === "should_find" && result.status === "pass"; +} + +function enforceRequiredRecallZero( + family: string, + arms: EvalCaseReport["arms"], + failures: ReportFailure[] +): void { + const baselineByKey = new Map(arms.A.intentToTreat.map((rate) => [`${rate.list}\0${rate.expectationId}`, rate])); + for (const arm of ["B", "C"] as const) { + for (const rate of arms[arm].intentToTreat) { + const baseline = baselineByKey.get(`${rate.list}\0${rate.expectationId}`); + if (baseline === undefined) { + failures.push(failure("expectation_join", `${family}/${arm} has an expectation absent from A: ${rate.expectationId}`)); + continue; + } + if ((baseline.candidateHits > 0 && rate.candidateHits === 0) || (baseline.finalHits > 0 && rate.finalHits === 0)) { + failures.push(failure("required_recall_zero", `${family}/${arm}/${rate.expectationId} falls from non-zero recall to zero`)); + } + } + } +} + +function cohortRecallEvidence(cases: EvalCaseReport[], arm: ExperimentArm): { + candidateHits: number; + finalHits: number; + earlyLosses: number; + observations: number; + lossByAtomCount: Record; +} { + const rates = cases.flatMap((entry) => entry.arms[arm].intentToTreat); + const buckets = sortedUnique(rates.flatMap((rate) => Object.keys(rate.lossByAtomCount))); + return { + candidateHits: sum(rates.map((rate) => rate.candidateHits)), + finalHits: sum(rates.map((rate) => rate.finalHits)), + earlyLosses: sum(rates.map((rate) => rate.lossHistogram["missed-before-candidate-generation"] ?? 0)), + observations: sum(rates.map((rate) => rate.denominator)), + lossByAtomCount: Object.fromEntries(buckets.map((bucket) => [bucket, { + denominator: sum(rates.map((rate) => rate.lossByAtomCount[bucket]?.denominator ?? 0)), + missedBeforeCandidate: sum(rates.map((rate) => rate.lossByAtomCount[bucket]?.missedBeforeCandidate ?? 0)) + }])) + }; +} + +function cohortArmRecallPasses(cases: EvalCaseReport[], arm: "B" | "C"): boolean { + const baseline = cohortRecallEvidence(cases, "A"); + const packed = cohortRecallEvidence(cases, arm); + const baselineEarlyRate = baseline.observations === 0 ? 0 : baseline.earlyLosses / baseline.observations; + return baseline.candidateHits - packed.candidateHits <= 1 && baseline.finalHits - packed.finalHits <= 1 && + packed.earlyLosses - baseline.earlyLosses <= 1 && Object.values(packed.lossByAtomCount).every((bucket) => + bucket.missedBeforeCandidate - baselineEarlyRate * bucket.denominator <= 1 + ); +} + +function enforceCohortRecallGates(cases: EvalCaseReport[], failures: ReportFailure[]): void { + const baseline = cohortRecallEvidence(cases, "A"); + const baselineEarlyRate = baseline.observations === 0 ? 0 : baseline.earlyLosses / baseline.observations; + for (const arm of ["B", "C"] as const) { + const packed = cohortRecallEvidence(cases, arm); + if (baseline.candidateHits - packed.candidateHits > 1 || baseline.finalHits - packed.finalHits > 1) { + failures.push(failure("recall_non_inferiority", `${arm} loses more than the single cohort-wide candidate or final hit allowance`, { + baselineCandidate: baseline.candidateHits, + packedCandidate: packed.candidateHits, + baselineFinal: baseline.finalHits, + packedFinal: packed.finalHits + })); + } + if (packed.earlyLosses - baseline.earlyLosses > 1) { + failures.push(failure("candidate_generation_loss", `${arm} exceeds the single cohort-wide allowance for losses before candidate generation`)); + } + for (const [bucket, evidence] of Object.entries(packed.lossByAtomCount)) { + if (evidence.missedBeforeCandidate - baselineEarlyRate * evidence.denominator > 1) { + failures.push(failure("candidate_generation_loss_by_atom_count", `${arm} exceeds the cohort-wide atom-count ${bucket} loss allowance`, { + atomCount: Number(bucket), ...evidence, baselineEarlyRate: round(baselineEarlyRate) + })); + } + } + } +} + +function enforcePressureGate( + family: string, + arms: EvalCaseReport["arms"], + failures: ReportFailure[] +): void { + const aRecall = sum(arms.A.intentToTreat.map((rate) => rate.candidateHits)); + for (const arm of ["B", "C"] as const) { + const packedRecall = sum(arms[arm].intentToTreat.map((rate) => rate.candidateHits)); + if (arms[arm].pressure.rejectionRatePerAtom - arms.A.pressure.rejectionRatePerAtom > 0.1 && packedRecall <= aRecall) { + failures.push(failure("tool_pressure", `${family}/${arm} exceeds A by more than 0.10 rejected attempts per reviewed atom without higher candidate recall`)); + } + } + if (arms.C.pressure.rejectionRatePerAtom > arms.B.pressure.rejectionRatePerAtom) { + failures.push(failure("atom_scaled_pressure", `${family}/C increases normalized tool rejections over B`)); + } +} + +function selectCohortPackedArm( + cases: EvalCaseReport[], + expectedRepeats: number, + failures: ReportFailure[] +): "B" | "C" { + let eligible = new Set<"B" | "C">(["B", "C"]); + const globalBRecallPasses = cohortArmRecallPasses(cases, "B"); + const globalCRecallPasses = cohortArmRecallPasses(cases, "C"); + const eligibilityByCase = cases.map((entry) => ({ + entry, + B: packedArmPasses(entry.arms, "B", expectedRepeats) && globalBRecallPasses, + C: packedArmPasses(entry.arms, "C", expectedRepeats) && globalCRecallPasses && cStrictlyQualifies(entry.arms) + })); + const bCohortEligible = eligibilityByCase.every((entry) => entry.B); + let cCohortEligible = eligibilityByCase.every((entry) => entry.C); + if (!bCohortEligible && cCohortEligible) { + const retention = cohortPacketCountServiceSavingsRetention(cases); + if (retention < 0.85) { + cCohortEligible = false; + failures.push(failure("atom_scaled_savings_retention", "C does not retain at least 85% of cohort packet-count-implied service-time savings while B is ineligible", { + retention: round(retention, 6) + })); + } + } + for (const { entry, B: bPasses, C: caseCPasses } of eligibilityByCase) { + const cPasses = caseCPasses && cCohortEligible; + const passing: Array<"B" | "C"> = []; + if (bPasses) { + passing.push("B"); + } + if (cPasses) { + passing.push("C"); + } + if (passing.length === 0) { + failures.push(failure("no_passing_arm", `${entry.case} has no packed arm that passes treatment, recall, and pressure gates`)); + } + eligible = new Set([...eligible].filter((arm) => passing.includes(arm))); + } + if (eligible.size === 0) { + failures.push(failure("conflicting_arm_selection", "case-level gates require conflicting product arms; one cohort-wide arm cannot be selected")); + eligible = new Set(["B", "C"]); + } + if (eligible.size === 1) { + return [...eligible][0]!; + } + const economics = Object.fromEntries((["B", "C"] as const).map((arm) => [arm, { + costUSD: round(sum(cases.map((entry) => entry.arms[arm].actualCostUSD))), + modelServiceSeconds: round(sum(cases.map((entry) => entry.arms[arm].pressure.modelServiceSeconds)), 3) + }])) as Record<"B" | "C", { costUSD: number; modelServiceSeconds: number }>; + const bDominates = economics.B.costUSD <= economics.C.costUSD && economics.B.modelServiceSeconds <= economics.C.modelServiceSeconds; + const cDominates = economics.C.costUSD <= economics.B.costUSD && economics.C.modelServiceSeconds <= economics.B.modelServiceSeconds; + if (bDominates && !cDominates) { + return "B"; + } + if (cDominates && !bDominates) { + return "C"; + } + if (!bDominates && !cDominates) { + failures.push(failure("conflicting_arm_economics", "B/C trade off actual cost against model-service speed; neither is globally cheaper and faster", economics)); + } + return "B"; +} + +function packedArmPasses(arms: EvalCaseReport["arms"], arm: "B" | "C", expectedRepeats: number): boolean { + if (!arms[arm].treatment.valid) { + return false; + } + if (expectedRepeats < 10) { + return true; + } + const baselineByKey = new Map(arms.A.intentToTreat.map((rate) => [`${rate.list}\0${rate.expectationId}`, rate])); + const requiredZeroPasses = arms[arm].intentToTreat.every((rate) => { + const baseline = baselineByKey.get(`${rate.list}\0${rate.expectationId}`); + if (baseline === undefined) { + return false; + } + return !((baseline.candidateHits > 0 && rate.candidateHits === 0) || (baseline.finalHits > 0 && rate.finalHits === 0)); + }); + const baselineCandidate = sum(arms.A.intentToTreat.map((rate) => rate.candidateHits)); + const packedCandidate = sum(arms[arm].intentToTreat.map((rate) => rate.candidateHits)); + const pressurePasses = arms[arm].pressure.rejectionRatePerAtom - arms.A.pressure.rejectionRatePerAtom <= 0.1 || packedCandidate > baselineCandidate; + const cPressurePasses = arm !== "C" || arms.C.pressure.rejectionRatePerAtom <= arms.B.pressure.rejectionRatePerAtom; + return requiredZeroPasses && pressurePasses && cPressurePasses; +} + +function cStrictlyQualifies(arms: EvalCaseReport["arms"]): boolean { + if (arms.C.pressure.rejectionRatePerAtom > arms.B.pressure.rejectionRatePerAtom) { + return false; + } + const bCandidate = sum(arms.B.intentToTreat.map((rate) => rate.candidateHits)); + const cCandidate = sum(arms.C.intentToTreat.map((rate) => rate.candidateHits)); + const bFinal = sum(arms.B.intentToTreat.map((rate) => rate.finalHits)); + const cFinal = sum(arms.C.intentToTreat.map((rate) => rate.finalHits)); + return arms.C.pressure.rejectionRatePerAtom < arms.B.pressure.rejectionRatePerAtom || cCandidate > bCandidate || cFinal > bFinal; +} + +function cohortPacketCountServiceSavingsRetention(cases: EvalCaseReport[]): number { + const baselinePackets = sum(cases.map((entry) => entry.arms.A.pressure.reviewedPackets)); + const packedPackets = sum(cases.map((entry) => entry.arms.C.pressure.reviewedPackets)); + const baselineService = sum(cases.map((entry) => entry.arms.A.pressure.modelServiceSeconds)); + const packedService = sum(cases.map((entry) => entry.arms.C.pressure.modelServiceSeconds)); + const packetCountSavings = baselinePackets > 0 + ? baselineService * Math.max(0, baselinePackets - packedPackets) / baselinePackets + : 0; + const actualSavings = baselineService - packedService; + return packetCountSavings > 0 ? actualSavings / packetCountSavings : 0; +} + +function duplicateValues(values: string[]): string[] { + return sortedUnique(values.filter((value, index) => values.indexOf(value) !== index)); +} + +function aggregateCohortCost(runs: EvalCaseRunInput[], failures: ReportFailure[]): number { + return round(sum(runs.map((run) => costForRun(run, failures)))); +} + +function costForRun(run: EvalCaseRunInput, failures: ReportFailure[]): number { + const costs = run.executions.map((execution) => { + const providerCalls = execution.modelCalls.filter((call) => call.cacheStatus !== "hit"); + return providerCalls.every((call) => call.costUSD !== undefined) + ? sum(providerCalls.map((call) => call.costUSD ?? 0)) + : undefined; + }); + if (costs.some((cost) => cost === undefined)) { + failures.push(failure("missing_spend_data", `run ${run.runNumber} lacks actual spend data for one or more executions`)); + } + return round(sum(costs.map((cost) => cost ?? 0))); +} + +function reviewedHunksForRun(run: EvalCaseRunInput): number { + return sum(run.executions.map((execution) => execution.reviewedHunkIds.length)); +} + +function validateProductionProvenance( + baseline: EvalCaseRunInput, + selected: EvalCaseRunInput, + failures: ReportFailure[] +): void { + for (const [label, run] of [["baseline", baseline], ["selected", selected]] as const) { + const repo = run.info.repo; + if ( + repo?.root !== PRODUCTION_REPO_ROOT || repo.baseSha !== PRODUCTION_BASE_SHA || + repo.mergeBase !== PRODUCTION_BASE_SHA || repo.headSha !== PRODUCTION_HEAD_SHA + ) { + failures.push(failure("production_refs", `${label} production run does not use the exact pinned Plan 102 refs`)); + } + const command = run.declaredCase.command; + if ( + run.declaredCase.repo?.external !== PRODUCTION_REPO_ROOT || command?.base !== PRODUCTION_BASE_SHA || command.head !== PRODUCTION_HEAD_SHA + ) { + failures.push(failure("production_refs", `${label} production case does not declare the exact pinned Plan 102 base/head`)); + } + const review = run.declaredCase.review; + const effective = run.info.effectiveConfig; + if ( + review?.cache !== false || run.info.cache.enabled !== false || review.concurrency !== PRODUCTION_CONCURRENCY || + review.maxTimeMinutes !== 60 || effective?.review.concurrency !== PRODUCTION_CONCURRENCY || + effective.review.timeoutMs !== PRODUCTION_TIMEOUT_MS || effective.llm.maxConcurrentCalls !== PRODUCTION_CONCURRENCY + ) { + failures.push(failure("production_run_shape", `${label} production arm must use the exact external repo, cache-off, concurrency-6, 60-minute declared/effective run shape`)); + } + } + const baselineRuntime = baseline.info.codegenieRuntime; + const selectedRuntime = selected.info.codegenieRuntime; + if ( + baselineRuntime?.commit === undefined || selectedRuntime?.commit === undefined || + baselineRuntime.commit !== selectedRuntime.commit || baselineRuntime.dirty !== false || selectedRuntime.dirty !== false + ) { + failures.push(failure("production_runtime_provenance", "production arms must share one present clean Codegenie runtime commit")); + } +} + +function validateProductionExpectations( + baseline: EvalCaseRunInput, + selected: EvalCaseRunInput, + failures: ReportFailure[] +): void { + const canonical = declaredExpectationKeys(baseline.declaredCase); + const selectedDeclared = declaredExpectationKeys(selected.declaredCase); + if ( + canonical.values.length === 0 || canonical.duplicates.length > 0 || selectedDeclared.duplicates.length > 0 || + stableJson(canonical.values) !== stableJson(selectedDeclared.values) + ) { + failures.push(failure("declared_expectation_join", "production arms do not declare the exact same unique expectation set")); + } + const requiredKeys = new Set((["should_find", "should_find_candidate", "should_not_find"] as const).flatMap((list) => + (baseline.declaredCase[list] ?? []).filter((expectation) => expectation.tier !== "optional").map((expectation) => `${list}\0${expectation.id}`) + )); + for (const [label, run] of [["baseline", baseline], ["selected", selected]] as const) { + for (const execution of run.executions) { + const actual = expectationKeys(execution.score); + if (actual.duplicates.length > 0 || stableJson(actual.values) !== stableJson(canonical.values)) { + failures.push(failure("expectation_join", `${label} production score does not match its declared expectations bidirectionally`)); + } + const candidateIds = new Set(execution.candidateFindings.map((finding) => finding.id)); + const finalIds = new Set(execution.finalFindings.map((finding) => finding.id)); + validateProductionFindingArtifacts(execution, failures, label); + for (const result of execution.score.expectationResults) { + const expectedArtifact = result.list === "should_find_candidate" ? "candidate-findings" : "final-findings"; + if (result.matched.some((match) => + match.artifact !== expectedArtifact || + (match.artifact === "candidate-findings" ? !candidateIds.has(match.findingId) : !finalIds.has(match.findingId)) + )) { + failures.push(failure("production_finding_artifact_join", `${label} production score references a finding absent from its strict candidate/final artifacts`)); + } + const requiredPositive = requiredKeys.has(`${result.list}\0${result.expectationId}`) && result.list !== "should_not_find"; + if ( + (requiredKeys.has(`${result.list}\0${result.expectationId}`) && result.status !== "pass") || + requiredPositive && result.matched.length === 0 + ) { + failures.push(failure("production_finding_preservation", `${label} production arm does not preserve a required declared finding outcome`)); + } + } + } + } +} + +function validateProductionFindingArtifacts( + execution: EvalExecutionInput, + failures: ReportFailure[], + label: string +): void { + const packetIds = new Set(execution.packets.map((packet) => packet.id)); + const hunkPaths = new Map(execution.diff.files.flatMap((file) => file.hunks.map((hunk) => [hunk.id, hunk.path] as const))); + const candidateIds = new Set(execution.candidateFindings.map((finding) => finding.id)); + const validAnchor = (anchor: { hunkId: string; path: string } | undefined): boolean => + anchor === undefined || hunkPaths.get(anchor.hunkId) === anchor.path; + const candidateRelationsValid = execution.candidateFindings.every((finding) => + packetIds.has(finding.producedBy.packetId) && validAnchor(finding.anchor) + ); + const finalRelationsValid = execution.finalFindings.every((finding) => + packetIds.has(finding.producedBy.packetId) && validAnchor(finding.anchor) && + finding.mergedCandidateIds.length > 0 && finding.mergedCandidateIds.every((id) => candidateIds.has(id)) && + (finding.mergedAnchors ?? []).every((anchor) => validAnchor(anchor)) + ); + const inline = execution.finalFindings.filter((finding) => finding.publication === "inline").length; + const summaryOnly = execution.finalFindings.filter((finding) => finding.publication === "summary-only").length; + const suppressed = execution.finalFindings.filter((finding) => finding.publication === "suppressed").length; + const metrics = execution.score.metrics; + const countsValid = + metrics.candidateFindings === execution.candidateFindings.length && metrics.inlineFindings === inline && + metrics.summaryOnlyFindings === summaryOnly && metrics.suppressedFindings === suppressed && + metrics.reportedFindings === inline + summaryOnly; + if (!candidateRelationsValid || !finalRelationsValid || !countsValid) { + failures.push(failure("production_finding_relations", `${label} production candidate/final artifacts do not join exactly to packets, hunks, lineage, and score counts`, { + candidates: execution.candidateFindings.length, + finals: execution.finalFindings.length, + packets: packetIds.size, + hunks: hunkPaths.size + })); + } +} + +function validateProductionModelAccounting( + run: EvalCaseRunInput, + treatments: TreatmentExecution[], + failures: ReportFailure[], + label: string +): void { + for (const execution of run.executions) { + const treatment = treatments.find((entry) => entry.repeat === execution.repeat); + const reviewedHunks = new Set(execution.reviewedHunkIds); + const reviewedPacketIds = new Set(execution.packets.filter((packet) => packet.hunks.some((hunk) => reviewedHunks.has(hunk.hunkId))).map((packet) => packet.id)); + const stage7 = execution.modelCalls.filter((call) => call.stage === 7 && call.role === "packetReview"); + const initialPacketIds = stage7.filter((call) => call.kind === "initial" && call.packetId !== undefined).map((call) => call.packetId as string); + const completeFields = execution.modelCalls.every((call) => + call.inputTokens !== undefined && call.outputTokens !== undefined && call.reasoningTokens !== undefined && call.totalTokens !== undefined && + call.costUSD !== undefined && call.durationMs > 0 && call.totalTokens >= call.inputTokens + call.outputTokens && call.status === "ok" + ); + const joined = stage7.length > 0 && stage7.every((call) => call.packetId !== undefined && reviewedPacketIds.has(call.packetId) && call.status === "ok") && + unique(initialPacketIds).length === initialPacketIds.length && stableJson([...initialPacketIds].sort()) === stableJson([...reviewedPacketIds].sort()); + const scoreCount = execution.score.metrics.modelCalls; + const scoreCost = execution.score.metrics.costUSD; + const recordedCost = sum(execution.modelCalls.map((call) => call.costUSD ?? 0)); + const scoreJoined = scoreCount === execution.modelCalls.length && scoreCost !== undefined && Math.abs(scoreCost - recordedCost) <= 1e-9; + if (!completeFields || !joined || !scoreJoined || reviewedPacketIds.size === 0 || treatment === undefined || treatment.pressure.reviewedAtoms <= 0) { + failures.push(failure("production_model_accounting", `${label} production model calls are incomplete or do not join exactly to score/reviewed packets/atoms`, { + repeat: execution.repeat, + modelCalls: execution.modelCalls.length, + scoreModelCalls: scoreCount ?? -1, + reviewedPackets: reviewedPacketIds.size, + reviewedAtoms: treatment?.pressure.reviewedAtoms ?? 0 + })); + } + } +} + +function analyzeProductionThroughput( + baselineRun: EvalCaseRunInput, + baselineTreatment: TreatmentExecution[], + selectedRun: EvalCaseRunInput, + selectedTreatment: TreatmentExecution[], + failures: ReportFailure[] +): ProductionThroughput { + const baseline = productionArmThroughput(baselineRun, baselineTreatment, failures, "baseline"); + const selected = productionArmThroughput(selectedRun, selectedTreatment, failures, "selected"); + if (stableJson(baseline.reviewedHunkIds) !== stableJson(selected.reviewedHunkIds)) { + failures.push(failure("production_hunk_loss", "selected production arm does not preserve the exact reviewed-hunk set from baseline", { + baselineCount: baseline.reviewedHunks, + selectedCount: selected.reviewedHunks, + baselineSetHash: sha256Hex(baseline.reviewedHunkIds.join("\n")), + selectedSetHash: sha256Hex(selected.reviewedHunkIds.join("\n")) + })); + } + if (baseline.reviewedHunks !== EQUIVALENT_TARGET_HUNKS || selected.reviewedHunks !== EQUIVALENT_TARGET_HUNKS) { + failures.push(failure("production_incomplete", `production comparison must review exactly ${EQUIVALENT_TARGET_HUNKS} hunks in both arms`, { + baseline: baseline.reviewedHunks, + selected: selected.reviewedHunks + })); + } + if (!(selected.reviewedHunksPerWallSecond > baseline.reviewedHunksPerWallSecond)) { + failures.push(failure("production_throughput", "selected production arm does not improve reviewed hunks per wall second")); + } + if (!(selected.wallTimeSeconds <= baseline.wallTimeSeconds)) { + failures.push(failure("production_wall_time", "selected production arm increases total wall time")); + } + if (!(selected.modelServiceSecondsPerReviewedHunk < baseline.modelServiceSecondsPerReviewedHunk)) { + failures.push(failure("production_model_service", "selected production arm does not strictly improve model-service seconds per reviewed hunk")); + } + if (!(selected.tokensPerReviewedHunk < baseline.tokensPerReviewedHunk)) { + failures.push(failure("production_tokens", "selected production arm does not strictly improve tokens per reviewed hunk")); + } + if (!(selected.reasoningTokensPerReviewedHunk <= baseline.reasoningTokensPerReviewedHunk)) { + failures.push(failure("production_reasoning_tokens", "selected production arm increases reasoning tokens per reviewed hunk")); + } + if (selected.continuations > baseline.continuations) { + const retention = productionPacketCountServiceSavingsRetention(baseline, selected); + if (retention < 0.85) { + failures.push(failure("production_continuation_savings_retention", "selected production arm adds continuations without retaining at least 85% of packet-count-implied all-stage model-service savings", { + retention: round(retention, 6), + baselineContinuations: baseline.continuations, + selectedContinuations: selected.continuations + })); + } + } + if (selected.pressure.rejectionRatePerAtom - baseline.pressure.rejectionRatePerAtom > 0.1) { + failures.push(failure("production_tool_pressure", "selected production arm increases normalized tool rejection pressure by more than 0.10")); + } + return { baseline, selected }; +} + +function productionPacketCountServiceSavingsRetention( + baseline: ProductionArmThroughput, + selected: ProductionArmThroughput +): number { + const packetCountSavings = baseline.reviewedPackets > 0 + ? baseline.modelServiceSeconds * Math.max(0, baseline.reviewedPackets - selected.reviewedPackets) / baseline.reviewedPackets + : 0; + const actualSavings = baseline.modelServiceSeconds - selected.modelServiceSeconds; + return packetCountSavings > 0 ? actualSavings / packetCountSavings : 0; +} + +function productionArmThroughput( + run: EvalCaseRunInput, + treatments: TreatmentExecution[], + failures: ReportFailure[], + label: string +): ProductionArmThroughput { + const rawReviewedHunkIds = run.executions.flatMap((execution) => execution.reviewedHunkIds); + if (unique(rawReviewedHunkIds).length !== rawReviewedHunkIds.length) { + failures.push(failure("production_hunk_bijection", `${label} production arm repeats reviewed hunk IDs`)); + } + const reviewedHunkIds = sortedUnique(rawReviewedHunkIds); + const pressure = combinePressure(treatments.map((entry) => entry.pressure)); + const allModelCalls = run.executions.flatMap((execution) => execution.modelCalls); + const reviewedHunks = reviewedHunkIds.length; + const wallTimeSeconds = round(sum(run.executions.map((execution) => execution.wallTimeSeconds)), 3); + if (!(wallTimeSeconds > 0)) { + failures.push(failure("missing_wall_time", `${label} production arm lacks positive wall-time evidence`)); + } + const modelServiceSeconds = round(sum(allModelCalls.map((call) => call.durationMs)) / 1000, 3); + const totalTokens = sum(allModelCalls.map((call) => call.totalTokens ?? 0)); + const reasoningTokens = sum(allModelCalls.map((call) => call.reasoningTokens ?? 0)); + return { + reviewedHunkIds, + reviewedHunks, + reviewedAtoms: pressure.reviewedAtoms, + reviewedPackets: pressure.reviewedPackets, + wallTimeSeconds, + reviewedHunksPerWallSecond: wallTimeSeconds > 0 ? round(reviewedHunks / wallTimeSeconds) : 0, + modelServiceSeconds, + modelServiceSecondsPerReviewedHunk: reviewedHunks > 0 ? round(modelServiceSeconds / reviewedHunks) : 0, + totalTokens, + tokensPerReviewedHunk: reviewedHunks > 0 ? round(totalTokens / reviewedHunks) : 0, + reasoningTokens, + reasoningTokensPerReviewedHunk: reviewedHunks > 0 ? round(reasoningTokens / reviewedHunks) : 0, + continuations: pressure.continuations, + continuationsPerReviewedAtom: pressure.reviewedAtoms > 0 ? round(pressure.continuations / pressure.reviewedAtoms) : 0, + pressure + }; +} + +export function computeProductionEconomics( + baseline: { actualCostUSD: number; reviewedHunks: number }, + selected: { actualCostUSD: number; reviewedHunks: number }, + validationCostInputUSD: number, + label: ProductionEconomics["validationCostInputLabel"] = "explicit_cumulative_validation_cost", + failures: ReportFailure[] = [] +): ProductionEconomics { + if (!(baseline.actualCostUSD >= 0) || !(selected.actualCostUSD >= 0)) { + failures.push(failure("missing_spend_data", "production arm costs must be finite non-negative values")); + } + if (!(baseline.reviewedHunks > 0) || !(selected.reviewedHunks > 0)) { + failures.push(failure("missing_reviewed_hunks", "production arms must report positive reviewed-hunk counts")); + } + const baselinePerHunk = baseline.reviewedHunks > 0 ? baseline.actualCostUSD / baseline.reviewedHunks : Number.POSITIVE_INFINITY; + const selectedPerHunk = selected.reviewedHunks > 0 ? selected.actualCostUSD / selected.reviewedHunks : Number.POSITIVE_INFINITY; + const baselineEquivalent = baselinePerHunk * EQUIVALENT_TARGET_HUNKS; + const selectedEquivalent = selectedPerHunk * EQUIVALENT_TARGET_HUNKS; + const savings = baselineEquivalent - selectedEquivalent; + if (!(savings > 0) || !Number.isFinite(savings)) { + failures.push(failure("non_positive_payback_denominator", "normalized equivalent-review savings must be positive", { + baselineEquivalent, + selectedEquivalent, + savings + })); + } + const breakEven = savings > 0 && Number.isFinite(savings) ? Math.ceil(validationCostInputUSD / savings) : 0; + return { + equivalentTargetHunks: EQUIVALENT_TARGET_HUNKS, + baseline: { + actualCostUSD: round(baseline.actualCostUSD), + reviewedHunks: baseline.reviewedHunks, + costPerReviewedHunkUSD: round(baselinePerHunk), + equivalentReviewCostUSD: round(baselineEquivalent), + equivalentCostExtrapolated: baseline.reviewedHunks < EQUIVALENT_TARGET_HUNKS + }, + selected: { + actualCostUSD: round(selected.actualCostUSD), + reviewedHunks: selected.reviewedHunks, + costPerReviewedHunkUSD: round(selectedPerHunk), + equivalentReviewCostUSD: round(selectedEquivalent), + equivalentCostExtrapolated: selected.reviewedHunks < EQUIVALENT_TARGET_HUNKS + }, + equivalentReviewSavingsUSD: round(savings), + validationCostInputUSD: round(validationCostInputUSD), + validationCostInputLabel: label, + breakEvenReviewCount: breakEven + }; +} + +export function analyzeRegressionCohorts( + baselineCohort: CohortSelection, + selectedCohort: CohortSelection, + expectedRepeats: number +): RegressionReport { + const failures: ReportFailure[] = []; + const baselineByName = new Map(baselineCohort.runs.map((run) => [run.info.caseName, run])); + const selectedByName = new Map(selectedCohort.runs.map((run) => [run.info.caseName, run])); + const allNames = sortedUnique([...baselineByName.keys(), ...selectedByName.keys()]); + const cases: RegressionReport["cases"] = []; + for (const caseName of allNames) { + const baseline = baselineByName.get(caseName); + const selected = selectedByName.get(caseName); + if (baseline === undefined || selected === undefined) { + failures.push(failure("regression_case_join", `case ${caseName} is missing from one regression cohort`)); + continue; + } + validateRunRepeatCount(baseline, expectedRepeats, failures); + validateRunRepeatCount(selected, expectedRepeats, failures); + const baselineEvidenceScores = validateRunEvidence(baseline, `${caseName}/baseline`, failures); + const selectedEvidenceScores = validateRunEvidence(selected, `${caseName}/selected`, failures); + enforceRequiredRegressionOutcomes(baseline, baselineEvidenceScores, `${caseName}/baseline`, failures); + enforceRequiredRegressionOutcomes(selected, selectedEvidenceScores, `${caseName}/selected`, failures); + validateRunScoreExpectations(`${caseName}/baseline`, baseline, failures); + validateRunScoreExpectations(`${caseName}/selected`, selected, failures); + let baselineArm: ExperimentArm | undefined; + let selectedArm: ExperimentArm | undefined; + try { + baselineArm = armForRun(baseline); + selectedArm = armForRun(selected); + } catch (error) { + if (error instanceof PacketPackingReportError) { + failures.push(...error.failures); + } else { + throw error; + } + } + if (baselineArm !== "A" || selectedArm === "A") { + failures.push(failure("regression_arm_config", `${caseName} must compare packing-off/base to one selected packed arm`)); + } + if (stableJson(normalizeExperimentSnapshot(baseline.declaredCase)) !== stableJson(normalizeExperimentSnapshot(selected.declaredCase))) { + failures.push(failure("regression_yaml_delta", `${caseName} differs outside repeat and packet-packing review settings`)); + } + if (baseline.info.score.status === "error" || selected.info.score.status === "error") { + failures.push(failure("eval_error", `${caseName} has an errored collateral eval`)); + } + const transitions = expectationTransitions(baseline.info.score, selected.info.score); + for (const transition of transitions) { + if (transition.from === "pass" && transition.to !== "pass") { + failures.push(failure("collateral_expectation_regression", `${caseName}/${transition.expectationId} regresses from pass to ${transition.to}`)); + } + } + const baselineTreatment = baseline.executions.map((execution) => analyzeTreatmentExecution(baseline, execution, "A")); + const effectiveSelectedArm = selectedArm === "C" ? "C" : "B"; + const selectedTreatment = selected.executions.map((execution) => analyzeTreatmentExecution( + selected, + execution, + effectiveSelectedArm, + baseline.executions.find((candidate) => candidate.repeat === execution.repeat), + false + )); + failures.push(...baselineTreatment.flatMap((entry) => entry.failures), ...selectedTreatment.flatMap((entry) => entry.failures)); + cases.push({ + caseName, + baselineRun: baseline.runNumber, + selectedRun: selected.runNumber, + expectationTransitions: transitions, + selectedTreatmentExecutions: selectedTreatment.filter((entry) => entry.treated).length, + dispatchOrderChanges: dispatchOrderChanges(baseline.executions, selected.executions), + baselinePressure: combinePressure(baselineTreatment.map((entry) => entry.pressure)), + selectedPressure: combinePressure(selectedTreatment.map((entry) => entry.pressure)) + }); + } + const baselineCost = aggregateCohortCost(baselineCohort.runs, failures); + const selectedCost = aggregateCohortCost(selectedCohort.runs, failures); + return { + schemaVersion: 1, + mode: "regression", + evidence: "one-repeat-collateral-only", + baselineCohort: { id: baselineCohort.id, runNumbers: baselineCohort.runs.map((run) => run.runNumber), actualCostUSD: baselineCost }, + selectedCohort: { id: selectedCohort.id, runNumbers: selectedCohort.runs.map((run) => run.runNumber), actualCostUSD: selectedCost }, + expectedRepeats, + cases, + failures + }; +} + +function validateRunScoreExpectations(label: string, run: EvalCaseRunInput, failures: ReportFailure[]): void { + const declared = declaredExpectationKeys(run.declaredCase); + const scores = [run.info.score, ...run.executions.map((execution) => execution.score)]; + for (const [index, score] of scores.entries()) { + const actual = expectationKeys(score); + if (declared.duplicates.length > 0 || actual.duplicates.length > 0 || stableJson(actual.values) !== stableJson(declared.values)) { + failures.push(failure("expectation_join", `${label}/score ${index} does not match the exact declared expectation set`)); + } + } +} + +function enforceRequiredRegressionOutcomes( + run: EvalCaseRunInput, + scores: EvalScore[], + label: string, + failures: ReportFailure[] +): void { + const required = new Set(([ + ...((run.declaredCase.should_find ?? []).map((expectation) => ({ list: "should_find" as const, expectation }))), + ...((run.declaredCase.should_find_candidate ?? []).map((expectation) => ({ list: "should_find_candidate" as const, expectation }))), + ...((run.declaredCase.should_not_find ?? []).map((expectation) => ({ list: "should_not_find" as const, expectation }))) + ]).filter((entry) => entry.expectation.tier !== "optional").map((entry) => `${entry.list}\0${entry.expectation.id}`)); + for (const [index, score] of scores.entries()) { + for (const result of score.expectationResults) { + if (!required.has(`${result.list}\0${result.expectationId}`)) { + continue; + } + const passes = result.status === "pass" && (result.list === "should_not_find" || result.matched.length > 0); + if (!passes) { + failures.push(failure("collateral_cohort_expectation_failure", `${label}/repeat ${run.executions[index]?.repeat ?? index + 1} fails a required ${result.list} expectation`, { + expectationId: result.expectationId, + list: result.list + })); + } + } + } +} + +function expectationTransitions(baseline: EvalScore, selected: EvalScore): RegressionReport["cases"][number]["expectationTransitions"] { + const baselineByKey = new Map(baseline.expectationResults.map((result) => [`${result.list}\0${result.expectationId}`, result])); + const selectedByKey = new Map(selected.expectationResults.map((result) => [`${result.list}\0${result.expectationId}`, result])); + return sortedUnique([...baselineByKey.keys(), ...selectedByKey.keys()]).flatMap((key) => { + const before = baselineByKey.get(key); + const after = selectedByKey.get(key); + const [list, expectationId] = key.split("\0"); + const from = before?.status ?? "missing"; + const to = after?.status ?? "missing"; + return from === to ? [] : [{ expectationId: expectationId ?? "unknown", list: list ?? "unknown", from, to }]; + }); +} + +function dispatchOrderChanges(baseline: EvalExecutionInput[], selected: EvalExecutionInput[]): number { + let changes = 0; + for (const baselineExecution of baseline) { + const selectedExecution = selected.find((entry) => entry.repeat === baselineExecution.repeat); + if (selectedExecution === undefined) { + continue; + } + const baselineOrder = schedulingOrder(baselineExecution.packets); + const selectedOrder = schedulingOrder(selectedExecution.packets); + const baselineByHunk = new Map(baselineOrder.flatMap((packet, index) => packet.hunks.map((hunk) => [hunk.hunkId, index] as const))); + const selectedByHunk = new Map(selectedOrder.flatMap((packet, index) => packet.hunks.map((hunk) => [hunk.hunkId, index] as const))); + changes += [...baselineByHunk].filter(([hunkId, index]) => selectedByHunk.get(hunkId) !== index).length; + } + return changes; +} + +type ReplayReport = { + schemaVersion: 1; + mode: "replay"; + noModelCalls: boolean; + repo: string; + rows: ReplayRow[]; + failures: ReportFailure[]; +}; + +async function loadJsonValue(filePath: string, description: string): Promise { + let raw: string; + try { + raw = await readFile(filePath, "utf8"); + } catch (error) { + fail("missing_artifact", `failed to read ${description}: ${filePath}`, { error: error instanceof Error ? error.message : String(error) }); + } + try { + return JSON.parse(raw) as unknown; + } catch (error) { + fail("corrupt_artifact", `failed to parse ${description}: ${filePath}`, { error: error instanceof Error ? error.message : String(error) }); + } +} + +async function loadValidatedJson(filePath: string, description: string, schema: S): Promise> { + const value = await loadJsonValue(filePath, description); + const parsed = schema.safeParse(value); + assertReport(parsed.success, "corrupt_artifact_schema", `invalid ${description}: ${filePath}`, { + issues: parsed.success ? [] : parsed.error.issues.map((issue) => ({ path: issue.path.join("."), code: issue.code, message: issue.message })) + }); + return parsed.data; +} + +async function loadValidatedJsonLines(filePath: string, description: string, schema: S): Promise>> { + if (!existsSync(filePath)) { + fail("missing_artifact", `failed to read ${description}: ${filePath}`); + } + const raw = await readFile(filePath, "utf8"); + const result: Array> = []; + for (const [index, line] of raw.split("\n").entries()) { + if (line.trim().length === 0) { + continue; + } + let value: unknown; + try { + value = JSON.parse(line) as unknown; + } catch (error) { + fail("corrupt_artifact", `failed to parse ${description} line ${index + 1}: ${filePath}`, { + error: error instanceof Error ? error.message : String(error) + }); + } + const parsed = schema.safeParse(value); + assertReport(parsed.success, "corrupt_artifact_schema", `invalid ${description} line ${index + 1}: ${filePath}`, { + issues: parsed.success ? [] : parsed.error.issues.map((issue) => ({ path: issue.path.join("."), code: issue.code, message: issue.message })) + }); + result.push(parsed.data); + } + return result; +} + +function scoringHintEvents(events: TelemetryEvent[]): EvalArtifacts["hintEvents"] { + return events.flatMap((event) => { + if (event.message !== "follow_up_hint" && event.message !== "uncertainty") { + return []; + } + const data = event.data ?? {}; + const question = typeof data.question === "string" ? data.question : ""; + if (question.trim().length === 0) { + return []; + } + const strings = (value: unknown): string[] => Array.isArray(value) ? value.filter((entry): entry is string => typeof entry === "string") : []; + const confidence = data.confidence === "high" || data.confidence === "medium" || data.confidence === "low" ? data.confidence : "low"; + const packetId = event.packetId ?? (typeof data.packetId === "string" ? data.packetId : undefined); + return [{ + ...(packetId === undefined ? {} : { packetId }), + question, + files: strings(data.files), + symbols: strings(data.symbols), + projectedSkillIds: strings(data.projectedSkillIds), + ...(typeof data.reason === "string" ? { reason: data.reason } : {}), + confidence + }]; + }); +} + +function scoringHumanAttentionNotes(raw: unknown): NonNullable { + const entries = Array.isArray(raw) + ? raw + : isRecord(raw) && Array.isArray(raw.groups) + ? raw.groups + : []; + return entries.flatMap((entry) => { + if (!isRecord(entry) || typeof entry.question !== "string" || entry.question.length === 0) { + return []; + } + const files = Array.isArray(entry.files) ? entry.files.filter((value): value is string => typeof value === "string") : []; + const reasons = Array.isArray(entry.reasons) + ? entry.reasons.filter((value): value is string => typeof value === "string") + : typeof entry.reason === "string" + ? [entry.reason] + : []; + return [{ question: entry.question, files, reasons }]; + }); +} + +async function loadPackets(packetDir: string, options: { allowLegacyDispatchRank?: boolean } = {}): Promise { + let entries: string[]; + try { + entries = (await readdir(packetDir)).filter((entry) => entry.endsWith(".json")).sort(); + } catch (error) { + fail("missing_artifact", `failed to read packet artifacts: ${packetDir}`, { error: error instanceof Error ? error.message : String(error) }); + } + assertReport(entries.length > 0, "missing_artifact", `no packet artifacts found in ${packetDir}`); + const parsedPackets = await Promise.all(entries.map((entry) => loadValidatedJson(path.join(packetDir, entry), "packet artifact", reviewPacketSchema))); + const packetIds = new Set(); + const hunkIds = new Set(); + for (const [index, packet] of parsedPackets.entries()) { + assertReport(entries[index] === `${packet.id}.json`, "packet_filename_mismatch", `packet filename does not match packet ID in ${packetDir}`); + assertReport(!packetIds.has(packet.id), "duplicate_packet_id", `duplicate packet ID ${packet.id} in ${packetDir}`); + packetIds.add(packet.id); + assertReport(options.allowLegacyDispatchRank === true || packet.dispatchRank !== undefined, "missing_dispatch_rank", `packet ${packet.id} has no dispatch rank`); + for (const hunk of packet.hunks) { + assertReport(!hunkIds.has(hunk.hunkId), "duplicate_packet_hunk", `hunk ${hunk.hunkId} appears in multiple packet artifacts`); + hunkIds.add(hunk.hunkId); + } + } + return parsedPackets as unknown as ReviewPacket[]; +} + +export async function loadEvalRuns(logsDir: string, selector?: string): Promise { + let locations: Array<{ logsRoot: string; runPath: string; runNumber: number }>; + try { + await readdir(logsDir); + } catch (error) { + fail("missing_logs", `failed to read eval log root: ${logsDir}`, { error: error instanceof Error ? error.message : String(error) }); + } + const selectedManifest = selector === undefined ? undefined : await loadSelectedInvocationManifest(logsDir, selector); + if (selectedManifest !== undefined) { + locations = selectedManifest.runs.map((run) => ({ logsRoot: path.resolve(run.logsRoot), runPath: run.runPath, runNumber: run.runNumber })); + for (const ownedRoot of sortedUnique(locations.map((location) => location.logsRoot))) { + const ownedManifest = await loadValidatedJson( + path.join(ownedRoot, "invocations", `${selectedManifest.invocationId}.json`), + "root-owned eval invocation manifest", + evalInvocationManifestSchema + ) as EvalInvocationManifest; + assertReport( + stableJson(ownedManifest) === stableJson(selectedManifest), + "invocation_root_ownership", + `invocation ${selectedManifest.invocationId} differs across its declared log roots` + ); + } + } else { + try { + locations = (await readdir(logsDir)).filter((entry) => /^\d+$/u.test(entry)).sort((a, b) => Number(a) - Number(b)) + .map((runPath) => ({ logsRoot: path.resolve(logsDir), runPath, runNumber: Number(runPath) })); + } catch (error) { + fail("missing_logs", `failed to read eval log root: ${logsDir}`, { error: error instanceof Error ? error.message : String(error) }); + } + } + const runs: EvalCaseRunInput[] = []; + const manifests = new Map(); + if (selectedManifest !== undefined) { + manifests.set(selectedManifest.invocationId, selectedManifest); + } + const declarations = new Map(); + for (const location of locations) { + const runDir = path.resolve(location.logsRoot, location.runPath); + assertReport( + runDir === path.join(location.logsRoot, String(location.runNumber)) && path.dirname(runDir) === location.logsRoot, + "invocation_root_ownership", + "eval invocation run path does not resolve to its exact declared owning root" + ); + const info = await loadValidatedJson(path.join(runDir, "info.json"), "eval info", evalRunInfoSchema) as unknown as EvalRunInfo; + assertReport(info.runNumber === location.runNumber, "corrupt_eval_info", `eval info run number does not match directory ${runDir}`); + assertReport(info.caseSnapshot.name === info.caseName, "corrupt_eval_info", `eval case snapshot name does not match case name in ${runDir}`); + let invocationManifest: EvalInvocationManifest | undefined; + let declaredCase = info.caseSnapshot; + if (info.invocation !== undefined) { + assertReport(info.invocation.manifest === `invocations/${info.invocation.id}.json`, "invocation_manifest_join", `eval invocation manifest path is not canonical in ${runDir}`); + invocationManifest = manifests.get(info.invocation.id); + if (invocationManifest === undefined) { + invocationManifest = await loadValidatedJson( + path.join(location.logsRoot, info.invocation.manifest), + "eval invocation manifest", + evalInvocationManifestSchema + ) as EvalInvocationManifest; + manifests.set(info.invocation.id, invocationManifest); + } + validateInvocationManifestRun(invocationManifest, info, runDir); + const declarationKey = `${invocationManifest.invocationId}\0${info.invocation.caseIndex}`; + declaredCase = declarations.get(declarationKey) ?? await loadDeclaredEvalCase(invocationManifest, info); + declarations.set(declarationKey, declaredCase); + } + const repeat = info.caseSnapshot.repeat ?? 1; + const executions: EvalExecutionInput[] = []; + for (let index = 1; index <= repeat; index += 1) { + const executionDir = repeat > 1 ? path.join(runDir, "repeats", String(index)) : runDir; + const telemetryDir = path.join(executionDir, "telemetry"); + const score = repeat > 1 + ? await loadValidatedJson(path.join(executionDir, "score.json"), "repeat score", evalScoreSchema) as unknown as EvalScore + : info.score; + const packets = await loadPackets(path.join(telemetryDir, "stages", "06-packets", "packets")); + const events = await loadValidatedJsonLines(path.join(telemetryDir, "events.jsonl"), "telemetry events", telemetryEventSchema) as TelemetryEvent[]; + const modelCalls = await loadValidatedJsonLines(path.join(telemetryDir, "model-calls.jsonl"), "model calls", llmCallRecordSchema) as LlmCallRecord[]; + const toolCalls = await loadValidatedJsonLines(path.join(telemetryDir, "tool-calls.jsonl"), "tool calls", toolCallRecordSchema) as ToolCallRecord[]; + const fileFacts = await loadValidatedJson(path.join(telemetryDir, "stages", "03-classify", "file-facts.json"), "eval file facts", z.array(fileFactsSchema)) as FileFacts[]; + const diff = await loadValidatedJson(path.join(telemetryDir, "stages", "02-diff", "diff.json"), "eval diff", unifiedDiffSchema) as unknown as UnifiedDiff; + const plan = await loadValidatedJson(path.join(telemetryDir, "stages", "05-planner", "review-plan.json"), "eval review plan", reviewPlanSchema) as unknown as ReviewPlan; + const candidates = await loadValidatedJson(path.join(telemetryDir, "stages", "09-verification", "candidate-findings.json"), "candidate findings", z.array(candidateFindingSchema)); + const verification = await loadValidatedJson(path.join(telemetryDir, "stages", "09-verification", "verification.json"), "verification records", z.array(verificationRecordSchema)); + const finalSelectionArtifact = await loadValidatedJson( + path.join(telemetryDir, "stages", "10-composition", "final-selection.json"), + "final selection", + finalSelectionArtifactSchema + ); + const finalFindings = await loadValidatedJson(path.join(telemetryDir, "stages", "10-composition", "final-findings.json"), "final findings", z.array(finalFindingSchema)); + const coverage = await loadValidatedJson(path.join(telemetryDir, "stages", "10-composition", "coverage.json"), "coverage", coverageArtifactSchema); + const attention = await loadValidatedJson(path.join(telemetryDir, "stages", "10-composition", "attention.json"), "attention records", z.array(attentionRecordSchema)); + const humanAttentionRaw = await loadValidatedJson( + path.join(telemetryDir, "stages", "10-composition", "human-attention-notes.json"), + "human attention notes", + humanAttentionArtifactSchema + ); + const budgetSummary = await loadValidatedJson(path.join(telemetryDir, "stages", "10-composition", "budget-summary.json"), "budget summary", budgetSummarySchema); + const costProfile = await loadValidatedJson(path.join(telemetryDir, "stages", "00-run", "cost-profile.json"), "cost profile", costProfileSchema); + const modelCallsSummary = await loadValidatedJson(path.join(telemetryDir, "stages", "00-run", "model-calls-summary.json"), "model calls summary", modelCallsSummarySchema); + const toolCallsSummary = await loadValidatedJson(path.join(telemetryDir, "stages", "00-run", "tool-calls-summary.json"), "tool calls summary", toolCallsSummarySchema); + const runJson = await loadValidatedJson(path.join(telemetryDir, "run.json"), "run telemetry", evalRunTelemetrySchema); + const telemetrySummary = await loadValidatedJson(path.join(telemetryDir, "telemetry.json"), "telemetry summary", telemetrySummarySchema); + const reviewedHunkIds = coverage.records.filter((record) => record.status === "reviewed").map((record) => record.hunkId).sort(); + assertReport(reviewedHunkIds.length === coverage.status.reviewedHunks, "coverage_join_mismatch", `reviewed-hunk records disagree with coverage status in ${executionDir}`); + const normalizedFinalFindings = finalFindings.map((finding) => ({ + ...finding, + mergedAnchors: (finding.mergedAnchors ?? []).filter((anchor): anchor is Exclude => typeof anchor !== "string") + })) as FinalFinding[]; + const scoringArtifacts: EvalArtifacts = { + candidates: candidates as CandidateFinding[], + verification: verification as unknown as EvalVerificationRecord[], + finalSelection: finalSelectionArtifact.records as EvalSelectionRecord[], + finalFindings: normalizedFinalFindings, + humanAttentionNotes: scoringHumanAttentionNotes(humanAttentionRaw), + attention: attention as unknown as NonNullable, + packets, + hintEvents: scoringHintEvents(events), + coverage: { ...(coverage.status as unknown as RunCoverageStatus), hunks: coverage.records }, + reviewPlan: plan, + metricsSources: { + costProfile, + modelCallsSummary, + toolCallsSummary, + budgetSummary: budgetSummary as unknown as NonNullable, + runJson, + telemetry: telemetrySummary, + modelCalls, + toolCalls + } + }; + const execution: EvalExecutionInput = { + repeat: index, + score, + telemetryDir, + packets, + events, + modelCalls, + toolCalls, + fileFacts, + diff, + plan, + candidateFindings: candidates as CandidateFinding[], + verification: verification as unknown as EvalVerificationRecord[], + finalSelection: finalSelectionArtifact.records as EvalSelectionRecord[], + finalSelectionArtifact, + finalFindings: normalizedFinalFindings, + scoringArtifacts, + summaryArtifacts: { + attention, + humanAttention: humanAttentionRaw, + budget: budgetSummary, + cost: costProfile, + model: modelCallsSummary, + tool: toolCallsSummary, + run: runJson, + telemetry: telemetrySummary + }, + reviewedHunkIds, + wallTimeSeconds: runJson.durationMs / 1000 + }; + validateEvalExecutionArtifacts(execution, coverage.records, info.reviewRunId); + executions.push(execution); + } + runs.push({ runNumber: info.runNumber, runDir, info, declaredCase, ...(invocationManifest === undefined ? {} : { invocationManifest }), executions }); + } + return runs; +} + +async function loadSelectedInvocationManifest(logsDir: string, selector: string): Promise { + const invocationDir = path.join(logsDir, "invocations"); + let files: string[]; + try { + files = (await readdir(invocationDir)).filter((entry) => entry.endsWith(".json")).sort(); + } catch (error) { + fail("missing_invocation_manifest", `failed to read eval invocation manifests: ${invocationDir}`, { + error: valueFingerprint(error instanceof Error ? error.message : String(error)) + }); + } + const manifests = await Promise.all(files.map((file) => + loadValidatedJson(path.join(invocationDir, file), "eval invocation manifest", evalInvocationManifestSchema) as Promise + )); + assertReport(manifests.length > 0, "missing_invocation_manifest", `no eval invocation manifests found in ${invocationDir}`); + const ordered = [...manifests].sort((left, right) => + left.startedAt.localeCompare(right.startedAt) || manifestLastRun(left) - manifestLastRun(right) + ); + let selected: EvalInvocationManifest | undefined; + if (manifests.some((manifest) => manifest.invocationId === selector)) { + selected = manifests.find((manifest) => manifest.invocationId === selector); + } else if (selector === "latest") { + selected = ordered.at(-1); + } else if (/^\d+$/u.test(selector)) { + const matches = manifests.filter((manifest) => manifestLastRun(manifest) === Number(selector)); + assertReport(matches.length <= 1, "ambiguous_cohort_selector", `run ${selector} matches multiple persisted invocations; use the invocation UUID`); + selected = matches[0]; + } else { + const match = /^(\d+)-(\d+)$/u.exec(selector); + assertReport(match !== null, "invalid_cohort_selector", `cohort must be latest, an invocation UUID, an ending run number, or a min-max range: ${selector}`); + const matches = manifests.filter((manifest) => manifest.runs[0]?.runNumber === Number(match[1]) && manifestLastRun(manifest) === Number(match[2])); + assertReport(matches.length <= 1, "ambiguous_cohort_selector", `cohort ${selector} matches multiple persisted invocations; use the invocation UUID`); + selected = matches[0]; + } + assertReport(selected !== undefined, "unknown_cohort", `cohort ${selector} is not an exact persisted invocation boundary`); + validateCompleteManifest(selected); + return selected; +} + +function manifestLastRun(manifest: EvalInvocationManifest): number { + return manifest.runs.at(-1)?.runNumber ?? -1; +} + +function validateCompleteManifest(manifest: EvalInvocationManifest): void { + const expectedIndexes = Array.from({ length: manifest.cases.length }, (_, index) => index); + assertReport( + manifest.status === "complete" && manifest.completedAt !== undefined && + stableJson(manifest.cases.map((entry) => entry.caseIndex)) === stableJson(expectedIndexes) && + stableJson(manifest.runs.map((entry) => entry.caseIndex)) === stableJson(expectedIndexes) && + manifest.runs.length === manifest.cases.length && + unique(manifest.runs.map((entry) => `${path.resolve(entry.logsRoot)}\0${entry.runPath}`)).length === manifest.runs.length && + manifest.runs.every((entry) => entry.runPath === String(entry.runNumber)), + "partial_latest_cohort", + "selected persisted eval invocation is incomplete or has an invalid exact ordered case boundary" + ); +} + +async function loadDeclaredEvalCase(manifest: EvalInvocationManifest, info: EvalRunInfo): Promise { + const reference = info.invocation; + assertReport(reference !== undefined, "declared_case_join", `eval run ${info.runNumber} has no invocation reference`); + const manifestCase = manifest.cases.find((entry) => entry.caseIndex === reference.caseIndex); + assertReport(manifestCase !== undefined, "declared_case_join", `eval run ${info.runNumber} has no declared manifest case`); + const suiteDir = path.resolve(manifest.suiteDir); + const casePath = path.resolve(suiteDir, manifestCase.caseFile); + const relative = path.relative(suiteDir, casePath); + assertReport(relative.length > 0 && !relative.startsWith("..") && !path.isAbsolute(relative), "declared_case_join", `eval run ${info.runNumber} case YAML escapes its declared suite`); + let declaration: Awaited>; + try { + declaration = await loadEvalCaseDeclaration(casePath, suiteDir); + } catch (error) { + fail("declared_case_join", `failed to load the exact declared eval YAML for run ${info.runNumber}`, { + error: valueFingerprint(error instanceof Error ? error.message : String(error)) + }); + } + assertReport( + declaration.file === manifestCase.caseFile && declaration.evalCase.name === manifestCase.caseName && + declaration.caseHash === manifestCase.caseHash && declaration.caseHash === info.caseHash && + stableJson(declaration.evalCase) === stableJson(info.caseSnapshot), + "declared_case_join", + `eval run ${info.runNumber} manifest, YAML, case hash, and snapshot do not join exactly`, + { + declaredHash: declaration.caseHash, + manifestHash: manifestCase.caseHash, + infoHash: info.caseHash, + declaredSnapshotHash: sha256Hex(stableJson(declaration.evalCase)), + persistedSnapshotHash: sha256Hex(stableJson(info.caseSnapshot)) + } + ); + return declaration.evalCase; +} + +function validateInvocationManifestRun(manifest: EvalInvocationManifest, info: EvalRunInfo, runDir?: string): void { + const reference = info.invocation; + assertReport(reference !== undefined && manifest.invocationId === reference.id, "invocation_manifest_join", `eval run ${info.runNumber} invocation identity disagrees with its manifest`); + const declared = manifest.cases.find((entry) => entry.caseIndex === reference.caseIndex); + const recorded = manifest.runs.find((entry) => entry.caseIndex === reference.caseIndex); + assertReport( + declared !== undefined && recorded !== undefined && + declared.caseName === info.caseName && declared.caseHash === info.caseHash && declared.caseFile === info.caseFile && + recorded.caseName === info.caseName && recorded.caseHash === info.caseHash && recorded.runNumber === info.runNumber, + "invocation_manifest_join", + `eval run ${info.runNumber} does not exactly match its invocation manifest entry` + ); + if (runDir !== undefined && recorded !== undefined) { + assertReport( + path.resolve(runDir) === path.resolve(recorded.logsRoot, recorded.runPath) && recorded.runPath === String(recorded.runNumber), + "invocation_root_ownership", + `eval run ${info.runNumber} was not loaded from its exact manifest-owned root and path` + ); + } +} + +function validateEvalExecutionArtifacts( + execution: EvalExecutionInput, + coverageRecords: Array["records"][number]>, + expectedRunId: string | undefined +): void { + const packetIds = new Set(execution.packets.map((packet) => packet.id)); + const packetHunkIds = new Set(execution.packets.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId))); + const diffHunks = execution.diff.files.flatMap((file) => file.hunks); + const diffHunkById = new Map(diffHunks.map((hunk) => [hunk.id, hunk])); + assertReport(diffHunkById.size === diffHunks.length, "duplicate_diff_hunk", "eval diff contains duplicate hunk IDs"); + assertReport(execution.packets.every((packet) => packet.hunks.every((hunk) => diffHunkById.get(hunk.hunkId)?.path === packet.path)), "packet_diff_join", "eval packet hunks do not join exactly to the recorded diff"); + const planHunkIds = execution.plan.coverage.map((decision) => decision.hunkId); + assertReport(unique(planHunkIds).length === planHunkIds.length, "plan_diff_join", "eval review plan contains duplicate hunk IDs"); + assertReport(execution.plan.coverage.every((decision) => diffHunkById.get(decision.hunkId)?.path === decision.path), "plan_diff_join", "eval review plan decisions do not join to the recorded diff"); + assertReport(unique(execution.candidateFindings.map((finding) => finding.id)).length === execution.candidateFindings.length, "duplicate_candidate_finding", "candidate findings contain duplicate IDs"); + assertReport(unique(execution.finalFindings.map((finding) => finding.id)).length === execution.finalFindings.length, "duplicate_final_finding", "final findings contain duplicate IDs"); + const factPaths = new Set(execution.fileFacts.map((facts) => facts.path)); + assertReport(factPaths.size === execution.fileFacts.length, "duplicate_file_facts", "eval file facts contain duplicate paths"); + assertReport(execution.packets.every((packet) => factPaths.has(packet.path)), "packet_facts_join", "one or more eval packets lack file facts"); + const coverageIds = coverageRecords.map((record) => record.hunkId); + assertReport(unique(coverageIds).length === coverageIds.length, "coverage_join_mismatch", "coverage records contain duplicate hunk IDs"); + assertReport([...packetHunkIds].every((hunkId) => coverageIds.includes(hunkId)), "coverage_join_mismatch", "packet hunks are missing from coverage records"); + assertReport(execution.reviewedHunkIds.every((hunkId) => packetHunkIds.has(hunkId)), "coverage_join_mismatch", "reviewed coverage records reference packet-less hunks"); + const modelCallIds = execution.modelCalls.map((call) => call.callId); + assertReport(unique(modelCallIds).length === modelCallIds.length, "duplicate_model_call", "model-call telemetry contains duplicate IDs"); + const toolCallIds = execution.toolCalls.map((call) => call.toolCallId); + assertReport(unique(toolCallIds).length === toolCallIds.length, "duplicate_tool_call", "tool-call telemetry contains duplicate IDs"); + assertReport(execution.events.every((event, index) => execution.events.findIndex((candidate) => candidate.eventId === event.eventId) === index), "duplicate_event", "event telemetry contains duplicate IDs"); + assertReport(execution.modelCalls.every((call) => call.packetId === undefined || packetIds.has(call.packetId)), "model_packet_join", "model-call telemetry references unknown packets"); + assertReport(execution.toolCalls.every((call) => call.packetId === undefined || packetIds.has(call.packetId)), "tool_packet_join", "tool-call telemetry references unknown packets"); + assertReport(execution.toolCalls.every((call) => call.modelCallId === undefined || modelCallIds.includes(call.modelCallId)), "tool_model_join", "tool-call telemetry references unknown model calls"); + assertReport(execution.events.every((event) => event.message !== "same_file_atoms_packed" || (typeof event.data?.packetId === "string" && packetIds.has(event.data.packetId))), "event_packet_join", "packing telemetry references unknown packets"); + const observedRunIds = sortedUnique([ + ...execution.events.map((event) => event.runId), + ...execution.modelCalls.map((call) => call.runId), + ...execution.toolCalls.map((call) => call.runId) + ]); + assertReport(observedRunIds.length <= 1, "telemetry_run_join", "eval telemetry streams mix review run IDs", { observedRunIds }); + if (expectedRunId !== undefined && observedRunIds.length === 1) { + assertReport(observedRunIds[0] === expectedRunId, "telemetry_run_join", "eval telemetry run ID differs from info.json reviewRunId"); + } +} + +export function reconstructEvidenceArtifacts(execution: EvalExecutionInput): EvalArtifacts { + const rawModel = rawModelEvidence(execution.modelCalls, execution.events); + const rawTool = rawToolEvidence(execution.toolCalls); + const existingBudget = execution.scoringArtifacts.metricsSources.budgetSummary; + const omittedNotes = humanAttentionOutputCounts(execution.summaryArtifacts?.humanAttention).omitted; + const rawContextPressure = rawContextPressureEvidence(execution, omittedNotes); + const rawBudget = existingBudget === undefined ? undefined : ({ + ...existingBudget, + completeness: execution.scoringArtifacts.coverage?.partial === true ? "partial" : "complete", + partialReasons: execution.scoringArtifacts.coverage?.partial === true ? [...execution.scoringArtifacts.coverage.reasons] : [], + overruns: rawBudgetEvents(execution.events, "budget_overrun"), + dispatchBlocks: rawBudgetEvents(execution.events, "budget_dispatch_blocked"), + ...(rawContextPressure === undefined ? {} : { contextPressure: rawContextPressure }) + } as unknown as NonNullable); + return { + ...execution.scoringArtifacts, + candidates: execution.candidateFindings, + verification: execution.verification, + finalSelection: execution.finalSelection, + finalFindings: execution.finalFindings, + packets: execution.packets, + reviewPlan: execution.plan, + metricsSources: { + ...execution.scoringArtifacts.metricsSources, + costProfile: rawCostEvidence(rawModel), + modelCallsSummary: rawModel, + toolCallsSummary: rawTool, + ...(rawBudget === undefined ? {} : { budgetSummary: rawBudget }), + runJson: { + ...(isRecord(execution.scoringArtifacts.metricsSources.runJson) ? execution.scoringArtifacts.metricsSources.runJson : {}), + durationMs: execution.wallTimeSeconds * 1000 + }, + modelCalls: execution.modelCalls, + toolCalls: execution.toolCalls, + } + }; +} + +function recomputeEvidenceScore(run: EvalCaseRunInput, execution: EvalExecutionInput): EvalScore { + return scoreEvalRun(run.declaredCase, reconstructEvidenceArtifacts(execution), "live"); +} + +function validChangedLineAnchor(anchor: CandidateFinding["anchor"], packet: ReviewPacket, diffByHunk: Map): boolean { + if (anchor === undefined || anchor.path !== packet.path || !packet.hunks.some((hunk) => hunk.hunkId === anchor.hunkId)) { + return false; + } + const hunk = diffByHunk.get(anchor.hunkId); + if (hunk === undefined || hunk.path !== anchor.path) { + return false; + } + return anchor.side === "RIGHT" + ? hunk.lines.some((line) => line.kind === "add" && line.newLineNumber === anchor.line) + : hunk.lines.some((line) => line.kind === "delete" && line.oldLineNumber === anchor.line); +} + +type NumericRecord = Record; + +function emptyRawCacheCounts(): NumericRecord { + return { hit: 0, miss: 0, disabled: 0, write: 0 }; +} + +function updateRawCacheCounts(target: NumericRecord, status: LlmCallRecord["cacheStatus"]): void { + if (status === "write") { + target.miss = (target.miss ?? 0) + 1; + target.write = (target.write ?? 0) + 1; + } else { + target[status] = (target[status] ?? 0) + 1; + } +} + +function emptyRawFinalize(): NumericRecord { + return { + compactCalls: 0, fullCalls: 0, noFindingCalls: 0, candidateOrUnknownCalls: 0, promptChars: 0, + noFindingPromptChars: 0, candidateOrUnknownPromptChars: 0, costUSD: 0, noFindingCostUSD: 0, + candidateOrUnknownCostUSD: 0, unknownCostCalls: 0 + }; +} + +function updateRawFinalize(target: NumericRecord, call: LlmCallRecord, providerCalls: number): void { + if (providerCalls === 0 || call.finalizeMode === undefined) { + return; + } + target[call.finalizeMode === "compact" ? "compactCalls" : "fullCalls"]! += 1; + if (call.finalizeTarget === "no_findings") { + target.noFindingCalls! += 1; + target.noFindingPromptChars! += call.promptChars; + target.noFindingCostUSD! += call.costUSD ?? 0; + } else if (call.finalizeTarget === "candidate_or_unknown") { + target.candidateOrUnknownCalls! += 1; + target.candidateOrUnknownPromptChars! += call.promptChars; + target.candidateOrUnknownCostUSD! += call.costUSD ?? 0; + } + target.promptChars! += call.promptChars; + if (call.costUSD === undefined) { + target.unknownCostCalls! += 1; + } else { + target.costUSD! += call.costUSD; + } +} + +function emptyRawModelBucket(stage = false): Record { + return { + ...(stage ? { recordCount: 0, count: 0, statuses: { ok: 0, schema_invalid: 0, transient_error: 0, auth_error: 0, timeout: 0, aborted: 0 } } : { totalRecords: 0, totalCalls: 0 }), + providerCalls: 0, inputTokens: 0, uncachedInputTokens: 0, cacheReadTokens: 0, cacheWriteTokens: 0, + billableInputTokens: 0, outputTokens: 0, reasoningTokens: 0, totalTokens: 0, costUSD: 0, + inputCostUSD: 0, outputCostUSD: 0, cacheReadCostUSD: 0, cacheWriteCostUSD: 0, unknownCostCalls: 0, + cache: emptyRawCacheCounts(), retryAttempts: 0, repairCalls: 0, schemaInvalidCalls: 0, + ...(stage ? {} : { toolChoiceDowngradedCalls: 0 }), finalize: emptyRawFinalize() + }; +} + +function updateRawModelBucket(bucket: Record, call: LlmCallRecord, stage: boolean): void { + const providerCalls = call.cacheStatus === "hit" ? 0 : 1; + bucket[stage ? "recordCount" : "totalRecords"] = Number(bucket[stage ? "recordCount" : "totalRecords"]) + 1; + bucket[stage ? "count" : "totalCalls"] = Number(bucket[stage ? "count" : "totalCalls"]) + providerCalls; + bucket.providerCalls = Number(bucket.providerCalls) + providerCalls; + if (providerCalls > 0) { + for (const key of ["inputTokens", "uncachedInputTokens", "cacheReadTokens", "cacheWriteTokens", "billableInputTokens", "outputTokens", "reasoningTokens", "totalTokens"] as const) { + bucket[key] = Number(bucket[key]) + (call[key] ?? 0); + } + } + updateRawCacheCounts(bucket.cache as NumericRecord, call.cacheStatus); + bucket.retryAttempts = Number(bucket.retryAttempts) + (providerCalls > 0 && call.attempt > 1 ? 1 : 0); + bucket.repairCalls = Number(bucket.repairCalls) + (call.kind === "repair" ? 1 : 0); + bucket.schemaInvalidCalls = Number(bucket.schemaInvalidCalls) + (call.status === "schema_invalid" ? 1 : 0); + if (!stage) { + bucket.toolChoiceDowngradedCalls = Number(bucket.toolChoiceDowngradedCalls) + (providerCalls > 0 && call.toolChoiceDowngraded === true ? 1 : 0); + } else { + const statuses = bucket.statuses as NumericRecord; + statuses[call.status] = (statuses[call.status] ?? 0) + 1; + } + updateRawFinalize(bucket.finalize as NumericRecord, call, providerCalls); + if (providerCalls === 0) { + return; + } + if (call.costUSD === undefined) { + bucket.unknownCostCalls = Number(bucket.unknownCostCalls) + 1; + } else { + for (const key of ["costUSD", "inputCostUSD", "outputCostUSD", "cacheReadCostUSD", "cacheWriteCostUSD"] as const) { + bucket[key] = Number(bucket[key]) + (call[key] ?? 0); + } + } +} + +function cacheAliases(bucket: Record): Record { + return { + ...bucket, + localModelCallCache: { ...(bucket.cache as NumericRecord) }, + providerPromptCache: { + readTokens: bucket.cacheReadTokens, writeTokens: bucket.cacheWriteTokens, + readCostUSD: bucket.cacheReadCostUSD, writeCostUSD: bucket.cacheWriteCostUSD + } + }; +} + +function emptyRawSchemaRecovery(): NumericRecord { + return { + schemaInvalidCalls: 0, schemaInvalidRecovered: 0, schemaInvalidUnrecovered: 0, + schemaRepairAttempts: 0, schemaRepairRecovered: 0, deterministicSchemaRecovered: 0, schemaRecoveryFailed: 0 + }; +} + +function rawSchemaRecoveryEvidence(calls: LlmCallRecord[], events: TelemetryEvent[]): Record { + const total = emptyRawSchemaRecovery(); + const byStage: Record = {}; + const add = (stage: number, delta: Partial): void => { + const targets = stage === 0 ? [total] : [total, byStage[String(stage)] ?? (byStage[String(stage)] = emptyRawSchemaRecovery())]; + for (const target of targets) { + for (const [key, value] of Object.entries(delta)) target[key] = (target[key] ?? 0) + (value ?? 0); + } + }; + for (const call of calls) { + if (call.status === "schema_invalid") add(call.stage, { schemaInvalidCalls: 1 }); + if (call.kind !== "repair") continue; + if (call.status === "ok") { + add(call.stage, { schemaRepairAttempts: 1, schemaRepairRecovered: 1, schemaInvalidRecovered: 1 }); + } else { + add(call.stage, { schemaRepairAttempts: 1 }); + if (call.status === "schema_invalid" && call.stage !== 7 && call.stage !== 9) add(call.stage, { schemaRecoveryFailed: 1 }); + } + } + for (const event of events) { + if (event.message === "schema_invalid_submit_recovered") { + const recovered = event.data?.schemaRepairUsed === true ? 2 : 1; + add(event.stage, { deterministicSchemaRecovered: recovered, schemaInvalidRecovered: recovered }); + } else if (event.message === "stage7_schema_cleanup_recovered") { + add(event.stage, { deterministicSchemaRecovered: 1, schemaInvalidRecovered: 1 }); + } else if ([ + "schema_invalid_submit_recovery_invalid", "stage7_schema_cleanup_rejected", "stage7_schema_repair_failed", + "verification_schema_repair_failed" + ].includes(event.message)) { + add(event.stage, { schemaRecoveryFailed: 1 }); + } + } + const finalize = (input: NumericRecord): NumericRecord => ({ + ...input, + schemaInvalidRecovered: Math.min(input.schemaInvalidRecovered ?? 0, input.schemaInvalidCalls ?? 0), + schemaInvalidUnrecovered: Math.max(0, (input.schemaInvalidCalls ?? 0) - Math.min(input.schemaInvalidRecovered ?? 0, input.schemaInvalidCalls ?? 0)) + }); + return { ...finalize(total), byStage: Object.fromEntries(Object.entries(byStage).map(([stage, counters]) => [stage, finalize(counters)])) }; +} + +function rawModelEvidence(calls: LlmCallRecord[], events: TelemetryEvent[] = []): Record { + const total = emptyRawModelBucket(); + const byStage: Record> = {}; + for (const call of calls) { + updateRawModelBucket(total, call, false); + const key = String(call.stage); + const bucket = byStage[key] ?? (byStage[key] = emptyRawModelBucket(true)); + updateRawModelBucket(bucket, call, true); + } + const schemaRecovery = rawSchemaRecoveryEvidence(calls, events); + const recoveryByStage = schemaRecovery.byStage as Record; + return { + ...cacheAliases(total), + schemaRecovery, + byStage: Object.fromEntries(Object.entries(byStage).map(([stage, bucket]) => [stage, { + ...cacheAliases(bucket), + schemaRecovery: recoveryByStage[stage] ?? emptyRawSchemaRecovery() + }])) + }; +} + +function modelEvidenceView(summary: Record): Record { + const topKeys = [...Object.keys(emptyRawModelBucket()), "localModelCallCache", "providerPromptCache", "schemaRecovery"]; + const stageKeys = [...Object.keys(emptyRawModelBucket(true)), "localModelCallCache", "providerPromptCache", "schemaRecovery"]; + const pick = (input: Record, keys: string[]) => Object.fromEntries(keys.map((key) => [key, input[key]])); + const stages = isRecord(summary.byStage) ? summary.byStage : {}; + return { + ...pick(summary, topKeys), + byStage: Object.fromEntries(Object.entries(stages).map(([stage, value]) => [stage, isRecord(value) ? pick(value, stageKeys) : value])) + }; +} + +function rawCostEvidence(model: Record): Record { + const breakdown = (source: Record) => ({ + uncachedInput: { tokens: source.uncachedInputTokens, costUSD: source.inputCostUSD }, + providerPromptCacheRead: { tokens: source.cacheReadTokens, costUSD: source.cacheReadCostUSD }, + providerPromptCacheWrite: { tokens: source.cacheWriteTokens, costUSD: source.cacheWriteCostUSD }, + output: { tokens: source.outputTokens, costUSD: source.outputCostUSD }, + total: { tokens: source.totalTokens, costUSD: source.costUSD } + }); + const byStage = model.byStage as Record>; + return { + totalCostUSD: model.costUSD, unknownCostCalls: model.unknownCostCalls, + localModelCallCache: model.localModelCallCache, providerPromptCache: model.providerPromptCache, + costBreakdown: breakdown(model), + tokens: Object.fromEntries(["inputTokens", "uncachedInputTokens", "cacheReadTokens", "cacheWriteTokens", "billableInputTokens", "outputTokens", "reasoningTokens", "totalTokens"].map((key) => [key, model[key]])), + cost: { + inputCostUSD: model.inputCostUSD, outputCostUSD: model.outputCostUSD, cacheReadCostUSD: model.cacheReadCostUSD, + cacheWriteCostUSD: model.cacheWriteCostUSD, totalCostUSD: model.costUSD + }, + byStage: Object.fromEntries(Object.entries(byStage).map(([stage, value]) => { + const { schemaRecovery: _schemaRecovery, ...costValue } = value; + return [stage, { ...costValue, costBreakdown: breakdown(costValue) }]; + })) + }; +} + +function emptyRawToolCache(): NumericRecord { + return { hits: 0, misses: 0, writes: 0, disabled: 0, inflightHits: 0, evictions: 0, backendExecutions: 0, savedBackendCalls: 0 }; +} + +function emptyRawToolBucket(): Record { + return { count: 0, errors: 0, rejections: 0, degraded: 0, backendExecutions: 0, savedBackendCalls: 0, totalDurationMs: 0, totalResultChars: 0, resultCache: emptyRawToolCache() }; +} + +function updateRawToolCache(cache: NumericRecord, call: ToolCallRecord): void { + if (call.backendExecuted === true) cache.backendExecutions! += 1; + if (call.cacheStatus === "hit") { + cache.hits! += 1; cache.savedBackendCalls! += 1; + if (call.cacheHitKind === "inflight") cache.inflightHits! += 1; + } else if (call.cacheStatus === "write") { + cache.misses! += 1; cache.writes! += 1; + } else if (call.cacheStatus === "miss") { + cache.misses! += 1; + } else { + cache.disabled! += 1; + } + cache.evictions! += call.cacheEvictedEntries ?? 0; +} + +function updateRawToolBucket(bucket: Record, call: ToolCallRecord): void { + bucket.count = Number(bucket.count) + 1; + bucket.errors = Number(bucket.errors) + (call.status === "error" ? 1 : 0); + bucket.rejections = Number(bucket.rejections) + (call.status === "rejected" ? 1 : 0); + bucket.degraded = Number(bucket.degraded) + (call.degraded ? 1 : 0); + bucket.backendExecutions = Number(bucket.backendExecutions) + (call.backendExecuted === true ? 1 : 0); + bucket.savedBackendCalls = Number(bucket.savedBackendCalls) + (call.cacheStatus === "hit" ? 1 : 0); + bucket.totalDurationMs = Number(bucket.totalDurationMs) + call.durationMs; + bucket.totalResultChars = Number(bucket.totalResultChars) + call.resultChars; + updateRawToolCache(bucket.resultCache as NumericRecord, call); +} + +function rawToolEvidence(calls: ToolCallRecord[]): Record { + const resultCache = emptyRawToolCache(); + const byTool: Record> = {}; + const byStage: Record> = {}; + for (const call of calls) { + updateRawToolCache(resultCache, call); + updateRawToolBucket(byTool[call.tool] ?? (byTool[call.tool] = emptyRawToolBucket()), call); + updateRawToolBucket(byStage[String(call.stage)] ?? (byStage[String(call.stage)] = emptyRawToolBucket()), call); + } + const average = (buckets: Record>) => Object.fromEntries(Object.entries(buckets).map(([key, bucket]) => [key, { + ...bucket, + averageDurationMs: Number(bucket.count) === 0 ? 0 : Number(bucket.totalDurationMs) / Number(bucket.count), + averageResultChars: Number(bucket.count) === 0 ? 0 : Number(bucket.totalResultChars) / Number(bucket.count) + }])); + return { totalCalls: calls.length, resultCache, byTool: average(byTool), byStage: average(byStage) }; +} + +function eventStageEvidence(events: TelemetryEvent[], recovery: Record): Record { + const recoveryByStage = recovery.byStage as Record; + const byStage: Record = {}; + for (let stage = 0; stage <= 11; stage += 1) { + byStage[String(stage)] = { + events: 0, levels: { debug: 0, info: 0, warn: 0, error: 0 }, cache: emptyRawCacheCounts(), runtimeMs: 0, + schemaRecovery: recoveryByStage[String(stage)] ?? emptyRawSchemaRecovery() + }; + } + for (const event of events) { + const bucket = byStage[String(event.stage)]!; + bucket.events += 1; + bucket.levels[event.level] = (bucket.levels[event.level] ?? 0) + 1; + if (event.cacheStatus !== undefined) bucket.cache[event.cacheStatus] = (bucket.cache[event.cacheStatus] ?? 0) + 1; + if (event.message === "stage_started" && bucket.startedAt === undefined) bucket.startedAt = event.timestamp; + if ((event.message === "stage_completed" || event.message === "stage_failed") && bucket.completedAt === undefined) { + bucket.completedAt = event.timestamp; + if (bucket.startedAt !== undefined) bucket.runtimeMs += Math.max(0, Date.parse(event.timestamp) - Date.parse(bucket.startedAt)); + } + } + return byStage; +} + +function telemetryStageEvidenceView(input: Record): Record { + const stages = isRecord(input.stages) ? input.stages : {}; + return Object.fromEntries(Object.entries(stages).map(([stage, value]) => { + if (!isRecord(value)) return [stage, value]; + return [stage, Object.fromEntries(["events", "levels", "cache", "startedAt", "completedAt", "runtimeMs", "schemaRecovery"] + .filter((key) => value[key] !== undefined).map((key) => [key, value[key]]))]; + })); +} + +function humanAttentionOutputCounts(input: unknown): { emitted: number; omitted: number } { + return isRecord(input) && Array.isArray(input.outputNotes) + ? { emitted: input.outputNotes.length, omitted: typeof input.omittedCount === "number" ? input.omittedCount : 0 } + : { emitted: Array.isArray(input) ? input.length : 0, omitted: 0 }; +} + +function rawBudgetEvents(events: TelemetryEvent[], message: "budget_overrun" | "budget_dispatch_blocked"): Array> { + return events.filter((event) => event.message === message).map((event) => ({ ...(event.data ?? {}) })); +} + +function incrementStageCount(target: NumericRecord, stage: number): void { + if (stage !== 0) { + target[String(stage)] = (target[String(stage)] ?? 0) + 1; + } +} + +function rawContextPressureEvidence(execution: EvalExecutionInput, omittedNotes: number): Record | undefined { + const localRejections = execution.toolCalls.filter((call) => call.status === "rejected" && + isLocalToolBudgetRejectionReason(call.degradationReason ?? call.errorCode ?? "rejected")); + const degradedCalls = execution.toolCalls.filter((call) => call.status !== "rejected" && call.degraded); + const rejectionByStage: NumericRecord = {}; + const degradedByStage: NumericRecord = {}; + const rejectionReasons: NumericRecord = {}; + for (const call of localRejections) { + incrementStageCount(rejectionByStage, call.stage); + const reason = call.degradationReason ?? call.errorCode ?? "rejected"; + rejectionReasons[reason] = (rejectionReasons[reason] ?? 0) + 1; + } + for (const call of degradedCalls) { + incrementStageCount(degradedByStage, call.stage); + } + const extensions = { + granted: 0, + denied: 0, + resultChars: 0, + grantedByStage: {} as NumericRecord, + deniedByStage: {} as NumericRecord + }; + for (const event of execution.events) { + if (event.message === "tool_budget_extension_granted") { + extensions.granted += 1; + incrementStageCount(extensions.grantedByStage, event.stage); + extensions.resultChars += typeof event.data?.resultChars === "number" ? event.data.resultChars : 0; + } else if (event.message === "tool_budget_extension_denied") { + extensions.denied += 1; + incrementStageCount(extensions.deniedByStage, event.stage); + } + } + const hasExtensionPressure = extensions.granted > 0 || extensions.denied > 0 || extensions.resultChars > 0; + if (localRejections.length === 0 && !hasExtensionPressure && degradedCalls.length === 0 && + execution.packets.every((packet) => packet.degraded === undefined) && omittedNotes === 0) { + return undefined; + } + return { + toolBudgetRejections: localRejections.length, + toolBudgetRejectionsByStage: rejectionByStage, + ...(hasExtensionPressure ? { toolBudgetExtensions: extensions } : {}), + degradedToolResults: degradedCalls.length, + degradedToolResultsByStage: degradedByStage, + degradedHunks: sum(execution.packets.filter((packet) => packet.degraded !== undefined).map((packet) => packet.hunks.length)), + rejectionReasons: Object.entries(rejectionReasons) + .map(([reason, count]) => ({ reason, count })) + .sort((left, right) => right.count - left.count || left.reason.localeCompare(right.reason)), + unresolvedNotes: { emitted: humanAttentionOutputCounts(execution.summaryArtifacts?.humanAttention).emitted, omitted: omittedNotes } + }; +} + +function attentionRelationsMatch( + records: Array>, + execution: EvalExecutionInput +): boolean { + const candidatePacketById = new Map(execution.candidateFindings.map((candidate) => [candidate.id, candidate.producedBy.packetId])); + const countByPacket = (predicate: (candidate: CandidateFinding) => boolean): NumericRecord => { + const counts: NumericRecord = {}; + for (const candidate of execution.candidateFindings.filter(predicate)) { + counts[candidate.producedBy.packetId] = (counts[candidate.producedBy.packetId] ?? 0) + 1; + } + return counts; + }; + const promoted = countByPacket((candidate) => candidate.provenance?.source === "uncertainty_promotion"); + const direct = countByPacket((candidate) => candidate.provenance?.source !== "uncertainty_promotion"); + const kept: NumericRecord = {}; + for (const record of execution.verification) { + if ("verdict" in record && (record.verdict.verdict === "keep" || record.verdict.verdict === "revise")) { + const packetId = candidatePacketById.get(record.candidateId); + if (packetId !== undefined) kept[packetId] = (kept[packetId] ?? 0) + 1; + } + } + const published: NumericRecord = {}; + for (const finding of execution.finalFindings.filter((candidate) => candidate.publication !== "suppressed")) { + const packetIds = new Set([finding.id, ...finding.mergedCandidateIds].flatMap((id) => candidatePacketById.get(id) ?? [])); + for (const packetId of packetIds) published[packetId] = (published[packetId] ?? 0) + 1; + } + const plannedHunkIds = new Set(execution.plan.coverage.map((decision) => decision.hunkId)); + return records.length === execution.packets.length && records.every((record, index) => { + const packet = execution.packets[index]; + if (packet === undefined) return false; + const coverageSource = packet.coverageEscalation !== undefined + ? `escalated:${packet.coverageEscalation.rule}` + : packet.hunks.some((hunk) => hunk.plannerFallbackReason !== undefined) || + packet.hunks.every((hunk) => !plannedHunkIds.has(hunk.hunkId)) + ? "deterministic_default" + : "planner"; + return record.packetId === packet.id && record.path === packet.path && record.coverage === packet.coverage && + record.coverageSource === coverageSource && record.ensemblePasses >= 1 && + record.directCandidates === (direct[packet.id] ?? 0) && record.promotedCandidates === (promoted[packet.id] ?? 0) && + record.keptVerified === (kept[packet.id] ?? 0) && record.published === (published[packet.id] ?? 0); + }); +} + +function humanAttentionRelationsMatch(input: z.infer): boolean { + if (Array.isArray(input)) return true; + const noteIds = input.notes.map((note) => note.id); + const groupKeys = input.groups.map((group) => group.key); + const noteIdSet = new Set(noteIds); + const groupByKey = new Map(input.groups.map((group) => [group.key, group])); + const uniqueReferences = (values: string[]): boolean => unique(values).length === values.length && values.every((value) => groupByKey.has(value)); + const findingSuppressed = input.suppressedByFindings.map((record) => record.groupKey); + const verificationSuppressed = input.suppressedByVerification.map((record) => record.groupKey); + const partition = [...input.keptForOutputGroupIds, ...findingSuppressed, ...verificationSuppressed].sort(); + return unique(noteIds).length === noteIds.length && unique(groupKeys).length === groupKeys.length && + input.groups.every((group) => group.count === group.noteIds.length && unique(group.noteIds).length === group.noteIds.length && + group.noteIds.every((id) => noteIdSet.has(id))) && + uniqueReferences(input.composerPromptGroupIds) && uniqueReferences(input.outputGroupIds) && uniqueReferences(input.keptForOutputGroupIds) && + uniqueReferences(findingSuppressed) && uniqueReferences(verificationSuppressed) && + stableJson([...groupKeys].sort()) === stableJson(partition) && + stableJson(input.outputGroupIds) === stableJson(input.keptForOutputGroupIds.slice(0, input.outputGroupIds.length)) && + input.outputNotes.length === input.outputGroupIds.length && + input.omittedCount === input.keptForOutputGroupIds.length - input.outputGroupIds.length && + input.suppressedByFindings.every((record) => stableJson(record.noteIds) === stableJson(groupByKey.get(record.groupKey)?.noteIds)) && + input.suppressedByVerification.every((record) => stableJson(record.noteIds) === stableJson(groupByKey.get(record.groupKey)?.noteIds)); +} + +function validateRawSummaryEvidence(run: EvalCaseRunInput, execution: EvalExecutionInput, label: string, failures: ReportFailure[]): void { + const artifacts = execution.summaryArtifacts; + if (artifacts === undefined) { + return; + } + const parsedModel = modelCallsSummarySchema.safeParse(artifacts.model); + const parsedCost = costProfileSchema.safeParse(artifacts.cost); + const parsedTool = toolCallsSummarySchema.safeParse(artifacts.tool); + const parsedBudget = budgetSummarySchema.safeParse(artifacts.budget); + const parsedRun = evalRunTelemetrySchema.safeParse(artifacts.run); + const parsedTelemetry = telemetrySummarySchema.safeParse(artifacts.telemetry); + const parsedAttention = z.array(attentionRecordSchema).safeParse(artifacts.attention); + const parsedHuman = humanAttentionArtifactSchema.safeParse(artifacts.humanAttention); + if (!parsedModel.success || !parsedCost.success || !parsedTool.success || !parsedBudget.success || !parsedRun.success || + !parsedTelemetry.success || !parsedAttention.success || !parsedHuman.success) { + failures.push(failure("paid_summary_schema", `${label}/repeat ${execution.repeat} summary evidence is not strictly shaped`)); + return; + } + const rawModel = rawModelEvidence(execution.modelCalls, execution.events); + const rawTool = rawToolEvidence(execution.toolCalls); + const rawDerived = reconstructRunTelemetryDerivedEvidence(execution.events, execution.modelCalls); + const rawCost = rawCostEvidence(rawModel); + const modelMatches = stableJson(modelEvidenceView(parsedModel.data as unknown as Record)) === stableJson(rawModel); + const telemetryModelMatches = stableJson(modelEvidenceView(parsedTelemetry.data.modelCalls as unknown as Record)) === stableJson(rawModel); + const costMatches = stableJson(parsedCost.data) === stableJson(rawCost); + const toolMatches = stableJson(parsedTool.data) === stableJson(rawTool) && stableJson(parsedTelemetry.data.toolCalls) === stableJson(rawTool); + const runTotals = parsedRun.data.totals; + const telemetryTotals = parsedTelemetry.data.totals; + const totalView = (totals: typeof runTotals) => ({ + events: totals.events, modelCallRecords: totals.modelCallRecords, modelCalls: totals.modelCalls, providerCalls: totals.providerCalls, + toolCalls: totals.toolCalls, toolResultCache: totals.toolResultCache, inputTokens: totals.inputTokens, + uncachedInputTokens: totals.uncachedInputTokens, cacheReadTokens: totals.cacheReadTokens, cacheWriteTokens: totals.cacheWriteTokens, + billableInputTokens: totals.billableInputTokens, outputTokens: totals.outputTokens, reasoningTokens: totals.reasoningTokens, + totalTokens: totals.totalTokens, totalCostUSD: totals.totalCostUSD, inputCostUSD: totals.inputCostUSD, + outputCostUSD: totals.outputCostUSD, cacheReadCostUSD: totals.cacheReadCostUSD, cacheWriteCostUSD: totals.cacheWriteCostUSD, + costBreakdown: totals.costBreakdown, + unknownCostCalls: totals.unknownCostCalls, cache: totals.cache, localModelCallCache: totals.localModelCallCache, + providerPromptCache: totals.providerPromptCache, retryAttempts: totals.retryAttempts, repairCalls: totals.repairCalls, + schemaInvalidCalls: totals.schemaInvalidCalls, schemaRecovery: totals.schemaRecovery, + stage7SchemaRepair: totals.stage7SchemaRepair, logOverflow: totals.logOverflow, + filesChanged: totals.filesChanged, hunks: totals.hunks, packets: totals.packets, packetReviews: totals.packetReviews, + candidates: totals.candidates, verified: totals.verified, finalFindings: totals.finalFindings, postedComments: totals.postedComments + }); + const rawTotals = { + events: execution.events.length, modelCallRecords: rawModel.totalRecords, modelCalls: rawModel.providerCalls, providerCalls: rawModel.providerCalls, + toolCalls: execution.toolCalls.length, toolResultCache: rawTool.resultCache, inputTokens: rawModel.inputTokens, + uncachedInputTokens: rawModel.uncachedInputTokens, cacheReadTokens: rawModel.cacheReadTokens, cacheWriteTokens: rawModel.cacheWriteTokens, + billableInputTokens: rawModel.billableInputTokens, outputTokens: rawModel.outputTokens, reasoningTokens: rawModel.reasoningTokens, + totalTokens: rawModel.totalTokens, totalCostUSD: rawModel.costUSD, inputCostUSD: rawModel.inputCostUSD, + outputCostUSD: rawModel.outputCostUSD, cacheReadCostUSD: rawModel.cacheReadCostUSD, cacheWriteCostUSD: rawModel.cacheWriteCostUSD, + costBreakdown: rawCost.costBreakdown, + unknownCostCalls: rawModel.unknownCostCalls, cache: rawModel.cache, localModelCallCache: rawModel.localModelCallCache, + providerPromptCache: rawModel.providerPromptCache, retryAttempts: rawModel.retryAttempts, repairCalls: rawModel.repairCalls, + schemaInvalidCalls: rawModel.schemaInvalidCalls, schemaRecovery: rawModel.schemaRecovery, + stage7SchemaRepair: rawDerived.stage7SchemaRepair, logOverflow: rawDerived.logOverflow, + ...rawDerived.pipelineTotals + }; + const totalsMatch = stableJson(totalView(runTotals)) === stableJson(rawTotals) && stableJson(totalView(telemetryTotals)) === stableJson(rawTotals); + const stagesMatch = stableJson(telemetryStageEvidenceView(parsedTelemetry.data as unknown as Record)) === + stableJson(eventStageEvidence(execution.events, rawModel.schemaRecovery as Record)); + const recoveryMatches = stableJson(parsedTelemetry.data.schemaRecovery) === stableJson(rawModel.schemaRecovery); + const pipelineMatches = stableJson({ + workers: parsedTelemetry.data.workers, + packets: parsedTelemetry.data.packets, + lenses: parsedTelemetry.data.lenses, + coverage: parsedTelemetry.data.coverage, + candidates: parsedTelemetry.data.candidates, + verdicts: parsedTelemetry.data.verdicts, + dedup: parsedTelemetry.data.dedup, + finalSelection: parsedTelemetry.data.finalSelection, + posting: parsedTelemetry.data.posting + }) === stableJson(rawDerived.pipeline); + const repairAndOverflowMatch = stableJson(parsedTelemetry.data.schemaRepair.stage7) === stableJson(rawDerived.stage7SchemaRepair) && + stableJson(parsedTelemetry.data.logs.bufferedOverflow) === stableJson(rawDerived.logOverflow) && + parsedTelemetry.data.events === execution.events.length; + const budget = parsedBudget.data; + const rawByStage = rawModel.byStage as Record>; + const budgetUsage = { + modelCalls: rawModel.providerCalls, + totalTokens: rawModel.totalTokens, + costUSD: Number(rawModel.costUSD) > 0 ? rawModel.costUSD : undefined, + byStage: Object.entries(rawByStage).map(([stage, entry]) => ({ stage: Number(stage), modelCalls: entry.providerCalls, totalTokens: entry.totalTokens })) + .sort((left, right) => left.stage - right.stage) + }; + const usageMatches = stableJson(budget.usage) === stableJson(budgetUsage) && + budget.effective.timeoutMs === run.info.effectiveConfig?.review.timeoutMs && + budget.effective.maxBudgetTokens === run.info.effectiveConfig?.review.maxBudgetTokens; + const noteCounts = humanAttentionOutputCounts(parsedHuman.data); + const contextMatches = stableJson(budget.contextPressure) === stableJson(rawContextPressureEvidence(execution, noteCounts.omitted)); + const budgetEventsMatch = stableJson(budget.overruns) === stableJson(rawBudgetEvents(execution.events, "budget_overrun")) && + stableJson(budget.dispatchBlocks) === stableJson(rawBudgetEvents(execution.events, "budget_dispatch_blocked")); + const rawRunIds = sortedUnique([ + ...execution.events.map((event) => event.runId), + ...execution.modelCalls.map((call) => call.runId), + ...execution.toolCalls.map((call) => call.runId) + ]); + const identityMatches = parsedRun.data.runId === parsedTelemetry.data.runId && + (run.info.reviewRunId === undefined || parsedRun.data.runId === run.info.reviewRunId) && + (rawRunIds.length === 0 || (rawRunIds.length === 1 && rawRunIds[0] === parsedRun.data.runId)) && + parsedRun.data.startedAt === parsedTelemetry.data.startedAt && parsedRun.data.finishedAt === parsedTelemetry.data.finishedAt && + parsedRun.data.completedAt === parsedRun.data.finishedAt && parsedTelemetry.data.completedAt === parsedTelemetry.data.finishedAt && + parsedRun.data.durationMs === Math.max(0, Date.parse(parsedRun.data.finishedAt) - Date.parse(parsedRun.data.startedAt)) && + parsedRun.data.durationMs === parsedTelemetry.data.durationMs && parsedRun.data.durationMs === execution.wallTimeSeconds * 1000; + const attentionMatches = attentionRelationsMatch(parsedAttention.data, execution) && humanAttentionRelationsMatch(parsedHuman.data); + if (!modelMatches || !telemetryModelMatches || !costMatches || !toolMatches || !totalsMatch || !stagesMatch || !recoveryMatches || + !pipelineMatches || !repairAndOverflowMatch || !usageMatches || !contextMatches || !budgetEventsMatch || !identityMatches || !attentionMatches) { + failures.push(failure("paid_summary_reconciliation", `${label}/repeat ${execution.repeat} summaries do not independently reconcile to raw records`, { + modelMatches, telemetryModelMatches, costMatches, toolMatches, totalsMatch, stagesMatch, recoveryMatches, pipelineMatches, + repairAndOverflowMatch, usageMatches, contextMatches, budgetEventsMatch, identityMatches, attentionMatches + })); + } +} + +function validateExecutionEvidence(run: EvalCaseRunInput, execution: EvalExecutionInput, label: string, failures: ReportFailure[]): EvalScore { + validateRawSummaryEvidence(run, execution, label, failures); + const packetById = new Map(execution.packets.map((packet) => [packet.id, packet])); + const diffHunks = execution.diff.files.flatMap((file) => file.hunks); + const diffByHunk = new Map(diffHunks.map((hunk) => [hunk.id, hunk])); + const candidateById = new Map(execution.candidateFindings.map((finding) => [finding.id, finding])); + const verificationById = new Map(execution.verification.map((record) => [record.candidateId, record])); + const selectionById = new Map(execution.finalSelection.map((record) => [record.findingId, record])); + const finalByFingerprint = new Map(execution.finalFindings.map((finding) => [finding.fingerprint, finding])); + const selectionArtifact = execution.finalSelectionArtifact; + const effectiveReview = run.info.effectiveConfig?.review; + const policyConfig: CodegenieConfig = { + ...structuredClone(defaultConfig), + review: { + ...structuredClone(defaultConfig.review), + ...(effectiveReview ?? {}) + } + }; + let relationsValid = + candidateById.size === execution.candidateFindings.length && verificationById.size === execution.verification.length && + selectionById.size === execution.finalSelection.length && finalByFingerprint.size === execution.finalFindings.length && + verificationById.size === candidateById.size && effectiveReview !== undefined && + stableJson(selectionArtifact.records) === stableJson(execution.finalSelection) && + ((selectionArtifact.composition.mode === "deterministic_fallback" || selectionArtifact.composition.mode === "schema_repair_fallback") + ? (selectionArtifact.composition.fallbackReason?.trim().length ?? 0) > 0 + : selectionArtifact.composition.fallbackReason === undefined); + + for (const candidate of execution.candidateFindings) { + const packet = packetById.get(candidate.producedBy.packetId); + const record = verificationById.get(candidate.id); + const producerValid = packet !== undefined && candidate.producedBy.kind === "packet" && + (candidate.producedBy.stage === 7 || candidate.producedBy.stage === 8) && candidate.producedBy.lensId.trim().length > 0 && + packet.lenses.includes(candidate.producedBy.lensId) && candidate.producedBy.skillIds.every((id) => id.trim().length > 0); + const provenanceValid = candidate.provenance === undefined || ( + packetById.has(candidate.provenance.sourcePacketId) && candidate.provenance.question.trim().length > 0 && candidate.provenance.reason.trim().length > 0 && + record?.candidateProvenance !== undefined && stableJson(record.candidateProvenance) === stableJson(candidate.provenance) + ); + const recordValid = record !== undefined && (!("verdict" in record) || record.verdict.candidateId === candidate.id); + relationsValid &&= producerValid && packet !== undefined && candidate.path === packet.path && provenanceValid && recordValid; + } + + const keptIds = new Set(execution.verification.flatMap((record) => + "verdict" in record && record.verdict.verdict !== "reject" && record.verdict.verdict !== "incomplete" && record.verdict.verificationIncomplete !== true + ? [record.candidateId] + : [] + )); + const reconstructedVerified = reconstructVerifiedFindingsFromArtifacts( + execution.candidateFindings, + execution.verification, + execution.packets, + execution.diff + ); + const verifiedById = new Map(reconstructedVerified.map((finding) => [finding.id, finding])); + const verifierPolicy = reconstructGatedVerifierCandidatesFromArtifacts( + execution.candidateFindings, + execution.packets, + execution.diff, + policyConfig + ); + const gateEvaluationById = new Map(verifierPolicy.evaluations.map((evaluation) => [evaluation.candidate.id, evaluation])); + const reconstructedVerifierCandidates = new Map(verifierPolicy.candidates.map((finding) => [finding.id, finding])); + relationsValid &&= verifiedById.size === reconstructedVerified.length && stableJson([...verifiedById.keys()].sort()) === stableJson([...keptIds].sort()); + relationsValid &&= selectionById.size === keptIds.size && [...keptIds].every((id) => selectionById.has(id)); + for (const record of execution.verification) { + const ownEvaluation = gateEvaluationById.get(record.candidateId); + const hasVerdict = "verdict" in record; + // Completed and budget-limited duplicate records are emitted with the + // representative's record metadata. Lane-limited duplicate records have + // no verdict and are emitted with the duplicate candidate's own metadata. + const gateFactsCandidateId = effectiveReview?.verify !== false && hasVerdict && record.duplicateOf !== undefined + ? record.duplicateOf + : record.candidateId; + const gateEvaluation = gateEvaluationById.get(gateFactsCandidateId); + const gateCandidate = gateEvaluation?.candidate; + if (gateCandidate === undefined || ownEvaluation === undefined) { + relationsValid = false; + } + const gatePacket = gateCandidate === undefined ? undefined : packetById.get(gateCandidate.producedBy.packetId); + relationsValid &&= gateCandidate !== undefined && stableJson(record.gateFacts) === stableJson( + gateEvaluation?.decision.facts + ); + const ownDecision = ownEvaluation?.decision; + const ownGate = ownEvaluation?.anchorStripped === true ? "gate_anchor_stripped" : "passed"; + const expectedScheduledDecision = gateEvaluation?.decision.outcome === "schedule" + ? gateEvaluation.decision.lane === "evidence_resolution" ? "scheduled_for_evidence_resolution" : "scheduled" + : undefined; + const laneLimited = !hasVerdict && ownDecision?.outcome === "schedule" && + ownDecision.lane === "evidence_resolution" && record.gateReason === "low_confidence_evidence_resolution_lane_limit"; + const gateStateValid = ownDecision?.outcome === "suppress" + ? !hasVerdict && record.gate === (ownEvaluation?.anchorStripped === true ? "gate_anchor_stripped" : "suppressed") && + record.gateDecision === "suppressed" && record.gateReason === (ownEvaluation?.anchorStripped === true + ? `invalid_anchor; ${ownDecision.reason}` + : ownDecision.reason) && record.verificationLane === undefined && record.gateFacts !== undefined + : laneLimited + ? record.gate === "suppressed" && record.gateDecision === "scheduled_for_evidence_resolution" && + record.verificationLane === "evidence_resolution" && record.gateFacts !== undefined + : hasVerdict && record.gate === ownGate && record.gateDecision === expectedScheduledDecision && + record.gateReason === (gateEvaluation?.decision.outcome === "schedule" ? gateEvaluation.decision.reason : undefined) && + record.verificationLane === (gateEvaluation?.decision.outcome === "schedule" ? gateEvaluation.decision.lane : undefined) && + record.gateFacts !== undefined; + relationsValid &&= gateStateValid; + const clusteredCandidate = reconstructedVerifierCandidates.get(record.candidateId); + if (ownDecision?.outcome === "suppress") { + relationsValid &&= clusteredCandidate === undefined && record.duplicateOf === undefined && record.clusterId === undefined; + } else if (effectiveReview?.verify === false) { + relationsValid &&= clusteredCandidate !== undefined && record.duplicateOf === undefined && record.clusterId === undefined && hasVerdict && + stableJson(record.verdict) === stableJson({ + candidateId: record.candidateId, + verdict: "keep", + reason: "verification disabled by config", + requiredEvidencePresent: true, + falsePositiveRisk: "low" + }); + } else { + relationsValid &&= clusteredCandidate !== undefined && + record.duplicateOf === clusteredCandidate?.duplicateOf && + record.clusterId === clusteredCandidate?.clusterId; + } + if ("verdict" in record) { + if (record.duplicateOf !== undefined) { + const representativeRecord = verificationById.get(record.duplicateOf); + relationsValid &&= representativeRecord !== undefined && "verdict" in representativeRecord && clusteredCandidate !== undefined && + stableJson(record.verdict) === stableJson(reconstructDuplicateVerificationVerdict(clusteredCandidate, representativeRecord.verdict)); + } + const packet = record.verdict.finalFinding === undefined ? undefined : packetById.get(record.verdict.finalFinding.producedBy.packetId); + if (record.verdict.finalFinding !== undefined && (packet === undefined || !validChangedLineAnchor(record.verdict.finalFinding.anchor, packet, diffByHunk))) { + relationsValid = false; + } + if (record.verdict.revisedAnchor !== undefined) { + const candidate = candidateById.get(record.candidateId); + const candidatePacket = candidate === undefined ? undefined : packetById.get(candidate.producedBy.packetId); + if (candidatePacket === undefined || !validChangedLineAnchor(record.verdict.revisedAnchor, candidatePacket, diffByHunk)) { + relationsValid = false; + } + } + if (keptIds.has(record.candidateId)) { + const source = verifiedById.get(record.candidateId); + const finals = execution.finalFindings.filter((finding) => finding.mergedCandidateIds.includes(record.candidateId)); + if (source === undefined || source.id !== record.candidateId || finals.length !== 1) { + relationsValid = false; + } else { + const target = finals[0]!; + const lineageAnchor = record.verdict.revisedAnchor ?? source.anchor; + const lineageAnchorRetained = source.anchorSource === "backfill_packet_representative" || + lineageAnchor === undefined || stableJson(target.anchor) === stableJson(lineageAnchor) || + (target.mergedAnchors ?? []).some((anchor) => stableJson(anchor) === stableJson(lineageAnchor)); + const sourcePath = lineageAnchor?.path ?? source.path; + const sourceFieldsRetained = target.id !== record.candidateId || ( + stableJson(target.producedBy) === stableJson(source.producedBy) && target.category === source.category && + target.failureMode === source.failureMode && target.whyThisMatters === source.whyThisMatters && + target.verification === source.verification + ); + relationsValid &&= lineageAnchorRetained && sourceFieldsRetained && + (target.mergedCategories === undefined || target.mergedCategories.includes(source.category)) && + (target.mergedPaths === undefined || target.mergedPaths.includes(sourcePath)) && + (target.mergedTitles === undefined || target.mergedTitles.includes(source.title)); + } + } + } + } + + const composerEvidence = reconstructComposerGroupsFromArtifacts(reconstructedVerified, execution.packets); + const composerCandidateById = new Map(composerEvidence.publishable.map((finding) => [finding.id, finding])); + const pretrimSuppressedIds = new Set(composerEvidence.pretrimSuppressedIds); + const capSuppressionReasons = new Set(["severity-threshold", "confidence-threshold", "report-cap"]); + const publicationDowngradeReasons = new Set(["min-inline-confidence", "soft-comment-cap", "unanchorable"]); + const policyDrafts = execution.finalFindings.flatMap((finding) => { + const mergedFindings = unique(finding.mergedCandidateIds).flatMap((id) => composerCandidateById.get(id) ?? []); + if (mergedFindings.length !== finding.mergedCandidateIds.length) { + return []; + } + const selections = finding.mergedCandidateIds.flatMap((id) => selectionById.get(id) ?? []); + const isPretrimmed = finding.mergedCandidateIds.some((id) => pretrimSuppressedIds.has(id)); + const hasPolicyDowngrade = selections.some((selection) => publicationDowngradeReasons.has(selection.reason)); + const requestedPublication = isPretrimmed + ? "suppressed" as const + : finding.publication === "inline" || hasPolicyDowngrade + ? "inline" as const + : finding.publication === "summary-only" + ? "summary-only" as const + : mergedFindings.some((candidate) => candidate.anchor !== undefined) + ? "inline" as const + : "summary-only" as const; + const policyReasonApplied = selections.some((selection) => + capSuppressionReasons.has(selection.reason) || publicationDowngradeReasons.has(selection.reason) + ); + const representativeSelection = selectionById.get(finding.id); + const representativeBaseReason = !policyReasonApplied && + (representativeSelection?.reason === "composer-selected" || representativeSelection?.reason === "composer_omitted_finding") + ? representativeSelection.reason + : "composer-selected"; + if (representativeBaseReason === "composer_omitted_finding" && selectionArtifact.composition.mode !== "llm_degraded") { + relationsValid = false; + } + return [{ + mergedFindings, + finalBody: finding.finalBody, + requestedPublication, + baseSelection: finding.mergedCandidateIds.map((id) => isPretrimmed + ? { findingId: id, decision: "suppressed" as const, reason: "composer-pre-trim" } + : id === finding.id + ? { findingId: id, decision: "published" as const, reason: representativeBaseReason } + : { findingId: id, decision: "merged" as const, reason: "composer-merged", mergedIntoFingerprint: finding.fingerprint }) + }]; + }); + const verifierVerdicts = execution.verification.flatMap((record) => + "verdict" in record && (effectiveReview?.verify === false || record.duplicateOf === undefined) ? [record.verdict] : [] + ); + const composerPolicy = reconstructComposerPolicyFromArtifacts( + policyDrafts, + execution.packets, + execution.diff, + policyConfig, + verifierVerdicts + ); + relationsValid &&= policyDrafts.length === execution.finalFindings.length && + stableJson(composerPolicy.selection) === stableJson(execution.finalSelection) && + stableJson(composerPolicy.findings.map((finding) => finding.id)) === stableJson(execution.finalFindings.map((finding) => finding.id)); + const reconstructedFinalById = new Map(composerPolicy.findings.map((finding) => [finding.id, finding])); + for (const finding of execution.finalFindings) { + const packet = packetById.get(finding.producedBy.packetId); + const representative = candidateById.get(finding.id); + const mergedIds = unique(finding.mergedCandidateIds); + const mergedValid = mergedIds.length > 0 && mergedIds.length === finding.mergedCandidateIds.length && + mergedIds.every((id) => candidateById.has(id) && keptIds.has(id) && selectionById.has(id)); + const publicationAnchorOwner = finding.anchor === undefined ? undefined : [...packetById.values()] + .find((candidatePacket) => candidatePacket.hunks.some((hunk) => hunk.hunkId === finding.anchor?.hunkId)); + const publicationAnchorValid = finding.anchor === undefined + ? finding.publication !== "inline" && finding.changedLine === false + : publicationAnchorOwner !== undefined && validChangedLineAnchor(finding.anchor, publicationAnchorOwner, diffByHunk) && finding.changedLine === true; + const anchorsValid = packet !== undefined && publicationAnchorValid && + (finding.mergedAnchors ?? []).every((anchor) => { + const owner = [...packetById.values()].find((candidatePacket) => candidatePacket.hunks.some((hunk) => hunk.hunkId === anchor.hunkId)); + return owner !== undefined && validChangedLineAnchor(anchor, owner, diffByHunk); + }); + const selections = mergedIds.flatMap((id) => selectionById.get(id) ?? []); + const decisionsValid = finding.publication === "suppressed" + ? selections.length === mergedIds.length && selections.every((selection) => selection.decision === "suppressed") + : selections.length === mergedIds.length && selections.filter((selection) => selection.decision === "published" && selection.findingId === finding.id).length === 1 && + selections.every((selection) => selection.findingId === finding.id + ? selection.decision === "published" + : selection.decision === "merged" && selection.mergedIntoFingerprint === finding.fingerprint); + const reconstructed = reconstructedFinalById.get(finding.id); + const withoutMergedAnchors = (value: FinalFinding): Omit => { + const { mergedAnchors: _anchors, ...rest } = value; + return rest; + }; + const exactFinalMetadata = reconstructed !== undefined && reconstructed.id === finding.id && + stableJson(withoutMergedAnchors(reconstructed)) === stableJson(withoutMergedAnchors(finding)) && + (finding.mergedAnchors ?? []).every((anchor) => (reconstructed.mergedAnchors ?? []) + .some((expected) => stableJson(expected) === stableJson(anchor))); + relationsValid &&= representative !== undefined && stableJson(representative.producedBy) === stableJson(finding.producedBy) && + finding.path === packet?.path && mergedValid && anchorsValid && decisionsValid && exactFinalMetadata; + } + + const publicationById = new Map(selectionArtifact.publicationAnchors.map((record) => [record.findingId, record])); + relationsValid &&= publicationById.size === selectionArtifact.publicationAnchors.length && + stableJson(selectionArtifact.publicationAnchors) === stableJson(composerPolicy.publicationAnchors) && + publicationById.size === execution.finalFindings.length && execution.finalFindings.every((finding) => { + const record = publicationById.get(finding.id); + if (record === undefined || record.fingerprint !== finding.fingerprint || record.publication !== finding.publication || + stableJson(record.anchor) !== stableJson(finding.anchor) || record.reason.trim().length === 0) { + return false; + } + if (record.source === "none") { + return record.sourceFindingId === undefined && record.anchor === undefined; + } + if (record.sourceFindingId === undefined || !finding.mergedCandidateIds.includes(record.sourceFindingId) || record.anchor === undefined) { + return false; + } + return record.source === "selected" ? record.sourceFindingId === finding.id : record.sourceFindingId !== finding.id; + }); + + const confidenceIds = selectionArtifact.confidenceSelections.map((record) => record.findingId); + relationsValid &&= unique(confidenceIds).length === confidenceIds.length && + stableJson(selectionArtifact.confidenceSelections) === stableJson(composerPolicy.confidenceSelections) && + selectionArtifact.confidenceSelections.every((record) => { + const finding = execution.finalFindings.find((candidate) => candidate.id === record.findingId); + const representative = verifiedById.get(record.findingId); + const sourceValid = record.reason === "representative" + ? record.sourceFindingId === undefined + : record.sourceFindingId !== undefined && record.sourceFindingId !== record.findingId; + return finding !== undefined && representative !== undefined && record.confidence === finding.confidence && sourceValid && + record.representativeConfidence === representative.confidence && + (record.sourceFindingId === undefined || finding.mergedCandidateIds.includes(record.sourceFindingId)); + }); + + const groupedIds = selectionArtifact.groups.flatMap((group) => group.findingIds); + relationsValid &&= unique(selectionArtifact.groups.map((group) => group.fingerprint)).length === selectionArtifact.groups.length && + unique(groupedIds).length === groupedIds.length && + stableJson(selectionArtifact.groups) === stableJson(composerEvidence.groups.map(({ representativeId: _representativeId, ...group }) => group)) && + stableJson(composerEvidence.pretrimSuppressedIds.sort()) === stableJson(execution.finalSelection + .filter((record) => record.decision === "suppressed" && record.reason === "composer-pre-trim") + .map((record) => record.findingId).sort()); + + if (!relationsValid) { + failures.push(failure("paid_evidence_relations", `${label}/repeat ${execution.repeat} candidate, verification, selection, and final evidence is not fully relational`, { + candidates: execution.candidateFindings.length, + verification: execution.verification.length, + selections: execution.finalSelection.length, + finals: execution.finalFindings.length + })); + } + + const recomputed = recomputeEvidenceScore(run, execution); + if (stableJson(execution.score) !== stableJson(recomputed)) { + failures.push(failure("paid_evidence_score_reconstruction", `${label}/repeat ${execution.repeat} score does not exactly reconstruct from relational artifacts`, { + persisted: valueFingerprint(execution.score), + reconstructed: valueFingerprint(recomputed) + })); + } + const providerCalls = execution.modelCalls.filter((call) => call.cacheStatus !== "hit"); + const callCostUSD = sum(providerCalls.map((call) => call.costUSD ?? Number.NaN)); + if (providerCalls.some((call) => call.costUSD === undefined || !Number.isFinite(call.costUSD)) || + !Number.isFinite(callCostUSD) || Math.abs((execution.score.metrics.costUSD ?? 0) - callCostUSD) > 1e-9) { + failures.push(failure("evidence_cost_accounting", `${label}/repeat ${execution.repeat} score cost does not exactly reconcile with finite model-call cost evidence`, { + persistedCostUSD: execution.score.metrics.costUSD, + modelCallCostUSD: callCostUSD + })); + } + return recomputed; +} + +function validateRunEvidence(run: EvalCaseRunInput, label: string, failures: ReportFailure[]): EvalScore[] { + const canonicalRepeats = run.executions.map((execution) => execution.repeat); + if (stableJson(canonicalRepeats) !== stableJson(Array.from({ length: run.executions.length }, (_, index) => index + 1))) { + failures.push(failure("paid_evidence_repeat_order", `${label} repeat evidence is not in canonical repeats/ order`)); + } + const scores = run.executions.map((execution) => validateExecutionEvidence(run, execution, label, failures)); + if (scores.length === 1) { + if (stableJson(run.info.score) !== stableJson(scores[0]!)) { + failures.push(failure("paid_evidence_aggregate", `${label} info score does not reconstruct from its execution evidence`)); + } + return scores; + } + const recomputed = aggregateRepeatScores(run.declaredCase, run.executions.map((execution, index) => ({ + runDir: `repeats/${execution.repeat}`, + score: scores[index]!, + artifacts: reconstructEvidenceArtifacts(execution) + }))); + if (run.info.repeats === undefined || stableJson(run.info.repeats) !== stableJson(recomputed.aggregate) || + stableJson(run.info.score) !== stableJson(recomputed.score)) { + failures.push(failure("paid_evidence_aggregate", `${label} repeat aggregate does not reconstruct exactly from per-repeat evidence`)); + } + return scores; +} + +export function validateReplayArtifacts( + runId: string, + summary: z.infer, + diff: UnifiedDiff, + filterDecisions: FileFilterDecision[], + fileFacts: FileFacts[], + plan: ReviewPlan, + dossier: PlannerDossier, + runInfo: Record +): { base: string; head: string } { + assertReport(dossier.runId === runId && runInfo.runId === runId, "replay_run_join", `recorded replay artifacts disagree on run ID for ${runId}`); + const review = isRecord(runInfo.review) ? runInfo.review : {}; + const baseRefs = sortedUnique([ + summary.baseRef, + summary.mergeBase, + summary.startCommit, + summary.pr?.baseSha, + dossier.target.baseRef, + dossier.target.mergeBase, + typeof review.baseRef === "string" ? review.baseRef : undefined, + typeof review.mergeBase === "string" ? review.mergeBase : undefined + ].filter((value): value is string => value !== undefined)); + const headRefs = sortedUnique([ + summary.headRef, + summary.headSha, + summary.endCommit, + summary.pr?.headSha, + dossier.target.headRef, + dossier.target.headSha, + typeof review.headRef === "string" ? review.headRef : undefined, + typeof review.headSha === "string" ? review.headSha : undefined + ].filter((value): value is string => value !== undefined)); + assertReport(baseRefs.length === 1 && headRefs.length === 1, "replay_ref_join", `recorded replay artifacts disagree on base/head refs for ${runId}`, { + baseRefHashes: baseRefs.map((value) => sha256Hex(value)), + headRefHashes: headRefs.map((value) => sha256Hex(value)) + }); + const base = baseRefs[0]; + const head = headRefs[0]; + assertReport(base !== undefined && head !== undefined, "missing_refs", `recorded run ${runId} does not contain explicit base/head refs`); + + const diffPaths = diff.files.map((file) => file.path); + const diffHunks = diff.files.flatMap((file) => file.hunks); + assertReport(unique(diffPaths).length === diffPaths.length, "duplicate_diff_path", `recorded diff contains duplicate file paths for ${runId}`); + assertReport(unique(diffHunks.map((hunk) => hunk.id)).length === diffHunks.length, "duplicate_diff_hunk", `recorded diff contains duplicate hunk IDs for ${runId}`); + assertReport(diff.files.every((file) => file.hunks.every((hunk) => hunk.path === file.path)), "diff_path_join", `recorded diff hunk/file paths disagree for ${runId}`); + + const decisionPaths = filterDecisions.map((decision) => decision.path); + assertReport(unique(decisionPaths).length === decisionPaths.length && stableJson([...decisionPaths].sort()) === stableJson([...diffPaths].sort()), "filter_diff_join", `filter decisions do not form an exact join with the recorded diff for ${runId}`); + const keptPaths = filterDecisions.filter((decision) => decision.action === "keep").map((decision) => decision.path).sort(); + const factPaths = fileFacts.map((facts) => facts.path); + assertReport(unique(factPaths).length === factPaths.length && stableJson([...factPaths].sort()) === stableJson(keptPaths), "facts_filter_join", `file facts do not form an exact join with kept files for ${runId}`); + const diffByPath = new Map(diff.files.map((file) => [file.path, file])); + assertReport(fileFacts.every((facts) => { + const file = diffByPath.get(facts.path); + return file !== undefined && facts.hunkCount === file.hunks.length && facts.changedLines === sum(file.hunks.flatMap((hunk) => hunk.lines).map((line) => line.kind === "context" ? 0 : 1)); + }), "facts_diff_join", `file-fact counts disagree with the recorded diff for ${runId}`); + + const diffHunkById = new Map(diffHunks.map((hunk) => [hunk.id, hunk])); + const coverageIds = plan.coverage.map((decision) => decision.hunkId); + assertReport(unique(coverageIds).length === coverageIds.length, "plan_diff_join", `review plan contains duplicate hunk IDs for ${runId}`); + assertReport(plan.coverage.every((decision) => { + const hunk = diffHunkById.get(decision.hunkId); + return hunk !== undefined && hunk.path === decision.path && keptPaths.includes(decision.path); + }), "plan_diff_join", `review plan references unknown or filtered hunks for ${runId}`); + return { base, head }; +} + +async function replayOne(repo: string, runDir: string): Promise { + const runId = path.basename(runDir); + const summary = await loadValidatedJson(path.join(runDir, "stages", "01-input", "resolved-input.json"), "resolved input", resolvedInputSchema); + const diff = await loadValidatedJson(path.join(runDir, "stages", "02-diff", "diff.json"), "recorded diff", unifiedDiffSchema) as unknown as UnifiedDiff; + const filterDecisions = await loadValidatedJson(path.join(runDir, "stages", "02-diff", "file-filter-decisions.json"), "file filter decisions", z.array(fileFilterDecisionSchema)) as FileFilterDecision[]; + const fileFacts = await loadValidatedJson(path.join(runDir, "stages", "03-classify", "file-facts.json"), "file facts", z.array(fileFactsSchema)) as FileFacts[]; + const plan = await loadValidatedJson(path.join(runDir, "stages", "05-planner", "review-plan.json"), "review plan", reviewPlanSchema) as unknown as ReviewPlan; + const dossier = await loadValidatedJson(path.join(runDir, "stages", "05-planner", "planner-dossier.json"), "planner dossier", plannerDossierSchema) as unknown as PlannerDossier; + const runInfo = await loadValidatedJson(path.join(runDir, "run.json"), "run metadata", runMetadataSchema) as Record; + const refs = validateReplayArtifacts(runId, summary, diff, filterDecisions, fileFacts, plan, dossier, runInfo); + const { base, head } = refs; + if (runId.includes("dca8d870")) { + assertReport( + base === "d1c49bdf6a8002ec2ec27faac94a932d736532b2" && head === "fbb5f8761c2c296e115af17e919a7c35d9de8373", + "stale_replay_refs", + "motivating run refs differ from Plan 102", + { base, head } + ); + } + const git = createGitClient(repo); + assertReport(await git.commitExists(base), "missing_ref", `base ref ${base} does not exist in ${repo}`); + assertReport(await git.commitExists(head), "missing_ref", `head ref ${head} does not exist in ${repo}`); + const actualBase = await git.revParse(base); + const actualHead = await git.revParse(head); + const rawDiff = await git.diff(actualBase, actualHead); + assertReport(rawDiff.length === summary.rawDiffChars, "recorded_diff_mismatch", `rebuilt diff length differs for ${runId}`, { + recorded: summary.rawDiffChars, + actual: rawDiff.length + }); + const rebuiltDiff = stripCredentials(parseDiff(rawDiff)) as UnifiedDiff; + validateRecordedDiffParity(runId, diff, rebuiltDiff); + const tempRoot = await mkdtemp(path.join(tmpdir(), "plan102-packet-replay-")); + const worktree = path.join(tempRoot, "head"); + let worktreeAttempted = false; + try { + // Set this before invoking git: worktree add can fail after partially + // registering or creating the target path. + worktreeAttempted = true; + await runGit(repo, ["worktree", "add", "--detach", worktree, actualHead], { errorCode: "invalid_args" }); + const resolved: ResolvedReviewInput = { + ...(summary as Omit), + repoRoot: worktree, + baseRef: actualBase, + mergeBase: actualBase, + headRef: actualHead, + headSha: actualHead, + commits: summary.commits as ResolvedReviewInput["commits"], + rawDiff + }; + const keptPaths = new Set(filterDecisions.filter((decision) => decision.action === "keep").map((decision) => decision.path)); + const kept = diff.files.filter((file) => keptPaths.has(file.path)); + const config = replayConfig(dossier, runInfo); + const indexCapture = createCaptureTelemetry(`${runId}-index`); + const repoIndex = await buildRepositoryIndex(resolved, kept, fileFacts, config, indexCapture.telemetry); + const enabledLenses = dossier.lenses.map((lens) => lens.id); + const reviewContext = packetReviewContextFromDossier(dossier); + const offCapture = createCaptureTelemetry(`${runId}-off`); + const offConfig = structuredClone(config) as CodegenieConfig; + offConfig.review.packSameFileHunks = false; + offConfig.review.packedToolBudgetMode = "base"; + const offPackets = await buildReviewPackets(plan, kept, fileFacts, repoIndex, offCapture.telemetry, { + config: offConfig, + enabledLenses, + reviewContext + }); + const artifactOffPackets = stripCredentials(offPackets); + const onCapture = createCaptureTelemetry(`${runId}-on`); + const onConfig = structuredClone(config) as CodegenieConfig; + onConfig.review.packSameFileHunks = true; + onConfig.review.packedToolBudgetMode = "base"; + const onPackets = await buildReviewPackets(plan, kept, fileFacts, repoIndex, onCapture.telemetry, { + config: onConfig, + enabledLenses, + reviewContext + }); + const artifactOnPackets = stripCredentials(onPackets); + const recordedPackets = await loadPackets(path.join(runDir, "stages", "06-packets", "packets"), { allowLegacyDispatchRank: true }); + const flagOffParity = historicalFlagOffParityView(runId, recordedPackets, artifactOffPackets); + return analyzeReplayComparison({ + runId, + recordedPackets, + offPackets: artifactOffPackets, + onPackets: artifactOnPackets, + onEvents: onCapture.captured.events, + fileFacts, + diff, + plan, + expectedRefs: { base, head }, + actualRefs: { base: actualBase, head: actualHead }, + flagOffParityPackets: flagOffParity.packets, + flagOffParityMigrations: flagOffParity.migrations, + modelCallsObserved: indexCapture.captured.modelCalls.length + offCapture.captured.modelCalls.length + onCapture.captured.modelCalls.length + }); + } finally { + await finalizeReplayCleanup( + runId, + worktreeAttempted, + async () => runGit(repo, ["worktree", "remove", "--force", worktree], { errorCode: "invalid_args" }).then(() => undefined), + async () => rm(tempRoot, { recursive: true, force: true }), + async () => { + const listing = await runGit(repo, ["worktree", "list", "--porcelain"], { errorCode: "invalid_args" }); + const registered = listing.split("\n").some((line) => line === `worktree ${worktree}`); + return !existsSync(worktree) && !registered; + } + ); + } +} + +export async function finalizeReplayCleanup( + runId: string, + worktreeAttempted: boolean, + removeWorktree: () => Promise, + removeTempRoot: () => Promise, + verifyWorktreeRemoved: () => Promise +): Promise { + const errors: unknown[] = []; + let removalError: unknown; + if (worktreeAttempted) { + try { + await removeWorktree(); + } catch (error) { + removalError = error; + } + } + try { + await removeTempRoot(); + } catch (error) { + errors.push(error); + } + if (worktreeAttempted) { + try { + if (!await verifyWorktreeRemoved()) { + errors.push(removalError ?? new Error("worktree remains registered or present on disk")); + } + } catch (error) { + errors.push(error); + } + } + if (errors.length > 0) { + fail("replay_cleanup_failed", `failed to remove replay worktree for ${runId}`, { + errors: errors.map((error) => valueFingerprint(error instanceof Error ? error.message : String(error))) + }); + } +} + +function replayConfig(dossier: PlannerDossier, runInfo: Record): CodegenieConfig { + const config = structuredClone(defaultConfig) as CodegenieConfig; + if (dossier.depth === "light" || dossier.depth === "normal" || dossier.depth === "deep") { + config.review.depth = dossier.depth; + } + if (isRecord(runInfo.review)) { + const boost = asFiniteNumber(runInfo.review.budgetBoost); + if (boost !== undefined && boost > 0) { + config.review.budgetBoost = boost; + } + } + return config; +} + +type ParsedArgs = { + mode: "replay" | "eval" | "regression"; + repo?: string; + runs: string[]; + logs?: string; + baselineLogs?: string; + selectedLogs?: string; + cohort: string; + expectedRepeats?: number; + actualValidationCostUSD?: number; + output: string; +}; + +function parseArgs(argv: string[]): ParsedArgs { + const mode = argv[0]; + assertReport(mode === "replay" || mode === "eval" || mode === "regression", "invalid_args", "expected replay, eval, or regression mode"); + const values = new Map(); + for (let index = 1; index < argv.length; index += 1) { + const key = argv[index]; + assertReport(key?.startsWith("--") === true, "invalid_args", `unexpected argument: ${key ?? ""}`); + const value = argv[index + 1]; + assertReport(value !== undefined && !value.startsWith("--"), "invalid_args", `missing value for ${key}`); + values.set(key, [...(values.get(key) ?? []), value]); + index += 1; + } + const allowedKeys: Record> = { + replay: new Set(["--repo", "--run", "--output"]), + eval: new Set(["--logs", "--cohort", "--expected-repeats", "--actual-validation-cost", "--output"]), + regression: new Set(["--baseline-logs", "--selected-logs", "--cohort", "--expected-repeats", "--output"]) + }; + const unknownKeys = [...values.keys()].filter((key) => !allowedKeys[mode].has(key)); + assertReport(unknownKeys.length === 0, "invalid_args", `${mode} received unsupported arguments`, { unknownKeys }); + const single = (key: string): string | undefined => { + const entries = values.get(key) ?? []; + assertReport(entries.length <= 1, "invalid_args", `${key} may be specified only once`); + return entries[0]; + }; + const output = single("--output"); + assertReport(output !== undefined, "invalid_args", "--output is required"); + const expectedRepeatsText = single("--expected-repeats"); + const expectedRepeats = expectedRepeatsText === undefined ? undefined : Number(expectedRepeatsText); + if (expectedRepeats !== undefined) { + assertReport(Number.isInteger(expectedRepeats) && expectedRepeats > 0, "invalid_args", "--expected-repeats must be a positive integer"); + } + const validationText = single("--actual-validation-cost"); + const actualValidationCostUSD = validationText === undefined ? undefined : Number(validationText); + if (actualValidationCostUSD !== undefined) { + assertReport(Number.isFinite(actualValidationCostUSD) && actualValidationCostUSD >= 0, "invalid_args", "--actual-validation-cost must be non-negative"); + } + const repo = single("--repo"); + const logs = single("--logs"); + const baselineLogs = single("--baseline-logs"); + const selectedLogs = single("--selected-logs"); + return { + mode, + ...(repo === undefined ? {} : { repo }), + runs: values.get("--run") ?? [], + ...(logs === undefined ? {} : { logs }), + ...(baselineLogs === undefined ? {} : { baselineLogs }), + ...(selectedLogs === undefined ? {} : { selectedLogs }), + cohort: single("--cohort") ?? "latest", + ...(expectedRepeats === undefined ? {} : { expectedRepeats }), + ...(actualValidationCostUSD === undefined ? {} : { actualValidationCostUSD }), + output + }; +} + +async function writeReport(output: string, report: unknown): Promise { + await writeFile(output, formatSafeReport(report)); +} + +export function formatSafeReport(report: unknown): string { + return `${JSON.stringify(stripCredentials(safeReportOutput(report)), null, 2)}\n`; +} + +function safeReportOutput(value: unknown): unknown { + if (Array.isArray(value)) { + return value.map(safeReportOutput); + } + if (!isRecord(value)) { + return value; + } + const output: Record = {}; + for (const [key, entry] of Object.entries(value)) { + if (key === "failures" && Array.isArray(entry)) { + output[key] = entry.map(safeFailureOutput); + } else { + output[key] = safeReportOutput(entry); + } + } + return output; +} + +function safeFailureOutput(value: unknown): Record { + if (!isRecord(value)) { + const fingerprint = valueFingerprint(value); + return { code: "invalid_failure", message: `sha256:${fingerprint.sha256}:${fingerprint.length}` }; + } + const code = typeof value.code === "string" ? value.code : "invalid_failure"; + const message = typeof value.message === "string" ? value.message : stableJson(value.message); + const fingerprint = valueFingerprint(message); + return { + code, + message: `sha256:${fingerprint.sha256}:${fingerprint.length}`, + ...(value.context === undefined ? {} : { context: safeFailureValue(value.context) }) + }; +} + +function safeFailureValue(value: unknown): unknown { + if (typeof value === "string") { + return valueFingerprint(value); + } + if (Array.isArray(value)) { + return value.map(safeFailureValue); + } + if (isRecord(value)) { + if ( + (value.kind === "string" || value.kind === "json") && + typeof value.length === "number" && Number.isInteger(value.length) && value.length >= 0 && + typeof value.sha256 === "string" && /^[a-f0-9]{64}$/u.test(value.sha256) && + Object.keys(value).every((key) => key === "kind" || key === "length" || key === "sha256") + ) { + return value; + } + return Object.fromEntries(Object.entries(value).map(([key, entry]) => [key, safeFailureValue(entry)])); + } + return value; +} + +function failuresForError(error: unknown, code: string): ReportFailure[] { + return error instanceof PacketPackingReportError + ? error.failures + : [failure(code, error instanceof Error ? error.message : String(error))]; +} + +export async function runPacketPackingReportCli(argv: string[]): Promise { + const fallbackOutput = rawOutputArgument(argv); + try { + return await runPacketPackingReportCliUnchecked(argv); + } catch (error) { + const mode = argv[0] === "replay" || argv[0] === "eval" || argv[0] === "regression" ? argv[0] : "unknown"; + const report = { schemaVersion: 1, mode, failures: failuresForError(error, "packet_report_error") }; + if (fallbackOutput !== undefined) { + try { + await writeReport(fallbackOutput, report); + return 1; + } catch (writeError) { + process.stderr.write(formatSafeReport({ + schemaVersion: 1, + mode, + failures: [...failuresForError(error, "packet_report_error"), ...failuresForError(writeError, "report_write_error")] + })); + return 1; + } + } + process.stderr.write(formatSafeReport(report)); + return 1; + } +} + +function rawOutputArgument(argv: string[]): string | undefined { + const indexes = argv.flatMap((entry, index) => entry === "--output" ? [index] : []); + if (indexes.length !== 1) { + return undefined; + } + const value = argv[indexes[0]! + 1]; + return value !== undefined && !value.startsWith("--") ? value : undefined; +} + +async function runPacketPackingReportCliUnchecked(argv: string[]): Promise { + const args = parseArgs(argv); + if (args.mode === "replay") { + assertReport(args.repo !== undefined, "invalid_args", "replay requires --repo"); + assertReport(args.runs.length > 0, "invalid_args", "replay requires at least one --run"); + const rows: ReplayRow[] = []; + const failures: ReportFailure[] = []; + for (const run of args.runs) { + try { + const row = await replayOne(path.resolve(args.repo), path.resolve(run)); + rows.push(row); + failures.push(...row.failures.map((entry) => ({ ...entry, context: { run: path.basename(run), ...entry.context } }))); + } catch (error) { + const entries = error instanceof PacketPackingReportError + ? error.failures + : [failure("replay_error", error instanceof Error ? error.message : String(error))]; + failures.push(...entries.map((entry) => ({ ...entry, context: { run: path.basename(run), ...entry.context } }))); + } + } + const report: ReplayReport = { + schemaVersion: 1, + mode: "replay", + noModelCalls: rows.length === args.runs.length && rows.every((row) => row.modelCallsObserved === 0), + repo: path.resolve(args.repo), + rows, + failures + }; + await writeReport(args.output, report); + return failures.length === 0 && rows.length === args.runs.length ? 0 : 1; + } + + assertReport(args.expectedRepeats !== undefined, "invalid_args", `${args.mode} requires --expected-repeats`); + if (args.mode === "eval") { + assertReport(args.logs !== undefined, "invalid_args", "eval requires --logs"); + try { + const cohort = selectExplicitCohort(await loadEvalRuns(path.resolve(args.logs), args.cohort), args.cohort); + const report = analyzeEvalCohort(cohort, args.expectedRepeats, { + ...(args.actualValidationCostUSD === undefined ? {} : { actualValidationCostUSD: args.actualValidationCostUSD }) + }); + await writeReport(args.output, report); + return report.failures.length === 0 ? 0 : 1; + } catch (error) { + await writeReport(args.output, { schemaVersion: 1, mode: "eval", failures: failuresForError(error, "eval_report_error") }); + return 1; + } + } + + assertReport(args.baselineLogs !== undefined && args.selectedLogs !== undefined, "invalid_args", "regression requires --baseline-logs and --selected-logs"); + try { + const baseline = selectExplicitCohort(await loadEvalRuns(path.resolve(args.baselineLogs), args.cohort), args.cohort); + const selected = selectExplicitCohort(await loadEvalRuns(path.resolve(args.selectedLogs), args.cohort), args.cohort); + const report = analyzeRegressionCohorts(baseline, selected, args.expectedRepeats); + await writeReport(args.output, report); + return report.failures.length === 0 ? 0 : 1; + } catch (error) { + await writeReport(args.output, { schemaVersion: 1, mode: "regression", failures: failuresForError(error, "regression_report_error") }); + return 1; + } +} + +const invokedPath = process.argv[1] === undefined ? undefined : pathToFileURL(path.resolve(process.argv[1])).href; +if (invokedPath === import.meta.url) { + runPacketPackingReportCli(process.argv.slice(2)).then((exitCode) => { + process.exitCode = exitCode; + }).catch((error) => { + process.stderr.write(formatSafeReport({ schemaVersion: 1, mode: "unknown", failures: failuresForError(error, "packet_report_error") })); + process.exitCode = 1; + }); +} diff --git a/specs/plans/102-issue-102-same-file-packet-packing.md b/specs/plans/102-issue-102-same-file-packet-packing.md index e1f41c5..7f49543 100644 --- a/specs/plans/102-issue-102-same-file-packet-packing.md +++ b/specs/plans/102-issue-102-same-file-packet-packing.md @@ -13,6 +13,25 @@ Recommended priority: next throughput plan. Plan 100 is complete, and run `dca8d > Working-tree check: `git status --short -- src/pipeline/packet-builder.ts src/config/schema.ts src/config/config-loader.ts src/types.ts src/evals/eval-runner.ts scripts/packet-packing-report.ts tests/fixtures/packet-packing-golden.json tests/pipeline-phase5.test.ts tests/config-loader.test.ts tests/evals.test.ts tests/packet-packing-report.test.ts` > If the drift check reports committed changes, reconcile the current-state claims below. If the working-tree check reports changes owned by another task, stop and wait for that work to land or move this plan to an isolated worktree; do not overwrite it. +## Phase 2 deterministic reconciliation (2026-07-24) + +Step 6's exact four-run command completed with exit `0`, `noModelCalls: true`, four explicit `modelCallsObserved: 0` row proofs, and no report failures. The frozen output is `/tmp/plan102-packet-shape.json` (SHA-256 `77af0c38937bd6957806f05ad201cbad32461414c1d667824096208473c276fa`). The authoritative motivating count is **75**: the real `combinedPatchChars()` builder produced 96→75, so the bounded 74/76 reconciliation was not used. The report's diagnostic reapplication of the old `contentWithLineNumbers` proxy produces 76 for the current atom stream while the real measurement produces 75; this does not change the golden because the authoritative builder result itself matches the plan's historical 75 target. The frozen hash was refreshed after the verifier stopped trusting serialized cap telemetry and independently rendered the source diff for every packet that actually combines multiple atoms. + +| Retained run | Packets off→on | Reduction | Hunks | Multi-atom packets | Hunk distribution off→on (1/2/3/4/5) | Atoms per on-packet (1/2/3/4) | +| --- | ---: | ---: | ---: | ---: | --- | --- | +| `740d73f2` | 109→85 | 22.018% | 153 | 16 | 86/12/5/2/4 → 54/13/6/5/7 | 69/9/6/1 | +| `fe1548ae` | 109→85 | 22.018% | 153 | 16 | 86/12/5/2/4 → 54/13/6/5/7 | 69/9/6/1 | +| `81f806a6` | 93→68 | 26.882% | 136 | 17 | 71/11/5/2/4 → 38/12/6/4/8 | 51/10/6/1 | +| `dca8d870` | 96→75 | 21.875% | 142 | 14 | 73/11/5/3/4 → 46/12/4/5/8 | 61/8/5/1 | + +All four rows have zero hunk/atom bijection failures, atom splits or reorderings, new coverage promotions, cap violations, planner-lens drops, high/critical focus omissions, deep-context downgrades, derived/effective profile downgrades, effective base-budget downgrades, and invalid dispatch ranks. The largest eligible packet has five hunks; independently rendered raw-diff patch use for every multi-atom packet is at most 11,755 characters in each run. Oversized single-hunk packets are intentionally windowed and are not atom-combination cap candidates. Flag-off and flag-on context truncation counts are identical (13/13, 13/13, 13/13, and 15/15). Each run internalizes two related-context excerpts because their target hunk is now in the same packet. Six standalone note strings per run are absent after the unchanged bounded note merge, but none belongs to a high/critical atom; this remains an explicit paid-recall concern rather than being hidden as deterministic parity. Lens omissions are zero. All treatment packets use the `base` budget and the eight direct file/whole-file bypass packets per run retain their baseline budget. + +Packing changes the complete scheduling position of 129, 129, 129, and 117 hunks respectively, while every recomputed rank satisfies the unchanged `[fileClassRank, -changedLines]` formula. The motivating row's 75 packets at concurrency 6 versus 96 at concurrency 4 gives a capacity multiplier of `(96 / 75) * (6 / 4) = 1.92×`. Its completion projection is `52.7 * (75 / 57) * (4 / 6) = 46.23` minutes. + +The motivating run has byte-identical flag-off artifacts after applying the same credential stripping used by artifact serialization. The three older retained artifacts predate Plan 100 and omit `dispatchRank`; the replay applies an explicit run-ID-pinned compatibility view that removes only that absent serialized field after validating every current rank. Those same three recorded diffs predate persisted `hunkHash` and the current derived hunk ID. Their versioned diff parity requires every hunk to use the old schema, then compares every other parsed field exactly against fresh `parseDiff(rawDiff)` output; partial migration or any semantic diff drift fails closed. The oldest clean artifact also predates the self-test context filter, so 16 exact redundant `path:path` test lines are removed in that compatibility view. Any other field drift still fails closed, and the current packing-off implementation remains covered by the packet-builder golden parity tests. + +Paid validation was not authorized or run in this phase: `actualValidationCostUSD = $0`; steps 7–12 are `not_run` pending an owner-supplied `approvedValidationCostUSD`. No paid model call may begin until that ceiling and the next-phase projection are recorded. + ## Decision Pack more same-file work, but only across boundaries that do not carry a deliberate review signal: @@ -202,7 +221,7 @@ Use staged measurements instead of guessing the final bill: 3. The two production cases run as one paired suite, so approve that phase before launching either case. Run D cost `$25.2852` in total: `$22.8517` in Stage 7 across 57 dispatched packets (`$0.4009` per dispatched packet) plus `$2.4335` outside Stage 7. For frozen packed count `P`, forecast a complete pair as `2 * $2.4335 + (96 + P) * $0.4009`, or `$73.02–$73.82` for `P in {74, 75, 76}`. Applying 25% contingency yields `$91.28–$92.28`; round upward and reserve **`$95`**. This budgets all 96 baseline packets plus all packed packets even if the 60-minute baseline later truncates. Replace the reservation with actual paired cost afterward. 4. Before starting any next phase, require `actualValidationCostUSD + projectedRemainingCostUSD <= approvedValidationCostUSD`. If it does not hold, stop for explicit owner approval rather than dropping recall or deterministic gates to save money. -The report script must aggregate cohort cost from `score.metrics.costUSD`, falling back to the actual value of the `maxCostUSD` budget result when necessary. It emits cohort `actualCostUSD`; the executor records cumulative actual/projected/approved amounts in the reconciliation note using the formulas above. +The report script must reconstruct cohort cost from every recorded model call and require the per-repeat and aggregate score cost/count fields to match that evidence. Candidate, verification, final-selection, and final artifacts must also form an exact lineage before any paid score is accepted. It emits cohort `actualCostUSD`; the executor records cumulative actual/projected/approved amounts in the reconciliation note using the formulas above. Normalize production payback to equivalent reviewed work because the baseline may truncate while the selected arm completes. For the pinned diff, set `equivalentTargetHunks = 142` and compute: @@ -270,7 +289,7 @@ Author six case files (`dilution-{a,b,c}.yml`, `consistency-{a,b,c}.yml`) with ` | B — isolate shape | on | base | | C — shape + capacity | on | atom-scaled | -Run the one-repeat treatment preflight, then use `scripts/packet-packing-report.ts eval` on the suite logs. For both cases it must prove: +Run the one-repeat treatment preflight, then use `scripts/packet-packing-report.ts eval` on the suite logs. Preflight proves treatment and economics only and must not select a rollout arm. Invocation manifests record each run's exact owning log root/path so a suite may span roots without ambiguous numeric run IDs. For both cases it must prove: - A's target packet has one source atom; - B and C put the target hunk in a packet with at least two source atoms; @@ -302,7 +321,7 @@ Define Stage-7 tool-pressure rate per arm as: rejectionRate = rejected repository-tool attempts / reviewed pre-existing source atoms ``` -Report the rejection cause/limit when telemetry supplies it, result characters, used calls, continuations, and model-service time per reviewed atom. C qualifies as a budget fix only when `rejectionRate(C) <= rejectionRate(B)` and it either strictly reduces rejection rate or improves candidate/final recall. If C and B have equal recall and pressure, choose B. If the selected packed arm exceeds A by more than `0.10` rejected attempts per reviewed atom without higher candidate recall, stop and keep packing dark. +Report the rejection cause/limit when telemetry supplies it, result characters, used calls, continuations, and model-service time per reviewed atom. C qualifies as a budget fix only when `rejectionRate(C) <= rejectionRate(B)` and it either strictly reduces rejection rate or improves candidate/final recall. If C and B have equal recall and pressure, choose B. If B fails and C passes, C must retain at least 85% of the model-service saving implied by its packet-count reduction. If the selected packed arm exceeds A by more than `0.10` rejected attempts per reviewed atom without higher candidate recall, stop and keep packing dark. ### C. Existing-suite collateral regression checks @@ -313,7 +332,7 @@ After the repeated packing-sensitive gate selects B or C, run two broader suites Materialize A and selected copies in disposable directories, copying the complete suite including `repos/`, and change only `repeat`, `review.packSameFileHunks`, and `review.packedToolBudgetMode`. The report must verify those are the only YAML differences, that the selected setting reached resolved config, and how many selected executions actually packed multiple atoms. Untreated cases remain valid collateral checks for flag/config parity but are explicitly not evidence about packing recall. -Add a `regression` mode to `scripts/packet-packing-report.ts` that compares the two explicit log roots and reports eval errors, expectation transitions, packet/hunk/cap/profile invariants, treatment counts, tool pressure, and dispatch-order changes. It must not merge these one-repeat outcomes into the repeated packing-sensitive recall gate. +Add a `regression` mode to `scripts/packet-packing-report.ts` that compares the two explicit log roots and reports eval errors, expectation transitions, packet/hunk/cap/profile invariants, treatment counts, tool pressure, and dispatch-order changes. Each baseline and selected cohort must independently pass every required positive/negative expectation even when both fail identically. It must not merge these one-repeat outcomes into the repeated packing-sensitive recall gate. ### D. One production-shaped capacity confirmation @@ -323,7 +342,7 @@ Confirm: - at least the same hunk set is reviewed, with the target being all 142 reviewable hunks; - accepted/candidate findings and known cross-hunk observations are not lost; -- total model-service time, total tokens, cost per reviewed hunk, and wall time improve; +- total model-service time, total/reasoning tokens, cost per reviewed hunk, and wall time improve, with totals covering planner, reviewer, verifier, composer, repair, and every other recorded call rather than Stage 7 alone; - raw arm costs/reviewed-hunk counts and 142-hunk equivalent costs are reported, with truncation extrapolations labeled and positive equivalent-review savings required; - context/tool pressure stays within the offline and eval gates; - profile floors hold, and dispatch-order plus reviewed-hunk-set changes are reported so a Plan-100 ordering interaction cannot masquerade as a packing-quality result. diff --git a/specs/plans/phase-plans/102/phase_2.md b/specs/plans/phase-plans/102/phase_2.md new file mode 100644 index 0000000..9d6c574 --- /dev/null +++ b/specs/plans/phase-plans/102/phase_2.md @@ -0,0 +1,33 @@ +--- +status: complete +--- + +# Phase 2: Packet-Packing Reports and Deterministic Replay + +## Overview + +Implement Plan 102 steps 5–6 only. This phase adds a fail-closed report tool for retained-run Stage-6 replay, repeated A/B/C evaluation analysis, collateral regression comparison, treatment/economics/tool-pressure accounting, and production-normalized payback. It then runs the exact four retained inputs without model calls, freezes `/tmp/plan102-packet-shape.json`, and records the deterministic packet/cap/context/lens/note reconciliation in Plan 102. Paid eval execution, private recall fixtures, rollout selection, and teardown remain out of scope. + +## Steps + +1. Add `scripts/packet-packing-report.ts` with an import-safe CLI and exported pure report builders/parsers. Define strict JSON boundary validation for retained-run artifacts, eval `info.json`/repeat execution artifacts, packets, events, model/tool calls, and cost sources. Make every missing/corrupt/ambiguous input or invariant violation a named report failure and non-zero CLI exit. +2. Implement `replay` mode. Parse `--repo`, repeated `--run`, and `--output`; verify each run directory and recorded base/head/merge-base refs; materialize the recorded head in a temporary detached Git worktree; rebuild the repository index and Stage 6 twice from the recorded diff, filter decisions, file facts, planner plan/dossier, and enabled lenses with packing off/base and on/base; capture all Stage-6 telemetry without any model runner; and remove the worktree in `finally`. +3. Compare replay outputs against recorded and off/on packet shapes. Enforce recorded flag-off artifact parity, reviewable-hunk bijection, atom preservation/order, file/language/source order, five-hunk and 12K caps, zero new coverage promotion, lens preservation, high/critical note preservation, profile/base-budget monotonicity, and the unchanged dispatch-rank formula. Emit stable per-run packet/atom counts, packet membership, changed scheduling position by hunk, derived-floor/profile/budget evidence, and cap/context/related-context/note/lens omissions. Apply the one-time 74/75/76 motivating-run rule only to real `combinedPatchChars()` measurement; reject all other target/count/ref failures. +4. Implement `eval` mode with explicit `--logs`, `--cohort latest|`, `--expected-repeats`, and `--output`. Persist each invocation run's exact owning log root/path so multi-root suites and repeated numeric run IDs resolve unambiguously. Select one complete latest cohort explicitly rather than pooling history, reject an interrupted newest invocation, validate A/B/C arm identity and effective config, and join executions bidirectionally by case, arm, repeat, and expectation. Resolve every B/C treatment atom to the matching A-repeat packet, require exact atom/hunk bijections, and independently recompute cap use, coverage/profile floors, base/effective budgets, and dispatch ranks from the artifacts. Require at least 8/10 treated executions when ten repeats are expected; repeat-1 preflight cannot select a rollout arm. +5. Compute evaluation views and economics. Strictly load candidate, verification, final-selection, and final artifacts and reconstruct their packet/hunk/changed-line/provenance/lineage joins, matched score IDs, per-repeat finding/count/cost metrics, and repeat aggregates. Use every execution for the primary intent-to-treat candidate/final rates and atom-count-bucketed loss histograms; emit a separately labeled treated-only view whose numerator and denominator both exclude untreated B/C executions. Aggregate actual cohort spend from all recorded model calls; aggregate reviewed atoms, repository-tool requested/used/rejected calls, rejection causes, result characters, continuations, service time, input/output tokens, and cost per reviewed atom. Enforce aggregate A-relative non-inferiority, zero-recall, treatment, per-atom pressure, strict C-improvement, and B-fail/C-pass 85%-saving-retention gates. +6. Add production economics and capacity gates to `eval` output when a baseline/selected production pair is present: exact external repo/refs, cache-off, concurrency-6, declared/effective 60-minute shape, exact bidirectional reviewed-hunk sets with 142/142 required, raw cost and cost per reviewed hunk, normalized 142-hunk equivalent costs, per-arm extrapolation labels, positive equivalent-review savings, cumulative-validation-cost break-even inputs/count, wall throughput/time, all-stage model-service seconds and total/reasoning tokens per hunk, Stage-7 continuations per atom, and normalized tool pressure. Fail closed on truncated work, missing evidence, non-improving throughput, regressions in any capacity metric, or a non-positive payback denominator. +7. Implement `regression` mode with explicit `--baseline-logs` and `--selected-logs`, independent explicit cohort selection, `--expected-repeats`, and `--output`. Verify only permitted YAML differences and selected effective config, independently require both cohorts to pass every required positive/negative expectation, then report eval errors/expectation transitions, deterministic negative/positive regressions, relational artifacts, packet/hunk/cap/profile/budget/rank invariants, treatment counts, dispatch-order movement, per-atom pressure, and actual cohort cost. Label the result as one-repeat collateral evidence and never combine it with repeated recall gates. +8. Add `tests/packet-packing-report.test.ts` using disposable synthetic retained runs and eval cohorts. Cover successful replay/eval/regression/production summaries plus fail-closed cases for strict packet/event/model/tool/coverage/info schemas, corrupt/missing or cross-artifact refs, interrupted cohorts, flag-off mismatch without source leakage, replay model calls and cleanup failure, hunk/atom loss or mismatch, cap/profile/base-budget/rank defects, bidirectional expectation mismatch, missing treatment, incorrect denominators, recall regression, valid untreated collateral, equal B/C tie selection, exact production work/capacity, missing spend data, and non-positive payback. +9. Run `pnpm test -- tests/packet-packing-report.test.ts`, the Plan 102 focused suite, `pnpm run check`, `pnpm test`, and `pnpm build`, iterating until clean. +10. Run the exact four-run no-LLM replay command against `/home/peter/Dev/0xsequence/trails-api` and the retained directories from Plan 102, writing `/tmp/plan102-packet-shape.json`. If and only if all deterministic gates pass, add a Plan 102 reconciliation note containing the JSON table, frozen motivating count (74/75/76 only), proxy-versus-real patch explanation when needed, per-run distribution, cap/context/related-context/lens/note/profile/budget/rank comparison, capacity multiplier, and completion projection. Stop with a roadblock on any unpermitted count/ref/invariant failure. + +## Tests + +- `replay rebuilds recorded Stage 6 without model calls`: verifies explicit refs/worktree cleanup, recorded inputs, stable off/on summaries, packet membership, cap/context/lens/note/profile/budget/rank reporting, and no LLM path. +- `replay fails closed on stale or corrupt evidence`: covers missing/corrupt artifacts and refs, recorded/off parity mismatch, hunk loss/duplication, atom split/reorder, cap/coverage/lens/focus/profile/budget/rank violations, and an unpermitted motivating packet count. +- `eval selects one explicit A/B/C cohort`: proves latest/id selection, arm/config identity, exact repeat joins, no mixed cohorts, and actual cost aggregation with the documented fallback. +- `eval reports intent-to-treat and treated-only recall separately`: proves all executions remain in the primary denominator while untreated B/C executions are removed from both treated-only numerator and denominator. +- `eval proves treatment and per-atom pressure`: proves Stage-6 provenance joins, target packet/coverage/lens/profile/budget/cap invariants, 8/10 thresholds, atom-normalized calls/rejections/results/continuations/service/tokens/cost, and rejection-cause reporting. +- `eval fails closed on recall, treatment, telemetry, and spend defects`: covers missing/mismatched telemetry, under-treatment, zero or out-of-band recall, missing cost, and invalid denominators. +- `production economics normalize equal reviewed work`: proves raw versus 142-hunk-equivalent cost, extrapolation labels, positive savings, validation break-even count, and failure on non-positive normalized savings. +- `regression compares explicit roots as collateral evidence`: proves separate cohort selection, permitted YAML/config deltas, expectation/eval transition detection, treatment/invariant/pressure/order/cost reporting, and no accidental recall-cohort pooling. diff --git a/src/evals/eval-artifacts.ts b/src/evals/eval-artifacts.ts index 791e772..007de6d 100644 --- a/src/evals/eval-artifacts.ts +++ b/src/evals/eval-artifacts.ts @@ -6,6 +6,7 @@ import type { EvalArtifacts, EvalHintEvent, EvalHumanAttentionNote, + EvalInvocationManifest, EvalRunInfo, EvalSelectionRecord, EvalVerificationRecord, @@ -138,6 +139,16 @@ export async function writeEvalRunInfo(dir: string, info: EvalRunInfo): Promise< await rename(tmp, target); } +export async function writeEvalInvocationManifest(logsDir: string, manifest: EvalInvocationManifest): Promise { + const relative = path.join("invocations", `${manifest.invocationId}.json`); + const target = path.join(logsDir, relative); + await mkdir(path.dirname(target), { recursive: true }); + const tmp = `${target}.${process.pid}.${Date.now()}.tmp`; + await writeFile(tmp, `${JSON.stringify(manifest, null, 2)}\n`); + await rename(tmp, target); + return relative; +} + export function resolveTelemetryDir(runOrTelemetryDir: string): string { const resolved = path.resolve(runOrTelemetryDir); if (path.basename(resolved) === "telemetry") { diff --git a/src/evals/eval-command.ts b/src/evals/eval-command.ts index c26ca1a..b4a12ee 100644 --- a/src/evals/eval-command.ts +++ b/src/evals/eval-command.ts @@ -1,10 +1,12 @@ +import { randomUUID } from "node:crypto"; import path from "node:path"; import { Command, CommanderError } from "commander"; import { loadConfig } from "../config/config-loader.js"; -import type { CodegenieConfig, EvalCaseResult, EvalLossLabel } from "../types.js"; +import type { CodegenieConfig, EvalCaseResult, EvalInvocationManifest, EvalLossLabel } from "../types.js"; import { CodegenieError } from "../util/errors.js"; import { CliDisplayExit } from "../cli/review-command.js"; -import { loadEvalSuite, replayFromArtifacts, runEvalCase } from "./eval-runner.js"; +import { writeEvalInvocationManifest } from "./eval-artifacts.js"; +import { loadEvalSuite, replayFromArtifacts, resolveLogsDir, runEvalCase } from "./eval-runner.js"; export type EvalCommandOptions = { evalDir?: string; @@ -59,15 +61,52 @@ export async function runEvalCommand( throw new CodegenieError("invalid_args", "--eval-dir is required when eval.defaultEvalDir is not configured"); } const suite = await loadEvalSuite(evalDir); + const invocationId = randomUUID(); + const startedAt = new Date().toISOString(); + const manifestRelativePath = `invocations/${invocationId}.json`; + const manifest: EvalInvocationManifest = { + schemaVersion: 1, + invocationId, + suiteDir: suite.dir, + status: "running", + startedAt, + cases: suite.cases.map((entry, caseIndex) => ({ + caseIndex, + caseName: entry.evalCase.name, + caseHash: entry.caseHash, + caseFile: entry.file + })), + runs: [] + }; + const manifestRoots = [...new Set(suite.cases.map((entry) => resolveLogsDir(suite.dir, entry.evalCase, config)))]; + const persistManifest = async (): Promise => { + await Promise.all(manifestRoots.map((logsDir) => writeEvalInvocationManifest(logsDir, manifest))); + }; + await persistManifest(); const results: EvalCaseResult[] = []; - for (const entry of suite.cases) { + for (const [caseIndex, entry] of suite.cases.entries()) { const result = await runEvalCase(suite, entry, { config, + invocation: { id: invocationId, caseIndex, manifest: manifestRelativePath }, ...(options.cache !== undefined ? { cacheOverride: options.cache } : {}) }); results.push(result); + const logsRoot = resolveLogsDir(suite.dir, entry.evalCase, config); + const runPath = path.relative(logsRoot, result.runDir); + manifest.runs.push({ + caseIndex, + caseName: entry.evalCase.name, + caseHash: entry.caseHash, + runNumber: result.info.runNumber, + logsRoot, + runPath + }); + await persistManifest(); write(renderCaseResult(result)); } + manifest.status = "complete"; + manifest.completedAt = new Date().toISOString(); + await persistManifest(); write(renderSuiteTotals(results)); return results.every((result) => result.status === "pass") ? 0 : 1; } diff --git a/src/evals/eval-runner.ts b/src/evals/eval-runner.ts index 01b974c..add3e49 100644 --- a/src/evals/eval-runner.ts +++ b/src/evals/eval-runner.ts @@ -50,6 +50,7 @@ export type EvalSuite = { export type EvalRunOptions = { cacheOverride?: boolean; config: CodegenieConfig; + invocation?: EvalRunInfo["invocation"]; }; const positiveNumberSchema = z.number().positive(); @@ -336,6 +337,7 @@ export async function replayFromArtifacts( finishedAt, score, config: options.config, + ...(options.invocation !== undefined ? { invocation: options.invocation } : {}), replay: { sourceArtifacts: source, caseSource: reread.source }, ...(sourceInfo.caseFile !== undefined ? { caseFile: sourceInfo.caseFile } : {}) }); @@ -353,7 +355,9 @@ export async function replayFromArtifacts( startedAt, error, "replay", - { sourceArtifacts: source, caseSource: reread.source } + { sourceArtifacts: source, caseSource: reread.source }, + undefined, + options.invocation ); } } @@ -417,6 +421,7 @@ async function runArtifactCase( finishedAt, score, config: options.config, + ...(options.invocation !== undefined ? { invocation: options.invocation } : {}), replay: { sourceArtifacts: source, caseSource: "yaml" } }); await writeRunOutputs(allocated.dir, path.dirname(allocated.dir), info, artifacts.finalFindings); @@ -429,7 +434,9 @@ async function runArtifactCase( startedAt, error, "replay", - { sourceArtifacts: source, caseSource: "yaml" } + { sourceArtifacts: source, caseSource: "yaml" }, + undefined, + options.invocation ); } } @@ -492,6 +499,7 @@ async function runLiveCase( finishedAt, score, config: caseConfig.config, + ...(options.invocation !== undefined ? { invocation: options.invocation } : {}), cache: caseConfig.cache, repo: await repoInfo(actualRepoRoot, telemetryDir), ...(reviewRunId !== undefined ? { reviewRunId } : {}) @@ -499,7 +507,7 @@ async function runLiveCase( await writeRunOutputs(allocated.dir, path.dirname(allocated.dir), info, artifacts.finalFindings); return { caseName: info.caseName, runDir: allocated.dir, status: info.score.status, info }; } catch (error) { - return writeErroredCase(allocated, entry, errorConfig, startedAt, error, "live", undefined, errorCache); + return writeErroredCase(allocated, entry, errorConfig, startedAt, error, "live", undefined, errorCache, options.invocation); } } @@ -581,6 +589,7 @@ async function runRepeatedLiveCase( score, repeats: aggregate, config: caseConfig.config, + ...(options.invocation !== undefined ? { invocation: options.invocation } : {}), cache: caseConfig.cache, repo: await repoInfo(actualRepoRoot, path.join(allocated.dir, "repeats", "1", "telemetry")) }); @@ -589,7 +598,7 @@ async function runRepeatedLiveCase( await writeEvalRunInfo(allocated.dir, info); return { caseName: info.caseName, runDir: allocated.dir, status: info.score.status, info }; } catch (error) { - return writeErroredCase(allocated, entry, errorConfig, startedAt, error, "live", undefined, errorCache); + return writeErroredCase(allocated, entry, errorConfig, startedAt, error, "live", undefined, errorCache, options.invocation); } } @@ -601,7 +610,8 @@ async function writeErroredCase( error: unknown, mode: "live" | "replay", replay?: EvalRunInfo["replay"], - cache?: EvalRunInfo["cache"] + cache?: EvalRunInfo["cache"], + invocation?: EvalRunInfo["invocation"] ): Promise { const finishedAt = new Date().toISOString(); const score = errorScore(error); @@ -616,6 +626,7 @@ async function writeErroredCase( finishedAt, score, config, + ...(invocation !== undefined ? { invocation } : {}), ...(cache !== undefined ? { cache } : {}) }); await writeFile(path.join(allocated.dir, "out.log"), `${JSON.stringify({ level: "error", message: score.error?.message, code: score.error?.code })}\n`); @@ -693,6 +704,13 @@ async function loadCaseFile(filePath: string, suiteDir: string): Promise { + return loadCaseFile(path.resolve(filePath), path.resolve(suiteDir)); +} + async function rereadReplayCase( sourceInfo: EvalRunInfo, suiteDir: string @@ -729,6 +747,7 @@ function buildRunInfo(input: { repeats?: EvalRunInfo["repeats"]; repo?: EvalRunInfo["repo"]; reviewRunId?: string; + invocation?: EvalRunInfo["invocation"]; startedAt: string; finishedAt: string; score: EvalScore; @@ -746,6 +765,7 @@ function buildRunInfo(input: { ...(input.repeats !== undefined ? { repeats: input.repeats } : {}), ...(input.repo !== undefined ? { repo: input.repo } : {}), ...(input.reviewRunId !== undefined ? { reviewRunId: input.reviewRunId } : {}), + ...(input.invocation !== undefined ? { invocation: input.invocation } : {}), codegenieRuntime: resolveCodegenieRuntimeProvenance(), cache: input.cache ?? { enabled: input.config.cache.enabled, source: "config", dir: input.config.cache.dir }, effectiveConfig: evalEffectiveConfig(input.config), @@ -875,6 +895,12 @@ function evalEffectiveConfig(config: CodegenieConfig): NonNullable; + pretrimSuppressedIds: string[]; + publishable: CandidateFinding[]; +} { + const publishable = verified.map((candidate) => candidate.anchorSource === "backfill_packet_representative" && candidate.anchor !== undefined + ? (() => { + const { anchor: _anchor, ...withoutAnchor } = candidate; + return { ...withoutAnchor, changedLine: false }; + })() + : candidate); + const pretrim = pretrimComposerInput(publishable); + const packetsById = new Map(packets.map((packet) => [packet.id, packet])); + return { + groups: groupFindings(pretrim.kept, packetsById).map((group) => ({ + fingerprint: group.fingerprint, + representativeId: group.representative.id, + findingIds: group.findings.map((finding) => finding.id) + })), + pretrimSuppressedIds: pretrim.suppressed.map((finding) => finding.id), + publishable + }; +} + +export function reconstructFinalFindingFromArtifacts( + mergedFindings: CandidateFinding[], + packets: ReviewPacket[], + diff: UnifiedDiff, + finalBody: string, + publication: FinalFinding["publication"] +): { + finding: FinalFinding; + publicationAnchor: PublicationAnchorDecision; + confidenceSelection: ConfidenceSelection; +} { + const representative = canonicalMergedRepresentative(mergedFindings, diff); + const packetsById = new Map(packets.map((packet) => [packet.id, packet])); + const fingerprint = fingerprintFinding(representative, packetsById); + const publicationAnchors = new Map(); + const confidenceSelections = new Map(); + const finding = toFinalFinding( + representative, + fingerprint, + finalBody, + publication, + mergedFindings, + diff, + publicationAnchors, + confidenceSelections + ); + return { + finding, + publicationAnchor: publicationAnchors.get(finding.id) ?? { source: "none", reason: "no publication anchor" }, + confidenceSelection: confidenceSelections.get(finding.id) ?? { confidence: representative.confidence, reason: "representative" } + }; +} + +export type ComposerArtifactPolicyDraft = { + mergedFindings: CandidateFinding[]; + finalBody: string; + requestedPublication: FinalFinding["publication"]; + baseSelection: SelectionRecord[]; +}; + +export function reconstructComposerPolicyFromArtifacts( + drafts: ComposerArtifactPolicyDraft[], + packets: ReviewPacket[], + diff: UnifiedDiff, + config: CodegenieConfig, + verdicts: VerificationVerdict[] +): { + findings: FinalFinding[]; + selection: SelectionRecord[]; + publicationAnchors: Array<{ + findingId: string; + fingerprint: string; + publication: FinalFinding["publication"]; + source: PublicationAnchorDecision["source"]; + reason: string; + sourceFindingId?: string; + anchor?: DiffAnchor; + }>; + confidenceSelections: Array; +} { + const publicationAnchorDecisions = new Map(); + const confidenceSelections = new Map(); + const anchorDowngradeReasons = new Map(); + const packetsById = new Map(packets.map((packet) => [packet.id, packet])); + const baseSelection = new Map(); + const findings = drafts.map((draft) => { + const representative = canonicalMergedRepresentative(draft.mergedFindings, diff); + const finding = toFinalFinding( + representative, + fingerprintFinding(representative, packetsById), + draft.finalBody, + draft.requestedPublication, + draft.mergedFindings, + diff, + publicationAnchorDecisions, + confidenceSelections + ); + recordAnchorDowngrade(finding, draft.requestedPublication, anchorDowngradeReasons); + for (const record of draft.baseSelection) { + baseSelection.set(record.findingId, record); + } + return finding; + }); + const quietTelemetry = { event: () => undefined } as unknown as TelemetryRecorder; + const capped = applyCaps(findings, config, { + lowConfidencePublishableIds: lowConfidencePublishableCandidateIds(verdicts), + telemetry: quietTelemetry + }); + for (const [id, reason] of anchorDowngradeReasons) { + if (!capped.downgradeReasons.has(id)) { + capped.downgradeReasons.set(id, reason); + } + } + return { + findings: capped.findings, + selection: buildSelectionRecords(capped.findings, baseSelection, capped.suppressedReasons, capped.downgradeReasons), + publicationAnchors: publicationAnchorSelectionRecords(capped.findings, publicationAnchorDecisions), + confidenceSelections: [...confidenceSelections.entries()] + .map(([findingId, selection]) => ({ findingId, ...selection })) + .sort((left, right) => left.findingId.localeCompare(right.findingId)) + }; +} + function mergeProximityGroups(groups: FindingGroup[], packetsById: Map): FindingGroup[] { const merged: FindingGroup[] = []; for (const group of groups) { diff --git a/src/pipeline/review-runner.ts b/src/pipeline/review-runner.ts index 38b07c8..499081b 100644 --- a/src/pipeline/review-runner.ts +++ b/src/pipeline/review-runner.ts @@ -921,7 +921,13 @@ async function maybeZeroWork( }); await run.telemetry.writeArtifact("candidate-findings.json", []); await run.telemetry.writeArtifact("verification.json", []); - await run.telemetry.writeArtifact("final-selection.json", { records: [], groups: [] }); + await run.telemetry.writeArtifact("final-selection.json", { + composition: { mode: "deterministic_fallback", fallbackReason: "zero-work short circuit" }, + records: [], + publicationAnchors: [], + confidenceSelections: [], + groups: [] + }); await run.telemetry.writeArtifact("final-findings.json", []); await run.telemetry.writeArtifact("budget-summary.json", result.budgetSummary); await maybePublishToGitHub(result, resolved, config, run.telemetry, { diff --git a/src/pipeline/verifier.ts b/src/pipeline/verifier.ts index c1e9937..0ac7665 100644 --- a/src/pipeline/verifier.ts +++ b/src/pipeline/verifier.ts @@ -7,6 +7,7 @@ import type { TelemetryRecorder } from "../telemetry/telemetry-recorder.js"; import type { CandidateFinding, CodegenieConfig, + EvalVerificationRecord, PacketReviewResult, RepositoryTools, ReviewPacket, @@ -141,6 +142,12 @@ type CandidateGateDecision = | { outcome: "suppress"; reason: string; facts: VerificationGateFacts } | { outcome: "schedule"; reason: string; lane: VerificationLane; facts: VerificationGateFacts }; +export type ReconstructedVerificationGateDecision = { + candidate: CandidateFinding; + anchorStripped: boolean; + decision: CandidateGateDecision; +}; + export async function verifyFindings( input: { packetResults: PacketReviewResult[]; packets: ReviewPacket[] }, tools: RepositoryTools, @@ -495,6 +502,101 @@ function applyVerificationVerdict(candidate: CandidateFinding, verdict: Verifica return applyVerdictIntentAssessment(applyVerdictAnchor(revised, verdict), verdict); } +export function reconstructVerifiedFindingsFromArtifacts( + candidates: CandidateFinding[], + records: EvalVerificationRecord[], + packets: ReviewPacket[], + diff: UnifiedDiff +): CandidateFinding[] { + const clusteredById = new Map(reconstructVerifierCandidatesFromArtifacts(candidates, records, packets, diff) + .map((candidate) => [candidate.id, candidate])); + return records.flatMap((record) => { + if (!("verdict" in record) || record.verdict.verificationIncomplete === true || + record.verdict.verdict === "reject" || record.verdict.verdict === "incomplete") { + return []; + } + const candidate = clusteredById.get(record.candidateId); + return candidate === undefined ? [] : [applyVerificationVerdict(candidate, record.verdict)]; + }); +} + +export function reconstructVerifierCandidatesFromArtifacts( + candidates: CandidateFinding[], + records: EvalVerificationRecord[], + packets: ReviewPacket[], + diff: UnifiedDiff +): CandidateFinding[] { + const packetsById = new Map(packets.map((packet) => [packet.id, packet])); + const recordsById = new Map(records.map((record) => [record.candidateId, record])); + const quietTelemetry = { event: () => undefined } as unknown as TelemetryRecorder; + const preGated = candidates.map((candidate) => + preGateAnchor(candidate, packetsById.get(candidate.producedBy.packetId), diff, quietTelemetry).candidate + ).filter((candidate) => recordsById.get(candidate.id)?.gateDecision !== "suppressed"); + const clustered = clusterCandidates(preGated, quietTelemetry); + return clustered.all; +} + +export function reconstructDuplicateVerificationVerdict( + duplicate: CandidateFinding, + representativeVerdict: VerificationVerdict +): VerificationVerdict { + return { + ...representativeVerdict, + candidateId: duplicate.id, + ...(representativeVerdict.finalFinding !== undefined + ? { finalFinding: applyVerificationVerdict(duplicate, representativeVerdict) } + : {}) + }; +} + +export function reconstructVerificationGateFactsFromArtifacts( + candidate: CandidateFinding, + packet: ReviewPacket | undefined, + diff: UnifiedDiff +): VerificationGateFacts { + const quietTelemetry = { event: () => undefined } as unknown as TelemetryRecorder; + return candidateGateFacts(preGateAnchor(candidate, packet, diff, quietTelemetry).candidate); +} + +export function reconstructVerificationGateDecisionFromArtifacts( + candidate: CandidateFinding, + packet: ReviewPacket | undefined, + diff: UnifiedDiff, + config: CodegenieConfig +): ReconstructedVerificationGateDecision { + const quietTelemetry = { event: () => undefined } as unknown as TelemetryRecorder; + const preGated = preGateAnchor(candidate, packet, diff, quietTelemetry); + return { + candidate: preGated.candidate, + anchorStripped: preGated.anchorStripped, + decision: gateCandidate(preGated.candidate, config) + }; +} + +export function reconstructGatedVerifierCandidatesFromArtifacts( + candidates: CandidateFinding[], + packets: ReviewPacket[], + diff: UnifiedDiff, + config: CodegenieConfig +): { + evaluations: ReconstructedVerificationGateDecision[]; + candidates: CandidateFinding[]; +} { + const packetsById = new Map(packets.map((packet) => [packet.id, packet])); + const evaluations = candidates.map((candidate) => reconstructVerificationGateDecisionFromArtifacts( + candidate, + packetsById.get(candidate.producedBy.packetId), + diff, + config + )); + const quietTelemetry = { event: () => undefined } as unknown as TelemetryRecorder; + const clustered = clusterCandidates( + evaluations.flatMap((evaluation) => evaluation.decision.outcome === "schedule" ? [evaluation.candidate] : []), + quietTelemetry + ); + return { evaluations, candidates: clustered.all }; +} + function applyFindingRevision(candidate: CandidateFinding, revision: CandidateFinding): CandidateFinding { const { clusterId: _revisionClusterId, duplicateOf: _revisionDuplicateOf, ...revisionFields } = revision; return { diff --git a/src/telemetry/run-artifacts.ts b/src/telemetry/run-artifacts.ts index 313f36e..9baa89d 100644 --- a/src/telemetry/run-artifacts.ts +++ b/src/telemetry/run-artifacts.ts @@ -197,7 +197,7 @@ type TelemetryStageSummary = { schemaRecovery: SchemaRecoveryCounters; }; -type Stage7SchemaRepairSummary = { +export type Stage7SchemaRepairSummary = { candidateInvalidSubmits: number; noFindingInvalidSubmits: number; cleanupAttempted: number; @@ -214,7 +214,7 @@ type Stage7SchemaRepairSummary = { actualRepairPromptChars: number; }; -type LogOverflowSummary = { +export type LogOverflowSummary = { droppedDebugInfo: number; droppedWarnError: number; }; @@ -274,7 +274,7 @@ type ArtifactManifest = { artifacts: ArtifactManifestEntry[]; }; -type PipelineTotals = { +export type PipelineTotals = { filesChanged: number; hunks: number; packets: number; @@ -285,7 +285,7 @@ type PipelineTotals = { postedComments: number; }; -type PipelineTelemetrySummary = { +export type PipelineTelemetrySummary = { workers: { started: number; completed: number; @@ -498,6 +498,14 @@ class RunTelemetryImpl { this.runDirectory = runDir; touchCoreFiles(runDir); this.flushBuffers(); + if (this.logOverflow.droppedDebugInfo > 0 || this.logOverflow.droppedWarnError > 0) { + this.recordEvent({ + stage: 0, + level: "warn", + message: "telemetry_log_overflow", + data: { ...this.logOverflow } + }); + } this.recordPruneResult(pruneRuns(runsRoot, runDir, this.config.retainRuns)); return { runId: this.runId, runDir }; } @@ -1164,83 +1172,15 @@ class RunTelemetryImpl { } private updateStage7SchemaRepairSummaryFromEvent(event: TelemetryEvent): void { - if (event.stage !== 7) { - return; - } - const data = objectField(event.data); - const classification = typeof data?.classification === "string" ? data.classification : ""; - const payloadKind = typeof data?.payloadKind === "string" ? data.payloadKind : ""; - if (event.message === "stage7_schema_repair_attempted") { - if (payloadKind === "no_findings" || classification === "empty_no_findings_missing_fields") { - this.stage7SchemaRepairSummary.noFindingInvalidSubmits += 1; - } else { - this.stage7SchemaRepairSummary.candidateInvalidSubmits += 1; - } - return; - } - if (event.message === "stage7_schema_cleanup_attempted") { - this.stage7SchemaRepairSummary.cleanupAttempted += 1; - return; - } - if (event.message === "stage7_schema_cleanup_recovered") { - this.stage7SchemaRepairSummary.cleanupRecovered += 1; - return; - } - if (event.message === "stage7_schema_cleanup_rejected") { - this.stage7SchemaRepairSummary.cleanupRejected += 1; - return; - } - if (event.message === "stage7_schema_compact_repair_scheduled") { - this.stage7SchemaRepairSummary.compactRepairScheduled += 1; - const promptChars = numberField(data?.repairPromptChars); - this.stage7SchemaRepairSummary.repairPromptChars += promptChars; - this.stage7SchemaRepairSummary.compactRepairPromptChars += promptChars; - return; - } - if (event.message === "schema_repair_scheduled") { - const replaceConversation = data?.replaceConversation === true; - if (!replaceConversation) { - this.stage7SchemaRepairSummary.appendRepairScheduled += 1; - const promptChars = numberField(data?.repairPromptChars); - this.stage7SchemaRepairSummary.repairPromptChars += promptChars; - this.stage7SchemaRepairSummary.appendRepairPromptChars += promptChars; - } - return; - } - if (event.message === "stage7_schema_repair_recovered") { - if (classification === "schema_valid_after_retry") { - this.stage7SchemaRepairSummary.repairRecovered += 1; - } - return; - } - if (event.message === "stage7_schema_repair_failed") { - this.stage7SchemaRepairSummary.repairFailed += 1; - } + updateStage7SchemaRepairFromEvent(this.stage7SchemaRepairSummary, event); } private updateStage7SchemaRepairSummaryFromModelCall(record: LlmCallRecord): void { - if (record.stage !== 7 || record.kind !== "repair") { - return; - } - this.stage7SchemaRepairSummary.actualRepairCalls += 1; - this.stage7SchemaRepairSummary.actualRepairPromptChars += record.promptChars; + updateStage7SchemaRepairFromModelCall(this.stage7SchemaRepairSummary, record); } private updatePipelineSummaryFromEvent(event: TelemetryEvent): void { - if (event.message !== "pipeline_metrics" || !event.data || typeof event.data !== "object") { - return; - } - const data = event.data as Record; - mergePipelineTotals(this.pipelineTotals, objectField(data.totals)); - mergePipelineWorkers(this.pipelineSummary.workers, objectField(data.workers)); - mergePipelinePackets(this.pipelineSummary.packets, objectField(data.packets)); - mergePipelineLenses(this.pipelineSummary.lenses, objectField(data.lenses)); - mergePipelineCoverage(this.pipelineSummary.coverage, objectField(data.coverage)); - mergePipelineCandidates(this.pipelineSummary.candidates, objectField(data.candidates)); - mergePipelineVerdicts(this.pipelineSummary.verdicts, objectField(data.verdicts)); - mergePipelineDedup(this.pipelineSummary.dedup, objectField(data.dedup)); - mergePipelineFinalSelection(this.pipelineSummary.finalSelection, objectField(data.finalSelection)); - mergePipelinePosting(this.pipelineSummary.posting, objectField(data.posting)); + updatePipelineSummaryFromEvent(this.pipelineTotals, this.pipelineSummary, event); } private runReviewMetadata(): RunReviewArtifactMetadata { @@ -1660,6 +1600,97 @@ function emptyStage7SchemaRepairSummary(): Stage7SchemaRepairSummary { }; } +function updateStage7SchemaRepairFromEvent(summary: Stage7SchemaRepairSummary, event: TelemetryEvent): void { + if (event.stage !== 7) { + return; + } + const data = objectField(event.data); + const classification = typeof data?.classification === "string" ? data.classification : ""; + const payloadKind = typeof data?.payloadKind === "string" ? data.payloadKind : ""; + if (event.message === "stage7_schema_repair_attempted") { + if (payloadKind === "no_findings" || classification === "empty_no_findings_missing_fields") { + summary.noFindingInvalidSubmits += 1; + } else { + summary.candidateInvalidSubmits += 1; + } + } else if (event.message === "stage7_schema_cleanup_attempted") { + summary.cleanupAttempted += 1; + } else if (event.message === "stage7_schema_cleanup_recovered") { + summary.cleanupRecovered += 1; + } else if (event.message === "stage7_schema_cleanup_rejected") { + summary.cleanupRejected += 1; + } else if (event.message === "stage7_schema_compact_repair_scheduled") { + summary.compactRepairScheduled += 1; + const promptChars = numberField(data?.repairPromptChars); + summary.repairPromptChars += promptChars; + summary.compactRepairPromptChars += promptChars; + } else if (event.message === "schema_repair_scheduled" && data?.replaceConversation !== true) { + summary.appendRepairScheduled += 1; + const promptChars = numberField(data?.repairPromptChars); + summary.repairPromptChars += promptChars; + summary.appendRepairPromptChars += promptChars; + } else if (event.message === "stage7_schema_repair_recovered" && classification === "schema_valid_after_retry") { + summary.repairRecovered += 1; + } else if (event.message === "stage7_schema_repair_failed") { + summary.repairFailed += 1; + } +} + +function updateStage7SchemaRepairFromModelCall(summary: Stage7SchemaRepairSummary, record: LlmCallRecord): void { + if (record.stage === 7 && record.kind === "repair") { + summary.actualRepairCalls += 1; + summary.actualRepairPromptChars += record.promptChars; + } +} + +function updatePipelineSummaryFromEvent( + totals: PipelineTotals, + summary: PipelineTelemetrySummary, + event: TelemetryEvent +): void { + if (event.message !== "pipeline_metrics" || !event.data || typeof event.data !== "object") { + return; + } + const data = event.data as Record; + mergePipelineTotals(totals, objectField(data.totals)); + mergePipelineWorkers(summary.workers, objectField(data.workers)); + mergePipelinePackets(summary.packets, objectField(data.packets)); + mergePipelineLenses(summary.lenses, objectField(data.lenses)); + mergePipelineCoverage(summary.coverage, objectField(data.coverage)); + mergePipelineCandidates(summary.candidates, objectField(data.candidates)); + mergePipelineVerdicts(summary.verdicts, objectField(data.verdicts)); + mergePipelineDedup(summary.dedup, objectField(data.dedup)); + mergePipelineFinalSelection(summary.finalSelection, objectField(data.finalSelection)); + mergePipelinePosting(summary.posting, objectField(data.posting)); +} + +export function reconstructRunTelemetryDerivedEvidence( + events: TelemetryEvent[], + modelCalls: LlmCallRecord[] +): { + pipelineTotals: PipelineTotals; + pipeline: PipelineTelemetrySummary; + stage7SchemaRepair: Stage7SchemaRepairSummary; + logOverflow: LogOverflowSummary; +} { + const pipelineTotals = emptyPipelineTotals(); + const pipeline = emptyPipelineTelemetrySummary(); + const stage7SchemaRepair = emptyStage7SchemaRepairSummary(); + const logOverflow: LogOverflowSummary = { droppedDebugInfo: 0, droppedWarnError: 0 }; + for (const event of events) { + updatePipelineSummaryFromEvent(pipelineTotals, pipeline, event); + updateStage7SchemaRepairFromEvent(stage7SchemaRepair, event); + if (event.message === "telemetry_log_overflow") { + logOverflow.droppedDebugInfo = numberField(event.data?.droppedDebugInfo); + logOverflow.droppedWarnError = numberField(event.data?.droppedWarnError); + } + } + for (const record of modelCalls) { + updateStage7SchemaRepairFromModelCall(stage7SchemaRepair, record); + } + return { pipelineTotals, pipeline, stage7SchemaRepair, logOverflow }; +} + function emptyToolBucket(): ToolBucket { return { count: 0, diff --git a/src/types.ts b/src/types.ts index a21679d..8d9875d 100644 --- a/src/types.ts +++ b/src/types.ts @@ -1260,12 +1260,23 @@ export type EvalRunInfo = { }; repo?: { root: string; baseSha?: string; headSha?: string; mergeBase?: string }; reviewRunId?: string; + invocation?: { + id: string; + caseIndex: number; + manifest: string; + }; codegenieRuntime?: CodegenieRuntimeProvenance; cache: { enabled: boolean; source: "cli" | "case" | "config"; dir?: string }; effectiveConfig?: { review: { concurrency: number; timeoutMs: number; + verify: boolean; + minSeverity?: Severity; + maxFindings: number; + softCommentCap: number; + minConfidence: Confidence; + minInlineConfidence: Confidence; packSameFileHunks: boolean; packedToolBudgetMode: PackedToolBudgetMode; maxBudgetTokens?: number; @@ -1282,6 +1293,29 @@ export type EvalRunInfo = { score: EvalScore; }; +export type EvalInvocationManifest = { + schemaVersion: 1; + invocationId: string; + suiteDir: string; + status: "running" | "complete"; + startedAt: string; + completedAt?: string; + cases: Array<{ + caseIndex: number; + caseName: string; + caseHash: string; + caseFile: string; + }>; + runs: Array<{ + caseIndex: number; + caseName: string; + caseHash: string; + runNumber: number; + logsRoot: string; + runPath: string; + }>; +}; + export type EvalCaseResult = { caseName: string; runDir: string; @@ -1612,6 +1646,7 @@ export type ToolCallRecord = { glob?: string; source?: string; contextMode?: string; + maxResults?: number; }; backend: ToolBackend; precision: ToolPrecision; diff --git a/tests/evals.test.ts b/tests/evals.test.ts index a5e7c33..4da84cf 100644 --- a/tests/evals.test.ts +++ b/tests/evals.test.ts @@ -9,10 +9,12 @@ import { compareToPrevious, renderEvalCompareText } from "../src/evals/eval-comp import { executeEvalCommand, renderCaseResult, runEvalCommand } from "../src/evals/eval-command.js"; import { loadEvalSuite, replayFromArtifacts, runEvalCase } from "../src/evals/eval-runner.js"; import { aggregateRepeatScores, assignExpectations, matchExpectation, scoreEvalRun } from "../src/evals/eval-scoring.js"; +import { loadEvalRuns } from "../scripts/packet-packing-report.js"; import type { CandidateFinding, EvalArtifacts, EvalCase, + EvalInvocationManifest, EvalRunInfo, EvalRunMetrics, EvalScore, @@ -1638,6 +1640,7 @@ describe("eval command fixture suite", () => { }); let output = ""; + const suite = await loadEvalSuite(suiteDir); const exitCode = await runEvalCommand({ evalDir: suiteDir, cache: false }, defaultConfig, { writeOutput: (text) => { output += text; @@ -1647,6 +1650,115 @@ describe("eval command fixture suite", () => { expect(exitCode).toBe(0); expect(output).toContain("Suite: 8 passed, 0 failed, 0 errored"); expect(existsSync(path.join(suiteDir, "logs", "1", "fixture-repo", ".git"))).toBe(true); + const invocationFiles = readdirSync(path.join(suiteDir, "logs", "invocations")); + expect(invocationFiles).toHaveLength(1); + const invocationFile = invocationFiles[0]!; + const manifest = JSON.parse(readFileSync(path.join(suiteDir, "logs", "invocations", invocationFile), "utf8")) as EvalInvocationManifest; + expect(manifest).toMatchObject({ + schemaVersion: 1, + suiteDir, + status: "complete", + completedAt: expect.any(String), + cases: suite.cases.map((entry, caseIndex) => ({ + caseIndex, + caseName: entry.evalCase.name, + caseHash: entry.caseHash, + caseFile: entry.file + })), + runs: suite.cases.map((entry, caseIndex) => ({ + caseIndex, + caseName: entry.evalCase.name, + caseHash: entry.caseHash, + runNumber: caseIndex + 1, + logsRoot: path.join(suiteDir, "logs"), + runPath: String(caseIndex + 1) + })) + }); + for (const [caseIndex, run] of manifest.runs.entries()) { + const info = JSON.parse(readFileSync(path.join(suiteDir, "logs", String(run.runNumber), "info.json"), "utf8")) as EvalRunInfo; + expect(info.invocation).toEqual({ + id: manifest.invocationId, + caseIndex, + manifest: path.join("invocations", invocationFile) + }); + } + const reportRuns = await loadEvalRuns(path.join(suiteDir, "logs")); + expect(reportRuns.map((run) => run.info.invocation?.caseIndex)).toEqual(suite.cases.map((_, index) => index)); + + const interruptedManifest: EvalInvocationManifest = { + schemaVersion: 1, + invocationId: "historical-interrupted", + suiteDir, + status: "running", + startedAt: "2000-01-01T00:00:00.000Z", + cases: [{ caseIndex: 0, caseName: "interrupted", caseHash: "interrupted", caseFile: suite.cases[0]!.file }], + runs: [{ + caseIndex: 0, + caseName: "interrupted", + caseHash: "interrupted", + runNumber: 999, + logsRoot: path.join(suiteDir, "logs"), + runPath: "999" + }] + }; + writeFileSync( + path.join(suiteDir, "logs", "invocations", `${interruptedManifest.invocationId}.json`), + `${JSON.stringify(interruptedManifest, null, 2)}\n` + ); + mkdirSync(path.join(suiteDir, "logs", "999")); + expect((await loadEvalRuns(path.join(suiteDir, "logs"), "latest")).map((run) => run.runNumber)).toEqual(manifest.runs.map((run) => run.runNumber)); + expect((await loadEvalRuns(path.join(suiteDir, "logs"), manifest.invocationId)).map((run) => run.runNumber)).toEqual(manifest.runs.map((run) => run.runNumber)); + + const firstRecordedRun = manifest.runs[0]!; + const firstInfoPath = path.join(suiteDir, "logs", String(firstRecordedRun.runNumber), "info.json"); + const mutuallyAlteredInfo = JSON.parse(readFileSync(firstInfoPath, "utf8")) as EvalRunInfo; + mutuallyAlteredInfo.caseHash = "mutually-altered-hash"; + mutuallyAlteredInfo.caseSnapshot.should_find![0]!.id = "mutually-altered-expectation"; + mutuallyAlteredInfo.score.expectationResults[0]!.expectationId = "mutually-altered-expectation"; + manifest.cases[0]!.caseHash = "mutually-altered-hash"; + manifest.runs[0]!.caseHash = "mutually-altered-hash"; + writeFileSync(firstInfoPath, `${JSON.stringify(mutuallyAlteredInfo, null, 2)}\n`); + writeFileSync(path.join(suiteDir, "logs", "invocations", invocationFile), `${JSON.stringify(manifest, null, 2)}\n`); + await expect(loadEvalRuns(path.join(suiteDir, "logs"), manifest.invocationId)).rejects.toMatchObject({ + failures: [expect.objectContaining({ code: "declared_case_join" })] + }); + }, 60_000); + + it("records exact per-entry root ownership and loads one invocation across multiple log roots", async () => { + const root = mkdtempSync(path.join(tmpdir(), "codegenie-multi-root-invocation-")); + const suiteDir = path.join(root, "fixtures"); + const firstRoot = path.join(root, "logs-a"); + const secondRoot = path.join(root, "logs-b"); + cpSync(path.join(process.cwd(), "evals", "fixtures"), suiteDir, { + recursive: true, + filter: (source) => path.basename(source) !== ".git" && path.basename(source) !== "logs" && !source.includes(`${path.sep}.git${path.sep}`) + }); + const caseFiles = readdirSync(suiteDir).filter((entry) => /\.ya?ml$/u.test(entry)).sort(); + for (const [index, file] of caseFiles.entries()) { + const logsRoot = index < caseFiles.length / 2 ? firstRoot : secondRoot; + const target = path.join(suiteDir, file); + writeFileSync(target, `${readFileSync(target, "utf8").trimEnd()}\nlogs:\n dir: ${logsRoot}\n`); + } + + expect(await runEvalCommand({ evalDir: suiteDir, cache: false }, defaultConfig)).toBe(0); + const manifestFile = readdirSync(path.join(firstRoot, "invocations"))[0]!; + const firstManifest = JSON.parse(readFileSync(path.join(firstRoot, "invocations", manifestFile), "utf8")) as EvalInvocationManifest; + const secondManifestPath = path.join(secondRoot, "invocations", manifestFile); + expect(JSON.parse(readFileSync(secondManifestPath, "utf8"))).toEqual(firstManifest); + expect(new Set(firstManifest.runs.map((entry) => entry.logsRoot))).toEqual(new Set([firstRoot, secondRoot])); + expect(new Set(firstManifest.runs.map((entry) => `${entry.logsRoot}\0${entry.runPath}`)).size).toBe(caseFiles.length); + expect(new Set(firstManifest.runs.map((entry) => entry.runNumber)).size).toBeLessThan(caseFiles.length); + + const loaded = await loadEvalRuns(firstRoot, firstManifest.invocationId); + expect(loaded).toHaveLength(caseFiles.length); + expect(new Set(loaded.map((run) => path.dirname(run.runDir)))).toEqual(new Set([firstRoot, secondRoot])); + + const altered = structuredClone(firstManifest); + altered.runs.at(-1)!.logsRoot = firstRoot; + writeFileSync(secondManifestPath, `${JSON.stringify(altered, null, 2)}\n`); + await expect(loadEvalRuns(firstRoot, firstManifest.invocationId)).rejects.toMatchObject({ + failures: [expect.objectContaining({ code: "invocation_root_ownership" })] + }); }, 60_000); it("does not leak the invocation directory repo config into live cases", async () => { diff --git a/tests/packet-packing-report.test.ts b/tests/packet-packing-report.test.ts new file mode 100644 index 0000000..6502746 --- /dev/null +++ b/tests/packet-packing-report.test.ts @@ -0,0 +1,2236 @@ +import { mkdtemp, mkdir, readFile, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { afterEach, describe, expect, it, vi } from "vitest"; +import { + analyzeEvalCohort, + analyzeRegressionCohorts, + analyzeReplayComparison, + computeProductionEconomics, + finalizeReplayCleanup, + historicalFlagOffParityView, + reconstructEvidenceArtifacts, + runPacketPackingReportCli, + selectExplicitCohort, + validateRecordedDiffParity, + validatePacketPackingArtifact, + validateReplayArtifacts, + type CohortSelection, + type EvalCaseRunInput, + type EvalExecutionInput, + type ReplayAnalysisInput, + type ReportFailure +} from "../scripts/packet-packing-report.js"; +import { aggregateRepeatScores, scoreEvalRun } from "../src/evals/eval-scoring.js"; +import { defaultConfig } from "../src/config/schema.js"; +import { + reconstructComposerGroupsFromArtifacts, + reconstructComposerPolicyFromArtifacts +} from "../src/pipeline/composer.js"; +import { + reconstructVerificationGateFactsFromArtifacts, + reconstructVerifiedFindingsFromArtifacts +} from "../src/pipeline/verifier.js"; +import { createRunTelemetry, reconstructRunTelemetryDerivedEvidence } from "../src/telemetry/run-artifacts.js"; +import type { + CandidateFinding, + EvalArtifacts, + EvalCase, + EvalExpectationResult, + EvalInvocationManifest, + EvalRunInfo, + EvalScore, + FileFacts, + FinalFinding, + ReviewPacket, + ReviewPlan, + TelemetryEvent, + ToolBudget, + UnifiedDiff +} from "../src/types.js"; +import { sha256Hex } from "../src/util/hashing.js"; + +const tempDirs: string[] = []; + +afterEach(async () => { + const { rm } = await import("node:fs/promises"); + await Promise.all(tempDirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); +}); + +const standardBudget: ToolBudget = { + maxToolCalls: 4, + maxInvestigationRounds: 2, + maxResultChars: 10_000 +}; + +function packet( + id: string, + hunkIds: string[], + overrides: Partial = {} +): ReviewPacket { + const hunks = hunkIds.map((hunkId, index) => ({ + // Synthetic atoms and their packed form must describe the same source + // coordinates. Derive the coordinate from canonical hN fixture IDs so a + // separately constructed h2 does not collide with h1 at line 1. + ...(() => { + const match = /(\d+)$/u.exec(hunkId); + const sourceIndex = match === null ? index : Number(match[1]) - 1; + const line = sourceIndex * 100 + 1; + return { + oldStart: line, + newStart: line, + contentWithLineNumbers: `${"".padStart(4)} ${String(line).padStart(4)} +change${sourceIndex}`, + lines: [{ kind: "add" as const, content: `change${sourceIndex}`, newLine: line }], + changedNewLineNumbers: [line] + }; + })(), + hunkId, + oldLines: 1, + newLines: 1, + changedOldLineNumbers: [] + })); + return { + id, + dispatchRank: [0, -hunks.length], + kind: hunks.length === 1 ? "hunk" : "coalesced-hunks", + prSummary: "test", + path: "app.ts", + fileStatus: "modified", + isDeletedContent: false, + language: "typescript", + reviewPriority: "normal", + coverage: "normal", + reviewProfile: "standard", + lenses: ["core/code-review", "lang/typescript"], + hunks, + symbolFacts: [], + context: { path: "app.ts" }, + contextText: "context", + contextQuality: "full", + relevantTests: [], + surroundingContextHints: [], + labels: [], + attentionNotes: [], + relatedChangedContext: [], + toolBudget: structuredClone(standardBudget), + ...overrides + }; +} + +function diffForPackets(packets: ReviewPacket[]): UnifiedDiff { + const hunks = packets.flatMap((target) => target.hunks.map((hunk) => ({ + id: hunk.hunkId, + hunkHash: sha256Hex(hunk.hunkId), + path: target.path, + oldStart: hunk.oldStart, + oldLines: hunk.oldLines, + newStart: hunk.newStart, + newLines: hunk.newLines, + header: hunk.header ?? "", + lines: hunk.lines.map((line) => ({ + kind: line.kind, + content: line.content, + ...(line.oldLine === undefined ? {} : { oldLineNumber: line.oldLine }), + ...(line.newLine === undefined ? {} : { newLineNumber: line.newLine }) + })) + }))); + return { + files: [{ path: "app.ts", status: "modified", language: "typescript", hunks }] + }; +} + +function atomId(hunkIds: string[]): string { + return sha256Hex(`hunk-first\n${hunkIds.join("\n")}`); +} + +function packingEvent( + target: ReviewPacket, + atoms: ReviewPacket[], + overrides: Record = {} +): TelemetryEvent { + const profiles = atoms.map((atom) => atom.reviewProfile); + const profileFloor = profiles.includes("investigate") ? "investigate" : profiles.includes("standard") ? "standard" : "simple"; + return { + runId: "eval-run", + eventId: `event-${target.id}`, + timestamp: "2026-07-24T00:00:00.000Z", + stage: 6, + level: "info", + message: "same_file_atoms_packed", + file: target.path, + data: { + packetId: target.id, + atomIds: atoms.map((atom) => atomId(atom.hunks.map((hunk) => hunk.hunkId))), + standaloneProfiles: profiles, + sourceAtomCount: atoms.length, + hunkCount: target.hunks.length, + effectiveCoverage: target.coverage, + requestedLensSignature: JSON.stringify(["core/code-review"]), + capUsage: { + hunks: target.hunks.length, + maxHunks: 5, + patchChars: target.hunks.reduce((total, hunk) => total + hunk.contentWithLineNumbers.length, 0), + maxPatchChars: 12_000 + }, + derivedPackedProfile: target.reviewProfile, + profileFloor, + effectiveProfile: target.reviewProfile, + profileFloorApplied: false, + plannerLensesPreserved: true, + toolBudgetMode: "base", + baseToolBudget: standardBudget, + effectiveToolBudget: target.toolBudget, + ...overrides + } + }; +} + +function replayFixture(): ReplayAnalysisInput { + const first = packet("off-1", ["h1"]); + const second = packet("off-2", ["h2"]); + const combined = packet("on-1", ["h1", "h2"]); + const diff: UnifiedDiff = { + files: [{ + path: "app.ts", + status: "modified", + language: "typescript", + hunks: [ + { + id: "h1", + hunkHash: "hash1", + path: "app.ts", + oldStart: 1, + oldLines: 1, + newStart: 1, + newLines: 1, + header: "@@ -1 +1 @@", + lines: [{ kind: "add", content: "change0", newLineNumber: 1 }] + }, + { + id: "h2", + hunkHash: "hash2", + path: "app.ts", + oldStart: 101, + oldLines: 1, + newStart: 101, + newLines: 1, + header: "@@ -101 +101 @@", + lines: [{ kind: "add", content: "change1", newLineNumber: 101 }] + } + ] + }] + }; + const facts: FileFacts[] = [{ + path: "app.ts", + language: "typescript", + processingMode: "per-hunk", + testStatus: "source", + isGenerated: false, + isVendored: false, + isLockfile: false, + isBinary: false, + changedLines: 2, + hunkCount: 2, + labels: [], + reviewPriority: "normal", + reasons: [], + provenance: [] + }]; + const plan: ReviewPlan = { + diffUnderstanding: { declaredIntent: "test", inferredBehavior: "test" }, + coverage: [ + { hunkId: "h1", path: "app.ts", coverage: "normal", lenses: ["core/code-review"], surroundingContextHints: [], reason: "test" }, + { hunkId: "h2", path: "app.ts", coverage: "normal", lenses: ["core/code-review"], surroundingContextHints: [], reason: "test" } + ] + }; + return { + runId: "retained-run", + recordedPackets: [first, second], + offPackets: [first, second], + onPackets: [combined], + onEvents: [packingEvent(combined, [first, second])], + fileFacts: facts, + diff, + plan, + expectedRefs: { base: "a".repeat(40), head: "b".repeat(40) }, + actualRefs: { base: "a".repeat(40), head: "b".repeat(40) } + }; +} + +function failureCodes(failures: ReportFailure[]): string[] { + return failures.map((entry) => entry.code); +} + +describe("packet packing replay analysis", () => { + it("reports packet shape, profile provenance, caps, and dispatch movement", () => { + const report = analyzeReplayComparison(replayFixture()); + expect(report.failures).toEqual([]); + expect(report).toMatchObject({ + offPackets: 2, + onPackets: 1, + sourceAtoms: 2, + packedMultiAtomPackets: 1, + reviewableHunks: 2, + newCoveragePromotions: 0, + capViolations: 0, + effectiveProfileDowngrades: 0, + effectiveBudgetDowngrades: 0, + invalidDispatchRanks: 0 + }); + expect(report.packetMembership[0]?.atomIds).toEqual([atomId(["h1"]), atomId(["h2"])]); + }); + + it("fails closed on stale refs and flag-off artifact drift", () => { + const input = replayFixture(); + input.actualRefs = { ...input.actualRefs!, head: "c".repeat(40) }; + input.recordedPackets[0] = { ...input.recordedPackets[0]!, contextText: "recorded-only" }; + const report = analyzeReplayComparison(input); + expect(failureCodes(report.failures)).toEqual(expect.arrayContaining(["stale_replay_refs", "flag_off_parity"])); + }); + + it("never includes proprietary source text in parity failures and rejects replay model calls", () => { + const proprietary = "PROPRIETARY_SOURCE_NEVER_REPORT_THIS"; + const input = replayFixture(); + input.recordedPackets[0] = { ...input.recordedPackets[0]!, contextText: proprietary }; + input.modelCallsObserved = 1; + const report = analyzeReplayComparison(input); + expect(failureCodes(report.failures)).toEqual(expect.arrayContaining(["flag_off_parity", "replay_model_call"])); + const serialized = JSON.stringify(report); + expect(serialized).not.toContain(proprietary); + expect(serialized).toContain("sha256"); + }); + + it("fails closed when replay cleanup cannot remove its worktree", async () => { + let tempCleanupCalled = false; + await expect(finalizeReplayCleanup( + "retained-run", + true, + async () => { throw new Error("worktree busy"); }, + async () => { tempCleanupCalled = true; }, + async () => false + )).rejects.toThrow(/failed to remove replay worktree/u); + expect(tempCleanupCalled).toBe(true); + }); + + it("unconditionally cleans and verifies a partially-added worktree", async () => { + let removalAttempted = false; + let tempCleanupCalled = false; + await expect(finalizeReplayCleanup( + "partial-worktree-add", + true, + async () => { removalAttempted = true; throw new Error("not registered"); }, + async () => { tempCleanupCalled = true; }, + async () => true + )).resolves.toBeUndefined(); + expect(removalAttempted).toBe(true); + expect(tempCleanupCalled).toBe(true); + }); + + it("admits only the pinned pre-Plan100 dispatch-rank artifact migration", () => { + const input = replayFixture(); + input.runId = "20260724-150405-fe1548ae"; + input.recordedPackets = structuredClone(input.recordedPackets); + for (const packet of input.recordedPackets) { + delete (packet as Partial).dispatchRank; + } + const parity = historicalFlagOffParityView(input.runId, input.recordedPackets, input.offPackets); + input.flagOffParityPackets = parity.packets; + input.flagOffParityMigrations = parity.migrations; + const report = analyzeReplayComparison(input); + expect(failureCodes(report.failures)).not.toContain("flag_off_parity"); + expect(report.flagOffParityDifferences.historicalMigrations).toEqual([ + expect.objectContaining({ code: "pre_plan100_dispatch_rank_schema", packets: 2 }) + ]); + + input.recordedPackets[0]!.contextText = "unrecognized historical drift"; + expect(failureCodes(analyzeReplayComparison(input).failures)).toContain("flag_off_parity"); + }); + + it("admits the legacy diff schema for exactly the three retained run IDs", () => { + const rebuilt = replayFixture().diff; + const legacy = structuredClone(rebuilt); + for (const hunk of legacy.files.flatMap((file) => file.hunks)) { + delete (hunk as Partial).hunkHash; + } + expect(() => validateRecordedDiffParity("20260724-135818-740d73f2", legacy, rebuilt)).not.toThrow(); + expect(() => validateRecordedDiffParity("20260724-000000-unknown000", legacy, rebuilt)).toThrow(/not allowlisted/u); + }); + + it("fails closed on hunk loss, atom reorder, caps, and coverage promotion", () => { + const input = replayFixture(); + const reversed = packet("bad-on", ["h2", "h1"], { coverage: "deep" }); + input.onPackets = [reversed]; + input.onEvents = [packingEvent(reversed, [input.offPackets[0]!, input.offPackets[1]!], { + effectiveCoverage: "deep", + capUsage: { hunks: 6, maxHunks: 5, patchChars: 12_001, maxPatchChars: 12_000 } + })]; + const report = analyzeReplayComparison(input); + expect(JSON.stringify(report)).not.toContain("inspect critical boundary"); + expect(failureCodes(report.failures)).toEqual(expect.arrayContaining([ + "source_order", + "atom_split_or_reorder", + "packet_cap", + "coverage_promotion" + ])); + + const lost = replayFixture(); + lost.onPackets = [lost.onPackets[0] = packet("lost", ["h1"])]; + lost.onEvents = [packingEvent(lost.onPackets[0]!, [lost.offPackets[0]!])]; + expect(failureCodes(analyzeReplayComparison(lost).failures)).toContain("hunk_bijection"); + }); + + it("derives eligible-packet patch size from the diff and enforces complete packet source order", () => { + const falseCap = replayFixture(); + const falseCapEvent = falseCap.onEvents[0]!; + falseCapEvent.data = { + ...falseCapEvent.data, + capUsage: { ...(falseCapEvent.data!.capUsage as object), patchChars: 1 } + }; + const capReport = analyzeReplayComparison(falseCap); + expect(failureCodes(capReport.failures)).toContain("packet_cap"); + expect(capReport.distribution.maxEligiblePackingPatchChars).toBeGreaterThan(1); + + const reordered = replayFixture(); + reordered.onPackets = [reordered.offPackets[1]!, reordered.offPackets[0]!]; + reordered.onEvents = []; + expect(failureCodes(analyzeReplayComparison(reordered).failures)).toContain("source_packet_order"); + }); + + it("fails closed on lens/focus/context/profile/budget/rank regressions", () => { + const input = replayFixture(); + input.offPackets[0] = { + ...input.offPackets[0]!, + reviewPriority: "high", + coverage: "deep", + reviewProfile: "investigate", + attentionNotes: ["inspect critical boundary"], + toolBudget: { ...standardBudget, maxToolCalls: 6, maxResultChars: 12_000 } + }; + input.recordedPackets = structuredClone(input.offPackets); + input.onPackets[0] = { + ...input.onPackets[0]!, + coverage: "deep", + reviewProfile: "standard", + lenses: ["core/code-review"], + contextQuality: "path_only", + dispatchRank: [0, -99] + }; + input.onEvents = [packingEvent(input.onPackets[0]!, input.offPackets, { + standaloneProfiles: ["investigate", "standard"], + effectiveCoverage: "deep", + profileFloor: "investigate", + effectiveProfile: "standard", + baseToolBudget: standardBudget, + effectiveToolBudget: { ...standardBudget, maxToolCalls: 2 } + })]; + const report = analyzeReplayComparison(input); + expect(failureCodes(report.failures)).toEqual(expect.arrayContaining([ + "planner_lens_drop", + "high_priority_focus_omitted", + "deep_context_downgrade", + "effective_profile_downgrade", + "effective_budget_downgrade", + "invalid_dispatch_rank" + ])); + }); + + it("rejects motivating counts outside the bounded 74/75/76 reconciliation", () => { + const input = replayFixture(); + input.runId = "20260724-184952-dca8d870"; + const report = analyzeReplayComparison(input); + expect(failureCodes(report.failures)).toEqual(expect.arrayContaining(["motivating_off_count", "unpermitted_packet_count"])); + }); + + it("writes a fail-closed report for corrupt retained-run artifacts", async () => { + const root = await mkdtemp(path.join(tmpdir(), "packet-report-corrupt-")); + tempDirs.push(root); + const run = path.join(root, "bad-run"); + await mkdir(path.join(run, "stages", "01-input"), { recursive: true }); + await writeFile(path.join(run, "stages", "01-input", "resolved-input.json"), "{not-json\n"); + const output = path.join(root, "report.json"); + const exitCode = await runPacketPackingReportCli(["replay", "--repo", process.cwd(), "--run", run, "--output", output]); + expect(exitCode).toBe(1); + const report = JSON.parse(await readFile(output, "utf8")) as { failures: ReportFailure[] }; + expect(failureCodes(report.failures)).toContain("corrupt_artifact"); + }); + + it("rejects unknown artifact fields and disagreement across replay refs", async () => { + const root = await mkdtemp(path.join(tmpdir(), "packet-report-schema-")); + tempDirs.push(root); + const run = path.join(root, "strict-run"); + await mkdir(path.join(run, "stages", "01-input"), { recursive: true }); + await writeFile(path.join(run, "stages", "01-input", "resolved-input.json"), JSON.stringify({ + mode: "head", + repoRoot: process.cwd(), + baseRef: "a".repeat(40), + headRef: "b".repeat(40), + commits: [], + rawDiffChars: 0, + unexpected: "PROPRIETARY_ARTIFACT_VALUE" + })); + const output = path.join(root, "report.json"); + expect(await runPacketPackingReportCli(["replay", "--repo", process.cwd(), "--run", run, "--output", output])).toBe(1); + const schemaReport = JSON.parse(await readFile(output, "utf8")) as { failures: ReportFailure[] }; + expect(failureCodes(schemaReport.failures)).toContain("corrupt_artifact_schema"); + expect(await readFile(output, "utf8")).not.toContain("PROPRIETARY_ARTIFACT_VALUE"); + + const replay = replayFixture(); + expect(() => validateReplayArtifacts( + "retained-run", + { + mode: "head", + repoRoot: process.cwd(), + baseRef: "a".repeat(40), + headRef: "b".repeat(40), + commits: [], + rawDiffChars: 0 + }, + replay.diff, + [{ path: "app.ts", action: "keep", reason: "test", provenance: [] }], + replay.fileFacts, + replay.plan, + { + runId: "retained-run", + mode: "head", + depth: "normal", + target: { baseRef: "c".repeat(40), headRef: "b".repeat(40) }, + commits: [], + policyFilesChanged: [], + hunkIndex: [], + files: [], + directories: [], + filterSummary: { keptFiles: 1, skippedFiles: 0, skipped: [] }, + lenses: [], + totals: { files: 1, keptFiles: 1, hunks: 2, addedLines: 2, deletedLines: 0 }, + compaction: {} as never + }, + { runId: "retained-run" } + )).toThrow(/disagree on base\/head refs/u); + }); + + it("fails closed at every packet/eval telemetry artifact schema boundary", () => { + for (const kind of [ + "packet", "event", "model-call", "tool-call", "coverage", "eval-info", "attention", "human-attention", + "budget-summary", "cost-profile", "model-summary", "tool-summary", "run-summary", "telemetry-summary" + ] as const) { + expect(() => validatePacketPackingArtifact(kind, { unexpected: true }), kind).toThrow(/invalid .* artifact/u); + } + + const target = packet("strict-packet", ["h1", "h2"]); + const event = packingEvent(target, [packet("strict-a", ["h1"]), packet("strict-b", ["h2"])]); + (event.data as Record).unexpected = true; + expect(() => validatePacketPackingArtifact("event", event)).toThrow(/invalid event artifact/u); + + const nestedCase = evalRun("strict", "A", 1, [execution("A", 1)]).info; + (nestedCase.caseSnapshot.review as Record).unexpected = true; + expect(() => validatePacketPackingArtifact("eval-info", nestedCase)).toThrow(/invalid eval-info artifact/u); + + const toolCall = { + runId: "run", + toolCallId: "tool-call", + timestamp: "2026-07-24T00:00:00.000Z", + stage: 7, + initiator: "model", + tool: "read_file", + args: { path: "app.ts", maxResults: 40 }, + backend: "text", + precision: "exact", + degraded: false, + resultChars: 1, + durationMs: 1, + status: "ok" + }; + expect(() => validatePacketPackingArtifact("tool-call", toolCall)).not.toThrow(); + expect(() => validatePacketPackingArtifact("tool-call", { + ...toolCall, + args: { ...toolCall.args, unexpected: "nested" } + })).toThrow(/invalid tool-call artifact/u); + }); + + it("accepts current producer-shaped model, tool, cost, run, and telemetry summaries", async () => { + const repoRoot = await mkdtemp(path.join(tmpdir(), "packet-report-summary-producer-")); + tempDirs.push(repoRoot); + const run = createRunTelemetry({ + telemetryConfig: { ...defaultConfig.telemetry, enabled: true, logLevel: "debug" }, + idFactory: () => "20260725-000000-report-summary" + }); + const attached = await run.attachRunDirectory(repoRoot); + run.recorder.event({ stage: 10, level: "info", message: "stage_started", cacheStatus: "write" }); + run.recorder.recordModelCall({ + callId: "summary-call", stage: 10, role: "composer", model: "model", provider: "provider", kind: "initial", attempt: 1, + promptChars: 12, promptHash: "prompt", outputChars: 5, outputHash: "output", inputTokens: 10, outputTokens: 2, + reasoningTokens: 1, totalTokens: 13, costUSD: 0.01, durationMs: 10, cacheStatus: "miss", schemaValid: false, + stopReason: "submit", status: "schema_invalid", errorCode: "llm_schema_invalid" + }); + run.recorder.event({ + stage: 10, level: "info", message: "schema_invalid_submit_recovered", + data: { submitTool: "submit_composition", invalidSubmitCallCount: 1 } + }); + run.recorder.recordToolCall({ + stage: 7, initiator: "harness", tool: "read_file", args: { path: "app.ts" }, backend: "text", precision: "exact", + degraded: false, cacheStatus: "write", backendExecuted: true, resultChars: 10, durationMs: 2, status: "ok" + }); + run.recorder.event({ stage: 10, level: "info", message: "stage_completed" }); + await run.finalize({ status: "completed_full", exitCode: 0 }); + for (const [kind, filename] of [ + ["model-summary", "stages/00-run/model-calls-summary.json"], + ["tool-summary", "stages/00-run/tool-calls-summary.json"], + ["cost-profile", "stages/00-run/cost-profile.json"], + ["run-summary", "run.json"], + ["telemetry-summary", "telemetry.json"] + ] as const) { + const value = JSON.parse(await readFile(path.join(attached.runDir, filename), "utf8")) as unknown; + expect(() => validatePacketPackingArtifact(kind, value), kind).not.toThrow(); + } + }); + + it("persists raw evidence for nonzero buffered log overflow", async () => { + const repoRoot = await mkdtemp(path.join(tmpdir(), "packet-report-log-overflow-")); + tempDirs.push(repoRoot); + const run = createRunTelemetry({ + telemetryConfig: { ...defaultConfig.telemetry, enabled: true, logLevel: "debug" }, + idFactory: () => "20260725-000000-log-overflow" + }); + for (let index = 0; index < 1_010; index += 1) { + run.logger.debug({ runId: run.recorder.runId, stage: 0, event: "buffered", message: `buffered ${index}` }); + } + const attached = await run.attachRunDirectory(repoRoot); + await run.finalize({ status: "completed_full", exitCode: 0 }); + const events = (await readFile(path.join(attached.runDir, "events.jsonl"), "utf8")) + .trim().split("\n").filter(Boolean).map((line) => JSON.parse(line) as TelemetryEvent); + const derived = reconstructRunTelemetryDerivedEvidence(events, []); + const summary = JSON.parse(await readFile(path.join(attached.runDir, "telemetry.json"), "utf8")) as { + logs: { bufferedOverflow: { droppedDebugInfo: number; droppedWarnError: number } }; + }; + expect(derived.logOverflow.droppedDebugInfo).toBeGreaterThan(0); + expect(derived.logOverflow).toEqual(summary.logs.bufferedOverflow); + }); +}); + +function evalCase(family: string, arm: "A" | "B" | "C", repeat: number): EvalCase { + const suffix = arm.toLowerCase(); + return { + name: `${family}-${suffix}`, + repeat, + repo: { fixture: `repos/${family}` }, + review: { + depth: "normal", + lenses: ["core/code-review"], + cache: false, + packSameFileHunks: arm !== "A", + packedToolBudgetMode: arm === "C" ? "atom-scaled" : "base" + }, + should_find: [{ id: "bug", path: "app.ts", lineRange: [1, 1], titlePattern: "boundary" }] + }; +} + +function expectation(hit: boolean, loss: EvalExpectationResult["loss"] = undefined): EvalExpectationResult { + return { + expectationId: "bug", + list: "should_find", + tier: "required", + status: hit ? "pass" : "fail", + fromReplayedArtifacts: false, + matched: hit ? [{ findingId: "finding", artifact: "final-findings" }] : [], + ...(loss === undefined ? {} : { loss }) + }; +} + +function score(hit: boolean, cost: number | undefined = 1, lossLabel = "missed-before-candidate-generation"): EvalScore { + const loss = hit ? undefined : { + label: lossLabel as "missed-before-candidate-generation", + nearestInstances: [] + }; + return { + status: hit ? "pass" : "fail", + expectationResults: [expectation(hit, loss)], + budgetResults: cost === undefined ? [] : [{ + check: "maxCostUSD", + status: "pass", + actual: cost, + limit: 100, + direction: "maximum" + }], + violations: [], + nearViolations: [], + metrics: { + reportedFindings: hit ? 1 : 0, + inlineFindings: hit ? 1 : 0, + summaryOnlyFindings: 0, + suppressedFindings: 0, + candidateFindings: hit ? 1 : 0, + duplicateGroups: 0, + ...(cost === undefined ? {} : { costUSD: cost }), + stageLossCounts: { + "missed-before-candidate-generation": hit ? 0 : 1, + "lost-at-verification": 0, + "lost-at-composition": 0, + "partial-match": 0 + } + } + }; +} + +function modelCall(packetId: string, repeat: number, costUSD = 0.1) { + return { + callId: `call-${packetId}-${repeat}`, + runId: "eval", + stage: 7 as const, + role: "packetReview" as const, + model: "fake", + provider: "fake", + packetId, + kind: "initial" as const, + attempt: 1, + promptChars: 100, + promptHash: "hash", + outputChars: 10, + outputHash: "hash", + inputTokens: 100, + outputTokens: 10, + reasoningTokens: 0, + totalTokens: 110, + costUSD, + durationMs: 1000, + cacheStatus: "disabled" as const, + stopReason: "submit" as const, + status: "ok" as const + }; +} + +function candidateFinding(id: string, packetId: string, hunkId: string, line = 1): CandidateFinding { + return { + id, + title: "Boundary bug", + severity: "medium", + confidence: "medium", + path: "app.ts", + anchor: { path: "app.ts", line, side: "RIGHT", hunkId }, + anchorSource: "model", + modelAnchorSubmitted: true, + changedLine: true, + category: "correctness", + evidence: { changedCode: "+change" }, + failureMode: "The boundary behavior is incorrect.", + whyThisMatters: "The changed code produces an incorrect result.", + verification: "The changed line demonstrates the failure.", + producedBy: { kind: "packet", stage: 7, packetId, lensId: "core/code-review", skillIds: ["core/code-review"] } + }; +} + +function finalFinding(candidate: CandidateFinding, publication: FinalFinding["publication"] = "inline"): FinalFinding { + return { + ...structuredClone(candidate), + fingerprint: `fingerprint-${candidate.id}`, + finalBody: candidate.failureMode, + publication, + mergedCandidateIds: [candidate.id], + mergedAnchors: candidate.anchor === undefined ? [] : [structuredClone(candidate.anchor)] + }; +} + +function findingEvidence(candidate: CandidateFinding | undefined): Pick { + if (candidate === undefined) { + return { candidateFindings: [], verification: [], finalSelection: [], finalFindings: [] }; + } + return { + candidateFindings: [candidate], + verification: [{ + candidateId: candidate.id, + gate: "passed", + gateDecision: "scheduled", + gateReason: "meets_confidence_threshold", + verificationLane: "standard", + gateFacts: { + anchorSource: "model", + category: candidate.category, + changedLine: true, + confidence: candidate.confidence, + failureModeConcrete: true, + hasChangedCode: true, + hasFailureMode: true, + modelAnchorSubmitted: true, + modelAnchorValid: true, + relatedEvidenceCount: 0, + severity: candidate.severity, + validAnchorPresent: true + }, + verdict: { + candidateId: candidate.id, + verdict: "keep", + reason: "evidence confirmed", + requiredEvidencePresent: true, + falsePositiveRisk: "low" + } + }], + finalSelection: [{ findingId: candidate.id, decision: "published", reason: "composer-selected" }], + finalFindings: [finalFinding(candidate)] + }; +} + +function executionArtifacts(execution: EvalExecutionInput): EvalArtifacts { + const existing = execution.scoringArtifacts as EvalArtifacts | undefined; + const persisted = { + ...existing, + candidates: execution.candidateFindings, + verification: execution.verification, + finalSelection: execution.finalSelection, + finalFindings: execution.finalFindings, + packets: execution.packets, + hintEvents: existing?.hintEvents ?? [], + reviewPlan: execution.plan, + metricsSources: { + ...existing?.metricsSources, + modelCalls: execution.modelCalls, + toolCalls: execution.toolCalls, + costProfile: { totalCostUSD: execution.modelCalls.reduce((total, call) => total + (call.costUSD ?? 0), 0) }, + runJson: { durationMs: execution.wallTimeSeconds * 1000 } + } + } as EvalArtifacts; + return reconstructEvidenceArtifacts({ ...execution, scoringArtifacts: persisted }); +} + +function refreshExecutionEvidence(execution: EvalExecutionInput, policyConfig: typeof defaultConfig = defaultConfig): void { + const candidateById = new Map(execution.candidateFindings.map((candidate) => [candidate.id, candidate])); + const packetById = new Map(execution.packets.map((packet) => [packet.id, packet])); + execution.verification = execution.verification.map((record) => { + const candidateId = "verdict" in record && record.duplicateOf !== undefined + ? record.duplicateOf + : record.candidateId; + const candidate = candidateById.get(candidateId); + if (candidate === undefined) { + return record; + } + return { + ...record, + gateFacts: reconstructVerificationGateFactsFromArtifacts( + candidate, + packetById.get(candidate.producedBy.packetId), + execution.diff + ) + }; + }); + const verified = reconstructVerifiedFindingsFromArtifacts(execution.candidateFindings, execution.verification, execution.packets, execution.diff); + const composer = reconstructComposerGroupsFromArtifacts(verified, execution.packets); + const publishableById = new Map(composer.publishable.map((finding) => [finding.id, finding])); + const selectionById = new Map(execution.finalSelection.map((record) => [record.findingId, record])); + const downgradeReasons = new Set(["min-inline-confidence", "soft-comment-cap", "unanchorable"]); + const suppressionReasons = new Set(["severity-threshold", "confidence-threshold", "report-cap"]); + const drafts = execution.finalFindings.map((finding) => { + const selections = finding.mergedCandidateIds.flatMap((id) => selectionById.get(id) ?? []); + const isPretrimmed = finding.mergedCandidateIds.some((id) => composer.pretrimSuppressedIds.includes(id)); + const hasDowngrade = selections.some((selection) => downgradeReasons.has(selection.reason)); + const hasPolicyReason = hasDowngrade || selections.some((selection) => suppressionReasons.has(selection.reason)); + return { + mergedFindings: finding.mergedCandidateIds.flatMap((id) => publishableById.get(id) ?? []), + finalBody: finding.finalBody, + requestedPublication: isPretrimmed + ? "suppressed" as const + : finding.publication === "inline" || hasDowngrade ? "inline" as const : finding.publication, + baseSelection: finding.mergedCandidateIds.map((id) => isPretrimmed + ? { findingId: id, decision: "suppressed" as const, reason: "composer-pre-trim" } + : id === finding.id + ? { + findingId: id, + decision: "published" as const, + reason: !hasPolicyReason && selectionById.get(id)?.reason === "composer_omitted_finding" + ? "composer_omitted_finding" + : "composer-selected" + } + : { findingId: id, decision: "merged" as const, reason: "composer-merged", mergedIntoFingerprint: finding.fingerprint }) + }; + }); + let policy = reconstructComposerPolicyFromArtifacts( + drafts, + execution.packets, + execution.diff, + policyConfig, + execution.verification.flatMap((record) => + "verdict" in record && (policyConfig.review.verify === false || record.duplicateOf === undefined) ? [record.verdict] : [] + ) + ); + const fingerprintByCandidateId = new Map(policy.findings.flatMap((finding) => + finding.mergedCandidateIds.map((id) => [id, finding.fingerprint] as const) + )); + if (drafts.some((draft) => draft.baseSelection.some((record) => record.decision === "merged"))) { + policy = reconstructComposerPolicyFromArtifacts( + drafts.map((draft) => ({ + ...draft, + baseSelection: draft.baseSelection.map((record) => record.decision === "merged" + ? { ...record, mergedIntoFingerprint: fingerprintByCandidateId.get(record.findingId) ?? record.mergedIntoFingerprint ?? "missing" } + : record) + })), + execution.packets, + execution.diff, + policyConfig, + execution.verification.flatMap((record) => + "verdict" in record && (policyConfig.review.verify === false || record.duplicateOf === undefined) ? [record.verdict] : [] + ) + ); + } + execution.finalFindings = policy.findings; + execution.finalSelection = policy.selection; + execution.finalSelectionArtifact = { + composition: { mode: "llm" }, + records: structuredClone(policy.selection), + publicationAnchors: structuredClone(policy.publicationAnchors), + confidenceSelections: structuredClone(policy.confidenceSelections), + groups: composer.groups.map(({ representativeId: _representativeId, ...group }) => group) + }; + execution.scoringArtifacts = executionArtifacts(execution); +} + +function refreshRunEvidenceScores(run: EvalCaseRunInput): void { + const policyConfig = structuredClone(defaultConfig); + if (run.info.effectiveConfig !== undefined) { + policyConfig.review = { ...policyConfig.review, ...run.info.effectiveConfig.review }; + } + for (const execution of run.executions) { + refreshExecutionEvidence(execution, policyConfig); + execution.score = scoreEvalRun(run.declaredCase, executionArtifacts(execution), "live"); + execution.scoringArtifacts = executionArtifacts(execution); + } + if (run.executions.length === 1) { + run.info.score = run.executions[0]!.score; + delete run.info.repeats; + return; + } + const repeated = aggregateRepeatScores(run.declaredCase, run.executions.map((execution) => ({ + runDir: `repeats/${execution.repeat}`, + score: execution.score, + artifacts: executionArtifacts(execution) + }))); + run.info.score = repeated.score; + run.info.repeats = repeated.aggregate; +} + +function planForHunks(hunkIds: string[]): ReviewPlan { + return { + diffUnderstanding: { declaredIntent: "test", inferredBehavior: "test" }, + coverage: hunkIds.map((hunkId) => ({ + hunkId, + path: "app.ts", + coverage: "normal" as const, + lenses: ["core/code-review"], + surroundingContextHints: [], + reason: "test" + })) + }; +} + +function execution( + arm: "A" | "B" | "C", + repeat: number, + options: { treated?: boolean; hit?: boolean; cost?: number | undefined; missingTelemetry?: boolean } = {} +): EvalExecutionInput { + const first = packet(`a1-${repeat}`, ["h1"]); + const second = packet(`a2-${repeat}`, ["h2"]); + const treated = options.treated ?? arm !== "A"; + const packets = arm === "A" || !treated + ? [first, second] + : [packet(`packed-${arm}-${repeat}`, ["h1", "h2"], { + toolBudget: arm === "C" ? { ...standardBudget, maxToolCalls: 5, maxResultChars: 12_000 } : standardBudget + })]; + const events = arm === "A" || options.missingTelemetry === true + ? [] + : treated + ? [packingEvent(packets[0]!, [first, second], { + toolBudgetMode: arm === "C" ? "atom-scaled" : "base", + effectiveToolBudget: packets[0]!.toolBudget + })] + : packets.map((target, index) => packingEvent(target, [index === 0 ? first : second], { + toolBudgetMode: arm === "C" ? "atom-scaled" : "base" + })); + const executionScore = score(options.hit ?? true, options.cost === undefined && "cost" in options ? undefined : options.cost ?? 1); + const hit = options.hit ?? true; + const calls = packets.map((target) => modelCall(target.id, repeat, (options.cost ?? 1) / packets.length)); + if ("cost" in options && options.cost === undefined) { + for (const call of calls) { + delete (call as Partial).costUSD; + } + } + executionScore.metrics.modelCalls = calls.length; + executionScore.metrics.verificationCalls = hit ? 1 : 0; + executionScore.metrics.elapsedSeconds = 60; + const candidate = hit ? candidateFinding("finding", packets[0]!.id, "h1") : undefined; + const result = { + repeat, + score: executionScore, + telemetryDir: `/tmp/${arm}/${repeat}`, + packets, + events, + modelCalls: calls, + toolCalls: [], + fileFacts: [{ + path: "app.ts", + language: "typescript", + processingMode: "per-hunk", + testStatus: "source", + isGenerated: false, + isVendored: false, + isLockfile: false, + isBinary: false, + changedLines: 2, + hunkCount: 2, + labels: [], + reviewPriority: "normal", + reasons: [], + provenance: [] + }], + diff: diffForPackets([first, second]), + plan: planForHunks(["h1", "h2"]), + ...findingEvidence(candidate), + reviewedHunkIds: ["h1", "h2"], + wallTimeSeconds: 60 + } as unknown as EvalExecutionInput; + refreshExecutionEvidence(result); + return result; +} + +function evalRun( + family: string, + arm: "A" | "B" | "C", + runNumber: number, + executions: EvalExecutionInput[] +): EvalCaseRunInput { + const snapshot = evalCase(family, arm, executions.length); + for (const entry of executions) { + refreshExecutionEvidence(entry); + entry.score = scoreEvalRun(snapshot, entry.scoringArtifacts, "live"); + } + const repeated = executions.length > 1 ? aggregateRepeatScores(snapshot, executions.map((entry) => ({ + runDir: `repeats/${entry.repeat}`, + score: entry.score, + artifacts: executionArtifacts(entry) + }))) : undefined; + const info: EvalRunInfo = { + runNumber, + caseName: snapshot.name, + caseFile: `${family}-${arm.toLowerCase()}.yml`, + caseHash: `hash-${family}-${arm}`, + caseSnapshot: snapshot, + mode: "live", + ...(repeated === undefined ? {} : { repeats: repeated.aggregate }), + cache: { enabled: false, source: "case" }, + effectiveConfig: { + review: { + concurrency: 1, + timeoutMs: 60_000, + verify: true, + maxFindings: defaultConfig.review.maxFindings, + softCommentCap: defaultConfig.review.softCommentCap, + minConfidence: defaultConfig.review.minConfidence, + minInlineConfidence: defaultConfig.review.minInlineConfidence, + packSameFileHunks: arm !== "A", + packedToolBudgetMode: arm === "C" ? "atom-scaled" : "base" + }, + llm: { maxConcurrentCalls: 1 } + }, + codegenieRuntime: { packageVersion: "0.5.0", commit: "a".repeat(40), source: "git" }, + startedAt: "2026-07-24T00:00:00.000Z", + finishedAt: "2026-07-24T00:01:00.000Z", + score: repeated?.score ?? executions[0]?.score ?? score(true) + }; + return { runNumber, runDir: `/logs/${runNumber}`, info, declaredCase: structuredClone(snapshot), executions }; +} + +async function attachProducerSummaryEvidence(run: EvalCaseRunInput): Promise { + const execution = run.executions[0]!; + const repoRoot = await mkdtemp(path.join(tmpdir(), "packet-report-paid-summary-")); + tempDirs.push(repoRoot); + let clockTick = 0; + const telemetry = createRunTelemetry({ + telemetryConfig: { ...defaultConfig.telemetry, enabled: true, logLevel: "debug" }, + idFactory: () => `paid-summary-${run.runNumber}`, + clock: () => new Date(Date.UTC(2026, 6, 25, 0, 0, clockTick++)) + }); + const attached = await telemetry.attachRunDirectory(repoRoot); + telemetry.recorder.event({ stage: 7, level: "info", message: "stage_started" }); + telemetry.recorder.event({ + stage: 7, + level: "warn", + message: "stage7_schema_repair_attempted", + data: { classification: "candidate_payload", payloadKind: "candidate" } + }); + telemetry.recorder.event({ + stage: 7, + level: "info", + message: "stage7_schema_compact_repair_scheduled", + data: { repairPromptChars: execution.modelCalls[0]?.promptChars ?? 0 } + }); + for (const [index, call] of execution.modelCalls.entries()) { + const { runId: _runId, ...record } = call; + telemetry.recorder.recordModelCall(index === 0 ? { ...record, kind: "repair" } : record); + } + telemetry.recorder.event({ + stage: 7, + level: "info", + message: "stage7_schema_repair_recovered", + data: { classification: "schema_valid_after_retry" } + }); + telemetry.recorder.event({ stage: 7, level: "info", message: "stage_completed" }); + const reviewedHunks = execution.reviewedHunkIds.length; + telemetry.recorder.event({ + stage: 10, + level: "info", + message: "pipeline_metrics", + data: { + totals: { + filesChanged: execution.diff.files.length, + hunks: execution.diff.files.flatMap((file) => file.hunks).length, + packets: execution.packets.length, + packetReviews: execution.packets.length, + candidates: execution.candidateFindings.length, + verified: execution.verification.filter((record) => "verdict" in record && record.verdict.verdict !== "reject").length, + finalFindings: execution.finalFindings.length, + postedComments: 0 + }, + workers: { started: 2, completed: 2, failed: 0, retried: 0, timedOut: 0 }, + packets: { generated: execution.packets.length, reviewed: execution.packets.length, failed: 0, degraded: 0 }, + lenses: { selected: 1, byLens: { "core/code-review": execution.packets.length } }, + coverage: { + byLevel: { deep: 0, normal: reviewedHunks, light: 0, skip: 0 }, + hunks: { total: reviewedHunks, reviewed: reviewedHunks, skipped: 0, failed: 0, degraded: 0 } + }, + candidates: { + generated: execution.candidateFindings.length, + gateRejected: 0, + verificationScheduled: execution.verification.length, + verificationBudgetLimited: 0, + clusteredDuplicates: 0, + verificationRepresentatives: execution.verification.length, + lowConfidenceSuppressed: 0, + lowConfidenceEvidenceEligible: 0, + lowConfidenceEvidenceScheduled: 0, + lowConfidenceEvidenceLaneLimited: 0, + lowConfidenceEvidenceKept: 0, + lowConfidenceEvidenceRejected: 0, + lowConfidenceEvidenceIncomplete: 0 + }, + verdicts: { accept: execution.verification.length, revise: 0, reject: 0, incomplete: 0 }, + dedup: { clusters: execution.finalFindings.length, duplicates: 0, suppressed: 0 }, + finalSelection: { + published: execution.finalSelection.filter((record) => record.decision === "published").length, + merged: execution.finalSelection.filter((record) => record.decision === "merged").length, + suppressed: execution.finalSelection.filter((record) => record.decision === "suppressed").length, + finalFindings: execution.finalFindings.length, + compositionMode: "llm" + }, + posting: { attempted: 0, postedComments: 0, skippedDuplicates: 0, failed: 0 } + } + }); + await telemetry.finalize({ status: "completed_full", exitCode: 0 }); + const readJson = async (relative: string): Promise => JSON.parse(await readFile(path.join(attached.runDir, relative), "utf8")); + const readJsonl = async (relative: string): Promise => (await readFile(path.join(attached.runDir, relative), "utf8")) + .trim().split("\n").filter((line) => line.length > 0).map((line) => JSON.parse(line) as unknown); + execution.events = await readJsonl("events.jsonl") as TelemetryEvent[]; + execution.modelCalls = await readJsonl("model-calls.jsonl") as EvalExecutionInput["modelCalls"]; + execution.toolCalls = await readJsonl("tool-calls.jsonl") as EvalExecutionInput["toolCalls"]; + const runSummary = await readJson("run.json") as { startedAt: string; finishedAt: string; durationMs: number }; + const modelByStage = new Map(); + for (const call of execution.modelCalls.filter((call) => call.cacheStatus !== "hit")) { + const bucket = modelByStage.get(call.stage) ?? { modelCalls: 0, totalTokens: 0 }; + bucket.modelCalls += 1; + bucket.totalTokens += call.totalTokens ?? 0; + modelByStage.set(call.stage, bucket); + } + const totalTokens = execution.modelCalls.reduce((total, call) => total + (call.totalTokens ?? 0), 0); + const totalCostUSD = execution.modelCalls.reduce((total, call) => total + (call.costUSD ?? 0), 0); + execution.wallTimeSeconds = runSummary.durationMs / 1000; + execution.summaryArtifacts = { + attention: execution.packets.map((packet) => ({ + packetId: packet.id, + path: packet.path, + coverage: packet.coverage, + coverageSource: "planner", + ensemblePasses: 1, + directCandidates: execution.candidateFindings.filter((candidate) => candidate.producedBy.packetId === packet.id && candidate.provenance === undefined).length, + promotedCandidates: execution.candidateFindings.filter((candidate) => candidate.producedBy.packetId === packet.id && candidate.provenance !== undefined).length, + hintsEmitted: 0, + uncertaintiesEmitted: 0, + keptVerified: execution.verification.filter((record) => record.candidateId === execution.candidateFindings.find((candidate) => candidate.producedBy.packetId === packet.id)?.id && "verdict" in record).length, + published: execution.finalFindings.filter((finding) => finding.publication !== "suppressed" && + finding.mergedCandidateIds.some((id) => execution.candidateFindings.find((candidate) => candidate.id === id)?.producedBy.packetId === packet.id)).length + })), + humanAttention: [], + budget: { + completeness: "complete", + partialReasons: [], + multiplier: 1, + configured: { timeoutMs: run.info.effectiveConfig!.review.timeoutMs }, + effective: { timeoutMs: run.info.effectiveConfig!.review.timeoutMs }, + usage: { + modelCalls: execution.modelCalls.filter((call) => call.cacheStatus !== "hit").length, + totalTokens, + ...(totalCostUSD > 0 ? { costUSD: totalCostUSD } : {}), + byStage: [...modelByStage.entries()].map(([stage, value]) => ({ stage, ...value })).sort((left, right) => left.stage - right.stage) + }, + overruns: [], + dispatchBlocks: [] + }, + cost: await readJson("stages/00-run/cost-profile.json"), + model: await readJson("stages/00-run/model-calls-summary.json"), + tool: await readJson("stages/00-run/tool-calls-summary.json"), + run: await readJson("run.json"), + telemetry: await readJson("telemetry.json") + }; + run.info.reviewRunId = telemetry.recorder.runId; + run.info.startedAt = runSummary.startedAt; + run.info.finishedAt = runSummary.finishedAt; + refreshRunEvidenceScores(run); +} + +function attachInvocation( + runs: EvalCaseRunInput[], + invocationId: string, + status: EvalInvocationManifest["status"] = "complete", + recordedRuns = runs.length +): EvalCaseRunInput[] { + const manifest: EvalInvocationManifest = { + schemaVersion: 1, + invocationId, + suiteDir: "/suite", + status, + startedAt: "2026-07-24T00:00:00.000Z", + ...(status === "complete" ? { completedAt: "2026-07-24T00:02:00.000Z" } : {}), + cases: runs.map((run, caseIndex) => ({ + caseIndex, + caseName: run.info.caseName, + caseHash: run.info.caseHash, + caseFile: run.info.caseFile ?? `${run.info.caseName}.yml` + })), + runs: runs.slice(0, recordedRuns).map((run, caseIndex) => ({ + caseIndex, + caseName: run.info.caseName, + caseHash: run.info.caseHash, + runNumber: run.runNumber, + logsRoot: "/logs", + runPath: String(run.runNumber) + })) + }; + for (const [caseIndex, run] of runs.entries()) { + run.runDir = `/logs/${run.runNumber}`; + run.info.invocation = { id: invocationId, caseIndex, manifest: `invocations/${invocationId}.json` }; + run.invocationManifest = manifest; + } + return runs; +} + +function abcCohort( + repeat: number, + mutate?: (arm: "A" | "B" | "C", index: number) => Partial[2]>, + family = "dilution", + runOffset = 0 +): CohortSelection { + const arms = (["A", "B", "C"] as const).map((arm, armIndex) => evalRun( + family, + arm, + runOffset + armIndex + 1, + Array.from({ length: repeat }, (_, index) => execution(arm, index + 1, mutate?.(arm, index) ?? {})) + )); + return { id: `1-${arms.length}`, runs: arms }; +} + +describe("packet packing eval analysis", () => { + it("selects only the explicit latest complete cohort", () => { + const old = attachInvocation(abcCohort(1).runs, "old-invocation"); + const latest = attachInvocation(abcCohort(1).runs.map((run) => ({ ...run, runNumber: run.runNumber + 3, info: { ...run.info, runNumber: run.runNumber + 3 } })), "latest-invocation"); + const selected = selectExplicitCohort([...old, ...latest], "latest"); + expect(selected.id).toBe("latest-invocation"); + expect(selected.runs.map((run) => run.runNumber)).toEqual([4, 5, 6]); + expect(() => selectExplicitCohort([...old, ...attachInvocation(abcCohort(1).runs, "same-ending-run")], "3")).toThrow(/multiple persisted invocations/u); + }); + + it("ignores unselected interrupted history and accepts an exact invocation UUID", () => { + const interrupted = attachInvocation(abcCohort(1).runs, "historical-interrupted", "running", 2).slice(0, 2); + const invocationId = "550e8400-e29b-41d4-a716-446655440000"; + const current = attachInvocation(abcCohort(1, undefined, "dilution", 3).runs, invocationId); + expect(selectExplicitCohort([...interrupted, ...current], "latest").id).toBe(invocationId); + expect(selectExplicitCohort([...interrupted, ...current], invocationId).runs.map((run) => run.runNumber)).toEqual([4, 5, 6]); + }); + + it("preserves manifest caseIndex order across roots with duplicate run numbers", () => { + const invocationId = "multi-root-order"; + const runs = attachInvocation([ + evalRun("root-a-one", "A", 1, [execution("A", 1)]), + evalRun("root-a-two", "A", 2, [execution("A", 1)]), + evalRun("root-b-one", "A", 1, [execution("A", 1)]), + evalRun("root-b-two", "A", 2, [execution("A", 1)]) + ], invocationId); + const locations = [ + { logsRoot: "/rootA", runPath: "1", runNumber: 1 }, + { logsRoot: "/rootA", runPath: "2", runNumber: 2 }, + { logsRoot: "/rootB", runPath: "1", runNumber: 1 }, + { logsRoot: "/rootB", runPath: "2", runNumber: 2 } + ]; + const manifest = runs[0]!.invocationManifest!; + for (const [caseIndex, run] of runs.entries()) { + const location = locations[caseIndex]!; + Object.assign(manifest.runs[caseIndex]!, location); + run.runNumber = location.runNumber; + run.info.runNumber = location.runNumber; + run.runDir = `${location.logsRoot}/${location.runPath}`; + } + const selected = selectExplicitCohort([runs[3]!, runs[1]!, runs[2]!, runs[0]!], invocationId); + expect(selected.runs.map((run) => run.runDir)).toEqual(["/rootA/1", "/rootA/2", "/rootB/1", "/rootB/2"]); + expect(selected.runs.map((run) => run.info.invocation!.caseIndex)).toEqual([0, 1, 2, 3]); + }); + + it("rejects an explicit mixed cohort with duplicate cases", () => { + const runs = attachInvocation([...abcCohort(1).runs, abcCohort(1).runs[0]!].map((run, index) => ({ + ...run, + runNumber: index + 1, + info: { ...run.info, runNumber: index + 1 } + })), "mixed-invocation"); + expect(() => selectExplicitCohort(runs, "1-4")).toThrow(/multiple generations/u); + }); + + it("rejects an interrupted newest invocation instead of mixing it into latest", () => { + const complete = attachInvocation(abcCohort(1).runs, "complete-invocation"); + const interrupted = attachInvocation(abcCohort(1).runs.map((run) => ({ + ...run, + runNumber: run.runNumber + 3, + info: { ...run.info, runNumber: run.runNumber + 3 } + })), "interrupted-invocation", "running", 2).slice(0, 2); + expect(() => selectExplicitCohort([...complete, ...interrupted], "latest")).toThrow(/exact ordered case set/u); + }); + + it("proves preflight treatment without selecting an arm and uses relational call cost", () => { + const report = analyzeEvalCohort(abcCohort(1), 1); + expect(report.failures).toEqual([]); + expect(report.cohort.actualCostUSD).toBe(3); + expect(report.selectedArm).toBeUndefined(); + expect(report.cases?.[0]?.arms.B.treatment).toMatchObject({ treated: 1, total: 1, valid: true }); + + const fallback = abcCohort(1); + delete fallback.runs[0]!.executions[0]!.score.metrics.costUSD; + expect(analyzeEvalCohort(fallback, 1).cohort.actualCostUSD).toBe(3); + + const failedFallback = abcCohort(1); + delete failedFallback.runs[0]!.executions[0]!.modelCalls[0]!.costUSD; + expect(failureCodes(analyzeEvalCohort(failedFallback, 1).failures)).toContain("missing_spend_data"); + }); + + it("fails closed on mismatched treatment lenses and effective configuration", () => { + const lenses = abcCohort(1); + const event = lenses.runs[1]!.executions[0]!.events[0]!; + event.data = { ...event.data, requestedLensSignature: "[]" }; + expect(failureCodes(analyzeEvalCohort(lenses, 1).failures)).toContain("target_lens_signature"); + + const config = abcCohort(1); + config.runs[1]!.info.effectiveConfig!.review.packedToolBudgetMode = "invalid" as never; + expect(() => analyzeEvalCohort(config, 1)).toThrow(/invalid effective packet-packing settings/u); + }); + + it("joins every B/C atom and hunk exactly to A and independently validates telemetry", () => { + const cohort = abcCohort(1); + const b = cohort.runs[1]!.executions[0]!; + const event = b.events[0]!; + event.data = { + ...event.data, + atomIds: ["unknown-atom", ...(event.data?.atomIds as string[]).slice(1)], + standaloneProfiles: ["investigate", "standard"], + capUsage: { ...(event.data?.capUsage as object), patchChars: 1 }, + baseToolBudget: { maxToolCalls: 99, maxInvestigationRounds: 99, maxResultChars: 99 } + }; + b.packets[0]!.dispatchRank = [3, -99]; + const codes = failureCodes(analyzeEvalCohort(cohort, 1).failures); + expect(codes).toEqual(expect.arrayContaining(["unknown_source_atom", "treatment_invariant", "invalid_dispatch_rank", "atom_bijection"])); + }); + + it("requires a bidirectional expectation join and selects B on equal B/C evidence", () => { + const tied = analyzeEvalCohort(abcCohort(10), 10); + expect(tied.failures).toEqual([]); + expect(tied.cases?.[0]?.selectedArm).toBe("B"); + + const missing = abcCohort(1); + missing.runs[1]!.executions[0]!.score.expectationResults = []; + expect(failureCodes(analyzeEvalCohort(missing, 1).failures)).toContain("expectation_join"); + + const colludingScores = abcCohort(1); + colludingScores.runs[0]!.declaredCase.should_find![0]!.id = "declared-only"; + expect(failureCodes(analyzeEvalCohort(colludingScores, 1).failures)).toEqual(expect.arrayContaining([ + "declared_expectation_join", + "expectation_join" + ])); + }); + + it("joins requested and routed lenses to the persisted Stage-5 plan and A atoms", () => { + const planned = abcCohort(1); + planned.runs[0]!.executions[0]!.plan.coverage[0]!.lenses = ["security/auth"]; + expect(failureCodes(analyzeEvalCohort(planned, 1).failures)).toContain("requested_lens_join"); + + const routed = abcCohort(1); + routed.runs[0]!.executions[0]!.packets[0]!.lenses.push("security/auth"); + expect(failureCodes(analyzeEvalCohort(routed, 1).failures)).toContain("routed_lens_join"); + + const missing = abcCohort(1); + missing.runs[0]!.executions[0]!.plan.coverage.shift(); + expect(failureCodes(analyzeEvalCohort(missing, 1).failures)).toEqual(expect.arrayContaining(["stage5_lens_join", "requested_lens_join"])); + + const empty = abcCohort(1); + empty.runs[0]!.executions[0]!.plan.coverage[0]!.lenses = []; + expect(failureCodes(analyzeEvalCohort(empty, 1).failures)).toEqual(expect.arrayContaining(["stage5_lens_join", "requested_lens_join"])); + + const whitespace = abcCohort(1); + whitespace.runs[0]!.executions[0]!.plan.coverage[0]!.lenses = [" "]; + expect(failureCodes(analyzeEvalCohort(whitespace, 1).failures)).toEqual(expect.arrayContaining(["stage5_lens_join", "requested_lens_join"])); + + const extra = abcCohort(1); + extra.runs[1]!.executions[0]!.packets[0]!.lenses.push("security/arbitrary-extra"); + expect(failureCodes(analyzeEvalCohort(extra, 1).failures)).toContain("routed_lens_join"); + }); + + it("selects one economical cohort-wide arm and fails conflicting case requirements", () => { + const economicalRuns = [ + ...abcCohort(10, (arm, index) => ({ hit: arm !== "B" || index > 0, cost: arm === "C" ? 0.5 : 1 }), "first", 0).runs, + ...abcCohort(10, (arm, index) => ({ hit: arm === "C" || index > 0, cost: arm === "C" ? 0.5 : 1 }), "second", 3).runs + ]; + for (const run of economicalRuns.filter((candidate) => candidate.info.caseName.endsWith("-c"))) { + for (const execution of run.executions) { + for (const call of execution.modelCalls) { + call.durationMs = 500; + } + } + } + const economical = analyzeEvalCohort({ id: "economic", runs: economicalRuns }, 10); + expect(economical.failures).toEqual([]); + expect(economical.selectedArm).toBe("C"); + expect(economical.cases?.every((entry) => entry.selectedArm === "C")).toBe(true); + + const conflictingRuns = [ + ...abcCohort(10, (arm, index) => ({ hit: arm === "B" ? index < 7 : true }), "requires-c", 0).runs, + ...abcCohort(10, (arm, index) => ({ hit: arm === "C" ? index < 7 : true }), "requires-b", 3).runs + ]; + expect(failureCodes(analyzeEvalCohort({ id: "conflicting", runs: conflictingRuns }, 10).failures)).toContain("conflicting_arm_selection"); + }); + + it("requires C to improve pressure or recall and retain 85% of packet-count service savings when B fails", () => { + const tied = analyzeEvalCohort(abcCohort(10), 10); + expect(tied.selectedArm).toBe("B"); + + const lowRetention = abcCohort(10, (arm, index) => ({ hit: arm !== "B" || index < 7 })); + for (const execution of lowRetention.runs[2]!.executions) { + execution.modelCalls[0]!.durationMs = 1_900; + } + const report = analyzeEvalCohort(lowRetention, 10); + expect(failureCodes(report.failures)).toEqual(expect.arrayContaining(["atom_scaled_savings_retention", "no_passing_arm"])); + }); + + it("applies savings retention cohort-wide when B is ineligible for pressure rather than recall", () => { + const cases = [ + abcCohort(10, undefined, "pressure-one", 0), + abcCohort(10, undefined, "pressure-two", 3) + ]; + for (const cohort of cases) { + const b = cohort.runs[1]!; + for (const execution of b.executions) { + execution.toolCalls.push({ + runId: "eval", + toolCallId: `rejected-${b.info.caseName}-${execution.repeat}`, + timestamp: "2026-07-24T00:00:00.000Z", + stage: 7, + initiator: "model", + packetId: execution.packets[0]!.id, + tool: "read_file", + args: { path: "app.ts" }, + backend: "text", + precision: "exact", + degraded: false, + degradationReason: "tool_call_budget_exhausted", + resultChars: 0, + durationMs: 0, + status: "rejected" + }); + } + refreshRunEvidenceScores(b); + for (const execution of cohort.runs[2]!.executions) { + execution.modelCalls[0]!.durationMs = 1_900; + } + } + const report = analyzeEvalCohort({ id: "cohort-pressure-retention", runs: cases.flatMap((cohort) => cohort.runs) }, 10); + expect(failureCodes(report.failures)).toEqual(expect.arrayContaining([ + "tool_pressure", + "atom_scaled_savings_retention", + "no_passing_arm" + ])); + }); + + it("reconstructs relational evidence and rejects deletions, altered matches, costs, and repeat aggregates", () => { + for (const mutate of [ + (run: EvalCaseRunInput) => { run.executions[0]!.verification = []; }, + (run: EvalCaseRunInput) => { run.executions[0]!.finalSelection = []; }, + (run: EvalCaseRunInput) => { run.executions[0]!.score.expectationResults[0]!.matched = [{ findingId: "invented", artifact: "final-findings" }]; }, + (run: EvalCaseRunInput) => { run.executions[0]!.score.metrics.costUSD = 99; } + ]) { + const cohort = abcCohort(1); + mutate(cohort.runs[1]!); + const codes = failureCodes(analyzeEvalCohort(cohort, 1).failures); + expect(codes.some((code) => code === "paid_evidence_relations" || code === "paid_evidence_score_reconstruction")).toBe(true); + } + + const repeated = abcCohort(10); + repeated.runs[1]!.info.repeats!.totals.costUSD += 1; + expect(failureCodes(analyzeEvalCohort(repeated, 10).failures)).toContain("paid_evidence_aggregate"); + }); + + it("rejects collusive score/summary mutations and non-canonical repeat paths", () => { + const collusive = abcCohort(1); + const run = collusive.runs[1]!; + const target = run.executions[0]!; + target.scoringArtifacts.metricsSources.costProfile = { totalCostUSD: 99 }; + target.scoringArtifacts.metricsSources.modelCallsSummary = { totalCalls: 99 }; + target.scoringArtifacts.metricsSources.toolCallsSummary = { totalCalls: 99 }; + target.scoringArtifacts.metricsSources.runJson = { durationMs: 99_000 }; + target.score.metrics.costUSD = 99; + target.score.metrics.modelCalls = 99; + target.score.metrics.toolCalls = 99; + target.score.metrics.elapsedSeconds = 99; + const costBudget = target.score.budgetResults.find((result) => result.check === "maxCostUSD"); + if (costBudget !== undefined) costBudget.actual = 99; + run.info.score = structuredClone(target.score); + expect(failureCodes(analyzeEvalCohort(collusive, 1).failures)).toEqual(expect.arrayContaining([ + "paid_evidence_score_reconstruction", + "evidence_cost_accounting" + ])); + + const paths = abcCohort(10); + paths.runs[1]!.info.repeats!.executions[0]!.runDir = "/arbitrary/repeat"; + expect(failureCodes(analyzeEvalCohort(paths, 10).failures)).toContain("paid_evidence_aggregate"); + + const reordered = abcCohort(10); + [reordered.runs[1]!.executions[0], reordered.runs[1]!.executions[1]] = + [reordered.runs[1]!.executions[1]!, reordered.runs[1]!.executions[0]!]; + expect(failureCodes(analyzeEvalCohort(reordered, 10).failures)).toContain("paid_evidence_repeat_order"); + }); + + it("reconciles every accepted producer summary field to raw calls, events, and artifacts", async () => { + const base = abcCohort(1); + await attachProducerSummaryEvidence(base.runs[0]!); + expect(failureCodes(analyzeEvalCohort(base, 1).failures)).not.toContain("paid_summary_reconciliation"); + + const paths = [ + ["cost", "costBreakdown", "total", "costUSD"], + ["run", "totals", "costBreakdown", "total", "costUSD"], + ["run", "totals", "stage7SchemaRepair", "actualRepairCalls"], + ["telemetry", "schemaRepair", "stage7", "actualRepairCalls"], + ["run", "totals", "logOverflow", "droppedDebugInfo"], + ["telemetry", "logs", "bufferedOverflow", "droppedWarnError"], + ...["filesChanged", "hunks", "packets", "packetReviews", "candidates", "verified", "finalFindings", "postedComments"] + .map((field) => ["run", "totals", field]), + ["telemetry", "workers", "started"], + ["telemetry", "packets", "generated"], + ["telemetry", "lenses", "selected"], + ["telemetry", "coverage", "hunks", "reviewed"], + ["telemetry", "candidates", "generated"], + ["telemetry", "verdicts", "accept"], + ["telemetry", "dedup", "clusters"], + ["telemetry", "finalSelection", "published"], + ["telemetry", "posting", "attempted"] + ]; + for (const fieldPath of paths) { + const cohort = structuredClone(base); + const summary = cohort.runs[0]!.executions[0]!.summaryArtifacts as Record; + let owner = summary; + for (const key of fieldPath.slice(0, -1)) { + owner = owner[key] as Record; + } + const key = fieldPath.at(-1)!; + owner[key] = Number(owner[key]) + 1; + expect( + failureCodes(analyzeEvalCohort(cohort, 1).failures), + fieldPath.join(".") + ).toContain("paid_summary_reconciliation"); + } + }); + + it("reconstructs note-bearing repeats and rejects corruption in every persisted score section", () => { + const notes = abcCohort(10, (arm, index) => ({ hit: arm !== "B" || index > 0 })); + notes.runs[1]!.executions[0]!.scoringArtifacts.humanAttentionNotes = [{ + question: "Does the boundary bug affect app.ts?", + files: ["app.ts"], + reasons: ["boundary behavior"] + }]; + refreshRunEvidenceScores(notes.runs[1]!); + expect(notes.runs[1]!.executions[0]!.score.expectationResults[0]!.loss?.surfacedAsNote).toBe(true); + expect(failureCodes(analyzeEvalCohort(notes, 10).failures)).not.toEqual(expect.arrayContaining([ + "paid_evidence_score_reconstruction", + "paid_evidence_aggregate" + ])); + + const corruptions: Array<(score: EvalScore) => void> = [ + (persisted) => { persisted.nearViolations.push({ expectationId: "bug", findingId: "invented", artifact: "final-findings" }); }, + (persisted) => { persisted.budgetResults.push({ check: "maxFindings", status: "pass", limit: 99, actual: 1, direction: "maximum" }); }, + (persisted) => { persisted.metrics.reasoningTokens = 999; }, + (persisted) => { persisted.expectationResults[0]!.note = "corrupted aggregate note"; } + ]; + for (const corrupt of corruptions) { + const cohort = abcCohort(1); + corrupt(cohort.runs[1]!.executions[0]!.score); + expect(failureCodes(analyzeEvalCohort(cohort, 1).failures)).toContain("paid_evidence_score_reconstruction"); + } + + const aggregate = abcCohort(10); + aggregate.runs[1]!.info.score.metrics.reasoningTokens = 123; + expect(failureCodes(analyzeEvalCohort(aggregate, 10).failures)).toContain("paid_evidence_aggregate"); + }); + + it("validates every final-selection section and verifier-to-final lineage", () => { + const mutations: Array<(execution: EvalExecutionInput) => void> = [ + (target) => { target.finalSelectionArtifact.records = []; }, + (target) => { target.finalSelectionArtifact.groups = []; }, + (target) => { target.finalSelectionArtifact.publicationAnchors[0]!.fingerprint = "invented"; }, + (target) => { target.finalSelectionArtifact.confidenceSelections.push({ + findingId: "finding", confidence: "high", representativeConfidence: "medium", reason: "representative" + }); }, + (target) => { + const record = target.verification[0]!; + if ("verdict" in record) { + record.verdict.finalFinding = { ...structuredClone(target.candidateFindings[0]!), failureMode: "unrelated revised failure" }; + } + }, + (target) => { + const record = target.verification[0]!; + if ("verdict" in record) { + record.verdict.revisedAnchor = { path: "app.ts", hunkId: "h2", line: 1, side: "RIGHT" }; + } + }, + (target) => { + const record = target.verification[0]!; + record.gate = "gate_anchor_stripped"; + record.gateDecision = "suppressed"; + record.gateReason = "invalid_anchor; fabricated mixed state"; + }, + (target) => { + target.verification[0]!.gateFacts = { ...target.verification[0]!.gateFacts, changedLine: false }; + } + ]; + for (const [index, mutate] of mutations.entries()) { + const cohort = abcCohort(1); + mutate(cohort.runs[1]!.executions[0]!); + expect(failureCodes(analyzeEvalCohort(cohort, 1).failures), `mutation ${index}`).toContain("paid_evidence_relations"); + } + }); + + it("replays verifier gates from raw candidates instead of trusting collusive persisted decisions", () => { + const scheduledInvalid = abcCohort(1); + const scheduledRun = scheduledInvalid.runs[1]!; + scheduledRun.executions[0]!.candidateFindings[0]!.evidence.changedCode = ""; + refreshRunEvidenceScores(scheduledRun); + expect(failureCodes(analyzeEvalCohort(scheduledInvalid, 1).failures)).toContain("paid_evidence_relations"); + + const suppressedValid = abcCohort(1); + const suppressedRun = suppressedValid.runs[1]!; + const suppressedExecution = suppressedRun.executions[0]!; + suppressedExecution.verification = [{ + candidateId: "finding", + gate: "suppressed", + gateDecision: "suppressed", + gateReason: "missing_evidence" + }]; + suppressedExecution.finalSelection = []; + suppressedExecution.finalFindings = []; + refreshRunEvidenceScores(suppressedRun); + expect(failureCodes(analyzeEvalCohort(suppressedValid, 1).failures)).toContain("paid_evidence_relations"); + }); + + it("accepts production anchor backfill and withholds representative-only anchors from inline publication", () => { + for (const invalidAnchor of [false, true]) { + const cohort = abcCohort(1); + const run = cohort.runs[1]!; + const execution = run.executions[0]!; + const candidate = execution.candidateFindings[0]!; + candidate.changedLine = false; + candidate.modelAnchorSubmitted = invalidAnchor; + if (invalidAnchor) { + candidate.anchor = { path: "app.ts", line: 999, side: "RIGHT", hunkId: "h1" }; + candidate.anchorSource = "model"; + execution.verification[0]!.gate = "gate_anchor_stripped"; + } else { + delete candidate.anchor; + delete candidate.anchorSource; + } + refreshRunEvidenceScores(run); + expect(execution.finalFindings[0]!.publication).toBe("summary-only"); + expect(execution.finalFindings[0]!.anchor).toBeUndefined(); + expect(execution.finalSelection[0]!.reason).toBe("unanchorable"); + expect(failureCodes(analyzeEvalCohort(cohort, 1).failures)).not.toContain("paid_evidence_relations"); + } + }); + + it("supports production verify:false duplicate records without persisted cluster metadata", () => { + const cohort = abcCohort(1); + const run = cohort.runs[1]!; + run.info.effectiveConfig!.review.verify = false; + run.declaredCase.review!.verify = false; + run.info.caseSnapshot.review!.verify = false; + const execution = run.executions[0]!; + const duplicate = { ...structuredClone(execution.candidateFindings[0]!), id: "finding-2" }; + execution.candidateFindings.push(duplicate); + const primaryRecord = execution.verification[0]!; + if (!("verdict" in primaryRecord)) throw new Error("expected verifier record"); + primaryRecord.verdict = { + candidateId: "finding", + verdict: "keep", + reason: "verification disabled by config", + requiredEvidencePresent: true, + falsePositiveRisk: "low" + }; + execution.verification.push({ + ...structuredClone(primaryRecord), + candidateId: duplicate.id, + verdict: { ...structuredClone(primaryRecord.verdict), candidateId: duplicate.id } + }); + execution.finalFindings[0]!.mergedCandidateIds = ["finding", duplicate.id]; + execution.finalSelection.push({ findingId: duplicate.id, decision: "merged", reason: "composer-merged" }); + refreshRunEvidenceScores(run); + expect(execution.verification.every((record) => record.duplicateOf === undefined && record.clusterId === undefined)).toBe(true); + expect(failureCodes(analyzeEvalCohort(cohort, 1).failures)).not.toContain("paid_evidence_relations"); + }); + + it("independently applies composer thresholds, caps, and deterministic ordering", () => { + const thresholdCases: Array<(run: EvalCaseRunInput) => void> = [ + (run) => { run.info.effectiveConfig!.review.minSeverity = "high"; }, + (run) => { run.info.effectiveConfig!.review.minConfidence = "high"; }, + (run) => { run.info.effectiveConfig!.review.minInlineConfidence = "high"; } + ]; + for (const mutatePolicy of thresholdCases) { + const cohort = abcCohort(1); + mutatePolicy(cohort.runs[1]!); + expect(failureCodes(analyzeEvalCohort(cohort, 1).failures)).toContain("paid_evidence_relations"); + } + + const multi = abcCohort(1); + const multiRun = multi.runs[1]!; + const execution = multiRun.executions[0]!; + const packetWithH2 = execution.packets.find((target) => target.hunks.some((hunk) => hunk.hunkId === "h2"))!; + const second = candidateFinding("finding-2", packetWithH2.id, "h2", 101); + second.title = "Independent second boundary defect"; + const evidence = findingEvidence(second); + execution.candidateFindings.push(...evidence.candidateFindings); + execution.verification.push(...evidence.verification); + execution.finalSelection.push(...evidence.finalSelection); + execution.finalFindings.push(...evidence.finalFindings); + refreshRunEvidenceScores(multiRun); + expect(failureCodes(analyzeEvalCohort(multi, 1).failures)).not.toContain("paid_evidence_relations"); + + const softCap = structuredClone(multi); + softCap.runs[1]!.info.effectiveConfig!.review.softCommentCap = 1; + expect(failureCodes(analyzeEvalCohort(softCap, 1).failures)).toContain("paid_evidence_relations"); + + const reportCap = structuredClone(multi); + reportCap.runs[1]!.info.effectiveConfig!.review.maxFindings = 1; + expect(failureCodes(analyzeEvalCohort(reportCap, 1).failures)).toContain("paid_evidence_relations"); + + const reordered = structuredClone(multi); + reordered.runs[1]!.executions[0]!.finalFindings.reverse(); + expect(failureCodes(analyzeEvalCohort(reordered, 1).failures)).toContain("paid_evidence_relations"); + }); + + it("enforces aggregate recall and atom-count loss attribution instead of allowing one loss per expectation", () => { + const cohort = abcCohort(10); + for (const run of cohort.runs) { + run.declaredCase.should_find![0]!.titlePattern = "^Boundary bug$"; + run.declaredCase.should_find!.push({ id: "bug-2", path: "app.ts", titlePattern: "^Secondary boundary bug$" }); + for (const execution of run.executions) { + const packetWithH2 = execution.packets.find((target) => target.hunks.some((hunk) => hunk.hunkId === "h2"))!; + const candidate = candidateFinding("finding-2", packetWithH2.id, "h2", 1); + candidate.title = "Secondary boundary bug"; + const evidence = findingEvidence(candidate); + execution.candidateFindings.push(...evidence.candidateFindings); + execution.verification.push(...evidence.verification); + execution.finalSelection.push(...evidence.finalSelection); + execution.finalFindings.push(...evidence.finalFindings); + } + refreshRunEvidenceScores(run); + } + for (const [repeatIndex, findingId] of [[0, "finding"], [1, "finding-2"]] as const) { + const execution = cohort.runs[1]!.executions[repeatIndex]!; + execution.candidateFindings = execution.candidateFindings.filter((finding) => finding.id !== findingId); + execution.verification = execution.verification.filter((record) => record.candidateId !== findingId); + execution.finalSelection = execution.finalSelection.filter((record) => record.findingId !== findingId); + execution.finalFindings = execution.finalFindings.filter((finding) => finding.id !== findingId); + } + refreshRunEvidenceScores(cohort.runs[1]!); + const codes = failureCodes(analyzeEvalCohort(cohort, 10).failures); + expect(codes).toContain("recall_non_inferiority"); + + const bucketed = abcCohort(10, (arm, index) => ({ hit: arm !== "B" || index > 1 })); + expect(failureCodes(analyzeEvalCohort(bucketed, 10).failures)).toContain("candidate_generation_loss_by_atom_count"); + }); + + it("spends the single recall and atom-loss allowance across the entire multi-case cohort", () => { + const runs = [ + ...abcCohort(10, (arm, index) => ({ hit: arm !== "B" || index > 0 }), "cohort-loss-one", 0).runs, + ...abcCohort(10, (arm, index) => ({ hit: arm !== "B" || index > 0 }), "cohort-loss-two", 3).runs + ]; + const codes = failureCodes(analyzeEvalCohort({ id: "cohort-loss", runs }, 10).failures); + expect(codes).toEqual(expect.arrayContaining([ + "recall_non_inferiority", + "candidate_generation_loss", + "candidate_generation_loss_by_atom_count" + ])); + }); + + it("keeps all executions in intent-to-treat and excludes untreated runs from both treated-only numerator and denominator", () => { + const report = analyzeEvalCohort(abcCohort(10, (arm, index) => ({ + treated: arm === "B" ? index !== 0 : arm !== "A", + hit: arm === "B" ? index !== 1 : true + })), 10); + expect(report.failures).toEqual([]); + const b = report.cases?.[0]?.arms.B; + expect(b?.intentToTreat[0]).toMatchObject({ denominator: 10, candidateHits: 9, finalHits: 9 }); + expect(b?.treatedOnly.rates[0]).toMatchObject({ denominator: 9, candidateHits: 8, finalHits: 8 }); + expect(b?.treatedOnly.denominatorByExpectation["should_find:bug"]).toBe(9); + }); + + it("fails closed below 8/10 treatment and on missing telemetry", () => { + const underTreated = analyzeEvalCohort(abcCohort(10, (arm, index) => ({ + treated: arm === "B" ? index < 7 : arm !== "A" + })), 10); + expect(failureCodes(underTreated.failures)).toContain("insufficient_treatment"); + + const missing = analyzeEvalCohort(abcCohort(1, (arm) => ({ missingTelemetry: arm === "C" })), 1); + expect(failureCodes(missing.failures)).toEqual(expect.arrayContaining(["missing_treatment_telemetry", "insufficient_treatment"])); + }); + + it("fails closed on recall regression and missing spend data", () => { + const recall = analyzeEvalCohort(abcCohort(10, (arm, index) => ({ + hit: arm === "B" ? index < 7 : true + })), 10); + expect(failureCodes(recall.failures)).toEqual(expect.arrayContaining(["recall_non_inferiority"])); + + const noSpend = abcCohort(1, (arm) => arm === "C" ? { cost: undefined } : {}); + delete noSpend.runs[2]!.executions[0]!.score.metrics.costUSD; + noSpend.runs[2]!.executions[0]!.score.budgetResults = []; + const spendReport = analyzeEvalCohort(noSpend, 1); + expect(failureCodes(spendReport.failures)).toContain("missing_spend_data"); + }); + + it("writes a failure report when eval artifacts cannot be loaded", async () => { + const root = await mkdtemp(path.join(tmpdir(), "packet-report-missing-eval-")); + tempDirs.push(root); + const output = path.join(root, "report.json"); + const exitCode = await runPacketPackingReportCli([ + "eval", + "--logs", + path.join(root, "missing"), + "--expected-repeats", + "1", + "--output", + output + ]); + expect(exitCode).toBe(1); + const report = JSON.parse(await readFile(output, "utf8")) as { failures: ReportFailure[] }; + expect(failureCodes(report.failures)).toContain("missing_logs"); + }); + + it("projects replay, eval, and regression failures through one prose-safe output boundary", async () => { + const root = await mkdtemp(path.join(tmpdir(), "packet-report-safe-output-")); + tempDirs.push(root); + const secret = "PROPRIETARY_FOCUS_AND_CREDENTIAL_NEVER_EMIT"; + const replayRun = path.join(root, secret); + await mkdir(path.join(replayRun, "stages", "01-input"), { recursive: true }); + await writeFile(path.join(replayRun, "stages", "01-input", "resolved-input.json"), "{bad-json\n"); + const invocations: Array<{ argv: string[]; output: string }> = [ + { + argv: ["replay", "--repo", process.cwd(), "--run", replayRun, "--output", path.join(root, "replay.json")], + output: path.join(root, "replay.json") + }, + { + argv: ["eval", "--logs", path.join(root, secret, "eval"), "--expected-repeats", "1", "--output", path.join(root, "eval.json")], + output: path.join(root, "eval.json") + }, + { + argv: [ + "regression", + "--baseline-logs", path.join(root, secret, "baseline"), + "--selected-logs", path.join(root, secret, "selected"), + "--expected-repeats", "1", + "--output", path.join(root, "regression.json") + ], + output: path.join(root, "regression.json") + } + ]; + for (const invocation of invocations) { + expect(await runPacketPackingReportCli(invocation.argv)).toBe(1); + const serialized = await readFile(invocation.output, "utf8"); + expect(serialized).not.toContain(secret); + expect(serialized).toContain("sha256:"); + expect((JSON.parse(serialized) as { failures: ReportFailure[] }).failures.length).toBeGreaterThan(0); + } + }); + + it("fingerprints argument, top-level, and report-write failures without raw stderr prose", async () => { + const root = await mkdtemp(path.join(tmpdir(), "packet-report-safe-top-level-")); + tempDirs.push(root); + const secret = "PROPRIETARY_TOP_LEVEL_MESSAGE_NEVER_EMIT"; + const parseOutput = path.join(root, "parse.json"); + expect(await runPacketPackingReportCli([ + "eval", "--logs", path.join(root, secret), "--unsupported", secret, + "--output", parseOutput + ])).toBe(1); + const parseReport = await readFile(parseOutput, "utf8"); + expect(parseReport).not.toContain(secret); + expect(parseReport).toContain("sha256:"); + + const stderrChunks: string[] = []; + const stderr = vi.spyOn(process.stderr, "write").mockImplementation(((chunk: string | Uint8Array) => { + stderrChunks.push(String(chunk)); + return true; + }) as typeof process.stderr.write); + try { + expect(await runPacketPackingReportCli(["unknown-mode", secret])).toBe(1); + const unwritableOutput = path.join(root, secret); + await mkdir(unwritableOutput); + expect(await runPacketPackingReportCli([ + "eval", "--logs", path.join(root, "missing"), "--expected-repeats", "1", "--output", unwritableOutput + ])).toBe(1); + } finally { + stderr.mockRestore(); + } + const stderrOutput = stderrChunks.join(""); + expect(stderrOutput).not.toContain(secret); + expect(stderrOutput).toContain("sha256:"); + }); +}); + +describe("production economics", () => { + function productionExecution(arm: "A" | "B", wallTimeSeconds: number, cost: number): EvalExecutionInput { + const hunkIds = Array.from({ length: 142 }, (_, index) => `production-hunk-${index + 1}`); + const atoms = hunkIds.map((hunkId, index) => packet(`production-atom-${index + 1}`, [hunkId])); + const packets = arm === "A" + ? atoms + : Array.from({ length: Math.ceil(atoms.length / 5) }, (_, index) => { + const members = atoms.slice(index * 5, index * 5 + 5); + return packet(`production-packed-${index + 1}`, members.flatMap((member) => member.hunks.map((hunk) => hunk.hunkId))); + }); + const events = arm === "A" ? [] : packets.map((target, index) => packingEvent(target, atoms.slice(index * 5, index * 5 + 5))); + const executionScore = score(true, cost); + const callCost = cost / (packets.length + 3); + const packetCalls = packets.map((target) => modelCall(target.id, 1, callCost)); + const planner = { ...modelCall(packets[0]!.id, 1, callCost), callId: `production-planner-${arm}`, stage: 5 as const, role: "planner" as const }; + delete (planner as Partial).packetId; + const verifier = { + ...modelCall(packets[0]!.id, 1, callCost), + callId: `production-verifier-${arm}`, + stage: 9 as const, + role: "verifier" as const, + kind: "repair" as const, + candidateId: "finding" + }; + delete (verifier as Partial).packetId; + const composer = { ...modelCall(packets[0]!.id, 1, callCost), callId: `production-composer-${arm}`, stage: 10 as const, role: "composer" as const }; + delete (composer as Partial).packetId; + const allCalls = [...packetCalls, planner, verifier, composer]; + executionScore.metrics.modelCalls = allCalls.length; + executionScore.metrics.verificationCalls = 1; + executionScore.metrics.elapsedSeconds = wallTimeSeconds; + const candidate = candidateFinding("finding", packets[0]!.id, hunkIds[0]!); + const result = { + repeat: 1, + score: executionScore, + telemetryDir: `/tmp/production/${arm}`, + packets, + events, + modelCalls: allCalls, + toolCalls: [], + fileFacts: [{ + path: "app.ts", + language: "typescript", + processingMode: "per-hunk", + testStatus: "source", + isGenerated: false, + isVendored: false, + isLockfile: false, + isBinary: false, + changedLines: 142, + hunkCount: 142, + labels: [], + reviewPriority: "normal", + reasons: [], + provenance: [] + }], + diff: diffForPackets(atoms), + plan: planForHunks(hunkIds), + ...findingEvidence(candidate), + reviewedHunkIds: hunkIds, + wallTimeSeconds + } as unknown as EvalExecutionInput; + refreshExecutionEvidence(result); + return result; + } + + function productionCohort(): CohortSelection { + const cohort = { + id: "1-2", + runs: [ + evalRun("production", "A", 1, [productionExecution("A", 200, 20)]), + evalRun("production", "B", 2, [productionExecution("B", 100, 10)]) + ] + }; + for (const run of cohort.runs) { + run.declaredCase.repo = { external: "/home/peter/Dev/0xsequence/trails-api" }; + run.declaredCase.command = { base: "d1c49bdf6a8002ec2ec27faac94a932d736532b2", head: "fbb5f8761c2c296e115af17e919a7c35d9de8373" }; + run.declaredCase.review = { + ...run.declaredCase.review, + cache: false, + concurrency: 6, + maxTimeMinutes: 60 + }; + run.info.repo = { + root: "/home/peter/Dev/0xsequence/trails-api", + baseSha: "d1c49bdf6a8002ec2ec27faac94a932d736532b2", + mergeBase: "d1c49bdf6a8002ec2ec27faac94a932d736532b2", + headSha: "fbb5f8761c2c296e115af17e919a7c35d9de8373" + }; + run.info.effectiveConfig!.review.concurrency = 6; + run.info.effectiveConfig!.review.timeoutMs = 3_600_000; + run.info.effectiveConfig!.llm.maxConcurrentCalls = 6; + run.info.codegenieRuntime = { packageVersion: "0.5.0", commit: "a".repeat(40), dirty: false, source: "git" }; + } + return cohort; + } + + it("reports raw and normalized 142-hunk economics with extrapolation and break-even", () => { + const failures: ReportFailure[] = []; + const economics = computeProductionEconomics( + { actualCostUSD: 20, reviewedHunks: 100 }, + { actualCostUSD: 20, reviewedHunks: 142 }, + 80, + "explicit_cumulative_validation_cost", + failures + ); + expect(failures).toEqual([]); + expect(economics).toMatchObject({ + equivalentTargetHunks: 142, + equivalentReviewSavingsUSD: 8.4, + breakEvenReviewCount: 10, + baseline: { equivalentCostExtrapolated: true }, + selected: { equivalentCostExtrapolated: false } + }); + }); + + it("fails closed on missing reviewed work and a non-positive payback denominator", () => { + const failures: ReportFailure[] = []; + computeProductionEconomics( + { actualCostUSD: 10, reviewedHunks: 0 }, + { actualCostUSD: 20, reviewedHunks: 100 }, + 50, + "explicit_cumulative_validation_cost", + failures + ); + expect(failureCodes(failures)).toEqual(expect.arrayContaining(["missing_reviewed_hunks", "non_positive_payback_denominator"])); + + const noSavings: ReportFailure[] = []; + computeProductionEconomics( + { actualCostUSD: 10, reviewedHunks: 100 }, + { actualCostUSD: 20, reviewedHunks: 100 }, + 50, + "explicit_cumulative_validation_cost", + noSavings + ); + expect(failureCodes(noSavings)).toContain("non_positive_payback_denominator"); + }); + + it("gates production on the exact 142-hunk set and measured wall/model/token capacity", () => { + const report = analyzeEvalCohort(productionCohort(), 1, { actualValidationCostUSD: 30 }); + expect(report.failures).toEqual([]); + const cohort = productionCohort(); + const baselineCalls = cohort.runs[0]!.executions[0]!.modelCalls; + const allStageReport = analyzeEvalCohort(cohort, 1, { actualValidationCostUSD: 30 }); + expect(report.productionThroughput).toMatchObject({ + baseline: { reviewedHunks: 142, wallTimeSeconds: 200 }, + selected: { reviewedHunks: 142, wallTimeSeconds: 100 } + }); + expect(allStageReport.productionThroughput?.baseline).toMatchObject({ + modelServiceSeconds: baselineCalls.reduce((total, call) => total + call.durationMs, 0) / 1000, + totalTokens: baselineCalls.reduce((total, call) => total + call.totalTokens!, 0), + reasoningTokens: 0, + reasoningTokensPerReviewedHunk: 0 + }); + + const truncated = productionCohort(); + truncated.runs[1]!.executions[0]!.reviewedHunkIds.pop(); + const codes = failureCodes(analyzeEvalCohort(truncated, 1).failures); + expect(codes).toEqual(expect.arrayContaining(["production_hunk_loss", "production_incomplete"])); + }); + + it("gates reasoning tokens per hunk and continuation savings using all-stage service time", () => { + const reasoning = productionCohort(); + reasoning.runs[1]!.executions[0]!.modelCalls[0]!.reasoningTokens = 100; + refreshRunEvidenceScores(reasoning.runs[1]!); + expect(failureCodes(analyzeEvalCohort(reasoning, 1).failures)).toContain("production_reasoning_tokens"); + + const continuation = productionCohort(); + const selectedExecution = continuation.runs[1]!.executions[0]!; + selectedExecution.modelCalls.push({ + ...modelCall(selectedExecution.packets[0]!.id, 1, 0), + callId: "production-added-continuation", + kind: "tool-continuation", + attempt: 2, + durationMs: 100_000 + }); + refreshRunEvidenceScores(continuation.runs[1]!); + expect(failureCodes(analyzeEvalCohort(continuation, 1).failures)).toContain("production_continuation_savings_retention"); + }); + + it("fails closed when production model telemetry is coverage-only or does not strictly improve capacity", () => { + const coverageOnly = productionCohort(); + delete coverageOnly.runs[1]!.executions[0]!.modelCalls[0]!.totalTokens; + expect(failureCodes(analyzeEvalCohort(coverageOnly, 1).failures)).toContain("production_model_accounting"); + + const equalCapacity = productionCohort(); + const selectedCalls = equalCapacity.runs[1]!.executions[0]!.modelCalls; + const baselineCalls = equalCapacity.runs[0]!.executions[0]!.modelCalls; + let remainingTokens = baselineCalls.reduce((total, call) => total + call.totalTokens!, 0); + const baselineServiceMs = baselineCalls.reduce((total, call) => total + call.durationMs, 0); + for (const [index, call] of selectedCalls.entries()) { + const totalTokens = index === selectedCalls.length - 1 ? remainingTokens : Math.floor(remainingTokens / (selectedCalls.length - index)); + remainingTokens -= totalTokens; + call.durationMs = baselineServiceMs / selectedCalls.length; + call.inputTokens = totalTokens - 10; + call.outputTokens = 10; + call.totalTokens = totalTokens; + } + expect(failureCodes(analyzeEvalCohort(equalCapacity, 1).failures)).toEqual(expect.arrayContaining([ + "production_model_service", + "production_tokens" + ])); + }); + + it("requires pinned clean production provenance and exact finding-artifact preservation", () => { + const cohort = productionCohort(); + cohort.runs[1]!.info.repo!.headSha = "c".repeat(40); + cohort.runs[1]!.info.codegenieRuntime!.dirty = true; + cohort.runs[1]!.executions[0]!.finalFindings = []; + cohort.runs[1]!.executions[0]!.score.expectationResults[0]!.status = "fail"; + const codes = failureCodes(analyzeEvalCohort(cohort, 1).failures); + expect(codes).toEqual(expect.arrayContaining([ + "production_refs", + "production_runtime_provenance", + "production_finding_artifact_join", + "production_finding_preservation" + ])); + }); + + it("requires the exact production repo, cache, concurrency, and 60-minute declared/effective shape", () => { + const cohort = productionCohort(); + cohort.runs[1]!.declaredCase.repo!.external = "/tmp/wrong-repo"; + cohort.runs[1]!.declaredCase.review!.cache = true; + cohort.runs[1]!.declaredCase.review!.concurrency = 1; + cohort.runs[1]!.declaredCase.review!.maxTimeMinutes = 1; + cohort.runs[1]!.info.effectiveConfig!.review.timeoutMs = 60_000; + const codes = failureCodes(analyzeEvalCohort(cohort, 1).failures); + expect(codes).toEqual(expect.arrayContaining(["production_refs", "production_run_shape"])); + }); + + it("requires nonempty required matches and relationally complete production finding artifacts", () => { + const emptyMatch = productionCohort(); + emptyMatch.runs[1]!.executions[0]!.score.expectationResults[0]!.matched = []; + expect(failureCodes(analyzeEvalCohort(emptyMatch, 1).failures)).toContain("production_finding_preservation"); + + for (const mutate of [ + (execution: EvalExecutionInput) => { execution.candidateFindings[0]!.producedBy.packetId = "unknown-packet"; }, + (execution: EvalExecutionInput) => { execution.finalFindings[0]!.mergedCandidateIds = ["unknown-candidate"]; }, + (execution: EvalExecutionInput) => { execution.finalFindings[0]!.anchor = { hunkId: "unknown-hunk", path: "app.ts", line: 1, side: "RIGHT" }; }, + (execution: EvalExecutionInput) => { execution.score.metrics.candidateFindings = 99; } + ]) { + const cohort = productionCohort(); + mutate(cohort.runs[1]!.executions[0]!); + expect(failureCodes(analyzeEvalCohort(cohort, 1).failures)).toContain("production_finding_relations"); + } + for (const mutate of [ + (execution: EvalExecutionInput) => { execution.verification = []; }, + (execution: EvalExecutionInput) => { execution.finalSelection = []; } + ]) { + const cohort = productionCohort(); + mutate(cohort.runs[1]!.executions[0]!); + expect(failureCodes(analyzeEvalCohort(cohort, 1).failures)).toContain("paid_evidence_relations"); + } + }); +}); + +describe("collateral regression analysis", () => { + function regressionPair(selectedHit = true, selectedError = false) { + const baseline = evalRun("fixture", "A", 1, [execution("A", 1)]); + const selected = evalRun("fixture", "B", 1, [execution("B", 1, { hit: selectedHit })]); + baseline.info.caseSnapshot.name = "fixture-baseline"; + baseline.declaredCase.name = "fixture-baseline"; + baseline.info.caseName = "fixture"; + baseline.info.caseFile = "fixture.yml"; + selected.info.caseSnapshot.name = "fixture-selected"; + selected.declaredCase.name = "fixture-selected"; + selected.info.caseName = "fixture"; + selected.info.caseFile = "fixture.yml"; + selected.info.score = selected.executions[0]!.score; + if (selectedError) { + selected.info.score = { ...selected.info.score, status: "error", error: { code: "llm_call_failed", message: "boom" } }; + } + return { + baseline: { id: "1-1", runs: [baseline] }, + selected: { id: "1-1", runs: [selected] } + }; + } + + it("compares explicit roots as one-repeat collateral evidence", () => { + const pair = regressionPair(); + const report = analyzeRegressionCohorts(pair.baseline, pair.selected, 1); + expect(report.failures).toEqual([]); + expect(report).toMatchObject({ + evidence: "one-repeat-collateral-only", + baselineCohort: { actualCostUSD: 1 }, + selectedCohort: { actualCostUSD: 1 } + }); + }); + + it("accepts a valid collateral case where packing is enabled but no atom is treated", () => { + const pair = regressionPair(); + pair.selected.runs[0]!.executions[0]!.packets = structuredClone(pair.baseline.runs[0]!.executions[0]!.packets); + pair.selected.runs[0]!.executions[0]!.events = []; + const selectedExecution = pair.selected.runs[0]!.executions[0]!; + selectedExecution.candidateFindings[0]!.producedBy.packetId = selectedExecution.packets[0]!.id; + selectedExecution.finalFindings[0]!.producedBy.packetId = selectedExecution.packets[0]!.id; + refreshExecutionEvidence(selectedExecution); + const report = analyzeRegressionCohorts(pair.baseline, pair.selected, 1); + expect(report.failures).toEqual([]); + expect(report.cases[0]?.selectedTreatmentExecutions).toBe(0); + }); + + it("fails closed on collateral expectation transitions and eval errors", () => { + const regressed = regressionPair(false); + const report = analyzeRegressionCohorts(regressed.baseline, regressed.selected, 1); + expect(failureCodes(report.failures)).toContain("collateral_expectation_regression"); + + const errored = regressionPair(true, true); + expect(failureCodes(analyzeRegressionCohorts(errored.baseline, errored.selected, 1).failures)).toContain("eval_error"); + + const mutuallyAlteredScore = regressionPair(); + mutuallyAlteredScore.selected.runs[0]!.info.score.expectationResults[0]!.expectationId = "score-only-expectation"; + expect(failureCodes(analyzeRegressionCohorts(mutuallyAlteredScore.baseline, mutuallyAlteredScore.selected, 1).failures)).toContain("expectation_join"); + }); + + it("fails when both collateral cohorts identically miss a requirement and when relational evidence is deleted", () => { + const bothMiss = regressionPair(false); + const baselineExecution = bothMiss.baseline.runs[0]!.executions[0]!; + baselineExecution.candidateFindings = []; + baselineExecution.verification = []; + baselineExecution.finalSelection = []; + baselineExecution.finalFindings = []; + refreshRunEvidenceScores(bothMiss.baseline.runs[0]!); + const bothMissReport = analyzeRegressionCohorts(bothMiss.baseline, bothMiss.selected, 1); + expect(bothMissReport.cases[0]!.expectationTransitions).toEqual([]); + expect(failureCodes(bothMissReport.failures)).toContain("collateral_cohort_expectation_failure"); + + const deleted = regressionPair(); + deleted.selected.runs[0]!.executions[0]!.verification = []; + expect(failureCodes(analyzeRegressionCohorts(deleted.baseline, deleted.selected, 1).failures)).toContain("paid_evidence_relations"); + }); + + it("fails closed when baseline and selected YAML differ beyond allowed experiment fields", () => { + const pair = regressionPair(); + pair.selected.runs[0]!.declaredCase.review!.depth = "deep"; + const report = analyzeRegressionCohorts(pair.baseline, pair.selected, 1); + expect(failureCodes(report.failures)).toContain("regression_yaml_delta"); + }); +}); From 7ebfd2f95325ba758940ec1d301f0960e4475335 Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 08:59:46 -0400 Subject: [PATCH 03/24] fix(evals): repair packet-packing preflight validation --- scripts/packet-packing-report.ts | 45 ++++++++-------- .../102-issue-102-same-file-packet-packing.md | 50 ++++++++++++++++++ specs/plans/phase-plans/102/phase_3.md | 30 +++++++++++ tests/packet-packing-report.test.ts | 52 +++++++++++++++++++ 4 files changed, 155 insertions(+), 22 deletions(-) create mode 100644 specs/plans/phase-plans/102/phase_3.md diff --git a/scripts/packet-packing-report.ts b/scripts/packet-packing-report.ts index 3e82fb2..07b658c 100644 --- a/scripts/packet-packing-report.ts +++ b/scripts/packet-packing-report.ts @@ -2511,15 +2511,11 @@ function analyzeTreatmentExecution( ) { failures.push(failure("requested_lens_join", `packet ${packetId} requested-lens telemetry does not match Stage-5 decisions for its A atoms`)); } - const aRequestedLenses = sortedUnique(resolvedAtoms.flatMap((atom) => requestedLensesForAtom(baselineExecution, atom) ?? [])); + // Stage 6 may validly prune a requested lens before constructing the A + // atom. Packing must preserve that routed A surface exactly; it must not + // reinterpret intentional A-side routing as a packing regression. const aRoutedLenses = sortedUnique(resolvedAtoms.flatMap((atom) => atom.lenses)); const selectedRoutedLenses = sortedUnique(packet.lenses); - const missingFromA = aRequestedLenses.filter((lens) => !aRoutedLenses.includes(lens)); - if (missingFromA.length > 0) { - failures.push(failure("a_lens_route_join", `A atoms omit one or more of their explicit Stage-5 lenses`, { - missingLensHashes: missingFromA.map((lens) => sha256Hex(lens)) - })); - } if (stableJson(selectedRoutedLenses) !== stableJson(aRoutedLenses)) { const missingRoutedLenses = aRoutedLenses.filter((lens) => !selectedRoutedLenses.includes(lens)); const extraRoutedLenses = selectedRoutedLenses.filter((lens) => !aRoutedLenses.includes(lens)); @@ -4330,6 +4326,7 @@ function attentionRelationsMatch( records: Array>, execution: EvalExecutionInput ): boolean { + const packetById = new Map(execution.packets.map((packet) => [packet.id, packet])); const candidatePacketById = new Map(execution.candidateFindings.map((candidate) => [candidate.id, candidate.producedBy.packetId])); const countByPacket = (predicate: (candidate: CandidateFinding) => boolean): NumericRecord => { const counts: NumericRecord = {}; @@ -4353,20 +4350,21 @@ function attentionRelationsMatch( for (const packetId of packetIds) published[packetId] = (published[packetId] ?? 0) + 1; } const plannedHunkIds = new Set(execution.plan.coverage.map((decision) => decision.hunkId)); - return records.length === execution.packets.length && records.every((record, index) => { - const packet = execution.packets[index]; - if (packet === undefined) return false; - const coverageSource = packet.coverageEscalation !== undefined - ? `escalated:${packet.coverageEscalation.rule}` - : packet.hunks.some((hunk) => hunk.plannerFallbackReason !== undefined) || - packet.hunks.every((hunk) => !plannedHunkIds.has(hunk.hunkId)) - ? "deterministic_default" - : "planner"; - return record.packetId === packet.id && record.path === packet.path && record.coverage === packet.coverage && - record.coverageSource === coverageSource && record.ensemblePasses >= 1 && - record.directCandidates === (direct[packet.id] ?? 0) && record.promotedCandidates === (promoted[packet.id] ?? 0) && - record.keptVerified === (kept[packet.id] ?? 0) && record.published === (published[packet.id] ?? 0); - }); + return packetById.size === execution.packets.length && records.length === execution.packets.length && + unique(records.map((record) => record.packetId)).length === records.length && records.every((record) => { + const packet = packetById.get(record.packetId); + if (packet === undefined) return false; + const coverageSource = packet.coverageEscalation !== undefined + ? `escalated:${packet.coverageEscalation.rule}` + : packet.hunks.some((hunk) => hunk.plannerFallbackReason !== undefined) || + packet.hunks.every((hunk) => !plannedHunkIds.has(hunk.hunkId)) + ? "deterministic_default" + : "planner"; + return record.packetId === packet.id && record.path === packet.path && record.coverage === packet.coverage && + record.coverageSource === coverageSource && record.ensemblePasses >= 1 && + record.directCandidates === (direct[packet.id] ?? 0) && record.promotedCandidates === (promoted[packet.id] ?? 0) && + record.keptVerified === (kept[packet.id] ?? 0) && record.published === (published[packet.id] ?? 0); + }); } function humanAttentionRelationsMatch(input: z.infer): boolean { @@ -4535,8 +4533,11 @@ function validateExecutionEvidence(run: EvalCaseRunInput, execution: EvalExecuti for (const candidate of execution.candidateFindings) { const packet = packetById.get(candidate.producedBy.packetId); const record = verificationById.get(candidate.id); + const producerStageValid = candidate.provenance?.source === "uncertainty_promotion" + ? candidate.producedBy.stage === 9 && candidate.provenance.sourcePacketId === candidate.producedBy.packetId + : candidate.producedBy.stage === 7 || candidate.producedBy.stage === 8; const producerValid = packet !== undefined && candidate.producedBy.kind === "packet" && - (candidate.producedBy.stage === 7 || candidate.producedBy.stage === 8) && candidate.producedBy.lensId.trim().length > 0 && + producerStageValid && candidate.producedBy.lensId.trim().length > 0 && packet.lenses.includes(candidate.producedBy.lensId) && candidate.producedBy.skillIds.every((id) => id.trim().length > 0); const provenanceValid = candidate.provenance === undefined || ( packetById.has(candidate.provenance.sourcePacketId) && candidate.provenance.question.trim().length > 0 && candidate.provenance.reason.trim().length > 0 && diff --git a/specs/plans/102-issue-102-same-file-packet-packing.md b/specs/plans/102-issue-102-same-file-packet-packing.md index 7f49543..1358f33 100644 --- a/specs/plans/102-issue-102-same-file-packet-packing.md +++ b/specs/plans/102-issue-102-same-file-packet-packing.md @@ -13,6 +13,56 @@ Recommended priority: next throughput plan. Plan 100 is complete, and run `dca8d > Working-tree check: `git status --short -- src/pipeline/packet-builder.ts src/config/schema.ts src/config/config-loader.ts src/types.ts src/evals/eval-runner.ts scripts/packet-packing-report.ts tests/fixtures/packet-packing-golden.json tests/pipeline-phase5.test.ts tests/config-loader.test.ts tests/evals.test.ts tests/packet-packing-report.test.ts` > If the drift check reports committed changes, reconcile the current-state claims below. If the working-tree check reports changes owned by another task, stop and wait for that work to land or move this plan to an isolated worktree; do not overwrite it. +## Phase 3 paid-validation authorization (2026-07-25; before first paid call) + +The owner supplied `approvedValidationCostUSD: $500`, which is the active total validation ceiling and supersedes the earlier `$200` authorization communicated for this phase. Before the one-repeat step-7 preflight, `actualValidationCostUSD = $0.00` and `projectedRemainingCostUSD = $60.00`, so `actualValidationCostUSD + projectedRemainingCostUSD = $60.00 <= $500.00`. The `$60.00` preflight reservation assumes six paid A/B/C executions capped at `expect.maxCostUSD: 10` each; the fixture cases use the same explicit provider/model/reasoning/concurrency/depth/lens/time/token configuration and run with cache disabled. Authorization in this phase is limited to this one-repeat suite. Repeat-10, collateral, production-capacity, arm selection, rollout, and teardown remain `not_run` and are not authorized by this phase. + +After the preflight, replace the reservation with reconstructed cohort spend, set the repeat-10 projection to exactly `10 * preflightCostUSD`, and record cumulative actual plus projected spend. The later production phase continues to reserve `$95` only when reached and separately authorized; real-model collateral selected-side cost remains deferred until its A-side measurement, as required by this plan. + +### Phase 3 one-repeat invalid cohort, diagnosis, and deterministic repair (2026-07-25) + +The private suite was created at `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/recall/` with materializable `dilution-control` and `cross-atom-consistency` Go fixtures plus `dilution-{a,b,c}.yml` and `consistency-{a,b,c}.yml`. All six cases remain at `repeat: 1`. Within each family the current definitions are identical except for the arm name suffix and the two permitted review settings: A is packing off/base, B is packing on/base, and C is packing on/atom-scaled. They use Anthropic `claude-opus-4-8`, reasoning `high`, normal depth, concurrency/provider concurrency 1, a 20-minute case limit, a 1,000,000-token budget, `expect.maxCostUSD: 10`, and cache disabled. + +The first paid cohort used the original three-lens configuration (`core/code-review`, `core/tests`, and `lang/go`). Its cache-off invocation was `ace65769-3a68-4bd1-bd36-662fa6827dc6`, started `2026-07-25T12:02:33.511Z` and completed `2026-07-25T12:23:15.259Z`. Its exact log root is `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/recall/logs`; the same ID is the selected cohort. It completed six live executions with zero eval/provider/configuration errors and 84 model calls: + +| Run | Case | Review run ID | Eval score | Actual cost USD | +| ---: | --- | --- | --- | ---: | +| 1 | `consistency-a` | `20260725-120233-6c490c44` | fail, 1/2 expectations | 1.270413 | +| 2 | `consistency-b` | `20260725-120706-36842f4b` | fail, 1/2 expectations | 0.901080 | +| 3 | `consistency-c` | `20260725-121006-87a8ed77` | fail, 0/2 expectations | 0.519151 | +| 4 | `dilution-a` | `20260725-121205-5a020015` | pass, 2/2 expectations | 1.102087 | +| 5 | `dilution-b` | `20260725-121737-6249976f` | fail, 0/2 expectations | 0.539328 | +| 6 | `dilution-c` | `20260725-121917-4c7e8289` | fail, 1/2 expectations | 0.875598 | + +The exact required report command selected that cohort and failed closed. The original invalid report is preserved immutably at `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/reports/plan102-eval-preflight-invalid-ace65769.json`, SHA-256 `3368178d57d6c4c46865fccb52b8aa160d1f9acf3536be9e1cde4d05d52c50ab`. Its authoritative reconstructed `preflightCostUSD` and cumulative `actualValidationCostUSD` are `$5.207657`; the unrounded sum of the 84 model-call records is `$5.207655`, with the `$0.000002` difference coming from summing the report's per-run six-decimal values. + +The invalid cohort's exact additional repeat-10 planning projection is `10 * preflightCostUSD = $52.076570`. Because a corrected repeat-one retry is now pending, reserve at most another `$60` before that retry and replace the repeat-10 estimate with `10 * correctedPreflightCostUSD` only after a valid retry. The currently known accounting projection is therefore `$60 + $52.076570 + $95 = $207.076570` (retry cap, placeholder repeat-10 estimate, and production-pair reservation), with fake-provider collateral `$0` and real-model collateral still deferred until its A-side measurement. Cumulative actual plus this known projection is `$212.284227 <= approvedValidationCostUSD: $500`, leaving `$287.715773` before deferred collateral. This is accounting only: Phase 3 does not authorize or start the retry. + +Paid target treatment was invalid in every packed arm/case: consistency B `0/1`, consistency C `0/1`, dilution B `0/1`, and dilution C `0/1`. Each consistency target remained a one-atom deep packet with the same requested/routed lenses as A while non-target atoms packed, reducing file packets from 5 to 3. Each dilution target likewise remained a one-atom deep packet; B emitted only one-atom events and stayed at 3 packets, while C packed two non-target light hunks and also stayed at 3. The target comparisons themselves did not change path, coverage, or requested lenses. + +The failed report had three independent artifact-contract defects, now fixed with focused regressions: + +- `paid_summary_reconciliation` failed six times because attention records were compared to filename-sorted packet artifacts by array index. Reconciliation now requires unique IDs and joins by `packetId`, so producer order is irrelevant but duplicates/missing packets still fail. +- `paid_evidence_relations` failed twice because the verifier admitted only Stage 7/8 producers even though real uncertainty promotions are intentionally produced at Stage 9. Stage 9 is now admitted only when `provenance.source === "uncertainty_promotion"`; an arbitrary Stage 9 candidate still fails. +- `a_lens_route_join` failed twice because it treated intentional Stage-6 pruning from the A planner request as packing loss. The actual packing contract remains strict: event requested signatures must join the Stage-5 decisions, and B/C routed lenses must exactly equal the union of their A source atoms' already-routed lenses. + +Re-running the report locally over the immutable cohort after those fixes removes all ten artifact-contract failures and leaves 17 treatment failures: four `requested_lens_join`, three `treatment_invariant`, two `effective_profile_downgrade`, one `routed_lens_join`, four `insufficient_treatment`, and one each of `hunk_bijection`, `unknown_source_atom`, and `atom_bijection`. These are not waived. The consistency planners varied the non-target coverage, requested lenses, and relationship hints across A/B/C, changing an A atom from normal to light and A standalone profiles from investigate to simple in packed arms. Dilution C changed an A-skipped hunk to light and changed the normal hunk's requested lenses, so its atom universe could not join A. Those are real independent Stage-5 samples. The four target-treatment misses also exposed a fixture/config defect: an obvious target regression was predictably assigned deep/tests attention while safe mechanical siblings were assigned normal/light attention, but packing correctly requires identical coverage and requested-lens signatures. + +That 17-failure diagnostic rerun occurred before changing the declared case files. After the deterministic fixture repair below, the historical CLI now stops earlier with `declared_case_join`, as designed, because the current YAML hashes no longer match invocation `ace65769...` and its persisted snapshots. Do not edit the old manifest or logs to bypass that provenance check; the immutable original report and case snapshots remain the authority for the invalid cohort, and the pending retry will create a new invocation with the repaired declarations. + +The deterministic repair keeps both required failure classes while reducing those avoidable compatibility axes. All six current cases enable only `lang/go`. In `cross-atom-consistency`, every separated sibling now contains the same safe `Amount == 0` to `Amount <= 0` boundary hardening while four also add the shared validation veto and `HandleWire` still omits it. In `dilution-control`, all four separated hunks are comparator-boundary changes: the target retains the inclusive-limit regression, while the attempt normalization, integer-equivalent retry guard, and equality-equivalent shard cap remain safe sibling changes. The original paid case snapshots/logs are untouched. + +The repaired suite passes strict six-case parsing and arm-structure comparison. Both fixture bases and both materialized main/feature repositories pass `go test ./...`. A no-model scripted normal-coverage validation proves: + +| Case | A packets / target atoms | B packets / target atoms | C packets / target atoms | Routed lens | Profile | B base budget | C effective budget | +| --- | --- | --- | --- | --- | --- | --- | --- | +| `cross-atom-consistency` | 5 / 1 | 1 / 5 | 1 / 5 | `lang/go` | `standard` | 4 calls, 2 rounds, 10,000 chars | 7 calls, 2 rounds, 17,500 chars | +| `dilution-control` | 4 / 1 | 1 / 4 | 1 / 4 | `lang/go` | `standard` | 4 calls, 2 rounds, 10,000 chars | 7 calls, 2 rounds, 16,000 chars | + +That validator observed zero model calls and passed exact hunk/atom order and bijection, five-hunk/12K caps, normal coverage, requested/routed lens, standalone/effective profile floor, base/effective budget, target membership, and dispatch-rank checks. It proves the repaired deterministic preconditions, not real-planner treatment. A second paid repeat-one cohort is still required and remains pending outside Phase 3. + +One-repeat recall from the invalid cohort is retained only as measurement: consistency A/B each hit the final expectation while C did not; dilution A hit candidate/final, B missed before candidate generation despite reviewing the target packet, and C generated a candidate that was rejected at verification. These samples do not select B or C and do not decide rollout. Plan 102 steps 8–12 are `not_run` because the cohort is treatment-invalid, not because packing has received a rollout verdict. No case was changed to repeat 10, no later paid suite ran, no arm was selected, and no experiment flag or product path was torn down. + ## Phase 2 deterministic reconciliation (2026-07-24) Step 6's exact four-run command completed with exit `0`, `noModelCalls: true`, four explicit `modelCallsObserved: 0` row proofs, and no report failures. The frozen output is `/tmp/plan102-packet-shape.json` (SHA-256 `77af0c38937bd6957806f05ad201cbad32461414c1d667824096208473c276fa`). The authoritative motivating count is **75**: the real `combinedPatchChars()` builder produced 96→75, so the bounded 74/76 reconciliation was not used. The report's diagnostic reapplication of the old `contentWithLineNumbers` proxy produces 76 for the current atom stream while the real measurement produces 75; this does not change the golden because the authoritative builder result itself matches the plan's historical 75 target. The frozen hash was refreshed after the verifier stopped trusting serialized cap telemetry and independently rendered the source diff for every packet that actually combines multiple atoms. diff --git a/specs/plans/phase-plans/102/phase_3.md b/specs/plans/phase-plans/102/phase_3.md new file mode 100644 index 0000000..77c5569 --- /dev/null +++ b/specs/plans/phase-plans/102/phase_3.md @@ -0,0 +1,30 @@ +--- +status: complete +--- + +# Phase 3: One-Repeat Packet-Packing Treatment Preflight + +## Overview + +Implement Plan 102 step 7 only. This phase creates two private, materializable Go fixtures and six repeat-one A/B/C recall cases, records the owner-approved paid-validation ceiling before model use, executes the cache-off one-repeat suite, and runs the exact cohort report preflight. The first cohort is retained as treatment-invalid evidence: it exposed real independent-planner variance, a fixture/config incompatibility, and three report artifact-contract bugs. This phase fixes the report bugs, repairs and revalidates the fixtures without model calls, and stops with a second paid repeat-one cohort pending. One-sample recall is measurement only; retry execution, repeat-10 execution, arm selection, collateral checks, production validation, rollout, and teardown remain out of scope. + +## Steps + +1. Reconcile current drift and working-tree state for Plan 102, inspect the private-eval conventions and existing eval schema, and record `approvedValidationCostUSD: $500` (superseding the earlier $200 authorization), current actual spend, preflight projection assumptions, and authorized scope in Plan 102 before any paid model call. +2. Create `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/recall/repos/dilution-control/{base,feature}` as a compact materializable Go repository. Put a locally detectable boundary regression in one separated function and safe unrelated edits in separated sibling functions so A keeps the target in one source atom while B/C combine it with unrelated atoms. +3. Create `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/recall/repos/cross-atom-consistency/{base,feature}` as a compact materializable Go repository. Change five separated sibling functions so four add a shared validation/veto while the fifth omits it, and keep baseline atom boundaries separate while allowing B/C to combine the inconsistent siblings. +4. Add `dilution-{a,b,c}.yml` and `consistency-{a,b,c}.yml` with `repeat: 1`, cache disabled, precise candidate/final expectations, and identical model/provider/reasoning/concurrency/depth/lens/time/token settings. Limit arm differences to A `false`/`base`, B `true`/`base`, and C `true`/`atom-scaled`. +5. Parse/materialize the private suite without model calls and compare all six YAMLs structurally to prove that only the permitted arm fields and case-specific identity/fixture/expectation fields differ. +6. Run `pnpm dev -- eval --eval-dir /home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/recall --no-cache` once, then run the exact `pnpm exec tsx scripts/packet-packing-report.ts eval --logs .../recall/logs --cohort latest --expected-repeats 1 --output /tmp/plan102-eval-preflight.json` command. +7. Inspect and preserve the invalid cohort evidence. Record invocation/cohort/run IDs, the immutable failed-report path/hash, actual preflight and cumulative costs, `10 * preflightCostUSD`, projected spend assumptions, the active $500 ceiling, target telemetry, root-cause classification, and one-repeat recall limitations. Stop paid work because all four B/C target treatments are invalid. +8. Fix only the three demonstrated report artifact-contract defects: packet-ID attention reconciliation, Stage-9 uncertainty-promotion provenance, and the distinction between valid A-side Stage-6 lens pruning and B/C routed-lens loss. Add focused positive/negative regressions and rerun the report locally over the immutable cohort to expose the remaining real treatment failures. +9. Repair the private deterministic preconditions without model calls. Restrict all arms to `lang/go`, make every sibling hunk a plausibly comparable boundary/validation-risk change while retaining the dilution boundary bug and consistency omission, rerun strict YAML/arm parsing, materialized Go tests, and the local atom/packing/cap/profile/budget/rank validator. Record the retry as pending; do not start it. +10. Run the focused Plan 102 test command, `pnpm run check`, the complete test suite, and the build. Do not change any case to repeat ten, run a later paid phase, select an arm, remove experiment flags, or commit. + +## Tests + +- `private packet-packing suite parses and materializes`: verifies all six strict YAML cases load with `lang/go` only, both Go repos materialize, and main/feature compile and pass `go test ./...` without model use. +- `A/B/C structural comparison`: verifies repeat/cache/model/provider/reasoning/concurrency/depth/lenses/time/token/fixture/expectation inputs match within each case and only the two permitted review fields differ across arms. +- `report artifact-contract regressions`: verifies attention joins by unique packet ID regardless of producer order, admits Stage 9 only for relational uncertainty promotion, and preserves the distinction between requested and validly routed lenses while still detecting B/C routed-lens loss. +- `local repaired treatment validator`: assigns identical normal coverage and `lang/go` to every hunk, then verifies A target packets contain one source atom, B/C targets combine all source atoms and reduce target-file packets, and all hunk/atom/cap/coverage/lens/profile/budget/rank invariants pass with zero model calls. +- `repository focused and complete gates`: verifies packet packing, configuration, eval loading, reporting, type/lint/format checks, all tests, and the production build remain clean after evidence is recorded. diff --git a/tests/packet-packing-report.test.ts b/tests/packet-packing-report.test.ts index 6502746..5dda531 100644 --- a/tests/packet-packing-report.test.ts +++ b/tests/packet-packing-report.test.ts @@ -1356,6 +1356,25 @@ describe("packet packing eval analysis", () => { }); it("joins requested and routed lenses to the persisted Stage-5 plan and A atoms", () => { + const pruned = abcCohort(1); + for (const run of pruned.runs) { + for (const execution of run.executions) { + for (const decision of execution.plan.coverage) { + decision.lenses = ["core/code-review", "core/tests"]; + } + for (const event of execution.events.filter((entry) => entry.message === "same_file_atoms_packed")) { + event.data = { + ...event.data, + requestedLensSignature: JSON.stringify(["core/code-review", "core/tests"]) + }; + } + } + } + expect(failureCodes(analyzeEvalCohort(pruned, 1).failures)).not.toEqual(expect.arrayContaining([ + "requested_lens_join", + "routed_lens_join" + ])); + const planned = abcCohort(1); planned.runs[0]!.executions[0]!.plan.coverage[0]!.lenses = ["security/auth"]; expect(failureCodes(analyzeEvalCohort(planned, 1).failures)).toContain("requested_lens_join"); @@ -1509,6 +1528,12 @@ describe("packet packing eval analysis", () => { await attachProducerSummaryEvidence(base.runs[0]!); expect(failureCodes(analyzeEvalCohort(base, 1).failures)).not.toContain("paid_summary_reconciliation"); + const reorderedAttention = structuredClone(base); + reorderedAttention.runs[0]!.executions[0]!.summaryArtifacts!.attention = [ + ...(reorderedAttention.runs[0]!.executions[0]!.summaryArtifacts!.attention as unknown[]) + ].reverse(); + expect(failureCodes(analyzeEvalCohort(reorderedAttention, 1).failures)).not.toContain("paid_summary_reconciliation"); + const paths = [ ["cost", "costBreakdown", "total", "costUSD"], ["run", "totals", "costBreakdown", "total", "costUSD"], @@ -1612,6 +1637,33 @@ describe("packet packing eval analysis", () => { } }); + it("accepts Stage 9 producers only for relational uncertainty promotions", () => { + const promoted = abcCohort(1); + const promotedRun = promoted.runs[1]!; + const promotedExecution = promotedRun.executions[0]!; + const promotedCandidate = promotedExecution.candidateFindings[0]!; + const provenance = { + source: "uncertainty_promotion" as const, + sourceKind: "uncertainty" as const, + sourcePacketId: promotedCandidate.producedBy.packetId, + question: "Does this changed boundary remain safe?", + files: [promotedCandidate.path], + symbols: [], + reason: "The packet review left the boundary unresolved." + }; + promotedCandidate.producedBy.stage = 9; + promotedCandidate.provenance = provenance; + promotedExecution.verification[0]!.candidateProvenance = structuredClone(provenance); + refreshRunEvidenceScores(promotedRun); + expect(failureCodes(analyzeEvalCohort(promoted, 1).failures)).not.toContain("paid_evidence_relations"); + + const arbitraryStage9 = abcCohort(1); + const arbitraryRun = arbitraryStage9.runs[1]!; + arbitraryRun.executions[0]!.candidateFindings[0]!.producedBy.stage = 9; + refreshRunEvidenceScores(arbitraryRun); + expect(failureCodes(analyzeEvalCohort(arbitraryStage9, 1).failures)).toContain("paid_evidence_relations"); + }); + it("replays verifier gates from raw candidates instead of trusting collusive persisted decisions", () => { const scheduledInvalid = abcCohort(1); const scheduledRun = scheduledInvalid.runs[1]!; From 87a5a10c912145eea33e253d6bf6e468a2b81150 Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 09:49:13 -0400 Subject: [PATCH 04/24] docs(evals): record failed packet-packing retry gate --- .../102-issue-102-same-file-packet-packing.md | 45 +++++++++++++++++++ specs/plans/phase-plans/102/phase_4.md | 32 +++++++++++++ 2 files changed, 77 insertions(+) create mode 100644 specs/plans/phase-plans/102/phase_4.md diff --git a/specs/plans/102-issue-102-same-file-packet-packing.md b/specs/plans/102-issue-102-same-file-packet-packing.md index 1358f33..94cc461 100644 --- a/specs/plans/102-issue-102-same-file-packet-packing.md +++ b/specs/plans/102-issue-102-same-file-packet-packing.md @@ -63,6 +63,51 @@ That validator observed zero model calls and passed exact hunk/atom order and bi One-repeat recall from the invalid cohort is retained only as measurement: consistency A/B each hit the final expectation while C did not; dilution A hit candidate/final, B missed before candidate generation despite reviewing the target packet, and C generated a candidate that was rejected at verification. These samples do not select B or C and do not decide rollout. Plan 102 steps 8–12 are `not_run` because the cohort is treatment-invalid, not because packing has received a rollout verdict. No case was changed to repeat 10, no later paid suite ran, no arm was selected, and no experiment flag or product path was torn down. +### Phase 4 repaired one-repeat retry and failed gate (2026-07-25) + +Phase 4 retried Plan 102 step 7 exactly once and stopped at the failed report gate. The runtime was exact clean commit `bb96fd3439c715130756a93efd8e679772f81a9b` on branch `next`, with `git status --porcelain=v1` empty, Node `v26.5.0`, pnpm `11.15.1`, and launch preflight timestamp `2026-07-25T13:06:52Z`. The draft phase note was stashed before launch and restored only after the paid process and report completed. The documented `pnpm dev -- eval ...` spelling passed a literal `--` through pnpm 11 and was rejected by Commander before suite allocation; it created no invocation, run directory, model call, or spend. The one actual invocation used the equivalent pnpm-11 spelling `pnpm dev eval --eval-dir /home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/recall --no-cache`. + +Immediately before the paid call, the repaired suite strictly parsed as six cases with `repeat: 1`, cache disabled, `lang/go` as the only lens, and the exact Phase 3 provider/model/reasoning/concurrency/depth/time/token/cost settings. Within each family, structural comparison found only the case-name suffix and the two permitted A/B/C review-field differences. The declaration SHA-256 values were: + +| Case | SHA-256 | +| --- | --- | +| `consistency-a.yml` | `15702c3e5c8c2d97845b39ff04618d26966174934ad1afe9765bdb7710bde590` | +| `consistency-b.yml` | `5b0e093db08b9cee8837d04d5236ceabd30727903ef18e13c74208022bca05bb` | +| `consistency-c.yml` | `69130fa21479fe25acd881bff1fc9e914ff470cb5efb70e89c1e70bb5f4e01c3` | +| `dilution-a.yml` | `fe2802d21333f6650fe4c4fc1a5685574638fe6036f19b6d8bfbcad088fbb871` | +| `dilution-b.yml` | `00a8d51ea6aee14c9ef39105c3bd7514ba6167ac642b35dc9b45114f67bdd757` | +| `dilution-c.yml` | `87883f0210d92551da43b860f47b98a169609ff902102b21b28f853039be12ef` | + +The six fixture source files had aggregate SHA-256 `d486e2601a1ea05f0872794be71f409b5296b78c2998ef39e973eedb57061da4`. Both materialized base and feature revisions passed `go test ./...`. The no-model validator observed exactly zero model calls and no failures. With normal coverage and the declared `lang/go` route, it reproduced consistency A `5` packets/one target atom versus B/C `1` packet/five target atoms, and dilution A `4`/one versus B/C `1`/four. Exact hunk/atom order and bijection, five-hunk/12K caps, coverage, requested/routed lenses, standalone/effective profile floors, base/effective budgets, target membership, and dispatch ranks all passed. This reconfirmed only deterministic fixture eligibility; it did not predict independently sampled Stage-5 plans. + +The sole live retry invocation was `5bd80f2c-865e-40f0-a605-07387138b904`, started `2026-07-25T13:10:14.113Z` and completed `2026-07-25T13:26:22.430Z`. Its exact log root is `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/recall/logs`. It completed all six cache-off executions with zero provider/configuration errors, zero incomplete reviews, zero budget violations, and 71 model calls: + +| Run | Case | Review run ID | Eval score | Model calls | Actual cost USD | +| ---: | --- | --- | --- | ---: | ---: | +| 7 | `consistency-a` | `20260725-131014-ff7643a9` | fail, 0/2 expectations | 13 | 0.669039 | +| 8 | `consistency-b` | `20260725-131217-78051528` | fail, 0/2 expectations | 11 | 0.564235 | +| 9 | `consistency-c` | `20260725-131407-46ee2073` | fail, 0/2 expectations | 11 | 0.618305 | +| 10 | `dilution-a` | `20260725-131627-fd970500` | fail, 1/2 expectations | 12 | 0.614135 | +| 11 | `dilution-b` | `20260725-131904-9623b816` | pass, 2/2 expectations | 13 | 0.713651 | +| 12 | `dilution-c` | `20260725-132212-896d115e` | fail, 1/2 expectations | 11 | 0.806856 | + +The exact report command selected that invocation UUID, not `latest`: `pnpm exec tsx scripts/packet-packing-report.ts eval --logs /home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/recall/logs --cohort 5bd80f2c-865e-40f0-a605-07387138b904 --expected-repeats 1 --output /tmp/plan102-eval-preflight.json`. It exited `1` with exactly six failures: consistency B and C were each treated `0/1`, and one B plus one C packet in each family failed the composite `treatment_invariant`. There were no other report failure codes. + +The treatment diagnosis is exact: + +- Consistency A produced five packets: one normal/investigate atom, three light/simple atoms, and the target `HandleWire` deep/investigate atom. B and C packed only the three non-target light atoms, reducing five packets to three, while the target stayed a one-atom deep packet. Both target treatments are therefore invalid at `0/1`. On the packed non-target packet `e33a8a1d...`, the independent B/C planner hints made all three standalone profiles `investigate`; the corresponding A atoms were `simple`. The event's profile-floor fields consequently could not exactly reconcile to the A atoms, producing one composite invariant failure in each packed arm. +- Dilution A produced deep/investigate, normal/investigate, light/simple, and deep/investigate atoms. B and C validly combined the target first deep atom with the last deep atom, reduced four packets to three, and treated the target at `1/1` with two source atoms. A separate one-atom light packet `3989cd85...` nevertheless changed from A `simple`/zero-tool to a B/C `investigate` standalone profile due independently sampled planner hints. Its event profile-floor fields likewise could not exactly reconcile to A, producing one composite invariant failure per packed arm. + +Thus caps, hunk/atom joins, coverage, requested/routed lenses, effective profiles/budgets, dispatch ranks, artifact lineage, summary reconciliation, and spend evidence produced no separate failures; the gate failed specifically on the two missing consistency target treatments and cross-arm standalone-profile reconciliation. No report check was waived. The failure is not a rollout verdict from recall, and there will be no third paid one-repeat invocation. + +One-repeat recall remains measurement only. All three consistency executions generated and published the missing-wire-validation finding as category `security`, while the declared expectation required `correctness`, so each recorded candidate/final partial matches rather than a hit. Dilution A and C hit the candidate expectation but were rejected at verification; dilution B hit both candidate and final expectations. These six samples do not select B or C and do not establish packing safety or harm. + +The report's authoritative reconstructed `retryPreflightCostUSD` is `$3.986221`; the unrounded sum of its 71 model-call records is `$3.986220`, with the `$0.000001` difference coming from the report's per-run six-decimal reconciliation. Cumulative `actualValidationCostUSD` is `$5.207657 + $3.986221 = $9.193878`. Because the retry is not valid, `validPreflightCostUSD` and therefore the required `10 * validPreflightCostUSD` repeat-10 projection are **not established** and cannot authorize step 8. For comparison only, `10 * retryPreflightCostUSD = $39.862210`; including the known later `$95` production-pair reservation would give `$9.193878 + $39.862210 + $95 = $144.056088`. The conservative Phase 3 placeholder remains `$52.076570`, yielding known projected remaining spend of `$147.076570` and cumulative actual plus projection of `$156.270448 <= approvedValidationCostUSD: $500`, leaving `$343.729552` before deferred real-model collateral. Fake-provider collateral remains `$0`. None of these accounting projections authorize a later paid phase after the failed gate. + +The failed retry report is preserved byte-for-byte at `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/reports/plan102-eval-preflight-retry-invalid-5bd80f2c.json`, SHA-256 `650113d24d092e6fd712303e1828b297010c76f672d7efb799f5081f79635517`. The new run/manifests aggregate SHA-256 is `dabb5d4700a0fe559490216da73d335bd9f28cec6d3c004d8b41b4cb7c77c3c1`. The original runs 1–6 plus manifest retain aggregate SHA-256 `2a3232651752a4f3652334d5126e2eebca6169da658e854214b6f98a158eb750`, and the original invalid report remains SHA-256 `3368178d57d6c4c46865fccb52b8aa160d1f9acf3536be9e1cde4d05d52c50ab`. No historical log, manifest, or report was edited or deleted. + +Plan 102 steps 8–11 are `not_run` because the repaired retry gate failed. Step 12's failure teardown is also `not_run` in Phase 4 and is reserved for the next separately reviewed phase; no case was changed to repeat 10, no arm was selected, no collateral or production suite ran, no rollout verdict was made, and no experiment code or evidence was torn down in this phase. + ## Phase 2 deterministic reconciliation (2026-07-24) Step 6's exact four-run command completed with exit `0`, `noModelCalls: true`, four explicit `modelCallsObserved: 0` row proofs, and no report failures. The frozen output is `/tmp/plan102-packet-shape.json` (SHA-256 `77af0c38937bd6957806f05ad201cbad32461414c1d667824096208473c276fa`). The authoritative motivating count is **75**: the real `combinedPatchChars()` builder produced 96→75, so the bounded 74/76 reconciliation was not used. The report's diagnostic reapplication of the old `contentWithLineNumbers` proxy produces 76 for the current atom stream while the real measurement produces 75; this does not change the golden because the authoritative builder result itself matches the plan's historical 75 target. The frozen hash was refreshed after the verifier stopped trusting serialized cap telemetry and independently rendered the source diff for every packet that actually combines multiple atoms. diff --git a/specs/plans/phase-plans/102/phase_4.md b/specs/plans/phase-plans/102/phase_4.md new file mode 100644 index 0000000..c4ea9e8 --- /dev/null +++ b/specs/plans/phase-plans/102/phase_4.md @@ -0,0 +1,32 @@ +--- +status: complete +--- + +# Phase 4: One-Repeat Treatment Retry + +## Overview + +Complete only the repaired retry of Plan 102 step 7. Re-verify the six current strict private cases, both materializable Go fixtures, the no-model treatment proof, budget authorization, and exact clean runtime provenance at commit `bb96fd3`. Then execute one cache-off repeat-one A/B/C invocation, analyze that exact invocation UUID with the packet-packing report, preserve the resulting JSON report immutably, and reconcile actual/projected validation spend. The retry gate failed: consistency B/C did not treat the target, and independently sampled B/C planner hints changed standalone profiles relative to A in both families. The phase stops without a third paid invocation. One-sample recall remains measurement only. Repeat ten, arm selection, collateral, production validation, rollout, and teardown remain out of scope. + +## Steps + +1. Confirm `HEAD` is exactly `bb96fd3439c715130756a93efd8e679772f81a9b`, the repository and Plan 102 in-scope paths are clean, all six private YAMLs remain `repeat: 1`, and the invalid cohort/logs/report retain their recorded hashes and contents. +2. Hash the six current YAMLs and both fixture trees, strictly load the suite, compare A/B/C definitions within each family, and prove that arm differences remain limited to name plus `review.packSameFileHunks` and `review.packedToolBudgetMode`. Verify cache disabled and the exact provider/model/reasoning/concurrency/depth/lens/time/token/cost settings. +3. Materialize both fixture base/feature repositories without model calls and run `go test ./...` in every materialized revision. Re-run the deterministic local treatment validator to prove exact hunk/atom order and bijection, five-hunk/12K caps, coverage/lens/profile/budget/rank invariants, A one-atom targets, B/C multi-atom targets, and reduced packed packet counts. +4. Reconcile the paid gate immediately before launch: cumulative actual spend is `$5.207657`; the retry reserves at most `$60`; the placeholder repeat-10 projection is `$52.076570`; the known later production reservation is `$95`; and `$212.284227 <= approvedValidationCostUSD: $500`. Record exact clean runtime provenance and launch exactly one complete `pnpm dev -- eval --eval-dir .../recall --no-cache` invocation. +5. Resolve the new invocation UUID and its six review run IDs from the invocation manifest. Run exactly `packet-packing-report.ts eval --cohort --expected-repeats 1`, never `latest`, and inspect the JSON for zero provider/configuration/evidence failures, valid B/C treatment in both families, and every deterministic/artifact invariant. +6. Copy the new JSON report without altering it into `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/reports/`, record its SHA-256, and leave the invalid cohort, all logs, and all reports untouched. If treatment or report validation fails, stop with the exact diagnosis and do not launch a third paid invocation. +7. Update Plan 102 reconciliation with invocation/cohort/run/review IDs, per-run and exact cohort cost, cumulative actual cost including `$5.207657`, `10 * validPreflightCostUSD`, the known future `$95` production reservation, total projected spend, and comparison with the `$500` ceiling. State that one-repeat recall is measurement only and that steps 8–12 remain `not_run`. +8. Run the focused Plan 102 tests, repository checks, complete test suite, and build. Do not change any private case to repeat ten, select an arm, start collateral or production validation, or tear down experiment scaffolding. + +## Tests + +- `clean runtime provenance and immutable evidence`: proves the paid command ran from exact commit `bb96fd3`, with a clean worktree, and that the prior invalid cohort/logs/report remained byte-identical. +- `strict repaired suite preflight`: proves all six repeat-one YAMLs parse, use only `lang/go`, differ only by the two permitted arm settings, disable cache, and retain identical paid configuration within each family. +- `fixture materialization and no-model treatment proof`: proves both fixture revisions pass `go test ./...` and the deterministic A/B/C treatment validator passes all atom, packet, cap, coverage, lens, profile, budget, and rank invariants without model calls. +- `exact paid cohort report`: proves the explicit new invocation contains six complete executions, both B/C arms receive the target treatment for both cases, paid evidence and costs reconcile, and all deterministic/artifact failure lists are empty. +- `repository focused and complete gates`: verifies packet packing, configuration, eval loading, reporting, lint/format/type checks, all tests, and the production build remain clean after reconciliation. + +## Outcome + +Invocation `5bd80f2c-865e-40f0-a605-07387138b904` completed six cache-off executions with zero errored cases and authoritative reconstructed cost `$3.986221`. The exact explicit-cohort report exited `1` with two consistency `insufficient_treatment` failures and four cross-arm standalone-profile `treatment_invariant` failures. It is preserved at `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/reports/plan102-eval-preflight-retry-invalid-5bd80f2c.json`, SHA-256 `650113d24d092e6fd712303e1828b297010c76f672d7efb799f5081f79635517`. Cumulative actual validation spend is `$9.193878`, within the `$500` ceiling. Steps 8–12 remain `not_run`; the next separately reviewed phase owns the required step-12 failure teardown. From f372f735267794bf160198c9b9b3a46c880a662b Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 10:08:58 -0400 Subject: [PATCH 05/24] fix(pipeline): restore baseline after packet-packing gate failure --- scripts/packet-packing-report.ts | 5319 ----------------- .../102-issue-102-same-file-packet-packing.md | 33 +- specs/plans/README.md | 2 +- specs/plans/phase-plans/102/phase_5.md | 33 + src/config/config-loader.ts | 26 +- src/config/schema.ts | 7 - src/evals/eval-artifacts.ts | 11 - src/evals/eval-command.ts | 45 +- src/evals/eval-runner.ts | 49 +- src/pipeline/composer.ts | 136 +- src/pipeline/packet-builder.ts | 375 +- src/pipeline/review-runner.ts | 8 +- src/pipeline/verifier.ts | 102 - src/telemetry/run-artifacts.ts | 181 +- src/types.ts | 42 - tests/config-loader.test.ts | 36 - tests/evals.test.ts | 148 +- tests/packet-packing-report.test.ts | 2288 ------- tests/pipeline-phase5.test.ts | 629 -- 19 files changed, 171 insertions(+), 9299 deletions(-) delete mode 100644 scripts/packet-packing-report.ts create mode 100644 specs/plans/phase-plans/102/phase_5.md delete mode 100644 tests/packet-packing-report.test.ts diff --git a/scripts/packet-packing-report.ts b/scripts/packet-packing-report.ts deleted file mode 100644 index 07b658c..0000000 --- a/scripts/packet-packing-report.ts +++ /dev/null @@ -1,5319 +0,0 @@ -import { mkdtemp, readdir, readFile, rm, writeFile } from "node:fs/promises"; -import { existsSync } from "node:fs"; -import { tmpdir } from "node:os"; -import path from "node:path"; -import { pathToFileURL } from "node:url"; -import { z } from "zod"; -import { defaultConfig } from "../src/config/schema.js"; -import { createGitClient } from "../src/git/git-client.js"; -import { parseDiff } from "../src/git/diff-parser.js"; -import { runGit } from "../src/git/subprocess.js"; -import { buildReviewPackets, packetDispatchRank, packetReviewContextFromDossier, toolBudget } from "../src/pipeline/packet-builder.js"; -import { reconstructComposerGroupsFromArtifacts, reconstructComposerPolicyFromArtifacts } from "../src/pipeline/composer.js"; -import { - reconstructGatedVerifierCandidatesFromArtifacts, - reconstructDuplicateVerificationVerdict, - reconstructVerifiedFindingsFromArtifacts -} from "../src/pipeline/verifier.js"; -import { buildRepositoryIndex } from "../src/repo/repository-index.js"; -import { stripCredentials } from "../src/telemetry/redaction.js"; -import { reconstructRunTelemetryDerivedEvidence } from "../src/telemetry/run-artifacts.js"; -import type { LlmCallRecord, TelemetryRecorder } from "../src/telemetry/telemetry-recorder.js"; -import type { - CodegenieConfig, - DiffFile, - DiffLine, - EvalBudgetResult, - EvalCase, - EvalArtifacts, - EvalExpectationList, - EvalExpectationResult, - EvalInvocationManifest, - EvalRunInfo, - EvalSelectionRecord, - EvalScore, - EvalVerificationRecord, - FileFacts, - FileFilterDecision, - PacketContextQuality, - PlannerDossier, - ResolvedReviewInput, - ReviewPacket, - ReviewPlan, - ReviewProfile, - RunCoverageStatus, - CandidateFinding, - FinalFinding, - TelemetryEvent, - ToolBudget, - ToolCallRecord, - UnifiedDiff -} from "../src/types.js"; -import { sha256Hex } from "../src/util/hashing.js"; -import { scaleToolBudget } from "../src/util/budget.js"; -import { isLocalToolBudgetRejectionReason } from "../src/util/context-pressure.js"; -import { loadEvalCaseDeclaration } from "../src/evals/eval-runner.js"; -import { aggregateRepeatScores, scoreEvalRun } from "../src/evals/eval-scoring.js"; - -const MAX_HUNKS_PER_PACKET = 5; -const MAX_PATCH_CHARS = 12_000; -const EQUIVALENT_TARGET_HUNKS = 142; -const PRODUCTION_BASE_SHA = "d1c49bdf6a8002ec2ec27faac94a932d736532b2"; -const PRODUCTION_HEAD_SHA = "fbb5f8761c2c296e115af17e919a7c35d9de8373"; -const PRODUCTION_REPO_ROOT = "/home/peter/Dev/0xsequence/trails-api"; -const PRODUCTION_TIMEOUT_MS = 60 * 60 * 1000; -const PRODUCTION_CONCURRENCY = 6; -const LEGACY_DIFF_SCHEMA_RUN_IDS = new Set([ - "20260724-135818-740d73f2", - "20260724-150405-fe1548ae", - "20260724-162739-81f806a6" -]); -const PROFILE_RANK: Record = { simple: 0, standard: 1, investigate: 2 }; -const CONTEXT_QUALITY_RANK: Record = { - path_only: 0, - outline_only: 1, - sliced: 2, - full: 3 -}; -const DOCS_CONFIG_EXTENSIONS = new Set([".md", ".yml", ".yaml", ".toml", ".conf", ".sample", ".txt"]); - -const commitShaSchema = z.string().regex(/^[a-f0-9]{40}$/iu); -const jsonObjectSchema = z.record(z.string(), z.unknown()); -const nonnegativeIntSchema = z.number().int().nonnegative(); -const positiveIntSchema = z.number().int().positive(); - -const pullRequestMetadataSchema = z.object({ - owner: z.string(), - repo: z.string(), - number: positiveIntSchema, - title: z.string(), - body: z.string(), - url: z.string(), - baseRefName: z.string(), - baseSha: commitShaSchema, - headRefName: z.string(), - headSha: commitShaSchema -}).strict(); - -const resolvedInputSchema = z.object({ - mode: z.enum(["github_pr", "branch", "head", "commit_range"]), - repoRoot: z.string(), - baseRef: commitShaSchema.optional(), - baseRefName: z.string().optional(), - headRef: commitShaSchema.optional(), - headRefName: z.string().optional(), - startCommit: commitShaSchema.optional(), - endCommit: commitShaSchema.optional(), - mergeBase: commitShaSchema.optional(), - headSha: commitShaSchema.optional(), - pr: pullRequestMetadataSchema.optional(), - commits: z.array(z.object({ - sha: commitShaSchema, - title: z.string(), - body: z.string(), - authorName: z.string().optional(), - authoredAt: z.string().optional() - }).strict()), - rawDiffChars: nonnegativeIntSchema -}).strict(); - -const diffLineSchema = z.object({ - kind: z.enum(["context", "add", "delete"]), - content: z.string(), - oldLineNumber: positiveIntSchema.optional(), - newLineNumber: positiveIntSchema.optional() -}).strict(); - -const diffHunkSchema = z.object({ - id: z.string().min(1), - // The pre-Plan-100 retained artifacts predate persisted hunkHash. Their - // versioned migration is checked against a freshly parsed diff below. - hunkHash: z.string().min(1).optional(), - path: z.string().min(1), - oldStart: nonnegativeIntSchema, - oldLines: nonnegativeIntSchema, - newStart: nonnegativeIntSchema, - newLines: nonnegativeIntSchema, - header: z.string(), - lines: z.array(diffLineSchema) -}).strict(); - -const diffFileSchema = z.object({ - path: z.string().min(1), - oldPath: z.string().min(1).optional(), - status: z.enum(["added", "modified", "deleted", "renamed", "copied"]), - isBinary: z.boolean().optional(), - modeOnly: z.boolean().optional(), - isSymlink: z.boolean().optional(), - isSubmodule: z.boolean().optional(), - language: z.string(), - hunks: z.array(diffHunkSchema) -}).strict(); - -const unifiedDiffSchema = z.object({ files: z.array(diffFileSchema) }).strict(); - -const factProvenanceSchema = z.object({ - fact: z.string(), - source: z.enum(["path", "filename", "extension", "parser", "git", "diff", "config", "generated_detector"]), - confidence: z.enum(["high", "medium", "low"]), - reason: z.string() -}).strict(); - -const fileFilterDecisionSchema = z.object({ - path: z.string().min(1), - action: z.enum(["skip", "keep"]), - reason: z.string(), - provenance: z.array(factProvenanceSchema) -}).strict(); - -const fileFactsSchema = z.object({ - path: z.string().min(1), - language: z.string(), - packageRoot: z.string().optional(), - processingMode: z.enum(["per-hunk", "whole-file", "skip"]), - testStatus: z.enum(["test", "source", "unknown"]), - isGenerated: z.boolean(), - isVendored: z.boolean(), - isLockfile: z.boolean(), - isBinary: z.boolean(), - changedLines: nonnegativeIntSchema, - hunkCount: nonnegativeIntSchema, - labels: z.array(z.string()), - reviewPriority: z.enum(["critical", "high", "normal", "low"]), - reasons: z.array(z.string()), - provenance: z.array(factProvenanceSchema), - degraded: z.object({ reason: z.string() }).strict().optional() -}).strict(); - -const surroundingContextHintSchema = z.object({ - kind: z.enum(["enclosing_symbol", "call_site", "test", "line_range", "other"]), - path: z.string().optional(), - symbol: z.string().optional(), - lineRange: z.tuple([positiveIntSchema, positiveIntSchema]).optional(), - reason: z.string(), - expectedUse: z.enum(["packet_context", "tool_lookup"]) -}).strict(); - -const hunkCoverageDecisionSchema = z.object({ - hunkId: z.string().min(1), - path: z.string().min(1), - coverage: z.enum(["deep", "normal", "light", "skip"]), - lenses: z.array(z.string()), - surroundingContextHints: z.array(surroundingContextHintSchema), - reason: z.string(), - focusNotes: z.array(z.string()).optional(), - relatedSymbols: z.array(z.string()).optional(), - relatedFiles: z.array(z.string()).optional() -}).strict(); - -const reviewPlanSchema = z.object({ - diffUnderstanding: z.object({ declaredIntent: z.string(), inferredBehavior: z.string() }).strict(), - intentSignals: jsonObjectSchema.optional(), - coverage: z.array(hunkCoverageDecisionSchema), - plannerRecovery: jsonObjectSchema.optional(), - partialReview: z.object({ - isPartial: z.boolean(), - reason: z.string(), - reviewedHunks: nonnegativeIntSchema, - totalHunks: nonnegativeIntSchema - }).strict().optional() -}).strict(); - -const plannerDossierSchema = z.object({ - commits: z.array(jsonObjectSchema), - compaction: jsonObjectSchema, - depth: z.enum(["light", "normal", "deep"]), - directories: z.array(z.unknown()), - files: z.array(z.unknown()), - filterSummary: jsonObjectSchema, - hunkIndex: z.array(z.unknown()).optional(), - intentSignals: jsonObjectSchema, - lenses: z.array(z.object({ id: z.string().min(1), summary: z.string() }).strict()), - mode: z.enum(["github_pr", "branch", "head", "commit_range"]), - policyFilesChanged: z.array(z.string()), - pr: z.object({ - title: z.string(), - body: z.string(), - url: z.string(), - baseRefName: z.string(), - headRefName: z.string() - }).strict().optional(), - runId: z.string().min(1), - target: z.object({ - baseRef: commitShaSchema.optional(), - headRef: commitShaSchema.optional(), - headSha: commitShaSchema.optional(), - mergeBase: commitShaSchema.optional() - }).strict(), - totals: z.object({ - addedLines: nonnegativeIntSchema, - deletedLines: nonnegativeIntSchema, - files: nonnegativeIntSchema, - hunks: nonnegativeIntSchema, - keptFiles: nonnegativeIntSchema - }).strict() -}).strict(); - -const runMetadataSchema = z.object({ - argv: z.array(z.string()), - budgetStop: z.unknown().nullable(), - codegenieRuntime: jsonObjectSchema, - codegenieVersion: z.string(), - completedAt: z.string().optional(), - durationMs: nonnegativeIntSchema, - finishedAt: z.string(), - nodeVersion: z.string(), - outcome: jsonObjectSchema, - repoRoot: z.string(), - review: jsonObjectSchema, - runId: z.string().min(1), - schemaVersion: positiveIntSchema, - startedAt: z.string(), - totals: jsonObjectSchema -}).strict(); - -const toolBudgetSchema = z.object({ - maxToolCalls: z.number().int().nonnegative(), - maxInvestigationRounds: z.number().int().nonnegative(), - maxResultChars: z.number().int().nonnegative(), - maxSingleToolResultChars: z.number().int().nonnegative().optional(), - reservedSourceResultChars: z.number().int().nonnegative().optional(), - sourceExtension: z.object({ - maxToolCalls: z.number().int().nonnegative(), - maxResultChars: z.number().int().nonnegative() - }).strict().optional() -}).strict(); - -const packetLineSchema = z.object({ - kind: z.enum(["context", "add", "delete"]), - oldLine: positiveIntSchema.optional(), - newLine: positiveIntSchema.optional(), - content: z.string() -}).strict(); - -const packetHunkSchema = z.object({ - hunkId: z.string().min(1), - oldStart: nonnegativeIntSchema, - oldLines: nonnegativeIntSchema, - newStart: nonnegativeIntSchema, - newLines: nonnegativeIntSchema, - header: z.string().optional(), - contentWithLineNumbers: z.string(), - lines: z.array(packetLineSchema), - changedNewLineNumbers: z.array(positiveIntSchema), - changedOldLineNumbers: z.array(positiveIntSchema), - staticSignals: z.array(jsonObjectSchema).optional(), - omittedSignalCount: nonnegativeIntSchema.optional(), - truncated: z.boolean().optional(), - omittedLineCount: nonnegativeIntSchema.optional(), - plannerFallbackReason: z.string().optional() -}).strict(); - -const reviewPacketSchema = z.object({ - id: z.string().min(1), - dispatchRank: z.tuple([nonnegativeIntSchema, z.number().int().nonpositive()]).optional(), - kind: z.enum(["hunk", "coalesced-hunks", "file-diff", "whole-file"]), - coverageEscalation: z.object({ rule: z.literal("test_coverage_delta"), reason: z.string() }).strict().optional(), - prSummary: z.string(), - intentText: z.string().optional(), - intentSignals: jsonObjectSchema.optional(), - path: z.string().min(1), - oldPath: z.string().min(1).optional(), - fileStatus: z.enum(["added", "modified", "deleted", "renamed", "copied"]), - isDeletedContent: z.boolean(), - language: z.string(), - reviewPriority: z.enum(["critical", "high", "normal", "low"]), - coverage: z.enum(["deep", "normal", "light"]), - reviewProfile: z.enum(["simple", "standard", "investigate"]), - lenses: z.array(z.string()), - hunks: z.array(packetHunkSchema).min(1), - symbolFacts: z.array(jsonObjectSchema), - context: jsonObjectSchema, - contextText: z.string(), - contextQuality: z.enum(["path_only", "outline_only", "sliced", "full"]).optional(), - contextDegradationReasons: z.array(z.string()).optional(), - testCoverageDelta: jsonObjectSchema.optional(), - packetSymbols: z.array(jsonObjectSchema).optional(), - relevantTests: z.array(jsonObjectSchema), - surroundingContextHints: z.array(surroundingContextHintSchema), - labels: z.array(z.string()), - attentionNotes: z.array(z.string()), - relatedChangedContext: z.array(jsonObjectSchema), - toolBudget: toolBudgetSchema, - degraded: z.object({ reason: z.string() }).strict().optional(), - fileContext: z.object({ mode: z.enum(["file-diff", "whole-file"]), reason: z.string() }).strict().optional() -}).strict(); - -const diffAnchorSchema = z.object({ - path: z.string().min(1), line: positiveIntSchema, side: z.enum(["RIGHT", "LEFT"]), hunkId: z.string().min(1), - startLine: positiveIntSchema.optional(), startSide: z.enum(["RIGHT", "LEFT"]).optional(), commitSha: commitShaSchema.optional() -}).strict(); - -const candidateFindingSchema = z.object({ - id: z.string().min(1), title: z.string(), severity: z.enum(["critical", "high", "medium", "low"]), - severityBeforeCap: z.enum(["critical", "high", "medium", "low"]).optional(), confidence: z.enum(["high", "medium", "low"]), - path: z.string().min(1), anchor: diffAnchorSchema.optional(), - anchorSource: z.enum(["model", "backfill_changed_code", "backfill_packet_representative", "verifier_revised"]).optional(), - modelAnchorSubmitted: z.boolean().optional(), changedLine: z.boolean(), - category: z.enum(["logic_bug", "correctness", "security", "performance", "architecture", "testing", "maintainability"]), - evidence: z.object({ - changedCode: z.string(), relatedCode: z.array(z.object({ path: z.string(), lines: z.string(), whyRelevant: z.string() }).strict()).optional() - }).strict(), - failureMode: z.string(), whyThisMatters: z.string(), suggestedFix: z.string().optional(), suggestedTest: z.string().optional(), - verification: z.string(), behaviorChange: z.enum(["accidental_regression", "intentional_needs_confirmation", "specified_change", "unknown"]).optional(), - intentEvidence: z.array(z.string()).optional(), - producedBy: z.object({ - kind: z.literal("packet"), stage: z.number().int().min(1).max(11), packetId: z.string().min(1), lensId: z.string().min(1), - skillIds: z.array(z.string()), workerId: z.string().optional(), ensemblePass: positiveIntSchema.optional() - }).strict(), - provenance: z.object({ - source: z.literal("uncertainty_promotion"), sourceKind: z.enum(["uncertainty", "follow_up_hint"]), sourcePacketId: z.string(), - question: z.string(), files: z.array(z.string()), symbols: z.array(z.string()), reason: z.string() - }).strict().optional(), - clusterId: z.string().optional(), duplicateOf: z.string().optional() -}).strict(); - -const finalFindingSchema = candidateFindingSchema.extend({ - fingerprint: z.string().min(1), finalBody: z.string(), publication: z.enum(["inline", "summary-only", "suppressed"]), - mergedCandidateIds: z.array(z.string()), - mergedCategories: z.array(z.enum(["logic_bug", "correctness", "security", "performance", "architecture", "testing", "maintainability"])).optional(), - mergedSeverities: z.array(z.enum(["critical", "high", "medium", "low"])).optional(), mergedPaths: z.array(z.string()).optional(), - mergedTitles: z.array(z.string()).optional(), - // Persisted artifacts pass through the shared redactor, which emits this - // controlled sentinel when mergedAnchors reuses the primary anchor object. - mergedAnchors: z.array(z.union([diffAnchorSchema, z.literal("[redacted:circular]")])).optional() -}).strict(); - -const verificationGateFactsSchema = z.object({ - anchorSource: z.enum(["model", "backfill_changed_code", "backfill_packet_representative", "verifier_revised"]).optional(), - category: z.enum(["logic_bug", "correctness", "security", "performance", "architecture", "testing", "maintainability"]), - changedLine: z.boolean(), confidence: z.enum(["high", "medium", "low"]), failureModeConcrete: z.boolean(), - hasChangedCode: z.boolean(), hasFailureMode: z.boolean(), modelAnchorSubmitted: z.boolean(), modelAnchorValid: z.boolean(), - relatedEvidenceCount: nonnegativeIntSchema, severity: z.enum(["critical", "high", "medium", "low"]), validAnchorPresent: z.boolean() -}).strict(); - -const candidateProvenanceSchema = z.object({ - source: z.literal("uncertainty_promotion"), sourceKind: z.enum(["uncertainty", "follow_up_hint"]), sourcePacketId: z.string().min(1), - question: z.string(), files: z.array(z.string()), symbols: z.array(z.string()), reason: z.string() -}).strict(); - -const verificationVerdictSchema = z.object({ - candidateId: z.string().min(1), verdict: z.enum(["keep", "reject", "revise", "incomplete"]), reason: z.string(), - requiredEvidencePresent: z.boolean(), falsePositiveRisk: z.enum(["low", "medium", "high"]), - finalFinding: candidateFindingSchema.optional(), revisedAnchor: diffAnchorSchema.optional(), verificationIncomplete: z.boolean().optional(), - behaviorChange: z.enum(["accidental_regression", "intentional_needs_confirmation", "specified_change", "unknown"]).optional(), - intentEvidence: z.array(z.string()).optional() -}).strict(); - -const verificationRecordBase = { - candidateId: z.string().min(1), gate: z.enum(["suppressed", "passed", "gate_anchor_stripped"]), - gateDecision: z.enum(["suppressed", "scheduled", "scheduled_for_evidence_resolution"]).optional(), gateReason: z.string().optional(), - verificationLane: z.enum(["standard", "evidence_resolution"]).optional(), gateFacts: verificationGateFactsSchema.optional(), - candidateProvenance: candidateProvenanceSchema.optional(), duplicateOf: z.string().optional(), clusterId: z.string().optional(), - verificationStatus: z.enum(["completed", "incomplete"]).optional(), incompleteReason: z.string().optional(), errorCode: z.string().optional() -}; - -const verificationRecordSchema = z.object({ ...verificationRecordBase, verdict: verificationVerdictSchema.optional() }).strict().superRefine((record, ctx) => { - if (record.gate === "suppressed" && record.gateReason === undefined) { - ctx.addIssue({ code: "custom", path: ["gateReason"], message: "suppressed verification records require gateReason" }); - } - if (record.gate === "passed" && record.verdict === undefined) { - ctx.addIssue({ code: "custom", path: ["verdict"], message: "passed verification records require a verdict" }); - } -}); - -const selectionRecordSchema = z.object({ - findingId: z.string().min(1), decision: z.enum(["published", "merged", "suppressed"]), reason: z.string(), - mergedIntoFingerprint: z.string().optional() -}).strict(); - -const finalSelectionArtifactSchema = z.object({ - composition: z.object({ - mode: z.enum(["llm", "llm_degraded", "deterministic_fallback", "schema_repair_fallback"]), fallbackReason: z.string().optional() - }).strict(), - records: z.array(selectionRecordSchema), - publicationAnchors: z.array(z.object({ - findingId: z.string().min(1), fingerprint: z.string().min(1), publication: z.enum(["inline", "summary-only", "suppressed"]), - source: z.enum(["selected", "merged", "none"]), reason: z.string(), sourceFindingId: z.string().optional(), anchor: diffAnchorSchema.optional() - }).strict()), - confidenceSelections: z.array(z.object({ - findingId: z.string().min(1), confidence: z.enum(["high", "medium", "low"]), representativeConfidence: z.enum(["high", "medium", "low"]), - sourceFindingId: z.string().optional(), reason: z.enum(["representative", "same_severity", "compatible_lower_severity"]) - }).strict()), - groups: z.array(z.object({ fingerprint: z.string().min(1), findingIds: z.array(z.string().min(1)).min(1) }).strict()) -}).strict(); - -type FinalSelectionArtifact = z.infer; - -const telemetryEventSchema = z.object({ - runId: z.string().min(1), - eventId: z.string().min(1), - timestamp: z.string().min(1), - stage: z.number().int().min(0).max(11), - level: z.enum(["debug", "info", "warn", "error"]), - message: z.string().min(1), - file: z.string().optional(), - lineRange: z.tuple([positiveIntSchema, positiveIntSchema]).optional(), - packetId: z.string().optional(), - lensId: z.string().optional(), - workerId: z.string().optional(), - durationMs: nonnegativeIntSchema.optional(), - cacheStatus: z.enum(["hit", "miss", "disabled", "write"]).optional(), - data: jsonObjectSchema.optional() -}).strict().superRefine((event, ctx) => { - if (event.message !== "same_file_atoms_packed") { - return; - } - const parsed = sameFilePackingEventDataSchema.safeParse(event.data); - if (!parsed.success) { - ctx.addIssue({ code: "custom", path: ["data"], message: "same_file_atoms_packed data must match the strict packing schema" }); - } -}); - -const llmCallRecordSchema = z.object({ - callId: z.string().min(1), runId: z.string().min(1), stage: z.number().int().min(0).max(11), - role: z.enum(["planner", "packetReview", "systemReview", "verifier", "composer"]), - model: z.string(), provider: z.string(), workerId: z.string().optional(), packetId: z.string().optional(), candidateId: z.string().optional(), - kind: z.enum(["initial", "tool-continuation", "repair", "finalize"]), - finalizeMode: z.enum(["compact", "full"]).optional(), finalizeTarget: z.enum(["no_findings", "candidate_or_unknown"]).optional(), - toolChoiceRequested: z.string().optional(), toolChoiceEffective: z.string().optional(), toolChoiceDowngraded: z.boolean().optional(), - reasoningRequested: z.string().optional(), reasoningMechanism: z.string().optional(), reasoningLevelEffective: z.string().optional(), - ttfbMs: nonnegativeIntSchema.optional(), providerHttpStatus: nonnegativeIntSchema.optional(), providerRequestId: z.string().optional(), - rateLimit: z.record(z.string(), z.string()).optional(), attempt: positiveIntSchema, promptChars: nonnegativeIntSchema, promptHash: z.string(), - outputChars: nonnegativeIntSchema, outputHash: z.string(), inputTokens: nonnegativeIntSchema.optional(), uncachedInputTokens: nonnegativeIntSchema.optional(), - cacheReadTokens: nonnegativeIntSchema.optional(), cacheWriteTokens: nonnegativeIntSchema.optional(), billableInputTokens: nonnegativeIntSchema.optional(), - outputTokens: nonnegativeIntSchema.optional(), reasoningTokens: nonnegativeIntSchema.optional(), totalTokens: nonnegativeIntSchema.optional(), - costUSD: z.number().nonnegative().optional(), inputCostUSD: z.number().nonnegative().optional(), outputCostUSD: z.number().nonnegative().optional(), - cacheReadCostUSD: z.number().nonnegative().optional(), cacheWriteCostUSD: z.number().nonnegative().optional(), durationMs: nonnegativeIntSchema, - cacheStatus: z.enum(["hit", "miss", "disabled", "write"]), schemaValid: z.boolean().optional(), - stopReason: z.enum(["submit", "tool_calls", "text", "error"]), - status: z.enum(["ok", "schema_invalid", "transient_error", "auth_error", "timeout", "aborted"]), - errorCode: z.string().optional(), errorMessage: z.string().optional(), retryable: z.boolean().optional(), retryReason: z.string().optional(), - maxAttempts: positiveIntSchema.optional(), retryExhausted: z.boolean().optional() -}).strict(); - -const toolCallRecordSchema = z.object({ - runId: z.string().min(1), toolCallId: z.string().min(1), timestamp: z.string().min(1), stage: z.number().int().min(1).max(11), - initiator: z.enum(["model", "harness"]), workerId: z.string().optional(), packetId: z.string().optional(), taskId: z.string().optional(), - candidateId: z.string().optional(), modelCallId: z.string().optional(), tool: z.string().min(1), args: z.object({ - path: z.string().optional(), symbolName: z.string().optional(), line: positiveIntSchema.optional(), startLine: positiveIntSchema.optional(), - endLine: positiveIntSchema.optional(), query: z.string().optional(), glob: z.string().optional(), source: z.string().optional(), - contextMode: z.string().optional(), maxResults: positiveIntSchema.optional() - }).strict(), - backend: z.enum(["tree-sitter", "text", "language-analyzer"]), precision: z.enum(["exact", "semantic", "syntactic", "heuristic", "text"]), - engine: z.literal("git-grep").optional(), degraded: z.boolean(), degradationReason: z.string().optional(), truncated: z.boolean().optional(), - omittedCount: nonnegativeIntSchema.optional(), lookupStatus: z.enum(["found", "not_found", "ambiguous", "file_missing", "unavailable"]).optional(), - deliveryStatus: z.enum(["full", "truncated", "budget_rejected", "empty"]).optional(), recovery: z.object({ - tool: z.literal("read_range"), path: z.string(), startLine: positiveIntSchema, endLine: positiveIntSchema, - source: z.enum(["head", "base"]), reason: z.string() - }).strict().optional(), - budgetState: z.object({ - toolCallsUsed: nonnegativeIntSchema, maxToolCalls: nonnegativeIntSchema, investigationRoundsUsed: nonnegativeIntSchema, - maxInvestigationRounds: nonnegativeIntSchema, resultCharsUsed: nonnegativeIntSchema, maxResultChars: nonnegativeIntSchema, - remainingResultChars: nonnegativeIntSchema, maxSingleToolResultChars: nonnegativeIntSchema.optional(), reservedSourceResultChars: nonnegativeIntSchema.optional(), - toolResultCharLimit: nonnegativeIntSchema.optional(), sourceExtensionCallsUsed: nonnegativeIntSchema.optional(), sourceExtensionMaxCalls: nonnegativeIntSchema.optional(), - sourceExtensionResultCharsUsed: nonnegativeIntSchema.optional(), sourceExtensionMaxResultChars: nonnegativeIntSchema.optional(), - sourceExtensionRemainingResultChars: nonnegativeIntSchema.optional(), sourceExtensionActive: z.boolean().optional() - }).strict().optional(), cacheStatus: z.enum(["hit", "miss", "disabled", "write"]).optional(), backendExecuted: z.boolean().optional(), - cacheHitKind: z.enum(["stored", "inflight"]).optional(), cacheEvictedEntries: nonnegativeIntSchema.optional(), resultCount: nonnegativeIntSchema.optional(), - resultChars: nonnegativeIntSchema, durationMs: nonnegativeIntSchema, status: z.enum(["ok", "error", "rejected", "skipped"]), errorCode: z.string().optional() -}).strict(); - -const cacheCountsSchema = z.object({ - hit: nonnegativeIntSchema, miss: nonnegativeIntSchema, disabled: nonnegativeIntSchema, write: nonnegativeIntSchema -}).strict(); -const providerPromptCacheSchema = z.object({ - readTokens: nonnegativeIntSchema, writeTokens: nonnegativeIntSchema, - readCostUSD: z.number().nonnegative(), writeCostUSD: z.number().nonnegative() -}).strict(); -const modelFinalizeSummarySchema = z.object({ - compactCalls: nonnegativeIntSchema, fullCalls: nonnegativeIntSchema, noFindingCalls: nonnegativeIntSchema, - candidateOrUnknownCalls: nonnegativeIntSchema, promptChars: nonnegativeIntSchema, noFindingPromptChars: nonnegativeIntSchema, - candidateOrUnknownPromptChars: nonnegativeIntSchema, costUSD: z.number().nonnegative(), noFindingCostUSD: z.number().nonnegative(), - candidateOrUnknownCostUSD: z.number().nonnegative(), unknownCostCalls: nonnegativeIntSchema -}).strict(); -const schemaRecoveryCountersSchema = z.object({ - schemaInvalidCalls: nonnegativeIntSchema, schemaInvalidRecovered: nonnegativeIntSchema, schemaInvalidUnrecovered: nonnegativeIntSchema, - schemaRepairAttempts: nonnegativeIntSchema, schemaRepairRecovered: nonnegativeIntSchema, - deterministicSchemaRecovered: nonnegativeIntSchema, schemaRecoveryFailed: nonnegativeIntSchema -}).strict(); -const schemaRecoverySummarySchema = schemaRecoveryCountersSchema.extend({ - byStage: z.record(z.string().regex(/^\d+$/u), schemaRecoveryCountersSchema) -}).strict(); -const modelSummaryBaseShape = { - totalRecords: nonnegativeIntSchema, totalCalls: nonnegativeIntSchema, providerCalls: nonnegativeIntSchema, - inputTokens: nonnegativeIntSchema, uncachedInputTokens: nonnegativeIntSchema, cacheReadTokens: nonnegativeIntSchema, - cacheWriteTokens: nonnegativeIntSchema, billableInputTokens: nonnegativeIntSchema, outputTokens: nonnegativeIntSchema, - reasoningTokens: nonnegativeIntSchema, totalTokens: nonnegativeIntSchema, costUSD: z.number().nonnegative(), - inputCostUSD: z.number().nonnegative(), outputCostUSD: z.number().nonnegative(), cacheReadCostUSD: z.number().nonnegative(), - cacheWriteCostUSD: z.number().nonnegative(), unknownCostCalls: nonnegativeIntSchema, cache: cacheCountsSchema, - retryAttempts: nonnegativeIntSchema, repairCalls: nonnegativeIntSchema, schemaInvalidCalls: nonnegativeIntSchema, - toolChoiceDowngradedCalls: nonnegativeIntSchema, finalize: modelFinalizeSummarySchema -}; -const modelStageSummarySchema = z.object({ - recordCount: nonnegativeIntSchema, count: nonnegativeIntSchema, providerCalls: nonnegativeIntSchema, - inputTokens: nonnegativeIntSchema, uncachedInputTokens: nonnegativeIntSchema, cacheReadTokens: nonnegativeIntSchema, - cacheWriteTokens: nonnegativeIntSchema, billableInputTokens: nonnegativeIntSchema, outputTokens: nonnegativeIntSchema, - reasoningTokens: nonnegativeIntSchema, totalTokens: nonnegativeIntSchema, costUSD: z.number().nonnegative(), - inputCostUSD: z.number().nonnegative(), outputCostUSD: z.number().nonnegative(), cacheReadCostUSD: z.number().nonnegative(), - cacheWriteCostUSD: z.number().nonnegative(), unknownCostCalls: nonnegativeIntSchema, cache: cacheCountsSchema, - retryAttempts: nonnegativeIntSchema, repairCalls: nonnegativeIntSchema, schemaInvalidCalls: nonnegativeIntSchema, - statuses: z.object({ - ok: nonnegativeIntSchema, schema_invalid: nonnegativeIntSchema, transient_error: nonnegativeIntSchema, - auth_error: nonnegativeIntSchema, timeout: nonnegativeIntSchema, aborted: nonnegativeIntSchema - }).strict(), - finalize: modelFinalizeSummarySchema, localModelCallCache: cacheCountsSchema, - providerPromptCache: providerPromptCacheSchema, schemaRecovery: schemaRecoveryCountersSchema -}).strict(); -const modelCallsSummarySchema = z.object({ - ...modelSummaryBaseShape, - localModelCallCache: cacheCountsSchema, - providerPromptCache: providerPromptCacheSchema, - schemaRecovery: schemaRecoverySummarySchema, - byStage: z.record(z.string().regex(/^\d+$/u), modelStageSummarySchema) -}).strict(); -const costBreakdownSchema = z.object({ - uncachedInput: z.object({ tokens: nonnegativeIntSchema, costUSD: z.number().nonnegative() }).strict(), - providerPromptCacheRead: z.object({ tokens: nonnegativeIntSchema, costUSD: z.number().nonnegative() }).strict(), - providerPromptCacheWrite: z.object({ tokens: nonnegativeIntSchema, costUSD: z.number().nonnegative() }).strict(), - output: z.object({ tokens: nonnegativeIntSchema, costUSD: z.number().nonnegative() }).strict(), - total: z.object({ tokens: nonnegativeIntSchema, costUSD: z.number().nonnegative() }).strict() -}).strict(); -const costProfileStageSchema = modelStageSummarySchema.omit({ schemaRecovery: true }).extend({ costBreakdown: costBreakdownSchema }).strict(); -const costProfileSchema = z.object({ - totalCostUSD: z.number().nonnegative(), unknownCostCalls: nonnegativeIntSchema, - localModelCallCache: cacheCountsSchema, providerPromptCache: providerPromptCacheSchema, costBreakdown: costBreakdownSchema, - tokens: z.object({ - inputTokens: nonnegativeIntSchema, uncachedInputTokens: nonnegativeIntSchema, cacheReadTokens: nonnegativeIntSchema, - cacheWriteTokens: nonnegativeIntSchema, billableInputTokens: nonnegativeIntSchema, outputTokens: nonnegativeIntSchema, - reasoningTokens: nonnegativeIntSchema, totalTokens: nonnegativeIntSchema - }).strict(), - cost: z.object({ - inputCostUSD: z.number().nonnegative(), outputCostUSD: z.number().nonnegative(), cacheReadCostUSD: z.number().nonnegative(), - cacheWriteCostUSD: z.number().nonnegative(), totalCostUSD: z.number().nonnegative() - }).strict(), - byStage: z.record(z.string().regex(/^\d+$/u), costProfileStageSchema) -}).strict(); -const toolResultCacheSummarySchema = z.object({ - hits: nonnegativeIntSchema, misses: nonnegativeIntSchema, writes: nonnegativeIntSchema, disabled: nonnegativeIntSchema, - inflightHits: nonnegativeIntSchema, evictions: nonnegativeIntSchema, backendExecutions: nonnegativeIntSchema, - savedBackendCalls: nonnegativeIntSchema -}).strict(); -const toolBucketSummarySchema = z.object({ - count: nonnegativeIntSchema, errors: nonnegativeIntSchema, rejections: nonnegativeIntSchema, degraded: nonnegativeIntSchema, - backendExecutions: nonnegativeIntSchema, savedBackendCalls: nonnegativeIntSchema, totalDurationMs: nonnegativeIntSchema, - totalResultChars: nonnegativeIntSchema, resultCache: toolResultCacheSummarySchema, - averageDurationMs: z.number().nonnegative(), averageResultChars: z.number().nonnegative() -}).strict(); -const toolCallsSummarySchema = z.object({ - totalCalls: nonnegativeIntSchema, resultCache: toolResultCacheSummarySchema, - byTool: z.record(z.string().min(1), toolBucketSummarySchema), byStage: z.record(z.string().regex(/^\d+$/u), toolBucketSummarySchema) -}).strict(); - -const evalDeclaredExpectationSchema = z.object({ - id: z.string().min(1), tier: z.enum(["required", "optional"]).optional(), path: z.string().min(1).optional(), - lineRange: z.tuple([positiveIntSchema, positiveIntSchema]).optional(), - category: z.enum(["logic_bug", "correctness", "security", "performance", "architecture", "testing", "maintainability"]).optional(), - severityAtLeast: z.enum(["critical", "high", "medium", "low"]).optional(), titlePattern: z.string().min(1).optional(), - failureModePattern: z.string().min(1).optional(), minRecallRate: z.number().min(0).max(1).optional(), minCandidateRate: z.number().min(0).max(1).optional() -}).strict(); - -const evalCaseArtifactSchema = z.object({ - name: z.string().min(1), repeat: positiveIntSchema.optional(), - repo: z.object({ external: z.string().min(1).optional(), fixture: z.string().min(1).optional() }).strict().optional(), - command: z.object({ - pr: positiveIntSchema.optional(), branch: z.string().min(1).optional(), head: z.string().min(1).optional(), - base: z.string().min(1).optional(), target: z.string().min(1).optional() - }).strict().optional(), - review: z.object({ - depth: z.enum(["light", "normal", "deep"]).optional(), lenses: z.array(z.string().min(1)).optional(), maxFindings: positiveIntSchema.optional(), - concurrency: positiveIntSchema.optional(), budgetBoost: z.number().positive().optional(), packSameFileHunks: z.boolean().optional(), - packedToolBudgetMode: z.enum(["base", "atom-scaled"]).optional(), maxTimeMinutes: z.number().positive().optional(), - maxBudgetTokens: positiveIntSchema.optional(), deepEnsemblePasses: positiveIntSchema.optional(), adaptiveSecondPass: z.boolean().optional(), - verify: z.boolean().optional(), cache: z.boolean().optional(), cacheDir: z.string().min(1).optional(), debug: z.boolean().optional(), - provider: z.string().min(1).optional(), model: z.string().min(1).optional(), reasoning: z.string().min(1).optional() - }).strict().optional(), - llm: z.object({ - provider: z.string().min(1).optional(), model: z.string().min(1).optional(), reasoning: z.string().min(1).optional(), - maxConcurrentCalls: positiveIntSchema.optional() - }).strict().optional(), - logs: z.object({ dir: z.string().min(1).optional() }).strict().optional(), - artifacts: z.object({ path: z.string().min(1) }).strict().optional(), - expect: z.object({ - minFindings: z.number().nonnegative().optional(), maxFindings: z.number().positive().optional(), maxDuplicateGroups: z.number().positive().optional(), - maxCostUSD: z.number().positive().optional(), maxElapsedSeconds: z.number().positive().optional(), maxModelCalls: z.number().positive().optional(), - maxToolCalls: z.number().positive().optional(), maxPromptCharsByStage: z.record(z.string(), positiveIntSchema).optional(), - reviewCompleteness: z.enum(["complete", "partial"]).optional(), maxBudgetOverruns: nonnegativeIntSchema.optional(), - maxToolBudgetRejections: nonnegativeIntSchema.optional(), maxDegradedHunks: nonnegativeIntSchema.optional(), - maxUnresolvedNotesSuppressed: nonnegativeIntSchema.optional() - }).strict().optional(), - should_find: z.array(evalDeclaredExpectationSchema).optional(), - should_find_candidate: z.array(evalDeclaredExpectationSchema).optional(), - should_not_find: z.array(evalDeclaredExpectationSchema).optional() -}).strict(); - -const evalExpectationResultSchema = z.object({ - expectationId: z.string().min(1), list: z.enum(["should_find", "should_find_candidate", "should_not_find"]), - tier: z.enum(["required", "optional"]).optional(), status: z.enum(["pass", "fail", "skipped"]), skipReason: z.string().optional(), - fromReplayedArtifacts: z.boolean().optional(), - matched: z.array(z.object({ findingId: z.string().min(1), artifact: z.enum(["final-findings", "candidate-findings"]) }).strict()), - loss: z.object({ - label: z.enum(["missed-before-candidate-generation", "lost-at-verification", "lost-at-composition", "partial-match"]), - subReason: z.string().optional(), nearestInstances: z.array(z.object({ - findingId: z.string().optional(), artifact: z.enum(["final-findings", "final-selection", "verification", "candidate-findings", "events"]), - outcome: z.string(), fieldMismatches: z.array(z.object({ - field: z.enum(["path", "lineRange", "category", "severityAtLeast", "titlePattern", "failureModePattern"]), - present: z.boolean(), matched: z.boolean(), expected: z.string().optional(), actual: z.string().optional(), via: z.string().optional() - }).strict()).optional() - }).strict()), - matchingHints: z.array(z.object({ - packetId: z.string().optional(), question: z.string(), files: z.array(z.string()), symbols: z.array(z.string()), - confidence: z.enum(["high", "medium", "low"]) - }).strict()).optional(), coveringPacketIds: z.array(z.string()).optional(), coveringPacketLenses: z.array(z.string()).optional(), - plannerCoverage: z.string().optional(), surfacedAsNote: z.boolean().optional() - }).strict().optional(), note: z.string().optional() -}).strict(); - -const evalBudgetResultSchema = z.object({ - check: z.string().min(1), stage: z.number().int().min(1).max(11).optional(), status: z.enum(["pass", "fail", "skipped"]), - skipReason: z.string().optional(), limit: z.number().optional(), actual: z.number().optional(), expected: z.string().optional(), - actualText: z.string().optional(), direction: z.enum(["minimum", "maximum", "equals"]), fromReplayedArtifacts: z.boolean().optional() -}).strict(); - -const evalMetricsSchema = z.object({ - reportedFindings: nonnegativeIntSchema, inlineFindings: nonnegativeIntSchema, summaryOnlyFindings: nonnegativeIntSchema, - suppressedFindings: nonnegativeIntSchema, candidateFindings: nonnegativeIntSchema, duplicateGroups: nonnegativeIntSchema, - costUSD: z.number().nonnegative().optional(), elapsedSeconds: z.number().nonnegative().optional(), modelCalls: nonnegativeIntSchema.optional(), - verificationCalls: nonnegativeIntSchema.optional(), toolCalls: nonnegativeIntSchema.optional(), toolChoiceDowngradedCalls: nonnegativeIntSchema.optional(), - attentionEfficiency: jsonObjectSchema.optional(), missingArtifacts: z.array(z.string()).optional(), maxPromptCharsByStage: jsonObjectSchema.optional(), - reviewCompleteness: z.enum(["complete", "partial"]).optional(), budgetOverruns: nonnegativeIntSchema.optional(), - toolBudgetRejections: nonnegativeIntSchema.optional(), toolBudgetExtensions: nonnegativeIntSchema.optional(), toolBudgetExtensionDenials: nonnegativeIntSchema.optional(), - degradedHunks: nonnegativeIntSchema.optional(), unresolvedNotesSuppressed: nonnegativeIntSchema.optional(), localModelCallCacheHits: nonnegativeIntSchema.optional(), - localModelCallCacheMisses: nonnegativeIntSchema.optional(), localModelCallCacheWrites: nonnegativeIntSchema.optional(), - providerPromptCacheReadTokens: nonnegativeIntSchema.optional(), providerPromptCacheWriteTokens: nonnegativeIntSchema.optional(), - providerPromptCacheReadCostUSD: z.number().nonnegative().optional(), providerPromptCacheWriteCostUSD: z.number().nonnegative().optional(), - reasoningTokens: nonnegativeIntSchema.optional(), schemaInvalidCalls: nonnegativeIntSchema.optional(), schemaInvalidRecovered: nonnegativeIntSchema.optional(), - schemaInvalidUnrecovered: nonnegativeIntSchema.optional(), schemaRepairAttempts: nonnegativeIntSchema.optional(), schemaRepairRecovered: nonnegativeIntSchema.optional(), - deterministicSchemaRecovered: nonnegativeIntSchema.optional(), schemaRecoveryFailed: nonnegativeIntSchema.optional(), cacheHits: nonnegativeIntSchema.optional(), - cacheMisses: nonnegativeIntSchema.optional(), - stageLossCounts: z.object({ - "missed-before-candidate-generation": nonnegativeIntSchema, - "lost-at-verification": nonnegativeIntSchema, - "lost-at-composition": nonnegativeIntSchema, - "partial-match": nonnegativeIntSchema - }).strict() -}).strict(); - -const evalScoreSchema = z.object({ - status: z.enum(["pass", "fail", "error"]), expectationResults: z.array(evalExpectationResultSchema), budgetResults: z.array(evalBudgetResultSchema), - violations: z.array(z.object({ expectationId: z.string(), findingId: z.string(), publication: z.enum(["inline", "summary-only"]) }).strict()), - nearViolations: z.array(z.object({ expectationId: z.string(), findingId: z.string(), artifact: z.string() }).strict()), - metrics: evalMetricsSchema, error: z.object({ code: z.string(), message: z.string() }).strict().optional() -}).strict(); - -const evalRepeatExpectationSchema = z.object({ - expectationId: z.string().min(1), list: z.enum(["should_find", "should_find_candidate", "should_not_find"]), - tier: z.enum(["required", "optional"]), finalMatched: nonnegativeIntSchema, candidateMatched: nonnegativeIntSchema, - noteSurfaced: nonnegativeIntSchema, finalRecallRate: z.number().min(0).max(1), candidateRecallRate: z.number().min(0).max(1), - noteRate: z.number().min(0).max(1), lossHistogram: z.record(z.string(), nonnegativeIntSchema), fingerprintsStable: z.boolean().optional(), - distinctFingerprints: nonnegativeIntSchema.optional(), gate: z.object({ - minRecallRate: z.number().min(0).max(1).optional(), minCandidateRate: z.number().min(0).max(1).optional(), passed: z.boolean() - }).strict().optional() -}).strict(); - -const evalRepeatAggregateSchema = z.object({ - repeat: positiveIntSchema, - executions: z.array(z.object({ runDir: z.string().min(1), status: z.enum(["pass", "fail", "error"]) }).strict()), - expectations: z.array(evalRepeatExpectationSchema), - totals: z.object({ costUSD: z.number().nonnegative(), elapsedSeconds: z.number().nonnegative(), errors: nonnegativeIntSchema }).strict() -}).strict(); - -const evalRepoProvenanceSchema = z.object({ - root: z.string().min(1), baseSha: commitShaSchema.optional(), headSha: commitShaSchema.optional(), mergeBase: commitShaSchema.optional() -}).strict(); - -const codegenieRuntimeSchema = z.object({ - packageVersion: z.string().min(1), commit: commitShaSchema.optional(), shortCommit: z.string().min(1).optional(), - branch: z.string().min(1).optional(), dirty: z.boolean().optional(), source: z.enum(["build_env", "git", "package", "unknown"]) -}).strict(); - -const attentionRecordSchema = z.object({ - packetId: z.string().min(1), path: z.string().min(1), coverage: z.enum(["deep", "normal", "light"]), - coverageSource: z.union([z.enum(["planner", "deterministic_default"]), z.string().regex(/^escalated:.+/u)]), - ensemblePasses: nonnegativeIntSchema, directCandidates: nonnegativeIntSchema, promotedCandidates: nonnegativeIntSchema, - hintsEmitted: nonnegativeIntSchema, uncertaintiesEmitted: nonnegativeIntSchema, keptVerified: nonnegativeIntSchema, - published: nonnegativeIntSchema -}).strict(); -const attentionDroppedPathSchema = z.object({ path: z.string(), reason: z.string() }).strict(); -const attentionNoteSchema = z.object({ - question: z.string(), files: z.array(z.string()), symbols: z.array(z.string()), reason: z.string(), - confidence: z.enum(["high", "medium"]), sourcePacketIds: z.array(z.string()).optional() -}).strict(); -const attentionRawHintSchema = z.object({ - id: z.string(), source: z.enum(["follow_up_hint", "uncertainty"]), packetId: z.string(), question: z.string(), reason: z.string(), - confidence: z.enum(["high", "medium", "low"]), files: z.array(z.string()), originalFiles: z.array(z.string()), - droppedPaths: z.array(attentionDroppedPathSchema), symbols: z.array(z.string()), suggestedLenses: z.array(z.string()) -}).strict(); -const attentionGroupSchema = z.object({ - key: z.string(), noteIds: z.array(z.string()), question: z.string(), reason: z.string(), confidence: z.enum(["high", "medium"]), - files: z.array(z.string()), droppedPaths: z.array(attentionDroppedPathSchema), invalidPathCount: nonnegativeIntSchema, - symbols: z.array(z.string()), reasons: z.array(z.string()), packetIds: z.array(z.string()), - sources: z.array(z.enum(["follow_up_hint", "uncertainty"])), count: positiveIntSchema -}).strict(); -const humanAttentionArtifactSchema = z.union([ - z.array(z.object({ question: z.string(), files: z.array(z.string()), reasons: z.array(z.string()).optional(), reason: z.string().optional() }).strict()), - z.object({ - schemaVersion: z.literal(2), notes: z.array(attentionRawHintSchema), groups: z.array(attentionGroupSchema), - mergeStats: z.object({ exactDuplicateHints: nonnegativeIntSchema, nearDuplicateHints: nonnegativeIntSchema, nearDuplicateGroupsMerged: nonnegativeIntSchema }).strict(), - composerPromptGroupIds: z.array(z.string()), outputGroupIds: z.array(z.string()), outputNotes: z.array(attentionNoteSchema), - omittedCount: nonnegativeIntSchema, - suppressedByFindings: z.array(z.object({ groupKey: z.string(), noteIds: z.array(z.string()) }).strict()), - suppressedByVerification: z.array(z.object({ - groupKey: z.string(), noteIds: z.array(z.string()), candidateId: z.string(), verdict: z.enum(["keep", "reject", "revise", "incomplete"]), - reason: z.string(), verdictReason: z.string(), match: z.object({ - sharedFiles: z.array(z.string()), sharedSymbols: z.array(z.string()), sharedTerms: nonnegativeIntSchema, - similarity: z.number().min(0).max(1), questionMatched: z.boolean(), provenanceMatched: z.boolean() - }).strict() - }).strict()), - keptForOutputGroupIds: z.array(z.string()) - }).strict() -]); -const budgetLimitEventSchema = z.object({ - stage: z.number().int().min(0).max(11), reason: z.enum(["runtime_reserved_tail", "max_model_calls", "max_budget_tokens", "hard_timeout"]), - elapsedMs: nonnegativeIntSchema, kind: z.enum(["runtime", "model_calls", "tokens"]), actual: z.number().nonnegative(), - limit: z.number().nonnegative(), totalTokens: nonnegativeIntSchema, modelCalls: nonnegativeIntSchema, afterDispatchedCall: z.boolean() -}).strict(); -const contextPressureSummarySchema = z.object({ - toolBudgetRejections: nonnegativeIntSchema, toolBudgetRejectionsByStage: z.record(z.string().regex(/^\d+$/u), nonnegativeIntSchema), - toolBudgetExtensions: z.object({ - granted: nonnegativeIntSchema, denied: nonnegativeIntSchema, resultChars: nonnegativeIntSchema, - grantedByStage: z.record(z.string().regex(/^\d+$/u), nonnegativeIntSchema), - deniedByStage: z.record(z.string().regex(/^\d+$/u), nonnegativeIntSchema) - }).strict().optional(), - degradedToolResults: nonnegativeIntSchema, degradedToolResultsByStage: z.record(z.string().regex(/^\d+$/u), nonnegativeIntSchema), - degradedHunks: nonnegativeIntSchema, rejectionReasons: z.array(z.object({ reason: z.string(), count: positiveIntSchema }).strict()), - unresolvedNotes: z.object({ emitted: nonnegativeIntSchema, omitted: nonnegativeIntSchema }).strict() -}).strict(); -const budgetSummarySchema = z.object({ - completeness: z.enum(["complete", "partial"]), partialReasons: z.array(z.string()), multiplier: z.number().positive(), - configured: z.object({ timeoutMs: positiveIntSchema, maxModelCalls: positiveIntSchema.optional(), maxBudgetTokens: positiveIntSchema.optional() }).strict(), - effective: z.object({ timeoutMs: positiveIntSchema, maxModelCalls: positiveIntSchema.optional(), maxBudgetTokens: positiveIntSchema.optional() }).strict(), - usage: z.object({ - modelCalls: nonnegativeIntSchema, totalTokens: nonnegativeIntSchema, costUSD: z.number().nonnegative().optional(), - byStage: z.array(z.object({ stage: z.number().int().min(1).max(11), modelCalls: nonnegativeIntSchema, totalTokens: nonnegativeIntSchema }).strict()) - }).strict(), - overruns: z.array(budgetLimitEventSchema), dispatchBlocks: z.array(budgetLimitEventSchema), contextPressure: contextPressureSummarySchema.optional() -}).strict(); -const budgetStopSchema = z.object({ - reason: z.enum(["runtime_reserved_tail", "max_model_calls", "max_budget_tokens", "hard_timeout"]), stage: z.number().int().min(0).max(11), - elapsedMs: nonnegativeIntSchema, timeoutMs: positiveIntSchema, hardTimeoutMs: positiveIntSchema, remainingRuntimeMs: nonnegativeIntSchema, - reservedTailRuntimeMs: nonnegativeIntSchema, modelCalls: nonnegativeIntSchema, inFlightModelCalls: nonnegativeIntSchema, - projectedModelCalls: nonnegativeIntSchema, maxModelCalls: positiveIntSchema.optional(), remainingModelCalls: nonnegativeIntSchema.optional(), - reservedModelCalls: nonnegativeIntSchema.optional(), totalTokens: nonnegativeIntSchema, inFlightTokens: nonnegativeIntSchema, - projectedTokens: nonnegativeIntSchema, maxBudgetTokens: positiveIntSchema.optional(), remainingTokens: nonnegativeIntSchema.optional(), - reservedTokens: nonnegativeIntSchema.optional() -}).strict(); -const runReviewSummarySchema = z.object({ - mode: z.string(), target: z.unknown().nullable(), prNumber: positiveIntSchema.nullable(), baseRef: z.string().nullable(), headRef: z.string().nullable(), - baseSha: z.string().nullable(), headSha: z.string().nullable(), depth: z.string().nullable(), concurrency: positiveIntSchema.nullable(), - budgetBoost: z.number().positive().nullable(), llmMaxConcurrentCalls: positiveIntSchema.nullable(), lenses: z.array(z.string()), - format: z.string().nullable(), postGithubComments: z.boolean() -}).strict(); -const modelTotalsSchema = z.object({ - events: nonnegativeIntSchema, modelCallRecords: nonnegativeIntSchema, modelCalls: nonnegativeIntSchema, providerCalls: nonnegativeIntSchema, - toolCalls: nonnegativeIntSchema, toolResultCache: toolResultCacheSummarySchema, - inputTokens: nonnegativeIntSchema, uncachedInputTokens: nonnegativeIntSchema, cacheReadTokens: nonnegativeIntSchema, - cacheWriteTokens: nonnegativeIntSchema, billableInputTokens: nonnegativeIntSchema, outputTokens: nonnegativeIntSchema, - reasoningTokens: nonnegativeIntSchema, totalTokens: nonnegativeIntSchema, totalCostUSD: z.number().nonnegative(), - inputCostUSD: z.number().nonnegative(), outputCostUSD: z.number().nonnegative(), cacheReadCostUSD: z.number().nonnegative(), - cacheWriteCostUSD: z.number().nonnegative(), costBreakdown: costBreakdownSchema, unknownCostCalls: nonnegativeIntSchema, - cache: cacheCountsSchema, localModelCallCache: cacheCountsSchema, providerPromptCache: providerPromptCacheSchema, - retryAttempts: nonnegativeIntSchema, repairCalls: nonnegativeIntSchema, schemaInvalidCalls: nonnegativeIntSchema, - schemaRecovery: schemaRecoverySummarySchema, - stage7SchemaRepair: z.object({ - candidateInvalidSubmits: nonnegativeIntSchema, noFindingInvalidSubmits: nonnegativeIntSchema, cleanupAttempted: nonnegativeIntSchema, - cleanupRecovered: nonnegativeIntSchema, cleanupRejected: nonnegativeIntSchema, compactRepairScheduled: nonnegativeIntSchema, - appendRepairScheduled: nonnegativeIntSchema, repairRecovered: nonnegativeIntSchema, repairFailed: nonnegativeIntSchema, - repairPromptChars: nonnegativeIntSchema, compactRepairPromptChars: nonnegativeIntSchema, appendRepairPromptChars: nonnegativeIntSchema, - actualRepairCalls: nonnegativeIntSchema, actualRepairPromptChars: nonnegativeIntSchema - }).strict(), - logOverflow: z.object({ droppedDebugInfo: nonnegativeIntSchema, droppedWarnError: nonnegativeIntSchema }).strict(), - filesChanged: nonnegativeIntSchema, hunks: nonnegativeIntSchema, packets: nonnegativeIntSchema, packetReviews: nonnegativeIntSchema, - candidates: nonnegativeIntSchema, verified: nonnegativeIntSchema, finalFindings: nonnegativeIntSchema, postedComments: nonnegativeIntSchema -}).strict(); -const evalRunTelemetrySchema = z.object({ - schemaVersion: z.literal(1), runId: z.string().min(1), codegenieVersion: z.string(), codegenieRuntime: codegenieRuntimeSchema, - nodeVersion: z.string(), argv: z.array(z.string()), repoRoot: z.string().nullable(), review: runReviewSummarySchema, - startedAt: z.string(), finishedAt: z.string(), completedAt: z.string(), durationMs: nonnegativeIntSchema, - outcome: z.object({ status: z.enum(["completed_full", "completed_partial", "failed"]), errorCode: z.string().nullable(), exitCode: z.number().int(), budgetStop: budgetStopSchema.nullable() }).strict(), - budgetStop: budgetStopSchema.optional(), totals: modelTotalsSchema -}).strict(); -const telemetryStageSummarySchema = z.object({ - events: nonnegativeIntSchema, - levels: z.object({ debug: nonnegativeIntSchema, info: nonnegativeIntSchema, warn: nonnegativeIntSchema, error: nonnegativeIntSchema }).strict(), - cache: cacheCountsSchema, startedAt: z.string().optional(), completedAt: z.string().optional(), runtimeMs: nonnegativeIntSchema, - schemaRecovery: schemaRecoveryCountersSchema -}).strict(); -const pipelineTelemetrySummaryShape = { - workers: z.object({ started: nonnegativeIntSchema, completed: nonnegativeIntSchema, failed: nonnegativeIntSchema, retried: nonnegativeIntSchema, timedOut: nonnegativeIntSchema }).strict(), - packets: z.object({ generated: nonnegativeIntSchema, reviewed: nonnegativeIntSchema, failed: nonnegativeIntSchema, degraded: nonnegativeIntSchema }).strict(), - lenses: z.object({ selected: nonnegativeIntSchema, byLens: z.record(z.string(), nonnegativeIntSchema) }).strict(), - coverage: z.object({ - byLevel: z.object({ deep: nonnegativeIntSchema, normal: nonnegativeIntSchema, light: nonnegativeIntSchema, skip: nonnegativeIntSchema }).strict(), - hunks: z.object({ total: nonnegativeIntSchema, reviewed: nonnegativeIntSchema, skipped: nonnegativeIntSchema, failed: nonnegativeIntSchema, degraded: nonnegativeIntSchema }).strict() - }).strict(), - candidates: z.object({ - generated: nonnegativeIntSchema, gateRejected: nonnegativeIntSchema, verificationScheduled: nonnegativeIntSchema, - verificationBudgetLimited: nonnegativeIntSchema, clusteredDuplicates: nonnegativeIntSchema, verificationRepresentatives: nonnegativeIntSchema, - lowConfidenceSuppressed: nonnegativeIntSchema, lowConfidenceEvidenceEligible: nonnegativeIntSchema, - lowConfidenceEvidenceScheduled: nonnegativeIntSchema, lowConfidenceEvidenceLaneLimited: nonnegativeIntSchema, - lowConfidenceEvidenceKept: nonnegativeIntSchema, lowConfidenceEvidenceRejected: nonnegativeIntSchema, - lowConfidenceEvidenceIncomplete: nonnegativeIntSchema - }).strict(), - verdicts: z.object({ accept: nonnegativeIntSchema, revise: nonnegativeIntSchema, reject: nonnegativeIntSchema, incomplete: nonnegativeIntSchema }).strict(), - dedup: z.object({ clusters: nonnegativeIntSchema, duplicates: nonnegativeIntSchema, suppressed: nonnegativeIntSchema }).strict(), - finalSelection: z.object({ published: nonnegativeIntSchema, merged: nonnegativeIntSchema, suppressed: nonnegativeIntSchema, finalFindings: nonnegativeIntSchema, compositionMode: z.string().nullable(), fallbackReason: z.string().nullable() }).strict(), - posting: z.object({ attempted: nonnegativeIntSchema, postedComments: nonnegativeIntSchema, skippedDuplicates: nonnegativeIntSchema, failed: nonnegativeIntSchema }).strict() -}; -const telemetrySummarySchema = z.object({ - schemaVersion: z.literal(1), runId: z.string().min(1), codegenieRuntime: codegenieRuntimeSchema, startedAt: z.string(), finishedAt: z.string(), - completedAt: z.string(), durationMs: nonnegativeIntSchema, logLevel: z.enum(["debug", "info", "warn", "error"]), debugTrace: z.boolean(), - events: nonnegativeIntSchema, logs: z.object({ bufferedOverflow: z.object({ droppedDebugInfo: nonnegativeIntSchema, droppedWarnError: nonnegativeIntSchema }).strict() }).strict(), - budgetStop: budgetStopSchema.optional(), totals: modelTotalsSchema, stages: z.record(z.string().regex(/^\d+$/u), telemetryStageSummarySchema), - ...pipelineTelemetrySummaryShape, - schemaRecovery: schemaRecoverySummarySchema, - schemaRepair: z.object({ stage7: modelTotalsSchema.shape.stage7SchemaRepair }).strict(), - modelCalls: modelCallsSummarySchema, toolCalls: toolCallsSummarySchema -}).strict(); - -const invocationIdSchema = z.string().regex(/^[a-zA-Z0-9_-]+$/u); -const evalInvocationReferenceSchema = z.object({ - id: invocationIdSchema, caseIndex: nonnegativeIntSchema, manifest: z.string().regex(/^invocations\/[a-zA-Z0-9_-]+\.json$/u) -}).strict(); - -const evalInvocationCaseSchema = z.object({ - caseIndex: nonnegativeIntSchema, caseName: z.string().min(1), caseHash: z.string().min(1), caseFile: z.string().min(1) -}).strict(); - -const evalInvocationRunSchema = z.object({ - caseIndex: nonnegativeIntSchema, caseName: z.string().min(1), caseHash: z.string().min(1), runNumber: positiveIntSchema, - logsRoot: z.string().min(1).refine((value) => path.isAbsolute(value), "logsRoot must be absolute"), - runPath: z.string().regex(/^\d+$/u) -}).strict(); - -const evalInvocationManifestSchema = z.object({ - schemaVersion: z.literal(1), invocationId: invocationIdSchema, - suiteDir: z.string().min(1).refine((value) => path.isAbsolute(value), "suiteDir must be absolute"), - status: z.enum(["running", "complete"]), - startedAt: z.string().min(1), completedAt: z.string().min(1).optional(), cases: z.array(evalInvocationCaseSchema).min(1), runs: z.array(evalInvocationRunSchema) -}).strict(); - -const evalRunInfoSchema = z.object({ - runNumber: positiveIntSchema, caseName: z.string().min(1), caseFile: z.string().optional(), caseHash: z.string().min(1), - caseSnapshot: evalCaseArtifactSchema, mode: z.enum(["live", "replay"]), repeats: evalRepeatAggregateSchema.optional(), - replay: z.object({ sourceArtifacts: z.string().min(1), caseSource: z.enum(["yaml", "snapshot"]) }).strict().optional(), - repo: evalRepoProvenanceSchema.optional(), reviewRunId: z.string().optional(), codegenieRuntime: codegenieRuntimeSchema.optional(), - invocation: evalInvocationReferenceSchema.optional(), - cache: z.object({ enabled: z.boolean(), source: z.enum(["cli", "case", "config"]), dir: z.string().optional() }).strict(), - effectiveConfig: z.object({ - review: z.object({ - concurrency: positiveIntSchema, timeoutMs: positiveIntSchema, verify: z.boolean(), - minSeverity: z.enum(["critical", "high", "medium", "low"]).optional(), maxFindings: positiveIntSchema, - softCommentCap: positiveIntSchema, minConfidence: z.enum(["high", "medium", "low"]), - minInlineConfidence: z.enum(["high", "medium", "low"]), packSameFileHunks: z.boolean(), - packedToolBudgetMode: z.enum(["base", "atom-scaled"]), maxBudgetTokens: positiveIntSchema.optional() - }).strict(), - llm: z.object({ - provider: z.string().optional(), model: z.string().optional(), reasoning: z.string().optional(), maxConcurrentCalls: positiveIntSchema - }).strict() - }).strict().optional(), - startedAt: z.string().min(1), finishedAt: z.string().min(1), score: evalScoreSchema -}).strict(); - -const coverageArtifactSchema = z.object({ - status: z.object({ - totalHunks: nonnegativeIntSchema, reviewedHunks: nonnegativeIntSchema, skippedHunks: nonnegativeIntSchema, failedHunks: nonnegativeIntSchema, - coverageByLevel: z.object({ deep: nonnegativeIntSchema, normal: nonnegativeIntSchema, light: nonnegativeIntSchema, skip: nonnegativeIntSchema }).strict(), - degradedPlanning: z.boolean(), budgetStopped: z.boolean(), budgetStop: jsonObjectSchema.optional(), - unreviewedHunksByPath: z.array(jsonObjectSchema).optional(), verificationIncompleteCount: nonnegativeIntSchema, - verificationSkipped: z.boolean().optional(), partial: z.boolean(), reasons: z.array(z.string()) - }).strict(), - records: z.array(z.object({ - hunkId: z.string().min(1), path: z.string().min(1), coverage: z.enum(["deep", "normal", "light", "skip"]), - source: z.enum(["planner", "deterministic_default", "config"]), status: z.enum(["reviewed", "skipped", "review_failed", "degraded"]), - reason: z.string().optional() - }).strict()) -}).strict(); - -export type PacketPackingArtifactKind = - | "packet" | "event" | "model-call" | "tool-call" | "coverage" | "eval-info" - | "attention" | "human-attention" | "budget-summary" | "cost-profile" | "model-summary" | "tool-summary" | "run-summary" | "telemetry-summary"; - -export function validatePacketPackingArtifact(kind: PacketPackingArtifactKind, value: unknown): void { - const schemas: Record = { - packet: reviewPacketSchema, - event: telemetryEventSchema, - "model-call": llmCallRecordSchema, - "tool-call": toolCallRecordSchema, - coverage: coverageArtifactSchema, - "eval-info": evalRunInfoSchema, - attention: z.array(attentionRecordSchema), - "human-attention": humanAttentionArtifactSchema, - "budget-summary": budgetSummarySchema, - "cost-profile": costProfileSchema, - "model-summary": modelCallsSummarySchema, - "tool-summary": toolCallsSummarySchema, - "run-summary": evalRunTelemetrySchema, - "telemetry-summary": telemetrySummarySchema - }; - const parsed = schemas[kind].safeParse(value); - assertReport(parsed.success, "corrupt_artifact_schema", `invalid ${kind} artifact`, { - issues: parsed.success ? [] : parsed.error.issues.map((issue) => ({ path: issue.path.join("."), code: issue.code, message: issue.message })) - }); -} - -const sameFilePackingEventDataSchema = z.object({ - packetId: z.string().min(1), - atomIds: z.array(z.string().min(1)).min(1), - standaloneProfiles: z.array(z.enum(["simple", "standard", "investigate"])).min(1), - sourceAtomCount: z.number().int().positive(), - hunkCount: z.number().int().positive(), - effectiveCoverage: z.enum(["deep", "normal", "light"]), - requestedLensSignature: z.string(), - capUsage: z.object({ - hunks: z.number().int().nonnegative(), - maxHunks: z.number().int().positive(), - patchChars: z.number().int().nonnegative(), - maxPatchChars: z.number().int().positive() - }), - derivedPackedProfile: z.enum(["simple", "standard", "investigate"]), - profileFloor: z.enum(["simple", "standard", "investigate"]), - effectiveProfile: z.enum(["simple", "standard", "investigate"]), - profileFloorApplied: z.boolean(), - plannerLensesPreserved: z.boolean(), - toolBudgetMode: z.enum(["base", "atom-scaled"]), - baseToolBudget: toolBudgetSchema, - effectiveToolBudget: toolBudgetSchema -}).strict(); - -type PackingEventData = z.infer & { - baseToolBudget: ToolBudget; - effectiveToolBudget: ToolBudget; -}; - -export type ReportFailure = { - code: string; - message: string; - context?: Record; -}; - -export type CapturedTelemetry = { - events: Array>; - modelCalls: Array>; - toolCalls: Array>; -}; - -export type ReplayAnalysisInput = { - runId: string; - recordedPackets: ReviewPacket[]; - offPackets: ReviewPacket[]; - onPackets: ReviewPacket[]; - onEvents: Array | TelemetryEvent>; - fileFacts: FileFacts[]; - diff: UnifiedDiff; - plan: ReviewPlan; - expectedRefs?: { base: string; head: string }; - actualRefs?: { base: string; head: string }; - flagOffParityPackets?: ReviewPacket[]; - flagOffParityMigrations?: Array<{ code: string; packets: number; explanation: string }>; - modelCallsObserved?: number; -}; - -export type ReplayRow = { - runId: string; - offPackets: number; - onPackets: number; - reductionPercent: number; - sourceAtoms: number; - packedMultiAtomPackets: number; - reviewableHunks: number; - newCoveragePromotions: number; - hunkBijectionFailures: number; - atomInvariantFailures: number; - capViolations: number; - lensDrops: number; - highPriorityNoteOmissions: number; - deepContextDowngrades: number; - relatedContextOmissions: number; - derivedProfileDowngrades: number; - profileFloorApplications: number; - effectiveProfileDowngrades: number; - effectiveBudgetDowngrades: number; - invalidDispatchRanks: number; - modelCallsObserved: number; - schedulingMoves: number; - proxyPackedPackets: number; - countReconciliation: "historical-target" | "real-patch-74" | "real-patch-76" | "not-motivating-run"; - flagOffParityDifferences: { - missingPacketIds: string[]; - extraPacketIds: string[]; - changedPackets: Array<{ - packetId: string; - paths: string[]; - samples: Array<{ path: string; recorded: ValueFingerprint; rebuilt: ValueFingerprint }>; - }>; - historicalMigrations: Array<{ code: string; packets: number; explanation: string }>; - }; - contextComparison: { - contextTruncationsOff: number; - contextTruncationsOn: number; - relatedContextOmissions: number; - attentionNoteOmissions: number; - lensDrops: number; - }; - distribution: { - offHunksPerPacket: Record; - onHunksPerPacket: Record; - sourceAtomsPerOnPacket: Record; - onCoverage: Record; - onProfiles: Record; - onBudgetModes: Record; - maxOnHunks: number; - eligiblePackingPackets: number; - bypassPackets: number; - maxEligiblePackingPatchChars: number; - }; - packetMembership: Array<{ - packetId: string; - path: string; - hunkIds: string[]; - coverage: string; - lenses: string[]; - profile: string; - budget: ToolBudget; - dispatchRank: [number, number]; - atomIds: string[]; - }>; - failures: ReportFailure[]; -}; - -type ValueFingerprint = { - kind: "string" | "json"; - length: number; - sha256: string; -}; - -export type EvalExecutionInput = { - repeat: number; - score: EvalScore; - telemetryDir: string; - packets: ReviewPacket[]; - events: TelemetryEvent[]; - modelCalls: LlmCallRecord[]; - toolCalls: ToolCallRecord[]; - fileFacts: FileFacts[]; - diff: UnifiedDiff; - plan: ReviewPlan; - candidateFindings: CandidateFinding[]; - verification: EvalVerificationRecord[]; - finalSelection: EvalSelectionRecord[]; - finalSelectionArtifact: FinalSelectionArtifact; - finalFindings: FinalFinding[]; - scoringArtifacts: EvalArtifacts; - summaryArtifacts?: { - attention: unknown; - humanAttention: unknown; - budget: unknown; - cost: unknown; - model: unknown; - tool: unknown; - run: unknown; - telemetry: unknown; - }; - reviewedHunkIds: string[]; - wallTimeSeconds: number; -}; - -export type EvalCaseRunInput = { - runNumber: number; - runDir: string; - info: EvalRunInfo; - declaredCase: EvalCase; - invocationManifest?: EvalInvocationManifest; - executions: EvalExecutionInput[]; -}; - -export type CohortSelection = { - id: string; - runs: EvalCaseRunInput[]; -}; - -export class PacketPackingReportError extends Error { - readonly failures: ReportFailure[]; - - constructor(message: string, failures: ReportFailure[]) { - super(message); - this.name = "PacketPackingReportError"; - this.failures = failures; - } -} - -function failure(code: string, message: string, context?: Record): ReportFailure { - return { code, message, ...(context === undefined ? {} : { context }) }; -} - -function fail(code: string, message: string, context?: Record): never { - throw new PacketPackingReportError(message, [failure(code, message, context)]); -} - -function assertReport(condition: unknown, code: string, message: string, context?: Record): asserts condition { - if (!condition) { - fail(code, message, context); - } -} - -function isRecord(value: unknown): value is Record { - return value !== null && typeof value === "object" && !Array.isArray(value); -} - -function asFiniteNumber(value: unknown): number | undefined { - return typeof value === "number" && Number.isFinite(value) ? value : undefined; -} - -function stableValue(value: unknown): unknown { - if (Array.isArray(value)) { - return value.map(stableValue); - } - if (!isRecord(value)) { - return value; - } - return Object.fromEntries(Object.keys(value).sort().map((key) => [key, stableValue(value[key])])); -} - -function stableJson(value: unknown): string { - return JSON.stringify(stableValue(value)); -} - -function unique(values: T[]): T[] { - return [...new Set(values)]; -} - -function sortedUnique(values: string[]): string[] { - return unique(values.filter((value) => value.length > 0)).sort(); -} - -function sum(values: number[]): number { - return values.reduce((total, value) => total + value, 0); -} - -function countDistribution(values: Array): Record { - const entries = new Map(); - for (const value of values) { - const key = String(value); - entries.set(key, (entries.get(key) ?? 0) + 1); - } - return Object.fromEntries([...entries].sort(([left], [right]) => left.localeCompare(right, undefined, { numeric: true }))); -} - -function atomIdForPacket(packet: ReviewPacket): string { - return sha256Hex(`hunk-first\n${packet.hunks.map((hunk) => hunk.hunkId).join("\n")}`); -} - -function changedLines(packet: ReviewPacket): number { - return packet.hunks.reduce((total, hunk) => total + hunk.changedNewLineNumbers.length + hunk.changedOldLineNumbers.length, 0); -} - -function budgetDowngraded(actual: ToolBudget, floor: ToolBudget): boolean { - if ( - actual.maxToolCalls < floor.maxToolCalls || - actual.maxInvestigationRounds < floor.maxInvestigationRounds || - actual.maxResultChars < floor.maxResultChars - ) { - return true; - } - const actualExtension = actual.sourceExtension; - const floorExtension = floor.sourceExtension; - return floorExtension !== undefined && ( - actualExtension === undefined || - actualExtension.maxToolCalls < floorExtension.maxToolCalls || - actualExtension.maxResultChars < floorExtension.maxResultChars - ); -} - -function packingEvents( - events: Array | TelemetryEvent>, - failures: ReportFailure[] -): Map { - const byPacket = new Map(); - for (const event of events) { - if (event.stage !== 6 || event.message !== "same_file_atoms_packed") { - continue; - } - const result = sameFilePackingEventDataSchema.safeParse(event.data); - if (!result.success) { - failures.push(failure("missing_or_corrupt_treatment_telemetry", "same_file_atoms_packed telemetry is invalid", { - issues: result.error.issues - })); - continue; - } - const data = result.data as PackingEventData; - if (byPacket.has(data.packetId)) { - failures.push(failure("duplicate_treatment_telemetry", `duplicate same_file_atoms_packed event for ${data.packetId}`)); - continue; - } - byPacket.set(data.packetId, data); - } - return byPacket; -} - -function hunkPacketMap(packets: ReviewPacket[], failures: ReportFailure[], side: string): Map { - const result = new Map(); - for (const packet of packets) { - for (const hunk of packet.hunks) { - if (result.has(hunk.hunkId)) { - failures.push(failure("hunk_bijection", `${side} duplicates hunk ${hunk.hunkId}`, { packetId: packet.id })); - } - result.set(hunk.hunkId, packet); - } - } - return result; -} - -function contextTruncated(packet: ReviewPacket): boolean { - return packet.contextDegradationReasons?.some((reason) => reason.includes("context truncated")) === true || - packet.contextText.includes("[... content truncated to fit packet context budget ...]"); -} - -function packetSourcePositions(diff: UnifiedDiff): Map { - const positions = new Map(); - let position = 0; - for (const file of diff.files) { - for (const hunk of file.hunks) { - positions.set(hunk.id, position); - position += 1; - } - } - return positions; -} - -function actualPatchCharsForHunk(diff: UnifiedDiff): Map { - const values = new Map(); - for (const file of diff.files) { - for (const hunk of file.hunks) { - values.set(hunk.id, renderDiffLines(hunk.lines).length); - } - } - return values; -} - -function renderDiffLines(lines: DiffLine[]): string { - return lines.map((line) => { - const oldLine = line.oldLineNumber === undefined ? " " : String(line.oldLineNumber); - const newLine = line.newLineNumber === undefined ? " " : String(line.newLineNumber); - const prefix = line.kind === "add" ? "+" : line.kind === "delete" ? "-" : " "; - return `${oldLine.padStart(4)} ${newLine.padStart(4)} ${prefix}${line.content}`; - }).join("\n"); -} - -function legacyDiffParityView(diff: UnifiedDiff): unknown { - return { - files: diff.files.map((file) => ({ - ...file, - hunks: file.hunks.map(({ hunkHash: _hunkHash, id: _id, ...hunk }) => hunk) - })) - }; -} - -export function validateRecordedDiffParity(runId: string, recorded: UnifiedDiff, rebuilt: UnifiedDiff): void { - const recordedHunks = recorded.files.flatMap((file) => file.hunks); - const missingHunkHashes = recordedHunks.filter((hunk) => hunk.hunkHash === undefined).length; - assertReport(missingHunkHashes === 0 || missingHunkHashes === recordedHunks.length, "recorded_diff_mismatch", `recorded diff partially persists hunk hashes for ${runId}`); - if (missingHunkHashes > 0) { - assertReport(LEGACY_DIFF_SCHEMA_RUN_IDS.has(runId), "unpermitted_legacy_diff_schema", `run ${runId} is not allowlisted for the legacy hunk identity schema`); - } - const rebuiltParity = missingHunkHashes === 0 ? rebuilt : legacyDiffParityView(rebuilt); - const recordedParity = missingHunkHashes === 0 ? recorded : legacyDiffParityView(recorded); - assertReport(stableJson(rebuiltParity) === stableJson(recordedParity), "recorded_diff_mismatch", `rebuilt diff content differs for ${runId}`, { - recorded: valueFingerprint(recorded), - actual: valueFingerprint(rebuilt) - }); -} - -function normalizedRequestedLenses(plan: ReviewPlan, hunkIds: string[]): string { - const ids = new Set(hunkIds); - return JSON.stringify(sortedUnique(plan.coverage.filter((decision) => ids.has(decision.hunkId)).flatMap((decision) => decision.lenses ?? []))); -} - -function proxyPackedPacketCount( - offPackets: ReviewPacket[], - events: Map, - plan: ReviewPlan, - diff: UnifiedDiff -): { proxy: number; actualMeasurement: number; changedSplits: number } { - const referencedAtomIds = new Set([...events.values()].flatMap((event) => event.atomIds)); - const eligible = offPackets.filter((packet) => referencedAtomIds.has(atomIdForPacket(packet))); - const ineligible = offPackets.length - eligible.length; - const realPatchChars = actualPatchCharsForHunk(diff); - type Group = { hunks: number; proxyChars: number; realChars: number }; - const proxyGroups = new Map(); - const realGroups = new Map(); - for (const packet of eligible) { - const key = `${packet.path}\0${packet.coverage}\0${normalizedRequestedLenses(plan, packet.hunks.map((hunk) => hunk.hunkId))}`; - const proxyChars = sum(packet.hunks.map((hunk) => hunk.contentWithLineNumbers.length)); - const realChars = sum(packet.hunks.map((hunk) => realPatchChars.get(hunk.hunkId) ?? Number.MAX_SAFE_INTEGER)); - appendGreedy(proxyGroups, key, packet.hunks.length, proxyChars, realChars, "proxyChars"); - appendGreedy(realGroups, key, packet.hunks.length, proxyChars, realChars, "realChars"); - } - const proxy = ineligible + sum([...proxyGroups.values()].map((groups) => groups.length)); - const actualMeasurement = ineligible + sum([...realGroups.values()].map((groups) => groups.length)); - return { proxy, actualMeasurement, changedSplits: Math.abs(proxy - actualMeasurement) }; -} - -function appendGreedy( - groupsByKey: Map>, - key: string, - hunks: number, - proxyChars: number, - realChars: number, - measurement: "proxyChars" | "realChars" -): void { - const groups = groupsByKey.get(key) ?? []; - const current = groups.at(-1); - if (current !== undefined && current.hunks + hunks <= MAX_HUNKS_PER_PACKET && current[measurement] + (measurement === "proxyChars" ? proxyChars : realChars) <= MAX_PATCH_CHARS) { - current.hunks += hunks; - current.proxyChars += proxyChars; - current.realChars += realChars; - } else { - groups.push({ hunks, proxyChars, realChars }); - } - groupsByKey.set(key, groups); -} - -export function analyzeReplayComparison(input: ReplayAnalysisInput): ReplayRow { - const failures: ReportFailure[] = []; - const modelCallsObserved = input.modelCallsObserved ?? 0; - if (modelCallsObserved !== 0) { - failures.push(failure("replay_model_call", "deterministic replay recorded model calls", { modelCallsObserved })); - } - if (input.expectedRefs !== undefined && input.actualRefs !== undefined) { - if (input.expectedRefs.base !== input.actualRefs.base || input.expectedRefs.head !== input.actualRefs.head) { - failures.push(failure("stale_replay_refs", "recorded replay refs do not match resolved refs", { - expected: input.expectedRefs, - actual: input.actualRefs - })); - } - } - - const recordedById = new Map(input.recordedPackets.map((packet) => [packet.id, packet])); - const parityOffById = new Map((input.flagOffParityPackets ?? input.offPackets).map((packet) => [packet.id, packet])); - const flagOffParityDifferences = { - missingPacketIds: [...recordedById.keys()].filter((id) => !parityOffById.has(id)).sort(), - extraPacketIds: [...parityOffById.keys()].filter((id) => !recordedById.has(id)).sort(), - changedPackets: [...recordedById].flatMap(([id, packet]) => { - const current = parityOffById.get(id); - return current === undefined || stableJson(packet) === stableJson(current) - ? [] - : [{ - packetId: id, - paths: changedValuePaths(packet, current).slice(0, 50), - samples: changedValueSamples(packet, current).slice(0, 10) - }]; - }), - historicalMigrations: input.flagOffParityMigrations ?? [] - }; - if (recordedById.size !== parityOffById.size || [...recordedById].some(([id, packet]) => stableJson(packet) !== stableJson(parityOffById.get(id)))) { - failures.push(failure("flag_off_parity", "packing-off packets are not artifact-identical to the recorded Stage-6 packets", flagOffParityDifferences)); - } - - const offById = new Map(input.offPackets.map((packet) => [packet.id, packet])); - const offByHunk = hunkPacketMap(input.offPackets, failures, "packing-off"); - const onByHunk = hunkPacketMap(input.onPackets, failures, "packing-on"); - const expectedHunks = [...offByHunk.keys()].sort(); - const actualHunks = [...onByHunk.keys()].sort(); - if (stableJson(expectedHunks) !== stableJson(actualHunks)) { - failures.push(failure("hunk_bijection", "packing-on hunk set differs from packing-off", { - missing: expectedHunks.filter((id) => !onByHunk.has(id)), - extra: actualHunks.filter((id) => !offByHunk.has(id)) - })); - } - - const positions = packetSourcePositions(input.diff); - const patchCharsByHunk = actualPatchCharsForHunk(input.diff); - const factsByPath = new Map(input.fileFacts.map((facts) => [facts.path, facts])); - const eventByPacket = packingEvents(input.onEvents, failures); - const offAtomById = new Map(input.offPackets.map((packet) => [atomIdForPacket(packet), packet])); - const observedAtomIds: string[] = []; - let atomInvariantFailures = 0; - let capViolations = 0; - let newCoveragePromotions = 0; - let lensDrops = 0; - let highPriorityNoteOmissions = 0; - let deepContextDowngrades = 0; - let relatedContextOmissions = 0; - let derivedProfileDowngrades = 0; - let profileFloorApplications = 0; - let effectiveProfileDowngrades = 0; - let effectiveBudgetDowngrades = 0; - let invalidDispatchRanks = 0; - let previousPacketSourcePosition = -1; - const eligiblePackingPatchChars: number[] = []; - - for (const packet of input.offPackets) { - const facts = factsByPath.get(packet.path); - if (facts === undefined) { - invalidDispatchRanks += 1; - failures.push(failure("missing_file_facts", `missing file facts for packing-off packet ${packet.path}`)); - continue; - } - const expectedRank = packetDispatchRank(packet.path, facts, changedLines(packet)); - if (stableJson(packet.dispatchRank) !== stableJson(expectedRank)) { - invalidDispatchRanks += 1; - failures.push(failure("invalid_dispatch_rank", `packing-off packet ${packet.id} has invalid dispatch rank`, { - expected: expectedRank, - actual: packet.dispatchRank - })); - } - } - - for (const packet of input.onPackets) { - const sourcePositions = packet.hunks.map((hunk) => positions.get(hunk.hunkId) ?? Number.MAX_SAFE_INTEGER); - if (sourcePositions.some((position, index) => index > 0 && position < (sourcePositions[index - 1] ?? -1))) { - atomInvariantFailures += 1; - failures.push(failure("source_order", `packet ${packet.id} reorders hunks`)); - } - const firstSourcePosition = Math.min(...sourcePositions); - if (firstSourcePosition < previousPacketSourcePosition) { - atomInvariantFailures += 1; - failures.push(failure("source_packet_order", `packet ${packet.id} appears before an earlier source packet`, { - previousPacketSourcePosition, - firstSourcePosition - })); - } - previousPacketSourcePosition = firstSourcePosition; - const facts = factsByPath.get(packet.path); - if (facts === undefined) { - invalidDispatchRanks += 1; - failures.push(failure("missing_file_facts", `missing file facts for ${packet.path}`)); - } else { - const expectedRank = packetDispatchRank(packet.path, facts, changedLines(packet)); - if (stableJson(packet.dispatchRank) !== stableJson(expectedRank)) { - invalidDispatchRanks += 1; - failures.push(failure("invalid_dispatch_rank", `packet ${packet.id} has invalid dispatch rank`, { - expected: expectedRank, - actual: packet.dispatchRank - })); - } - } - - const event = eventByPacket.get(packet.id); - const atoms = event?.atomIds.map((id) => offAtomById.get(id)).filter((atom): atom is ReviewPacket => atom !== undefined) ?? []; - if (event !== undefined) { - observedAtomIds.push(...event.atomIds); - const actualPatchChars = sum(packet.hunks.map((hunk) => patchCharsByHunk.get(hunk.hunkId) ?? Number.MAX_SAFE_INTEGER)); - const combinesMultipleAtoms = event.sourceAtomCount > 1; - if (combinesMultipleAtoms) { - eligiblePackingPatchChars.push(actualPatchChars); - } - if (event.atomIds.length !== event.sourceAtomCount || event.standaloneProfiles.length !== event.sourceAtomCount || atoms.length !== event.sourceAtomCount) { - atomInvariantFailures += 1; - failures.push(failure("atom_bijection", `packet ${packet.id} has missing or inconsistent atom provenance`, { - atomIds: event.atomIds, - sourceAtomCount: event.sourceAtomCount, - resolvedAtoms: atoms.length - })); - } - const flattened = atoms.flatMap((atom) => atom.hunks.map((hunk) => hunk.hunkId)); - if (stableJson(flattened) !== stableJson(packet.hunks.map((hunk) => hunk.hunkId))) { - atomInvariantFailures += 1; - failures.push(failure("atom_split_or_reorder", `packet ${packet.id} does not preserve source atom membership/order`)); - } - if ( - event.hunkCount !== packet.hunks.length || - event.capUsage.hunks !== packet.hunks.length || - event.capUsage.maxHunks !== MAX_HUNKS_PER_PACKET || - event.capUsage.maxPatchChars !== MAX_PATCH_CHARS || - event.capUsage.hunks > MAX_HUNKS_PER_PACKET || - event.capUsage.patchChars > MAX_PATCH_CHARS || - (combinesMultipleAtoms && (event.capUsage.patchChars !== actualPatchChars || actualPatchChars > MAX_PATCH_CHARS)) - ) { - capViolations += 1; - failures.push(failure("packet_cap", `packet ${packet.id} violates packing caps`, { - capUsage: event.capUsage, - actualPatchChars - })); - } - if (atoms.some((atom) => atom.path !== packet.path || atom.language !== packet.language)) { - atomInvariantFailures += 1; - failures.push(failure("file_language_boundary", `packet ${packet.id} crosses a file or language boundary`)); - } - if (atoms.some((atom) => atom.coverage !== packet.coverage) || event.effectiveCoverage !== packet.coverage) { - newCoveragePromotions += 1; - failures.push(failure("coverage_promotion", `packet ${packet.id} promotes or mixes atom coverage`)); - } - const missingLenses = sortedUnique(atoms.flatMap((atom) => atom.lenses)).filter((lens) => !packet.lenses.includes(lens)); - if (!event.plannerLensesPreserved || missingLenses.length > 0) { - lensDrops += missingLenses.length > 0 ? missingLenses.length : 1; - failures.push(failure("planner_lens_drop", `packet ${packet.id} drops a standalone routed lens`, { missingLenses })); - } - const floor = event.standaloneProfiles.reduce((highest, profile) => PROFILE_RANK[profile] > PROFILE_RANK[highest] ? profile : highest, "simple" as ReviewProfile); - if (event.profileFloor !== floor) { - failures.push(failure("profile_floor_mismatch", `packet ${packet.id} reports the wrong profile floor`, { expected: floor, actual: event.profileFloor })); - } - if (PROFILE_RANK[event.derivedPackedProfile] < PROFILE_RANK[floor]) { - derivedProfileDowngrades += 1; - } - if (event.profileFloorApplied) { - profileFloorApplications += 1; - } - if (PROFILE_RANK[event.effectiveProfile] < PROFILE_RANK[floor] || packet.reviewProfile !== event.effectiveProfile) { - effectiveProfileDowngrades += 1; - failures.push(failure("effective_profile_downgrade", `packet ${packet.id} falls below its standalone profile floor`, { - floor, - effective: event.effectiveProfile - })); - } - if (atoms.some((atom) => budgetDowngraded(event.effectiveToolBudget, atom.toolBudget)) || stableJson(event.effectiveToolBudget) !== stableJson(packet.toolBudget)) { - effectiveBudgetDowngrades += 1; - failures.push(failure("effective_budget_downgrade", `packet ${packet.id} has a downgraded or inconsistent effective tool budget`)); - } - } else { - observedAtomIds.push(atomIdForPacket(packet)); - const off = offById.get(packet.id); - if (off === undefined || stableJson(off) !== stableJson(packet)) { - atomInvariantFailures += 1; - failures.push(failure("missing_treatment_telemetry", `packing-on packet ${packet.id} has no provenance and is not an unchanged bypass packet`)); - } - } - - for (const hunk of packet.hunks) { - const off = offByHunk.get(hunk.hunkId); - if (off === undefined) { - continue; - } - const omittedLenses = off.lenses.filter((lens) => !packet.lenses.includes(lens)); - lensDrops += omittedLenses.length; - const omittedNotes = off.attentionNotes.filter((note) => !packet.attentionNotes.includes(note)); - if ((off.reviewPriority === "critical" || off.reviewPriority === "high") && omittedNotes.length > 0) { - highPriorityNoteOmissions += omittedNotes.length; - failures.push(failure("high_priority_focus_omitted", `packing omits a high/critical attention note for hunk ${hunk.hunkId}`, { - omittedNoteCount: omittedNotes.length, - omittedNoteFingerprints: omittedNotes.map(valueFingerprint) - })); - } - if (off.coverage === "deep" && contextQualityRank(packet.contextQuality) < contextQualityRank(off.contextQuality)) { - deepContextDowngrades += 1; - failures.push(failure("deep_context_downgrade", `deep hunk ${hunk.hunkId} loses context quality`, { - from: off.contextQuality, - to: packet.contextQuality - })); - } - const offRelated = off.relatedChangedContext.map(relatedContextKey); - const onRelated = new Set(packet.relatedChangedContext.map(relatedContextKey)); - relatedContextOmissions += offRelated.filter((key) => !onRelated.has(key)).length; - } - } - - const expectedAtomIds = [...offAtomById.keys()].sort(); - const actualAtomIds = observedAtomIds.sort(); - if (stableJson(expectedAtomIds) !== stableJson(actualAtomIds)) { - atomInvariantFailures += 1; - failures.push(failure("atom_bijection", "packing-on atom set differs from packing-off atoms", { - missing: expectedAtomIds.filter((id) => !actualAtomIds.includes(id)), - extra: actualAtomIds.filter((id) => !expectedAtomIds.includes(id)) - })); - } - - const offOrder = schedulingOrder(input.offPackets); - const onOrder = schedulingOrder(input.onPackets); - const offPositionByHunk = new Map(offOrder.flatMap((packet, index) => packet.hunks.map((hunk) => [hunk.hunkId, index] as const))); - const onPositionByHunk = new Map(onOrder.flatMap((packet, index) => packet.hunks.map((hunk) => [hunk.hunkId, index] as const))); - const schedulingMoves = expectedHunks.filter((hunkId) => offPositionByHunk.get(hunkId) !== onPositionByHunk.get(hunkId)).length; - const proxy = proxyPackedPacketCount(input.offPackets, eventByPacket, input.plan, input.diff); - const motivating = input.runId.includes("dca8d870"); - let countReconciliation: ReplayRow["countReconciliation"] = "not-motivating-run"; - if (input.onPackets.length > input.offPackets.length) { - failures.push(failure("packet_count_increase", `${input.runId} gains packets with packing enabled`)); - } - if (motivating) { - if (input.offPackets.length !== 96) { - failures.push(failure("motivating_off_count", `motivating run must rebuild 96 packing-off packets, got ${input.offPackets.length}`)); - } - if (input.onPackets.length === 75) { - countReconciliation = "historical-target"; - } else if (input.onPackets.length === 74 || input.onPackets.length === 76) { - countReconciliation = input.onPackets.length === 74 ? "real-patch-74" : "real-patch-76"; - if (proxy.proxy !== 75 || proxy.actualMeasurement !== input.onPackets.length || proxy.changedSplits !== 1) { - failures.push(failure("unexplained_packet_count", "74/76 motivating count is not attributable solely to proxy-versus-real patch measurement", { proxy })); - } - } else { - failures.push(failure("unpermitted_packet_count", `motivating run packed count must be 74, 75, or 76; got ${input.onPackets.length}`)); - } - if ((1 - input.onPackets.length / Math.max(1, input.offPackets.length)) < 0.2) { - failures.push(failure("insufficient_packet_reduction", "motivating run packet reduction is below 20%")); - } - } - - const attentionNoteOmissions = sum(expectedHunks.map((hunkId) => { - const off = offByHunk.get(hunkId); - const on = onByHunk.get(hunkId); - return off === undefined || on === undefined ? 0 : off.attentionNotes.filter((note) => !on.attentionNotes.includes(note)).length; - })); - const contextComparison = { - contextTruncationsOff: input.offPackets.filter(contextTruncated).length, - contextTruncationsOn: input.onPackets.filter(contextTruncated).length, - relatedContextOmissions, - attentionNoteOmissions, - lensDrops - }; - const packingEventValues = [...eventByPacket.values()]; - const distribution = { - offHunksPerPacket: countDistribution(input.offPackets.map((packet) => packet.hunks.length)), - onHunksPerPacket: countDistribution(input.onPackets.map((packet) => packet.hunks.length)), - sourceAtomsPerOnPacket: countDistribution(input.onPackets.map((packet) => eventByPacket.get(packet.id)?.sourceAtomCount ?? 1)), - onCoverage: countDistribution(input.onPackets.map((packet) => packet.coverage)), - onProfiles: countDistribution(input.onPackets.map((packet) => packet.reviewProfile)), - onBudgetModes: countDistribution(input.onPackets.map((packet) => eventByPacket.get(packet.id)?.toolBudgetMode ?? "bypass-base")), - maxOnHunks: Math.max(0, ...input.onPackets.map((packet) => packet.hunks.length)), - eligiblePackingPackets: packingEventValues.length, - bypassPackets: input.onPackets.length - packingEventValues.length, - maxEligiblePackingPatchChars: Math.max(0, ...eligiblePackingPatchChars) - }; - - return { - runId: input.runId, - offPackets: input.offPackets.length, - onPackets: input.onPackets.length, - reductionPercent: round((1 - input.onPackets.length / Math.max(1, input.offPackets.length)) * 100, 3), - sourceAtoms: input.offPackets.length, - packedMultiAtomPackets: [...eventByPacket.values()].filter((event) => event.sourceAtomCount > 1).length, - reviewableHunks: expectedHunks.length, - newCoveragePromotions, - hunkBijectionFailures: failures.filter((entry) => entry.code === "hunk_bijection").length, - atomInvariantFailures, - capViolations, - lensDrops, - highPriorityNoteOmissions, - deepContextDowngrades, - relatedContextOmissions, - derivedProfileDowngrades, - profileFloorApplications, - effectiveProfileDowngrades, - effectiveBudgetDowngrades, - invalidDispatchRanks, - modelCallsObserved, - schedulingMoves, - proxyPackedPackets: proxy.proxy, - countReconciliation, - flagOffParityDifferences, - contextComparison, - distribution, - packetMembership: input.onPackets.map((packet) => ({ - packetId: packet.id, - path: packet.path, - hunkIds: packet.hunks.map((hunk) => hunk.hunkId), - coverage: packet.coverage, - lenses: [...packet.lenses], - profile: packet.reviewProfile, - budget: packet.toolBudget, - dispatchRank: packet.dispatchRank, - atomIds: eventByPacket.get(packet.id)?.atomIds ?? [atomIdForPacket(packet)] - })), - failures - }; -} - -function changedValuePaths(before: unknown, after: unknown, prefix = ""): string[] { - if (Object.is(before, after)) { - return []; - } - if (Array.isArray(before) && Array.isArray(after)) { - if (before.length !== after.length) { - return [`${prefix}.length`]; - } - return before.flatMap((value, index) => changedValuePaths(value, after[index], `${prefix}[${index}]`)); - } - if (isRecord(before) && isRecord(after)) { - return sortedUnique([...Object.keys(before), ...Object.keys(after)]).flatMap((key) => - changedValuePaths(before[key], after[key], prefix.length === 0 ? key : `${prefix}.${key}`) - ); - } - return [prefix.length === 0 ? "" : prefix]; -} - -function changedValueSamples( - before: unknown, - after: unknown, - prefix = "" -): Array<{ path: string; recorded: ValueFingerprint; rebuilt: ValueFingerprint }> { - if (Object.is(before, after)) { - return []; - } - if (Array.isArray(before) && Array.isArray(after)) { - if (before.length !== after.length) { - return [{ path: `${prefix}.length`, recorded: valueFingerprint(before.length), rebuilt: valueFingerprint(after.length) }]; - } - return before.flatMap((value, index) => changedValueSamples(value, after[index], `${prefix}[${index}]`)); - } - if (isRecord(before) && isRecord(after)) { - return sortedUnique([...Object.keys(before), ...Object.keys(after)]).flatMap((key) => - changedValueSamples(before[key], after[key], prefix.length === 0 ? key : `${prefix}.${key}`) - ); - } - return [{ - path: prefix.length === 0 ? "" : prefix, - recorded: valueFingerprint(before), - rebuilt: valueFingerprint(after) - }]; -} - -function valueFingerprint(value: unknown): ValueFingerprint { - const serialized = typeof value === "string" ? value : JSON.stringify(stableValue(value)) ?? String(value); - return { - kind: typeof value === "string" ? "string" : "json", - length: serialized.length, - sha256: sha256Hex(serialized) - }; -} - -export function historicalFlagOffParityView( - runId: string, - recordedPackets: ReviewPacket[], - rebuiltPackets: ReviewPacket[] -): { - packets: ReviewPacket[]; - migrations: Array<{ code: string; packets: number; explanation: string }>; -} { - const legacyDispatchRuns = new Set([ - "20260724-135818-740d73f2", - "20260724-150405-fe1548ae", - "20260724-162739-81f806a6" - ]); - const packets = structuredClone(rebuiltPackets); - const migrations: Array<{ code: string; packets: number; explanation: string }> = []; - const recordedById = new Map(recordedPackets.map((packet) => [packet.id, packet])); - - if ( - legacyDispatchRuns.has(runId) && - recordedPackets.length > 0 && - recordedPackets.every((packet) => !Object.prototype.hasOwnProperty.call(packet, "dispatchRank")) - ) { - for (const packet of packets) { - delete (packet as Partial).dispatchRank; - } - migrations.push({ - code: "pre_plan100_dispatch_rank_schema", - packets: packets.length, - explanation: "recorded artifacts predate dispatchRank; parity omits only that absent field after validating the rebuilt current rank formula" - }); - } - - if (runId === "20260724-135818-740d73f2") { - let migratedPackets = 0; - for (const packet of packets) { - const recorded = recordedById.get(packet.id); - if (recorded === undefined || stableJson(recorded) === stableJson(packet)) { - continue; - } - const onlyKnownPaths = changedValuePaths(recorded, packet).every((entry) => entry === "contextText" || entry === "relevantTests.length"); - const selfTest = packet.relevantTests.length === 1 ? packet.relevantTests[0] : undefined; - const expectedContext = selfTest === undefined - ? undefined - : `${recorded.contextText}\nLikely tests: ${selfTest.path}:${selfTest.name}`; - if ( - onlyKnownPaths && - recorded.relevantTests.length === 0 && - selfTest?.path === packet.path && - selfTest.name === packet.path && - packet.contextText === expectedContext - ) { - packet.relevantTests = []; - packet.contextText = recorded.contextText; - migratedPackets += 1; - } - } - if (migratedPackets > 0) { - migrations.push({ - code: "pre_self_test_context_filter", - packets: migratedPackets, - explanation: "the oldest clean artifact predates the self-test context filter; parity removes only the exact redundant path:path test line" - }); - } - } - - return { packets, migrations }; -} - -function relatedContextKey(context: ReviewPacket["relatedChangedContext"][number]): string { - return stableJson({ - path: context.path, - hunkId: context.hunkId, - relatedHunkIds: context.relatedHunkIds, - symbol: context.symbol, - source: context.relationshipSource, - strength: context.relationshipStrength - }); -} - -function contextQualityRank(quality: PacketContextQuality | undefined): number { - return quality === undefined ? CONTEXT_QUALITY_RANK.path_only : CONTEXT_QUALITY_RANK[quality]; -} - -function schedulingOrder(packets: ReviewPacket[]): ReviewPacket[] { - const priorityRank = { critical: 0, high: 1, normal: 2, low: 3 } as const; - const coverageRank = { deep: 0, normal: 1, light: 2 } as const; - return packets.map((packet, index) => ({ packet, index })).sort((a, b) => - priorityRank[a.packet.reviewPriority] - priorityRank[b.packet.reviewPriority] || - coverageRank[a.packet.coverage] - coverageRank[b.packet.coverage] || - a.packet.dispatchRank[0] - b.packet.dispatchRank[0] || - a.packet.dispatchRank[1] - b.packet.dispatchRank[1] || - a.index - b.index - ).map((entry) => entry.packet); -} - -function round(value: number, digits = 6): number { - const factor = 10 ** digits; - return Math.round(value * factor) / factor; -} - -function createCaptureTelemetry(runId: string): { telemetry: TelemetryRecorder; captured: CapturedTelemetry } { - const captured: CapturedTelemetry = { events: [], modelCalls: [], toolCalls: [] }; - let toolCall = 0; - return { - captured, - telemetry: { - runId, - runDir: undefined, - event: (event) => captured.events.push(structuredClone(event)), - recordModelCall: (record) => captured.modelCalls.push(structuredClone(record)), - recordToolCall: (record) => { - captured.toolCalls.push(structuredClone(record)); - toolCall += 1; - return `replay-tool-${String(toolCall).padStart(6, "0")}`; - }, - writeArtifact: async () => undefined, - writeDebug: async () => undefined, - flush: async () => undefined - } - }; -} - -type ExperimentArm = "A" | "B" | "C"; - -type ExecutionPressure = { - reviewedPackets: number; - reviewedAtoms: number; - requestedToolCalls: number; - usedToolCalls: number; - rejectedToolCalls: number; - rejectionRatePerAtom: number; - rejectionReasons: Record; - resultChars: number; - continuations: number; - modelServiceSeconds: number; - inputTokens: number; - outputTokens: number; - stage7CostUSD: number; - costPerReviewedAtomUSD: number | null; -}; - -type TreatmentExecution = { - repeat: number; - treated: boolean; - atomIds: string[]; - hunkIds: string[]; - targetPacketId?: string; - targetPath?: string; - targetFilePackets?: number; - targetAtomCount?: number; - targetCoverage?: string; - targetRequestedLensSignature?: string; - pressure: ExecutionPressure; - failures: ReportFailure[]; -}; - -type ExpectationRate = { - expectationId: string; - list: EvalExpectationList; - candidateHits: number; - finalHits: number; - denominator: number; - candidateRate: number; - finalRate: number; - lossHistogram: Record; - lossByAtomCount: Record; -}; - -type EvalArmReport = { - runNumber: number; - treatment: { treated: number; total: number; minimum: number; valid: boolean }; - intentToTreat: ExpectationRate[]; - treatedOnly: { label: "secondary-treated-only"; denominatorByExpectation: Record; rates: ExpectationRate[] }; - pressure: ExecutionPressure; - actualCostUSD: number; -}; - -type EvalCaseReport = { - case: string; - selectedArm?: "B" | "C"; - arms: Record; -}; - -export type EvalReport = { - schemaVersion: 1; - mode: "eval"; - cohort: { id: string; runNumbers: number[]; actualCostUSD: number }; - evidence: "repeated-packing-sensitive" | "production-capacity"; - expectedRepeats: number; - selectedArm?: "B" | "C"; - cases?: EvalCaseReport[]; - productionEconomics?: ProductionEconomics; - productionThroughput?: ProductionThroughput; - failures: ReportFailure[]; -}; - -export type ProductionThroughput = { - baseline: ProductionArmThroughput; - selected: ProductionArmThroughput; -}; - -type ProductionArmThroughput = { - reviewedHunkIds: string[]; - reviewedHunks: number; - reviewedAtoms: number; - reviewedPackets: number; - wallTimeSeconds: number; - reviewedHunksPerWallSecond: number; - modelServiceSeconds: number; - modelServiceSecondsPerReviewedHunk: number; - totalTokens: number; - tokensPerReviewedHunk: number; - reasoningTokens: number; - reasoningTokensPerReviewedHunk: number; - continuations: number; - continuationsPerReviewedAtom: number; - pressure: ExecutionPressure; -}; - -export type ProductionEconomics = { - equivalentTargetHunks: 142; - baseline: ProductionArmEconomics; - selected: ProductionArmEconomics; - equivalentReviewSavingsUSD: number; - validationCostInputUSD: number; - validationCostInputLabel: "cohort_actual_cost_minimum" | "explicit_cumulative_validation_cost"; - breakEvenReviewCount: number; -}; - -type ProductionArmEconomics = { - actualCostUSD: number; - reviewedHunks: number; - costPerReviewedHunkUSD: number; - equivalentReviewCostUSD: number; - equivalentCostExtrapolated: boolean; -}; - -export type RegressionReport = { - schemaVersion: 1; - mode: "regression"; - evidence: "one-repeat-collateral-only"; - baselineCohort: { id: string; runNumbers: number[]; actualCostUSD: number }; - selectedCohort: { id: string; runNumbers: number[]; actualCostUSD: number }; - expectedRepeats: number; - cases: Array<{ - caseName: string; - baselineRun: number; - selectedRun: number; - expectationTransitions: Array<{ expectationId: string; list: string; from: string; to: string }>; - selectedTreatmentExecutions: number; - dispatchOrderChanges: number; - baselinePressure: ExecutionPressure; - selectedPressure: ExecutionPressure; - }>; - failures: ReportFailure[]; -}; - -function armForRun(run: EvalCaseRunInput): ExperimentArm { - const effective = run.info.effectiveConfig?.review; - assertReport(effective !== undefined, "missing_effective_config", `run ${run.runNumber} has no effective packet-packing config`); - assertReport( - typeof effective.packSameFileHunks === "boolean" && - (effective.packedToolBudgetMode === "base" || effective.packedToolBudgetMode === "atom-scaled"), - "invalid_arm_config", - `run ${run.runNumber} has invalid effective packet-packing settings` - ); - if (!effective.packSameFileHunks) { - assertReport(effective.packedToolBudgetMode === "base", "invalid_arm_config", `baseline run ${run.runNumber} must use base tool budget`); - return "A"; - } - return effective.packedToolBudgetMode === "base" ? "B" : "C"; -} - -function caseFamily(run: EvalCaseRunInput): string { - const source = run.info.caseFile ?? run.info.caseName; - return source - .replace(/\.ya?ml$/u, "") - .replace(/-(?:a|b|c|baseline|selected)$/iu, ""); -} - -export function selectExplicitCohort(runs: EvalCaseRunInput[], selector: string): CohortSelection { - assertReport(runs.length > 0, "empty_cohort", "no eval run info files were found"); - const byInvocation = new Map(); - for (const run of runs.filter((candidate) => candidate.info.invocation !== undefined && candidate.invocationManifest !== undefined)) { - const id = run.info.invocation?.id; - assertReport(id !== undefined, "missing_invocation_manifest", `run ${run.runNumber} lacks invocation identity`); - byInvocation.set(id, [...(byInvocation.get(id) ?? []), run]); - } - for (const [id, invocation] of byInvocation) { - byInvocation.set(id, [...invocation].sort((left, right) => - (left.info.invocation?.caseIndex ?? Number.MAX_SAFE_INTEGER) - (right.info.invocation?.caseIndex ?? Number.MAX_SAFE_INTEGER) - )); - } - assertReport(byInvocation.size > 0, "missing_invocation_manifest", "no cohort-eligible eval invocation manifests were found"); - const invocations = [...byInvocation.values()].sort((left, right) => - (left[0]?.invocationManifest?.startedAt ?? "").localeCompare(right[0]?.invocationManifest?.startedAt ?? "") || - Math.max(...left.map((run) => run.runNumber)) - Math.max(...right.map((run) => run.runNumber)) || - (left[0]?.info.invocation?.id ?? "").localeCompare(right[0]?.info.invocation?.id ?? "") - ); - let selected: EvalCaseRunInput[] | undefined; - if (byInvocation.has(selector)) { - selected = byInvocation.get(selector); - } else if (selector === "latest") { - selected = invocations.at(-1); - } else if (/^\d+$/u.test(selector)) { - const endingRun = Number(selector); - const matches = invocations.filter((invocation) => invocation.at(-1)?.runNumber === endingRun); - assertReport(matches.length <= 1, "ambiguous_cohort_selector", `run ${selector} matches multiple persisted invocations; use the invocation UUID`); - selected = matches[0]; - assertReport(selected !== undefined, "unknown_cohort", `run ${selector} is not an exact persisted invocation boundary`); - } else { - const match = /^(\d+)-(\d+)$/u.exec(selector); - assertReport(match !== null, "invalid_cohort_selector", `cohort must be latest, an invocation UUID, an ending run number, or a min-max range: ${selector}`); - const start = Number(match[1]); - const end = Number(match[2]); - const matches = invocations.filter((invocation) => invocation[0]?.runNumber === start && invocation.at(-1)?.runNumber === end); - assertReport(matches.length <= 1, "ambiguous_cohort_selector", `cohort ${selector} matches multiple persisted invocations; use the invocation UUID`); - selected = matches[0]; - assertReport(selected !== undefined, "unknown_cohort", `cohort ${selector} is not an exact persisted invocation boundary`); - } - assertReport(selected !== undefined && selected.length > 0, "unknown_cohort", `cohort ${selector} is empty`); - validateCompleteInvocation(selected); - assertNoMixedRuns(selected); - return { id: selected[0]?.info.invocation?.id ?? "unknown", runs: selected }; -} - -function validateCompleteInvocation(runs: EvalCaseRunInput[]): void { - const manifest = runs[0]?.invocationManifest; - assertReport(manifest !== undefined, "missing_invocation_manifest", "cohort invocation manifest is missing"); - const expectedIndexes = Array.from({ length: manifest.cases.length }, (_, index) => index); - const caseIndexes = manifest.cases.map((entry) => entry.caseIndex); - const runIndexes = manifest.runs.map((entry) => entry.caseIndex); - const loadedIndexes = runs.map((run) => run.info.invocation?.caseIndex).filter((index): index is number => index !== undefined); - const loadedRunDirs = runs.map((run) => path.resolve(run.runDir)); - assertReport( - manifest.status === "complete" && manifest.completedAt !== undefined && - stableJson(caseIndexes) === stableJson(expectedIndexes) && - stableJson(runIndexes) === stableJson(expectedIndexes) && - stableJson(loadedIndexes) === stableJson(expectedIndexes) && - manifest.runs.length === manifest.cases.length && runs.length === manifest.cases.length && - unique(loadedRunDirs).length === loadedRunDirs.length, - "partial_latest_cohort", - "persisted eval invocation is incomplete or its exact ordered case set does not match loaded runs", - { invocationId: manifest.invocationId, expectedCount: manifest.cases.length, recordedCount: manifest.runs.length, loadedCount: runs.length } - ); - for (const [index, run] of runs.entries()) { - validateInvocationManifestRun(manifest, run.info, run.runDir); - assertReport(run.info.invocation?.caseIndex === index, "invocation_manifest_join", `run ${run.runNumber} is out of invocation order`); - } -} - -function assertNoMixedRuns(runs: EvalCaseRunInput[]): void { - const caseNames = runs.map((run) => run.info.caseName); - assertReport(unique(caseNames).length === caseNames.length, "mixed_cohort", "selected cohort contains multiple generations of the same case", { caseNames }); - const runtimeCommits = sortedUnique(runs.flatMap((run) => run.info.codegenieRuntime?.commit ?? [])); - if (runtimeCommits.length > 1) { - fail("mixed_cohort", "selected cohort mixes Codegenie commits", { runtimeCommits }); - } -} - -export function analyzeEvalCohort( - cohort: CohortSelection, - expectedRepeats: number, - options: { actualValidationCostUSD?: number } = {} -): EvalReport { - const failures: ReportFailure[] = []; - assertReport(Number.isInteger(expectedRepeats) && expectedRepeats > 0, "invalid_expected_repeats", "expected repeats must be a positive integer"); - const arms = cohort.runs.map((run) => ({ run, arm: armForRun(run), family: caseFamily(run) })); - const armSet = new Set(arms.map((entry) => entry.arm)); - const production = cohort.runs.length === 2 && armSet.has("A") && (armSet.has("B") || armSet.has("C")); - const actualCostUSD = aggregateCohortCost(cohort.runs, failures); - - if (production) { - const baseline = arms.find((entry) => entry.arm === "A"); - const selected = arms.find((entry) => entry.arm !== "A"); - assertReport(baseline !== undefined && selected !== undefined, "production_pair", "production comparison requires one baseline and one selected run"); - if (stableJson(normalizeExperimentSnapshot(baseline.run.declaredCase)) !== stableJson(normalizeExperimentSnapshot(selected.run.declaredCase))) { - failures.push(failure("production_case_mismatch", "production baseline and selected cases differ outside packet-packing review settings")); - } - validateRunRepeatCount(baseline.run, expectedRepeats, failures); - validateRunRepeatCount(selected.run, expectedRepeats, failures); - validateRunEvidence(baseline.run, "production/baseline", failures); - validateRunEvidence(selected.run, "production/selected", failures); - const baselineTreatment = baseline.run.executions.map((execution) => analyzeTreatmentExecution(baseline.run, execution, "A")); - const selectedTreatment = selected.run.executions.map((execution) => analyzeTreatmentExecution( - selected.run, - execution, - selected.arm, - baseline.run.executions.find((candidate) => candidate.repeat === execution.repeat) - )); - failures.push(...baselineTreatment.flatMap((entry) => entry.failures), ...selectedTreatment.flatMap((entry) => entry.failures)); - validateProductionProvenance(baseline.run, selected.run, failures); - validateRunScoreExpectations("production/baseline", baseline.run, failures); - validateRunScoreExpectations("production/selected", selected.run, failures); - validateProductionExpectations(baseline.run, selected.run, failures); - validateProductionModelAccounting(baseline.run, baselineTreatment, failures, "baseline"); - validateProductionModelAccounting(selected.run, selectedTreatment, failures, "selected"); - if (selectedTreatment.every((entry) => !entry.treated)) { - failures.push(failure("missing_treatment", "selected production run never packed multiple atoms")); - } - const productionThroughput = analyzeProductionThroughput( - baseline.run, - baselineTreatment, - selected.run, - selectedTreatment, - failures - ); - const economics = computeProductionEconomics( - { - actualCostUSD: costForRun(baseline.run, failures), - reviewedHunks: reviewedHunksForRun(baseline.run) - }, - { - actualCostUSD: costForRun(selected.run, failures), - reviewedHunks: reviewedHunksForRun(selected.run) - }, - options.actualValidationCostUSD ?? actualCostUSD, - options.actualValidationCostUSD === undefined ? "cohort_actual_cost_minimum" : "explicit_cumulative_validation_cost", - failures - ); - if (selected.run.info.score.status === "error" || baseline.run.info.score.status === "error") { - failures.push(failure("eval_error", "production cohort contains an errored eval run")); - } - return { - schemaVersion: 1, - mode: "eval", - cohort: { id: cohort.id, runNumbers: cohort.runs.map((run) => run.runNumber), actualCostUSD }, - evidence: "production-capacity", - expectedRepeats, - productionEconomics: economics, - productionThroughput, - failures - }; - } - - const grouped = new Map>>(); - for (const entry of arms) { - const group = grouped.get(entry.family) ?? {}; - if (group[entry.arm] !== undefined) { - failures.push(failure("mixed_cohort", `cohort has duplicate ${entry.arm} arm for ${entry.family}`)); - } - group[entry.arm] = entry.run; - grouped.set(entry.family, group); - } - - const cases: NonNullable = []; - for (const [family, group] of [...grouped].sort(([a], [b]) => a.localeCompare(b))) { - if (group.A === undefined || group.B === undefined || group.C === undefined) { - failures.push(failure("incomplete_abc_cohort", `case ${family} does not have exactly one A/B/C run`)); - continue; - } - const runByArm = group as Record; - const evidenceScores = {} as Record>; - for (const arm of ["A", "B", "C"] as const) { - validateRunRepeatCount(runByArm[arm], expectedRepeats, failures); - evidenceScores[arm] = new Map(validateRunEvidence(runByArm[arm], `${family}/${arm}`, failures) - .map((score, index) => [runByArm[arm].executions[index]!.repeat, score])); - if (runByArm[arm].info.score.status === "error") { - failures.push(failure("eval_error", `${family}/${arm} is an errored eval run`)); - } - } - assertMatchingCaseDefinitions(runByArm, failures); - validateExpectationJoins(family, runByArm, failures); - const treatmentByArm = {} as Record; - treatmentByArm.A = runByArm.A.executions.map((execution) => analyzeTreatmentExecution(runByArm.A, execution, "A")); - for (const arm of ["B", "C"] as const) { - treatmentByArm[arm] = runByArm[arm].executions.map((execution) => analyzeTreatmentExecution( - runByArm[arm], - execution, - arm, - runByArm.A.executions.find((candidate) => candidate.repeat === execution.repeat) - )); - } - failures.push(...Object.values(treatmentByArm).flatMap((entries) => entries.flatMap((entry) => entry.failures))); - compareTargetTreatment(family, treatmentByArm, failures); - - const minimumTreatment = expectedRepeats >= 10 ? 8 : expectedRepeats; - const armReports = {} as EvalCaseReport["arms"]; - for (const arm of ["A", "B", "C"] as const) { - const executions = runByArm[arm].executions; - const treatedCount = arm === "A" ? executions.length : treatmentByArm[arm].filter((entry) => entry.treated).length; - if (arm !== "A" && treatedCount < minimumTreatment) { - failures.push(failure("insufficient_treatment", `${family}/${arm} treated ${treatedCount}/${executions.length}; minimum is ${minimumTreatment}`)); - } - const atomCounts = new Map(treatmentByArm[arm].map((entry) => [entry.repeat, entry.targetAtomCount ?? 1])); - const intentToTreat = expectationRates(executions, () => true, evidenceScores[arm], atomCounts); - const treatedOnly = expectationRates(executions, (execution) => { - if (arm === "A") { - return true; - } - return treatmentByArm[arm].find((entry) => entry.repeat === execution.repeat)?.treated === true; - }, evidenceScores[arm], atomCounts); - armReports[arm] = { - runNumber: runByArm[arm].runNumber, - treatment: { - treated: treatedCount, - total: executions.length, - minimum: arm === "A" ? executions.length : minimumTreatment, - valid: arm === "A" || treatedCount >= minimumTreatment - }, - intentToTreat, - treatedOnly: { - label: "secondary-treated-only", - denominatorByExpectation: Object.fromEntries(treatedOnly.map((rate) => [`${rate.list}:${rate.expectationId}`, rate.denominator])), - rates: treatedOnly - }, - pressure: combinePressure(treatmentByArm[arm].map((entry) => entry.pressure)), - actualCostUSD: costForRun(runByArm[arm], failures) - }; - } - if (expectedRepeats >= 10) { - enforceRequiredRecallZero(family, armReports, failures); - enforcePressureGate(family, armReports, failures); - } - cases.push({ case: family, arms: armReports }); - } - - if (expectedRepeats >= 10) { - enforceCohortRecallGates(cases, failures); - } - const selectedArm = expectedRepeats >= 10 ? selectCohortPackedArm(cases, expectedRepeats, failures) : undefined; - if (selectedArm !== undefined) { - for (const entry of cases) { - entry.selectedArm = selectedArm; - } - } - - return { - schemaVersion: 1, - mode: "eval", - cohort: { id: cohort.id, runNumbers: cohort.runs.map((run) => run.runNumber), actualCostUSD }, - evidence: "repeated-packing-sensitive", - expectedRepeats, - ...(selectedArm === undefined ? {} : { selectedArm }), - cases, - failures - }; -} - -function validateRunRepeatCount(run: EvalCaseRunInput, expectedRepeats: number, failures: ReportFailure[]): void { - const declared = run.declaredCase.repeat ?? 1; - if (declared !== expectedRepeats || run.executions.length !== expectedRepeats) { - failures.push(failure("repeat_count_mismatch", `run ${run.runNumber} has ${run.executions.length} executions, expected ${expectedRepeats}`, { - declared - })); - } - const indexes = run.executions.map((execution) => execution.repeat).sort((a, b) => a - b); - if (stableJson(indexes) !== stableJson(Array.from({ length: expectedRepeats }, (_, index) => index + 1))) { - failures.push(failure("repeat_join_mismatch", `run ${run.runNumber} has missing or duplicate repeat indexes`, { indexes })); - } -} - -function assertMatchingCaseDefinitions( - runs: Record, - failures: ReportFailure[] -): void { - const normalized = (["A", "B", "C"] as const).map((arm) => stableJson(normalizeExperimentSnapshot(runs[arm].declaredCase))); - if (unique(normalized).length !== 1) { - failures.push(failure("arm_case_mismatch", "A/B/C case definitions differ outside packet-packing review fields")); - } -} - -function validateExpectationJoins( - family: string, - runs: Record, - failures: ReportFailure[] -): void { - const canonical = declaredExpectationKeys(runs.A.declaredCase); - if (canonical.duplicates.length > 0) { - failures.push(failure("declared_expectation_join", `${family}/A declares duplicate expectation keys`, { duplicateKeys: canonical.duplicates })); - } - for (const arm of ["B", "C"] as const) { - const declared = declaredExpectationKeys(runs[arm].declaredCase); - if (declared.duplicates.length > 0 || stableJson(declared.values) !== stableJson(canonical.values)) { - failures.push(failure("declared_expectation_join", `${family}/${arm} declared expectations differ from A`, { - expected: canonical.values, - actual: declared.values, - duplicateKeys: declared.duplicates - })); - } - } - for (const arm of ["A", "B", "C"] as const) { - const scores = [runs[arm].info.score, ...runs[arm].executions.map((execution) => execution.score)]; - for (const [scoreIndex, score] of scores.entries()) { - const keys = expectationKeys(score); - if (keys.duplicates.length > 0 || stableJson(keys.values) !== stableJson(canonical.values)) { - failures.push(failure("expectation_join", `${family}/${arm}/score ${scoreIndex} does not have the exact bidirectional declared expectation join`, { - expected: canonical.values, - actual: keys.values, - duplicateKeys: keys.duplicates - })); - } - } - } -} - -function expectationKeys(score: EvalScore | undefined): { values: string[]; duplicates: string[] } { - const raw = score?.expectationResults.map((result) => `${result.list}\0${result.expectationId}`) ?? []; - return { values: [...raw].sort(), duplicates: duplicateValues(raw) }; -} - -function declaredExpectationKeys(evalCase: EvalCase): { values: string[]; duplicates: string[] } { - const raw = (["should_find", "should_find_candidate", "should_not_find"] as const).flatMap((list) => - (evalCase[list] ?? []).map((expectation) => `${list}\0${expectation.id}`) - ); - return { values: [...raw].sort(), duplicates: duplicateValues(raw) }; -} - -function normalizeExperimentSnapshot(snapshot: EvalCase): unknown { - const copy = structuredClone(snapshot); - copy.name = copy.name.replace(/-(?:a|b|c|baseline|selected)$/iu, ""); - delete copy.repeat; - if (copy.review !== undefined) { - delete copy.review.packSameFileHunks; - delete copy.review.packedToolBudgetMode; - } - return copy; -} - -function analyzeTreatmentExecution( - run: EvalCaseRunInput, - execution: EvalExecutionInput, - arm: ExperimentArm, - baselineExecution?: EvalExecutionInput, - requireTreatment = true -): TreatmentExecution { - const failures: ReportFailure[] = []; - const eventByPacket = packingEvents(execution.events, failures); - const packetById = new Map(execution.packets.map((packet) => [packet.id, packet])); - const atoms = arm === "A" ? execution.packets : baselineExecution?.packets ?? []; - const atomById = new Map(atoms.map((packet) => [atomIdForPacket(packet), packet])); - if (atomById.size !== atoms.length) { - failures.push(failure("atom_bijection", `run ${run.runNumber}/repeat ${execution.repeat} has ambiguous A atom IDs`)); - } - if (arm !== "A" && baselineExecution === undefined) { - failures.push(failure("missing_baseline_atoms", `run ${run.runNumber}/repeat ${execution.repeat} cannot join treatment to A atoms`)); - } - const duplicateHunks = duplicateValues(execution.packets.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId))); - if (duplicateHunks.length > 0) { - failures.push(failure("hunk_bijection", `run ${run.runNumber}/repeat ${execution.repeat} duplicates hunks`, { duplicateHunks })); - } - const factsByPath = new Map(execution.fileFacts.map((facts) => [facts.path, facts])); - if (factsByPath.size !== execution.fileFacts.length) { - failures.push(failure("duplicate_file_facts", `run ${run.runNumber}/repeat ${execution.repeat} has duplicate file facts`)); - } - for (const packet of execution.packets) { - if (packet.hunks.length > MAX_HUNKS_PER_PACKET) { - failures.push(failure("packet_cap", `packet ${packet.id} exceeds the five-hunk cap`)); - } - const facts = factsByPath.get(packet.path); - if (facts === undefined || stableJson(packet.dispatchRank) !== stableJson(packetDispatchRank(packet.path, facts, changedLines(packet)))) { - failures.push(failure("invalid_dispatch_rank", `packet ${packet.id} has an invalid dispatch rank`)); - } - if (requestedLensSignatureForAtom(execution, packet) === undefined) { - failures.push(failure("stage5_lens_join", `packet ${packet.id} lacks a non-empty Stage-5 lens decision for every hunk`)); - } - } - if (arm !== "A") { - const baselineHunks = atoms.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId)); - const selectedHunks = execution.packets.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId)); - if (stableJson([...baselineHunks].sort()) !== stableJson([...selectedHunks].sort())) { - failures.push(failure("hunk_bijection", `run ${run.runNumber}/repeat ${execution.repeat} does not preserve the exact A hunk set`)); - } - } - const usedAtomIds: string[] = []; - for (const [packetId, event] of eventByPacket) { - const packet = packetById.get(packetId); - if (packet === undefined) { - failures.push(failure("missing_packet", `treatment telemetry references missing packet ${packetId}`)); - continue; - } - const sourceAtoms = event.atomIds.map((atomId) => atomById.get(atomId)); - const allAtomsResolved = sourceAtoms.every((atom): atom is ReviewPacket => atom !== undefined); - if (!allAtomsResolved) { - failures.push(failure("unknown_source_atom", `packet ${packetId} references an atom absent from the matching A repeat`)); - } - const resolvedAtoms = sourceAtoms.filter((atom): atom is ReviewPacket => atom !== undefined); - const expectedHunks = resolvedAtoms.flatMap((atom) => atom.hunks.map((hunk) => hunk.hunkId)); - const packetHunks = packet.hunks.map((hunk) => hunk.hunkId); - const standaloneProfiles = resolvedAtoms.map((atom) => atom.reviewProfile); - const profileFloor = standaloneProfiles.reduce((highest, profile) => PROFILE_RANK[profile] > PROFILE_RANK[highest] ? profile : highest, "simple" as ReviewProfile); - const rawPatchChars = actualPatchCharsForHunk(execution.diff); - const patchChars = sum(packet.hunks.map((hunk) => rawPatchChars.get(hunk.hunkId) ?? Number.MAX_SAFE_INTEGER)); - const depth = run.declaredCase.review?.depth ?? "normal"; - const baseBudget = toolBudget(packet.coverage, depth, packet.reviewProfile); - const additionalAtoms = Math.max(0, resolvedAtoms.length - 1); - const packedBudget = arm === "C" && resolvedAtoms.length > 1 && packet.reviewProfile !== "simple" - ? { - ...baseBudget, - maxToolCalls: Math.min(baseBudget.maxToolCalls + additionalAtoms, Math.ceil(1.75 * baseBudget.maxToolCalls)), - maxResultChars: Math.min(baseBudget.maxResultChars + additionalAtoms * 2_000, Math.ceil(1.75 * baseBudget.maxResultChars)) - } - : baseBudget; - const expectedEffectiveBudget = scaleToolBudget(packedBudget, run.declaredCase.review?.budgetBoost ?? 1); - if ( - event.sourceAtomCount !== event.atomIds.length || - unique(event.atomIds).length !== event.atomIds.length || - stableJson(event.standaloneProfiles) !== stableJson(standaloneProfiles) || - stableJson(expectedHunks) !== stableJson(packetHunks) || - event.hunkCount !== packet.hunks.length || - event.capUsage.hunks !== packet.hunks.length || - event.capUsage.maxHunks !== MAX_HUNKS_PER_PACKET || - event.capUsage.maxPatchChars !== MAX_PATCH_CHARS || - event.capUsage.patchChars !== patchChars || - event.capUsage.hunks > MAX_HUNKS_PER_PACKET || - event.capUsage.patchChars > MAX_PATCH_CHARS || - resolvedAtoms.some((atom) => atom.path !== packet.path || atom.language !== packet.language || atom.coverage !== packet.coverage) || - event.effectiveCoverage !== packet.coverage || - event.effectiveProfile !== packet.reviewProfile || - event.profileFloor !== profileFloor || - stableJson(event.baseToolBudget) !== stableJson(baseBudget) || - stableJson(event.effectiveToolBudget) !== stableJson(expectedEffectiveBudget) || - stableJson(event.effectiveToolBudget) !== stableJson(packet.toolBudget) || - event.toolBudgetMode !== (arm === "C" ? "atom-scaled" : "base") - ) { - failures.push(failure("treatment_invariant", `packet ${packetId} has invalid atom/cap telemetry`)); - } - if (PROFILE_RANK[event.effectiveProfile] < PROFILE_RANK[profileFloor]) { - failures.push(failure("effective_profile_downgrade", `packet ${packetId} falls below its standalone profile floor`)); - } - if (budgetDowngraded(event.effectiveToolBudget, event.baseToolBudget)) { - failures.push(failure("effective_budget_downgrade", `packet ${packetId} effective budget falls below its base budget`)); - } - if (!event.plannerLensesPreserved) { - failures.push(failure("planner_lens_drop", `packet ${packetId} reports a planner-selected lens drop`)); - } - if (baselineExecution !== undefined) { - const requestedSignatures = resolvedAtoms.map((atom) => requestedLensSignatureForAtom(baselineExecution, atom)); - const selectedRequestedSignature = requestedLensSignatureForAtom(execution, packet); - if ( - requestedSignatures.some((signature) => signature === undefined) || selectedRequestedSignature === undefined || - unique(requestedSignatures).length !== 1 || requestedSignatures[0] !== event.requestedLensSignature || - selectedRequestedSignature !== event.requestedLensSignature - ) { - failures.push(failure("requested_lens_join", `packet ${packetId} requested-lens telemetry does not match Stage-5 decisions for its A atoms`)); - } - // Stage 6 may validly prune a requested lens before constructing the A - // atom. Packing must preserve that routed A surface exactly; it must not - // reinterpret intentional A-side routing as a packing regression. - const aRoutedLenses = sortedUnique(resolvedAtoms.flatMap((atom) => atom.lenses)); - const selectedRoutedLenses = sortedUnique(packet.lenses); - if (stableJson(selectedRoutedLenses) !== stableJson(aRoutedLenses)) { - const missingRoutedLenses = aRoutedLenses.filter((lens) => !selectedRoutedLenses.includes(lens)); - const extraRoutedLenses = selectedRoutedLenses.filter((lens) => !aRoutedLenses.includes(lens)); - failures.push(failure("routed_lens_join", `packet ${packetId} does not exactly preserve the allowed routed-lens set from its A atoms`, { - missingLensHashes: missingRoutedLenses.map((lens) => sha256Hex(lens)), - extraLensHashes: extraRoutedLenses.map((lens) => sha256Hex(lens)) - })); - } - } - usedAtomIds.push(...event.atomIds); - } - if (arm !== "A") { - for (const packet of execution.packets.filter((candidate) => !eventByPacket.has(candidate.id))) { - const matchingAtom = atomById.get(atomIdForPacket(packet)); - if (matchingAtom === undefined || stableJson(matchingAtom) !== stableJson(packet)) { - failures.push(failure("untreated_packet_mismatch", `packet ${packet.id} has no treatment event and is not an exact A atom`)); - } else { - usedAtomIds.push(atomIdForPacket(matchingAtom)); - } - } - if (stableJson([...usedAtomIds].sort()) !== stableJson([...atomById.keys()].sort())) { - failures.push(failure("atom_bijection", `run ${run.runNumber}/repeat ${execution.repeat} does not consume each A atom exactly once`)); - } - } - if (arm === "A" && eventByPacket.size > 0) { - failures.push(failure("baseline_treatment", `baseline run ${run.runNumber} emitted packing treatment telemetry`)); - } - if (arm !== "A" && requireTreatment && eventByPacket.size === 0) { - failures.push(failure("missing_treatment_telemetry", `packed run ${run.runNumber}/repeat ${execution.repeat} has no same_file_atoms_packed telemetry`)); - } - - const target = targetPacket(run.declaredCase, execution.packets); - const targetEvent = target === undefined ? undefined : eventByPacket.get(target.id); - const treated = arm !== "A" && targetEvent !== undefined && targetEvent.sourceAtomCount >= 2; - if (arm !== "A" && target === undefined) { - failures.push(failure("missing_target_packet", `run ${run.runNumber}/repeat ${execution.repeat} has no packet for its positive expectation`)); - } - const pressure = executionPressure(execution, eventByPacket, failures); - const targetAtomCount = target === undefined - ? undefined - : arm === "A" - ? 1 - : targetEvent?.atomIds.length ?? atoms.filter((atom) => atom.hunks.some((hunk) => target.hunks.some((targetHunk) => targetHunk.hunkId === hunk.hunkId))).length; - const targetRequestedLensSignature = arm === "A" && target !== undefined - ? requestedLensSignatureForAtom(execution, target) - : targetEvent?.requestedLensSignature; - return { - repeat: execution.repeat, - treated, - atomIds: [...atomById.keys()].sort(), - hunkIds: execution.packets.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId)).sort(), - ...(target === undefined ? {} : { - targetPacketId: target.id, - targetPath: target.path, - targetFilePackets: execution.packets.filter((packet) => packet.path === target.path).length, - ...(targetAtomCount === undefined ? {} : { targetAtomCount }), - targetCoverage: target.coverage, - ...(targetRequestedLensSignature === undefined ? {} : { targetRequestedLensSignature }) - }), - pressure, - failures - }; -} - -function targetPacket(evalCase: EvalCase, packets: ReviewPacket[]): ReviewPacket | undefined { - const expectation = evalCase.should_find_candidate?.[0] ?? evalCase.should_find?.[0]; - if (expectation === undefined) { - return packets[0]; - } - const pathMatches = packets.filter((packet) => expectation.path === undefined || globMatches(expectation.path, packet.path)); - if (expectation.lineRange === undefined) { - return pathMatches[0]; - } - return pathMatches.find((packet) => packet.hunks.some((hunk) => - [...hunk.changedNewLineNumbers, ...hunk.changedOldLineNumbers].some((line) => line >= expectation.lineRange![0] && line <= expectation.lineRange![1]) - )); -} - -function globMatches(pattern: string, value: string): boolean { - const escaped = pattern.replace(/[.+?^${}()|[\]\\]/gu, "\\$&").replace(/\*\*/gu, "\u0000").replace(/\*/gu, "[^/]*").replace(/\u0000/gu, ".*"); - return new RegExp(`^${escaped}$`, "u").test(value); -} - -function requestedLensSignatureForAtom(execution: EvalExecutionInput, atom: ReviewPacket): string | undefined { - const lenses = requestedLensesForAtom(execution, atom); - return lenses === undefined ? undefined : JSON.stringify(lenses); -} - -function requestedLensesForAtom(execution: EvalExecutionInput, atom: ReviewPacket): string[] | undefined { - const decisions = new Map(execution.plan.coverage.map((decision) => [decision.hunkId, decision])); - const lenses = atom.hunks.flatMap((hunk) => { - const decision = decisions.get(hunk.hunkId); - return decision?.lenses ?? []; - }); - if (atom.hunks.some((hunk) => { - const decision = decisions.get(hunk.hunkId); - return decision === undefined || decision.lenses.length === 0 || decision.lenses.some((lens) => lens.trim().length === 0); - })) { - return undefined; - } - return sortedUnique(lenses.map((lens) => lens.trim())); -} - -function compareTargetTreatment( - family: string, - treatments: Record, - failures: ReportFailure[] -): void { - for (const repeat of treatments.A.map((entry) => entry.repeat)) { - const a = treatments.A.find((entry) => entry.repeat === repeat); - const b = treatments.B.find((entry) => entry.repeat === repeat); - const c = treatments.C.find((entry) => entry.repeat === repeat); - if (a === undefined || b === undefined || c === undefined) { - continue; - } - for (const [arm, packed] of [["B", b], ["C", c]] as const) { - if (packed.targetPath !== a.targetPath || packed.targetCoverage !== a.targetCoverage) { - failures.push(failure("target_semantics_mismatch", `${family}/${arm}/repeat ${repeat} changes target path or coverage`)); - } - if (packed.targetFilePackets !== undefined && a.targetFilePackets !== undefined && packed.treated && packed.targetFilePackets >= a.targetFilePackets) { - failures.push(failure("target_packet_count", `${family}/${arm}/repeat ${repeat} treatment does not reduce target-file packet count`)); - } - } - if (b.targetRequestedLensSignature !== a.targetRequestedLensSignature || c.targetRequestedLensSignature !== a.targetRequestedLensSignature) { - failures.push(failure("target_lens_signature", `${family}/repeat ${repeat} packed requested-lens signatures differ from A`)); - } - } -} - -function executionPressure( - execution: EvalExecutionInput, - eventByPacket: Map, - failures: ReportFailure[] -): ExecutionPressure { - const reviewedPacketIds = new Set(execution.modelCalls.filter((call) => call.stage === 7 && call.role === "packetReview" && call.packetId !== undefined).map((call) => call.packetId as string)); - const reviewedAtoms = sum([...reviewedPacketIds].map((packetId) => eventByPacket.get(packetId)?.sourceAtomCount ?? 1)); - const tools = execution.toolCalls.filter((call) => call.stage === 7 && call.initiator === "model" && (call.packetId === undefined || reviewedPacketIds.has(call.packetId))); - const modelCalls = execution.modelCalls.filter((call) => call.stage === 7 && call.role === "packetReview" && call.cacheStatus !== "hit"); - const rejected = tools.filter((call) => call.status === "rejected"); - const reasons: Record = {}; - for (const call of rejected) { - const reason = call.degradationReason ?? call.errorCode ?? "unknown"; - reasons[reason] = (reasons[reason] ?? 0) + 1; - } - if (modelCalls.length > 0 && reviewedAtoms === 0) { - failures.push(failure("missing_reviewed_atom_telemetry", "Stage-7 model calls cannot be joined to reviewed packets/atoms")); - } - const stage7CostUSD = sum(modelCalls.map((call) => call.costUSD ?? 0)); - return { - reviewedPackets: reviewedPacketIds.size, - reviewedAtoms, - requestedToolCalls: tools.length, - usedToolCalls: tools.filter((call) => call.status !== "rejected" && call.status !== "skipped").length, - rejectedToolCalls: rejected.length, - rejectionRatePerAtom: reviewedAtoms === 0 ? 0 : round(rejected.length / reviewedAtoms), - rejectionReasons: reasons, - resultChars: sum(tools.map((call) => call.resultChars)), - continuations: modelCalls.filter((call) => call.kind === "tool-continuation").length, - modelServiceSeconds: round(sum(modelCalls.map((call) => call.durationMs)) / 1000), - inputTokens: sum(modelCalls.map((call) => call.inputTokens ?? 0)), - outputTokens: sum(modelCalls.map((call) => call.outputTokens ?? 0)), - stage7CostUSD: round(stage7CostUSD), - costPerReviewedAtomUSD: reviewedAtoms === 0 ? null : round(stage7CostUSD / reviewedAtoms) - }; -} - -function combinePressure(entries: ExecutionPressure[]): ExecutionPressure { - const reviewedAtoms = sum(entries.map((entry) => entry.reviewedAtoms)); - const rejected = sum(entries.map((entry) => entry.rejectedToolCalls)); - const stage7CostUSD = sum(entries.map((entry) => entry.stage7CostUSD)); - const reasons: Record = {}; - for (const entry of entries) { - for (const [reason, count] of Object.entries(entry.rejectionReasons)) { - reasons[reason] = (reasons[reason] ?? 0) + count; - } - } - return { - reviewedPackets: sum(entries.map((entry) => entry.reviewedPackets)), - reviewedAtoms, - requestedToolCalls: sum(entries.map((entry) => entry.requestedToolCalls)), - usedToolCalls: sum(entries.map((entry) => entry.usedToolCalls)), - rejectedToolCalls: rejected, - rejectionRatePerAtom: reviewedAtoms === 0 ? 0 : round(rejected / reviewedAtoms), - rejectionReasons: reasons, - resultChars: sum(entries.map((entry) => entry.resultChars)), - continuations: sum(entries.map((entry) => entry.continuations)), - modelServiceSeconds: round(sum(entries.map((entry) => entry.modelServiceSeconds)), 3), - inputTokens: sum(entries.map((entry) => entry.inputTokens)), - outputTokens: sum(entries.map((entry) => entry.outputTokens)), - stage7CostUSD: round(stage7CostUSD), - costPerReviewedAtomUSD: reviewedAtoms === 0 ? null : round(stage7CostUSD / reviewedAtoms) - }; -} - -function expectationRates( - executions: EvalExecutionInput[], - include: (execution: EvalExecutionInput) => boolean, - evidenceScores: Map, - atomCounts: Map -): ExpectationRate[] { - const selected = executions.filter(include); - const keys = sortedUnique(executions.flatMap((execution) => (evidenceScores.get(execution.repeat) ?? execution.score).expectationResults - .filter((result) => result.list !== "should_not_find") - .map((result) => `${result.list}\0${result.expectationId}`))); - return keys.map((key) => { - const [list, expectationId] = key.split("\0") as [EvalExpectationList, string]; - let candidateHits = 0; - let finalHits = 0; - const lossHistogram: Record = {}; - const lossByAtomCount: Record = {}; - for (const execution of selected) { - const result = (evidenceScores.get(execution.repeat) ?? execution.score).expectationResults.find((entry) => entry.list === list && entry.expectationId === expectationId); - const atomBucket = String(atomCounts.get(execution.repeat) ?? 1); - const bucket = lossByAtomCount[atomBucket] ?? { denominator: 0, missedBeforeCandidate: 0 }; - bucket.denominator += 1; - lossByAtomCount[atomBucket] = bucket; - if (result === undefined) { - lossHistogram.error = (lossHistogram.error ?? 0) + 1; - continue; - } - if (candidateMatched(result)) { - candidateHits += 1; - } - if (finalMatched(result)) { - finalHits += 1; - } - if (result.loss !== undefined) { - lossHistogram[result.loss.label] = (lossHistogram[result.loss.label] ?? 0) + 1; - if (result.loss.label === "missed-before-candidate-generation") { - bucket.missedBeforeCandidate += 1; - } - } - } - return { - expectationId, - list, - candidateHits, - finalHits, - denominator: selected.length, - candidateRate: selected.length === 0 ? 0 : round(candidateHits / selected.length), - finalRate: selected.length === 0 ? 0 : round(finalHits / selected.length), - lossHistogram, - lossByAtomCount - }; - }); -} - -function candidateMatched(result: EvalExpectationResult): boolean { - if (result.status === "pass") { - return true; - } - return result.list === "should_find" && - (result.loss?.label === "lost-at-verification" || result.loss?.label === "lost-at-composition"); -} - -function finalMatched(result: EvalExpectationResult): boolean { - return result.list === "should_find" && result.status === "pass"; -} - -function enforceRequiredRecallZero( - family: string, - arms: EvalCaseReport["arms"], - failures: ReportFailure[] -): void { - const baselineByKey = new Map(arms.A.intentToTreat.map((rate) => [`${rate.list}\0${rate.expectationId}`, rate])); - for (const arm of ["B", "C"] as const) { - for (const rate of arms[arm].intentToTreat) { - const baseline = baselineByKey.get(`${rate.list}\0${rate.expectationId}`); - if (baseline === undefined) { - failures.push(failure("expectation_join", `${family}/${arm} has an expectation absent from A: ${rate.expectationId}`)); - continue; - } - if ((baseline.candidateHits > 0 && rate.candidateHits === 0) || (baseline.finalHits > 0 && rate.finalHits === 0)) { - failures.push(failure("required_recall_zero", `${family}/${arm}/${rate.expectationId} falls from non-zero recall to zero`)); - } - } - } -} - -function cohortRecallEvidence(cases: EvalCaseReport[], arm: ExperimentArm): { - candidateHits: number; - finalHits: number; - earlyLosses: number; - observations: number; - lossByAtomCount: Record; -} { - const rates = cases.flatMap((entry) => entry.arms[arm].intentToTreat); - const buckets = sortedUnique(rates.flatMap((rate) => Object.keys(rate.lossByAtomCount))); - return { - candidateHits: sum(rates.map((rate) => rate.candidateHits)), - finalHits: sum(rates.map((rate) => rate.finalHits)), - earlyLosses: sum(rates.map((rate) => rate.lossHistogram["missed-before-candidate-generation"] ?? 0)), - observations: sum(rates.map((rate) => rate.denominator)), - lossByAtomCount: Object.fromEntries(buckets.map((bucket) => [bucket, { - denominator: sum(rates.map((rate) => rate.lossByAtomCount[bucket]?.denominator ?? 0)), - missedBeforeCandidate: sum(rates.map((rate) => rate.lossByAtomCount[bucket]?.missedBeforeCandidate ?? 0)) - }])) - }; -} - -function cohortArmRecallPasses(cases: EvalCaseReport[], arm: "B" | "C"): boolean { - const baseline = cohortRecallEvidence(cases, "A"); - const packed = cohortRecallEvidence(cases, arm); - const baselineEarlyRate = baseline.observations === 0 ? 0 : baseline.earlyLosses / baseline.observations; - return baseline.candidateHits - packed.candidateHits <= 1 && baseline.finalHits - packed.finalHits <= 1 && - packed.earlyLosses - baseline.earlyLosses <= 1 && Object.values(packed.lossByAtomCount).every((bucket) => - bucket.missedBeforeCandidate - baselineEarlyRate * bucket.denominator <= 1 - ); -} - -function enforceCohortRecallGates(cases: EvalCaseReport[], failures: ReportFailure[]): void { - const baseline = cohortRecallEvidence(cases, "A"); - const baselineEarlyRate = baseline.observations === 0 ? 0 : baseline.earlyLosses / baseline.observations; - for (const arm of ["B", "C"] as const) { - const packed = cohortRecallEvidence(cases, arm); - if (baseline.candidateHits - packed.candidateHits > 1 || baseline.finalHits - packed.finalHits > 1) { - failures.push(failure("recall_non_inferiority", `${arm} loses more than the single cohort-wide candidate or final hit allowance`, { - baselineCandidate: baseline.candidateHits, - packedCandidate: packed.candidateHits, - baselineFinal: baseline.finalHits, - packedFinal: packed.finalHits - })); - } - if (packed.earlyLosses - baseline.earlyLosses > 1) { - failures.push(failure("candidate_generation_loss", `${arm} exceeds the single cohort-wide allowance for losses before candidate generation`)); - } - for (const [bucket, evidence] of Object.entries(packed.lossByAtomCount)) { - if (evidence.missedBeforeCandidate - baselineEarlyRate * evidence.denominator > 1) { - failures.push(failure("candidate_generation_loss_by_atom_count", `${arm} exceeds the cohort-wide atom-count ${bucket} loss allowance`, { - atomCount: Number(bucket), ...evidence, baselineEarlyRate: round(baselineEarlyRate) - })); - } - } - } -} - -function enforcePressureGate( - family: string, - arms: EvalCaseReport["arms"], - failures: ReportFailure[] -): void { - const aRecall = sum(arms.A.intentToTreat.map((rate) => rate.candidateHits)); - for (const arm of ["B", "C"] as const) { - const packedRecall = sum(arms[arm].intentToTreat.map((rate) => rate.candidateHits)); - if (arms[arm].pressure.rejectionRatePerAtom - arms.A.pressure.rejectionRatePerAtom > 0.1 && packedRecall <= aRecall) { - failures.push(failure("tool_pressure", `${family}/${arm} exceeds A by more than 0.10 rejected attempts per reviewed atom without higher candidate recall`)); - } - } - if (arms.C.pressure.rejectionRatePerAtom > arms.B.pressure.rejectionRatePerAtom) { - failures.push(failure("atom_scaled_pressure", `${family}/C increases normalized tool rejections over B`)); - } -} - -function selectCohortPackedArm( - cases: EvalCaseReport[], - expectedRepeats: number, - failures: ReportFailure[] -): "B" | "C" { - let eligible = new Set<"B" | "C">(["B", "C"]); - const globalBRecallPasses = cohortArmRecallPasses(cases, "B"); - const globalCRecallPasses = cohortArmRecallPasses(cases, "C"); - const eligibilityByCase = cases.map((entry) => ({ - entry, - B: packedArmPasses(entry.arms, "B", expectedRepeats) && globalBRecallPasses, - C: packedArmPasses(entry.arms, "C", expectedRepeats) && globalCRecallPasses && cStrictlyQualifies(entry.arms) - })); - const bCohortEligible = eligibilityByCase.every((entry) => entry.B); - let cCohortEligible = eligibilityByCase.every((entry) => entry.C); - if (!bCohortEligible && cCohortEligible) { - const retention = cohortPacketCountServiceSavingsRetention(cases); - if (retention < 0.85) { - cCohortEligible = false; - failures.push(failure("atom_scaled_savings_retention", "C does not retain at least 85% of cohort packet-count-implied service-time savings while B is ineligible", { - retention: round(retention, 6) - })); - } - } - for (const { entry, B: bPasses, C: caseCPasses } of eligibilityByCase) { - const cPasses = caseCPasses && cCohortEligible; - const passing: Array<"B" | "C"> = []; - if (bPasses) { - passing.push("B"); - } - if (cPasses) { - passing.push("C"); - } - if (passing.length === 0) { - failures.push(failure("no_passing_arm", `${entry.case} has no packed arm that passes treatment, recall, and pressure gates`)); - } - eligible = new Set([...eligible].filter((arm) => passing.includes(arm))); - } - if (eligible.size === 0) { - failures.push(failure("conflicting_arm_selection", "case-level gates require conflicting product arms; one cohort-wide arm cannot be selected")); - eligible = new Set(["B", "C"]); - } - if (eligible.size === 1) { - return [...eligible][0]!; - } - const economics = Object.fromEntries((["B", "C"] as const).map((arm) => [arm, { - costUSD: round(sum(cases.map((entry) => entry.arms[arm].actualCostUSD))), - modelServiceSeconds: round(sum(cases.map((entry) => entry.arms[arm].pressure.modelServiceSeconds)), 3) - }])) as Record<"B" | "C", { costUSD: number; modelServiceSeconds: number }>; - const bDominates = economics.B.costUSD <= economics.C.costUSD && economics.B.modelServiceSeconds <= economics.C.modelServiceSeconds; - const cDominates = economics.C.costUSD <= economics.B.costUSD && economics.C.modelServiceSeconds <= economics.B.modelServiceSeconds; - if (bDominates && !cDominates) { - return "B"; - } - if (cDominates && !bDominates) { - return "C"; - } - if (!bDominates && !cDominates) { - failures.push(failure("conflicting_arm_economics", "B/C trade off actual cost against model-service speed; neither is globally cheaper and faster", economics)); - } - return "B"; -} - -function packedArmPasses(arms: EvalCaseReport["arms"], arm: "B" | "C", expectedRepeats: number): boolean { - if (!arms[arm].treatment.valid) { - return false; - } - if (expectedRepeats < 10) { - return true; - } - const baselineByKey = new Map(arms.A.intentToTreat.map((rate) => [`${rate.list}\0${rate.expectationId}`, rate])); - const requiredZeroPasses = arms[arm].intentToTreat.every((rate) => { - const baseline = baselineByKey.get(`${rate.list}\0${rate.expectationId}`); - if (baseline === undefined) { - return false; - } - return !((baseline.candidateHits > 0 && rate.candidateHits === 0) || (baseline.finalHits > 0 && rate.finalHits === 0)); - }); - const baselineCandidate = sum(arms.A.intentToTreat.map((rate) => rate.candidateHits)); - const packedCandidate = sum(arms[arm].intentToTreat.map((rate) => rate.candidateHits)); - const pressurePasses = arms[arm].pressure.rejectionRatePerAtom - arms.A.pressure.rejectionRatePerAtom <= 0.1 || packedCandidate > baselineCandidate; - const cPressurePasses = arm !== "C" || arms.C.pressure.rejectionRatePerAtom <= arms.B.pressure.rejectionRatePerAtom; - return requiredZeroPasses && pressurePasses && cPressurePasses; -} - -function cStrictlyQualifies(arms: EvalCaseReport["arms"]): boolean { - if (arms.C.pressure.rejectionRatePerAtom > arms.B.pressure.rejectionRatePerAtom) { - return false; - } - const bCandidate = sum(arms.B.intentToTreat.map((rate) => rate.candidateHits)); - const cCandidate = sum(arms.C.intentToTreat.map((rate) => rate.candidateHits)); - const bFinal = sum(arms.B.intentToTreat.map((rate) => rate.finalHits)); - const cFinal = sum(arms.C.intentToTreat.map((rate) => rate.finalHits)); - return arms.C.pressure.rejectionRatePerAtom < arms.B.pressure.rejectionRatePerAtom || cCandidate > bCandidate || cFinal > bFinal; -} - -function cohortPacketCountServiceSavingsRetention(cases: EvalCaseReport[]): number { - const baselinePackets = sum(cases.map((entry) => entry.arms.A.pressure.reviewedPackets)); - const packedPackets = sum(cases.map((entry) => entry.arms.C.pressure.reviewedPackets)); - const baselineService = sum(cases.map((entry) => entry.arms.A.pressure.modelServiceSeconds)); - const packedService = sum(cases.map((entry) => entry.arms.C.pressure.modelServiceSeconds)); - const packetCountSavings = baselinePackets > 0 - ? baselineService * Math.max(0, baselinePackets - packedPackets) / baselinePackets - : 0; - const actualSavings = baselineService - packedService; - return packetCountSavings > 0 ? actualSavings / packetCountSavings : 0; -} - -function duplicateValues(values: string[]): string[] { - return sortedUnique(values.filter((value, index) => values.indexOf(value) !== index)); -} - -function aggregateCohortCost(runs: EvalCaseRunInput[], failures: ReportFailure[]): number { - return round(sum(runs.map((run) => costForRun(run, failures)))); -} - -function costForRun(run: EvalCaseRunInput, failures: ReportFailure[]): number { - const costs = run.executions.map((execution) => { - const providerCalls = execution.modelCalls.filter((call) => call.cacheStatus !== "hit"); - return providerCalls.every((call) => call.costUSD !== undefined) - ? sum(providerCalls.map((call) => call.costUSD ?? 0)) - : undefined; - }); - if (costs.some((cost) => cost === undefined)) { - failures.push(failure("missing_spend_data", `run ${run.runNumber} lacks actual spend data for one or more executions`)); - } - return round(sum(costs.map((cost) => cost ?? 0))); -} - -function reviewedHunksForRun(run: EvalCaseRunInput): number { - return sum(run.executions.map((execution) => execution.reviewedHunkIds.length)); -} - -function validateProductionProvenance( - baseline: EvalCaseRunInput, - selected: EvalCaseRunInput, - failures: ReportFailure[] -): void { - for (const [label, run] of [["baseline", baseline], ["selected", selected]] as const) { - const repo = run.info.repo; - if ( - repo?.root !== PRODUCTION_REPO_ROOT || repo.baseSha !== PRODUCTION_BASE_SHA || - repo.mergeBase !== PRODUCTION_BASE_SHA || repo.headSha !== PRODUCTION_HEAD_SHA - ) { - failures.push(failure("production_refs", `${label} production run does not use the exact pinned Plan 102 refs`)); - } - const command = run.declaredCase.command; - if ( - run.declaredCase.repo?.external !== PRODUCTION_REPO_ROOT || command?.base !== PRODUCTION_BASE_SHA || command.head !== PRODUCTION_HEAD_SHA - ) { - failures.push(failure("production_refs", `${label} production case does not declare the exact pinned Plan 102 base/head`)); - } - const review = run.declaredCase.review; - const effective = run.info.effectiveConfig; - if ( - review?.cache !== false || run.info.cache.enabled !== false || review.concurrency !== PRODUCTION_CONCURRENCY || - review.maxTimeMinutes !== 60 || effective?.review.concurrency !== PRODUCTION_CONCURRENCY || - effective.review.timeoutMs !== PRODUCTION_TIMEOUT_MS || effective.llm.maxConcurrentCalls !== PRODUCTION_CONCURRENCY - ) { - failures.push(failure("production_run_shape", `${label} production arm must use the exact external repo, cache-off, concurrency-6, 60-minute declared/effective run shape`)); - } - } - const baselineRuntime = baseline.info.codegenieRuntime; - const selectedRuntime = selected.info.codegenieRuntime; - if ( - baselineRuntime?.commit === undefined || selectedRuntime?.commit === undefined || - baselineRuntime.commit !== selectedRuntime.commit || baselineRuntime.dirty !== false || selectedRuntime.dirty !== false - ) { - failures.push(failure("production_runtime_provenance", "production arms must share one present clean Codegenie runtime commit")); - } -} - -function validateProductionExpectations( - baseline: EvalCaseRunInput, - selected: EvalCaseRunInput, - failures: ReportFailure[] -): void { - const canonical = declaredExpectationKeys(baseline.declaredCase); - const selectedDeclared = declaredExpectationKeys(selected.declaredCase); - if ( - canonical.values.length === 0 || canonical.duplicates.length > 0 || selectedDeclared.duplicates.length > 0 || - stableJson(canonical.values) !== stableJson(selectedDeclared.values) - ) { - failures.push(failure("declared_expectation_join", "production arms do not declare the exact same unique expectation set")); - } - const requiredKeys = new Set((["should_find", "should_find_candidate", "should_not_find"] as const).flatMap((list) => - (baseline.declaredCase[list] ?? []).filter((expectation) => expectation.tier !== "optional").map((expectation) => `${list}\0${expectation.id}`) - )); - for (const [label, run] of [["baseline", baseline], ["selected", selected]] as const) { - for (const execution of run.executions) { - const actual = expectationKeys(execution.score); - if (actual.duplicates.length > 0 || stableJson(actual.values) !== stableJson(canonical.values)) { - failures.push(failure("expectation_join", `${label} production score does not match its declared expectations bidirectionally`)); - } - const candidateIds = new Set(execution.candidateFindings.map((finding) => finding.id)); - const finalIds = new Set(execution.finalFindings.map((finding) => finding.id)); - validateProductionFindingArtifacts(execution, failures, label); - for (const result of execution.score.expectationResults) { - const expectedArtifact = result.list === "should_find_candidate" ? "candidate-findings" : "final-findings"; - if (result.matched.some((match) => - match.artifact !== expectedArtifact || - (match.artifact === "candidate-findings" ? !candidateIds.has(match.findingId) : !finalIds.has(match.findingId)) - )) { - failures.push(failure("production_finding_artifact_join", `${label} production score references a finding absent from its strict candidate/final artifacts`)); - } - const requiredPositive = requiredKeys.has(`${result.list}\0${result.expectationId}`) && result.list !== "should_not_find"; - if ( - (requiredKeys.has(`${result.list}\0${result.expectationId}`) && result.status !== "pass") || - requiredPositive && result.matched.length === 0 - ) { - failures.push(failure("production_finding_preservation", `${label} production arm does not preserve a required declared finding outcome`)); - } - } - } - } -} - -function validateProductionFindingArtifacts( - execution: EvalExecutionInput, - failures: ReportFailure[], - label: string -): void { - const packetIds = new Set(execution.packets.map((packet) => packet.id)); - const hunkPaths = new Map(execution.diff.files.flatMap((file) => file.hunks.map((hunk) => [hunk.id, hunk.path] as const))); - const candidateIds = new Set(execution.candidateFindings.map((finding) => finding.id)); - const validAnchor = (anchor: { hunkId: string; path: string } | undefined): boolean => - anchor === undefined || hunkPaths.get(anchor.hunkId) === anchor.path; - const candidateRelationsValid = execution.candidateFindings.every((finding) => - packetIds.has(finding.producedBy.packetId) && validAnchor(finding.anchor) - ); - const finalRelationsValid = execution.finalFindings.every((finding) => - packetIds.has(finding.producedBy.packetId) && validAnchor(finding.anchor) && - finding.mergedCandidateIds.length > 0 && finding.mergedCandidateIds.every((id) => candidateIds.has(id)) && - (finding.mergedAnchors ?? []).every((anchor) => validAnchor(anchor)) - ); - const inline = execution.finalFindings.filter((finding) => finding.publication === "inline").length; - const summaryOnly = execution.finalFindings.filter((finding) => finding.publication === "summary-only").length; - const suppressed = execution.finalFindings.filter((finding) => finding.publication === "suppressed").length; - const metrics = execution.score.metrics; - const countsValid = - metrics.candidateFindings === execution.candidateFindings.length && metrics.inlineFindings === inline && - metrics.summaryOnlyFindings === summaryOnly && metrics.suppressedFindings === suppressed && - metrics.reportedFindings === inline + summaryOnly; - if (!candidateRelationsValid || !finalRelationsValid || !countsValid) { - failures.push(failure("production_finding_relations", `${label} production candidate/final artifacts do not join exactly to packets, hunks, lineage, and score counts`, { - candidates: execution.candidateFindings.length, - finals: execution.finalFindings.length, - packets: packetIds.size, - hunks: hunkPaths.size - })); - } -} - -function validateProductionModelAccounting( - run: EvalCaseRunInput, - treatments: TreatmentExecution[], - failures: ReportFailure[], - label: string -): void { - for (const execution of run.executions) { - const treatment = treatments.find((entry) => entry.repeat === execution.repeat); - const reviewedHunks = new Set(execution.reviewedHunkIds); - const reviewedPacketIds = new Set(execution.packets.filter((packet) => packet.hunks.some((hunk) => reviewedHunks.has(hunk.hunkId))).map((packet) => packet.id)); - const stage7 = execution.modelCalls.filter((call) => call.stage === 7 && call.role === "packetReview"); - const initialPacketIds = stage7.filter((call) => call.kind === "initial" && call.packetId !== undefined).map((call) => call.packetId as string); - const completeFields = execution.modelCalls.every((call) => - call.inputTokens !== undefined && call.outputTokens !== undefined && call.reasoningTokens !== undefined && call.totalTokens !== undefined && - call.costUSD !== undefined && call.durationMs > 0 && call.totalTokens >= call.inputTokens + call.outputTokens && call.status === "ok" - ); - const joined = stage7.length > 0 && stage7.every((call) => call.packetId !== undefined && reviewedPacketIds.has(call.packetId) && call.status === "ok") && - unique(initialPacketIds).length === initialPacketIds.length && stableJson([...initialPacketIds].sort()) === stableJson([...reviewedPacketIds].sort()); - const scoreCount = execution.score.metrics.modelCalls; - const scoreCost = execution.score.metrics.costUSD; - const recordedCost = sum(execution.modelCalls.map((call) => call.costUSD ?? 0)); - const scoreJoined = scoreCount === execution.modelCalls.length && scoreCost !== undefined && Math.abs(scoreCost - recordedCost) <= 1e-9; - if (!completeFields || !joined || !scoreJoined || reviewedPacketIds.size === 0 || treatment === undefined || treatment.pressure.reviewedAtoms <= 0) { - failures.push(failure("production_model_accounting", `${label} production model calls are incomplete or do not join exactly to score/reviewed packets/atoms`, { - repeat: execution.repeat, - modelCalls: execution.modelCalls.length, - scoreModelCalls: scoreCount ?? -1, - reviewedPackets: reviewedPacketIds.size, - reviewedAtoms: treatment?.pressure.reviewedAtoms ?? 0 - })); - } - } -} - -function analyzeProductionThroughput( - baselineRun: EvalCaseRunInput, - baselineTreatment: TreatmentExecution[], - selectedRun: EvalCaseRunInput, - selectedTreatment: TreatmentExecution[], - failures: ReportFailure[] -): ProductionThroughput { - const baseline = productionArmThroughput(baselineRun, baselineTreatment, failures, "baseline"); - const selected = productionArmThroughput(selectedRun, selectedTreatment, failures, "selected"); - if (stableJson(baseline.reviewedHunkIds) !== stableJson(selected.reviewedHunkIds)) { - failures.push(failure("production_hunk_loss", "selected production arm does not preserve the exact reviewed-hunk set from baseline", { - baselineCount: baseline.reviewedHunks, - selectedCount: selected.reviewedHunks, - baselineSetHash: sha256Hex(baseline.reviewedHunkIds.join("\n")), - selectedSetHash: sha256Hex(selected.reviewedHunkIds.join("\n")) - })); - } - if (baseline.reviewedHunks !== EQUIVALENT_TARGET_HUNKS || selected.reviewedHunks !== EQUIVALENT_TARGET_HUNKS) { - failures.push(failure("production_incomplete", `production comparison must review exactly ${EQUIVALENT_TARGET_HUNKS} hunks in both arms`, { - baseline: baseline.reviewedHunks, - selected: selected.reviewedHunks - })); - } - if (!(selected.reviewedHunksPerWallSecond > baseline.reviewedHunksPerWallSecond)) { - failures.push(failure("production_throughput", "selected production arm does not improve reviewed hunks per wall second")); - } - if (!(selected.wallTimeSeconds <= baseline.wallTimeSeconds)) { - failures.push(failure("production_wall_time", "selected production arm increases total wall time")); - } - if (!(selected.modelServiceSecondsPerReviewedHunk < baseline.modelServiceSecondsPerReviewedHunk)) { - failures.push(failure("production_model_service", "selected production arm does not strictly improve model-service seconds per reviewed hunk")); - } - if (!(selected.tokensPerReviewedHunk < baseline.tokensPerReviewedHunk)) { - failures.push(failure("production_tokens", "selected production arm does not strictly improve tokens per reviewed hunk")); - } - if (!(selected.reasoningTokensPerReviewedHunk <= baseline.reasoningTokensPerReviewedHunk)) { - failures.push(failure("production_reasoning_tokens", "selected production arm increases reasoning tokens per reviewed hunk")); - } - if (selected.continuations > baseline.continuations) { - const retention = productionPacketCountServiceSavingsRetention(baseline, selected); - if (retention < 0.85) { - failures.push(failure("production_continuation_savings_retention", "selected production arm adds continuations without retaining at least 85% of packet-count-implied all-stage model-service savings", { - retention: round(retention, 6), - baselineContinuations: baseline.continuations, - selectedContinuations: selected.continuations - })); - } - } - if (selected.pressure.rejectionRatePerAtom - baseline.pressure.rejectionRatePerAtom > 0.1) { - failures.push(failure("production_tool_pressure", "selected production arm increases normalized tool rejection pressure by more than 0.10")); - } - return { baseline, selected }; -} - -function productionPacketCountServiceSavingsRetention( - baseline: ProductionArmThroughput, - selected: ProductionArmThroughput -): number { - const packetCountSavings = baseline.reviewedPackets > 0 - ? baseline.modelServiceSeconds * Math.max(0, baseline.reviewedPackets - selected.reviewedPackets) / baseline.reviewedPackets - : 0; - const actualSavings = baseline.modelServiceSeconds - selected.modelServiceSeconds; - return packetCountSavings > 0 ? actualSavings / packetCountSavings : 0; -} - -function productionArmThroughput( - run: EvalCaseRunInput, - treatments: TreatmentExecution[], - failures: ReportFailure[], - label: string -): ProductionArmThroughput { - const rawReviewedHunkIds = run.executions.flatMap((execution) => execution.reviewedHunkIds); - if (unique(rawReviewedHunkIds).length !== rawReviewedHunkIds.length) { - failures.push(failure("production_hunk_bijection", `${label} production arm repeats reviewed hunk IDs`)); - } - const reviewedHunkIds = sortedUnique(rawReviewedHunkIds); - const pressure = combinePressure(treatments.map((entry) => entry.pressure)); - const allModelCalls = run.executions.flatMap((execution) => execution.modelCalls); - const reviewedHunks = reviewedHunkIds.length; - const wallTimeSeconds = round(sum(run.executions.map((execution) => execution.wallTimeSeconds)), 3); - if (!(wallTimeSeconds > 0)) { - failures.push(failure("missing_wall_time", `${label} production arm lacks positive wall-time evidence`)); - } - const modelServiceSeconds = round(sum(allModelCalls.map((call) => call.durationMs)) / 1000, 3); - const totalTokens = sum(allModelCalls.map((call) => call.totalTokens ?? 0)); - const reasoningTokens = sum(allModelCalls.map((call) => call.reasoningTokens ?? 0)); - return { - reviewedHunkIds, - reviewedHunks, - reviewedAtoms: pressure.reviewedAtoms, - reviewedPackets: pressure.reviewedPackets, - wallTimeSeconds, - reviewedHunksPerWallSecond: wallTimeSeconds > 0 ? round(reviewedHunks / wallTimeSeconds) : 0, - modelServiceSeconds, - modelServiceSecondsPerReviewedHunk: reviewedHunks > 0 ? round(modelServiceSeconds / reviewedHunks) : 0, - totalTokens, - tokensPerReviewedHunk: reviewedHunks > 0 ? round(totalTokens / reviewedHunks) : 0, - reasoningTokens, - reasoningTokensPerReviewedHunk: reviewedHunks > 0 ? round(reasoningTokens / reviewedHunks) : 0, - continuations: pressure.continuations, - continuationsPerReviewedAtom: pressure.reviewedAtoms > 0 ? round(pressure.continuations / pressure.reviewedAtoms) : 0, - pressure - }; -} - -export function computeProductionEconomics( - baseline: { actualCostUSD: number; reviewedHunks: number }, - selected: { actualCostUSD: number; reviewedHunks: number }, - validationCostInputUSD: number, - label: ProductionEconomics["validationCostInputLabel"] = "explicit_cumulative_validation_cost", - failures: ReportFailure[] = [] -): ProductionEconomics { - if (!(baseline.actualCostUSD >= 0) || !(selected.actualCostUSD >= 0)) { - failures.push(failure("missing_spend_data", "production arm costs must be finite non-negative values")); - } - if (!(baseline.reviewedHunks > 0) || !(selected.reviewedHunks > 0)) { - failures.push(failure("missing_reviewed_hunks", "production arms must report positive reviewed-hunk counts")); - } - const baselinePerHunk = baseline.reviewedHunks > 0 ? baseline.actualCostUSD / baseline.reviewedHunks : Number.POSITIVE_INFINITY; - const selectedPerHunk = selected.reviewedHunks > 0 ? selected.actualCostUSD / selected.reviewedHunks : Number.POSITIVE_INFINITY; - const baselineEquivalent = baselinePerHunk * EQUIVALENT_TARGET_HUNKS; - const selectedEquivalent = selectedPerHunk * EQUIVALENT_TARGET_HUNKS; - const savings = baselineEquivalent - selectedEquivalent; - if (!(savings > 0) || !Number.isFinite(savings)) { - failures.push(failure("non_positive_payback_denominator", "normalized equivalent-review savings must be positive", { - baselineEquivalent, - selectedEquivalent, - savings - })); - } - const breakEven = savings > 0 && Number.isFinite(savings) ? Math.ceil(validationCostInputUSD / savings) : 0; - return { - equivalentTargetHunks: EQUIVALENT_TARGET_HUNKS, - baseline: { - actualCostUSD: round(baseline.actualCostUSD), - reviewedHunks: baseline.reviewedHunks, - costPerReviewedHunkUSD: round(baselinePerHunk), - equivalentReviewCostUSD: round(baselineEquivalent), - equivalentCostExtrapolated: baseline.reviewedHunks < EQUIVALENT_TARGET_HUNKS - }, - selected: { - actualCostUSD: round(selected.actualCostUSD), - reviewedHunks: selected.reviewedHunks, - costPerReviewedHunkUSD: round(selectedPerHunk), - equivalentReviewCostUSD: round(selectedEquivalent), - equivalentCostExtrapolated: selected.reviewedHunks < EQUIVALENT_TARGET_HUNKS - }, - equivalentReviewSavingsUSD: round(savings), - validationCostInputUSD: round(validationCostInputUSD), - validationCostInputLabel: label, - breakEvenReviewCount: breakEven - }; -} - -export function analyzeRegressionCohorts( - baselineCohort: CohortSelection, - selectedCohort: CohortSelection, - expectedRepeats: number -): RegressionReport { - const failures: ReportFailure[] = []; - const baselineByName = new Map(baselineCohort.runs.map((run) => [run.info.caseName, run])); - const selectedByName = new Map(selectedCohort.runs.map((run) => [run.info.caseName, run])); - const allNames = sortedUnique([...baselineByName.keys(), ...selectedByName.keys()]); - const cases: RegressionReport["cases"] = []; - for (const caseName of allNames) { - const baseline = baselineByName.get(caseName); - const selected = selectedByName.get(caseName); - if (baseline === undefined || selected === undefined) { - failures.push(failure("regression_case_join", `case ${caseName} is missing from one regression cohort`)); - continue; - } - validateRunRepeatCount(baseline, expectedRepeats, failures); - validateRunRepeatCount(selected, expectedRepeats, failures); - const baselineEvidenceScores = validateRunEvidence(baseline, `${caseName}/baseline`, failures); - const selectedEvidenceScores = validateRunEvidence(selected, `${caseName}/selected`, failures); - enforceRequiredRegressionOutcomes(baseline, baselineEvidenceScores, `${caseName}/baseline`, failures); - enforceRequiredRegressionOutcomes(selected, selectedEvidenceScores, `${caseName}/selected`, failures); - validateRunScoreExpectations(`${caseName}/baseline`, baseline, failures); - validateRunScoreExpectations(`${caseName}/selected`, selected, failures); - let baselineArm: ExperimentArm | undefined; - let selectedArm: ExperimentArm | undefined; - try { - baselineArm = armForRun(baseline); - selectedArm = armForRun(selected); - } catch (error) { - if (error instanceof PacketPackingReportError) { - failures.push(...error.failures); - } else { - throw error; - } - } - if (baselineArm !== "A" || selectedArm === "A") { - failures.push(failure("regression_arm_config", `${caseName} must compare packing-off/base to one selected packed arm`)); - } - if (stableJson(normalizeExperimentSnapshot(baseline.declaredCase)) !== stableJson(normalizeExperimentSnapshot(selected.declaredCase))) { - failures.push(failure("regression_yaml_delta", `${caseName} differs outside repeat and packet-packing review settings`)); - } - if (baseline.info.score.status === "error" || selected.info.score.status === "error") { - failures.push(failure("eval_error", `${caseName} has an errored collateral eval`)); - } - const transitions = expectationTransitions(baseline.info.score, selected.info.score); - for (const transition of transitions) { - if (transition.from === "pass" && transition.to !== "pass") { - failures.push(failure("collateral_expectation_regression", `${caseName}/${transition.expectationId} regresses from pass to ${transition.to}`)); - } - } - const baselineTreatment = baseline.executions.map((execution) => analyzeTreatmentExecution(baseline, execution, "A")); - const effectiveSelectedArm = selectedArm === "C" ? "C" : "B"; - const selectedTreatment = selected.executions.map((execution) => analyzeTreatmentExecution( - selected, - execution, - effectiveSelectedArm, - baseline.executions.find((candidate) => candidate.repeat === execution.repeat), - false - )); - failures.push(...baselineTreatment.flatMap((entry) => entry.failures), ...selectedTreatment.flatMap((entry) => entry.failures)); - cases.push({ - caseName, - baselineRun: baseline.runNumber, - selectedRun: selected.runNumber, - expectationTransitions: transitions, - selectedTreatmentExecutions: selectedTreatment.filter((entry) => entry.treated).length, - dispatchOrderChanges: dispatchOrderChanges(baseline.executions, selected.executions), - baselinePressure: combinePressure(baselineTreatment.map((entry) => entry.pressure)), - selectedPressure: combinePressure(selectedTreatment.map((entry) => entry.pressure)) - }); - } - const baselineCost = aggregateCohortCost(baselineCohort.runs, failures); - const selectedCost = aggregateCohortCost(selectedCohort.runs, failures); - return { - schemaVersion: 1, - mode: "regression", - evidence: "one-repeat-collateral-only", - baselineCohort: { id: baselineCohort.id, runNumbers: baselineCohort.runs.map((run) => run.runNumber), actualCostUSD: baselineCost }, - selectedCohort: { id: selectedCohort.id, runNumbers: selectedCohort.runs.map((run) => run.runNumber), actualCostUSD: selectedCost }, - expectedRepeats, - cases, - failures - }; -} - -function validateRunScoreExpectations(label: string, run: EvalCaseRunInput, failures: ReportFailure[]): void { - const declared = declaredExpectationKeys(run.declaredCase); - const scores = [run.info.score, ...run.executions.map((execution) => execution.score)]; - for (const [index, score] of scores.entries()) { - const actual = expectationKeys(score); - if (declared.duplicates.length > 0 || actual.duplicates.length > 0 || stableJson(actual.values) !== stableJson(declared.values)) { - failures.push(failure("expectation_join", `${label}/score ${index} does not match the exact declared expectation set`)); - } - } -} - -function enforceRequiredRegressionOutcomes( - run: EvalCaseRunInput, - scores: EvalScore[], - label: string, - failures: ReportFailure[] -): void { - const required = new Set(([ - ...((run.declaredCase.should_find ?? []).map((expectation) => ({ list: "should_find" as const, expectation }))), - ...((run.declaredCase.should_find_candidate ?? []).map((expectation) => ({ list: "should_find_candidate" as const, expectation }))), - ...((run.declaredCase.should_not_find ?? []).map((expectation) => ({ list: "should_not_find" as const, expectation }))) - ]).filter((entry) => entry.expectation.tier !== "optional").map((entry) => `${entry.list}\0${entry.expectation.id}`)); - for (const [index, score] of scores.entries()) { - for (const result of score.expectationResults) { - if (!required.has(`${result.list}\0${result.expectationId}`)) { - continue; - } - const passes = result.status === "pass" && (result.list === "should_not_find" || result.matched.length > 0); - if (!passes) { - failures.push(failure("collateral_cohort_expectation_failure", `${label}/repeat ${run.executions[index]?.repeat ?? index + 1} fails a required ${result.list} expectation`, { - expectationId: result.expectationId, - list: result.list - })); - } - } - } -} - -function expectationTransitions(baseline: EvalScore, selected: EvalScore): RegressionReport["cases"][number]["expectationTransitions"] { - const baselineByKey = new Map(baseline.expectationResults.map((result) => [`${result.list}\0${result.expectationId}`, result])); - const selectedByKey = new Map(selected.expectationResults.map((result) => [`${result.list}\0${result.expectationId}`, result])); - return sortedUnique([...baselineByKey.keys(), ...selectedByKey.keys()]).flatMap((key) => { - const before = baselineByKey.get(key); - const after = selectedByKey.get(key); - const [list, expectationId] = key.split("\0"); - const from = before?.status ?? "missing"; - const to = after?.status ?? "missing"; - return from === to ? [] : [{ expectationId: expectationId ?? "unknown", list: list ?? "unknown", from, to }]; - }); -} - -function dispatchOrderChanges(baseline: EvalExecutionInput[], selected: EvalExecutionInput[]): number { - let changes = 0; - for (const baselineExecution of baseline) { - const selectedExecution = selected.find((entry) => entry.repeat === baselineExecution.repeat); - if (selectedExecution === undefined) { - continue; - } - const baselineOrder = schedulingOrder(baselineExecution.packets); - const selectedOrder = schedulingOrder(selectedExecution.packets); - const baselineByHunk = new Map(baselineOrder.flatMap((packet, index) => packet.hunks.map((hunk) => [hunk.hunkId, index] as const))); - const selectedByHunk = new Map(selectedOrder.flatMap((packet, index) => packet.hunks.map((hunk) => [hunk.hunkId, index] as const))); - changes += [...baselineByHunk].filter(([hunkId, index]) => selectedByHunk.get(hunkId) !== index).length; - } - return changes; -} - -type ReplayReport = { - schemaVersion: 1; - mode: "replay"; - noModelCalls: boolean; - repo: string; - rows: ReplayRow[]; - failures: ReportFailure[]; -}; - -async function loadJsonValue(filePath: string, description: string): Promise { - let raw: string; - try { - raw = await readFile(filePath, "utf8"); - } catch (error) { - fail("missing_artifact", `failed to read ${description}: ${filePath}`, { error: error instanceof Error ? error.message : String(error) }); - } - try { - return JSON.parse(raw) as unknown; - } catch (error) { - fail("corrupt_artifact", `failed to parse ${description}: ${filePath}`, { error: error instanceof Error ? error.message : String(error) }); - } -} - -async function loadValidatedJson(filePath: string, description: string, schema: S): Promise> { - const value = await loadJsonValue(filePath, description); - const parsed = schema.safeParse(value); - assertReport(parsed.success, "corrupt_artifact_schema", `invalid ${description}: ${filePath}`, { - issues: parsed.success ? [] : parsed.error.issues.map((issue) => ({ path: issue.path.join("."), code: issue.code, message: issue.message })) - }); - return parsed.data; -} - -async function loadValidatedJsonLines(filePath: string, description: string, schema: S): Promise>> { - if (!existsSync(filePath)) { - fail("missing_artifact", `failed to read ${description}: ${filePath}`); - } - const raw = await readFile(filePath, "utf8"); - const result: Array> = []; - for (const [index, line] of raw.split("\n").entries()) { - if (line.trim().length === 0) { - continue; - } - let value: unknown; - try { - value = JSON.parse(line) as unknown; - } catch (error) { - fail("corrupt_artifact", `failed to parse ${description} line ${index + 1}: ${filePath}`, { - error: error instanceof Error ? error.message : String(error) - }); - } - const parsed = schema.safeParse(value); - assertReport(parsed.success, "corrupt_artifact_schema", `invalid ${description} line ${index + 1}: ${filePath}`, { - issues: parsed.success ? [] : parsed.error.issues.map((issue) => ({ path: issue.path.join("."), code: issue.code, message: issue.message })) - }); - result.push(parsed.data); - } - return result; -} - -function scoringHintEvents(events: TelemetryEvent[]): EvalArtifacts["hintEvents"] { - return events.flatMap((event) => { - if (event.message !== "follow_up_hint" && event.message !== "uncertainty") { - return []; - } - const data = event.data ?? {}; - const question = typeof data.question === "string" ? data.question : ""; - if (question.trim().length === 0) { - return []; - } - const strings = (value: unknown): string[] => Array.isArray(value) ? value.filter((entry): entry is string => typeof entry === "string") : []; - const confidence = data.confidence === "high" || data.confidence === "medium" || data.confidence === "low" ? data.confidence : "low"; - const packetId = event.packetId ?? (typeof data.packetId === "string" ? data.packetId : undefined); - return [{ - ...(packetId === undefined ? {} : { packetId }), - question, - files: strings(data.files), - symbols: strings(data.symbols), - projectedSkillIds: strings(data.projectedSkillIds), - ...(typeof data.reason === "string" ? { reason: data.reason } : {}), - confidence - }]; - }); -} - -function scoringHumanAttentionNotes(raw: unknown): NonNullable { - const entries = Array.isArray(raw) - ? raw - : isRecord(raw) && Array.isArray(raw.groups) - ? raw.groups - : []; - return entries.flatMap((entry) => { - if (!isRecord(entry) || typeof entry.question !== "string" || entry.question.length === 0) { - return []; - } - const files = Array.isArray(entry.files) ? entry.files.filter((value): value is string => typeof value === "string") : []; - const reasons = Array.isArray(entry.reasons) - ? entry.reasons.filter((value): value is string => typeof value === "string") - : typeof entry.reason === "string" - ? [entry.reason] - : []; - return [{ question: entry.question, files, reasons }]; - }); -} - -async function loadPackets(packetDir: string, options: { allowLegacyDispatchRank?: boolean } = {}): Promise { - let entries: string[]; - try { - entries = (await readdir(packetDir)).filter((entry) => entry.endsWith(".json")).sort(); - } catch (error) { - fail("missing_artifact", `failed to read packet artifacts: ${packetDir}`, { error: error instanceof Error ? error.message : String(error) }); - } - assertReport(entries.length > 0, "missing_artifact", `no packet artifacts found in ${packetDir}`); - const parsedPackets = await Promise.all(entries.map((entry) => loadValidatedJson(path.join(packetDir, entry), "packet artifact", reviewPacketSchema))); - const packetIds = new Set(); - const hunkIds = new Set(); - for (const [index, packet] of parsedPackets.entries()) { - assertReport(entries[index] === `${packet.id}.json`, "packet_filename_mismatch", `packet filename does not match packet ID in ${packetDir}`); - assertReport(!packetIds.has(packet.id), "duplicate_packet_id", `duplicate packet ID ${packet.id} in ${packetDir}`); - packetIds.add(packet.id); - assertReport(options.allowLegacyDispatchRank === true || packet.dispatchRank !== undefined, "missing_dispatch_rank", `packet ${packet.id} has no dispatch rank`); - for (const hunk of packet.hunks) { - assertReport(!hunkIds.has(hunk.hunkId), "duplicate_packet_hunk", `hunk ${hunk.hunkId} appears in multiple packet artifacts`); - hunkIds.add(hunk.hunkId); - } - } - return parsedPackets as unknown as ReviewPacket[]; -} - -export async function loadEvalRuns(logsDir: string, selector?: string): Promise { - let locations: Array<{ logsRoot: string; runPath: string; runNumber: number }>; - try { - await readdir(logsDir); - } catch (error) { - fail("missing_logs", `failed to read eval log root: ${logsDir}`, { error: error instanceof Error ? error.message : String(error) }); - } - const selectedManifest = selector === undefined ? undefined : await loadSelectedInvocationManifest(logsDir, selector); - if (selectedManifest !== undefined) { - locations = selectedManifest.runs.map((run) => ({ logsRoot: path.resolve(run.logsRoot), runPath: run.runPath, runNumber: run.runNumber })); - for (const ownedRoot of sortedUnique(locations.map((location) => location.logsRoot))) { - const ownedManifest = await loadValidatedJson( - path.join(ownedRoot, "invocations", `${selectedManifest.invocationId}.json`), - "root-owned eval invocation manifest", - evalInvocationManifestSchema - ) as EvalInvocationManifest; - assertReport( - stableJson(ownedManifest) === stableJson(selectedManifest), - "invocation_root_ownership", - `invocation ${selectedManifest.invocationId} differs across its declared log roots` - ); - } - } else { - try { - locations = (await readdir(logsDir)).filter((entry) => /^\d+$/u.test(entry)).sort((a, b) => Number(a) - Number(b)) - .map((runPath) => ({ logsRoot: path.resolve(logsDir), runPath, runNumber: Number(runPath) })); - } catch (error) { - fail("missing_logs", `failed to read eval log root: ${logsDir}`, { error: error instanceof Error ? error.message : String(error) }); - } - } - const runs: EvalCaseRunInput[] = []; - const manifests = new Map(); - if (selectedManifest !== undefined) { - manifests.set(selectedManifest.invocationId, selectedManifest); - } - const declarations = new Map(); - for (const location of locations) { - const runDir = path.resolve(location.logsRoot, location.runPath); - assertReport( - runDir === path.join(location.logsRoot, String(location.runNumber)) && path.dirname(runDir) === location.logsRoot, - "invocation_root_ownership", - "eval invocation run path does not resolve to its exact declared owning root" - ); - const info = await loadValidatedJson(path.join(runDir, "info.json"), "eval info", evalRunInfoSchema) as unknown as EvalRunInfo; - assertReport(info.runNumber === location.runNumber, "corrupt_eval_info", `eval info run number does not match directory ${runDir}`); - assertReport(info.caseSnapshot.name === info.caseName, "corrupt_eval_info", `eval case snapshot name does not match case name in ${runDir}`); - let invocationManifest: EvalInvocationManifest | undefined; - let declaredCase = info.caseSnapshot; - if (info.invocation !== undefined) { - assertReport(info.invocation.manifest === `invocations/${info.invocation.id}.json`, "invocation_manifest_join", `eval invocation manifest path is not canonical in ${runDir}`); - invocationManifest = manifests.get(info.invocation.id); - if (invocationManifest === undefined) { - invocationManifest = await loadValidatedJson( - path.join(location.logsRoot, info.invocation.manifest), - "eval invocation manifest", - evalInvocationManifestSchema - ) as EvalInvocationManifest; - manifests.set(info.invocation.id, invocationManifest); - } - validateInvocationManifestRun(invocationManifest, info, runDir); - const declarationKey = `${invocationManifest.invocationId}\0${info.invocation.caseIndex}`; - declaredCase = declarations.get(declarationKey) ?? await loadDeclaredEvalCase(invocationManifest, info); - declarations.set(declarationKey, declaredCase); - } - const repeat = info.caseSnapshot.repeat ?? 1; - const executions: EvalExecutionInput[] = []; - for (let index = 1; index <= repeat; index += 1) { - const executionDir = repeat > 1 ? path.join(runDir, "repeats", String(index)) : runDir; - const telemetryDir = path.join(executionDir, "telemetry"); - const score = repeat > 1 - ? await loadValidatedJson(path.join(executionDir, "score.json"), "repeat score", evalScoreSchema) as unknown as EvalScore - : info.score; - const packets = await loadPackets(path.join(telemetryDir, "stages", "06-packets", "packets")); - const events = await loadValidatedJsonLines(path.join(telemetryDir, "events.jsonl"), "telemetry events", telemetryEventSchema) as TelemetryEvent[]; - const modelCalls = await loadValidatedJsonLines(path.join(telemetryDir, "model-calls.jsonl"), "model calls", llmCallRecordSchema) as LlmCallRecord[]; - const toolCalls = await loadValidatedJsonLines(path.join(telemetryDir, "tool-calls.jsonl"), "tool calls", toolCallRecordSchema) as ToolCallRecord[]; - const fileFacts = await loadValidatedJson(path.join(telemetryDir, "stages", "03-classify", "file-facts.json"), "eval file facts", z.array(fileFactsSchema)) as FileFacts[]; - const diff = await loadValidatedJson(path.join(telemetryDir, "stages", "02-diff", "diff.json"), "eval diff", unifiedDiffSchema) as unknown as UnifiedDiff; - const plan = await loadValidatedJson(path.join(telemetryDir, "stages", "05-planner", "review-plan.json"), "eval review plan", reviewPlanSchema) as unknown as ReviewPlan; - const candidates = await loadValidatedJson(path.join(telemetryDir, "stages", "09-verification", "candidate-findings.json"), "candidate findings", z.array(candidateFindingSchema)); - const verification = await loadValidatedJson(path.join(telemetryDir, "stages", "09-verification", "verification.json"), "verification records", z.array(verificationRecordSchema)); - const finalSelectionArtifact = await loadValidatedJson( - path.join(telemetryDir, "stages", "10-composition", "final-selection.json"), - "final selection", - finalSelectionArtifactSchema - ); - const finalFindings = await loadValidatedJson(path.join(telemetryDir, "stages", "10-composition", "final-findings.json"), "final findings", z.array(finalFindingSchema)); - const coverage = await loadValidatedJson(path.join(telemetryDir, "stages", "10-composition", "coverage.json"), "coverage", coverageArtifactSchema); - const attention = await loadValidatedJson(path.join(telemetryDir, "stages", "10-composition", "attention.json"), "attention records", z.array(attentionRecordSchema)); - const humanAttentionRaw = await loadValidatedJson( - path.join(telemetryDir, "stages", "10-composition", "human-attention-notes.json"), - "human attention notes", - humanAttentionArtifactSchema - ); - const budgetSummary = await loadValidatedJson(path.join(telemetryDir, "stages", "10-composition", "budget-summary.json"), "budget summary", budgetSummarySchema); - const costProfile = await loadValidatedJson(path.join(telemetryDir, "stages", "00-run", "cost-profile.json"), "cost profile", costProfileSchema); - const modelCallsSummary = await loadValidatedJson(path.join(telemetryDir, "stages", "00-run", "model-calls-summary.json"), "model calls summary", modelCallsSummarySchema); - const toolCallsSummary = await loadValidatedJson(path.join(telemetryDir, "stages", "00-run", "tool-calls-summary.json"), "tool calls summary", toolCallsSummarySchema); - const runJson = await loadValidatedJson(path.join(telemetryDir, "run.json"), "run telemetry", evalRunTelemetrySchema); - const telemetrySummary = await loadValidatedJson(path.join(telemetryDir, "telemetry.json"), "telemetry summary", telemetrySummarySchema); - const reviewedHunkIds = coverage.records.filter((record) => record.status === "reviewed").map((record) => record.hunkId).sort(); - assertReport(reviewedHunkIds.length === coverage.status.reviewedHunks, "coverage_join_mismatch", `reviewed-hunk records disagree with coverage status in ${executionDir}`); - const normalizedFinalFindings = finalFindings.map((finding) => ({ - ...finding, - mergedAnchors: (finding.mergedAnchors ?? []).filter((anchor): anchor is Exclude => typeof anchor !== "string") - })) as FinalFinding[]; - const scoringArtifacts: EvalArtifacts = { - candidates: candidates as CandidateFinding[], - verification: verification as unknown as EvalVerificationRecord[], - finalSelection: finalSelectionArtifact.records as EvalSelectionRecord[], - finalFindings: normalizedFinalFindings, - humanAttentionNotes: scoringHumanAttentionNotes(humanAttentionRaw), - attention: attention as unknown as NonNullable, - packets, - hintEvents: scoringHintEvents(events), - coverage: { ...(coverage.status as unknown as RunCoverageStatus), hunks: coverage.records }, - reviewPlan: plan, - metricsSources: { - costProfile, - modelCallsSummary, - toolCallsSummary, - budgetSummary: budgetSummary as unknown as NonNullable, - runJson, - telemetry: telemetrySummary, - modelCalls, - toolCalls - } - }; - const execution: EvalExecutionInput = { - repeat: index, - score, - telemetryDir, - packets, - events, - modelCalls, - toolCalls, - fileFacts, - diff, - plan, - candidateFindings: candidates as CandidateFinding[], - verification: verification as unknown as EvalVerificationRecord[], - finalSelection: finalSelectionArtifact.records as EvalSelectionRecord[], - finalSelectionArtifact, - finalFindings: normalizedFinalFindings, - scoringArtifacts, - summaryArtifacts: { - attention, - humanAttention: humanAttentionRaw, - budget: budgetSummary, - cost: costProfile, - model: modelCallsSummary, - tool: toolCallsSummary, - run: runJson, - telemetry: telemetrySummary - }, - reviewedHunkIds, - wallTimeSeconds: runJson.durationMs / 1000 - }; - validateEvalExecutionArtifacts(execution, coverage.records, info.reviewRunId); - executions.push(execution); - } - runs.push({ runNumber: info.runNumber, runDir, info, declaredCase, ...(invocationManifest === undefined ? {} : { invocationManifest }), executions }); - } - return runs; -} - -async function loadSelectedInvocationManifest(logsDir: string, selector: string): Promise { - const invocationDir = path.join(logsDir, "invocations"); - let files: string[]; - try { - files = (await readdir(invocationDir)).filter((entry) => entry.endsWith(".json")).sort(); - } catch (error) { - fail("missing_invocation_manifest", `failed to read eval invocation manifests: ${invocationDir}`, { - error: valueFingerprint(error instanceof Error ? error.message : String(error)) - }); - } - const manifests = await Promise.all(files.map((file) => - loadValidatedJson(path.join(invocationDir, file), "eval invocation manifest", evalInvocationManifestSchema) as Promise - )); - assertReport(manifests.length > 0, "missing_invocation_manifest", `no eval invocation manifests found in ${invocationDir}`); - const ordered = [...manifests].sort((left, right) => - left.startedAt.localeCompare(right.startedAt) || manifestLastRun(left) - manifestLastRun(right) - ); - let selected: EvalInvocationManifest | undefined; - if (manifests.some((manifest) => manifest.invocationId === selector)) { - selected = manifests.find((manifest) => manifest.invocationId === selector); - } else if (selector === "latest") { - selected = ordered.at(-1); - } else if (/^\d+$/u.test(selector)) { - const matches = manifests.filter((manifest) => manifestLastRun(manifest) === Number(selector)); - assertReport(matches.length <= 1, "ambiguous_cohort_selector", `run ${selector} matches multiple persisted invocations; use the invocation UUID`); - selected = matches[0]; - } else { - const match = /^(\d+)-(\d+)$/u.exec(selector); - assertReport(match !== null, "invalid_cohort_selector", `cohort must be latest, an invocation UUID, an ending run number, or a min-max range: ${selector}`); - const matches = manifests.filter((manifest) => manifest.runs[0]?.runNumber === Number(match[1]) && manifestLastRun(manifest) === Number(match[2])); - assertReport(matches.length <= 1, "ambiguous_cohort_selector", `cohort ${selector} matches multiple persisted invocations; use the invocation UUID`); - selected = matches[0]; - } - assertReport(selected !== undefined, "unknown_cohort", `cohort ${selector} is not an exact persisted invocation boundary`); - validateCompleteManifest(selected); - return selected; -} - -function manifestLastRun(manifest: EvalInvocationManifest): number { - return manifest.runs.at(-1)?.runNumber ?? -1; -} - -function validateCompleteManifest(manifest: EvalInvocationManifest): void { - const expectedIndexes = Array.from({ length: manifest.cases.length }, (_, index) => index); - assertReport( - manifest.status === "complete" && manifest.completedAt !== undefined && - stableJson(manifest.cases.map((entry) => entry.caseIndex)) === stableJson(expectedIndexes) && - stableJson(manifest.runs.map((entry) => entry.caseIndex)) === stableJson(expectedIndexes) && - manifest.runs.length === manifest.cases.length && - unique(manifest.runs.map((entry) => `${path.resolve(entry.logsRoot)}\0${entry.runPath}`)).length === manifest.runs.length && - manifest.runs.every((entry) => entry.runPath === String(entry.runNumber)), - "partial_latest_cohort", - "selected persisted eval invocation is incomplete or has an invalid exact ordered case boundary" - ); -} - -async function loadDeclaredEvalCase(manifest: EvalInvocationManifest, info: EvalRunInfo): Promise { - const reference = info.invocation; - assertReport(reference !== undefined, "declared_case_join", `eval run ${info.runNumber} has no invocation reference`); - const manifestCase = manifest.cases.find((entry) => entry.caseIndex === reference.caseIndex); - assertReport(manifestCase !== undefined, "declared_case_join", `eval run ${info.runNumber} has no declared manifest case`); - const suiteDir = path.resolve(manifest.suiteDir); - const casePath = path.resolve(suiteDir, manifestCase.caseFile); - const relative = path.relative(suiteDir, casePath); - assertReport(relative.length > 0 && !relative.startsWith("..") && !path.isAbsolute(relative), "declared_case_join", `eval run ${info.runNumber} case YAML escapes its declared suite`); - let declaration: Awaited>; - try { - declaration = await loadEvalCaseDeclaration(casePath, suiteDir); - } catch (error) { - fail("declared_case_join", `failed to load the exact declared eval YAML for run ${info.runNumber}`, { - error: valueFingerprint(error instanceof Error ? error.message : String(error)) - }); - } - assertReport( - declaration.file === manifestCase.caseFile && declaration.evalCase.name === manifestCase.caseName && - declaration.caseHash === manifestCase.caseHash && declaration.caseHash === info.caseHash && - stableJson(declaration.evalCase) === stableJson(info.caseSnapshot), - "declared_case_join", - `eval run ${info.runNumber} manifest, YAML, case hash, and snapshot do not join exactly`, - { - declaredHash: declaration.caseHash, - manifestHash: manifestCase.caseHash, - infoHash: info.caseHash, - declaredSnapshotHash: sha256Hex(stableJson(declaration.evalCase)), - persistedSnapshotHash: sha256Hex(stableJson(info.caseSnapshot)) - } - ); - return declaration.evalCase; -} - -function validateInvocationManifestRun(manifest: EvalInvocationManifest, info: EvalRunInfo, runDir?: string): void { - const reference = info.invocation; - assertReport(reference !== undefined && manifest.invocationId === reference.id, "invocation_manifest_join", `eval run ${info.runNumber} invocation identity disagrees with its manifest`); - const declared = manifest.cases.find((entry) => entry.caseIndex === reference.caseIndex); - const recorded = manifest.runs.find((entry) => entry.caseIndex === reference.caseIndex); - assertReport( - declared !== undefined && recorded !== undefined && - declared.caseName === info.caseName && declared.caseHash === info.caseHash && declared.caseFile === info.caseFile && - recorded.caseName === info.caseName && recorded.caseHash === info.caseHash && recorded.runNumber === info.runNumber, - "invocation_manifest_join", - `eval run ${info.runNumber} does not exactly match its invocation manifest entry` - ); - if (runDir !== undefined && recorded !== undefined) { - assertReport( - path.resolve(runDir) === path.resolve(recorded.logsRoot, recorded.runPath) && recorded.runPath === String(recorded.runNumber), - "invocation_root_ownership", - `eval run ${info.runNumber} was not loaded from its exact manifest-owned root and path` - ); - } -} - -function validateEvalExecutionArtifacts( - execution: EvalExecutionInput, - coverageRecords: Array["records"][number]>, - expectedRunId: string | undefined -): void { - const packetIds = new Set(execution.packets.map((packet) => packet.id)); - const packetHunkIds = new Set(execution.packets.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId))); - const diffHunks = execution.diff.files.flatMap((file) => file.hunks); - const diffHunkById = new Map(diffHunks.map((hunk) => [hunk.id, hunk])); - assertReport(diffHunkById.size === diffHunks.length, "duplicate_diff_hunk", "eval diff contains duplicate hunk IDs"); - assertReport(execution.packets.every((packet) => packet.hunks.every((hunk) => diffHunkById.get(hunk.hunkId)?.path === packet.path)), "packet_diff_join", "eval packet hunks do not join exactly to the recorded diff"); - const planHunkIds = execution.plan.coverage.map((decision) => decision.hunkId); - assertReport(unique(planHunkIds).length === planHunkIds.length, "plan_diff_join", "eval review plan contains duplicate hunk IDs"); - assertReport(execution.plan.coverage.every((decision) => diffHunkById.get(decision.hunkId)?.path === decision.path), "plan_diff_join", "eval review plan decisions do not join to the recorded diff"); - assertReport(unique(execution.candidateFindings.map((finding) => finding.id)).length === execution.candidateFindings.length, "duplicate_candidate_finding", "candidate findings contain duplicate IDs"); - assertReport(unique(execution.finalFindings.map((finding) => finding.id)).length === execution.finalFindings.length, "duplicate_final_finding", "final findings contain duplicate IDs"); - const factPaths = new Set(execution.fileFacts.map((facts) => facts.path)); - assertReport(factPaths.size === execution.fileFacts.length, "duplicate_file_facts", "eval file facts contain duplicate paths"); - assertReport(execution.packets.every((packet) => factPaths.has(packet.path)), "packet_facts_join", "one or more eval packets lack file facts"); - const coverageIds = coverageRecords.map((record) => record.hunkId); - assertReport(unique(coverageIds).length === coverageIds.length, "coverage_join_mismatch", "coverage records contain duplicate hunk IDs"); - assertReport([...packetHunkIds].every((hunkId) => coverageIds.includes(hunkId)), "coverage_join_mismatch", "packet hunks are missing from coverage records"); - assertReport(execution.reviewedHunkIds.every((hunkId) => packetHunkIds.has(hunkId)), "coverage_join_mismatch", "reviewed coverage records reference packet-less hunks"); - const modelCallIds = execution.modelCalls.map((call) => call.callId); - assertReport(unique(modelCallIds).length === modelCallIds.length, "duplicate_model_call", "model-call telemetry contains duplicate IDs"); - const toolCallIds = execution.toolCalls.map((call) => call.toolCallId); - assertReport(unique(toolCallIds).length === toolCallIds.length, "duplicate_tool_call", "tool-call telemetry contains duplicate IDs"); - assertReport(execution.events.every((event, index) => execution.events.findIndex((candidate) => candidate.eventId === event.eventId) === index), "duplicate_event", "event telemetry contains duplicate IDs"); - assertReport(execution.modelCalls.every((call) => call.packetId === undefined || packetIds.has(call.packetId)), "model_packet_join", "model-call telemetry references unknown packets"); - assertReport(execution.toolCalls.every((call) => call.packetId === undefined || packetIds.has(call.packetId)), "tool_packet_join", "tool-call telemetry references unknown packets"); - assertReport(execution.toolCalls.every((call) => call.modelCallId === undefined || modelCallIds.includes(call.modelCallId)), "tool_model_join", "tool-call telemetry references unknown model calls"); - assertReport(execution.events.every((event) => event.message !== "same_file_atoms_packed" || (typeof event.data?.packetId === "string" && packetIds.has(event.data.packetId))), "event_packet_join", "packing telemetry references unknown packets"); - const observedRunIds = sortedUnique([ - ...execution.events.map((event) => event.runId), - ...execution.modelCalls.map((call) => call.runId), - ...execution.toolCalls.map((call) => call.runId) - ]); - assertReport(observedRunIds.length <= 1, "telemetry_run_join", "eval telemetry streams mix review run IDs", { observedRunIds }); - if (expectedRunId !== undefined && observedRunIds.length === 1) { - assertReport(observedRunIds[0] === expectedRunId, "telemetry_run_join", "eval telemetry run ID differs from info.json reviewRunId"); - } -} - -export function reconstructEvidenceArtifacts(execution: EvalExecutionInput): EvalArtifacts { - const rawModel = rawModelEvidence(execution.modelCalls, execution.events); - const rawTool = rawToolEvidence(execution.toolCalls); - const existingBudget = execution.scoringArtifacts.metricsSources.budgetSummary; - const omittedNotes = humanAttentionOutputCounts(execution.summaryArtifacts?.humanAttention).omitted; - const rawContextPressure = rawContextPressureEvidence(execution, omittedNotes); - const rawBudget = existingBudget === undefined ? undefined : ({ - ...existingBudget, - completeness: execution.scoringArtifacts.coverage?.partial === true ? "partial" : "complete", - partialReasons: execution.scoringArtifacts.coverage?.partial === true ? [...execution.scoringArtifacts.coverage.reasons] : [], - overruns: rawBudgetEvents(execution.events, "budget_overrun"), - dispatchBlocks: rawBudgetEvents(execution.events, "budget_dispatch_blocked"), - ...(rawContextPressure === undefined ? {} : { contextPressure: rawContextPressure }) - } as unknown as NonNullable); - return { - ...execution.scoringArtifacts, - candidates: execution.candidateFindings, - verification: execution.verification, - finalSelection: execution.finalSelection, - finalFindings: execution.finalFindings, - packets: execution.packets, - reviewPlan: execution.plan, - metricsSources: { - ...execution.scoringArtifacts.metricsSources, - costProfile: rawCostEvidence(rawModel), - modelCallsSummary: rawModel, - toolCallsSummary: rawTool, - ...(rawBudget === undefined ? {} : { budgetSummary: rawBudget }), - runJson: { - ...(isRecord(execution.scoringArtifacts.metricsSources.runJson) ? execution.scoringArtifacts.metricsSources.runJson : {}), - durationMs: execution.wallTimeSeconds * 1000 - }, - modelCalls: execution.modelCalls, - toolCalls: execution.toolCalls, - } - }; -} - -function recomputeEvidenceScore(run: EvalCaseRunInput, execution: EvalExecutionInput): EvalScore { - return scoreEvalRun(run.declaredCase, reconstructEvidenceArtifacts(execution), "live"); -} - -function validChangedLineAnchor(anchor: CandidateFinding["anchor"], packet: ReviewPacket, diffByHunk: Map): boolean { - if (anchor === undefined || anchor.path !== packet.path || !packet.hunks.some((hunk) => hunk.hunkId === anchor.hunkId)) { - return false; - } - const hunk = diffByHunk.get(anchor.hunkId); - if (hunk === undefined || hunk.path !== anchor.path) { - return false; - } - return anchor.side === "RIGHT" - ? hunk.lines.some((line) => line.kind === "add" && line.newLineNumber === anchor.line) - : hunk.lines.some((line) => line.kind === "delete" && line.oldLineNumber === anchor.line); -} - -type NumericRecord = Record; - -function emptyRawCacheCounts(): NumericRecord { - return { hit: 0, miss: 0, disabled: 0, write: 0 }; -} - -function updateRawCacheCounts(target: NumericRecord, status: LlmCallRecord["cacheStatus"]): void { - if (status === "write") { - target.miss = (target.miss ?? 0) + 1; - target.write = (target.write ?? 0) + 1; - } else { - target[status] = (target[status] ?? 0) + 1; - } -} - -function emptyRawFinalize(): NumericRecord { - return { - compactCalls: 0, fullCalls: 0, noFindingCalls: 0, candidateOrUnknownCalls: 0, promptChars: 0, - noFindingPromptChars: 0, candidateOrUnknownPromptChars: 0, costUSD: 0, noFindingCostUSD: 0, - candidateOrUnknownCostUSD: 0, unknownCostCalls: 0 - }; -} - -function updateRawFinalize(target: NumericRecord, call: LlmCallRecord, providerCalls: number): void { - if (providerCalls === 0 || call.finalizeMode === undefined) { - return; - } - target[call.finalizeMode === "compact" ? "compactCalls" : "fullCalls"]! += 1; - if (call.finalizeTarget === "no_findings") { - target.noFindingCalls! += 1; - target.noFindingPromptChars! += call.promptChars; - target.noFindingCostUSD! += call.costUSD ?? 0; - } else if (call.finalizeTarget === "candidate_or_unknown") { - target.candidateOrUnknownCalls! += 1; - target.candidateOrUnknownPromptChars! += call.promptChars; - target.candidateOrUnknownCostUSD! += call.costUSD ?? 0; - } - target.promptChars! += call.promptChars; - if (call.costUSD === undefined) { - target.unknownCostCalls! += 1; - } else { - target.costUSD! += call.costUSD; - } -} - -function emptyRawModelBucket(stage = false): Record { - return { - ...(stage ? { recordCount: 0, count: 0, statuses: { ok: 0, schema_invalid: 0, transient_error: 0, auth_error: 0, timeout: 0, aborted: 0 } } : { totalRecords: 0, totalCalls: 0 }), - providerCalls: 0, inputTokens: 0, uncachedInputTokens: 0, cacheReadTokens: 0, cacheWriteTokens: 0, - billableInputTokens: 0, outputTokens: 0, reasoningTokens: 0, totalTokens: 0, costUSD: 0, - inputCostUSD: 0, outputCostUSD: 0, cacheReadCostUSD: 0, cacheWriteCostUSD: 0, unknownCostCalls: 0, - cache: emptyRawCacheCounts(), retryAttempts: 0, repairCalls: 0, schemaInvalidCalls: 0, - ...(stage ? {} : { toolChoiceDowngradedCalls: 0 }), finalize: emptyRawFinalize() - }; -} - -function updateRawModelBucket(bucket: Record, call: LlmCallRecord, stage: boolean): void { - const providerCalls = call.cacheStatus === "hit" ? 0 : 1; - bucket[stage ? "recordCount" : "totalRecords"] = Number(bucket[stage ? "recordCount" : "totalRecords"]) + 1; - bucket[stage ? "count" : "totalCalls"] = Number(bucket[stage ? "count" : "totalCalls"]) + providerCalls; - bucket.providerCalls = Number(bucket.providerCalls) + providerCalls; - if (providerCalls > 0) { - for (const key of ["inputTokens", "uncachedInputTokens", "cacheReadTokens", "cacheWriteTokens", "billableInputTokens", "outputTokens", "reasoningTokens", "totalTokens"] as const) { - bucket[key] = Number(bucket[key]) + (call[key] ?? 0); - } - } - updateRawCacheCounts(bucket.cache as NumericRecord, call.cacheStatus); - bucket.retryAttempts = Number(bucket.retryAttempts) + (providerCalls > 0 && call.attempt > 1 ? 1 : 0); - bucket.repairCalls = Number(bucket.repairCalls) + (call.kind === "repair" ? 1 : 0); - bucket.schemaInvalidCalls = Number(bucket.schemaInvalidCalls) + (call.status === "schema_invalid" ? 1 : 0); - if (!stage) { - bucket.toolChoiceDowngradedCalls = Number(bucket.toolChoiceDowngradedCalls) + (providerCalls > 0 && call.toolChoiceDowngraded === true ? 1 : 0); - } else { - const statuses = bucket.statuses as NumericRecord; - statuses[call.status] = (statuses[call.status] ?? 0) + 1; - } - updateRawFinalize(bucket.finalize as NumericRecord, call, providerCalls); - if (providerCalls === 0) { - return; - } - if (call.costUSD === undefined) { - bucket.unknownCostCalls = Number(bucket.unknownCostCalls) + 1; - } else { - for (const key of ["costUSD", "inputCostUSD", "outputCostUSD", "cacheReadCostUSD", "cacheWriteCostUSD"] as const) { - bucket[key] = Number(bucket[key]) + (call[key] ?? 0); - } - } -} - -function cacheAliases(bucket: Record): Record { - return { - ...bucket, - localModelCallCache: { ...(bucket.cache as NumericRecord) }, - providerPromptCache: { - readTokens: bucket.cacheReadTokens, writeTokens: bucket.cacheWriteTokens, - readCostUSD: bucket.cacheReadCostUSD, writeCostUSD: bucket.cacheWriteCostUSD - } - }; -} - -function emptyRawSchemaRecovery(): NumericRecord { - return { - schemaInvalidCalls: 0, schemaInvalidRecovered: 0, schemaInvalidUnrecovered: 0, - schemaRepairAttempts: 0, schemaRepairRecovered: 0, deterministicSchemaRecovered: 0, schemaRecoveryFailed: 0 - }; -} - -function rawSchemaRecoveryEvidence(calls: LlmCallRecord[], events: TelemetryEvent[]): Record { - const total = emptyRawSchemaRecovery(); - const byStage: Record = {}; - const add = (stage: number, delta: Partial): void => { - const targets = stage === 0 ? [total] : [total, byStage[String(stage)] ?? (byStage[String(stage)] = emptyRawSchemaRecovery())]; - for (const target of targets) { - for (const [key, value] of Object.entries(delta)) target[key] = (target[key] ?? 0) + (value ?? 0); - } - }; - for (const call of calls) { - if (call.status === "schema_invalid") add(call.stage, { schemaInvalidCalls: 1 }); - if (call.kind !== "repair") continue; - if (call.status === "ok") { - add(call.stage, { schemaRepairAttempts: 1, schemaRepairRecovered: 1, schemaInvalidRecovered: 1 }); - } else { - add(call.stage, { schemaRepairAttempts: 1 }); - if (call.status === "schema_invalid" && call.stage !== 7 && call.stage !== 9) add(call.stage, { schemaRecoveryFailed: 1 }); - } - } - for (const event of events) { - if (event.message === "schema_invalid_submit_recovered") { - const recovered = event.data?.schemaRepairUsed === true ? 2 : 1; - add(event.stage, { deterministicSchemaRecovered: recovered, schemaInvalidRecovered: recovered }); - } else if (event.message === "stage7_schema_cleanup_recovered") { - add(event.stage, { deterministicSchemaRecovered: 1, schemaInvalidRecovered: 1 }); - } else if ([ - "schema_invalid_submit_recovery_invalid", "stage7_schema_cleanup_rejected", "stage7_schema_repair_failed", - "verification_schema_repair_failed" - ].includes(event.message)) { - add(event.stage, { schemaRecoveryFailed: 1 }); - } - } - const finalize = (input: NumericRecord): NumericRecord => ({ - ...input, - schemaInvalidRecovered: Math.min(input.schemaInvalidRecovered ?? 0, input.schemaInvalidCalls ?? 0), - schemaInvalidUnrecovered: Math.max(0, (input.schemaInvalidCalls ?? 0) - Math.min(input.schemaInvalidRecovered ?? 0, input.schemaInvalidCalls ?? 0)) - }); - return { ...finalize(total), byStage: Object.fromEntries(Object.entries(byStage).map(([stage, counters]) => [stage, finalize(counters)])) }; -} - -function rawModelEvidence(calls: LlmCallRecord[], events: TelemetryEvent[] = []): Record { - const total = emptyRawModelBucket(); - const byStage: Record> = {}; - for (const call of calls) { - updateRawModelBucket(total, call, false); - const key = String(call.stage); - const bucket = byStage[key] ?? (byStage[key] = emptyRawModelBucket(true)); - updateRawModelBucket(bucket, call, true); - } - const schemaRecovery = rawSchemaRecoveryEvidence(calls, events); - const recoveryByStage = schemaRecovery.byStage as Record; - return { - ...cacheAliases(total), - schemaRecovery, - byStage: Object.fromEntries(Object.entries(byStage).map(([stage, bucket]) => [stage, { - ...cacheAliases(bucket), - schemaRecovery: recoveryByStage[stage] ?? emptyRawSchemaRecovery() - }])) - }; -} - -function modelEvidenceView(summary: Record): Record { - const topKeys = [...Object.keys(emptyRawModelBucket()), "localModelCallCache", "providerPromptCache", "schemaRecovery"]; - const stageKeys = [...Object.keys(emptyRawModelBucket(true)), "localModelCallCache", "providerPromptCache", "schemaRecovery"]; - const pick = (input: Record, keys: string[]) => Object.fromEntries(keys.map((key) => [key, input[key]])); - const stages = isRecord(summary.byStage) ? summary.byStage : {}; - return { - ...pick(summary, topKeys), - byStage: Object.fromEntries(Object.entries(stages).map(([stage, value]) => [stage, isRecord(value) ? pick(value, stageKeys) : value])) - }; -} - -function rawCostEvidence(model: Record): Record { - const breakdown = (source: Record) => ({ - uncachedInput: { tokens: source.uncachedInputTokens, costUSD: source.inputCostUSD }, - providerPromptCacheRead: { tokens: source.cacheReadTokens, costUSD: source.cacheReadCostUSD }, - providerPromptCacheWrite: { tokens: source.cacheWriteTokens, costUSD: source.cacheWriteCostUSD }, - output: { tokens: source.outputTokens, costUSD: source.outputCostUSD }, - total: { tokens: source.totalTokens, costUSD: source.costUSD } - }); - const byStage = model.byStage as Record>; - return { - totalCostUSD: model.costUSD, unknownCostCalls: model.unknownCostCalls, - localModelCallCache: model.localModelCallCache, providerPromptCache: model.providerPromptCache, - costBreakdown: breakdown(model), - tokens: Object.fromEntries(["inputTokens", "uncachedInputTokens", "cacheReadTokens", "cacheWriteTokens", "billableInputTokens", "outputTokens", "reasoningTokens", "totalTokens"].map((key) => [key, model[key]])), - cost: { - inputCostUSD: model.inputCostUSD, outputCostUSD: model.outputCostUSD, cacheReadCostUSD: model.cacheReadCostUSD, - cacheWriteCostUSD: model.cacheWriteCostUSD, totalCostUSD: model.costUSD - }, - byStage: Object.fromEntries(Object.entries(byStage).map(([stage, value]) => { - const { schemaRecovery: _schemaRecovery, ...costValue } = value; - return [stage, { ...costValue, costBreakdown: breakdown(costValue) }]; - })) - }; -} - -function emptyRawToolCache(): NumericRecord { - return { hits: 0, misses: 0, writes: 0, disabled: 0, inflightHits: 0, evictions: 0, backendExecutions: 0, savedBackendCalls: 0 }; -} - -function emptyRawToolBucket(): Record { - return { count: 0, errors: 0, rejections: 0, degraded: 0, backendExecutions: 0, savedBackendCalls: 0, totalDurationMs: 0, totalResultChars: 0, resultCache: emptyRawToolCache() }; -} - -function updateRawToolCache(cache: NumericRecord, call: ToolCallRecord): void { - if (call.backendExecuted === true) cache.backendExecutions! += 1; - if (call.cacheStatus === "hit") { - cache.hits! += 1; cache.savedBackendCalls! += 1; - if (call.cacheHitKind === "inflight") cache.inflightHits! += 1; - } else if (call.cacheStatus === "write") { - cache.misses! += 1; cache.writes! += 1; - } else if (call.cacheStatus === "miss") { - cache.misses! += 1; - } else { - cache.disabled! += 1; - } - cache.evictions! += call.cacheEvictedEntries ?? 0; -} - -function updateRawToolBucket(bucket: Record, call: ToolCallRecord): void { - bucket.count = Number(bucket.count) + 1; - bucket.errors = Number(bucket.errors) + (call.status === "error" ? 1 : 0); - bucket.rejections = Number(bucket.rejections) + (call.status === "rejected" ? 1 : 0); - bucket.degraded = Number(bucket.degraded) + (call.degraded ? 1 : 0); - bucket.backendExecutions = Number(bucket.backendExecutions) + (call.backendExecuted === true ? 1 : 0); - bucket.savedBackendCalls = Number(bucket.savedBackendCalls) + (call.cacheStatus === "hit" ? 1 : 0); - bucket.totalDurationMs = Number(bucket.totalDurationMs) + call.durationMs; - bucket.totalResultChars = Number(bucket.totalResultChars) + call.resultChars; - updateRawToolCache(bucket.resultCache as NumericRecord, call); -} - -function rawToolEvidence(calls: ToolCallRecord[]): Record { - const resultCache = emptyRawToolCache(); - const byTool: Record> = {}; - const byStage: Record> = {}; - for (const call of calls) { - updateRawToolCache(resultCache, call); - updateRawToolBucket(byTool[call.tool] ?? (byTool[call.tool] = emptyRawToolBucket()), call); - updateRawToolBucket(byStage[String(call.stage)] ?? (byStage[String(call.stage)] = emptyRawToolBucket()), call); - } - const average = (buckets: Record>) => Object.fromEntries(Object.entries(buckets).map(([key, bucket]) => [key, { - ...bucket, - averageDurationMs: Number(bucket.count) === 0 ? 0 : Number(bucket.totalDurationMs) / Number(bucket.count), - averageResultChars: Number(bucket.count) === 0 ? 0 : Number(bucket.totalResultChars) / Number(bucket.count) - }])); - return { totalCalls: calls.length, resultCache, byTool: average(byTool), byStage: average(byStage) }; -} - -function eventStageEvidence(events: TelemetryEvent[], recovery: Record): Record { - const recoveryByStage = recovery.byStage as Record; - const byStage: Record = {}; - for (let stage = 0; stage <= 11; stage += 1) { - byStage[String(stage)] = { - events: 0, levels: { debug: 0, info: 0, warn: 0, error: 0 }, cache: emptyRawCacheCounts(), runtimeMs: 0, - schemaRecovery: recoveryByStage[String(stage)] ?? emptyRawSchemaRecovery() - }; - } - for (const event of events) { - const bucket = byStage[String(event.stage)]!; - bucket.events += 1; - bucket.levels[event.level] = (bucket.levels[event.level] ?? 0) + 1; - if (event.cacheStatus !== undefined) bucket.cache[event.cacheStatus] = (bucket.cache[event.cacheStatus] ?? 0) + 1; - if (event.message === "stage_started" && bucket.startedAt === undefined) bucket.startedAt = event.timestamp; - if ((event.message === "stage_completed" || event.message === "stage_failed") && bucket.completedAt === undefined) { - bucket.completedAt = event.timestamp; - if (bucket.startedAt !== undefined) bucket.runtimeMs += Math.max(0, Date.parse(event.timestamp) - Date.parse(bucket.startedAt)); - } - } - return byStage; -} - -function telemetryStageEvidenceView(input: Record): Record { - const stages = isRecord(input.stages) ? input.stages : {}; - return Object.fromEntries(Object.entries(stages).map(([stage, value]) => { - if (!isRecord(value)) return [stage, value]; - return [stage, Object.fromEntries(["events", "levels", "cache", "startedAt", "completedAt", "runtimeMs", "schemaRecovery"] - .filter((key) => value[key] !== undefined).map((key) => [key, value[key]]))]; - })); -} - -function humanAttentionOutputCounts(input: unknown): { emitted: number; omitted: number } { - return isRecord(input) && Array.isArray(input.outputNotes) - ? { emitted: input.outputNotes.length, omitted: typeof input.omittedCount === "number" ? input.omittedCount : 0 } - : { emitted: Array.isArray(input) ? input.length : 0, omitted: 0 }; -} - -function rawBudgetEvents(events: TelemetryEvent[], message: "budget_overrun" | "budget_dispatch_blocked"): Array> { - return events.filter((event) => event.message === message).map((event) => ({ ...(event.data ?? {}) })); -} - -function incrementStageCount(target: NumericRecord, stage: number): void { - if (stage !== 0) { - target[String(stage)] = (target[String(stage)] ?? 0) + 1; - } -} - -function rawContextPressureEvidence(execution: EvalExecutionInput, omittedNotes: number): Record | undefined { - const localRejections = execution.toolCalls.filter((call) => call.status === "rejected" && - isLocalToolBudgetRejectionReason(call.degradationReason ?? call.errorCode ?? "rejected")); - const degradedCalls = execution.toolCalls.filter((call) => call.status !== "rejected" && call.degraded); - const rejectionByStage: NumericRecord = {}; - const degradedByStage: NumericRecord = {}; - const rejectionReasons: NumericRecord = {}; - for (const call of localRejections) { - incrementStageCount(rejectionByStage, call.stage); - const reason = call.degradationReason ?? call.errorCode ?? "rejected"; - rejectionReasons[reason] = (rejectionReasons[reason] ?? 0) + 1; - } - for (const call of degradedCalls) { - incrementStageCount(degradedByStage, call.stage); - } - const extensions = { - granted: 0, - denied: 0, - resultChars: 0, - grantedByStage: {} as NumericRecord, - deniedByStage: {} as NumericRecord - }; - for (const event of execution.events) { - if (event.message === "tool_budget_extension_granted") { - extensions.granted += 1; - incrementStageCount(extensions.grantedByStage, event.stage); - extensions.resultChars += typeof event.data?.resultChars === "number" ? event.data.resultChars : 0; - } else if (event.message === "tool_budget_extension_denied") { - extensions.denied += 1; - incrementStageCount(extensions.deniedByStage, event.stage); - } - } - const hasExtensionPressure = extensions.granted > 0 || extensions.denied > 0 || extensions.resultChars > 0; - if (localRejections.length === 0 && !hasExtensionPressure && degradedCalls.length === 0 && - execution.packets.every((packet) => packet.degraded === undefined) && omittedNotes === 0) { - return undefined; - } - return { - toolBudgetRejections: localRejections.length, - toolBudgetRejectionsByStage: rejectionByStage, - ...(hasExtensionPressure ? { toolBudgetExtensions: extensions } : {}), - degradedToolResults: degradedCalls.length, - degradedToolResultsByStage: degradedByStage, - degradedHunks: sum(execution.packets.filter((packet) => packet.degraded !== undefined).map((packet) => packet.hunks.length)), - rejectionReasons: Object.entries(rejectionReasons) - .map(([reason, count]) => ({ reason, count })) - .sort((left, right) => right.count - left.count || left.reason.localeCompare(right.reason)), - unresolvedNotes: { emitted: humanAttentionOutputCounts(execution.summaryArtifacts?.humanAttention).emitted, omitted: omittedNotes } - }; -} - -function attentionRelationsMatch( - records: Array>, - execution: EvalExecutionInput -): boolean { - const packetById = new Map(execution.packets.map((packet) => [packet.id, packet])); - const candidatePacketById = new Map(execution.candidateFindings.map((candidate) => [candidate.id, candidate.producedBy.packetId])); - const countByPacket = (predicate: (candidate: CandidateFinding) => boolean): NumericRecord => { - const counts: NumericRecord = {}; - for (const candidate of execution.candidateFindings.filter(predicate)) { - counts[candidate.producedBy.packetId] = (counts[candidate.producedBy.packetId] ?? 0) + 1; - } - return counts; - }; - const promoted = countByPacket((candidate) => candidate.provenance?.source === "uncertainty_promotion"); - const direct = countByPacket((candidate) => candidate.provenance?.source !== "uncertainty_promotion"); - const kept: NumericRecord = {}; - for (const record of execution.verification) { - if ("verdict" in record && (record.verdict.verdict === "keep" || record.verdict.verdict === "revise")) { - const packetId = candidatePacketById.get(record.candidateId); - if (packetId !== undefined) kept[packetId] = (kept[packetId] ?? 0) + 1; - } - } - const published: NumericRecord = {}; - for (const finding of execution.finalFindings.filter((candidate) => candidate.publication !== "suppressed")) { - const packetIds = new Set([finding.id, ...finding.mergedCandidateIds].flatMap((id) => candidatePacketById.get(id) ?? [])); - for (const packetId of packetIds) published[packetId] = (published[packetId] ?? 0) + 1; - } - const plannedHunkIds = new Set(execution.plan.coverage.map((decision) => decision.hunkId)); - return packetById.size === execution.packets.length && records.length === execution.packets.length && - unique(records.map((record) => record.packetId)).length === records.length && records.every((record) => { - const packet = packetById.get(record.packetId); - if (packet === undefined) return false; - const coverageSource = packet.coverageEscalation !== undefined - ? `escalated:${packet.coverageEscalation.rule}` - : packet.hunks.some((hunk) => hunk.plannerFallbackReason !== undefined) || - packet.hunks.every((hunk) => !plannedHunkIds.has(hunk.hunkId)) - ? "deterministic_default" - : "planner"; - return record.packetId === packet.id && record.path === packet.path && record.coverage === packet.coverage && - record.coverageSource === coverageSource && record.ensemblePasses >= 1 && - record.directCandidates === (direct[packet.id] ?? 0) && record.promotedCandidates === (promoted[packet.id] ?? 0) && - record.keptVerified === (kept[packet.id] ?? 0) && record.published === (published[packet.id] ?? 0); - }); -} - -function humanAttentionRelationsMatch(input: z.infer): boolean { - if (Array.isArray(input)) return true; - const noteIds = input.notes.map((note) => note.id); - const groupKeys = input.groups.map((group) => group.key); - const noteIdSet = new Set(noteIds); - const groupByKey = new Map(input.groups.map((group) => [group.key, group])); - const uniqueReferences = (values: string[]): boolean => unique(values).length === values.length && values.every((value) => groupByKey.has(value)); - const findingSuppressed = input.suppressedByFindings.map((record) => record.groupKey); - const verificationSuppressed = input.suppressedByVerification.map((record) => record.groupKey); - const partition = [...input.keptForOutputGroupIds, ...findingSuppressed, ...verificationSuppressed].sort(); - return unique(noteIds).length === noteIds.length && unique(groupKeys).length === groupKeys.length && - input.groups.every((group) => group.count === group.noteIds.length && unique(group.noteIds).length === group.noteIds.length && - group.noteIds.every((id) => noteIdSet.has(id))) && - uniqueReferences(input.composerPromptGroupIds) && uniqueReferences(input.outputGroupIds) && uniqueReferences(input.keptForOutputGroupIds) && - uniqueReferences(findingSuppressed) && uniqueReferences(verificationSuppressed) && - stableJson([...groupKeys].sort()) === stableJson(partition) && - stableJson(input.outputGroupIds) === stableJson(input.keptForOutputGroupIds.slice(0, input.outputGroupIds.length)) && - input.outputNotes.length === input.outputGroupIds.length && - input.omittedCount === input.keptForOutputGroupIds.length - input.outputGroupIds.length && - input.suppressedByFindings.every((record) => stableJson(record.noteIds) === stableJson(groupByKey.get(record.groupKey)?.noteIds)) && - input.suppressedByVerification.every((record) => stableJson(record.noteIds) === stableJson(groupByKey.get(record.groupKey)?.noteIds)); -} - -function validateRawSummaryEvidence(run: EvalCaseRunInput, execution: EvalExecutionInput, label: string, failures: ReportFailure[]): void { - const artifacts = execution.summaryArtifacts; - if (artifacts === undefined) { - return; - } - const parsedModel = modelCallsSummarySchema.safeParse(artifacts.model); - const parsedCost = costProfileSchema.safeParse(artifacts.cost); - const parsedTool = toolCallsSummarySchema.safeParse(artifacts.tool); - const parsedBudget = budgetSummarySchema.safeParse(artifacts.budget); - const parsedRun = evalRunTelemetrySchema.safeParse(artifacts.run); - const parsedTelemetry = telemetrySummarySchema.safeParse(artifacts.telemetry); - const parsedAttention = z.array(attentionRecordSchema).safeParse(artifacts.attention); - const parsedHuman = humanAttentionArtifactSchema.safeParse(artifacts.humanAttention); - if (!parsedModel.success || !parsedCost.success || !parsedTool.success || !parsedBudget.success || !parsedRun.success || - !parsedTelemetry.success || !parsedAttention.success || !parsedHuman.success) { - failures.push(failure("paid_summary_schema", `${label}/repeat ${execution.repeat} summary evidence is not strictly shaped`)); - return; - } - const rawModel = rawModelEvidence(execution.modelCalls, execution.events); - const rawTool = rawToolEvidence(execution.toolCalls); - const rawDerived = reconstructRunTelemetryDerivedEvidence(execution.events, execution.modelCalls); - const rawCost = rawCostEvidence(rawModel); - const modelMatches = stableJson(modelEvidenceView(parsedModel.data as unknown as Record)) === stableJson(rawModel); - const telemetryModelMatches = stableJson(modelEvidenceView(parsedTelemetry.data.modelCalls as unknown as Record)) === stableJson(rawModel); - const costMatches = stableJson(parsedCost.data) === stableJson(rawCost); - const toolMatches = stableJson(parsedTool.data) === stableJson(rawTool) && stableJson(parsedTelemetry.data.toolCalls) === stableJson(rawTool); - const runTotals = parsedRun.data.totals; - const telemetryTotals = parsedTelemetry.data.totals; - const totalView = (totals: typeof runTotals) => ({ - events: totals.events, modelCallRecords: totals.modelCallRecords, modelCalls: totals.modelCalls, providerCalls: totals.providerCalls, - toolCalls: totals.toolCalls, toolResultCache: totals.toolResultCache, inputTokens: totals.inputTokens, - uncachedInputTokens: totals.uncachedInputTokens, cacheReadTokens: totals.cacheReadTokens, cacheWriteTokens: totals.cacheWriteTokens, - billableInputTokens: totals.billableInputTokens, outputTokens: totals.outputTokens, reasoningTokens: totals.reasoningTokens, - totalTokens: totals.totalTokens, totalCostUSD: totals.totalCostUSD, inputCostUSD: totals.inputCostUSD, - outputCostUSD: totals.outputCostUSD, cacheReadCostUSD: totals.cacheReadCostUSD, cacheWriteCostUSD: totals.cacheWriteCostUSD, - costBreakdown: totals.costBreakdown, - unknownCostCalls: totals.unknownCostCalls, cache: totals.cache, localModelCallCache: totals.localModelCallCache, - providerPromptCache: totals.providerPromptCache, retryAttempts: totals.retryAttempts, repairCalls: totals.repairCalls, - schemaInvalidCalls: totals.schemaInvalidCalls, schemaRecovery: totals.schemaRecovery, - stage7SchemaRepair: totals.stage7SchemaRepair, logOverflow: totals.logOverflow, - filesChanged: totals.filesChanged, hunks: totals.hunks, packets: totals.packets, packetReviews: totals.packetReviews, - candidates: totals.candidates, verified: totals.verified, finalFindings: totals.finalFindings, postedComments: totals.postedComments - }); - const rawTotals = { - events: execution.events.length, modelCallRecords: rawModel.totalRecords, modelCalls: rawModel.providerCalls, providerCalls: rawModel.providerCalls, - toolCalls: execution.toolCalls.length, toolResultCache: rawTool.resultCache, inputTokens: rawModel.inputTokens, - uncachedInputTokens: rawModel.uncachedInputTokens, cacheReadTokens: rawModel.cacheReadTokens, cacheWriteTokens: rawModel.cacheWriteTokens, - billableInputTokens: rawModel.billableInputTokens, outputTokens: rawModel.outputTokens, reasoningTokens: rawModel.reasoningTokens, - totalTokens: rawModel.totalTokens, totalCostUSD: rawModel.costUSD, inputCostUSD: rawModel.inputCostUSD, - outputCostUSD: rawModel.outputCostUSD, cacheReadCostUSD: rawModel.cacheReadCostUSD, cacheWriteCostUSD: rawModel.cacheWriteCostUSD, - costBreakdown: rawCost.costBreakdown, - unknownCostCalls: rawModel.unknownCostCalls, cache: rawModel.cache, localModelCallCache: rawModel.localModelCallCache, - providerPromptCache: rawModel.providerPromptCache, retryAttempts: rawModel.retryAttempts, repairCalls: rawModel.repairCalls, - schemaInvalidCalls: rawModel.schemaInvalidCalls, schemaRecovery: rawModel.schemaRecovery, - stage7SchemaRepair: rawDerived.stage7SchemaRepair, logOverflow: rawDerived.logOverflow, - ...rawDerived.pipelineTotals - }; - const totalsMatch = stableJson(totalView(runTotals)) === stableJson(rawTotals) && stableJson(totalView(telemetryTotals)) === stableJson(rawTotals); - const stagesMatch = stableJson(telemetryStageEvidenceView(parsedTelemetry.data as unknown as Record)) === - stableJson(eventStageEvidence(execution.events, rawModel.schemaRecovery as Record)); - const recoveryMatches = stableJson(parsedTelemetry.data.schemaRecovery) === stableJson(rawModel.schemaRecovery); - const pipelineMatches = stableJson({ - workers: parsedTelemetry.data.workers, - packets: parsedTelemetry.data.packets, - lenses: parsedTelemetry.data.lenses, - coverage: parsedTelemetry.data.coverage, - candidates: parsedTelemetry.data.candidates, - verdicts: parsedTelemetry.data.verdicts, - dedup: parsedTelemetry.data.dedup, - finalSelection: parsedTelemetry.data.finalSelection, - posting: parsedTelemetry.data.posting - }) === stableJson(rawDerived.pipeline); - const repairAndOverflowMatch = stableJson(parsedTelemetry.data.schemaRepair.stage7) === stableJson(rawDerived.stage7SchemaRepair) && - stableJson(parsedTelemetry.data.logs.bufferedOverflow) === stableJson(rawDerived.logOverflow) && - parsedTelemetry.data.events === execution.events.length; - const budget = parsedBudget.data; - const rawByStage = rawModel.byStage as Record>; - const budgetUsage = { - modelCalls: rawModel.providerCalls, - totalTokens: rawModel.totalTokens, - costUSD: Number(rawModel.costUSD) > 0 ? rawModel.costUSD : undefined, - byStage: Object.entries(rawByStage).map(([stage, entry]) => ({ stage: Number(stage), modelCalls: entry.providerCalls, totalTokens: entry.totalTokens })) - .sort((left, right) => left.stage - right.stage) - }; - const usageMatches = stableJson(budget.usage) === stableJson(budgetUsage) && - budget.effective.timeoutMs === run.info.effectiveConfig?.review.timeoutMs && - budget.effective.maxBudgetTokens === run.info.effectiveConfig?.review.maxBudgetTokens; - const noteCounts = humanAttentionOutputCounts(parsedHuman.data); - const contextMatches = stableJson(budget.contextPressure) === stableJson(rawContextPressureEvidence(execution, noteCounts.omitted)); - const budgetEventsMatch = stableJson(budget.overruns) === stableJson(rawBudgetEvents(execution.events, "budget_overrun")) && - stableJson(budget.dispatchBlocks) === stableJson(rawBudgetEvents(execution.events, "budget_dispatch_blocked")); - const rawRunIds = sortedUnique([ - ...execution.events.map((event) => event.runId), - ...execution.modelCalls.map((call) => call.runId), - ...execution.toolCalls.map((call) => call.runId) - ]); - const identityMatches = parsedRun.data.runId === parsedTelemetry.data.runId && - (run.info.reviewRunId === undefined || parsedRun.data.runId === run.info.reviewRunId) && - (rawRunIds.length === 0 || (rawRunIds.length === 1 && rawRunIds[0] === parsedRun.data.runId)) && - parsedRun.data.startedAt === parsedTelemetry.data.startedAt && parsedRun.data.finishedAt === parsedTelemetry.data.finishedAt && - parsedRun.data.completedAt === parsedRun.data.finishedAt && parsedTelemetry.data.completedAt === parsedTelemetry.data.finishedAt && - parsedRun.data.durationMs === Math.max(0, Date.parse(parsedRun.data.finishedAt) - Date.parse(parsedRun.data.startedAt)) && - parsedRun.data.durationMs === parsedTelemetry.data.durationMs && parsedRun.data.durationMs === execution.wallTimeSeconds * 1000; - const attentionMatches = attentionRelationsMatch(parsedAttention.data, execution) && humanAttentionRelationsMatch(parsedHuman.data); - if (!modelMatches || !telemetryModelMatches || !costMatches || !toolMatches || !totalsMatch || !stagesMatch || !recoveryMatches || - !pipelineMatches || !repairAndOverflowMatch || !usageMatches || !contextMatches || !budgetEventsMatch || !identityMatches || !attentionMatches) { - failures.push(failure("paid_summary_reconciliation", `${label}/repeat ${execution.repeat} summaries do not independently reconcile to raw records`, { - modelMatches, telemetryModelMatches, costMatches, toolMatches, totalsMatch, stagesMatch, recoveryMatches, pipelineMatches, - repairAndOverflowMatch, usageMatches, contextMatches, budgetEventsMatch, identityMatches, attentionMatches - })); - } -} - -function validateExecutionEvidence(run: EvalCaseRunInput, execution: EvalExecutionInput, label: string, failures: ReportFailure[]): EvalScore { - validateRawSummaryEvidence(run, execution, label, failures); - const packetById = new Map(execution.packets.map((packet) => [packet.id, packet])); - const diffHunks = execution.diff.files.flatMap((file) => file.hunks); - const diffByHunk = new Map(diffHunks.map((hunk) => [hunk.id, hunk])); - const candidateById = new Map(execution.candidateFindings.map((finding) => [finding.id, finding])); - const verificationById = new Map(execution.verification.map((record) => [record.candidateId, record])); - const selectionById = new Map(execution.finalSelection.map((record) => [record.findingId, record])); - const finalByFingerprint = new Map(execution.finalFindings.map((finding) => [finding.fingerprint, finding])); - const selectionArtifact = execution.finalSelectionArtifact; - const effectiveReview = run.info.effectiveConfig?.review; - const policyConfig: CodegenieConfig = { - ...structuredClone(defaultConfig), - review: { - ...structuredClone(defaultConfig.review), - ...(effectiveReview ?? {}) - } - }; - let relationsValid = - candidateById.size === execution.candidateFindings.length && verificationById.size === execution.verification.length && - selectionById.size === execution.finalSelection.length && finalByFingerprint.size === execution.finalFindings.length && - verificationById.size === candidateById.size && effectiveReview !== undefined && - stableJson(selectionArtifact.records) === stableJson(execution.finalSelection) && - ((selectionArtifact.composition.mode === "deterministic_fallback" || selectionArtifact.composition.mode === "schema_repair_fallback") - ? (selectionArtifact.composition.fallbackReason?.trim().length ?? 0) > 0 - : selectionArtifact.composition.fallbackReason === undefined); - - for (const candidate of execution.candidateFindings) { - const packet = packetById.get(candidate.producedBy.packetId); - const record = verificationById.get(candidate.id); - const producerStageValid = candidate.provenance?.source === "uncertainty_promotion" - ? candidate.producedBy.stage === 9 && candidate.provenance.sourcePacketId === candidate.producedBy.packetId - : candidate.producedBy.stage === 7 || candidate.producedBy.stage === 8; - const producerValid = packet !== undefined && candidate.producedBy.kind === "packet" && - producerStageValid && candidate.producedBy.lensId.trim().length > 0 && - packet.lenses.includes(candidate.producedBy.lensId) && candidate.producedBy.skillIds.every((id) => id.trim().length > 0); - const provenanceValid = candidate.provenance === undefined || ( - packetById.has(candidate.provenance.sourcePacketId) && candidate.provenance.question.trim().length > 0 && candidate.provenance.reason.trim().length > 0 && - record?.candidateProvenance !== undefined && stableJson(record.candidateProvenance) === stableJson(candidate.provenance) - ); - const recordValid = record !== undefined && (!("verdict" in record) || record.verdict.candidateId === candidate.id); - relationsValid &&= producerValid && packet !== undefined && candidate.path === packet.path && provenanceValid && recordValid; - } - - const keptIds = new Set(execution.verification.flatMap((record) => - "verdict" in record && record.verdict.verdict !== "reject" && record.verdict.verdict !== "incomplete" && record.verdict.verificationIncomplete !== true - ? [record.candidateId] - : [] - )); - const reconstructedVerified = reconstructVerifiedFindingsFromArtifacts( - execution.candidateFindings, - execution.verification, - execution.packets, - execution.diff - ); - const verifiedById = new Map(reconstructedVerified.map((finding) => [finding.id, finding])); - const verifierPolicy = reconstructGatedVerifierCandidatesFromArtifacts( - execution.candidateFindings, - execution.packets, - execution.diff, - policyConfig - ); - const gateEvaluationById = new Map(verifierPolicy.evaluations.map((evaluation) => [evaluation.candidate.id, evaluation])); - const reconstructedVerifierCandidates = new Map(verifierPolicy.candidates.map((finding) => [finding.id, finding])); - relationsValid &&= verifiedById.size === reconstructedVerified.length && stableJson([...verifiedById.keys()].sort()) === stableJson([...keptIds].sort()); - relationsValid &&= selectionById.size === keptIds.size && [...keptIds].every((id) => selectionById.has(id)); - for (const record of execution.verification) { - const ownEvaluation = gateEvaluationById.get(record.candidateId); - const hasVerdict = "verdict" in record; - // Completed and budget-limited duplicate records are emitted with the - // representative's record metadata. Lane-limited duplicate records have - // no verdict and are emitted with the duplicate candidate's own metadata. - const gateFactsCandidateId = effectiveReview?.verify !== false && hasVerdict && record.duplicateOf !== undefined - ? record.duplicateOf - : record.candidateId; - const gateEvaluation = gateEvaluationById.get(gateFactsCandidateId); - const gateCandidate = gateEvaluation?.candidate; - if (gateCandidate === undefined || ownEvaluation === undefined) { - relationsValid = false; - } - const gatePacket = gateCandidate === undefined ? undefined : packetById.get(gateCandidate.producedBy.packetId); - relationsValid &&= gateCandidate !== undefined && stableJson(record.gateFacts) === stableJson( - gateEvaluation?.decision.facts - ); - const ownDecision = ownEvaluation?.decision; - const ownGate = ownEvaluation?.anchorStripped === true ? "gate_anchor_stripped" : "passed"; - const expectedScheduledDecision = gateEvaluation?.decision.outcome === "schedule" - ? gateEvaluation.decision.lane === "evidence_resolution" ? "scheduled_for_evidence_resolution" : "scheduled" - : undefined; - const laneLimited = !hasVerdict && ownDecision?.outcome === "schedule" && - ownDecision.lane === "evidence_resolution" && record.gateReason === "low_confidence_evidence_resolution_lane_limit"; - const gateStateValid = ownDecision?.outcome === "suppress" - ? !hasVerdict && record.gate === (ownEvaluation?.anchorStripped === true ? "gate_anchor_stripped" : "suppressed") && - record.gateDecision === "suppressed" && record.gateReason === (ownEvaluation?.anchorStripped === true - ? `invalid_anchor; ${ownDecision.reason}` - : ownDecision.reason) && record.verificationLane === undefined && record.gateFacts !== undefined - : laneLimited - ? record.gate === "suppressed" && record.gateDecision === "scheduled_for_evidence_resolution" && - record.verificationLane === "evidence_resolution" && record.gateFacts !== undefined - : hasVerdict && record.gate === ownGate && record.gateDecision === expectedScheduledDecision && - record.gateReason === (gateEvaluation?.decision.outcome === "schedule" ? gateEvaluation.decision.reason : undefined) && - record.verificationLane === (gateEvaluation?.decision.outcome === "schedule" ? gateEvaluation.decision.lane : undefined) && - record.gateFacts !== undefined; - relationsValid &&= gateStateValid; - const clusteredCandidate = reconstructedVerifierCandidates.get(record.candidateId); - if (ownDecision?.outcome === "suppress") { - relationsValid &&= clusteredCandidate === undefined && record.duplicateOf === undefined && record.clusterId === undefined; - } else if (effectiveReview?.verify === false) { - relationsValid &&= clusteredCandidate !== undefined && record.duplicateOf === undefined && record.clusterId === undefined && hasVerdict && - stableJson(record.verdict) === stableJson({ - candidateId: record.candidateId, - verdict: "keep", - reason: "verification disabled by config", - requiredEvidencePresent: true, - falsePositiveRisk: "low" - }); - } else { - relationsValid &&= clusteredCandidate !== undefined && - record.duplicateOf === clusteredCandidate?.duplicateOf && - record.clusterId === clusteredCandidate?.clusterId; - } - if ("verdict" in record) { - if (record.duplicateOf !== undefined) { - const representativeRecord = verificationById.get(record.duplicateOf); - relationsValid &&= representativeRecord !== undefined && "verdict" in representativeRecord && clusteredCandidate !== undefined && - stableJson(record.verdict) === stableJson(reconstructDuplicateVerificationVerdict(clusteredCandidate, representativeRecord.verdict)); - } - const packet = record.verdict.finalFinding === undefined ? undefined : packetById.get(record.verdict.finalFinding.producedBy.packetId); - if (record.verdict.finalFinding !== undefined && (packet === undefined || !validChangedLineAnchor(record.verdict.finalFinding.anchor, packet, diffByHunk))) { - relationsValid = false; - } - if (record.verdict.revisedAnchor !== undefined) { - const candidate = candidateById.get(record.candidateId); - const candidatePacket = candidate === undefined ? undefined : packetById.get(candidate.producedBy.packetId); - if (candidatePacket === undefined || !validChangedLineAnchor(record.verdict.revisedAnchor, candidatePacket, diffByHunk)) { - relationsValid = false; - } - } - if (keptIds.has(record.candidateId)) { - const source = verifiedById.get(record.candidateId); - const finals = execution.finalFindings.filter((finding) => finding.mergedCandidateIds.includes(record.candidateId)); - if (source === undefined || source.id !== record.candidateId || finals.length !== 1) { - relationsValid = false; - } else { - const target = finals[0]!; - const lineageAnchor = record.verdict.revisedAnchor ?? source.anchor; - const lineageAnchorRetained = source.anchorSource === "backfill_packet_representative" || - lineageAnchor === undefined || stableJson(target.anchor) === stableJson(lineageAnchor) || - (target.mergedAnchors ?? []).some((anchor) => stableJson(anchor) === stableJson(lineageAnchor)); - const sourcePath = lineageAnchor?.path ?? source.path; - const sourceFieldsRetained = target.id !== record.candidateId || ( - stableJson(target.producedBy) === stableJson(source.producedBy) && target.category === source.category && - target.failureMode === source.failureMode && target.whyThisMatters === source.whyThisMatters && - target.verification === source.verification - ); - relationsValid &&= lineageAnchorRetained && sourceFieldsRetained && - (target.mergedCategories === undefined || target.mergedCategories.includes(source.category)) && - (target.mergedPaths === undefined || target.mergedPaths.includes(sourcePath)) && - (target.mergedTitles === undefined || target.mergedTitles.includes(source.title)); - } - } - } - } - - const composerEvidence = reconstructComposerGroupsFromArtifacts(reconstructedVerified, execution.packets); - const composerCandidateById = new Map(composerEvidence.publishable.map((finding) => [finding.id, finding])); - const pretrimSuppressedIds = new Set(composerEvidence.pretrimSuppressedIds); - const capSuppressionReasons = new Set(["severity-threshold", "confidence-threshold", "report-cap"]); - const publicationDowngradeReasons = new Set(["min-inline-confidence", "soft-comment-cap", "unanchorable"]); - const policyDrafts = execution.finalFindings.flatMap((finding) => { - const mergedFindings = unique(finding.mergedCandidateIds).flatMap((id) => composerCandidateById.get(id) ?? []); - if (mergedFindings.length !== finding.mergedCandidateIds.length) { - return []; - } - const selections = finding.mergedCandidateIds.flatMap((id) => selectionById.get(id) ?? []); - const isPretrimmed = finding.mergedCandidateIds.some((id) => pretrimSuppressedIds.has(id)); - const hasPolicyDowngrade = selections.some((selection) => publicationDowngradeReasons.has(selection.reason)); - const requestedPublication = isPretrimmed - ? "suppressed" as const - : finding.publication === "inline" || hasPolicyDowngrade - ? "inline" as const - : finding.publication === "summary-only" - ? "summary-only" as const - : mergedFindings.some((candidate) => candidate.anchor !== undefined) - ? "inline" as const - : "summary-only" as const; - const policyReasonApplied = selections.some((selection) => - capSuppressionReasons.has(selection.reason) || publicationDowngradeReasons.has(selection.reason) - ); - const representativeSelection = selectionById.get(finding.id); - const representativeBaseReason = !policyReasonApplied && - (representativeSelection?.reason === "composer-selected" || representativeSelection?.reason === "composer_omitted_finding") - ? representativeSelection.reason - : "composer-selected"; - if (representativeBaseReason === "composer_omitted_finding" && selectionArtifact.composition.mode !== "llm_degraded") { - relationsValid = false; - } - return [{ - mergedFindings, - finalBody: finding.finalBody, - requestedPublication, - baseSelection: finding.mergedCandidateIds.map((id) => isPretrimmed - ? { findingId: id, decision: "suppressed" as const, reason: "composer-pre-trim" } - : id === finding.id - ? { findingId: id, decision: "published" as const, reason: representativeBaseReason } - : { findingId: id, decision: "merged" as const, reason: "composer-merged", mergedIntoFingerprint: finding.fingerprint }) - }]; - }); - const verifierVerdicts = execution.verification.flatMap((record) => - "verdict" in record && (effectiveReview?.verify === false || record.duplicateOf === undefined) ? [record.verdict] : [] - ); - const composerPolicy = reconstructComposerPolicyFromArtifacts( - policyDrafts, - execution.packets, - execution.diff, - policyConfig, - verifierVerdicts - ); - relationsValid &&= policyDrafts.length === execution.finalFindings.length && - stableJson(composerPolicy.selection) === stableJson(execution.finalSelection) && - stableJson(composerPolicy.findings.map((finding) => finding.id)) === stableJson(execution.finalFindings.map((finding) => finding.id)); - const reconstructedFinalById = new Map(composerPolicy.findings.map((finding) => [finding.id, finding])); - for (const finding of execution.finalFindings) { - const packet = packetById.get(finding.producedBy.packetId); - const representative = candidateById.get(finding.id); - const mergedIds = unique(finding.mergedCandidateIds); - const mergedValid = mergedIds.length > 0 && mergedIds.length === finding.mergedCandidateIds.length && - mergedIds.every((id) => candidateById.has(id) && keptIds.has(id) && selectionById.has(id)); - const publicationAnchorOwner = finding.anchor === undefined ? undefined : [...packetById.values()] - .find((candidatePacket) => candidatePacket.hunks.some((hunk) => hunk.hunkId === finding.anchor?.hunkId)); - const publicationAnchorValid = finding.anchor === undefined - ? finding.publication !== "inline" && finding.changedLine === false - : publicationAnchorOwner !== undefined && validChangedLineAnchor(finding.anchor, publicationAnchorOwner, diffByHunk) && finding.changedLine === true; - const anchorsValid = packet !== undefined && publicationAnchorValid && - (finding.mergedAnchors ?? []).every((anchor) => { - const owner = [...packetById.values()].find((candidatePacket) => candidatePacket.hunks.some((hunk) => hunk.hunkId === anchor.hunkId)); - return owner !== undefined && validChangedLineAnchor(anchor, owner, diffByHunk); - }); - const selections = mergedIds.flatMap((id) => selectionById.get(id) ?? []); - const decisionsValid = finding.publication === "suppressed" - ? selections.length === mergedIds.length && selections.every((selection) => selection.decision === "suppressed") - : selections.length === mergedIds.length && selections.filter((selection) => selection.decision === "published" && selection.findingId === finding.id).length === 1 && - selections.every((selection) => selection.findingId === finding.id - ? selection.decision === "published" - : selection.decision === "merged" && selection.mergedIntoFingerprint === finding.fingerprint); - const reconstructed = reconstructedFinalById.get(finding.id); - const withoutMergedAnchors = (value: FinalFinding): Omit => { - const { mergedAnchors: _anchors, ...rest } = value; - return rest; - }; - const exactFinalMetadata = reconstructed !== undefined && reconstructed.id === finding.id && - stableJson(withoutMergedAnchors(reconstructed)) === stableJson(withoutMergedAnchors(finding)) && - (finding.mergedAnchors ?? []).every((anchor) => (reconstructed.mergedAnchors ?? []) - .some((expected) => stableJson(expected) === stableJson(anchor))); - relationsValid &&= representative !== undefined && stableJson(representative.producedBy) === stableJson(finding.producedBy) && - finding.path === packet?.path && mergedValid && anchorsValid && decisionsValid && exactFinalMetadata; - } - - const publicationById = new Map(selectionArtifact.publicationAnchors.map((record) => [record.findingId, record])); - relationsValid &&= publicationById.size === selectionArtifact.publicationAnchors.length && - stableJson(selectionArtifact.publicationAnchors) === stableJson(composerPolicy.publicationAnchors) && - publicationById.size === execution.finalFindings.length && execution.finalFindings.every((finding) => { - const record = publicationById.get(finding.id); - if (record === undefined || record.fingerprint !== finding.fingerprint || record.publication !== finding.publication || - stableJson(record.anchor) !== stableJson(finding.anchor) || record.reason.trim().length === 0) { - return false; - } - if (record.source === "none") { - return record.sourceFindingId === undefined && record.anchor === undefined; - } - if (record.sourceFindingId === undefined || !finding.mergedCandidateIds.includes(record.sourceFindingId) || record.anchor === undefined) { - return false; - } - return record.source === "selected" ? record.sourceFindingId === finding.id : record.sourceFindingId !== finding.id; - }); - - const confidenceIds = selectionArtifact.confidenceSelections.map((record) => record.findingId); - relationsValid &&= unique(confidenceIds).length === confidenceIds.length && - stableJson(selectionArtifact.confidenceSelections) === stableJson(composerPolicy.confidenceSelections) && - selectionArtifact.confidenceSelections.every((record) => { - const finding = execution.finalFindings.find((candidate) => candidate.id === record.findingId); - const representative = verifiedById.get(record.findingId); - const sourceValid = record.reason === "representative" - ? record.sourceFindingId === undefined - : record.sourceFindingId !== undefined && record.sourceFindingId !== record.findingId; - return finding !== undefined && representative !== undefined && record.confidence === finding.confidence && sourceValid && - record.representativeConfidence === representative.confidence && - (record.sourceFindingId === undefined || finding.mergedCandidateIds.includes(record.sourceFindingId)); - }); - - const groupedIds = selectionArtifact.groups.flatMap((group) => group.findingIds); - relationsValid &&= unique(selectionArtifact.groups.map((group) => group.fingerprint)).length === selectionArtifact.groups.length && - unique(groupedIds).length === groupedIds.length && - stableJson(selectionArtifact.groups) === stableJson(composerEvidence.groups.map(({ representativeId: _representativeId, ...group }) => group)) && - stableJson(composerEvidence.pretrimSuppressedIds.sort()) === stableJson(execution.finalSelection - .filter((record) => record.decision === "suppressed" && record.reason === "composer-pre-trim") - .map((record) => record.findingId).sort()); - - if (!relationsValid) { - failures.push(failure("paid_evidence_relations", `${label}/repeat ${execution.repeat} candidate, verification, selection, and final evidence is not fully relational`, { - candidates: execution.candidateFindings.length, - verification: execution.verification.length, - selections: execution.finalSelection.length, - finals: execution.finalFindings.length - })); - } - - const recomputed = recomputeEvidenceScore(run, execution); - if (stableJson(execution.score) !== stableJson(recomputed)) { - failures.push(failure("paid_evidence_score_reconstruction", `${label}/repeat ${execution.repeat} score does not exactly reconstruct from relational artifacts`, { - persisted: valueFingerprint(execution.score), - reconstructed: valueFingerprint(recomputed) - })); - } - const providerCalls = execution.modelCalls.filter((call) => call.cacheStatus !== "hit"); - const callCostUSD = sum(providerCalls.map((call) => call.costUSD ?? Number.NaN)); - if (providerCalls.some((call) => call.costUSD === undefined || !Number.isFinite(call.costUSD)) || - !Number.isFinite(callCostUSD) || Math.abs((execution.score.metrics.costUSD ?? 0) - callCostUSD) > 1e-9) { - failures.push(failure("evidence_cost_accounting", `${label}/repeat ${execution.repeat} score cost does not exactly reconcile with finite model-call cost evidence`, { - persistedCostUSD: execution.score.metrics.costUSD, - modelCallCostUSD: callCostUSD - })); - } - return recomputed; -} - -function validateRunEvidence(run: EvalCaseRunInput, label: string, failures: ReportFailure[]): EvalScore[] { - const canonicalRepeats = run.executions.map((execution) => execution.repeat); - if (stableJson(canonicalRepeats) !== stableJson(Array.from({ length: run.executions.length }, (_, index) => index + 1))) { - failures.push(failure("paid_evidence_repeat_order", `${label} repeat evidence is not in canonical repeats/ order`)); - } - const scores = run.executions.map((execution) => validateExecutionEvidence(run, execution, label, failures)); - if (scores.length === 1) { - if (stableJson(run.info.score) !== stableJson(scores[0]!)) { - failures.push(failure("paid_evidence_aggregate", `${label} info score does not reconstruct from its execution evidence`)); - } - return scores; - } - const recomputed = aggregateRepeatScores(run.declaredCase, run.executions.map((execution, index) => ({ - runDir: `repeats/${execution.repeat}`, - score: scores[index]!, - artifacts: reconstructEvidenceArtifacts(execution) - }))); - if (run.info.repeats === undefined || stableJson(run.info.repeats) !== stableJson(recomputed.aggregate) || - stableJson(run.info.score) !== stableJson(recomputed.score)) { - failures.push(failure("paid_evidence_aggregate", `${label} repeat aggregate does not reconstruct exactly from per-repeat evidence`)); - } - return scores; -} - -export function validateReplayArtifacts( - runId: string, - summary: z.infer, - diff: UnifiedDiff, - filterDecisions: FileFilterDecision[], - fileFacts: FileFacts[], - plan: ReviewPlan, - dossier: PlannerDossier, - runInfo: Record -): { base: string; head: string } { - assertReport(dossier.runId === runId && runInfo.runId === runId, "replay_run_join", `recorded replay artifacts disagree on run ID for ${runId}`); - const review = isRecord(runInfo.review) ? runInfo.review : {}; - const baseRefs = sortedUnique([ - summary.baseRef, - summary.mergeBase, - summary.startCommit, - summary.pr?.baseSha, - dossier.target.baseRef, - dossier.target.mergeBase, - typeof review.baseRef === "string" ? review.baseRef : undefined, - typeof review.mergeBase === "string" ? review.mergeBase : undefined - ].filter((value): value is string => value !== undefined)); - const headRefs = sortedUnique([ - summary.headRef, - summary.headSha, - summary.endCommit, - summary.pr?.headSha, - dossier.target.headRef, - dossier.target.headSha, - typeof review.headRef === "string" ? review.headRef : undefined, - typeof review.headSha === "string" ? review.headSha : undefined - ].filter((value): value is string => value !== undefined)); - assertReport(baseRefs.length === 1 && headRefs.length === 1, "replay_ref_join", `recorded replay artifacts disagree on base/head refs for ${runId}`, { - baseRefHashes: baseRefs.map((value) => sha256Hex(value)), - headRefHashes: headRefs.map((value) => sha256Hex(value)) - }); - const base = baseRefs[0]; - const head = headRefs[0]; - assertReport(base !== undefined && head !== undefined, "missing_refs", `recorded run ${runId} does not contain explicit base/head refs`); - - const diffPaths = diff.files.map((file) => file.path); - const diffHunks = diff.files.flatMap((file) => file.hunks); - assertReport(unique(diffPaths).length === diffPaths.length, "duplicate_diff_path", `recorded diff contains duplicate file paths for ${runId}`); - assertReport(unique(diffHunks.map((hunk) => hunk.id)).length === diffHunks.length, "duplicate_diff_hunk", `recorded diff contains duplicate hunk IDs for ${runId}`); - assertReport(diff.files.every((file) => file.hunks.every((hunk) => hunk.path === file.path)), "diff_path_join", `recorded diff hunk/file paths disagree for ${runId}`); - - const decisionPaths = filterDecisions.map((decision) => decision.path); - assertReport(unique(decisionPaths).length === decisionPaths.length && stableJson([...decisionPaths].sort()) === stableJson([...diffPaths].sort()), "filter_diff_join", `filter decisions do not form an exact join with the recorded diff for ${runId}`); - const keptPaths = filterDecisions.filter((decision) => decision.action === "keep").map((decision) => decision.path).sort(); - const factPaths = fileFacts.map((facts) => facts.path); - assertReport(unique(factPaths).length === factPaths.length && stableJson([...factPaths].sort()) === stableJson(keptPaths), "facts_filter_join", `file facts do not form an exact join with kept files for ${runId}`); - const diffByPath = new Map(diff.files.map((file) => [file.path, file])); - assertReport(fileFacts.every((facts) => { - const file = diffByPath.get(facts.path); - return file !== undefined && facts.hunkCount === file.hunks.length && facts.changedLines === sum(file.hunks.flatMap((hunk) => hunk.lines).map((line) => line.kind === "context" ? 0 : 1)); - }), "facts_diff_join", `file-fact counts disagree with the recorded diff for ${runId}`); - - const diffHunkById = new Map(diffHunks.map((hunk) => [hunk.id, hunk])); - const coverageIds = plan.coverage.map((decision) => decision.hunkId); - assertReport(unique(coverageIds).length === coverageIds.length, "plan_diff_join", `review plan contains duplicate hunk IDs for ${runId}`); - assertReport(plan.coverage.every((decision) => { - const hunk = diffHunkById.get(decision.hunkId); - return hunk !== undefined && hunk.path === decision.path && keptPaths.includes(decision.path); - }), "plan_diff_join", `review plan references unknown or filtered hunks for ${runId}`); - return { base, head }; -} - -async function replayOne(repo: string, runDir: string): Promise { - const runId = path.basename(runDir); - const summary = await loadValidatedJson(path.join(runDir, "stages", "01-input", "resolved-input.json"), "resolved input", resolvedInputSchema); - const diff = await loadValidatedJson(path.join(runDir, "stages", "02-diff", "diff.json"), "recorded diff", unifiedDiffSchema) as unknown as UnifiedDiff; - const filterDecisions = await loadValidatedJson(path.join(runDir, "stages", "02-diff", "file-filter-decisions.json"), "file filter decisions", z.array(fileFilterDecisionSchema)) as FileFilterDecision[]; - const fileFacts = await loadValidatedJson(path.join(runDir, "stages", "03-classify", "file-facts.json"), "file facts", z.array(fileFactsSchema)) as FileFacts[]; - const plan = await loadValidatedJson(path.join(runDir, "stages", "05-planner", "review-plan.json"), "review plan", reviewPlanSchema) as unknown as ReviewPlan; - const dossier = await loadValidatedJson(path.join(runDir, "stages", "05-planner", "planner-dossier.json"), "planner dossier", plannerDossierSchema) as unknown as PlannerDossier; - const runInfo = await loadValidatedJson(path.join(runDir, "run.json"), "run metadata", runMetadataSchema) as Record; - const refs = validateReplayArtifacts(runId, summary, diff, filterDecisions, fileFacts, plan, dossier, runInfo); - const { base, head } = refs; - if (runId.includes("dca8d870")) { - assertReport( - base === "d1c49bdf6a8002ec2ec27faac94a932d736532b2" && head === "fbb5f8761c2c296e115af17e919a7c35d9de8373", - "stale_replay_refs", - "motivating run refs differ from Plan 102", - { base, head } - ); - } - const git = createGitClient(repo); - assertReport(await git.commitExists(base), "missing_ref", `base ref ${base} does not exist in ${repo}`); - assertReport(await git.commitExists(head), "missing_ref", `head ref ${head} does not exist in ${repo}`); - const actualBase = await git.revParse(base); - const actualHead = await git.revParse(head); - const rawDiff = await git.diff(actualBase, actualHead); - assertReport(rawDiff.length === summary.rawDiffChars, "recorded_diff_mismatch", `rebuilt diff length differs for ${runId}`, { - recorded: summary.rawDiffChars, - actual: rawDiff.length - }); - const rebuiltDiff = stripCredentials(parseDiff(rawDiff)) as UnifiedDiff; - validateRecordedDiffParity(runId, diff, rebuiltDiff); - const tempRoot = await mkdtemp(path.join(tmpdir(), "plan102-packet-replay-")); - const worktree = path.join(tempRoot, "head"); - let worktreeAttempted = false; - try { - // Set this before invoking git: worktree add can fail after partially - // registering or creating the target path. - worktreeAttempted = true; - await runGit(repo, ["worktree", "add", "--detach", worktree, actualHead], { errorCode: "invalid_args" }); - const resolved: ResolvedReviewInput = { - ...(summary as Omit), - repoRoot: worktree, - baseRef: actualBase, - mergeBase: actualBase, - headRef: actualHead, - headSha: actualHead, - commits: summary.commits as ResolvedReviewInput["commits"], - rawDiff - }; - const keptPaths = new Set(filterDecisions.filter((decision) => decision.action === "keep").map((decision) => decision.path)); - const kept = diff.files.filter((file) => keptPaths.has(file.path)); - const config = replayConfig(dossier, runInfo); - const indexCapture = createCaptureTelemetry(`${runId}-index`); - const repoIndex = await buildRepositoryIndex(resolved, kept, fileFacts, config, indexCapture.telemetry); - const enabledLenses = dossier.lenses.map((lens) => lens.id); - const reviewContext = packetReviewContextFromDossier(dossier); - const offCapture = createCaptureTelemetry(`${runId}-off`); - const offConfig = structuredClone(config) as CodegenieConfig; - offConfig.review.packSameFileHunks = false; - offConfig.review.packedToolBudgetMode = "base"; - const offPackets = await buildReviewPackets(plan, kept, fileFacts, repoIndex, offCapture.telemetry, { - config: offConfig, - enabledLenses, - reviewContext - }); - const artifactOffPackets = stripCredentials(offPackets); - const onCapture = createCaptureTelemetry(`${runId}-on`); - const onConfig = structuredClone(config) as CodegenieConfig; - onConfig.review.packSameFileHunks = true; - onConfig.review.packedToolBudgetMode = "base"; - const onPackets = await buildReviewPackets(plan, kept, fileFacts, repoIndex, onCapture.telemetry, { - config: onConfig, - enabledLenses, - reviewContext - }); - const artifactOnPackets = stripCredentials(onPackets); - const recordedPackets = await loadPackets(path.join(runDir, "stages", "06-packets", "packets"), { allowLegacyDispatchRank: true }); - const flagOffParity = historicalFlagOffParityView(runId, recordedPackets, artifactOffPackets); - return analyzeReplayComparison({ - runId, - recordedPackets, - offPackets: artifactOffPackets, - onPackets: artifactOnPackets, - onEvents: onCapture.captured.events, - fileFacts, - diff, - plan, - expectedRefs: { base, head }, - actualRefs: { base: actualBase, head: actualHead }, - flagOffParityPackets: flagOffParity.packets, - flagOffParityMigrations: flagOffParity.migrations, - modelCallsObserved: indexCapture.captured.modelCalls.length + offCapture.captured.modelCalls.length + onCapture.captured.modelCalls.length - }); - } finally { - await finalizeReplayCleanup( - runId, - worktreeAttempted, - async () => runGit(repo, ["worktree", "remove", "--force", worktree], { errorCode: "invalid_args" }).then(() => undefined), - async () => rm(tempRoot, { recursive: true, force: true }), - async () => { - const listing = await runGit(repo, ["worktree", "list", "--porcelain"], { errorCode: "invalid_args" }); - const registered = listing.split("\n").some((line) => line === `worktree ${worktree}`); - return !existsSync(worktree) && !registered; - } - ); - } -} - -export async function finalizeReplayCleanup( - runId: string, - worktreeAttempted: boolean, - removeWorktree: () => Promise, - removeTempRoot: () => Promise, - verifyWorktreeRemoved: () => Promise -): Promise { - const errors: unknown[] = []; - let removalError: unknown; - if (worktreeAttempted) { - try { - await removeWorktree(); - } catch (error) { - removalError = error; - } - } - try { - await removeTempRoot(); - } catch (error) { - errors.push(error); - } - if (worktreeAttempted) { - try { - if (!await verifyWorktreeRemoved()) { - errors.push(removalError ?? new Error("worktree remains registered or present on disk")); - } - } catch (error) { - errors.push(error); - } - } - if (errors.length > 0) { - fail("replay_cleanup_failed", `failed to remove replay worktree for ${runId}`, { - errors: errors.map((error) => valueFingerprint(error instanceof Error ? error.message : String(error))) - }); - } -} - -function replayConfig(dossier: PlannerDossier, runInfo: Record): CodegenieConfig { - const config = structuredClone(defaultConfig) as CodegenieConfig; - if (dossier.depth === "light" || dossier.depth === "normal" || dossier.depth === "deep") { - config.review.depth = dossier.depth; - } - if (isRecord(runInfo.review)) { - const boost = asFiniteNumber(runInfo.review.budgetBoost); - if (boost !== undefined && boost > 0) { - config.review.budgetBoost = boost; - } - } - return config; -} - -type ParsedArgs = { - mode: "replay" | "eval" | "regression"; - repo?: string; - runs: string[]; - logs?: string; - baselineLogs?: string; - selectedLogs?: string; - cohort: string; - expectedRepeats?: number; - actualValidationCostUSD?: number; - output: string; -}; - -function parseArgs(argv: string[]): ParsedArgs { - const mode = argv[0]; - assertReport(mode === "replay" || mode === "eval" || mode === "regression", "invalid_args", "expected replay, eval, or regression mode"); - const values = new Map(); - for (let index = 1; index < argv.length; index += 1) { - const key = argv[index]; - assertReport(key?.startsWith("--") === true, "invalid_args", `unexpected argument: ${key ?? ""}`); - const value = argv[index + 1]; - assertReport(value !== undefined && !value.startsWith("--"), "invalid_args", `missing value for ${key}`); - values.set(key, [...(values.get(key) ?? []), value]); - index += 1; - } - const allowedKeys: Record> = { - replay: new Set(["--repo", "--run", "--output"]), - eval: new Set(["--logs", "--cohort", "--expected-repeats", "--actual-validation-cost", "--output"]), - regression: new Set(["--baseline-logs", "--selected-logs", "--cohort", "--expected-repeats", "--output"]) - }; - const unknownKeys = [...values.keys()].filter((key) => !allowedKeys[mode].has(key)); - assertReport(unknownKeys.length === 0, "invalid_args", `${mode} received unsupported arguments`, { unknownKeys }); - const single = (key: string): string | undefined => { - const entries = values.get(key) ?? []; - assertReport(entries.length <= 1, "invalid_args", `${key} may be specified only once`); - return entries[0]; - }; - const output = single("--output"); - assertReport(output !== undefined, "invalid_args", "--output is required"); - const expectedRepeatsText = single("--expected-repeats"); - const expectedRepeats = expectedRepeatsText === undefined ? undefined : Number(expectedRepeatsText); - if (expectedRepeats !== undefined) { - assertReport(Number.isInteger(expectedRepeats) && expectedRepeats > 0, "invalid_args", "--expected-repeats must be a positive integer"); - } - const validationText = single("--actual-validation-cost"); - const actualValidationCostUSD = validationText === undefined ? undefined : Number(validationText); - if (actualValidationCostUSD !== undefined) { - assertReport(Number.isFinite(actualValidationCostUSD) && actualValidationCostUSD >= 0, "invalid_args", "--actual-validation-cost must be non-negative"); - } - const repo = single("--repo"); - const logs = single("--logs"); - const baselineLogs = single("--baseline-logs"); - const selectedLogs = single("--selected-logs"); - return { - mode, - ...(repo === undefined ? {} : { repo }), - runs: values.get("--run") ?? [], - ...(logs === undefined ? {} : { logs }), - ...(baselineLogs === undefined ? {} : { baselineLogs }), - ...(selectedLogs === undefined ? {} : { selectedLogs }), - cohort: single("--cohort") ?? "latest", - ...(expectedRepeats === undefined ? {} : { expectedRepeats }), - ...(actualValidationCostUSD === undefined ? {} : { actualValidationCostUSD }), - output - }; -} - -async function writeReport(output: string, report: unknown): Promise { - await writeFile(output, formatSafeReport(report)); -} - -export function formatSafeReport(report: unknown): string { - return `${JSON.stringify(stripCredentials(safeReportOutput(report)), null, 2)}\n`; -} - -function safeReportOutput(value: unknown): unknown { - if (Array.isArray(value)) { - return value.map(safeReportOutput); - } - if (!isRecord(value)) { - return value; - } - const output: Record = {}; - for (const [key, entry] of Object.entries(value)) { - if (key === "failures" && Array.isArray(entry)) { - output[key] = entry.map(safeFailureOutput); - } else { - output[key] = safeReportOutput(entry); - } - } - return output; -} - -function safeFailureOutput(value: unknown): Record { - if (!isRecord(value)) { - const fingerprint = valueFingerprint(value); - return { code: "invalid_failure", message: `sha256:${fingerprint.sha256}:${fingerprint.length}` }; - } - const code = typeof value.code === "string" ? value.code : "invalid_failure"; - const message = typeof value.message === "string" ? value.message : stableJson(value.message); - const fingerprint = valueFingerprint(message); - return { - code, - message: `sha256:${fingerprint.sha256}:${fingerprint.length}`, - ...(value.context === undefined ? {} : { context: safeFailureValue(value.context) }) - }; -} - -function safeFailureValue(value: unknown): unknown { - if (typeof value === "string") { - return valueFingerprint(value); - } - if (Array.isArray(value)) { - return value.map(safeFailureValue); - } - if (isRecord(value)) { - if ( - (value.kind === "string" || value.kind === "json") && - typeof value.length === "number" && Number.isInteger(value.length) && value.length >= 0 && - typeof value.sha256 === "string" && /^[a-f0-9]{64}$/u.test(value.sha256) && - Object.keys(value).every((key) => key === "kind" || key === "length" || key === "sha256") - ) { - return value; - } - return Object.fromEntries(Object.entries(value).map(([key, entry]) => [key, safeFailureValue(entry)])); - } - return value; -} - -function failuresForError(error: unknown, code: string): ReportFailure[] { - return error instanceof PacketPackingReportError - ? error.failures - : [failure(code, error instanceof Error ? error.message : String(error))]; -} - -export async function runPacketPackingReportCli(argv: string[]): Promise { - const fallbackOutput = rawOutputArgument(argv); - try { - return await runPacketPackingReportCliUnchecked(argv); - } catch (error) { - const mode = argv[0] === "replay" || argv[0] === "eval" || argv[0] === "regression" ? argv[0] : "unknown"; - const report = { schemaVersion: 1, mode, failures: failuresForError(error, "packet_report_error") }; - if (fallbackOutput !== undefined) { - try { - await writeReport(fallbackOutput, report); - return 1; - } catch (writeError) { - process.stderr.write(formatSafeReport({ - schemaVersion: 1, - mode, - failures: [...failuresForError(error, "packet_report_error"), ...failuresForError(writeError, "report_write_error")] - })); - return 1; - } - } - process.stderr.write(formatSafeReport(report)); - return 1; - } -} - -function rawOutputArgument(argv: string[]): string | undefined { - const indexes = argv.flatMap((entry, index) => entry === "--output" ? [index] : []); - if (indexes.length !== 1) { - return undefined; - } - const value = argv[indexes[0]! + 1]; - return value !== undefined && !value.startsWith("--") ? value : undefined; -} - -async function runPacketPackingReportCliUnchecked(argv: string[]): Promise { - const args = parseArgs(argv); - if (args.mode === "replay") { - assertReport(args.repo !== undefined, "invalid_args", "replay requires --repo"); - assertReport(args.runs.length > 0, "invalid_args", "replay requires at least one --run"); - const rows: ReplayRow[] = []; - const failures: ReportFailure[] = []; - for (const run of args.runs) { - try { - const row = await replayOne(path.resolve(args.repo), path.resolve(run)); - rows.push(row); - failures.push(...row.failures.map((entry) => ({ ...entry, context: { run: path.basename(run), ...entry.context } }))); - } catch (error) { - const entries = error instanceof PacketPackingReportError - ? error.failures - : [failure("replay_error", error instanceof Error ? error.message : String(error))]; - failures.push(...entries.map((entry) => ({ ...entry, context: { run: path.basename(run), ...entry.context } }))); - } - } - const report: ReplayReport = { - schemaVersion: 1, - mode: "replay", - noModelCalls: rows.length === args.runs.length && rows.every((row) => row.modelCallsObserved === 0), - repo: path.resolve(args.repo), - rows, - failures - }; - await writeReport(args.output, report); - return failures.length === 0 && rows.length === args.runs.length ? 0 : 1; - } - - assertReport(args.expectedRepeats !== undefined, "invalid_args", `${args.mode} requires --expected-repeats`); - if (args.mode === "eval") { - assertReport(args.logs !== undefined, "invalid_args", "eval requires --logs"); - try { - const cohort = selectExplicitCohort(await loadEvalRuns(path.resolve(args.logs), args.cohort), args.cohort); - const report = analyzeEvalCohort(cohort, args.expectedRepeats, { - ...(args.actualValidationCostUSD === undefined ? {} : { actualValidationCostUSD: args.actualValidationCostUSD }) - }); - await writeReport(args.output, report); - return report.failures.length === 0 ? 0 : 1; - } catch (error) { - await writeReport(args.output, { schemaVersion: 1, mode: "eval", failures: failuresForError(error, "eval_report_error") }); - return 1; - } - } - - assertReport(args.baselineLogs !== undefined && args.selectedLogs !== undefined, "invalid_args", "regression requires --baseline-logs and --selected-logs"); - try { - const baseline = selectExplicitCohort(await loadEvalRuns(path.resolve(args.baselineLogs), args.cohort), args.cohort); - const selected = selectExplicitCohort(await loadEvalRuns(path.resolve(args.selectedLogs), args.cohort), args.cohort); - const report = analyzeRegressionCohorts(baseline, selected, args.expectedRepeats); - await writeReport(args.output, report); - return report.failures.length === 0 ? 0 : 1; - } catch (error) { - await writeReport(args.output, { schemaVersion: 1, mode: "regression", failures: failuresForError(error, "regression_report_error") }); - return 1; - } -} - -const invokedPath = process.argv[1] === undefined ? undefined : pathToFileURL(path.resolve(process.argv[1])).href; -if (invokedPath === import.meta.url) { - runPacketPackingReportCli(process.argv.slice(2)).then((exitCode) => { - process.exitCode = exitCode; - }).catch((error) => { - process.stderr.write(formatSafeReport({ schemaVersion: 1, mode: "unknown", failures: failuresForError(error, "packet_report_error") })); - process.exitCode = 1; - }); -} diff --git a/specs/plans/102-issue-102-same-file-packet-packing.md b/specs/plans/102-issue-102-same-file-packet-packing.md index 94cc461..048cc8e 100644 --- a/specs/plans/102-issue-102-same-file-packet-packing.md +++ b/specs/plans/102-issue-102-same-file-packet-packing.md @@ -1,11 +1,11 @@ # Issue 102: Same-File Packet Packing -Status: PENDING +Status: COMPLETE — failed treatment gate; baseline restored Related: Plan 100 (COMPLETE; dispatch rank), Plans 40/44 (recall calibration), Plan 79 (repeat/recall harness) Planned from: production run `.codegenie/runs/20260724-184952-dca8d870` against `0xsequence/trails-api` PR 846 (88 files, 217 hunks, `--max-time 60`, concurrency 4), plus retained runs `740d73f2`, `fe1548ae`, and `81f806a6`, 2026-07-24 Production replay refs: base/merge-base `d1c49bdf6a8002ec2ec27faac94a932d736532b2`; head `fbb5f8761c2c296e115af17e919a7c35d9de8373` Planned at: commit `6909e1a` (branch `next`) -Recommended priority: next throughput plan. Plan 100 is complete, and run `dca8d870` is the clean planner-survival/dispatch-order baseline: 32/32 planner entries survived and all 19 deep hunks were dispatched. Its dependency is satisfied. +Final outcome: the repaired one-repeat treatment gate failed, so same-file packet packing did not roll out. Step 12 restored the baseline packet path and removed the experiment; the immutable reports and paid logs retain the rejected design's measurement record. > Executor instructions: preserve the output of today's semantic hunk grouper as indivisible **atoms**. Do not replace `canJoinGroup` with an affinity sort: proximity is not transitive, and a sort cannot preserve its semantics. Never combine atoms with different effective coverage levels. Preserve each atom's standalone review profile as a monotonic floor when packing internalizes relationship context. Keep source order, `MAX_HUNKS_PER_PACKET = 5`, and `MAX_PATCH_CHARS = 12_000`. Land packing dark, validate deterministic packet shape before spending model calls, then record treatment for every paid execution and require at least 8/10 treated B/C executions per arm/case. Use paired repeated A/B/C evals to select the production behavior and tool-budget calculation, preserve the paid evidence, then remove every experiment-only flag and alternate path in the dedicated teardown step. > @@ -108,6 +108,35 @@ The failed retry report is preserved byte-for-byte at `/home/peter/Dev/0xPolygon Plan 102 steps 8–11 are `not_run` because the repaired retry gate failed. Step 12's failure teardown is also `not_run` in Phase 4 and is reserved for the next separately reviewed phase; no case was changed to repeat 10, no arm was selected, no collateral or production suite ran, no rollout verdict was made, and no experiment code or evidence was torn down in this phase. +### Phase 5 failed-outcome teardown and final gate (2026-07-25) + +Plan 102 is complete through the explicit failed-outcome branch, not through rollout. The repaired retry never established a valid preflight: consistency B/C each treated the target `0/1`, and independently sampled planner hints prevented exact A-to-B/C standalone-profile reconciliation in both case families. No report condition was waived and no third paid attempt was made. The product decision is therefore to reject this packing design for this iteration and restore the original baseline path. + +Before teardown, every JSON report actually produced by the completed phases was preserved under `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/reports/`: + +| Preserved report | SHA-256 | +| --- | --- | +| `plan102-packet-shape.json` | `77af0c38937bd6957806f05ad201cbad32461414c1d667824096208473c276fa` | +| `plan102-eval-preflight-invalid-ace65769.json` | `3368178d57d6c4c46865fccb52b8aa160d1f9acf3536be9e1cde4d05d52c50ab` | +| `plan102-eval-preflight-retry-invalid-5bd80f2c.json` | `650113d24d092e6fd712303e1828b297010c76f672d7efb799f5081f79635517` | + +`reports/manifest.sha256`, itself SHA-256 `7c6a1dfb923aea7a36986e50c40068eb878688611f22bc31e4c3c20128970667`, covers exactly those three report basenames. `sha256sum -c manifest.sha256` passes for all three. The deterministic replay was copied byte-for-byte from `/tmp/plan102-packet-shape.json`; the two existing invalid-preflight reports were hash-verified and never rewritten. No repeat-10, collateral, or production-capacity report exists because those phases were not reached. + +Both paid invocations use the immutable log root `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/recall/logs` and retain their invocation manifests and run artifacts unchanged: + +| Invocation UUID | Eval runs | Review run IDs | Authoritative cost USD | +| --- | --- | --- | ---: | +| `ace65769-3a68-4bd1-bd36-662fa6827dc6` | 1–6 | `20260725-120233-6c490c44`, `20260725-120706-36842f4b`, `20260725-121006-87a8ed77`, `20260725-121205-5a020015`, `20260725-121737-6249976f`, `20260725-121917-4c7e8289` | 5.207657 | +| `5bd80f2c-865e-40f0-a605-07387138b904` | 7–12 | `20260725-131014-ff7643a9`, `20260725-131217-78051528`, `20260725-131407-46ee2073`, `20260725-131627-fd970500`, `20260725-131904-9623b816`, `20260725-132212-896d115e` | 3.986221 | + +Final cumulative `actualValidationCostUSD` is `$9.193878`, below the owner-approved `approvedValidationCostUSD: $500`. There is no projected remaining validation spend: step 8 (repeat-10 recall), step 9 (deterministic and real-model collateral), step 10 (production-capacity pair), and step 11 (arm selection and rollout documentation) are all `not_run` because the repaired step-7 treatment gate failed. Their absent reports and `$0` later-phase spend are intentional, not missing evidence. No paid model call occurred during teardown or the final repository gate. + +Step 12 completed through the failure branch. The same-file packing pass, atom-scaled and base experiment modes, both temporary config fields and eval/config plumbing, experiment-only atom wrappers/IDs/provenance/profile floor/telemetry, report script and tests, and dead artifact-reconstruction exports/metadata were removed. `scripts/`, `src/`, and `tests/` now match the pre-experiment product baseline at commit `5aca256` for every Plan 102 code path, restoring the original `hunkFirstGroups()` packet behavior and ordinary packet tool budget rather than retaining a dark alternate path. No golden packet-packing fixture was created. + +The private active suite no longer contains A/B/C case YAMLs. B/C declarations were retired with the experiment. `consistency-a` was also removed because its repaired baseline run met `0/2` declared expectations, and `dilution-a` was removed because it met only `1/2`; neither satisfied the suite's existing all-expectations-pass policy. Both fixture repositories under `recall/repos/`, all twelve run directories, both invocation manifests, and every log/report artifact remain unchanged. No production or collateral case was created merely for teardown. + +The exact step-12 evidence, retired-field grep, report-absence, strict-suite no-model parse, and focused baseline tests pass. Step 13's complete `pnpm run check`, `pnpm test`, and `pnpm build` gate plus `git diff --check` also pass. This completed status records a failed experiment with verified baseline restoration; it does not claim rollout, recall safety, throughput gain, or production economics. + ## Phase 2 deterministic reconciliation (2026-07-24) Step 6's exact four-run command completed with exit `0`, `noModelCalls: true`, four explicit `modelCallsObserved: 0` row proofs, and no report failures. The frozen output is `/tmp/plan102-packet-shape.json` (SHA-256 `77af0c38937bd6957806f05ad201cbad32461414c1d667824096208473c276fa`). The authoritative motivating count is **75**: the real `combinedPatchChars()` builder produced 96→75, so the bounded 74/76 reconciliation was not used. The report's diagnostic reapplication of the old `contentWithLineNumbers` proxy produces 76 for the current atom stream while the real measurement produces 75; this does not change the golden because the authoritative builder result itself matches the plan's historical 75 target. The frozen hash was refreshed after the verifier stopped trusting serialized cap telemetry and independently rendered the source diff for every packet that actually combines multiple atoms. diff --git a/specs/plans/README.md b/specs/plans/README.md index cd876cc..8f3cea8 100644 --- a/specs/plans/README.md +++ b/specs/plans/README.md @@ -105,4 +105,4 @@ This directory tracks implementation plans for confirmed improvements. Status va | 99 | COMPLETE | Issue 99: Plan 98 Correctness Follow-Up — Unicode-Safe Tree-sitter Extraction and Likely-Test Compatibility | | 100 | COMPLETE | Issue 100: Short Hunk IDs — Planner Coverage Survival and Dispatch Resilience | | 101 | IN PROGRESS (paid semantic A/B pending) | Issue 101: Exact Skill Provenance and Evidence-Gated Bundled-Skill Revision | -| 102 | PENDING | Issue 102: Same-File Packet Packing | +| 102 | COMPLETE (failed gate; baseline restored) | Issue 102: Same-File Packet Packing | diff --git a/specs/plans/phase-plans/102/phase_5.md b/specs/plans/phase-plans/102/phase_5.md new file mode 100644 index 0000000..d1b0f3b --- /dev/null +++ b/specs/plans/phase-plans/102/phase_5.md @@ -0,0 +1,33 @@ +--- +status: complete +--- + +# Phase 5: Failed-Gate Teardown and Final Repository Gate + +## Overview + +Complete Plan 102 step 12 through its failed-outcome branch, then run step 13. Preserve every JSON report actually produced by completed phases and prove its immutable checksum manifest before deleting experiment scaffolding. Restore the original single packet-builder path, remove both temporary configuration fields and all experiment-only reporting, atom/profile-floor/provenance/budget/telemetry surfaces, and keep only generally useful hardening that remains independently used and tested. Retire private B/C declarations and retain an A declaration only when its baseline result satisfied the suite's existing expectation policy; keep all private fixture repositories, logs, reports, and invocation manifests unchanged. Reconcile Plan 102 as a complete failed outcome without claiming rollout, and execute the exact teardown checks plus the full repository gate without model calls. + +## Steps + +1. Reconcile the clean Phase 4 commit against pre-experiment commit `5aca256`, classify every experiment-series change as required teardown, dead report support, or independently used hardening, and verify that no unrelated working-tree changes overlap the phase. +2. Before product deletion, stage `/tmp/plan102-packet-shape.json` in the workspace together with a private-evidence application manifest. Verify the deterministic replay hash `77af0c38937bd6957806f05ad201cbad32461414c1d667824096208473c276fa` and both immutable invalid-preflight hashes already under the approved private reports root. Define `manifest.sha256` over exactly those three preserved JSON report basenames, then ask the primary context to mechanically copy the replay and install the manifest without changing any existing report, log, or invocation artifact. +3. Restore the pre-experiment packet-building product path in `src/pipeline/packet-builder.ts`: remove compatible-atom packing, atom-scaled budgets, atom wrappers/IDs/provenance, member-atom profile floors, treatment telemetry, and experiment-only exported helpers or metadata while preserving the original `hunkFirstGroups()` behavior and packet budgets. +4. Remove `packSameFileHunks` and `packedToolBudgetMode` from `src/types.ts`, `src/config/schema.ts`, `src/config/config-loader.ts`, `src/evals/eval-runner.ts`, and every remaining eval/config/runtime artifact path. Remove dead report-only support from eval artifacts/commands, pipeline composition/review/verification, telemetry artifacts, and tests unless a surface remains generally used and independently covered after the report is gone. +5. Delete `scripts/packet-packing-report.ts` and `tests/packet-packing-report.test.ts`; remove all packet-packing-only cases from `tests/pipeline-phase5.test.ts`, `tests/config-loader.test.ts`, and `tests/evals.test.ts`, preserving focused tests for the restored baseline behavior. Confirm no packet-packing golden fixture exists. +6. Stage private suite mutations in the workspace for primary-context mechanical application. Remove `consistency-{b,c}.yml` and `dilution-{b,c}.yml`. Because `consistency-a` failed both declared expectations and `dilution-a` failed its final expectation, remove both active A declarations under the suite's existing pass policy. Do not touch `repos/`, `logs/`, `reports/`, invocation manifests, or create production/collateral cases. +7. Update Plan 102 with the final evidence inventory, both paid invocation UUIDs and all twelve review run IDs, private log/report paths and hashes, final cumulative cost `$9.193878` against the `$500` ceiling, steps 8–11 as `not_run` for the failed repaired retry gate, and step 12 complete through baseline restoration. Mark the plan and plan index `COMPLETE (failed gate; baseline restored)` according to repository conventions and update any affected normative project documentation only where it currently describes the temporary experiment as shipped behavior. +8. Run the exact evidence-manifest checksum check, both live-code/private-active-YAML retired-field greps, strict private suite parsing with no model calls, report script/test absence, and focused baseline tests. Verify the private report directory contains exactly the three produced JSON reports covered by `manifest.sha256` and that fixture repositories, logs, reports, and invocation manifests remain present and unchanged. +9. Run the complete step-13 gate: `pnpm run check`, `pnpm test`, `pnpm build`, and `git diff --check`. Iterate until every check passes and report the failed-outcome teardown ready for review without committing. + +## Tests + +- `evidence manifest`: `sha256sum -c manifest.sha256` passes for exactly the deterministic replay and two invalid one-repeat preflight reports. +- `live-code teardown`: retired config names have no matches under `src`, `scripts`, `tests`, or `evals`; report script and report tests do not exist. +- `private active declarations`: no retired fields remain in active YAML, B/C and failing A declarations are absent, retained suite directories strictly parse when active YAML exists, and no model call is made. +- `focused baseline behavior`: `pnpm test -- tests/pipeline-phase5.test.ts tests/evals.test.ts` passes with the original `hunkFirstGroups()` packet path and ordinary tool-budget behavior. +- `complete repository gate`: `pnpm run check`, `pnpm test`, `pnpm build`, and `git diff --check` all exit zero. + +## Outcome + +The failed-outcome branch is complete. Three produced JSON reports are preserved under the private reports root and pass their exact three-entry manifest; both fixture repositories, all 12 paid run directories, and both invocation manifests remain unchanged. All six active A/B/C declarations were retired because neither A baseline satisfied the suite's all-expectations-pass policy. The live product, configuration, eval, telemetry, report, and test surfaces are restored to the pre-experiment baseline with no dark packing path. Plan 102 and its index record `COMPLETE` as a failed gate with baseline restoration, steps 8–11 remain `not_run`, cumulative spend is `$9.193878` against the `$500` ceiling, and the focused plus complete repository gates pass without model calls. diff --git a/src/config/config-loader.ts b/src/config/config-loader.ts index 0b7490b..bb8d2de 100644 --- a/src/config/config-loader.ts +++ b/src/config/config-loader.ts @@ -64,8 +64,6 @@ const DEFAULT_SOURCE_PATHS = [ "review.maxTime", "review.perPassTimeoutMs", "review.budgetBoost", - "review.packSameFileHunks", - "review.packedToolBudgetMode", "review.maxBudgetTokens", "github.summaryWhenNoFindings", "classification.pathRules", @@ -81,15 +79,7 @@ const DEFAULT_SOURCE_PATHS = [ "eval.logsDir" ]; -const REPO_SAFE_REVIEW_KEYS = new Set([ - "depth", - "maxFindings", - "softCommentCap", - "budgetBoost", - "maxTime", - "packSameFileHunks", - "packedToolBudgetMode" -]); +const REPO_SAFE_REVIEW_KEYS = new Set(["depth", "maxFindings", "softCommentCap", "budgetBoost", "maxTime"]); const CREDENTIAL_KEY_PATTERN = /(?:api[_-]?key|apikey|secret|token|password|passwd|authorization|credentials|auth)/i; export function loadConfig(opts: LoadConfigOptions): LoadedConfig { @@ -264,14 +254,6 @@ function applyRawConfig( config.review.budgetBoost = raw.review.budgetBoost; sources["review.budgetBoost"] = source; } - if (raw.review?.packSameFileHunks !== undefined) { - config.review.packSameFileHunks = raw.review.packSameFileHunks; - sources["review.packSameFileHunks"] = source; - } - if (raw.review?.packedToolBudgetMode !== undefined) { - config.review.packedToolBudgetMode = raw.review.packedToolBudgetMode; - sources["review.packedToolBudgetMode"] = source; - } if (raw.review?.maxBudgetTokens !== undefined) { config.review.maxBudgetTokens = raw.review.maxBudgetTokens; sources["review.maxBudgetTokens"] = source; @@ -383,12 +365,6 @@ function filterRepoConfig(raw: RawCodegenieConfig, warnings: ConfigWarning[]): R if (raw.review.budgetBoost !== undefined) { safe.review.budgetBoost = raw.review.budgetBoost; } - if (raw.review.packSameFileHunks !== undefined) { - safe.review.packSameFileHunks = raw.review.packSameFileHunks; - } - if (raw.review.packedToolBudgetMode !== undefined) { - safe.review.packedToolBudgetMode = raw.review.packedToolBudgetMode; - } if (raw.review.maxTime !== undefined) { safe.review.maxTime = raw.review.maxTime; } diff --git a/src/config/schema.ts b/src/config/schema.ts index 7b639c7..d000099 100644 --- a/src/config/schema.ts +++ b/src/config/schema.ts @@ -6,7 +6,6 @@ export const reasoningLevelSchema = z.enum(["low", "medium", "high", "xhigh"]); export const severitySchema = z.enum(["critical", "high", "medium", "low"]); export const confidenceSchema = z.enum(["high", "medium", "low"]); export const logLevelSchema = z.enum(["debug", "info", "warn", "error"]); -export const packedToolBudgetModeSchema = z.enum(["base", "atom-scaled"]); const positiveIntSchema = z.number().int().positive(); const nonNegativeIntSchema = z.number().int().nonnegative(); @@ -52,8 +51,6 @@ export const rawConfigSchema = z maxTime: reviewMaxTimeMinutesSchema.optional(), perPassTimeoutMs: positiveIntSchema.optional(), budgetBoost: positiveFiniteNumberSchema.optional(), - packSameFileHunks: z.boolean().optional(), - packedToolBudgetMode: packedToolBudgetModeSchema.optional(), maxBudgetTokens: positiveIntSchema.optional(), maxModelCalls: positiveIntSchema.optional(), deepEnsemblePasses: positiveIntSchema.max(MAX_DEEP_ENSEMBLE_PASSES).optional(), @@ -141,8 +138,6 @@ export const codegenieConfigSchema = z maxTimeMs: positiveFiniteNumberSchema.max(MAX_REVIEW_TIME_MS), perPassTimeoutMs: positiveIntSchema, budgetBoost: positiveFiniteNumberSchema, - packSameFileHunks: z.boolean(), - packedToolBudgetMode: packedToolBudgetModeSchema, maxBudgetTokens: positiveIntSchema.optional(), maxModelCalls: positiveIntSchema.optional(), deepEnsemblePasses: positiveIntSchema.max(MAX_DEEP_ENSEMBLE_PASSES).optional(), @@ -214,8 +209,6 @@ export const defaultConfig: CodegenieConfig = { maxTimeMs: 30 * 60 * 1000, perPassTimeoutMs: 8 * 60 * 1000, budgetBoost: 1, - packSameFileHunks: false, - packedToolBudgetMode: "base", // Primary coverage budget (plan 90): work-denominated so provider latency // can never shrink a review. Re-derived 2026-07-04 after run 0c4d5213/53 // (5,921,791 tokens of legitimate work — planner-deep + escalator diff --git a/src/evals/eval-artifacts.ts b/src/evals/eval-artifacts.ts index 007de6d..791e772 100644 --- a/src/evals/eval-artifacts.ts +++ b/src/evals/eval-artifacts.ts @@ -6,7 +6,6 @@ import type { EvalArtifacts, EvalHintEvent, EvalHumanAttentionNote, - EvalInvocationManifest, EvalRunInfo, EvalSelectionRecord, EvalVerificationRecord, @@ -139,16 +138,6 @@ export async function writeEvalRunInfo(dir: string, info: EvalRunInfo): Promise< await rename(tmp, target); } -export async function writeEvalInvocationManifest(logsDir: string, manifest: EvalInvocationManifest): Promise { - const relative = path.join("invocations", `${manifest.invocationId}.json`); - const target = path.join(logsDir, relative); - await mkdir(path.dirname(target), { recursive: true }); - const tmp = `${target}.${process.pid}.${Date.now()}.tmp`; - await writeFile(tmp, `${JSON.stringify(manifest, null, 2)}\n`); - await rename(tmp, target); - return relative; -} - export function resolveTelemetryDir(runOrTelemetryDir: string): string { const resolved = path.resolve(runOrTelemetryDir); if (path.basename(resolved) === "telemetry") { diff --git a/src/evals/eval-command.ts b/src/evals/eval-command.ts index b4a12ee..c26ca1a 100644 --- a/src/evals/eval-command.ts +++ b/src/evals/eval-command.ts @@ -1,12 +1,10 @@ -import { randomUUID } from "node:crypto"; import path from "node:path"; import { Command, CommanderError } from "commander"; import { loadConfig } from "../config/config-loader.js"; -import type { CodegenieConfig, EvalCaseResult, EvalInvocationManifest, EvalLossLabel } from "../types.js"; +import type { CodegenieConfig, EvalCaseResult, EvalLossLabel } from "../types.js"; import { CodegenieError } from "../util/errors.js"; import { CliDisplayExit } from "../cli/review-command.js"; -import { writeEvalInvocationManifest } from "./eval-artifacts.js"; -import { loadEvalSuite, replayFromArtifacts, resolveLogsDir, runEvalCase } from "./eval-runner.js"; +import { loadEvalSuite, replayFromArtifacts, runEvalCase } from "./eval-runner.js"; export type EvalCommandOptions = { evalDir?: string; @@ -61,52 +59,15 @@ export async function runEvalCommand( throw new CodegenieError("invalid_args", "--eval-dir is required when eval.defaultEvalDir is not configured"); } const suite = await loadEvalSuite(evalDir); - const invocationId = randomUUID(); - const startedAt = new Date().toISOString(); - const manifestRelativePath = `invocations/${invocationId}.json`; - const manifest: EvalInvocationManifest = { - schemaVersion: 1, - invocationId, - suiteDir: suite.dir, - status: "running", - startedAt, - cases: suite.cases.map((entry, caseIndex) => ({ - caseIndex, - caseName: entry.evalCase.name, - caseHash: entry.caseHash, - caseFile: entry.file - })), - runs: [] - }; - const manifestRoots = [...new Set(suite.cases.map((entry) => resolveLogsDir(suite.dir, entry.evalCase, config)))]; - const persistManifest = async (): Promise => { - await Promise.all(manifestRoots.map((logsDir) => writeEvalInvocationManifest(logsDir, manifest))); - }; - await persistManifest(); const results: EvalCaseResult[] = []; - for (const [caseIndex, entry] of suite.cases.entries()) { + for (const entry of suite.cases) { const result = await runEvalCase(suite, entry, { config, - invocation: { id: invocationId, caseIndex, manifest: manifestRelativePath }, ...(options.cache !== undefined ? { cacheOverride: options.cache } : {}) }); results.push(result); - const logsRoot = resolveLogsDir(suite.dir, entry.evalCase, config); - const runPath = path.relative(logsRoot, result.runDir); - manifest.runs.push({ - caseIndex, - caseName: entry.evalCase.name, - caseHash: entry.caseHash, - runNumber: result.info.runNumber, - logsRoot, - runPath - }); - await persistManifest(); write(renderCaseResult(result)); } - manifest.status = "complete"; - manifest.completedAt = new Date().toISOString(); - await persistManifest(); write(renderSuiteTotals(results)); return results.every((result) => result.status === "pass") ? 0 : 1; } diff --git a/src/evals/eval-runner.ts b/src/evals/eval-runner.ts index add3e49..8470145 100644 --- a/src/evals/eval-runner.ts +++ b/src/evals/eval-runner.ts @@ -6,7 +6,6 @@ import { z } from "zod"; import { applyRepoConfigLayer } from "../config/config-loader.js"; import { MAX_DEEP_ENSEMBLE_PASSES, - packedToolBudgetModeSchema, reasoningLevelSchema, reviewDepthSchema, reviewMaxTimeMinutesSchema, @@ -50,7 +49,6 @@ export type EvalSuite = { export type EvalRunOptions = { cacheOverride?: boolean; config: CodegenieConfig; - invocation?: EvalRunInfo["invocation"]; }; const positiveNumberSchema = z.number().positive(); @@ -136,8 +134,6 @@ const caseSchema = z maxFindings: positiveIntSchema.optional(), concurrency: positiveIntSchema.optional(), budgetBoost: positiveNumberSchema.optional(), - packSameFileHunks: z.boolean().optional(), - packedToolBudgetMode: packedToolBudgetModeSchema.optional(), maxTimeMinutes: reviewMaxTimeMinutesSchema.optional(), maxBudgetTokens: positiveIntSchema.optional(), deepEnsemblePasses: positiveIntSchema.max(MAX_DEEP_ENSEMBLE_PASSES).optional(), @@ -337,7 +333,6 @@ export async function replayFromArtifacts( finishedAt, score, config: options.config, - ...(options.invocation !== undefined ? { invocation: options.invocation } : {}), replay: { sourceArtifacts: source, caseSource: reread.source }, ...(sourceInfo.caseFile !== undefined ? { caseFile: sourceInfo.caseFile } : {}) }); @@ -355,9 +350,7 @@ export async function replayFromArtifacts( startedAt, error, "replay", - { sourceArtifacts: source, caseSource: reread.source }, - undefined, - options.invocation + { sourceArtifacts: source, caseSource: reread.source } ); } } @@ -421,7 +414,6 @@ async function runArtifactCase( finishedAt, score, config: options.config, - ...(options.invocation !== undefined ? { invocation: options.invocation } : {}), replay: { sourceArtifacts: source, caseSource: "yaml" } }); await writeRunOutputs(allocated.dir, path.dirname(allocated.dir), info, artifacts.finalFindings); @@ -434,9 +426,7 @@ async function runArtifactCase( startedAt, error, "replay", - { sourceArtifacts: source, caseSource: "yaml" }, - undefined, - options.invocation + { sourceArtifacts: source, caseSource: "yaml" } ); } } @@ -499,7 +489,6 @@ async function runLiveCase( finishedAt, score, config: caseConfig.config, - ...(options.invocation !== undefined ? { invocation: options.invocation } : {}), cache: caseConfig.cache, repo: await repoInfo(actualRepoRoot, telemetryDir), ...(reviewRunId !== undefined ? { reviewRunId } : {}) @@ -507,7 +496,7 @@ async function runLiveCase( await writeRunOutputs(allocated.dir, path.dirname(allocated.dir), info, artifacts.finalFindings); return { caseName: info.caseName, runDir: allocated.dir, status: info.score.status, info }; } catch (error) { - return writeErroredCase(allocated, entry, errorConfig, startedAt, error, "live", undefined, errorCache, options.invocation); + return writeErroredCase(allocated, entry, errorConfig, startedAt, error, "live", undefined, errorCache); } } @@ -589,7 +578,6 @@ async function runRepeatedLiveCase( score, repeats: aggregate, config: caseConfig.config, - ...(options.invocation !== undefined ? { invocation: options.invocation } : {}), cache: caseConfig.cache, repo: await repoInfo(actualRepoRoot, path.join(allocated.dir, "repeats", "1", "telemetry")) }); @@ -598,7 +586,7 @@ async function runRepeatedLiveCase( await writeEvalRunInfo(allocated.dir, info); return { caseName: info.caseName, runDir: allocated.dir, status: info.score.status, info }; } catch (error) { - return writeErroredCase(allocated, entry, errorConfig, startedAt, error, "live", undefined, errorCache, options.invocation); + return writeErroredCase(allocated, entry, errorConfig, startedAt, error, "live", undefined, errorCache); } } @@ -610,8 +598,7 @@ async function writeErroredCase( error: unknown, mode: "live" | "replay", replay?: EvalRunInfo["replay"], - cache?: EvalRunInfo["cache"], - invocation?: EvalRunInfo["invocation"] + cache?: EvalRunInfo["cache"] ): Promise { const finishedAt = new Date().toISOString(); const score = errorScore(error); @@ -626,7 +613,6 @@ async function writeErroredCase( finishedAt, score, config, - ...(invocation !== undefined ? { invocation } : {}), ...(cache !== undefined ? { cache } : {}) }); await writeFile(path.join(allocated.dir, "out.log"), `${JSON.stringify({ level: "error", message: score.error?.message, code: score.error?.code })}\n`); @@ -704,13 +690,6 @@ async function loadCaseFile(filePath: string, suiteDir: string): Promise { - return loadCaseFile(path.resolve(filePath), path.resolve(suiteDir)); -} - async function rereadReplayCase( sourceInfo: EvalRunInfo, suiteDir: string @@ -747,7 +726,6 @@ function buildRunInfo(input: { repeats?: EvalRunInfo["repeats"]; repo?: EvalRunInfo["repo"]; reviewRunId?: string; - invocation?: EvalRunInfo["invocation"]; startedAt: string; finishedAt: string; score: EvalScore; @@ -765,7 +743,6 @@ function buildRunInfo(input: { ...(input.repeats !== undefined ? { repeats: input.repeats } : {}), ...(input.repo !== undefined ? { repo: input.repo } : {}), ...(input.reviewRunId !== undefined ? { reviewRunId: input.reviewRunId } : {}), - ...(input.invocation !== undefined ? { invocation: input.invocation } : {}), codegenieRuntime: resolveCodegenieRuntimeProvenance(), cache: input.cache ?? { enabled: input.config.cache.enabled, source: "config", dir: input.config.cache.dir }, effectiveConfig: evalEffectiveConfig(input.config), @@ -836,12 +813,6 @@ function applyCaseReviewConfig( if (review?.budgetBoost !== undefined) { config.review.budgetBoost = review.budgetBoost; } - if (review?.packSameFileHunks !== undefined) { - config.review.packSameFileHunks = review.packSameFileHunks; - } - if (review?.packedToolBudgetMode !== undefined) { - config.review.packedToolBudgetMode = review.packedToolBudgetMode; - } if (review?.maxTimeMinutes !== undefined) { config.review.maxTimeMs = Math.round(review.maxTimeMinutes * 60_000); } @@ -895,14 +866,6 @@ function evalEffectiveConfig(config: CodegenieConfig): NonNullable; - pretrimSuppressedIds: string[]; - publishable: CandidateFinding[]; -} { - const publishable = verified.map((candidate) => candidate.anchorSource === "backfill_packet_representative" && candidate.anchor !== undefined - ? (() => { - const { anchor: _anchor, ...withoutAnchor } = candidate; - return { ...withoutAnchor, changedLine: false }; - })() - : candidate); - const pretrim = pretrimComposerInput(publishable); - const packetsById = new Map(packets.map((packet) => [packet.id, packet])); - return { - groups: groupFindings(pretrim.kept, packetsById).map((group) => ({ - fingerprint: group.fingerprint, - representativeId: group.representative.id, - findingIds: group.findings.map((finding) => finding.id) - })), - pretrimSuppressedIds: pretrim.suppressed.map((finding) => finding.id), - publishable - }; -} - -export function reconstructFinalFindingFromArtifacts( - mergedFindings: CandidateFinding[], - packets: ReviewPacket[], - diff: UnifiedDiff, - finalBody: string, - publication: FinalFinding["publication"] -): { - finding: FinalFinding; - publicationAnchor: PublicationAnchorDecision; - confidenceSelection: ConfidenceSelection; -} { - const representative = canonicalMergedRepresentative(mergedFindings, diff); - const packetsById = new Map(packets.map((packet) => [packet.id, packet])); - const fingerprint = fingerprintFinding(representative, packetsById); - const publicationAnchors = new Map(); - const confidenceSelections = new Map(); - const finding = toFinalFinding( - representative, - fingerprint, - finalBody, - publication, - mergedFindings, - diff, - publicationAnchors, - confidenceSelections - ); - return { - finding, - publicationAnchor: publicationAnchors.get(finding.id) ?? { source: "none", reason: "no publication anchor" }, - confidenceSelection: confidenceSelections.get(finding.id) ?? { confidence: representative.confidence, reason: "representative" } - }; -} - -export type ComposerArtifactPolicyDraft = { - mergedFindings: CandidateFinding[]; - finalBody: string; - requestedPublication: FinalFinding["publication"]; - baseSelection: SelectionRecord[]; -}; - -export function reconstructComposerPolicyFromArtifacts( - drafts: ComposerArtifactPolicyDraft[], - packets: ReviewPacket[], - diff: UnifiedDiff, - config: CodegenieConfig, - verdicts: VerificationVerdict[] -): { - findings: FinalFinding[]; - selection: SelectionRecord[]; - publicationAnchors: Array<{ - findingId: string; - fingerprint: string; - publication: FinalFinding["publication"]; - source: PublicationAnchorDecision["source"]; - reason: string; - sourceFindingId?: string; - anchor?: DiffAnchor; - }>; - confidenceSelections: Array; -} { - const publicationAnchorDecisions = new Map(); - const confidenceSelections = new Map(); - const anchorDowngradeReasons = new Map(); - const packetsById = new Map(packets.map((packet) => [packet.id, packet])); - const baseSelection = new Map(); - const findings = drafts.map((draft) => { - const representative = canonicalMergedRepresentative(draft.mergedFindings, diff); - const finding = toFinalFinding( - representative, - fingerprintFinding(representative, packetsById), - draft.finalBody, - draft.requestedPublication, - draft.mergedFindings, - diff, - publicationAnchorDecisions, - confidenceSelections - ); - recordAnchorDowngrade(finding, draft.requestedPublication, anchorDowngradeReasons); - for (const record of draft.baseSelection) { - baseSelection.set(record.findingId, record); - } - return finding; - }); - const quietTelemetry = { event: () => undefined } as unknown as TelemetryRecorder; - const capped = applyCaps(findings, config, { - lowConfidencePublishableIds: lowConfidencePublishableCandidateIds(verdicts), - telemetry: quietTelemetry - }); - for (const [id, reason] of anchorDowngradeReasons) { - if (!capped.downgradeReasons.has(id)) { - capped.downgradeReasons.set(id, reason); - } - } - return { - findings: capped.findings, - selection: buildSelectionRecords(capped.findings, baseSelection, capped.suppressedReasons, capped.downgradeReasons), - publicationAnchors: publicationAnchorSelectionRecords(capped.findings, publicationAnchorDecisions), - confidenceSelections: [...confidenceSelections.entries()] - .map(([findingId, selection]) => ({ findingId, ...selection })) - .sort((left, right) => left.findingId.localeCompare(right.findingId)) - }; -} - function mergeProximityGroups(groups: FindingGroup[], packetsById: Map): FindingGroup[] { const merged: FindingGroup[] = []; for (const group of groups) { diff --git a/src/pipeline/packet-builder.ts b/src/pipeline/packet-builder.ts index 88bd129..f39b77f 100644 --- a/src/pipeline/packet-builder.ts +++ b/src/pipeline/packet-builder.ts @@ -64,42 +64,6 @@ type PacketGroup = { degradationReason?: string; }; -type HunkFirstGroup = PacketGroup & { - origin: "hunk-first"; -}; - -type PacketAtom = { - origin: "hunk-first"; - atomId: string; - group: HunkFirstGroup; - hunkCount: number; - patchChars: number; - firstSourcePosition: number; - effectiveCoverage: Exclude; - requestedLensSignature: string; - standalonePacket: ReviewPacket; - protectedFocusNotes: string[]; -}; - -type PackedAtomGroup = { - group: HunkFirstGroup; - atoms: PacketAtom[]; - firstSourcePosition: number; - stableOrder: number; -}; - -type PacketBuildOverrides = { - profileFloor?: ReviewProfile; - atomCount?: number; -}; - -type BuiltPacket = { - packet: ReviewPacket; - derivedReviewProfile: ReviewProfile; - baseToolBudget: ToolBudget; - patchChars: number; -}; - type HunkRelationshipSource = "same_symbol" | "symbol_mention" | "planner_hint"; type HunkRelationshipStrength = "strong" | "medium" | "weak"; @@ -226,108 +190,15 @@ export async function buildReviewPackets( } const allowWholeFileContext = includedPlanned.length === planned.length; - const groups = await groupHunks(includedPlanned, repoIndex, telemetry, { allowWholeFileContext }); - if (!opts.config.review.packSameFileHunks || !groups.every(isHunkFirstGroup)) { - for (const group of groups) { - const first = group.hunks[0]; - if (!first) { - continue; - } - const includedDecisions = decisionsForGroup(group, effectiveByHunk); - const built = await buildPacket(group.hunks, includedDecisions, group, relationshipGraph, repoIndex, opts.config, telemetry, opts.reviewContext, symbolContextMetrics, packetBuildMetrics); - packets.push(built.packet); - } - continue; - } - - const sourcePositionByHunk = new Map(includedPlanned.map((entry, index) => [entry.hunk.id, index])); - const atoms: PacketAtom[] = []; - for (const group of groups) { + for (const group of await groupHunks(includedPlanned, repoIndex, telemetry, { allowWholeFileContext })) { const first = group.hunks[0]; if (!first) { continue; } - const includedDecisions = decisionsForGroup(group, effectiveByHunk); - const standalone = await buildPacket( - group.hunks, - includedDecisions, - group, - isolatedRelationshipGraph(relationshipGraph), - repoIndex, - opts.config, - probeTelemetry(telemetry), - opts.reviewContext, - emptySymbolContextMetrics(), - emptyPacketBuildMetrics() - ); - atoms.push(packetAtom(group, includedDecisions, standalone.packet, sourcePositionByHunk.get(first.hunk.id) ?? atoms.length)); - } - - const candidates = packCompatibleAtoms(atoms); - const accepted: PackedAtomGroup[] = []; - for (const candidate of candidates) { - if (candidate.atoms.length === 1) { - accepted.push(candidate); - continue; - } - const profileFloor = maxReviewProfile(candidate.atoms.map((atom) => atom.standalonePacket.reviewProfile)); - const includedDecisions = decisionsForGroup(candidate.group, effectiveByHunk); - const probe = await buildPacket( - candidate.group.hunks, - includedDecisions, - candidate.group, - isolatedRelationshipGraph(relationshipGraph), - repoIndex, - opts.config, - probeTelemetry(telemetry), - opts.reviewContext, - emptySymbolContextMetrics(), - emptyPacketBuildMetrics(), - { profileFloor, atomCount: candidate.atoms.length } - ); - const rejection = packingCandidateRejection(candidate.atoms, probe.packet); - if (rejection === undefined) { - accepted.push(candidate); - continue; - } - telemetry.event({ - stage: 6, - level: "info", - message: "same_file_atom_pack_rejected", - ...(candidate.group.hunks[0]?.file.path !== undefined ? { file: candidate.group.hunks[0].file.path } : {}), - data: { reason: rejection, atomIds: candidate.atoms.map((atom) => atom.atomId) } - }); - accepted.push(...candidate.atoms.map((atom, index) => ({ - group: atom.group, - atoms: [atom], - firstSourcePosition: atom.firstSourcePosition, - stableOrder: candidate.stableOrder + index / Math.max(1, candidate.atoms.length) - }))); - } - - accepted.sort((a, b) => a.firstSourcePosition - b.firstSourcePosition || a.stableOrder - b.stableOrder); - for (const packed of accepted) { - const profileFloor = maxReviewProfile(packed.atoms.map((atom) => atom.standalonePacket.reviewProfile)); - const includedDecisions = decisionsForGroup(packed.group, effectiveByHunk); - const built = await buildPacket( - packed.group.hunks, - includedDecisions, - packed.group, - relationshipGraph, - repoIndex, - opts.config, - telemetry, - opts.reviewContext, - symbolContextMetrics, - packetBuildMetrics, - { profileFloor, atomCount: packed.atoms.length } - ); - const rejection = packingCandidateRejection(packed.atoms, built.packet); - if (rejection !== undefined) { - throw new Error(`same-file packet packing invariant failed: ${rejection}`); - } - emitSameFileAtomsPacked(telemetry, packed.atoms, built, profileFloor, opts.config.review.packedToolBudgetMode); - packets.push(built.packet); + const groupDecisions = group.hunks.map((entry) => effectiveByHunk.get(entry.hunk.id)).filter((decision): decision is EffectiveDecision => decision !== undefined); + const includedDecisions = groupDecisions.filter(isNonSkipDecision); + const packet = await buildPacket(group.hunks, includedDecisions, group, relationshipGraph, repoIndex, opts.config, telemetry, opts.reviewContext, symbolContextMetrics, packetBuildMetrics); + packets.push(packet); } } @@ -400,143 +271,6 @@ async function groupHunks( return hunkFirstGroups(planned); } -function isHunkFirstGroup(group: PacketGroup): group is HunkFirstGroup { - return "origin" in group && group.origin === "hunk-first"; -} - -function decisionsForGroup( - group: PacketGroup, - effectiveByHunk: Map -): NonSkipDecision[] { - return group.hunks - .map((entry) => effectiveByHunk.get(entry.hunk.id)) - .filter((decision): decision is NonSkipDecision => decision !== undefined && isNonSkipDecision(decision)); -} - -function packetAtom( - group: HunkFirstGroup, - decisions: NonSkipDecision[], - standalonePacket: ReviewPacket, - firstSourcePosition: number -): PacketAtom { - const hunkIds = group.hunks.map((entry) => entry.hunk.id); - const protectedFocusNotes = dedupe(group.hunks.flatMap((entry, index) => { - if (entry.facts.reviewPriority !== "critical" && entry.facts.reviewPriority !== "high") { - return []; - } - return (decisions[index]?.focusNotes ?? []).map(normalizeNote); - })).filter((note) => standalonePacket.attentionNotes.includes(note)); - return { - origin: "hunk-first", - atomId: sha256Hex(`hunk-first\n${hunkIds.join("\n")}`), - group, - hunkCount: group.hunks.length, - patchChars: combinedPatchChars(group.hunks), - firstSourcePosition, - effectiveCoverage: maxCoverage(decisions.map((decision) => decision.coverage)), - requestedLensSignature: normalizedLensSignature(decisions), - standalonePacket, - protectedFocusNotes - }; -} - -function normalizedLensSignature(decisions: NonSkipDecision[]): string { - return JSON.stringify(cleanStrings(decisions.flatMap((decision) => decision.lenses))); -} - -function packCompatibleAtoms(atoms: PacketAtom[]): PackedAtomGroup[] { - const partitions = new Map(); - for (const atom of atoms) { - const key = `${atom.effectiveCoverage}\0${atom.requestedLensSignature}`; - const groups = partitions.get(key) ?? []; - const current = groups.at(-1); - if ( - current !== undefined && - current.reduce((sum, member) => sum + member.hunkCount, 0) + atom.hunkCount <= MAX_HUNKS_PER_PACKET && - current.reduce((sum, member) => sum + member.patchChars, 0) + atom.patchChars <= MAX_PATCH_CHARS - ) { - current.push(atom); - } else { - groups.push([atom]); - } - partitions.set(key, groups); - } - - let stableOrder = 0; - return [...partitions.values()] - .flatMap((groups) => groups.map((members): PackedAtomGroup => { - const degradationReason = cleanStrings(members.flatMap((atom) => atom.group.degradationReason ?? [])); - const hunks = members.flatMap((atom) => atom.group.hunks); - return { - group: packetGroup(hunks, degradationReason.length > 0 ? degradationReason.join("; ") : undefined), - atoms: members, - firstSourcePosition: members[0]?.firstSourcePosition ?? Number.MAX_SAFE_INTEGER, - stableOrder: stableOrder++ - }; - })) - .sort((a, b) => a.firstSourcePosition - b.firstSourcePosition || a.stableOrder - b.stableOrder); -} - -function packingCandidateRejection(atoms: PacketAtom[], packet: ReviewPacket): string | undefined { - const expectedHunkIds = atoms.flatMap((atom) => atom.group.hunks.map((entry) => entry.hunk.id)); - const actualHunkIds = packet.hunks.map((hunk) => hunk.hunkId); - if (JSON.stringify(actualHunkIds) !== JSON.stringify(expectedHunkIds)) { - return "source atom hunks were split, duplicated, or reordered"; - } - if ( - packet.hunks.length > MAX_HUNKS_PER_PACKET || - (atoms.length > 1 && atoms.reduce((sum, atom) => sum + atom.patchChars, 0) > MAX_PATCH_CHARS) - ) { - return "packet cap exceeded"; - } - if (atoms.some((atom) => atom.effectiveCoverage !== packet.coverage)) { - return "effective coverage promotion"; - } - const missingLenses = cleanStrings(atoms.flatMap((atom) => atom.standalonePacket.lenses)) - .filter((lens) => !packet.lenses.includes(lens)); - if (missingLenses.length > 0) { - return `standalone routed lenses omitted: ${missingLenses.join(", ")}`; - } - const missingFocus = dedupe(atoms.flatMap((atom) => atom.protectedFocusNotes)) - .filter((note) => !packet.attentionNotes.includes(note)); - if (missingFocus.length > 0) { - return `high-priority planner focus omitted: ${missingFocus.join(" | ")}`; - } - const profileFloor = maxReviewProfile(atoms.map((atom) => atom.standalonePacket.reviewProfile)); - if (reviewProfileRank(packet.reviewProfile) < reviewProfileRank(profileFloor)) { - return `effective profile ${packet.reviewProfile} is below ${profileFloor}`; - } - return undefined; -} - -function isolatedRelationshipGraph(graph: HunkRelationshipGraph): HunkRelationshipGraph { - return { - ...graph, - relatedContextAttached: [], - relatedContextOmitted: [] - }; -} - -function emptyPacketBuildMetrics(): PacketBuildMetrics { - return { - relatedContextBudgetNudges: 0, - relatedContextBudgetNudgeSources: new Set() - }; -} - -function probeTelemetry(telemetry: TelemetryRecorder): TelemetryRecorder { - return { - runId: telemetry.runId, - runDir: telemetry.runDir, - event: () => undefined, - recordModelCall: () => undefined, - recordToolCall: () => "stage6-packet-probe", - writeArtifact: async () => undefined, - writeDebug: async () => undefined, - flush: async () => undefined - }; -} - async function buildPacket( planned: PlannedHunk[], decisions: NonSkipDecision[], @@ -547,9 +281,8 @@ async function buildPacket( telemetry: TelemetryRecorder, reviewContext: PacketReviewContext | undefined, symbolContextMetrics: SymbolContextMetrics, - packetBuildMetrics: PacketBuildMetrics, - overrides: PacketBuildOverrides = {} -): Promise { + packetBuildMetrics: PacketBuildMetrics +): Promise { const first = planned[0]; if (!first) { throw new Error("cannot build empty packet"); @@ -620,7 +353,7 @@ async function buildPacket( hasRelatedChangedContext, strongRelatedContext: false }); - const derivedReviewProfile = packetReviewProfile({ + const reviewProfile = packetReviewProfile({ coverage, reviewPriority, planned, @@ -629,8 +362,7 @@ async function buildPacket( hasRelatedChangedContext, strongRelatedContext }); - const reviewProfile = maxReviewProfile([derivedReviewProfile, overrides.profileFloor]); - if (strongRelatedContext && baseReviewProfile !== "investigate" && derivedReviewProfile === "investigate") { + if (strongRelatedContext && baseReviewProfile !== "investigate" && reviewProfile === "investigate") { const relationshipSources = cleanStrings(relatedChangedContext.flatMap((context) => context.relationshipSource ?? [])); packetBuildMetrics.relatedContextBudgetNudges += 1; for (const source of relationshipSources) { @@ -644,7 +376,7 @@ async function buildPacket( data: { coverage, baseReviewProfile, - reviewProfile: derivedReviewProfile, + reviewProfile, relatedContextCount: relatedChangedContext.length, relationshipSources } @@ -664,7 +396,6 @@ async function buildPacket( telemetry }); emitPacketContextQuality(telemetry, first.file.path, coverage, reviewPriority, contextQuality, contextDegradationReasons); - const baseToolBudget = toolBudget(coverage, config.review.depth, reviewProfile); const packet: ReviewPacket = { id: sha256Hex(`${first.file.path}\n${[...hunkIds].sort().join("\n")}\n${kind}`), dispatchRank: packetDispatchRank(first.file.path, first.facts, packetChangedLines), @@ -692,10 +423,7 @@ async function buildPacket( labels: first.facts.labels, attentionNotes, relatedChangedContext, - toolBudget: scaleToolBudget( - packedToolBudget(baseToolBudget, reviewProfile, overrides.atomCount ?? 1, config.review.packedToolBudgetMode), - config.review.budgetBoost - ), + toolBudget: scaleToolBudget(toolBudget(coverage, config.review.depth, reviewProfile), config.review.budgetBoost), ...(reviewContext?.intentText !== undefined ? { intentText: reviewContext.intentText } : {}), ...(reviewContext?.intentSignals !== undefined ? { intentSignals: reviewContext.intentSignals } : {}), ...(context.degradation !== undefined || truncationReason.length > 0 || contextDropReason !== undefined || contextTruncationReason !== undefined || group.degradationReason !== undefined @@ -707,45 +435,7 @@ async function buildPacket( ? { fileContext: { mode: "file-diff", reason: "grouped file hunks" } } : {}) }; - return { packet, derivedReviewProfile, baseToolBudget, patchChars }; -} - -function emitSameFileAtomsPacked( - telemetry: TelemetryRecorder, - atoms: PacketAtom[], - built: BuiltPacket, - profileFloor: ReviewProfile, - toolBudgetMode: CodegenieConfig["review"]["packedToolBudgetMode"] -): void { - telemetry.event({ - stage: 6, - level: "info", - message: "same_file_atoms_packed", - file: built.packet.path, - data: { - packetId: built.packet.id, - atomIds: atoms.map((atom) => atom.atomId), - standaloneProfiles: atoms.map((atom) => atom.standalonePacket.reviewProfile), - sourceAtomCount: atoms.length, - hunkCount: built.packet.hunks.length, - effectiveCoverage: built.packet.coverage, - requestedLensSignature: atoms[0]?.requestedLensSignature ?? "[]", - capUsage: { - hunks: built.packet.hunks.length, - maxHunks: MAX_HUNKS_PER_PACKET, - patchChars: built.patchChars, - maxPatchChars: MAX_PATCH_CHARS - }, - derivedPackedProfile: built.derivedReviewProfile, - profileFloor, - effectiveProfile: built.packet.reviewProfile, - profileFloorApplied: reviewProfileRank(built.derivedReviewProfile) < reviewProfileRank(profileFloor), - plannerLensesPreserved: true, - toolBudgetMode, - baseToolBudget: built.baseToolBudget, - effectiveToolBudget: built.packet.toolBudget - } - }); + return packet; } const DOCS_CONFIG_EXTENSIONS = new Set([".md", ".yml", ".yaml", ".toml", ".conf", ".sample", ".txt"]); @@ -1489,8 +1179,8 @@ function dedupe(values: T[]): T[] { return [...new Set(values)]; } -function hunkFirstGroups(planned: PlannedHunk[], degradationReason?: string): HunkFirstGroup[] { - const groups: HunkFirstGroup[] = []; +function hunkFirstGroups(planned: PlannedHunk[], degradationReason?: string): PacketGroup[] { + const groups: PacketGroup[] = []; let current: PlannedHunk[] = []; for (const entry of planned) { @@ -1576,7 +1266,7 @@ function lineInRange(line: number, start: number, lines: number): boolean { return line >= start && line <= start + lines - 1; } -function packetGroup(hunks: PlannedHunk[], degradationReason?: string): HunkFirstGroup { +function packetGroup(hunks: PlannedHunk[], degradationReason?: string): PacketGroup { const first = hunks[0]; const kind: ReviewPacket["kind"] = first && hunks.length > 1 && hunks.length === first.file.hunks.length @@ -1584,7 +1274,7 @@ function packetGroup(hunks: PlannedHunk[], degradationReason?: string): HunkFirs : hunks.length > 1 ? "coalesced-hunks" : "hunk"; - return { origin: "hunk-first", hunks, kind, ...(degradationReason !== undefined ? { degradationReason } : {}) }; + return { hunks, kind, ...(degradationReason !== undefined ? { degradationReason } : {}) }; } function packetKind(group: PacketGroup, planned: PlannedHunk[], file: DiffFile): ReviewPacket["kind"] { @@ -3258,39 +2948,6 @@ export function toolBudget(coverage: Exclude, depth: Code }; } -function packedToolBudget( - base: ToolBudget, - profile: ReviewProfile, - atomCount: number, - mode: CodegenieConfig["review"]["packedToolBudgetMode"] -): ToolBudget { - if (mode === "base" || atomCount <= 1 || profile === "simple") { - return base; - } - const additionalAtoms = atomCount - 1; - return { - ...base, - maxToolCalls: Math.min(base.maxToolCalls + additionalAtoms, Math.ceil(1.75 * base.maxToolCalls)), - maxResultChars: Math.min(base.maxResultChars + additionalAtoms * 2_000, Math.ceil(1.75 * base.maxResultChars)) - }; -} - -const REVIEW_PROFILE_RANK: Record = { - simple: 0, - standard: 1, - investigate: 2 -}; - -function reviewProfileRank(profile: ReviewProfile): number { - return REVIEW_PROFILE_RANK[profile]; -} - -function maxReviewProfile(profiles: Array): ReviewProfile { - return profiles - .filter((profile): profile is ReviewProfile => profile !== undefined) - .reduce((highest, profile) => reviewProfileRank(profile) > reviewProfileRank(highest) ? profile : highest, "simple"); -} - function packetReviewProfile(input: { coverage: Exclude; reviewPriority: ReviewPriority; diff --git a/src/pipeline/review-runner.ts b/src/pipeline/review-runner.ts index 499081b..38b07c8 100644 --- a/src/pipeline/review-runner.ts +++ b/src/pipeline/review-runner.ts @@ -921,13 +921,7 @@ async function maybeZeroWork( }); await run.telemetry.writeArtifact("candidate-findings.json", []); await run.telemetry.writeArtifact("verification.json", []); - await run.telemetry.writeArtifact("final-selection.json", { - composition: { mode: "deterministic_fallback", fallbackReason: "zero-work short circuit" }, - records: [], - publicationAnchors: [], - confidenceSelections: [], - groups: [] - }); + await run.telemetry.writeArtifact("final-selection.json", { records: [], groups: [] }); await run.telemetry.writeArtifact("final-findings.json", []); await run.telemetry.writeArtifact("budget-summary.json", result.budgetSummary); await maybePublishToGitHub(result, resolved, config, run.telemetry, { diff --git a/src/pipeline/verifier.ts b/src/pipeline/verifier.ts index 0ac7665..c1e9937 100644 --- a/src/pipeline/verifier.ts +++ b/src/pipeline/verifier.ts @@ -7,7 +7,6 @@ import type { TelemetryRecorder } from "../telemetry/telemetry-recorder.js"; import type { CandidateFinding, CodegenieConfig, - EvalVerificationRecord, PacketReviewResult, RepositoryTools, ReviewPacket, @@ -142,12 +141,6 @@ type CandidateGateDecision = | { outcome: "suppress"; reason: string; facts: VerificationGateFacts } | { outcome: "schedule"; reason: string; lane: VerificationLane; facts: VerificationGateFacts }; -export type ReconstructedVerificationGateDecision = { - candidate: CandidateFinding; - anchorStripped: boolean; - decision: CandidateGateDecision; -}; - export async function verifyFindings( input: { packetResults: PacketReviewResult[]; packets: ReviewPacket[] }, tools: RepositoryTools, @@ -502,101 +495,6 @@ function applyVerificationVerdict(candidate: CandidateFinding, verdict: Verifica return applyVerdictIntentAssessment(applyVerdictAnchor(revised, verdict), verdict); } -export function reconstructVerifiedFindingsFromArtifacts( - candidates: CandidateFinding[], - records: EvalVerificationRecord[], - packets: ReviewPacket[], - diff: UnifiedDiff -): CandidateFinding[] { - const clusteredById = new Map(reconstructVerifierCandidatesFromArtifacts(candidates, records, packets, diff) - .map((candidate) => [candidate.id, candidate])); - return records.flatMap((record) => { - if (!("verdict" in record) || record.verdict.verificationIncomplete === true || - record.verdict.verdict === "reject" || record.verdict.verdict === "incomplete") { - return []; - } - const candidate = clusteredById.get(record.candidateId); - return candidate === undefined ? [] : [applyVerificationVerdict(candidate, record.verdict)]; - }); -} - -export function reconstructVerifierCandidatesFromArtifacts( - candidates: CandidateFinding[], - records: EvalVerificationRecord[], - packets: ReviewPacket[], - diff: UnifiedDiff -): CandidateFinding[] { - const packetsById = new Map(packets.map((packet) => [packet.id, packet])); - const recordsById = new Map(records.map((record) => [record.candidateId, record])); - const quietTelemetry = { event: () => undefined } as unknown as TelemetryRecorder; - const preGated = candidates.map((candidate) => - preGateAnchor(candidate, packetsById.get(candidate.producedBy.packetId), diff, quietTelemetry).candidate - ).filter((candidate) => recordsById.get(candidate.id)?.gateDecision !== "suppressed"); - const clustered = clusterCandidates(preGated, quietTelemetry); - return clustered.all; -} - -export function reconstructDuplicateVerificationVerdict( - duplicate: CandidateFinding, - representativeVerdict: VerificationVerdict -): VerificationVerdict { - return { - ...representativeVerdict, - candidateId: duplicate.id, - ...(representativeVerdict.finalFinding !== undefined - ? { finalFinding: applyVerificationVerdict(duplicate, representativeVerdict) } - : {}) - }; -} - -export function reconstructVerificationGateFactsFromArtifacts( - candidate: CandidateFinding, - packet: ReviewPacket | undefined, - diff: UnifiedDiff -): VerificationGateFacts { - const quietTelemetry = { event: () => undefined } as unknown as TelemetryRecorder; - return candidateGateFacts(preGateAnchor(candidate, packet, diff, quietTelemetry).candidate); -} - -export function reconstructVerificationGateDecisionFromArtifacts( - candidate: CandidateFinding, - packet: ReviewPacket | undefined, - diff: UnifiedDiff, - config: CodegenieConfig -): ReconstructedVerificationGateDecision { - const quietTelemetry = { event: () => undefined } as unknown as TelemetryRecorder; - const preGated = preGateAnchor(candidate, packet, diff, quietTelemetry); - return { - candidate: preGated.candidate, - anchorStripped: preGated.anchorStripped, - decision: gateCandidate(preGated.candidate, config) - }; -} - -export function reconstructGatedVerifierCandidatesFromArtifacts( - candidates: CandidateFinding[], - packets: ReviewPacket[], - diff: UnifiedDiff, - config: CodegenieConfig -): { - evaluations: ReconstructedVerificationGateDecision[]; - candidates: CandidateFinding[]; -} { - const packetsById = new Map(packets.map((packet) => [packet.id, packet])); - const evaluations = candidates.map((candidate) => reconstructVerificationGateDecisionFromArtifacts( - candidate, - packetsById.get(candidate.producedBy.packetId), - diff, - config - )); - const quietTelemetry = { event: () => undefined } as unknown as TelemetryRecorder; - const clustered = clusterCandidates( - evaluations.flatMap((evaluation) => evaluation.decision.outcome === "schedule" ? [evaluation.candidate] : []), - quietTelemetry - ); - return { evaluations, candidates: clustered.all }; -} - function applyFindingRevision(candidate: CandidateFinding, revision: CandidateFinding): CandidateFinding { const { clusterId: _revisionClusterId, duplicateOf: _revisionDuplicateOf, ...revisionFields } = revision; return { diff --git a/src/telemetry/run-artifacts.ts b/src/telemetry/run-artifacts.ts index 9baa89d..313f36e 100644 --- a/src/telemetry/run-artifacts.ts +++ b/src/telemetry/run-artifacts.ts @@ -197,7 +197,7 @@ type TelemetryStageSummary = { schemaRecovery: SchemaRecoveryCounters; }; -export type Stage7SchemaRepairSummary = { +type Stage7SchemaRepairSummary = { candidateInvalidSubmits: number; noFindingInvalidSubmits: number; cleanupAttempted: number; @@ -214,7 +214,7 @@ export type Stage7SchemaRepairSummary = { actualRepairPromptChars: number; }; -export type LogOverflowSummary = { +type LogOverflowSummary = { droppedDebugInfo: number; droppedWarnError: number; }; @@ -274,7 +274,7 @@ type ArtifactManifest = { artifacts: ArtifactManifestEntry[]; }; -export type PipelineTotals = { +type PipelineTotals = { filesChanged: number; hunks: number; packets: number; @@ -285,7 +285,7 @@ export type PipelineTotals = { postedComments: number; }; -export type PipelineTelemetrySummary = { +type PipelineTelemetrySummary = { workers: { started: number; completed: number; @@ -498,14 +498,6 @@ class RunTelemetryImpl { this.runDirectory = runDir; touchCoreFiles(runDir); this.flushBuffers(); - if (this.logOverflow.droppedDebugInfo > 0 || this.logOverflow.droppedWarnError > 0) { - this.recordEvent({ - stage: 0, - level: "warn", - message: "telemetry_log_overflow", - data: { ...this.logOverflow } - }); - } this.recordPruneResult(pruneRuns(runsRoot, runDir, this.config.retainRuns)); return { runId: this.runId, runDir }; } @@ -1172,15 +1164,83 @@ class RunTelemetryImpl { } private updateStage7SchemaRepairSummaryFromEvent(event: TelemetryEvent): void { - updateStage7SchemaRepairFromEvent(this.stage7SchemaRepairSummary, event); + if (event.stage !== 7) { + return; + } + const data = objectField(event.data); + const classification = typeof data?.classification === "string" ? data.classification : ""; + const payloadKind = typeof data?.payloadKind === "string" ? data.payloadKind : ""; + if (event.message === "stage7_schema_repair_attempted") { + if (payloadKind === "no_findings" || classification === "empty_no_findings_missing_fields") { + this.stage7SchemaRepairSummary.noFindingInvalidSubmits += 1; + } else { + this.stage7SchemaRepairSummary.candidateInvalidSubmits += 1; + } + return; + } + if (event.message === "stage7_schema_cleanup_attempted") { + this.stage7SchemaRepairSummary.cleanupAttempted += 1; + return; + } + if (event.message === "stage7_schema_cleanup_recovered") { + this.stage7SchemaRepairSummary.cleanupRecovered += 1; + return; + } + if (event.message === "stage7_schema_cleanup_rejected") { + this.stage7SchemaRepairSummary.cleanupRejected += 1; + return; + } + if (event.message === "stage7_schema_compact_repair_scheduled") { + this.stage7SchemaRepairSummary.compactRepairScheduled += 1; + const promptChars = numberField(data?.repairPromptChars); + this.stage7SchemaRepairSummary.repairPromptChars += promptChars; + this.stage7SchemaRepairSummary.compactRepairPromptChars += promptChars; + return; + } + if (event.message === "schema_repair_scheduled") { + const replaceConversation = data?.replaceConversation === true; + if (!replaceConversation) { + this.stage7SchemaRepairSummary.appendRepairScheduled += 1; + const promptChars = numberField(data?.repairPromptChars); + this.stage7SchemaRepairSummary.repairPromptChars += promptChars; + this.stage7SchemaRepairSummary.appendRepairPromptChars += promptChars; + } + return; + } + if (event.message === "stage7_schema_repair_recovered") { + if (classification === "schema_valid_after_retry") { + this.stage7SchemaRepairSummary.repairRecovered += 1; + } + return; + } + if (event.message === "stage7_schema_repair_failed") { + this.stage7SchemaRepairSummary.repairFailed += 1; + } } private updateStage7SchemaRepairSummaryFromModelCall(record: LlmCallRecord): void { - updateStage7SchemaRepairFromModelCall(this.stage7SchemaRepairSummary, record); + if (record.stage !== 7 || record.kind !== "repair") { + return; + } + this.stage7SchemaRepairSummary.actualRepairCalls += 1; + this.stage7SchemaRepairSummary.actualRepairPromptChars += record.promptChars; } private updatePipelineSummaryFromEvent(event: TelemetryEvent): void { - updatePipelineSummaryFromEvent(this.pipelineTotals, this.pipelineSummary, event); + if (event.message !== "pipeline_metrics" || !event.data || typeof event.data !== "object") { + return; + } + const data = event.data as Record; + mergePipelineTotals(this.pipelineTotals, objectField(data.totals)); + mergePipelineWorkers(this.pipelineSummary.workers, objectField(data.workers)); + mergePipelinePackets(this.pipelineSummary.packets, objectField(data.packets)); + mergePipelineLenses(this.pipelineSummary.lenses, objectField(data.lenses)); + mergePipelineCoverage(this.pipelineSummary.coverage, objectField(data.coverage)); + mergePipelineCandidates(this.pipelineSummary.candidates, objectField(data.candidates)); + mergePipelineVerdicts(this.pipelineSummary.verdicts, objectField(data.verdicts)); + mergePipelineDedup(this.pipelineSummary.dedup, objectField(data.dedup)); + mergePipelineFinalSelection(this.pipelineSummary.finalSelection, objectField(data.finalSelection)); + mergePipelinePosting(this.pipelineSummary.posting, objectField(data.posting)); } private runReviewMetadata(): RunReviewArtifactMetadata { @@ -1600,97 +1660,6 @@ function emptyStage7SchemaRepairSummary(): Stage7SchemaRepairSummary { }; } -function updateStage7SchemaRepairFromEvent(summary: Stage7SchemaRepairSummary, event: TelemetryEvent): void { - if (event.stage !== 7) { - return; - } - const data = objectField(event.data); - const classification = typeof data?.classification === "string" ? data.classification : ""; - const payloadKind = typeof data?.payloadKind === "string" ? data.payloadKind : ""; - if (event.message === "stage7_schema_repair_attempted") { - if (payloadKind === "no_findings" || classification === "empty_no_findings_missing_fields") { - summary.noFindingInvalidSubmits += 1; - } else { - summary.candidateInvalidSubmits += 1; - } - } else if (event.message === "stage7_schema_cleanup_attempted") { - summary.cleanupAttempted += 1; - } else if (event.message === "stage7_schema_cleanup_recovered") { - summary.cleanupRecovered += 1; - } else if (event.message === "stage7_schema_cleanup_rejected") { - summary.cleanupRejected += 1; - } else if (event.message === "stage7_schema_compact_repair_scheduled") { - summary.compactRepairScheduled += 1; - const promptChars = numberField(data?.repairPromptChars); - summary.repairPromptChars += promptChars; - summary.compactRepairPromptChars += promptChars; - } else if (event.message === "schema_repair_scheduled" && data?.replaceConversation !== true) { - summary.appendRepairScheduled += 1; - const promptChars = numberField(data?.repairPromptChars); - summary.repairPromptChars += promptChars; - summary.appendRepairPromptChars += promptChars; - } else if (event.message === "stage7_schema_repair_recovered" && classification === "schema_valid_after_retry") { - summary.repairRecovered += 1; - } else if (event.message === "stage7_schema_repair_failed") { - summary.repairFailed += 1; - } -} - -function updateStage7SchemaRepairFromModelCall(summary: Stage7SchemaRepairSummary, record: LlmCallRecord): void { - if (record.stage === 7 && record.kind === "repair") { - summary.actualRepairCalls += 1; - summary.actualRepairPromptChars += record.promptChars; - } -} - -function updatePipelineSummaryFromEvent( - totals: PipelineTotals, - summary: PipelineTelemetrySummary, - event: TelemetryEvent -): void { - if (event.message !== "pipeline_metrics" || !event.data || typeof event.data !== "object") { - return; - } - const data = event.data as Record; - mergePipelineTotals(totals, objectField(data.totals)); - mergePipelineWorkers(summary.workers, objectField(data.workers)); - mergePipelinePackets(summary.packets, objectField(data.packets)); - mergePipelineLenses(summary.lenses, objectField(data.lenses)); - mergePipelineCoverage(summary.coverage, objectField(data.coverage)); - mergePipelineCandidates(summary.candidates, objectField(data.candidates)); - mergePipelineVerdicts(summary.verdicts, objectField(data.verdicts)); - mergePipelineDedup(summary.dedup, objectField(data.dedup)); - mergePipelineFinalSelection(summary.finalSelection, objectField(data.finalSelection)); - mergePipelinePosting(summary.posting, objectField(data.posting)); -} - -export function reconstructRunTelemetryDerivedEvidence( - events: TelemetryEvent[], - modelCalls: LlmCallRecord[] -): { - pipelineTotals: PipelineTotals; - pipeline: PipelineTelemetrySummary; - stage7SchemaRepair: Stage7SchemaRepairSummary; - logOverflow: LogOverflowSummary; -} { - const pipelineTotals = emptyPipelineTotals(); - const pipeline = emptyPipelineTelemetrySummary(); - const stage7SchemaRepair = emptyStage7SchemaRepairSummary(); - const logOverflow: LogOverflowSummary = { droppedDebugInfo: 0, droppedWarnError: 0 }; - for (const event of events) { - updatePipelineSummaryFromEvent(pipelineTotals, pipeline, event); - updateStage7SchemaRepairFromEvent(stage7SchemaRepair, event); - if (event.message === "telemetry_log_overflow") { - logOverflow.droppedDebugInfo = numberField(event.data?.droppedDebugInfo); - logOverflow.droppedWarnError = numberField(event.data?.droppedWarnError); - } - } - for (const record of modelCalls) { - updateStage7SchemaRepairFromModelCall(stage7SchemaRepair, record); - } - return { pipelineTotals, pipeline, stage7SchemaRepair, logOverflow }; -} - function emptyToolBucket(): ToolBucket { return { count: 0, diff --git a/src/types.ts b/src/types.ts index 8d9875d..2666377 100644 --- a/src/types.ts +++ b/src/types.ts @@ -8,7 +8,6 @@ export type ProcessingMode = "per-hunk" | "whole-file" | "skip"; export type ReviewPriority = "critical" | "high" | "normal" | "low"; export type OutputFormat = "markdown" | "json"; export type LogLevel = "debug" | "info" | "warn" | "error"; -export type PackedToolBudgetMode = "base" | "atom-scaled"; export type ConfigSource = | "defaults" @@ -45,8 +44,6 @@ export type CodegenieConfig = { maxTimeMs: number; perPassTimeoutMs: number; budgetBoost: number; - packSameFileHunks: boolean; - packedToolBudgetMode: PackedToolBudgetMode; maxBudgetTokens?: number; maxModelCalls?: number; // Plan 84: K independent Stage-7 passes for deep-coverage packets, union @@ -1011,8 +1008,6 @@ export type EvalCase = { maxFindings?: number; concurrency?: number; budgetBoost?: number; - packSameFileHunks?: boolean; - packedToolBudgetMode?: PackedToolBudgetMode; maxTimeMinutes?: number; maxBudgetTokens?: number; deepEnsemblePasses?: number; @@ -1260,25 +1255,12 @@ export type EvalRunInfo = { }; repo?: { root: string; baseSha?: string; headSha?: string; mergeBase?: string }; reviewRunId?: string; - invocation?: { - id: string; - caseIndex: number; - manifest: string; - }; codegenieRuntime?: CodegenieRuntimeProvenance; cache: { enabled: boolean; source: "cli" | "case" | "config"; dir?: string }; effectiveConfig?: { review: { concurrency: number; timeoutMs: number; - verify: boolean; - minSeverity?: Severity; - maxFindings: number; - softCommentCap: number; - minConfidence: Confidence; - minInlineConfidence: Confidence; - packSameFileHunks: boolean; - packedToolBudgetMode: PackedToolBudgetMode; maxBudgetTokens?: number; }; llm: { @@ -1293,29 +1275,6 @@ export type EvalRunInfo = { score: EvalScore; }; -export type EvalInvocationManifest = { - schemaVersion: 1; - invocationId: string; - suiteDir: string; - status: "running" | "complete"; - startedAt: string; - completedAt?: string; - cases: Array<{ - caseIndex: number; - caseName: string; - caseHash: string; - caseFile: string; - }>; - runs: Array<{ - caseIndex: number; - caseName: string; - caseHash: string; - runNumber: number; - logsRoot: string; - runPath: string; - }>; -}; - export type EvalCaseResult = { caseName: string; runDir: string; @@ -1646,7 +1605,6 @@ export type ToolCallRecord = { glob?: string; source?: string; contextMode?: string; - maxResults?: number; }; backend: ToolBackend; precision: ToolPrecision; diff --git a/tests/config-loader.test.ts b/tests/config-loader.test.ts index 3895121..89e20b6 100644 --- a/tests/config-loader.test.ts +++ b/tests/config-loader.test.ts @@ -14,42 +14,6 @@ import { loadProviderSettings, saveProviderSettings } from "../src/provider/prov import { CodegenieError } from "../src/util/errors.js"; describe("config loader", () => { - it("loads dark packet-packing defaults and tracks safe user and repository overrides", () => { - const repoRoot = tempDir(); - const home = tempDir(); - - const defaults = loadConfig({ repoRoot, homeOverride: home }); - expect(defaults.config.review.packSameFileHunks).toBe(false); - expect(defaults.config.review.packedToolBudgetMode).toBe("base"); - expect(defaults.sources["review.packSameFileHunks"]).toBe("defaults"); - expect(defaults.sources["review.packedToolBudgetMode"]).toBe("defaults"); - - writeFileSync(path.join(home, "config.toml"), [ - "[review]", - "packSameFileHunks = true", - 'packedToolBudgetMode = "atom-scaled"' - ].join("\n")); - const userConfigured = loadConfig({ repoRoot, homeOverride: home }); - expect(userConfigured.config.review.packSameFileHunks).toBe(true); - expect(userConfigured.config.review.packedToolBudgetMode).toBe("atom-scaled"); - expect(userConfigured.sources["review.packSameFileHunks"]).toBe("user-config"); - expect(userConfigured.sources["review.packedToolBudgetMode"]).toBe("user-config"); - - writeFileSync(path.join(repoRoot, "codegenie.toml"), [ - "[review]", - "packSameFileHunks = false", - 'packedToolBudgetMode = "base"' - ].join("\n")); - const repoConfigured = loadConfig({ repoRoot, homeOverride: home }); - expect(repoConfigured.config.review.packSameFileHunks).toBe(false); - expect(repoConfigured.config.review.packedToolBudgetMode).toBe("base"); - expect(repoConfigured.sources["review.packSameFileHunks"]).toBe("repo-config"); - expect(repoConfigured.sources["review.packedToolBudgetMode"]).toBe("repo-config"); - expect(repoConfigured.warnings).toEqual([]); - - expect(rawConfigSchema.safeParse({ review: { packedToolBudgetMode: "linear" } }).success).toBe(false); - }); - it("resolves maxTime minutes from defaults, user config, repo config, and CLI in precedence order", () => { const repoRoot = tempDir(); const home = tempDir(); diff --git a/tests/evals.test.ts b/tests/evals.test.ts index 4da84cf..64cdd97 100644 --- a/tests/evals.test.ts +++ b/tests/evals.test.ts @@ -9,12 +9,10 @@ import { compareToPrevious, renderEvalCompareText } from "../src/evals/eval-comp import { executeEvalCommand, renderCaseResult, runEvalCommand } from "../src/evals/eval-command.js"; import { loadEvalSuite, replayFromArtifacts, runEvalCase } from "../src/evals/eval-runner.js"; import { aggregateRepeatScores, assignExpectations, matchExpectation, scoreEvalRun } from "../src/evals/eval-scoring.js"; -import { loadEvalRuns } from "../scripts/packet-packing-report.js"; import type { CandidateFinding, EvalArtifacts, EvalCase, - EvalInvocationManifest, EvalRunInfo, EvalRunMetrics, EvalScore, @@ -128,34 +126,6 @@ describe("eval suite validation", () => { }); }); - it("strictly parses packet-packing eval overrides", async () => { - const suiteDir = mkdtempSync(path.join(tmpdir(), "codegenie-eval-packet-packing-")); - writeFileSync(path.join(suiteDir, "packing.yml"), [ - "name: packet-packing", - "artifacts:", - " path: logs/1", - "review:", - " packSameFileHunks: true", - " packedToolBudgetMode: atom-scaled" - ].join("\n")); - - const suite = await loadEvalSuite(suiteDir); - expect(suite.cases[0]?.evalCase.review).toMatchObject({ - packSameFileHunks: true, - packedToolBudgetMode: "atom-scaled" - }); - - const invalidDir = mkdtempSync(path.join(tmpdir(), "codegenie-eval-packet-packing-invalid-")); - writeFileSync(path.join(invalidDir, "packing.yml"), [ - "name: packet-packing-invalid", - "artifacts:", - " path: logs/1", - "review:", - " packedToolBudgetMode: linear" - ].join("\n")); - await expect(loadEvalSuite(invalidDir)).rejects.toMatchObject({ code: "config_error" }); - }); - it("accepts pinned head/base eval commands", async () => { const suiteDir = mkdtempSync(path.join(tmpdir(), "codegenie-eval-head-base-")); writeFileSync(path.join(suiteDir, "head.yml"), [ @@ -1640,7 +1610,6 @@ describe("eval command fixture suite", () => { }); let output = ""; - const suite = await loadEvalSuite(suiteDir); const exitCode = await runEvalCommand({ evalDir: suiteDir, cache: false }, defaultConfig, { writeOutput: (text) => { output += text; @@ -1650,115 +1619,6 @@ describe("eval command fixture suite", () => { expect(exitCode).toBe(0); expect(output).toContain("Suite: 8 passed, 0 failed, 0 errored"); expect(existsSync(path.join(suiteDir, "logs", "1", "fixture-repo", ".git"))).toBe(true); - const invocationFiles = readdirSync(path.join(suiteDir, "logs", "invocations")); - expect(invocationFiles).toHaveLength(1); - const invocationFile = invocationFiles[0]!; - const manifest = JSON.parse(readFileSync(path.join(suiteDir, "logs", "invocations", invocationFile), "utf8")) as EvalInvocationManifest; - expect(manifest).toMatchObject({ - schemaVersion: 1, - suiteDir, - status: "complete", - completedAt: expect.any(String), - cases: suite.cases.map((entry, caseIndex) => ({ - caseIndex, - caseName: entry.evalCase.name, - caseHash: entry.caseHash, - caseFile: entry.file - })), - runs: suite.cases.map((entry, caseIndex) => ({ - caseIndex, - caseName: entry.evalCase.name, - caseHash: entry.caseHash, - runNumber: caseIndex + 1, - logsRoot: path.join(suiteDir, "logs"), - runPath: String(caseIndex + 1) - })) - }); - for (const [caseIndex, run] of manifest.runs.entries()) { - const info = JSON.parse(readFileSync(path.join(suiteDir, "logs", String(run.runNumber), "info.json"), "utf8")) as EvalRunInfo; - expect(info.invocation).toEqual({ - id: manifest.invocationId, - caseIndex, - manifest: path.join("invocations", invocationFile) - }); - } - const reportRuns = await loadEvalRuns(path.join(suiteDir, "logs")); - expect(reportRuns.map((run) => run.info.invocation?.caseIndex)).toEqual(suite.cases.map((_, index) => index)); - - const interruptedManifest: EvalInvocationManifest = { - schemaVersion: 1, - invocationId: "historical-interrupted", - suiteDir, - status: "running", - startedAt: "2000-01-01T00:00:00.000Z", - cases: [{ caseIndex: 0, caseName: "interrupted", caseHash: "interrupted", caseFile: suite.cases[0]!.file }], - runs: [{ - caseIndex: 0, - caseName: "interrupted", - caseHash: "interrupted", - runNumber: 999, - logsRoot: path.join(suiteDir, "logs"), - runPath: "999" - }] - }; - writeFileSync( - path.join(suiteDir, "logs", "invocations", `${interruptedManifest.invocationId}.json`), - `${JSON.stringify(interruptedManifest, null, 2)}\n` - ); - mkdirSync(path.join(suiteDir, "logs", "999")); - expect((await loadEvalRuns(path.join(suiteDir, "logs"), "latest")).map((run) => run.runNumber)).toEqual(manifest.runs.map((run) => run.runNumber)); - expect((await loadEvalRuns(path.join(suiteDir, "logs"), manifest.invocationId)).map((run) => run.runNumber)).toEqual(manifest.runs.map((run) => run.runNumber)); - - const firstRecordedRun = manifest.runs[0]!; - const firstInfoPath = path.join(suiteDir, "logs", String(firstRecordedRun.runNumber), "info.json"); - const mutuallyAlteredInfo = JSON.parse(readFileSync(firstInfoPath, "utf8")) as EvalRunInfo; - mutuallyAlteredInfo.caseHash = "mutually-altered-hash"; - mutuallyAlteredInfo.caseSnapshot.should_find![0]!.id = "mutually-altered-expectation"; - mutuallyAlteredInfo.score.expectationResults[0]!.expectationId = "mutually-altered-expectation"; - manifest.cases[0]!.caseHash = "mutually-altered-hash"; - manifest.runs[0]!.caseHash = "mutually-altered-hash"; - writeFileSync(firstInfoPath, `${JSON.stringify(mutuallyAlteredInfo, null, 2)}\n`); - writeFileSync(path.join(suiteDir, "logs", "invocations", invocationFile), `${JSON.stringify(manifest, null, 2)}\n`); - await expect(loadEvalRuns(path.join(suiteDir, "logs"), manifest.invocationId)).rejects.toMatchObject({ - failures: [expect.objectContaining({ code: "declared_case_join" })] - }); - }, 60_000); - - it("records exact per-entry root ownership and loads one invocation across multiple log roots", async () => { - const root = mkdtempSync(path.join(tmpdir(), "codegenie-multi-root-invocation-")); - const suiteDir = path.join(root, "fixtures"); - const firstRoot = path.join(root, "logs-a"); - const secondRoot = path.join(root, "logs-b"); - cpSync(path.join(process.cwd(), "evals", "fixtures"), suiteDir, { - recursive: true, - filter: (source) => path.basename(source) !== ".git" && path.basename(source) !== "logs" && !source.includes(`${path.sep}.git${path.sep}`) - }); - const caseFiles = readdirSync(suiteDir).filter((entry) => /\.ya?ml$/u.test(entry)).sort(); - for (const [index, file] of caseFiles.entries()) { - const logsRoot = index < caseFiles.length / 2 ? firstRoot : secondRoot; - const target = path.join(suiteDir, file); - writeFileSync(target, `${readFileSync(target, "utf8").trimEnd()}\nlogs:\n dir: ${logsRoot}\n`); - } - - expect(await runEvalCommand({ evalDir: suiteDir, cache: false }, defaultConfig)).toBe(0); - const manifestFile = readdirSync(path.join(firstRoot, "invocations"))[0]!; - const firstManifest = JSON.parse(readFileSync(path.join(firstRoot, "invocations", manifestFile), "utf8")) as EvalInvocationManifest; - const secondManifestPath = path.join(secondRoot, "invocations", manifestFile); - expect(JSON.parse(readFileSync(secondManifestPath, "utf8"))).toEqual(firstManifest); - expect(new Set(firstManifest.runs.map((entry) => entry.logsRoot))).toEqual(new Set([firstRoot, secondRoot])); - expect(new Set(firstManifest.runs.map((entry) => `${entry.logsRoot}\0${entry.runPath}`)).size).toBe(caseFiles.length); - expect(new Set(firstManifest.runs.map((entry) => entry.runNumber)).size).toBeLessThan(caseFiles.length); - - const loaded = await loadEvalRuns(firstRoot, firstManifest.invocationId); - expect(loaded).toHaveLength(caseFiles.length); - expect(new Set(loaded.map((run) => path.dirname(run.runDir)))).toEqual(new Set([firstRoot, secondRoot])); - - const altered = structuredClone(firstManifest); - altered.runs.at(-1)!.logsRoot = firstRoot; - writeFileSync(secondManifestPath, `${JSON.stringify(altered, null, 2)}\n`); - await expect(loadEvalRuns(firstRoot, firstManifest.invocationId)).rejects.toMatchObject({ - failures: [expect.objectContaining({ code: "invocation_root_ownership" })] - }); }, 60_000); it("does not leak the invocation directory repo config into live cases", async () => { @@ -1865,8 +1725,6 @@ describe("eval command fixture suite", () => { " model: not-real-model", " reasoning: low", " concurrency: 3", - " packSameFileHunks: true", - " packedToolBudgetMode: atom-scaled", " lenses:", " - core/code-review", "llm:", @@ -1890,11 +1748,7 @@ describe("eval command fixture suite", () => { expect(result.status).toBe("pass"); expect(result.info.effectiveConfig).toMatchObject({ - review: { - concurrency: 3, - packSameFileHunks: true, - packedToolBudgetMode: "atom-scaled" - }, + review: { concurrency: 3 }, llm: { provider: "fake", model: "fake-model", reasoning: "high", maxConcurrentCalls: 2 } }); expect(result.info.codegenieRuntime).toMatchObject({ diff --git a/tests/packet-packing-report.test.ts b/tests/packet-packing-report.test.ts deleted file mode 100644 index 5dda531..0000000 --- a/tests/packet-packing-report.test.ts +++ /dev/null @@ -1,2288 +0,0 @@ -import { mkdtemp, mkdir, readFile, writeFile } from "node:fs/promises"; -import { tmpdir } from "node:os"; -import path from "node:path"; -import { afterEach, describe, expect, it, vi } from "vitest"; -import { - analyzeEvalCohort, - analyzeRegressionCohorts, - analyzeReplayComparison, - computeProductionEconomics, - finalizeReplayCleanup, - historicalFlagOffParityView, - reconstructEvidenceArtifacts, - runPacketPackingReportCli, - selectExplicitCohort, - validateRecordedDiffParity, - validatePacketPackingArtifact, - validateReplayArtifacts, - type CohortSelection, - type EvalCaseRunInput, - type EvalExecutionInput, - type ReplayAnalysisInput, - type ReportFailure -} from "../scripts/packet-packing-report.js"; -import { aggregateRepeatScores, scoreEvalRun } from "../src/evals/eval-scoring.js"; -import { defaultConfig } from "../src/config/schema.js"; -import { - reconstructComposerGroupsFromArtifacts, - reconstructComposerPolicyFromArtifacts -} from "../src/pipeline/composer.js"; -import { - reconstructVerificationGateFactsFromArtifacts, - reconstructVerifiedFindingsFromArtifacts -} from "../src/pipeline/verifier.js"; -import { createRunTelemetry, reconstructRunTelemetryDerivedEvidence } from "../src/telemetry/run-artifacts.js"; -import type { - CandidateFinding, - EvalArtifacts, - EvalCase, - EvalExpectationResult, - EvalInvocationManifest, - EvalRunInfo, - EvalScore, - FileFacts, - FinalFinding, - ReviewPacket, - ReviewPlan, - TelemetryEvent, - ToolBudget, - UnifiedDiff -} from "../src/types.js"; -import { sha256Hex } from "../src/util/hashing.js"; - -const tempDirs: string[] = []; - -afterEach(async () => { - const { rm } = await import("node:fs/promises"); - await Promise.all(tempDirs.splice(0).map((dir) => rm(dir, { recursive: true, force: true }))); -}); - -const standardBudget: ToolBudget = { - maxToolCalls: 4, - maxInvestigationRounds: 2, - maxResultChars: 10_000 -}; - -function packet( - id: string, - hunkIds: string[], - overrides: Partial = {} -): ReviewPacket { - const hunks = hunkIds.map((hunkId, index) => ({ - // Synthetic atoms and their packed form must describe the same source - // coordinates. Derive the coordinate from canonical hN fixture IDs so a - // separately constructed h2 does not collide with h1 at line 1. - ...(() => { - const match = /(\d+)$/u.exec(hunkId); - const sourceIndex = match === null ? index : Number(match[1]) - 1; - const line = sourceIndex * 100 + 1; - return { - oldStart: line, - newStart: line, - contentWithLineNumbers: `${"".padStart(4)} ${String(line).padStart(4)} +change${sourceIndex}`, - lines: [{ kind: "add" as const, content: `change${sourceIndex}`, newLine: line }], - changedNewLineNumbers: [line] - }; - })(), - hunkId, - oldLines: 1, - newLines: 1, - changedOldLineNumbers: [] - })); - return { - id, - dispatchRank: [0, -hunks.length], - kind: hunks.length === 1 ? "hunk" : "coalesced-hunks", - prSummary: "test", - path: "app.ts", - fileStatus: "modified", - isDeletedContent: false, - language: "typescript", - reviewPriority: "normal", - coverage: "normal", - reviewProfile: "standard", - lenses: ["core/code-review", "lang/typescript"], - hunks, - symbolFacts: [], - context: { path: "app.ts" }, - contextText: "context", - contextQuality: "full", - relevantTests: [], - surroundingContextHints: [], - labels: [], - attentionNotes: [], - relatedChangedContext: [], - toolBudget: structuredClone(standardBudget), - ...overrides - }; -} - -function diffForPackets(packets: ReviewPacket[]): UnifiedDiff { - const hunks = packets.flatMap((target) => target.hunks.map((hunk) => ({ - id: hunk.hunkId, - hunkHash: sha256Hex(hunk.hunkId), - path: target.path, - oldStart: hunk.oldStart, - oldLines: hunk.oldLines, - newStart: hunk.newStart, - newLines: hunk.newLines, - header: hunk.header ?? "", - lines: hunk.lines.map((line) => ({ - kind: line.kind, - content: line.content, - ...(line.oldLine === undefined ? {} : { oldLineNumber: line.oldLine }), - ...(line.newLine === undefined ? {} : { newLineNumber: line.newLine }) - })) - }))); - return { - files: [{ path: "app.ts", status: "modified", language: "typescript", hunks }] - }; -} - -function atomId(hunkIds: string[]): string { - return sha256Hex(`hunk-first\n${hunkIds.join("\n")}`); -} - -function packingEvent( - target: ReviewPacket, - atoms: ReviewPacket[], - overrides: Record = {} -): TelemetryEvent { - const profiles = atoms.map((atom) => atom.reviewProfile); - const profileFloor = profiles.includes("investigate") ? "investigate" : profiles.includes("standard") ? "standard" : "simple"; - return { - runId: "eval-run", - eventId: `event-${target.id}`, - timestamp: "2026-07-24T00:00:00.000Z", - stage: 6, - level: "info", - message: "same_file_atoms_packed", - file: target.path, - data: { - packetId: target.id, - atomIds: atoms.map((atom) => atomId(atom.hunks.map((hunk) => hunk.hunkId))), - standaloneProfiles: profiles, - sourceAtomCount: atoms.length, - hunkCount: target.hunks.length, - effectiveCoverage: target.coverage, - requestedLensSignature: JSON.stringify(["core/code-review"]), - capUsage: { - hunks: target.hunks.length, - maxHunks: 5, - patchChars: target.hunks.reduce((total, hunk) => total + hunk.contentWithLineNumbers.length, 0), - maxPatchChars: 12_000 - }, - derivedPackedProfile: target.reviewProfile, - profileFloor, - effectiveProfile: target.reviewProfile, - profileFloorApplied: false, - plannerLensesPreserved: true, - toolBudgetMode: "base", - baseToolBudget: standardBudget, - effectiveToolBudget: target.toolBudget, - ...overrides - } - }; -} - -function replayFixture(): ReplayAnalysisInput { - const first = packet("off-1", ["h1"]); - const second = packet("off-2", ["h2"]); - const combined = packet("on-1", ["h1", "h2"]); - const diff: UnifiedDiff = { - files: [{ - path: "app.ts", - status: "modified", - language: "typescript", - hunks: [ - { - id: "h1", - hunkHash: "hash1", - path: "app.ts", - oldStart: 1, - oldLines: 1, - newStart: 1, - newLines: 1, - header: "@@ -1 +1 @@", - lines: [{ kind: "add", content: "change0", newLineNumber: 1 }] - }, - { - id: "h2", - hunkHash: "hash2", - path: "app.ts", - oldStart: 101, - oldLines: 1, - newStart: 101, - newLines: 1, - header: "@@ -101 +101 @@", - lines: [{ kind: "add", content: "change1", newLineNumber: 101 }] - } - ] - }] - }; - const facts: FileFacts[] = [{ - path: "app.ts", - language: "typescript", - processingMode: "per-hunk", - testStatus: "source", - isGenerated: false, - isVendored: false, - isLockfile: false, - isBinary: false, - changedLines: 2, - hunkCount: 2, - labels: [], - reviewPriority: "normal", - reasons: [], - provenance: [] - }]; - const plan: ReviewPlan = { - diffUnderstanding: { declaredIntent: "test", inferredBehavior: "test" }, - coverage: [ - { hunkId: "h1", path: "app.ts", coverage: "normal", lenses: ["core/code-review"], surroundingContextHints: [], reason: "test" }, - { hunkId: "h2", path: "app.ts", coverage: "normal", lenses: ["core/code-review"], surroundingContextHints: [], reason: "test" } - ] - }; - return { - runId: "retained-run", - recordedPackets: [first, second], - offPackets: [first, second], - onPackets: [combined], - onEvents: [packingEvent(combined, [first, second])], - fileFacts: facts, - diff, - plan, - expectedRefs: { base: "a".repeat(40), head: "b".repeat(40) }, - actualRefs: { base: "a".repeat(40), head: "b".repeat(40) } - }; -} - -function failureCodes(failures: ReportFailure[]): string[] { - return failures.map((entry) => entry.code); -} - -describe("packet packing replay analysis", () => { - it("reports packet shape, profile provenance, caps, and dispatch movement", () => { - const report = analyzeReplayComparison(replayFixture()); - expect(report.failures).toEqual([]); - expect(report).toMatchObject({ - offPackets: 2, - onPackets: 1, - sourceAtoms: 2, - packedMultiAtomPackets: 1, - reviewableHunks: 2, - newCoveragePromotions: 0, - capViolations: 0, - effectiveProfileDowngrades: 0, - effectiveBudgetDowngrades: 0, - invalidDispatchRanks: 0 - }); - expect(report.packetMembership[0]?.atomIds).toEqual([atomId(["h1"]), atomId(["h2"])]); - }); - - it("fails closed on stale refs and flag-off artifact drift", () => { - const input = replayFixture(); - input.actualRefs = { ...input.actualRefs!, head: "c".repeat(40) }; - input.recordedPackets[0] = { ...input.recordedPackets[0]!, contextText: "recorded-only" }; - const report = analyzeReplayComparison(input); - expect(failureCodes(report.failures)).toEqual(expect.arrayContaining(["stale_replay_refs", "flag_off_parity"])); - }); - - it("never includes proprietary source text in parity failures and rejects replay model calls", () => { - const proprietary = "PROPRIETARY_SOURCE_NEVER_REPORT_THIS"; - const input = replayFixture(); - input.recordedPackets[0] = { ...input.recordedPackets[0]!, contextText: proprietary }; - input.modelCallsObserved = 1; - const report = analyzeReplayComparison(input); - expect(failureCodes(report.failures)).toEqual(expect.arrayContaining(["flag_off_parity", "replay_model_call"])); - const serialized = JSON.stringify(report); - expect(serialized).not.toContain(proprietary); - expect(serialized).toContain("sha256"); - }); - - it("fails closed when replay cleanup cannot remove its worktree", async () => { - let tempCleanupCalled = false; - await expect(finalizeReplayCleanup( - "retained-run", - true, - async () => { throw new Error("worktree busy"); }, - async () => { tempCleanupCalled = true; }, - async () => false - )).rejects.toThrow(/failed to remove replay worktree/u); - expect(tempCleanupCalled).toBe(true); - }); - - it("unconditionally cleans and verifies a partially-added worktree", async () => { - let removalAttempted = false; - let tempCleanupCalled = false; - await expect(finalizeReplayCleanup( - "partial-worktree-add", - true, - async () => { removalAttempted = true; throw new Error("not registered"); }, - async () => { tempCleanupCalled = true; }, - async () => true - )).resolves.toBeUndefined(); - expect(removalAttempted).toBe(true); - expect(tempCleanupCalled).toBe(true); - }); - - it("admits only the pinned pre-Plan100 dispatch-rank artifact migration", () => { - const input = replayFixture(); - input.runId = "20260724-150405-fe1548ae"; - input.recordedPackets = structuredClone(input.recordedPackets); - for (const packet of input.recordedPackets) { - delete (packet as Partial).dispatchRank; - } - const parity = historicalFlagOffParityView(input.runId, input.recordedPackets, input.offPackets); - input.flagOffParityPackets = parity.packets; - input.flagOffParityMigrations = parity.migrations; - const report = analyzeReplayComparison(input); - expect(failureCodes(report.failures)).not.toContain("flag_off_parity"); - expect(report.flagOffParityDifferences.historicalMigrations).toEqual([ - expect.objectContaining({ code: "pre_plan100_dispatch_rank_schema", packets: 2 }) - ]); - - input.recordedPackets[0]!.contextText = "unrecognized historical drift"; - expect(failureCodes(analyzeReplayComparison(input).failures)).toContain("flag_off_parity"); - }); - - it("admits the legacy diff schema for exactly the three retained run IDs", () => { - const rebuilt = replayFixture().diff; - const legacy = structuredClone(rebuilt); - for (const hunk of legacy.files.flatMap((file) => file.hunks)) { - delete (hunk as Partial).hunkHash; - } - expect(() => validateRecordedDiffParity("20260724-135818-740d73f2", legacy, rebuilt)).not.toThrow(); - expect(() => validateRecordedDiffParity("20260724-000000-unknown000", legacy, rebuilt)).toThrow(/not allowlisted/u); - }); - - it("fails closed on hunk loss, atom reorder, caps, and coverage promotion", () => { - const input = replayFixture(); - const reversed = packet("bad-on", ["h2", "h1"], { coverage: "deep" }); - input.onPackets = [reversed]; - input.onEvents = [packingEvent(reversed, [input.offPackets[0]!, input.offPackets[1]!], { - effectiveCoverage: "deep", - capUsage: { hunks: 6, maxHunks: 5, patchChars: 12_001, maxPatchChars: 12_000 } - })]; - const report = analyzeReplayComparison(input); - expect(JSON.stringify(report)).not.toContain("inspect critical boundary"); - expect(failureCodes(report.failures)).toEqual(expect.arrayContaining([ - "source_order", - "atom_split_or_reorder", - "packet_cap", - "coverage_promotion" - ])); - - const lost = replayFixture(); - lost.onPackets = [lost.onPackets[0] = packet("lost", ["h1"])]; - lost.onEvents = [packingEvent(lost.onPackets[0]!, [lost.offPackets[0]!])]; - expect(failureCodes(analyzeReplayComparison(lost).failures)).toContain("hunk_bijection"); - }); - - it("derives eligible-packet patch size from the diff and enforces complete packet source order", () => { - const falseCap = replayFixture(); - const falseCapEvent = falseCap.onEvents[0]!; - falseCapEvent.data = { - ...falseCapEvent.data, - capUsage: { ...(falseCapEvent.data!.capUsage as object), patchChars: 1 } - }; - const capReport = analyzeReplayComparison(falseCap); - expect(failureCodes(capReport.failures)).toContain("packet_cap"); - expect(capReport.distribution.maxEligiblePackingPatchChars).toBeGreaterThan(1); - - const reordered = replayFixture(); - reordered.onPackets = [reordered.offPackets[1]!, reordered.offPackets[0]!]; - reordered.onEvents = []; - expect(failureCodes(analyzeReplayComparison(reordered).failures)).toContain("source_packet_order"); - }); - - it("fails closed on lens/focus/context/profile/budget/rank regressions", () => { - const input = replayFixture(); - input.offPackets[0] = { - ...input.offPackets[0]!, - reviewPriority: "high", - coverage: "deep", - reviewProfile: "investigate", - attentionNotes: ["inspect critical boundary"], - toolBudget: { ...standardBudget, maxToolCalls: 6, maxResultChars: 12_000 } - }; - input.recordedPackets = structuredClone(input.offPackets); - input.onPackets[0] = { - ...input.onPackets[0]!, - coverage: "deep", - reviewProfile: "standard", - lenses: ["core/code-review"], - contextQuality: "path_only", - dispatchRank: [0, -99] - }; - input.onEvents = [packingEvent(input.onPackets[0]!, input.offPackets, { - standaloneProfiles: ["investigate", "standard"], - effectiveCoverage: "deep", - profileFloor: "investigate", - effectiveProfile: "standard", - baseToolBudget: standardBudget, - effectiveToolBudget: { ...standardBudget, maxToolCalls: 2 } - })]; - const report = analyzeReplayComparison(input); - expect(failureCodes(report.failures)).toEqual(expect.arrayContaining([ - "planner_lens_drop", - "high_priority_focus_omitted", - "deep_context_downgrade", - "effective_profile_downgrade", - "effective_budget_downgrade", - "invalid_dispatch_rank" - ])); - }); - - it("rejects motivating counts outside the bounded 74/75/76 reconciliation", () => { - const input = replayFixture(); - input.runId = "20260724-184952-dca8d870"; - const report = analyzeReplayComparison(input); - expect(failureCodes(report.failures)).toEqual(expect.arrayContaining(["motivating_off_count", "unpermitted_packet_count"])); - }); - - it("writes a fail-closed report for corrupt retained-run artifacts", async () => { - const root = await mkdtemp(path.join(tmpdir(), "packet-report-corrupt-")); - tempDirs.push(root); - const run = path.join(root, "bad-run"); - await mkdir(path.join(run, "stages", "01-input"), { recursive: true }); - await writeFile(path.join(run, "stages", "01-input", "resolved-input.json"), "{not-json\n"); - const output = path.join(root, "report.json"); - const exitCode = await runPacketPackingReportCli(["replay", "--repo", process.cwd(), "--run", run, "--output", output]); - expect(exitCode).toBe(1); - const report = JSON.parse(await readFile(output, "utf8")) as { failures: ReportFailure[] }; - expect(failureCodes(report.failures)).toContain("corrupt_artifact"); - }); - - it("rejects unknown artifact fields and disagreement across replay refs", async () => { - const root = await mkdtemp(path.join(tmpdir(), "packet-report-schema-")); - tempDirs.push(root); - const run = path.join(root, "strict-run"); - await mkdir(path.join(run, "stages", "01-input"), { recursive: true }); - await writeFile(path.join(run, "stages", "01-input", "resolved-input.json"), JSON.stringify({ - mode: "head", - repoRoot: process.cwd(), - baseRef: "a".repeat(40), - headRef: "b".repeat(40), - commits: [], - rawDiffChars: 0, - unexpected: "PROPRIETARY_ARTIFACT_VALUE" - })); - const output = path.join(root, "report.json"); - expect(await runPacketPackingReportCli(["replay", "--repo", process.cwd(), "--run", run, "--output", output])).toBe(1); - const schemaReport = JSON.parse(await readFile(output, "utf8")) as { failures: ReportFailure[] }; - expect(failureCodes(schemaReport.failures)).toContain("corrupt_artifact_schema"); - expect(await readFile(output, "utf8")).not.toContain("PROPRIETARY_ARTIFACT_VALUE"); - - const replay = replayFixture(); - expect(() => validateReplayArtifacts( - "retained-run", - { - mode: "head", - repoRoot: process.cwd(), - baseRef: "a".repeat(40), - headRef: "b".repeat(40), - commits: [], - rawDiffChars: 0 - }, - replay.diff, - [{ path: "app.ts", action: "keep", reason: "test", provenance: [] }], - replay.fileFacts, - replay.plan, - { - runId: "retained-run", - mode: "head", - depth: "normal", - target: { baseRef: "c".repeat(40), headRef: "b".repeat(40) }, - commits: [], - policyFilesChanged: [], - hunkIndex: [], - files: [], - directories: [], - filterSummary: { keptFiles: 1, skippedFiles: 0, skipped: [] }, - lenses: [], - totals: { files: 1, keptFiles: 1, hunks: 2, addedLines: 2, deletedLines: 0 }, - compaction: {} as never - }, - { runId: "retained-run" } - )).toThrow(/disagree on base\/head refs/u); - }); - - it("fails closed at every packet/eval telemetry artifact schema boundary", () => { - for (const kind of [ - "packet", "event", "model-call", "tool-call", "coverage", "eval-info", "attention", "human-attention", - "budget-summary", "cost-profile", "model-summary", "tool-summary", "run-summary", "telemetry-summary" - ] as const) { - expect(() => validatePacketPackingArtifact(kind, { unexpected: true }), kind).toThrow(/invalid .* artifact/u); - } - - const target = packet("strict-packet", ["h1", "h2"]); - const event = packingEvent(target, [packet("strict-a", ["h1"]), packet("strict-b", ["h2"])]); - (event.data as Record).unexpected = true; - expect(() => validatePacketPackingArtifact("event", event)).toThrow(/invalid event artifact/u); - - const nestedCase = evalRun("strict", "A", 1, [execution("A", 1)]).info; - (nestedCase.caseSnapshot.review as Record).unexpected = true; - expect(() => validatePacketPackingArtifact("eval-info", nestedCase)).toThrow(/invalid eval-info artifact/u); - - const toolCall = { - runId: "run", - toolCallId: "tool-call", - timestamp: "2026-07-24T00:00:00.000Z", - stage: 7, - initiator: "model", - tool: "read_file", - args: { path: "app.ts", maxResults: 40 }, - backend: "text", - precision: "exact", - degraded: false, - resultChars: 1, - durationMs: 1, - status: "ok" - }; - expect(() => validatePacketPackingArtifact("tool-call", toolCall)).not.toThrow(); - expect(() => validatePacketPackingArtifact("tool-call", { - ...toolCall, - args: { ...toolCall.args, unexpected: "nested" } - })).toThrow(/invalid tool-call artifact/u); - }); - - it("accepts current producer-shaped model, tool, cost, run, and telemetry summaries", async () => { - const repoRoot = await mkdtemp(path.join(tmpdir(), "packet-report-summary-producer-")); - tempDirs.push(repoRoot); - const run = createRunTelemetry({ - telemetryConfig: { ...defaultConfig.telemetry, enabled: true, logLevel: "debug" }, - idFactory: () => "20260725-000000-report-summary" - }); - const attached = await run.attachRunDirectory(repoRoot); - run.recorder.event({ stage: 10, level: "info", message: "stage_started", cacheStatus: "write" }); - run.recorder.recordModelCall({ - callId: "summary-call", stage: 10, role: "composer", model: "model", provider: "provider", kind: "initial", attempt: 1, - promptChars: 12, promptHash: "prompt", outputChars: 5, outputHash: "output", inputTokens: 10, outputTokens: 2, - reasoningTokens: 1, totalTokens: 13, costUSD: 0.01, durationMs: 10, cacheStatus: "miss", schemaValid: false, - stopReason: "submit", status: "schema_invalid", errorCode: "llm_schema_invalid" - }); - run.recorder.event({ - stage: 10, level: "info", message: "schema_invalid_submit_recovered", - data: { submitTool: "submit_composition", invalidSubmitCallCount: 1 } - }); - run.recorder.recordToolCall({ - stage: 7, initiator: "harness", tool: "read_file", args: { path: "app.ts" }, backend: "text", precision: "exact", - degraded: false, cacheStatus: "write", backendExecuted: true, resultChars: 10, durationMs: 2, status: "ok" - }); - run.recorder.event({ stage: 10, level: "info", message: "stage_completed" }); - await run.finalize({ status: "completed_full", exitCode: 0 }); - for (const [kind, filename] of [ - ["model-summary", "stages/00-run/model-calls-summary.json"], - ["tool-summary", "stages/00-run/tool-calls-summary.json"], - ["cost-profile", "stages/00-run/cost-profile.json"], - ["run-summary", "run.json"], - ["telemetry-summary", "telemetry.json"] - ] as const) { - const value = JSON.parse(await readFile(path.join(attached.runDir, filename), "utf8")) as unknown; - expect(() => validatePacketPackingArtifact(kind, value), kind).not.toThrow(); - } - }); - - it("persists raw evidence for nonzero buffered log overflow", async () => { - const repoRoot = await mkdtemp(path.join(tmpdir(), "packet-report-log-overflow-")); - tempDirs.push(repoRoot); - const run = createRunTelemetry({ - telemetryConfig: { ...defaultConfig.telemetry, enabled: true, logLevel: "debug" }, - idFactory: () => "20260725-000000-log-overflow" - }); - for (let index = 0; index < 1_010; index += 1) { - run.logger.debug({ runId: run.recorder.runId, stage: 0, event: "buffered", message: `buffered ${index}` }); - } - const attached = await run.attachRunDirectory(repoRoot); - await run.finalize({ status: "completed_full", exitCode: 0 }); - const events = (await readFile(path.join(attached.runDir, "events.jsonl"), "utf8")) - .trim().split("\n").filter(Boolean).map((line) => JSON.parse(line) as TelemetryEvent); - const derived = reconstructRunTelemetryDerivedEvidence(events, []); - const summary = JSON.parse(await readFile(path.join(attached.runDir, "telemetry.json"), "utf8")) as { - logs: { bufferedOverflow: { droppedDebugInfo: number; droppedWarnError: number } }; - }; - expect(derived.logOverflow.droppedDebugInfo).toBeGreaterThan(0); - expect(derived.logOverflow).toEqual(summary.logs.bufferedOverflow); - }); -}); - -function evalCase(family: string, arm: "A" | "B" | "C", repeat: number): EvalCase { - const suffix = arm.toLowerCase(); - return { - name: `${family}-${suffix}`, - repeat, - repo: { fixture: `repos/${family}` }, - review: { - depth: "normal", - lenses: ["core/code-review"], - cache: false, - packSameFileHunks: arm !== "A", - packedToolBudgetMode: arm === "C" ? "atom-scaled" : "base" - }, - should_find: [{ id: "bug", path: "app.ts", lineRange: [1, 1], titlePattern: "boundary" }] - }; -} - -function expectation(hit: boolean, loss: EvalExpectationResult["loss"] = undefined): EvalExpectationResult { - return { - expectationId: "bug", - list: "should_find", - tier: "required", - status: hit ? "pass" : "fail", - fromReplayedArtifacts: false, - matched: hit ? [{ findingId: "finding", artifact: "final-findings" }] : [], - ...(loss === undefined ? {} : { loss }) - }; -} - -function score(hit: boolean, cost: number | undefined = 1, lossLabel = "missed-before-candidate-generation"): EvalScore { - const loss = hit ? undefined : { - label: lossLabel as "missed-before-candidate-generation", - nearestInstances: [] - }; - return { - status: hit ? "pass" : "fail", - expectationResults: [expectation(hit, loss)], - budgetResults: cost === undefined ? [] : [{ - check: "maxCostUSD", - status: "pass", - actual: cost, - limit: 100, - direction: "maximum" - }], - violations: [], - nearViolations: [], - metrics: { - reportedFindings: hit ? 1 : 0, - inlineFindings: hit ? 1 : 0, - summaryOnlyFindings: 0, - suppressedFindings: 0, - candidateFindings: hit ? 1 : 0, - duplicateGroups: 0, - ...(cost === undefined ? {} : { costUSD: cost }), - stageLossCounts: { - "missed-before-candidate-generation": hit ? 0 : 1, - "lost-at-verification": 0, - "lost-at-composition": 0, - "partial-match": 0 - } - } - }; -} - -function modelCall(packetId: string, repeat: number, costUSD = 0.1) { - return { - callId: `call-${packetId}-${repeat}`, - runId: "eval", - stage: 7 as const, - role: "packetReview" as const, - model: "fake", - provider: "fake", - packetId, - kind: "initial" as const, - attempt: 1, - promptChars: 100, - promptHash: "hash", - outputChars: 10, - outputHash: "hash", - inputTokens: 100, - outputTokens: 10, - reasoningTokens: 0, - totalTokens: 110, - costUSD, - durationMs: 1000, - cacheStatus: "disabled" as const, - stopReason: "submit" as const, - status: "ok" as const - }; -} - -function candidateFinding(id: string, packetId: string, hunkId: string, line = 1): CandidateFinding { - return { - id, - title: "Boundary bug", - severity: "medium", - confidence: "medium", - path: "app.ts", - anchor: { path: "app.ts", line, side: "RIGHT", hunkId }, - anchorSource: "model", - modelAnchorSubmitted: true, - changedLine: true, - category: "correctness", - evidence: { changedCode: "+change" }, - failureMode: "The boundary behavior is incorrect.", - whyThisMatters: "The changed code produces an incorrect result.", - verification: "The changed line demonstrates the failure.", - producedBy: { kind: "packet", stage: 7, packetId, lensId: "core/code-review", skillIds: ["core/code-review"] } - }; -} - -function finalFinding(candidate: CandidateFinding, publication: FinalFinding["publication"] = "inline"): FinalFinding { - return { - ...structuredClone(candidate), - fingerprint: `fingerprint-${candidate.id}`, - finalBody: candidate.failureMode, - publication, - mergedCandidateIds: [candidate.id], - mergedAnchors: candidate.anchor === undefined ? [] : [structuredClone(candidate.anchor)] - }; -} - -function findingEvidence(candidate: CandidateFinding | undefined): Pick { - if (candidate === undefined) { - return { candidateFindings: [], verification: [], finalSelection: [], finalFindings: [] }; - } - return { - candidateFindings: [candidate], - verification: [{ - candidateId: candidate.id, - gate: "passed", - gateDecision: "scheduled", - gateReason: "meets_confidence_threshold", - verificationLane: "standard", - gateFacts: { - anchorSource: "model", - category: candidate.category, - changedLine: true, - confidence: candidate.confidence, - failureModeConcrete: true, - hasChangedCode: true, - hasFailureMode: true, - modelAnchorSubmitted: true, - modelAnchorValid: true, - relatedEvidenceCount: 0, - severity: candidate.severity, - validAnchorPresent: true - }, - verdict: { - candidateId: candidate.id, - verdict: "keep", - reason: "evidence confirmed", - requiredEvidencePresent: true, - falsePositiveRisk: "low" - } - }], - finalSelection: [{ findingId: candidate.id, decision: "published", reason: "composer-selected" }], - finalFindings: [finalFinding(candidate)] - }; -} - -function executionArtifacts(execution: EvalExecutionInput): EvalArtifacts { - const existing = execution.scoringArtifacts as EvalArtifacts | undefined; - const persisted = { - ...existing, - candidates: execution.candidateFindings, - verification: execution.verification, - finalSelection: execution.finalSelection, - finalFindings: execution.finalFindings, - packets: execution.packets, - hintEvents: existing?.hintEvents ?? [], - reviewPlan: execution.plan, - metricsSources: { - ...existing?.metricsSources, - modelCalls: execution.modelCalls, - toolCalls: execution.toolCalls, - costProfile: { totalCostUSD: execution.modelCalls.reduce((total, call) => total + (call.costUSD ?? 0), 0) }, - runJson: { durationMs: execution.wallTimeSeconds * 1000 } - } - } as EvalArtifacts; - return reconstructEvidenceArtifacts({ ...execution, scoringArtifacts: persisted }); -} - -function refreshExecutionEvidence(execution: EvalExecutionInput, policyConfig: typeof defaultConfig = defaultConfig): void { - const candidateById = new Map(execution.candidateFindings.map((candidate) => [candidate.id, candidate])); - const packetById = new Map(execution.packets.map((packet) => [packet.id, packet])); - execution.verification = execution.verification.map((record) => { - const candidateId = "verdict" in record && record.duplicateOf !== undefined - ? record.duplicateOf - : record.candidateId; - const candidate = candidateById.get(candidateId); - if (candidate === undefined) { - return record; - } - return { - ...record, - gateFacts: reconstructVerificationGateFactsFromArtifacts( - candidate, - packetById.get(candidate.producedBy.packetId), - execution.diff - ) - }; - }); - const verified = reconstructVerifiedFindingsFromArtifacts(execution.candidateFindings, execution.verification, execution.packets, execution.diff); - const composer = reconstructComposerGroupsFromArtifacts(verified, execution.packets); - const publishableById = new Map(composer.publishable.map((finding) => [finding.id, finding])); - const selectionById = new Map(execution.finalSelection.map((record) => [record.findingId, record])); - const downgradeReasons = new Set(["min-inline-confidence", "soft-comment-cap", "unanchorable"]); - const suppressionReasons = new Set(["severity-threshold", "confidence-threshold", "report-cap"]); - const drafts = execution.finalFindings.map((finding) => { - const selections = finding.mergedCandidateIds.flatMap((id) => selectionById.get(id) ?? []); - const isPretrimmed = finding.mergedCandidateIds.some((id) => composer.pretrimSuppressedIds.includes(id)); - const hasDowngrade = selections.some((selection) => downgradeReasons.has(selection.reason)); - const hasPolicyReason = hasDowngrade || selections.some((selection) => suppressionReasons.has(selection.reason)); - return { - mergedFindings: finding.mergedCandidateIds.flatMap((id) => publishableById.get(id) ?? []), - finalBody: finding.finalBody, - requestedPublication: isPretrimmed - ? "suppressed" as const - : finding.publication === "inline" || hasDowngrade ? "inline" as const : finding.publication, - baseSelection: finding.mergedCandidateIds.map((id) => isPretrimmed - ? { findingId: id, decision: "suppressed" as const, reason: "composer-pre-trim" } - : id === finding.id - ? { - findingId: id, - decision: "published" as const, - reason: !hasPolicyReason && selectionById.get(id)?.reason === "composer_omitted_finding" - ? "composer_omitted_finding" - : "composer-selected" - } - : { findingId: id, decision: "merged" as const, reason: "composer-merged", mergedIntoFingerprint: finding.fingerprint }) - }; - }); - let policy = reconstructComposerPolicyFromArtifacts( - drafts, - execution.packets, - execution.diff, - policyConfig, - execution.verification.flatMap((record) => - "verdict" in record && (policyConfig.review.verify === false || record.duplicateOf === undefined) ? [record.verdict] : [] - ) - ); - const fingerprintByCandidateId = new Map(policy.findings.flatMap((finding) => - finding.mergedCandidateIds.map((id) => [id, finding.fingerprint] as const) - )); - if (drafts.some((draft) => draft.baseSelection.some((record) => record.decision === "merged"))) { - policy = reconstructComposerPolicyFromArtifacts( - drafts.map((draft) => ({ - ...draft, - baseSelection: draft.baseSelection.map((record) => record.decision === "merged" - ? { ...record, mergedIntoFingerprint: fingerprintByCandidateId.get(record.findingId) ?? record.mergedIntoFingerprint ?? "missing" } - : record) - })), - execution.packets, - execution.diff, - policyConfig, - execution.verification.flatMap((record) => - "verdict" in record && (policyConfig.review.verify === false || record.duplicateOf === undefined) ? [record.verdict] : [] - ) - ); - } - execution.finalFindings = policy.findings; - execution.finalSelection = policy.selection; - execution.finalSelectionArtifact = { - composition: { mode: "llm" }, - records: structuredClone(policy.selection), - publicationAnchors: structuredClone(policy.publicationAnchors), - confidenceSelections: structuredClone(policy.confidenceSelections), - groups: composer.groups.map(({ representativeId: _representativeId, ...group }) => group) - }; - execution.scoringArtifacts = executionArtifacts(execution); -} - -function refreshRunEvidenceScores(run: EvalCaseRunInput): void { - const policyConfig = structuredClone(defaultConfig); - if (run.info.effectiveConfig !== undefined) { - policyConfig.review = { ...policyConfig.review, ...run.info.effectiveConfig.review }; - } - for (const execution of run.executions) { - refreshExecutionEvidence(execution, policyConfig); - execution.score = scoreEvalRun(run.declaredCase, executionArtifacts(execution), "live"); - execution.scoringArtifacts = executionArtifacts(execution); - } - if (run.executions.length === 1) { - run.info.score = run.executions[0]!.score; - delete run.info.repeats; - return; - } - const repeated = aggregateRepeatScores(run.declaredCase, run.executions.map((execution) => ({ - runDir: `repeats/${execution.repeat}`, - score: execution.score, - artifacts: executionArtifacts(execution) - }))); - run.info.score = repeated.score; - run.info.repeats = repeated.aggregate; -} - -function planForHunks(hunkIds: string[]): ReviewPlan { - return { - diffUnderstanding: { declaredIntent: "test", inferredBehavior: "test" }, - coverage: hunkIds.map((hunkId) => ({ - hunkId, - path: "app.ts", - coverage: "normal" as const, - lenses: ["core/code-review"], - surroundingContextHints: [], - reason: "test" - })) - }; -} - -function execution( - arm: "A" | "B" | "C", - repeat: number, - options: { treated?: boolean; hit?: boolean; cost?: number | undefined; missingTelemetry?: boolean } = {} -): EvalExecutionInput { - const first = packet(`a1-${repeat}`, ["h1"]); - const second = packet(`a2-${repeat}`, ["h2"]); - const treated = options.treated ?? arm !== "A"; - const packets = arm === "A" || !treated - ? [first, second] - : [packet(`packed-${arm}-${repeat}`, ["h1", "h2"], { - toolBudget: arm === "C" ? { ...standardBudget, maxToolCalls: 5, maxResultChars: 12_000 } : standardBudget - })]; - const events = arm === "A" || options.missingTelemetry === true - ? [] - : treated - ? [packingEvent(packets[0]!, [first, second], { - toolBudgetMode: arm === "C" ? "atom-scaled" : "base", - effectiveToolBudget: packets[0]!.toolBudget - })] - : packets.map((target, index) => packingEvent(target, [index === 0 ? first : second], { - toolBudgetMode: arm === "C" ? "atom-scaled" : "base" - })); - const executionScore = score(options.hit ?? true, options.cost === undefined && "cost" in options ? undefined : options.cost ?? 1); - const hit = options.hit ?? true; - const calls = packets.map((target) => modelCall(target.id, repeat, (options.cost ?? 1) / packets.length)); - if ("cost" in options && options.cost === undefined) { - for (const call of calls) { - delete (call as Partial).costUSD; - } - } - executionScore.metrics.modelCalls = calls.length; - executionScore.metrics.verificationCalls = hit ? 1 : 0; - executionScore.metrics.elapsedSeconds = 60; - const candidate = hit ? candidateFinding("finding", packets[0]!.id, "h1") : undefined; - const result = { - repeat, - score: executionScore, - telemetryDir: `/tmp/${arm}/${repeat}`, - packets, - events, - modelCalls: calls, - toolCalls: [], - fileFacts: [{ - path: "app.ts", - language: "typescript", - processingMode: "per-hunk", - testStatus: "source", - isGenerated: false, - isVendored: false, - isLockfile: false, - isBinary: false, - changedLines: 2, - hunkCount: 2, - labels: [], - reviewPriority: "normal", - reasons: [], - provenance: [] - }], - diff: diffForPackets([first, second]), - plan: planForHunks(["h1", "h2"]), - ...findingEvidence(candidate), - reviewedHunkIds: ["h1", "h2"], - wallTimeSeconds: 60 - } as unknown as EvalExecutionInput; - refreshExecutionEvidence(result); - return result; -} - -function evalRun( - family: string, - arm: "A" | "B" | "C", - runNumber: number, - executions: EvalExecutionInput[] -): EvalCaseRunInput { - const snapshot = evalCase(family, arm, executions.length); - for (const entry of executions) { - refreshExecutionEvidence(entry); - entry.score = scoreEvalRun(snapshot, entry.scoringArtifacts, "live"); - } - const repeated = executions.length > 1 ? aggregateRepeatScores(snapshot, executions.map((entry) => ({ - runDir: `repeats/${entry.repeat}`, - score: entry.score, - artifacts: executionArtifacts(entry) - }))) : undefined; - const info: EvalRunInfo = { - runNumber, - caseName: snapshot.name, - caseFile: `${family}-${arm.toLowerCase()}.yml`, - caseHash: `hash-${family}-${arm}`, - caseSnapshot: snapshot, - mode: "live", - ...(repeated === undefined ? {} : { repeats: repeated.aggregate }), - cache: { enabled: false, source: "case" }, - effectiveConfig: { - review: { - concurrency: 1, - timeoutMs: 60_000, - verify: true, - maxFindings: defaultConfig.review.maxFindings, - softCommentCap: defaultConfig.review.softCommentCap, - minConfidence: defaultConfig.review.minConfidence, - minInlineConfidence: defaultConfig.review.minInlineConfidence, - packSameFileHunks: arm !== "A", - packedToolBudgetMode: arm === "C" ? "atom-scaled" : "base" - }, - llm: { maxConcurrentCalls: 1 } - }, - codegenieRuntime: { packageVersion: "0.5.0", commit: "a".repeat(40), source: "git" }, - startedAt: "2026-07-24T00:00:00.000Z", - finishedAt: "2026-07-24T00:01:00.000Z", - score: repeated?.score ?? executions[0]?.score ?? score(true) - }; - return { runNumber, runDir: `/logs/${runNumber}`, info, declaredCase: structuredClone(snapshot), executions }; -} - -async function attachProducerSummaryEvidence(run: EvalCaseRunInput): Promise { - const execution = run.executions[0]!; - const repoRoot = await mkdtemp(path.join(tmpdir(), "packet-report-paid-summary-")); - tempDirs.push(repoRoot); - let clockTick = 0; - const telemetry = createRunTelemetry({ - telemetryConfig: { ...defaultConfig.telemetry, enabled: true, logLevel: "debug" }, - idFactory: () => `paid-summary-${run.runNumber}`, - clock: () => new Date(Date.UTC(2026, 6, 25, 0, 0, clockTick++)) - }); - const attached = await telemetry.attachRunDirectory(repoRoot); - telemetry.recorder.event({ stage: 7, level: "info", message: "stage_started" }); - telemetry.recorder.event({ - stage: 7, - level: "warn", - message: "stage7_schema_repair_attempted", - data: { classification: "candidate_payload", payloadKind: "candidate" } - }); - telemetry.recorder.event({ - stage: 7, - level: "info", - message: "stage7_schema_compact_repair_scheduled", - data: { repairPromptChars: execution.modelCalls[0]?.promptChars ?? 0 } - }); - for (const [index, call] of execution.modelCalls.entries()) { - const { runId: _runId, ...record } = call; - telemetry.recorder.recordModelCall(index === 0 ? { ...record, kind: "repair" } : record); - } - telemetry.recorder.event({ - stage: 7, - level: "info", - message: "stage7_schema_repair_recovered", - data: { classification: "schema_valid_after_retry" } - }); - telemetry.recorder.event({ stage: 7, level: "info", message: "stage_completed" }); - const reviewedHunks = execution.reviewedHunkIds.length; - telemetry.recorder.event({ - stage: 10, - level: "info", - message: "pipeline_metrics", - data: { - totals: { - filesChanged: execution.diff.files.length, - hunks: execution.diff.files.flatMap((file) => file.hunks).length, - packets: execution.packets.length, - packetReviews: execution.packets.length, - candidates: execution.candidateFindings.length, - verified: execution.verification.filter((record) => "verdict" in record && record.verdict.verdict !== "reject").length, - finalFindings: execution.finalFindings.length, - postedComments: 0 - }, - workers: { started: 2, completed: 2, failed: 0, retried: 0, timedOut: 0 }, - packets: { generated: execution.packets.length, reviewed: execution.packets.length, failed: 0, degraded: 0 }, - lenses: { selected: 1, byLens: { "core/code-review": execution.packets.length } }, - coverage: { - byLevel: { deep: 0, normal: reviewedHunks, light: 0, skip: 0 }, - hunks: { total: reviewedHunks, reviewed: reviewedHunks, skipped: 0, failed: 0, degraded: 0 } - }, - candidates: { - generated: execution.candidateFindings.length, - gateRejected: 0, - verificationScheduled: execution.verification.length, - verificationBudgetLimited: 0, - clusteredDuplicates: 0, - verificationRepresentatives: execution.verification.length, - lowConfidenceSuppressed: 0, - lowConfidenceEvidenceEligible: 0, - lowConfidenceEvidenceScheduled: 0, - lowConfidenceEvidenceLaneLimited: 0, - lowConfidenceEvidenceKept: 0, - lowConfidenceEvidenceRejected: 0, - lowConfidenceEvidenceIncomplete: 0 - }, - verdicts: { accept: execution.verification.length, revise: 0, reject: 0, incomplete: 0 }, - dedup: { clusters: execution.finalFindings.length, duplicates: 0, suppressed: 0 }, - finalSelection: { - published: execution.finalSelection.filter((record) => record.decision === "published").length, - merged: execution.finalSelection.filter((record) => record.decision === "merged").length, - suppressed: execution.finalSelection.filter((record) => record.decision === "suppressed").length, - finalFindings: execution.finalFindings.length, - compositionMode: "llm" - }, - posting: { attempted: 0, postedComments: 0, skippedDuplicates: 0, failed: 0 } - } - }); - await telemetry.finalize({ status: "completed_full", exitCode: 0 }); - const readJson = async (relative: string): Promise => JSON.parse(await readFile(path.join(attached.runDir, relative), "utf8")); - const readJsonl = async (relative: string): Promise => (await readFile(path.join(attached.runDir, relative), "utf8")) - .trim().split("\n").filter((line) => line.length > 0).map((line) => JSON.parse(line) as unknown); - execution.events = await readJsonl("events.jsonl") as TelemetryEvent[]; - execution.modelCalls = await readJsonl("model-calls.jsonl") as EvalExecutionInput["modelCalls"]; - execution.toolCalls = await readJsonl("tool-calls.jsonl") as EvalExecutionInput["toolCalls"]; - const runSummary = await readJson("run.json") as { startedAt: string; finishedAt: string; durationMs: number }; - const modelByStage = new Map(); - for (const call of execution.modelCalls.filter((call) => call.cacheStatus !== "hit")) { - const bucket = modelByStage.get(call.stage) ?? { modelCalls: 0, totalTokens: 0 }; - bucket.modelCalls += 1; - bucket.totalTokens += call.totalTokens ?? 0; - modelByStage.set(call.stage, bucket); - } - const totalTokens = execution.modelCalls.reduce((total, call) => total + (call.totalTokens ?? 0), 0); - const totalCostUSD = execution.modelCalls.reduce((total, call) => total + (call.costUSD ?? 0), 0); - execution.wallTimeSeconds = runSummary.durationMs / 1000; - execution.summaryArtifacts = { - attention: execution.packets.map((packet) => ({ - packetId: packet.id, - path: packet.path, - coverage: packet.coverage, - coverageSource: "planner", - ensemblePasses: 1, - directCandidates: execution.candidateFindings.filter((candidate) => candidate.producedBy.packetId === packet.id && candidate.provenance === undefined).length, - promotedCandidates: execution.candidateFindings.filter((candidate) => candidate.producedBy.packetId === packet.id && candidate.provenance !== undefined).length, - hintsEmitted: 0, - uncertaintiesEmitted: 0, - keptVerified: execution.verification.filter((record) => record.candidateId === execution.candidateFindings.find((candidate) => candidate.producedBy.packetId === packet.id)?.id && "verdict" in record).length, - published: execution.finalFindings.filter((finding) => finding.publication !== "suppressed" && - finding.mergedCandidateIds.some((id) => execution.candidateFindings.find((candidate) => candidate.id === id)?.producedBy.packetId === packet.id)).length - })), - humanAttention: [], - budget: { - completeness: "complete", - partialReasons: [], - multiplier: 1, - configured: { timeoutMs: run.info.effectiveConfig!.review.timeoutMs }, - effective: { timeoutMs: run.info.effectiveConfig!.review.timeoutMs }, - usage: { - modelCalls: execution.modelCalls.filter((call) => call.cacheStatus !== "hit").length, - totalTokens, - ...(totalCostUSD > 0 ? { costUSD: totalCostUSD } : {}), - byStage: [...modelByStage.entries()].map(([stage, value]) => ({ stage, ...value })).sort((left, right) => left.stage - right.stage) - }, - overruns: [], - dispatchBlocks: [] - }, - cost: await readJson("stages/00-run/cost-profile.json"), - model: await readJson("stages/00-run/model-calls-summary.json"), - tool: await readJson("stages/00-run/tool-calls-summary.json"), - run: await readJson("run.json"), - telemetry: await readJson("telemetry.json") - }; - run.info.reviewRunId = telemetry.recorder.runId; - run.info.startedAt = runSummary.startedAt; - run.info.finishedAt = runSummary.finishedAt; - refreshRunEvidenceScores(run); -} - -function attachInvocation( - runs: EvalCaseRunInput[], - invocationId: string, - status: EvalInvocationManifest["status"] = "complete", - recordedRuns = runs.length -): EvalCaseRunInput[] { - const manifest: EvalInvocationManifest = { - schemaVersion: 1, - invocationId, - suiteDir: "/suite", - status, - startedAt: "2026-07-24T00:00:00.000Z", - ...(status === "complete" ? { completedAt: "2026-07-24T00:02:00.000Z" } : {}), - cases: runs.map((run, caseIndex) => ({ - caseIndex, - caseName: run.info.caseName, - caseHash: run.info.caseHash, - caseFile: run.info.caseFile ?? `${run.info.caseName}.yml` - })), - runs: runs.slice(0, recordedRuns).map((run, caseIndex) => ({ - caseIndex, - caseName: run.info.caseName, - caseHash: run.info.caseHash, - runNumber: run.runNumber, - logsRoot: "/logs", - runPath: String(run.runNumber) - })) - }; - for (const [caseIndex, run] of runs.entries()) { - run.runDir = `/logs/${run.runNumber}`; - run.info.invocation = { id: invocationId, caseIndex, manifest: `invocations/${invocationId}.json` }; - run.invocationManifest = manifest; - } - return runs; -} - -function abcCohort( - repeat: number, - mutate?: (arm: "A" | "B" | "C", index: number) => Partial[2]>, - family = "dilution", - runOffset = 0 -): CohortSelection { - const arms = (["A", "B", "C"] as const).map((arm, armIndex) => evalRun( - family, - arm, - runOffset + armIndex + 1, - Array.from({ length: repeat }, (_, index) => execution(arm, index + 1, mutate?.(arm, index) ?? {})) - )); - return { id: `1-${arms.length}`, runs: arms }; -} - -describe("packet packing eval analysis", () => { - it("selects only the explicit latest complete cohort", () => { - const old = attachInvocation(abcCohort(1).runs, "old-invocation"); - const latest = attachInvocation(abcCohort(1).runs.map((run) => ({ ...run, runNumber: run.runNumber + 3, info: { ...run.info, runNumber: run.runNumber + 3 } })), "latest-invocation"); - const selected = selectExplicitCohort([...old, ...latest], "latest"); - expect(selected.id).toBe("latest-invocation"); - expect(selected.runs.map((run) => run.runNumber)).toEqual([4, 5, 6]); - expect(() => selectExplicitCohort([...old, ...attachInvocation(abcCohort(1).runs, "same-ending-run")], "3")).toThrow(/multiple persisted invocations/u); - }); - - it("ignores unselected interrupted history and accepts an exact invocation UUID", () => { - const interrupted = attachInvocation(abcCohort(1).runs, "historical-interrupted", "running", 2).slice(0, 2); - const invocationId = "550e8400-e29b-41d4-a716-446655440000"; - const current = attachInvocation(abcCohort(1, undefined, "dilution", 3).runs, invocationId); - expect(selectExplicitCohort([...interrupted, ...current], "latest").id).toBe(invocationId); - expect(selectExplicitCohort([...interrupted, ...current], invocationId).runs.map((run) => run.runNumber)).toEqual([4, 5, 6]); - }); - - it("preserves manifest caseIndex order across roots with duplicate run numbers", () => { - const invocationId = "multi-root-order"; - const runs = attachInvocation([ - evalRun("root-a-one", "A", 1, [execution("A", 1)]), - evalRun("root-a-two", "A", 2, [execution("A", 1)]), - evalRun("root-b-one", "A", 1, [execution("A", 1)]), - evalRun("root-b-two", "A", 2, [execution("A", 1)]) - ], invocationId); - const locations = [ - { logsRoot: "/rootA", runPath: "1", runNumber: 1 }, - { logsRoot: "/rootA", runPath: "2", runNumber: 2 }, - { logsRoot: "/rootB", runPath: "1", runNumber: 1 }, - { logsRoot: "/rootB", runPath: "2", runNumber: 2 } - ]; - const manifest = runs[0]!.invocationManifest!; - for (const [caseIndex, run] of runs.entries()) { - const location = locations[caseIndex]!; - Object.assign(manifest.runs[caseIndex]!, location); - run.runNumber = location.runNumber; - run.info.runNumber = location.runNumber; - run.runDir = `${location.logsRoot}/${location.runPath}`; - } - const selected = selectExplicitCohort([runs[3]!, runs[1]!, runs[2]!, runs[0]!], invocationId); - expect(selected.runs.map((run) => run.runDir)).toEqual(["/rootA/1", "/rootA/2", "/rootB/1", "/rootB/2"]); - expect(selected.runs.map((run) => run.info.invocation!.caseIndex)).toEqual([0, 1, 2, 3]); - }); - - it("rejects an explicit mixed cohort with duplicate cases", () => { - const runs = attachInvocation([...abcCohort(1).runs, abcCohort(1).runs[0]!].map((run, index) => ({ - ...run, - runNumber: index + 1, - info: { ...run.info, runNumber: index + 1 } - })), "mixed-invocation"); - expect(() => selectExplicitCohort(runs, "1-4")).toThrow(/multiple generations/u); - }); - - it("rejects an interrupted newest invocation instead of mixing it into latest", () => { - const complete = attachInvocation(abcCohort(1).runs, "complete-invocation"); - const interrupted = attachInvocation(abcCohort(1).runs.map((run) => ({ - ...run, - runNumber: run.runNumber + 3, - info: { ...run.info, runNumber: run.runNumber + 3 } - })), "interrupted-invocation", "running", 2).slice(0, 2); - expect(() => selectExplicitCohort([...complete, ...interrupted], "latest")).toThrow(/exact ordered case set/u); - }); - - it("proves preflight treatment without selecting an arm and uses relational call cost", () => { - const report = analyzeEvalCohort(abcCohort(1), 1); - expect(report.failures).toEqual([]); - expect(report.cohort.actualCostUSD).toBe(3); - expect(report.selectedArm).toBeUndefined(); - expect(report.cases?.[0]?.arms.B.treatment).toMatchObject({ treated: 1, total: 1, valid: true }); - - const fallback = abcCohort(1); - delete fallback.runs[0]!.executions[0]!.score.metrics.costUSD; - expect(analyzeEvalCohort(fallback, 1).cohort.actualCostUSD).toBe(3); - - const failedFallback = abcCohort(1); - delete failedFallback.runs[0]!.executions[0]!.modelCalls[0]!.costUSD; - expect(failureCodes(analyzeEvalCohort(failedFallback, 1).failures)).toContain("missing_spend_data"); - }); - - it("fails closed on mismatched treatment lenses and effective configuration", () => { - const lenses = abcCohort(1); - const event = lenses.runs[1]!.executions[0]!.events[0]!; - event.data = { ...event.data, requestedLensSignature: "[]" }; - expect(failureCodes(analyzeEvalCohort(lenses, 1).failures)).toContain("target_lens_signature"); - - const config = abcCohort(1); - config.runs[1]!.info.effectiveConfig!.review.packedToolBudgetMode = "invalid" as never; - expect(() => analyzeEvalCohort(config, 1)).toThrow(/invalid effective packet-packing settings/u); - }); - - it("joins every B/C atom and hunk exactly to A and independently validates telemetry", () => { - const cohort = abcCohort(1); - const b = cohort.runs[1]!.executions[0]!; - const event = b.events[0]!; - event.data = { - ...event.data, - atomIds: ["unknown-atom", ...(event.data?.atomIds as string[]).slice(1)], - standaloneProfiles: ["investigate", "standard"], - capUsage: { ...(event.data?.capUsage as object), patchChars: 1 }, - baseToolBudget: { maxToolCalls: 99, maxInvestigationRounds: 99, maxResultChars: 99 } - }; - b.packets[0]!.dispatchRank = [3, -99]; - const codes = failureCodes(analyzeEvalCohort(cohort, 1).failures); - expect(codes).toEqual(expect.arrayContaining(["unknown_source_atom", "treatment_invariant", "invalid_dispatch_rank", "atom_bijection"])); - }); - - it("requires a bidirectional expectation join and selects B on equal B/C evidence", () => { - const tied = analyzeEvalCohort(abcCohort(10), 10); - expect(tied.failures).toEqual([]); - expect(tied.cases?.[0]?.selectedArm).toBe("B"); - - const missing = abcCohort(1); - missing.runs[1]!.executions[0]!.score.expectationResults = []; - expect(failureCodes(analyzeEvalCohort(missing, 1).failures)).toContain("expectation_join"); - - const colludingScores = abcCohort(1); - colludingScores.runs[0]!.declaredCase.should_find![0]!.id = "declared-only"; - expect(failureCodes(analyzeEvalCohort(colludingScores, 1).failures)).toEqual(expect.arrayContaining([ - "declared_expectation_join", - "expectation_join" - ])); - }); - - it("joins requested and routed lenses to the persisted Stage-5 plan and A atoms", () => { - const pruned = abcCohort(1); - for (const run of pruned.runs) { - for (const execution of run.executions) { - for (const decision of execution.plan.coverage) { - decision.lenses = ["core/code-review", "core/tests"]; - } - for (const event of execution.events.filter((entry) => entry.message === "same_file_atoms_packed")) { - event.data = { - ...event.data, - requestedLensSignature: JSON.stringify(["core/code-review", "core/tests"]) - }; - } - } - } - expect(failureCodes(analyzeEvalCohort(pruned, 1).failures)).not.toEqual(expect.arrayContaining([ - "requested_lens_join", - "routed_lens_join" - ])); - - const planned = abcCohort(1); - planned.runs[0]!.executions[0]!.plan.coverage[0]!.lenses = ["security/auth"]; - expect(failureCodes(analyzeEvalCohort(planned, 1).failures)).toContain("requested_lens_join"); - - const routed = abcCohort(1); - routed.runs[0]!.executions[0]!.packets[0]!.lenses.push("security/auth"); - expect(failureCodes(analyzeEvalCohort(routed, 1).failures)).toContain("routed_lens_join"); - - const missing = abcCohort(1); - missing.runs[0]!.executions[0]!.plan.coverage.shift(); - expect(failureCodes(analyzeEvalCohort(missing, 1).failures)).toEqual(expect.arrayContaining(["stage5_lens_join", "requested_lens_join"])); - - const empty = abcCohort(1); - empty.runs[0]!.executions[0]!.plan.coverage[0]!.lenses = []; - expect(failureCodes(analyzeEvalCohort(empty, 1).failures)).toEqual(expect.arrayContaining(["stage5_lens_join", "requested_lens_join"])); - - const whitespace = abcCohort(1); - whitespace.runs[0]!.executions[0]!.plan.coverage[0]!.lenses = [" "]; - expect(failureCodes(analyzeEvalCohort(whitespace, 1).failures)).toEqual(expect.arrayContaining(["stage5_lens_join", "requested_lens_join"])); - - const extra = abcCohort(1); - extra.runs[1]!.executions[0]!.packets[0]!.lenses.push("security/arbitrary-extra"); - expect(failureCodes(analyzeEvalCohort(extra, 1).failures)).toContain("routed_lens_join"); - }); - - it("selects one economical cohort-wide arm and fails conflicting case requirements", () => { - const economicalRuns = [ - ...abcCohort(10, (arm, index) => ({ hit: arm !== "B" || index > 0, cost: arm === "C" ? 0.5 : 1 }), "first", 0).runs, - ...abcCohort(10, (arm, index) => ({ hit: arm === "C" || index > 0, cost: arm === "C" ? 0.5 : 1 }), "second", 3).runs - ]; - for (const run of economicalRuns.filter((candidate) => candidate.info.caseName.endsWith("-c"))) { - for (const execution of run.executions) { - for (const call of execution.modelCalls) { - call.durationMs = 500; - } - } - } - const economical = analyzeEvalCohort({ id: "economic", runs: economicalRuns }, 10); - expect(economical.failures).toEqual([]); - expect(economical.selectedArm).toBe("C"); - expect(economical.cases?.every((entry) => entry.selectedArm === "C")).toBe(true); - - const conflictingRuns = [ - ...abcCohort(10, (arm, index) => ({ hit: arm === "B" ? index < 7 : true }), "requires-c", 0).runs, - ...abcCohort(10, (arm, index) => ({ hit: arm === "C" ? index < 7 : true }), "requires-b", 3).runs - ]; - expect(failureCodes(analyzeEvalCohort({ id: "conflicting", runs: conflictingRuns }, 10).failures)).toContain("conflicting_arm_selection"); - }); - - it("requires C to improve pressure or recall and retain 85% of packet-count service savings when B fails", () => { - const tied = analyzeEvalCohort(abcCohort(10), 10); - expect(tied.selectedArm).toBe("B"); - - const lowRetention = abcCohort(10, (arm, index) => ({ hit: arm !== "B" || index < 7 })); - for (const execution of lowRetention.runs[2]!.executions) { - execution.modelCalls[0]!.durationMs = 1_900; - } - const report = analyzeEvalCohort(lowRetention, 10); - expect(failureCodes(report.failures)).toEqual(expect.arrayContaining(["atom_scaled_savings_retention", "no_passing_arm"])); - }); - - it("applies savings retention cohort-wide when B is ineligible for pressure rather than recall", () => { - const cases = [ - abcCohort(10, undefined, "pressure-one", 0), - abcCohort(10, undefined, "pressure-two", 3) - ]; - for (const cohort of cases) { - const b = cohort.runs[1]!; - for (const execution of b.executions) { - execution.toolCalls.push({ - runId: "eval", - toolCallId: `rejected-${b.info.caseName}-${execution.repeat}`, - timestamp: "2026-07-24T00:00:00.000Z", - stage: 7, - initiator: "model", - packetId: execution.packets[0]!.id, - tool: "read_file", - args: { path: "app.ts" }, - backend: "text", - precision: "exact", - degraded: false, - degradationReason: "tool_call_budget_exhausted", - resultChars: 0, - durationMs: 0, - status: "rejected" - }); - } - refreshRunEvidenceScores(b); - for (const execution of cohort.runs[2]!.executions) { - execution.modelCalls[0]!.durationMs = 1_900; - } - } - const report = analyzeEvalCohort({ id: "cohort-pressure-retention", runs: cases.flatMap((cohort) => cohort.runs) }, 10); - expect(failureCodes(report.failures)).toEqual(expect.arrayContaining([ - "tool_pressure", - "atom_scaled_savings_retention", - "no_passing_arm" - ])); - }); - - it("reconstructs relational evidence and rejects deletions, altered matches, costs, and repeat aggregates", () => { - for (const mutate of [ - (run: EvalCaseRunInput) => { run.executions[0]!.verification = []; }, - (run: EvalCaseRunInput) => { run.executions[0]!.finalSelection = []; }, - (run: EvalCaseRunInput) => { run.executions[0]!.score.expectationResults[0]!.matched = [{ findingId: "invented", artifact: "final-findings" }]; }, - (run: EvalCaseRunInput) => { run.executions[0]!.score.metrics.costUSD = 99; } - ]) { - const cohort = abcCohort(1); - mutate(cohort.runs[1]!); - const codes = failureCodes(analyzeEvalCohort(cohort, 1).failures); - expect(codes.some((code) => code === "paid_evidence_relations" || code === "paid_evidence_score_reconstruction")).toBe(true); - } - - const repeated = abcCohort(10); - repeated.runs[1]!.info.repeats!.totals.costUSD += 1; - expect(failureCodes(analyzeEvalCohort(repeated, 10).failures)).toContain("paid_evidence_aggregate"); - }); - - it("rejects collusive score/summary mutations and non-canonical repeat paths", () => { - const collusive = abcCohort(1); - const run = collusive.runs[1]!; - const target = run.executions[0]!; - target.scoringArtifacts.metricsSources.costProfile = { totalCostUSD: 99 }; - target.scoringArtifacts.metricsSources.modelCallsSummary = { totalCalls: 99 }; - target.scoringArtifacts.metricsSources.toolCallsSummary = { totalCalls: 99 }; - target.scoringArtifacts.metricsSources.runJson = { durationMs: 99_000 }; - target.score.metrics.costUSD = 99; - target.score.metrics.modelCalls = 99; - target.score.metrics.toolCalls = 99; - target.score.metrics.elapsedSeconds = 99; - const costBudget = target.score.budgetResults.find((result) => result.check === "maxCostUSD"); - if (costBudget !== undefined) costBudget.actual = 99; - run.info.score = structuredClone(target.score); - expect(failureCodes(analyzeEvalCohort(collusive, 1).failures)).toEqual(expect.arrayContaining([ - "paid_evidence_score_reconstruction", - "evidence_cost_accounting" - ])); - - const paths = abcCohort(10); - paths.runs[1]!.info.repeats!.executions[0]!.runDir = "/arbitrary/repeat"; - expect(failureCodes(analyzeEvalCohort(paths, 10).failures)).toContain("paid_evidence_aggregate"); - - const reordered = abcCohort(10); - [reordered.runs[1]!.executions[0], reordered.runs[1]!.executions[1]] = - [reordered.runs[1]!.executions[1]!, reordered.runs[1]!.executions[0]!]; - expect(failureCodes(analyzeEvalCohort(reordered, 10).failures)).toContain("paid_evidence_repeat_order"); - }); - - it("reconciles every accepted producer summary field to raw calls, events, and artifacts", async () => { - const base = abcCohort(1); - await attachProducerSummaryEvidence(base.runs[0]!); - expect(failureCodes(analyzeEvalCohort(base, 1).failures)).not.toContain("paid_summary_reconciliation"); - - const reorderedAttention = structuredClone(base); - reorderedAttention.runs[0]!.executions[0]!.summaryArtifacts!.attention = [ - ...(reorderedAttention.runs[0]!.executions[0]!.summaryArtifacts!.attention as unknown[]) - ].reverse(); - expect(failureCodes(analyzeEvalCohort(reorderedAttention, 1).failures)).not.toContain("paid_summary_reconciliation"); - - const paths = [ - ["cost", "costBreakdown", "total", "costUSD"], - ["run", "totals", "costBreakdown", "total", "costUSD"], - ["run", "totals", "stage7SchemaRepair", "actualRepairCalls"], - ["telemetry", "schemaRepair", "stage7", "actualRepairCalls"], - ["run", "totals", "logOverflow", "droppedDebugInfo"], - ["telemetry", "logs", "bufferedOverflow", "droppedWarnError"], - ...["filesChanged", "hunks", "packets", "packetReviews", "candidates", "verified", "finalFindings", "postedComments"] - .map((field) => ["run", "totals", field]), - ["telemetry", "workers", "started"], - ["telemetry", "packets", "generated"], - ["telemetry", "lenses", "selected"], - ["telemetry", "coverage", "hunks", "reviewed"], - ["telemetry", "candidates", "generated"], - ["telemetry", "verdicts", "accept"], - ["telemetry", "dedup", "clusters"], - ["telemetry", "finalSelection", "published"], - ["telemetry", "posting", "attempted"] - ]; - for (const fieldPath of paths) { - const cohort = structuredClone(base); - const summary = cohort.runs[0]!.executions[0]!.summaryArtifacts as Record; - let owner = summary; - for (const key of fieldPath.slice(0, -1)) { - owner = owner[key] as Record; - } - const key = fieldPath.at(-1)!; - owner[key] = Number(owner[key]) + 1; - expect( - failureCodes(analyzeEvalCohort(cohort, 1).failures), - fieldPath.join(".") - ).toContain("paid_summary_reconciliation"); - } - }); - - it("reconstructs note-bearing repeats and rejects corruption in every persisted score section", () => { - const notes = abcCohort(10, (arm, index) => ({ hit: arm !== "B" || index > 0 })); - notes.runs[1]!.executions[0]!.scoringArtifacts.humanAttentionNotes = [{ - question: "Does the boundary bug affect app.ts?", - files: ["app.ts"], - reasons: ["boundary behavior"] - }]; - refreshRunEvidenceScores(notes.runs[1]!); - expect(notes.runs[1]!.executions[0]!.score.expectationResults[0]!.loss?.surfacedAsNote).toBe(true); - expect(failureCodes(analyzeEvalCohort(notes, 10).failures)).not.toEqual(expect.arrayContaining([ - "paid_evidence_score_reconstruction", - "paid_evidence_aggregate" - ])); - - const corruptions: Array<(score: EvalScore) => void> = [ - (persisted) => { persisted.nearViolations.push({ expectationId: "bug", findingId: "invented", artifact: "final-findings" }); }, - (persisted) => { persisted.budgetResults.push({ check: "maxFindings", status: "pass", limit: 99, actual: 1, direction: "maximum" }); }, - (persisted) => { persisted.metrics.reasoningTokens = 999; }, - (persisted) => { persisted.expectationResults[0]!.note = "corrupted aggregate note"; } - ]; - for (const corrupt of corruptions) { - const cohort = abcCohort(1); - corrupt(cohort.runs[1]!.executions[0]!.score); - expect(failureCodes(analyzeEvalCohort(cohort, 1).failures)).toContain("paid_evidence_score_reconstruction"); - } - - const aggregate = abcCohort(10); - aggregate.runs[1]!.info.score.metrics.reasoningTokens = 123; - expect(failureCodes(analyzeEvalCohort(aggregate, 10).failures)).toContain("paid_evidence_aggregate"); - }); - - it("validates every final-selection section and verifier-to-final lineage", () => { - const mutations: Array<(execution: EvalExecutionInput) => void> = [ - (target) => { target.finalSelectionArtifact.records = []; }, - (target) => { target.finalSelectionArtifact.groups = []; }, - (target) => { target.finalSelectionArtifact.publicationAnchors[0]!.fingerprint = "invented"; }, - (target) => { target.finalSelectionArtifact.confidenceSelections.push({ - findingId: "finding", confidence: "high", representativeConfidence: "medium", reason: "representative" - }); }, - (target) => { - const record = target.verification[0]!; - if ("verdict" in record) { - record.verdict.finalFinding = { ...structuredClone(target.candidateFindings[0]!), failureMode: "unrelated revised failure" }; - } - }, - (target) => { - const record = target.verification[0]!; - if ("verdict" in record) { - record.verdict.revisedAnchor = { path: "app.ts", hunkId: "h2", line: 1, side: "RIGHT" }; - } - }, - (target) => { - const record = target.verification[0]!; - record.gate = "gate_anchor_stripped"; - record.gateDecision = "suppressed"; - record.gateReason = "invalid_anchor; fabricated mixed state"; - }, - (target) => { - target.verification[0]!.gateFacts = { ...target.verification[0]!.gateFacts, changedLine: false }; - } - ]; - for (const [index, mutate] of mutations.entries()) { - const cohort = abcCohort(1); - mutate(cohort.runs[1]!.executions[0]!); - expect(failureCodes(analyzeEvalCohort(cohort, 1).failures), `mutation ${index}`).toContain("paid_evidence_relations"); - } - }); - - it("accepts Stage 9 producers only for relational uncertainty promotions", () => { - const promoted = abcCohort(1); - const promotedRun = promoted.runs[1]!; - const promotedExecution = promotedRun.executions[0]!; - const promotedCandidate = promotedExecution.candidateFindings[0]!; - const provenance = { - source: "uncertainty_promotion" as const, - sourceKind: "uncertainty" as const, - sourcePacketId: promotedCandidate.producedBy.packetId, - question: "Does this changed boundary remain safe?", - files: [promotedCandidate.path], - symbols: [], - reason: "The packet review left the boundary unresolved." - }; - promotedCandidate.producedBy.stage = 9; - promotedCandidate.provenance = provenance; - promotedExecution.verification[0]!.candidateProvenance = structuredClone(provenance); - refreshRunEvidenceScores(promotedRun); - expect(failureCodes(analyzeEvalCohort(promoted, 1).failures)).not.toContain("paid_evidence_relations"); - - const arbitraryStage9 = abcCohort(1); - const arbitraryRun = arbitraryStage9.runs[1]!; - arbitraryRun.executions[0]!.candidateFindings[0]!.producedBy.stage = 9; - refreshRunEvidenceScores(arbitraryRun); - expect(failureCodes(analyzeEvalCohort(arbitraryStage9, 1).failures)).toContain("paid_evidence_relations"); - }); - - it("replays verifier gates from raw candidates instead of trusting collusive persisted decisions", () => { - const scheduledInvalid = abcCohort(1); - const scheduledRun = scheduledInvalid.runs[1]!; - scheduledRun.executions[0]!.candidateFindings[0]!.evidence.changedCode = ""; - refreshRunEvidenceScores(scheduledRun); - expect(failureCodes(analyzeEvalCohort(scheduledInvalid, 1).failures)).toContain("paid_evidence_relations"); - - const suppressedValid = abcCohort(1); - const suppressedRun = suppressedValid.runs[1]!; - const suppressedExecution = suppressedRun.executions[0]!; - suppressedExecution.verification = [{ - candidateId: "finding", - gate: "suppressed", - gateDecision: "suppressed", - gateReason: "missing_evidence" - }]; - suppressedExecution.finalSelection = []; - suppressedExecution.finalFindings = []; - refreshRunEvidenceScores(suppressedRun); - expect(failureCodes(analyzeEvalCohort(suppressedValid, 1).failures)).toContain("paid_evidence_relations"); - }); - - it("accepts production anchor backfill and withholds representative-only anchors from inline publication", () => { - for (const invalidAnchor of [false, true]) { - const cohort = abcCohort(1); - const run = cohort.runs[1]!; - const execution = run.executions[0]!; - const candidate = execution.candidateFindings[0]!; - candidate.changedLine = false; - candidate.modelAnchorSubmitted = invalidAnchor; - if (invalidAnchor) { - candidate.anchor = { path: "app.ts", line: 999, side: "RIGHT", hunkId: "h1" }; - candidate.anchorSource = "model"; - execution.verification[0]!.gate = "gate_anchor_stripped"; - } else { - delete candidate.anchor; - delete candidate.anchorSource; - } - refreshRunEvidenceScores(run); - expect(execution.finalFindings[0]!.publication).toBe("summary-only"); - expect(execution.finalFindings[0]!.anchor).toBeUndefined(); - expect(execution.finalSelection[0]!.reason).toBe("unanchorable"); - expect(failureCodes(analyzeEvalCohort(cohort, 1).failures)).not.toContain("paid_evidence_relations"); - } - }); - - it("supports production verify:false duplicate records without persisted cluster metadata", () => { - const cohort = abcCohort(1); - const run = cohort.runs[1]!; - run.info.effectiveConfig!.review.verify = false; - run.declaredCase.review!.verify = false; - run.info.caseSnapshot.review!.verify = false; - const execution = run.executions[0]!; - const duplicate = { ...structuredClone(execution.candidateFindings[0]!), id: "finding-2" }; - execution.candidateFindings.push(duplicate); - const primaryRecord = execution.verification[0]!; - if (!("verdict" in primaryRecord)) throw new Error("expected verifier record"); - primaryRecord.verdict = { - candidateId: "finding", - verdict: "keep", - reason: "verification disabled by config", - requiredEvidencePresent: true, - falsePositiveRisk: "low" - }; - execution.verification.push({ - ...structuredClone(primaryRecord), - candidateId: duplicate.id, - verdict: { ...structuredClone(primaryRecord.verdict), candidateId: duplicate.id } - }); - execution.finalFindings[0]!.mergedCandidateIds = ["finding", duplicate.id]; - execution.finalSelection.push({ findingId: duplicate.id, decision: "merged", reason: "composer-merged" }); - refreshRunEvidenceScores(run); - expect(execution.verification.every((record) => record.duplicateOf === undefined && record.clusterId === undefined)).toBe(true); - expect(failureCodes(analyzeEvalCohort(cohort, 1).failures)).not.toContain("paid_evidence_relations"); - }); - - it("independently applies composer thresholds, caps, and deterministic ordering", () => { - const thresholdCases: Array<(run: EvalCaseRunInput) => void> = [ - (run) => { run.info.effectiveConfig!.review.minSeverity = "high"; }, - (run) => { run.info.effectiveConfig!.review.minConfidence = "high"; }, - (run) => { run.info.effectiveConfig!.review.minInlineConfidence = "high"; } - ]; - for (const mutatePolicy of thresholdCases) { - const cohort = abcCohort(1); - mutatePolicy(cohort.runs[1]!); - expect(failureCodes(analyzeEvalCohort(cohort, 1).failures)).toContain("paid_evidence_relations"); - } - - const multi = abcCohort(1); - const multiRun = multi.runs[1]!; - const execution = multiRun.executions[0]!; - const packetWithH2 = execution.packets.find((target) => target.hunks.some((hunk) => hunk.hunkId === "h2"))!; - const second = candidateFinding("finding-2", packetWithH2.id, "h2", 101); - second.title = "Independent second boundary defect"; - const evidence = findingEvidence(second); - execution.candidateFindings.push(...evidence.candidateFindings); - execution.verification.push(...evidence.verification); - execution.finalSelection.push(...evidence.finalSelection); - execution.finalFindings.push(...evidence.finalFindings); - refreshRunEvidenceScores(multiRun); - expect(failureCodes(analyzeEvalCohort(multi, 1).failures)).not.toContain("paid_evidence_relations"); - - const softCap = structuredClone(multi); - softCap.runs[1]!.info.effectiveConfig!.review.softCommentCap = 1; - expect(failureCodes(analyzeEvalCohort(softCap, 1).failures)).toContain("paid_evidence_relations"); - - const reportCap = structuredClone(multi); - reportCap.runs[1]!.info.effectiveConfig!.review.maxFindings = 1; - expect(failureCodes(analyzeEvalCohort(reportCap, 1).failures)).toContain("paid_evidence_relations"); - - const reordered = structuredClone(multi); - reordered.runs[1]!.executions[0]!.finalFindings.reverse(); - expect(failureCodes(analyzeEvalCohort(reordered, 1).failures)).toContain("paid_evidence_relations"); - }); - - it("enforces aggregate recall and atom-count loss attribution instead of allowing one loss per expectation", () => { - const cohort = abcCohort(10); - for (const run of cohort.runs) { - run.declaredCase.should_find![0]!.titlePattern = "^Boundary bug$"; - run.declaredCase.should_find!.push({ id: "bug-2", path: "app.ts", titlePattern: "^Secondary boundary bug$" }); - for (const execution of run.executions) { - const packetWithH2 = execution.packets.find((target) => target.hunks.some((hunk) => hunk.hunkId === "h2"))!; - const candidate = candidateFinding("finding-2", packetWithH2.id, "h2", 1); - candidate.title = "Secondary boundary bug"; - const evidence = findingEvidence(candidate); - execution.candidateFindings.push(...evidence.candidateFindings); - execution.verification.push(...evidence.verification); - execution.finalSelection.push(...evidence.finalSelection); - execution.finalFindings.push(...evidence.finalFindings); - } - refreshRunEvidenceScores(run); - } - for (const [repeatIndex, findingId] of [[0, "finding"], [1, "finding-2"]] as const) { - const execution = cohort.runs[1]!.executions[repeatIndex]!; - execution.candidateFindings = execution.candidateFindings.filter((finding) => finding.id !== findingId); - execution.verification = execution.verification.filter((record) => record.candidateId !== findingId); - execution.finalSelection = execution.finalSelection.filter((record) => record.findingId !== findingId); - execution.finalFindings = execution.finalFindings.filter((finding) => finding.id !== findingId); - } - refreshRunEvidenceScores(cohort.runs[1]!); - const codes = failureCodes(analyzeEvalCohort(cohort, 10).failures); - expect(codes).toContain("recall_non_inferiority"); - - const bucketed = abcCohort(10, (arm, index) => ({ hit: arm !== "B" || index > 1 })); - expect(failureCodes(analyzeEvalCohort(bucketed, 10).failures)).toContain("candidate_generation_loss_by_atom_count"); - }); - - it("spends the single recall and atom-loss allowance across the entire multi-case cohort", () => { - const runs = [ - ...abcCohort(10, (arm, index) => ({ hit: arm !== "B" || index > 0 }), "cohort-loss-one", 0).runs, - ...abcCohort(10, (arm, index) => ({ hit: arm !== "B" || index > 0 }), "cohort-loss-two", 3).runs - ]; - const codes = failureCodes(analyzeEvalCohort({ id: "cohort-loss", runs }, 10).failures); - expect(codes).toEqual(expect.arrayContaining([ - "recall_non_inferiority", - "candidate_generation_loss", - "candidate_generation_loss_by_atom_count" - ])); - }); - - it("keeps all executions in intent-to-treat and excludes untreated runs from both treated-only numerator and denominator", () => { - const report = analyzeEvalCohort(abcCohort(10, (arm, index) => ({ - treated: arm === "B" ? index !== 0 : arm !== "A", - hit: arm === "B" ? index !== 1 : true - })), 10); - expect(report.failures).toEqual([]); - const b = report.cases?.[0]?.arms.B; - expect(b?.intentToTreat[0]).toMatchObject({ denominator: 10, candidateHits: 9, finalHits: 9 }); - expect(b?.treatedOnly.rates[0]).toMatchObject({ denominator: 9, candidateHits: 8, finalHits: 8 }); - expect(b?.treatedOnly.denominatorByExpectation["should_find:bug"]).toBe(9); - }); - - it("fails closed below 8/10 treatment and on missing telemetry", () => { - const underTreated = analyzeEvalCohort(abcCohort(10, (arm, index) => ({ - treated: arm === "B" ? index < 7 : arm !== "A" - })), 10); - expect(failureCodes(underTreated.failures)).toContain("insufficient_treatment"); - - const missing = analyzeEvalCohort(abcCohort(1, (arm) => ({ missingTelemetry: arm === "C" })), 1); - expect(failureCodes(missing.failures)).toEqual(expect.arrayContaining(["missing_treatment_telemetry", "insufficient_treatment"])); - }); - - it("fails closed on recall regression and missing spend data", () => { - const recall = analyzeEvalCohort(abcCohort(10, (arm, index) => ({ - hit: arm === "B" ? index < 7 : true - })), 10); - expect(failureCodes(recall.failures)).toEqual(expect.arrayContaining(["recall_non_inferiority"])); - - const noSpend = abcCohort(1, (arm) => arm === "C" ? { cost: undefined } : {}); - delete noSpend.runs[2]!.executions[0]!.score.metrics.costUSD; - noSpend.runs[2]!.executions[0]!.score.budgetResults = []; - const spendReport = analyzeEvalCohort(noSpend, 1); - expect(failureCodes(spendReport.failures)).toContain("missing_spend_data"); - }); - - it("writes a failure report when eval artifacts cannot be loaded", async () => { - const root = await mkdtemp(path.join(tmpdir(), "packet-report-missing-eval-")); - tempDirs.push(root); - const output = path.join(root, "report.json"); - const exitCode = await runPacketPackingReportCli([ - "eval", - "--logs", - path.join(root, "missing"), - "--expected-repeats", - "1", - "--output", - output - ]); - expect(exitCode).toBe(1); - const report = JSON.parse(await readFile(output, "utf8")) as { failures: ReportFailure[] }; - expect(failureCodes(report.failures)).toContain("missing_logs"); - }); - - it("projects replay, eval, and regression failures through one prose-safe output boundary", async () => { - const root = await mkdtemp(path.join(tmpdir(), "packet-report-safe-output-")); - tempDirs.push(root); - const secret = "PROPRIETARY_FOCUS_AND_CREDENTIAL_NEVER_EMIT"; - const replayRun = path.join(root, secret); - await mkdir(path.join(replayRun, "stages", "01-input"), { recursive: true }); - await writeFile(path.join(replayRun, "stages", "01-input", "resolved-input.json"), "{bad-json\n"); - const invocations: Array<{ argv: string[]; output: string }> = [ - { - argv: ["replay", "--repo", process.cwd(), "--run", replayRun, "--output", path.join(root, "replay.json")], - output: path.join(root, "replay.json") - }, - { - argv: ["eval", "--logs", path.join(root, secret, "eval"), "--expected-repeats", "1", "--output", path.join(root, "eval.json")], - output: path.join(root, "eval.json") - }, - { - argv: [ - "regression", - "--baseline-logs", path.join(root, secret, "baseline"), - "--selected-logs", path.join(root, secret, "selected"), - "--expected-repeats", "1", - "--output", path.join(root, "regression.json") - ], - output: path.join(root, "regression.json") - } - ]; - for (const invocation of invocations) { - expect(await runPacketPackingReportCli(invocation.argv)).toBe(1); - const serialized = await readFile(invocation.output, "utf8"); - expect(serialized).not.toContain(secret); - expect(serialized).toContain("sha256:"); - expect((JSON.parse(serialized) as { failures: ReportFailure[] }).failures.length).toBeGreaterThan(0); - } - }); - - it("fingerprints argument, top-level, and report-write failures without raw stderr prose", async () => { - const root = await mkdtemp(path.join(tmpdir(), "packet-report-safe-top-level-")); - tempDirs.push(root); - const secret = "PROPRIETARY_TOP_LEVEL_MESSAGE_NEVER_EMIT"; - const parseOutput = path.join(root, "parse.json"); - expect(await runPacketPackingReportCli([ - "eval", "--logs", path.join(root, secret), "--unsupported", secret, - "--output", parseOutput - ])).toBe(1); - const parseReport = await readFile(parseOutput, "utf8"); - expect(parseReport).not.toContain(secret); - expect(parseReport).toContain("sha256:"); - - const stderrChunks: string[] = []; - const stderr = vi.spyOn(process.stderr, "write").mockImplementation(((chunk: string | Uint8Array) => { - stderrChunks.push(String(chunk)); - return true; - }) as typeof process.stderr.write); - try { - expect(await runPacketPackingReportCli(["unknown-mode", secret])).toBe(1); - const unwritableOutput = path.join(root, secret); - await mkdir(unwritableOutput); - expect(await runPacketPackingReportCli([ - "eval", "--logs", path.join(root, "missing"), "--expected-repeats", "1", "--output", unwritableOutput - ])).toBe(1); - } finally { - stderr.mockRestore(); - } - const stderrOutput = stderrChunks.join(""); - expect(stderrOutput).not.toContain(secret); - expect(stderrOutput).toContain("sha256:"); - }); -}); - -describe("production economics", () => { - function productionExecution(arm: "A" | "B", wallTimeSeconds: number, cost: number): EvalExecutionInput { - const hunkIds = Array.from({ length: 142 }, (_, index) => `production-hunk-${index + 1}`); - const atoms = hunkIds.map((hunkId, index) => packet(`production-atom-${index + 1}`, [hunkId])); - const packets = arm === "A" - ? atoms - : Array.from({ length: Math.ceil(atoms.length / 5) }, (_, index) => { - const members = atoms.slice(index * 5, index * 5 + 5); - return packet(`production-packed-${index + 1}`, members.flatMap((member) => member.hunks.map((hunk) => hunk.hunkId))); - }); - const events = arm === "A" ? [] : packets.map((target, index) => packingEvent(target, atoms.slice(index * 5, index * 5 + 5))); - const executionScore = score(true, cost); - const callCost = cost / (packets.length + 3); - const packetCalls = packets.map((target) => modelCall(target.id, 1, callCost)); - const planner = { ...modelCall(packets[0]!.id, 1, callCost), callId: `production-planner-${arm}`, stage: 5 as const, role: "planner" as const }; - delete (planner as Partial).packetId; - const verifier = { - ...modelCall(packets[0]!.id, 1, callCost), - callId: `production-verifier-${arm}`, - stage: 9 as const, - role: "verifier" as const, - kind: "repair" as const, - candidateId: "finding" - }; - delete (verifier as Partial).packetId; - const composer = { ...modelCall(packets[0]!.id, 1, callCost), callId: `production-composer-${arm}`, stage: 10 as const, role: "composer" as const }; - delete (composer as Partial).packetId; - const allCalls = [...packetCalls, planner, verifier, composer]; - executionScore.metrics.modelCalls = allCalls.length; - executionScore.metrics.verificationCalls = 1; - executionScore.metrics.elapsedSeconds = wallTimeSeconds; - const candidate = candidateFinding("finding", packets[0]!.id, hunkIds[0]!); - const result = { - repeat: 1, - score: executionScore, - telemetryDir: `/tmp/production/${arm}`, - packets, - events, - modelCalls: allCalls, - toolCalls: [], - fileFacts: [{ - path: "app.ts", - language: "typescript", - processingMode: "per-hunk", - testStatus: "source", - isGenerated: false, - isVendored: false, - isLockfile: false, - isBinary: false, - changedLines: 142, - hunkCount: 142, - labels: [], - reviewPriority: "normal", - reasons: [], - provenance: [] - }], - diff: diffForPackets(atoms), - plan: planForHunks(hunkIds), - ...findingEvidence(candidate), - reviewedHunkIds: hunkIds, - wallTimeSeconds - } as unknown as EvalExecutionInput; - refreshExecutionEvidence(result); - return result; - } - - function productionCohort(): CohortSelection { - const cohort = { - id: "1-2", - runs: [ - evalRun("production", "A", 1, [productionExecution("A", 200, 20)]), - evalRun("production", "B", 2, [productionExecution("B", 100, 10)]) - ] - }; - for (const run of cohort.runs) { - run.declaredCase.repo = { external: "/home/peter/Dev/0xsequence/trails-api" }; - run.declaredCase.command = { base: "d1c49bdf6a8002ec2ec27faac94a932d736532b2", head: "fbb5f8761c2c296e115af17e919a7c35d9de8373" }; - run.declaredCase.review = { - ...run.declaredCase.review, - cache: false, - concurrency: 6, - maxTimeMinutes: 60 - }; - run.info.repo = { - root: "/home/peter/Dev/0xsequence/trails-api", - baseSha: "d1c49bdf6a8002ec2ec27faac94a932d736532b2", - mergeBase: "d1c49bdf6a8002ec2ec27faac94a932d736532b2", - headSha: "fbb5f8761c2c296e115af17e919a7c35d9de8373" - }; - run.info.effectiveConfig!.review.concurrency = 6; - run.info.effectiveConfig!.review.timeoutMs = 3_600_000; - run.info.effectiveConfig!.llm.maxConcurrentCalls = 6; - run.info.codegenieRuntime = { packageVersion: "0.5.0", commit: "a".repeat(40), dirty: false, source: "git" }; - } - return cohort; - } - - it("reports raw and normalized 142-hunk economics with extrapolation and break-even", () => { - const failures: ReportFailure[] = []; - const economics = computeProductionEconomics( - { actualCostUSD: 20, reviewedHunks: 100 }, - { actualCostUSD: 20, reviewedHunks: 142 }, - 80, - "explicit_cumulative_validation_cost", - failures - ); - expect(failures).toEqual([]); - expect(economics).toMatchObject({ - equivalentTargetHunks: 142, - equivalentReviewSavingsUSD: 8.4, - breakEvenReviewCount: 10, - baseline: { equivalentCostExtrapolated: true }, - selected: { equivalentCostExtrapolated: false } - }); - }); - - it("fails closed on missing reviewed work and a non-positive payback denominator", () => { - const failures: ReportFailure[] = []; - computeProductionEconomics( - { actualCostUSD: 10, reviewedHunks: 0 }, - { actualCostUSD: 20, reviewedHunks: 100 }, - 50, - "explicit_cumulative_validation_cost", - failures - ); - expect(failureCodes(failures)).toEqual(expect.arrayContaining(["missing_reviewed_hunks", "non_positive_payback_denominator"])); - - const noSavings: ReportFailure[] = []; - computeProductionEconomics( - { actualCostUSD: 10, reviewedHunks: 100 }, - { actualCostUSD: 20, reviewedHunks: 100 }, - 50, - "explicit_cumulative_validation_cost", - noSavings - ); - expect(failureCodes(noSavings)).toContain("non_positive_payback_denominator"); - }); - - it("gates production on the exact 142-hunk set and measured wall/model/token capacity", () => { - const report = analyzeEvalCohort(productionCohort(), 1, { actualValidationCostUSD: 30 }); - expect(report.failures).toEqual([]); - const cohort = productionCohort(); - const baselineCalls = cohort.runs[0]!.executions[0]!.modelCalls; - const allStageReport = analyzeEvalCohort(cohort, 1, { actualValidationCostUSD: 30 }); - expect(report.productionThroughput).toMatchObject({ - baseline: { reviewedHunks: 142, wallTimeSeconds: 200 }, - selected: { reviewedHunks: 142, wallTimeSeconds: 100 } - }); - expect(allStageReport.productionThroughput?.baseline).toMatchObject({ - modelServiceSeconds: baselineCalls.reduce((total, call) => total + call.durationMs, 0) / 1000, - totalTokens: baselineCalls.reduce((total, call) => total + call.totalTokens!, 0), - reasoningTokens: 0, - reasoningTokensPerReviewedHunk: 0 - }); - - const truncated = productionCohort(); - truncated.runs[1]!.executions[0]!.reviewedHunkIds.pop(); - const codes = failureCodes(analyzeEvalCohort(truncated, 1).failures); - expect(codes).toEqual(expect.arrayContaining(["production_hunk_loss", "production_incomplete"])); - }); - - it("gates reasoning tokens per hunk and continuation savings using all-stage service time", () => { - const reasoning = productionCohort(); - reasoning.runs[1]!.executions[0]!.modelCalls[0]!.reasoningTokens = 100; - refreshRunEvidenceScores(reasoning.runs[1]!); - expect(failureCodes(analyzeEvalCohort(reasoning, 1).failures)).toContain("production_reasoning_tokens"); - - const continuation = productionCohort(); - const selectedExecution = continuation.runs[1]!.executions[0]!; - selectedExecution.modelCalls.push({ - ...modelCall(selectedExecution.packets[0]!.id, 1, 0), - callId: "production-added-continuation", - kind: "tool-continuation", - attempt: 2, - durationMs: 100_000 - }); - refreshRunEvidenceScores(continuation.runs[1]!); - expect(failureCodes(analyzeEvalCohort(continuation, 1).failures)).toContain("production_continuation_savings_retention"); - }); - - it("fails closed when production model telemetry is coverage-only or does not strictly improve capacity", () => { - const coverageOnly = productionCohort(); - delete coverageOnly.runs[1]!.executions[0]!.modelCalls[0]!.totalTokens; - expect(failureCodes(analyzeEvalCohort(coverageOnly, 1).failures)).toContain("production_model_accounting"); - - const equalCapacity = productionCohort(); - const selectedCalls = equalCapacity.runs[1]!.executions[0]!.modelCalls; - const baselineCalls = equalCapacity.runs[0]!.executions[0]!.modelCalls; - let remainingTokens = baselineCalls.reduce((total, call) => total + call.totalTokens!, 0); - const baselineServiceMs = baselineCalls.reduce((total, call) => total + call.durationMs, 0); - for (const [index, call] of selectedCalls.entries()) { - const totalTokens = index === selectedCalls.length - 1 ? remainingTokens : Math.floor(remainingTokens / (selectedCalls.length - index)); - remainingTokens -= totalTokens; - call.durationMs = baselineServiceMs / selectedCalls.length; - call.inputTokens = totalTokens - 10; - call.outputTokens = 10; - call.totalTokens = totalTokens; - } - expect(failureCodes(analyzeEvalCohort(equalCapacity, 1).failures)).toEqual(expect.arrayContaining([ - "production_model_service", - "production_tokens" - ])); - }); - - it("requires pinned clean production provenance and exact finding-artifact preservation", () => { - const cohort = productionCohort(); - cohort.runs[1]!.info.repo!.headSha = "c".repeat(40); - cohort.runs[1]!.info.codegenieRuntime!.dirty = true; - cohort.runs[1]!.executions[0]!.finalFindings = []; - cohort.runs[1]!.executions[0]!.score.expectationResults[0]!.status = "fail"; - const codes = failureCodes(analyzeEvalCohort(cohort, 1).failures); - expect(codes).toEqual(expect.arrayContaining([ - "production_refs", - "production_runtime_provenance", - "production_finding_artifact_join", - "production_finding_preservation" - ])); - }); - - it("requires the exact production repo, cache, concurrency, and 60-minute declared/effective shape", () => { - const cohort = productionCohort(); - cohort.runs[1]!.declaredCase.repo!.external = "/tmp/wrong-repo"; - cohort.runs[1]!.declaredCase.review!.cache = true; - cohort.runs[1]!.declaredCase.review!.concurrency = 1; - cohort.runs[1]!.declaredCase.review!.maxTimeMinutes = 1; - cohort.runs[1]!.info.effectiveConfig!.review.timeoutMs = 60_000; - const codes = failureCodes(analyzeEvalCohort(cohort, 1).failures); - expect(codes).toEqual(expect.arrayContaining(["production_refs", "production_run_shape"])); - }); - - it("requires nonempty required matches and relationally complete production finding artifacts", () => { - const emptyMatch = productionCohort(); - emptyMatch.runs[1]!.executions[0]!.score.expectationResults[0]!.matched = []; - expect(failureCodes(analyzeEvalCohort(emptyMatch, 1).failures)).toContain("production_finding_preservation"); - - for (const mutate of [ - (execution: EvalExecutionInput) => { execution.candidateFindings[0]!.producedBy.packetId = "unknown-packet"; }, - (execution: EvalExecutionInput) => { execution.finalFindings[0]!.mergedCandidateIds = ["unknown-candidate"]; }, - (execution: EvalExecutionInput) => { execution.finalFindings[0]!.anchor = { hunkId: "unknown-hunk", path: "app.ts", line: 1, side: "RIGHT" }; }, - (execution: EvalExecutionInput) => { execution.score.metrics.candidateFindings = 99; } - ]) { - const cohort = productionCohort(); - mutate(cohort.runs[1]!.executions[0]!); - expect(failureCodes(analyzeEvalCohort(cohort, 1).failures)).toContain("production_finding_relations"); - } - for (const mutate of [ - (execution: EvalExecutionInput) => { execution.verification = []; }, - (execution: EvalExecutionInput) => { execution.finalSelection = []; } - ]) { - const cohort = productionCohort(); - mutate(cohort.runs[1]!.executions[0]!); - expect(failureCodes(analyzeEvalCohort(cohort, 1).failures)).toContain("paid_evidence_relations"); - } - }); -}); - -describe("collateral regression analysis", () => { - function regressionPair(selectedHit = true, selectedError = false) { - const baseline = evalRun("fixture", "A", 1, [execution("A", 1)]); - const selected = evalRun("fixture", "B", 1, [execution("B", 1, { hit: selectedHit })]); - baseline.info.caseSnapshot.name = "fixture-baseline"; - baseline.declaredCase.name = "fixture-baseline"; - baseline.info.caseName = "fixture"; - baseline.info.caseFile = "fixture.yml"; - selected.info.caseSnapshot.name = "fixture-selected"; - selected.declaredCase.name = "fixture-selected"; - selected.info.caseName = "fixture"; - selected.info.caseFile = "fixture.yml"; - selected.info.score = selected.executions[0]!.score; - if (selectedError) { - selected.info.score = { ...selected.info.score, status: "error", error: { code: "llm_call_failed", message: "boom" } }; - } - return { - baseline: { id: "1-1", runs: [baseline] }, - selected: { id: "1-1", runs: [selected] } - }; - } - - it("compares explicit roots as one-repeat collateral evidence", () => { - const pair = regressionPair(); - const report = analyzeRegressionCohorts(pair.baseline, pair.selected, 1); - expect(report.failures).toEqual([]); - expect(report).toMatchObject({ - evidence: "one-repeat-collateral-only", - baselineCohort: { actualCostUSD: 1 }, - selectedCohort: { actualCostUSD: 1 } - }); - }); - - it("accepts a valid collateral case where packing is enabled but no atom is treated", () => { - const pair = regressionPair(); - pair.selected.runs[0]!.executions[0]!.packets = structuredClone(pair.baseline.runs[0]!.executions[0]!.packets); - pair.selected.runs[0]!.executions[0]!.events = []; - const selectedExecution = pair.selected.runs[0]!.executions[0]!; - selectedExecution.candidateFindings[0]!.producedBy.packetId = selectedExecution.packets[0]!.id; - selectedExecution.finalFindings[0]!.producedBy.packetId = selectedExecution.packets[0]!.id; - refreshExecutionEvidence(selectedExecution); - const report = analyzeRegressionCohorts(pair.baseline, pair.selected, 1); - expect(report.failures).toEqual([]); - expect(report.cases[0]?.selectedTreatmentExecutions).toBe(0); - }); - - it("fails closed on collateral expectation transitions and eval errors", () => { - const regressed = regressionPair(false); - const report = analyzeRegressionCohorts(regressed.baseline, regressed.selected, 1); - expect(failureCodes(report.failures)).toContain("collateral_expectation_regression"); - - const errored = regressionPair(true, true); - expect(failureCodes(analyzeRegressionCohorts(errored.baseline, errored.selected, 1).failures)).toContain("eval_error"); - - const mutuallyAlteredScore = regressionPair(); - mutuallyAlteredScore.selected.runs[0]!.info.score.expectationResults[0]!.expectationId = "score-only-expectation"; - expect(failureCodes(analyzeRegressionCohorts(mutuallyAlteredScore.baseline, mutuallyAlteredScore.selected, 1).failures)).toContain("expectation_join"); - }); - - it("fails when both collateral cohorts identically miss a requirement and when relational evidence is deleted", () => { - const bothMiss = regressionPair(false); - const baselineExecution = bothMiss.baseline.runs[0]!.executions[0]!; - baselineExecution.candidateFindings = []; - baselineExecution.verification = []; - baselineExecution.finalSelection = []; - baselineExecution.finalFindings = []; - refreshRunEvidenceScores(bothMiss.baseline.runs[0]!); - const bothMissReport = analyzeRegressionCohorts(bothMiss.baseline, bothMiss.selected, 1); - expect(bothMissReport.cases[0]!.expectationTransitions).toEqual([]); - expect(failureCodes(bothMissReport.failures)).toContain("collateral_cohort_expectation_failure"); - - const deleted = regressionPair(); - deleted.selected.runs[0]!.executions[0]!.verification = []; - expect(failureCodes(analyzeRegressionCohorts(deleted.baseline, deleted.selected, 1).failures)).toContain("paid_evidence_relations"); - }); - - it("fails closed when baseline and selected YAML differ beyond allowed experiment fields", () => { - const pair = regressionPair(); - pair.selected.runs[0]!.declaredCase.review!.depth = "deep"; - const report = analyzeRegressionCohorts(pair.baseline, pair.selected, 1); - expect(failureCodes(report.failures)).toContain("regression_yaml_delta"); - }); -}); diff --git a/tests/pipeline-phase5.test.ts b/tests/pipeline-phase5.test.ts index df06f39..51e457e 100644 --- a/tests/pipeline-phase5.test.ts +++ b/tests/pipeline-phase5.test.ts @@ -3980,617 +3980,6 @@ describe("phase 5 pipeline regressions", () => { }); }); - it("keeps packet atom flag-off parity and bypasses whole-file groups", async () => { - const file = fakeMultiHunkFile([ - { id: "h1", newStart: 1, content: "export const one = 1;" }, - { id: "h2", newStart: 100, content: "export const two = 2;" } - ]); - const facts = { ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }; - const baselineArtifacts = new Map(); - const baseline = await buildReviewPackets( - fakePlanForHunks(["h1", "h2"]), - [file], - [facts], - fakeRepositoryIndex(), - { - ...nullTelemetry(), - writeArtifact: async (name, data) => { - baselineArtifacts.set(name, data); - } - }, - { config: config(), enabledLenses: ["core/code-review"] } - ); - const darkArtifacts = new Map(); - const dark = await buildReviewPackets( - fakePlanForHunks(["h1", "h2"]), - [file], - [facts], - fakeRepositoryIndex(), - { - ...nullTelemetry(), - writeArtifact: async (name, data) => { - darkArtifacts.set(name, data); - } - }, - { - config: { - ...config(), - review: { ...config().review, packSameFileHunks: false, packedToolBudgetMode: "atom-scaled" } - }, - enabledLenses: ["core/code-review"] - } - ); - - expect(dark).toEqual(baseline); - expect(darkArtifacts).toEqual(baselineArtifacts); - - const events: Array> = []; - const wholeFilePackets = await buildReviewPackets( - fakePlan(), - [fakeDiffFile("app.ts")], - [fakeFacts("app.ts", "whole-file")], - fakeRepositoryIndex(fakeTools("export const value = 1;\n")), - { ...nullTelemetry(), event: (event) => events.push(event) }, - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review"] - } - ); - expect(wholeFilePackets[0]?.kind).toBe("whole-file"); - expect(events.some((event) => event.message === "same_file_atoms_packed")).toBe(false); - - const probedFileDiff = await buildReviewPackets( - fakePlan(), - [fakeDiffFile("app.ts")], - [fakeFacts("app.ts", "whole-file")], - fakeRepositoryIndex(fakeTools("x".repeat(9_000))), - { ...nullTelemetry(), event: (event) => events.push(event) }, - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review"] - } - ); - expect(probedFileDiff[0]?.kind).toBe("file-diff"); - expect(events.some((event) => event.message === "same_file_atoms_packed")).toBe(false); - }); - - it("applies stable same-file packing with source-order, cap, telemetry, and dispatch-rank invariants", async () => { - const file = fakeMultiHunkFile(Array.from({ length: 6 }, (_, index) => ({ - id: `h${String(index + 1)}`, - newStart: index * 100 + 1, - content: `export const value${String(index + 1)} = ${String(index + 1)};` - }))); - const facts = { ...fakeFacts("app.ts", "per-hunk"), hunkCount: 6, changedLines: 6 }; - const events: Array> = []; - const packets = await buildReviewPackets( - fakePlanForHunks(["h1", "h2", "h3", "h4", "h5", "h6"]), - [file], - [facts], - fakeRepositoryIndex(), - { ...nullTelemetry(), event: (event) => events.push(event) }, - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review"] - } - ); - - expect(packets.map((packet) => packet.hunks.map((hunk) => hunk.hunkId))).toEqual([ - ["h1", "h2", "h3", "h4", "h5"], - ["h6"] - ]); - expect(packets.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId))).toEqual(["h1", "h2", "h3", "h4", "h5", "h6"]); - expect(packets[0]?.dispatchRank).toEqual(packetDispatchRank("app.ts", facts, 5)); - const packedEvents = events.filter((event) => event.message === "same_file_atoms_packed"); - expect(packedEvents).toHaveLength(2); - expect(packedEvents.map((event) => event.data)).toEqual([ - expect.objectContaining({ - atomIds: ["h1", "h2", "h3", "h4", "h5"].map((hunkId) => sha256Hex(`hunk-first\n${hunkId}`)), - sourceAtomCount: 5, - hunkCount: 5, - effectiveCoverage: "normal", - requestedLensSignature: '["core/code-review"]', - standaloneProfiles: ["standard", "standard", "standard", "standard", "standard"], - plannerLensesPreserved: true, - capUsage: expect.objectContaining({ hunks: 5, maxHunks: 5, maxPatchChars: 12_000 }) - }), - expect.objectContaining({ - atomIds: [sha256Hex("hunk-first\nh6")], - sourceAtomCount: 1, - hunkCount: 1 - }) - ]); - - const nearbyEvents: Array> = []; - await buildReviewPackets( - fakePlanForHunks(["h1", "h2"]), - [fakeMultiHunkFile([ - { id: "h1", newStart: 1, content: "export const one = 1;" }, - { id: "h2", newStart: 10, content: "export const two = 2;" } - ])], - [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }], - fakeRepositoryIndex(), - { ...nullTelemetry(), event: (event) => nearbyEvents.push(event) }, - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review"] - } - ); - const nearbyAtomIds = nearbyEvents.find((event) => event.message === "same_file_atoms_packed")?.data?.atomIds; - expect(nearbyAtomIds).toEqual([sha256Hex("hunk-first\nh1\nh2")]); - expect(nearbyAtomIds).not.toEqual([sha256Hex("hunk-first\nh1"), sha256Hex("hunk-first\nh2")]); - expect(nearbyAtomIds).not.toEqual([sha256Hex("hunk-first\nh2\nh1")]); - }); - - it("partitions same-file packing by coverage and lens signature before restoring packet order", async () => { - const ids = ["h1", "h2", "h3", "h4", "h5", "h6"]; - const file = fakeMultiHunkFile(ids.map((id, index) => ({ - id, - newStart: index * 100 + 1, - content: `export function value${String(index)}() { return ${String(index)}; }` - }))); - const plan = fakePlanForHunks(ids); - plan.coverage = plan.coverage.map((decision, index) => ({ - ...decision, - coverage: index === 1 || index === 3 ? "light" : "normal", - lenses: index >= 4 ? ["core/security"] : ["core/code-review"] - })); - const packets = await buildReviewPackets( - plan, - [file], - [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 6, changedLines: 6 }], - fakeRepositoryIndex(), - nullTelemetry(), - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review", "core/security"] - } - ); - - expect(packets.map((packet) => ({ - coverage: packet.coverage, - lenses: packet.lenses, - hunks: packet.hunks.map((hunk) => hunk.hunkId) - }))).toEqual([ - { coverage: "normal", lenses: ["core/code-review"], hunks: ["h1", "h3"] }, - { coverage: "light", lenses: ["core/code-review"], hunks: ["h2", "h4"] }, - { coverage: "normal", lenses: ["core/security"], hunks: ["h5", "h6"] } - ]); - }); - - it("leaves same-file atoms separate when packing would omit a standalone routed lens", async () => { - const ids = ["h1", "h2", "h3", "h4"]; - const file = fakeMultiHunkFile(ids.map((id, index) => ({ - id, - newStart: index * 100 + 1, - content: `export function value${String(index)}() { return ${String(index)}; }` - }))); - const plan = fakePlanForHunks(ids); - plan.coverage = plan.coverage.map((decision, index) => ({ - ...decision, - lenses: ["core/code-review", "core/tests"], - reason: "default_coverage", - focusNotes: [index === 3 ? "Check regression tests for this branch." : `Inspect behavior ${String(index + 1)}.`] - })); - const events: Array> = []; - const packets = await buildReviewPackets( - plan, - [file], - [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 4, changedLines: 4 }], - fakeRepositoryIndex(), - { ...nullTelemetry(), event: (event) => events.push(event) }, - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review", "core/tests"] - } - ); - - expect(packets.map((packet) => packet.hunks.map((hunk) => hunk.hunkId))).toEqual([ - ["h1"], - ["h2"], - ["h3"], - ["h4"] - ]); - expect(packets.map((packet) => packet.lenses)).toEqual([ - ["core/code-review"], - ["core/code-review"], - ["core/code-review"], - ["core/code-review", "core/tests"] - ]); - expect(events).toContainEqual(expect.objectContaining({ - stage: 6, - message: "same_file_atom_pack_rejected", - data: expect.objectContaining({ reason: "standalone routed lenses omitted: core/tests" }) - })); - }); - - it("preserves mixed-coverage baseline atoms and scales by newly combined atom count", async () => { - const file = fakeMultiHunkFile([ - { id: "h1", newStart: 1, content: "export function one() { return 1; }" }, - { id: "h2", newStart: 10, content: "export function two() { return 2; }" }, - { id: "h3", newStart: 100, content: "export function three() { return 3; }" } - ]); - const plan = fakePlanForHunks(["h1", "h2", "h3"]); - plan.coverage[1] = { ...plan.coverage[1]!, coverage: "deep" }; - plan.coverage[2] = { ...plan.coverage[2]!, coverage: "deep" }; - const facts = { ...fakeFacts("app.ts", "per-hunk"), hunkCount: 3, changedLines: 3 }; - const off = await buildReviewPackets( - plan, - [file], - [facts], - fakeRepositoryIndex(), - nullTelemetry(), - { config: config(), enabledLenses: ["core/code-review"] } - ); - expect(off.map((packet) => packet.hunks.map((hunk) => hunk.hunkId))).toEqual([["h1", "h2"], ["h3"]]); - expect(off.map((packet) => packet.coverage)).toEqual(["deep", "deep"]); - - const events: Array> = []; - const packed = await buildReviewPackets( - plan, - [file], - [facts], - fakeRepositoryIndex(), - { ...nullTelemetry(), event: (event) => events.push(event) }, - { - config: { - ...config(), - review: { ...config().review, packSameFileHunks: true, packedToolBudgetMode: "atom-scaled" } - }, - enabledLenses: ["core/code-review"] - } - ); - expect(packed).toHaveLength(1); - expect(packed[0]?.hunks.map((hunk) => hunk.hunkId)).toEqual(["h1", "h2", "h3"]); - expect(packed[0]?.coverage).toBe("deep"); - expect(packed[0]?.toolBudget.maxToolCalls).toBe(16); - expect(events).toContainEqual(expect.objectContaining({ - message: "same_file_atoms_packed", - data: expect.objectContaining({ sourceAtomCount: 2, hunkCount: 3, effectiveCoverage: "deep" }) - })); - }); - - it("leaves same-file atoms separate when packing would omit high-priority planner attention", async () => { - const ids = ["h1", "h2", "h3", "h4"]; - const file = fakeMultiHunkFile(ids.map((id, index) => ({ - id, - newStart: index * 100 + 1, - content: `export function value${String(index)}() { return ${String(index)}; }` - }))); - const plan = fakePlanForHunks(ids); - plan.coverage = plan.coverage.map((decision, index) => ({ - ...decision, - focusNotes: [`Protect focus ${String(index + 1)}.`] - })); - const events: Array> = []; - const packets = await buildReviewPackets( - plan, - [file], - [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 4, changedLines: 4, reviewPriority: "high" }], - fakeRepositoryIndex(), - { ...nullTelemetry(), event: (event) => events.push(event) }, - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review"] - } - ); - - expect(packets).toHaveLength(4); - expect(packets.map((packet) => packet.attentionNotes)).toEqual([ - expect.arrayContaining(["Protect focus 1."]), - expect.arrayContaining(["Protect focus 2."]), - expect.arrayContaining(["Protect focus 3."]), - expect.arrayContaining(["Protect focus 4."]) - ]); - expect(events).toContainEqual(expect.objectContaining({ - stage: 6, - message: "same_file_atom_pack_rejected", - data: expect.objectContaining({ reason: expect.stringContaining("high-priority planner focus omitted") }) - })); - }); - - it("merges degradation reasons deterministically while respecting the real patch cap", async () => { - const file = fakeMultiHunkFile([ - { id: "h1", newStart: 1, content: "a".repeat(5_000) }, - { id: "h2", newStart: 100, content: "b".repeat(5_000) }, - { id: "h3", newStart: 200, content: "c".repeat(5_000) } - ]); - const packets = await buildReviewPackets( - fakePlanForHunks(["h1", "h2", "h3"]), - [file], - [{ ...fakeFacts("app.ts", "whole-file"), hunkCount: 3, changedLines: 3 }], - fakeRepositoryIndex(), - nullTelemetry(), - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review"] - } - ); - - expect(packets.map((packet) => packet.hunks.length)).toEqual([2, 1]); - expect(packets[0]?.degraded?.reason.match(/whole-file downgraded/gu)).toHaveLength(1); - expect(packets[0]?.hunks.reduce((sum, hunk) => sum + hunk.contentWithLineNumbers.length, 0)).toBeLessThanOrEqual(12_000); - - const oversizedAtom = await buildReviewPackets( - fakePlan(), - [fakeDiffFile("app.ts", "x".repeat(20_000))], - [fakeFacts("app.ts", "per-hunk")], - fakeRepositoryIndex(), - nullTelemetry(), - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review"] - } - ); - expect(oversizedAtom).toHaveLength(1); - expect(oversizedAtom[0]?.hunks[0]).toMatchObject({ hunkId: "h1", truncated: true }); - }); - - it("applies a planner-hint profile floor when same-file packing absorbs strong related context", async () => { - const file = fakeMultiHunkFile([ - { id: "h1", newStart: 1, content: "export function alpha() { return beta(); }" }, - { id: "h2", newStart: 100, content: "export function beta() { return 2; }" } - ]); - const symbolFacts: HunkSymbolFacts[] = [ - packetPackingSymbolFact("h1", "alpha", [1, 3]), - packetPackingSymbolFact("h2", "beta", [100, 102]) - ]; - const plan = fakePlanForHunks(["h1", "h2"]); - plan.coverage[0] = { ...plan.coverage[0]!, reason: "default_coverage", relatedSymbols: ["beta"] }; - const off = await buildReviewPackets( - plan, - [file], - [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }], - { ...fakeRepositoryIndex(), symbolFacts }, - nullTelemetry(), - { config: config(), enabledLenses: ["core/code-review"] } - ); - expect(off.map((packet) => packet.reviewProfile)).toEqual(["investigate", "standard"]); - - const events: Array> = []; - const packed = await buildReviewPackets( - plan, - [file], - [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }], - { ...fakeRepositoryIndex(), symbolFacts }, - { ...nullTelemetry(), event: (event) => events.push(event) }, - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review"] - } - ); - expect(packed).toHaveLength(1); - expect(packed[0]).toMatchObject({ - reviewProfile: "investigate", - relatedChangedContext: [], - toolBudget: { maxToolCalls: 6, maxInvestigationRounds: 2, maxResultChars: 12_000 } - }); - expect(events).toContainEqual(expect.objectContaining({ - message: "same_file_atoms_packed", - data: expect.objectContaining({ - standaloneProfiles: ["investigate", "standard"], - derivedPackedProfile: "standard", - profileFloor: "investigate", - effectiveProfile: "investigate", - profileFloorApplied: true - }) - })); - }); - - it("applies an ordinary-context profile floor instead of demoting a packed light packet to simple", async () => { - const file = fakeMultiHunkFile([ - { id: "h1", newStart: 1, content: "import { one } from './one';" }, - { id: "h2", newStart: 100, content: "import { two } from './two';" } - ]); - const plan = fakePlanForHunks(["h1", "h2"]); - plan.coverage = plan.coverage.map((decision) => ({ ...decision, coverage: "light" })); - plan.coverage[0] = { ...plan.coverage[0]!, reason: "default_coverage", relatedFiles: ["app.ts"] }; - const events: Array> = []; - const packets = await buildReviewPackets( - plan, - [file], - [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }], - fakeRepositoryIndex(), - { ...nullTelemetry(), event: (event) => events.push(event) }, - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review"] - } - ); - - expect(packets).toHaveLength(1); - expect(packets[0]).toMatchObject({ reviewProfile: "standard", toolBudget: { maxToolCalls: 1 } }); - expect(events).toContainEqual(expect.objectContaining({ - message: "same_file_atoms_packed", - data: expect.objectContaining({ - standaloneProfiles: ["standard", "simple"], - derivedPackedProfile: "simple", - profileFloor: "standard", - effectiveProfile: "standard", - profileFloorApplied: true - }) - })); - }); - - it("applies a symbol-mention profile floor and does not invent one for same-symbol atoms", async () => { - const file = fakeMultiHunkFile([ - { id: "h-helper", newStart: 1, content: "export function scaleAmount(value) { return value / 10; }" }, - { id: "h-caller", newStart: 100, content: "export function quote(value) { return scaleAmount(value); }" } - ]); - const symbolFacts: HunkSymbolFacts[] = [ - packetPackingSymbolFact("h-helper", "scaleAmount", [1, 3]), - packetPackingSymbolFact("h-caller", "quote", [100, 102]) - ]; - const meta = { backend: "tree-sitter" as const, precision: "syntactic" as const, degraded: false }; - const tools = { - ...fakeTools(), - findSymbolMentions: async (symbolName: string, options: SymbolMentionOptions = {}) => ({ - results: symbolName === "scaleAmount" && options.contextMode === "symbols" - ? [{ - path: "app.ts", - line: 101, - matchText: "return scaleAmount(value);", - enclosingSymbol: { path: "app.ts", name: "quote", kind: "function" as const, lineRange: [100, 102] as [number, number] } - }] - : [], - meta - }) - }; - const events: Array> = []; - const packets = await buildReviewPackets( - fakePlanForHunks(["h-helper", "h-caller"]), - [file], - [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }], - { ...fakeRepositoryIndex(tools), symbolFacts }, - { ...nullTelemetry(), event: (event) => events.push(event) }, - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review"] - } - ); - expect(packets).toHaveLength(1); - expect(packets[0]?.reviewProfile).toBe("investigate"); - expect(events).toContainEqual(expect.objectContaining({ - message: "same_file_atoms_packed", - data: expect.objectContaining({ derivedPackedProfile: "standard", profileFloor: "investigate", profileFloorApplied: true }) - })); - - const sameSymbolEvents: Array> = []; - const sameSymbolFacts = [ - packetPackingSymbolFact("h-helper", "shared", [1, 200]), - packetPackingSymbolFact("h-caller", "shared", [1, 200]) - ]; - const sameSymbolPackets = await buildReviewPackets( - fakePlanForHunks(["h-helper", "h-caller"]), - [file], - [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 2, changedLines: 2 }], - { ...fakeRepositoryIndex(), symbolFacts: sameSymbolFacts }, - { ...nullTelemetry(), event: (event) => sameSymbolEvents.push(event) }, - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review"] - } - ); - expect(sameSymbolPackets).toHaveLength(1); - expect(sameSymbolPackets[0]?.reviewProfile).toBe("standard"); - expect(sameSymbolEvents).toContainEqual(expect.objectContaining({ - message: "same_file_atoms_packed", - data: expect.objectContaining({ sourceAtomCount: 1, standaloneProfiles: ["standard"], profileFloorApplied: false }) - })); - }); - - it("preserves the standalone profile in the same-name primarySymbols packing edge case", async () => { - const file = fakeMultiHunkFile([ - { id: "h-x", newStart: 1, content: "export function source() { return shared(); }" }, - { id: "h-y", newStart: 100, content: "export function shared() { return 1; }" }, - { id: "h-z", newStart: 200, content: "export function shared() { return 2; }" } - ]); - const symbolFacts = [ - packetPackingSymbolFact("h-x", "source", [1, 3]), - packetPackingSymbolFact("h-y", "shared", [100, 102]), - packetPackingSymbolFact("h-z", "shared", [200, 202]) - ]; - const plan = fakePlanForHunks(["h-x", "h-y", "h-z"]); - plan.coverage[0] = { ...plan.coverage[0]!, reason: "default_coverage", relatedSymbols: ["shared"] }; - plan.coverage[2] = { ...plan.coverage[2]!, coverage: "deep" }; - const events: Array> = []; - const packets = await buildReviewPackets( - plan, - [file], - [{ ...fakeFacts("app.ts", "per-hunk"), hunkCount: 3, changedLines: 3 }], - { ...fakeRepositoryIndex(), symbolFacts }, - { ...nullTelemetry(), event: (event) => events.push(event) }, - { - config: { ...config(), review: { ...config().review, packSameFileHunks: true } }, - enabledLenses: ["core/code-review"] - } - ); - const packed = packets.find((packet) => packet.hunks.some((hunk) => hunk.hunkId === "h-x")); - expect(packed?.hunks.map((hunk) => hunk.hunkId)).toEqual(["h-x", "h-y"]); - expect(packed?.reviewProfile).toBe("investigate"); - expect(events).toContainEqual(expect.objectContaining({ - message: "same_file_atoms_packed", - data: expect.objectContaining({ - packetId: packed?.id, - standaloneProfiles: ["investigate", "standard"], - derivedPackedProfile: "standard", - profileFloorApplied: true - }) - })); - }); - - it("scales packed tool budgets by additional atoms while preserving base, simple, rounds, and boost order", async () => { - const ids = ["h1", "h2", "h3", "h4", "h5"]; - const distantFile = fakeMultiHunkFile(ids.map((id, index) => ({ - id, - newStart: index * 100 + 1, - content: `export function value${String(index)}() { return ${String(index)}; }` - }))); - const facts = { ...fakeFacts("app.ts", "per-hunk"), hunkCount: 5, changedLines: 5 }; - const build = (review: Partial, plan = fakePlanForHunks(ids), file = distantFile, fileFacts = facts) => - buildReviewPackets( - plan, - [file], - [fileFacts], - fakeRepositoryIndex(), - nullTelemetry(), - { - config: { - ...config(), - review: { ...config().review, packSameFileHunks: true, packedToolBudgetMode: "atom-scaled", ...review } - }, - enabledLenses: ["core/code-review"] - } - ); - - const base = await build({ packedToolBudgetMode: "base" }); - expect(base[0]?.toolBudget).toEqual({ maxToolCalls: 4, maxInvestigationRounds: 2, maxResultChars: 10_000 }); - - const scaled = await build({}); - expect(scaled[0]?.toolBudget).toEqual({ maxToolCalls: 7, maxInvestigationRounds: 2, maxResultChars: 17_500 }); - - const investigate = await build({}, fakePlanForHunks(ids), distantFile, { ...facts, reviewPriority: "high" }); - expect(investigate[0]?.toolBudget).toEqual({ - maxToolCalls: 10, - maxInvestigationRounds: 2, - maxResultChars: 20_000, - sourceExtension: { maxToolCalls: 1, maxResultChars: 4_000 } - }); - - const boosted = await build({ budgetBoost: 1.5 }, fakePlanForHunks(ids), distantFile, { ...facts, reviewPriority: "high" }); - expect(boosted[0]?.toolBudget).toEqual({ - maxToolCalls: 15, - maxInvestigationRounds: 3, - maxResultChars: 30_000, - sourceExtension: { maxToolCalls: 2, maxResultChars: 6_000 } - }); - - const deepPlan = fakePlanForHunks(ids); - deepPlan.coverage = deepPlan.coverage.map((decision) => ({ ...decision, coverage: "deep" })); - const deep = await build({}, deepPlan); - expect(deep[0]?.toolBudget).toEqual({ - maxToolCalls: 19, - maxInvestigationRounds: 5, - maxResultChars: 40_000, - sourceExtension: { maxToolCalls: 1, maxResultChars: 4_000 } - }); - - const nearbyFile = fakeMultiHunkFile(ids.map((id, index) => ({ - id, - newStart: index * 10 + 1, - content: `export function nearby${String(index)}() { return ${String(index)}; }` - }))); - const oneAtom = await build({}, fakePlanForHunks(ids), nearbyFile); - expect(oneAtom[0]?.toolBudget).toEqual({ maxToolCalls: 4, maxInvestigationRounds: 2, maxResultChars: 10_000 }); - - const simpleFile = fakeMultiHunkFile([ - { id: "h1", newStart: 1, content: "import { one } from './one';" }, - { id: "h2", newStart: 100, content: "import { two } from './two';" } - ]); - const simple = await build({}, fakePlanForHunks(["h1", "h2"]), simpleFile, { ...facts, hunkCount: 2, changedLines: 2 }); - expect(simple[0]?.reviewProfile).toBe("simple"); - expect(simple[0]?.toolBudget).toEqual({ maxToolCalls: 0, maxInvestigationRounds: 0, maxResultChars: 0 }); - }); - it("does not coalesce add and deletion-only hunks by comparing new and old coordinates", async () => { const file: DiffFile = { path: "app.ts", @@ -13992,24 +13381,6 @@ function fakeMultiHunkFile(hunks: Array<{ id: string; newStart: number; content: }; } -function packetPackingSymbolFact( - hunkId: string, - enclosingSymbol: string, - symbolRange: [number, number] -): HunkSymbolFacts { - return { - path: "app.ts", - hunkId, - enclosingSymbol, - symbolKind: "function", - symbolRange, - changedLines: [symbolRange[0]], - changedLinesSide: "new", - source: "tree-sitter", - confidence: "syntactic" - }; -} - function fakeFacts(path: string, processingMode: FileFacts["processingMode"]): FileFacts { return { path, From 7b61b34cd20f193e50303a2aaba913f115931be0 Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 11:26:00 -0400 Subject: [PATCH 06/24] docs(plans): add Plan 103 relationship-driven packet packing Plan 102 failed its treatment gate because it partitioned packing candidates by the planner's per-hunk coverage grade. That predicate is self-defeating for recall validation: a hunk interesting enough to carry an expectation is graded differently from its safe siblings, so the target could never pack. Both paid cohorts died on treatment validity, never on recall, while the free deterministic replay passed every invariant across four production runs. Plan 103 changes the predicate. Atoms combine when the existing hunk relationship graph relates them - same_symbol, strong symbol_mention, uniquely-resolved planner_hint, plus a new tree-sitter same_container source keyed on structural owner identity. Packets grow by connectivity-preserving greedy expansion, so they stay connected without requiring source contiguity. Coverage promotion is permitted and recorded per member rather than banned, leaving Plan 92's packet-level escalation field untouched. The container signal lives in a separate packing-affinity view so flag-off context, notes, profiles, and the relationship-graph artifact stay byte-identical. Symbol context follows membership: multi-symbol packets read every member's primary symbol under a shared budget instead of silently keeping the top-ranked one. Each paid pair consumes one hash-pinned Stage-5 plan, so packing is the only within-pair difference and cross-arm reconciliation becomes exact. A pre-registered decision table with explicit Void conditions replaces the inherited non-inferiority bound, which could have passed on two zeroes. Free replay gates are numeric and counted on distinct diffs. Single flag, no budget-mode arm, teardown to one path under either outcome. Co-Authored-By: Claude Opus 5 (1M context) --- ...-103-relationship-driven-packet-packing.md | 435 ++++++++++++++++++ specs/plans/README.md | 1 + 2 files changed, 436 insertions(+) create mode 100644 specs/plans/103-issue-103-relationship-driven-packet-packing.md diff --git a/specs/plans/103-issue-103-relationship-driven-packet-packing.md b/specs/plans/103-issue-103-relationship-driven-packet-packing.md new file mode 100644 index 0000000..b29ac9a --- /dev/null +++ b/specs/plans/103-issue-103-relationship-driven-packet-packing.md @@ -0,0 +1,435 @@ +# Issue 103: Relationship-Driven Same-File Packet Packing + +Status: PENDING +Related: Plan 102 (`102-issue-102-same-file-packet-packing.md`) — COMPLETE as a failed treatment gate; supplies the atom implementation, the deterministic replay harness, and the negative result this plan is built from. Plan 67 (hunk relationship context) built the graph this plan spends differently. Plan 92 (coverage calibration and escalation) supplies the coverage-promotion precedent and the noisy-grade evidence. Plan 100 (dispatch rank), Plan 79 (repeat/recall harness), Plan 32 (adaptive Stage-6 symbol context). +Planned from: Plan 102's preserved evidence — `plan102-packet-shape.json`, `plan102-eval-preflight-invalid-ace65769.json`, and `plan102-eval-preflight-retry-invalid-5bd80f2c.json` under `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/reports/` (manifest verified) — plus the same four retained runs, 2026-07-25 +Production replay refs: base/merge-base `d1c49bdf6a8002ec2ec27faac94a932d736532b2`; head `fbb5f8761c2c296e115af17e919a7c35d9de8373` +Planned at: commit `f372f73` (branch `next`) +Recommended priority: after Plan 101's paid A/B settles. Plan 102's deterministic half already passed; its paid half never asked the question. This plan changes the packing predicate so the question becomes askable, and isolates Stage 6 so the answer is attributable. + +> Executor instructions: pack by the relationship graph that already exists, not by the planner's coverage grade. Preserve today's semantic grouper output as indivisible **atoms** exactly as Plan 102 did. Grow packets by connectivity-preserving greedy expansion — never chunk a component by source order, and never require contiguity. Keep the container-affinity signal out of the shared prompt-context graph entirely. Allow coverage promotion only through a packing-eligible edge, and record it per member, not on Plan 92's packet-level field. Extend symbol context to every member atom before packing multi-symbol atoms. Prove the treatment surface deterministically and for free before spending a model call, and pin one Stage-5 plan across both arms of every paid pair so packing is the only difference. Ship exactly one temporary flag; there is no budget-mode arm. +> +> **Commit references are rebase-unstable.** Branch `next` was rebased between Plan 102's teardown and this plan. The Plan 102 series is currently `5551547` (implementation), `8fceba9` (report tool), `7ebfd2f` (report fixes), `87a5a10` (retry docs), `f372f73` (teardown). If those hashes are unreachable, locate them by commit subject rather than trusting the hash. +> +> Drift check: `git diff --stat f372f73..HEAD -- src/pipeline/packet-builder.ts src/pipeline/review-runner.ts src/repo/symbol-extraction.ts src/repo/tree-sitter/ src/config/schema.ts src/config/config-loader.ts src/types.ts src/evals/eval-runner.ts scripts/packet-packing-report.ts tests/pipeline-phase5.test.ts tests/config-loader.test.ts tests/evals.test.ts tests/packet-packing-report.test.ts` +> Working-tree check: `git status --short -- src/pipeline/ src/repo/ src/config/ src/types.ts src/evals/` +> If the drift check reports committed changes, reconcile the current-state claims below. If the working-tree check reports changes owned by another task, stop and wait for that work to land or move this plan to an isolated worktree. + +## Decision + +Pack same-file atoms that the existing hunk relationship graph already relates, and stop using the planner's per-hunk coverage grade as a partition key. + +- Run the existing symbol/proximity grouper unchanged. Its output groups remain indivisible **atoms**, exactly as in Plan 102. +- Build a **separate packing-affinity view** over the atoms of one file. It is derived from the relationship graph but never written back into it: the prompt-context graph, its per-hunk edge cap, its excerpts, and its attention notes stay byte-identical with the flag off. +- An atom pair is adjacent when any hunk of one has a **packing-eligible** edge to any hunk of the other: `same_symbol`, strong `symbol_mention`, uniquely-resolved strong `planner_hint`, or the new `same_container`. +- Grow packets by **connectivity-preserving greedy expansion**: start from the source-earliest unassigned atom, then repeatedly admit the source-earliest atom that has an eligible edge into the current packet set and fits the caps. Never chunk a connected component by source order; never require source contiguity. +- **Coverage promotion is permitted and recorded per member**, not banned and not folded into Plan 92's packet-level escalation field. +- **Symbol context follows membership.** Packing multiple primary symbols into one packet extends symbol-source reading to every member atom under a shared budget, rather than silently keeping only the top-ranked one. +- Lens intent, focus notes, and per-member context quality are protected by **transactional candidate rejection**, not by partitioning. +- Plan 102's monotonic profile floor is retained unchanged. +- One temporary flag, `review.packRelatedHunks`. One budget policy — today's. No `atom-scaled` arm. +- Every paid pair consumes **one pinned Stage-5 plan**, so the arms differ only by packing. + +This is a different bet from Plan 102. That plan optimized packet count and treated relatedness as a nice-to-have; this plan optimizes cross-hunk reasoning and treats packet reduction as a secondary, reported effect that may well be negative. + +## Evidence + +### Plan 102's deterministic mechanics worked; its predicate did not + +The four-run replay passed every invariant with zero model calls: `740d73f2` 109→85, `fe1548ae` 109→85, `81f806a6` 93→68, `dca8d870` 96→75, with zero coverage promotions, zero effective profile or budget downgrades, zero cap violations, zero lens drops, and zero invalid dispatch ranks. Atom preservation, hunk bijection, source ordering, profile floors, and dispatch-rank recomputation are proven machinery. This plan reuses them and changes which atoms are eligible to combine. + +Note that `740d73f2` and `fe1548ae` are the same commit and PR — identical 109→85 rows. There are two distinct diffs among the four retained runs, and every threshold in this plan counts distinct diffs. + +### The coverage/lens predicate was self-defeating + +Both paid cohorts failed on treatment validity, never on recall. In the repaired retry (`5bd80f2c`), consistency B and C were each treated `0/1`: the planner graded the target `HandleWire` hunk `deep` and its three safe siblings `light`, so the compatibility partition correctly refused to pack them. The first cohort failed the same way. + +This is structural. A recall fixture needs a detectably risky hunk among safe siblings; the planner grades the risky one differently. Under a coverage-equality predicate, **any hunk interesting enough to carry a recall expectation is by construction unpackable**. Freezing the planner output does not fix that; only changing the predicate does. + +### The codebase already contradicts the predicate + +`applyCoverageEscalations()` exists because of Plan 92, and its own comment states the case: + +> the planner's deep/normal assignment is a one-draw LLM judgment with measured run-to-run variance (the erc20 packet: normal in runs 46-50, deep in 51); these rules floor structurally suspicious packets to deep so attention amplification (budgets, ensemble passes) stops depending on the planner's mood + +Plan 92 concluded that grade is too noisy to trust and added structural correction. Plan 102 then used the same grade as an inviolable partition key. This plan resolves the inconsistency in Plan 92's direction. + +### The relationship signal is already computed and currently spent on lossy excerpts + +`buildHunkRelationshipGraph()` runs before packet construction and produces strength-graded `same_symbol`, `symbol_mention`, and `planner_hint` edges from tree-sitter enclosing-symbol facts and `findSymbolMentions(..., contextMode: "symbols")`. Those edges buy a `relatedChangedContext` excerpt — capped at three per packet, 2,500 source and 1,500 patch characters each — plus a profile nudge. `buildRelatedChangedContext()` explicitly *skips* an edge whose target already sits in the packet, because membership makes the excerpt redundant. + +Plan 102 made this argument and did not act on it: *"The relationship graph and related-context machinery try to reconstruct this adjacency later, with caps and omissions; safe same-file packing supplies it directly."* + +### Promotion through an edge is cheaper than the split it replaces + +A deep atom and four related light atoms cost five model conversations and `10 + 4×1 = 14` tool calls today (`toolBudget()` at normal depth, `standard` profile). Packed into one promoted deep packet they cost one conversation and 10 tool calls. Promotion collapses total capacity rather than multiplying it, because tool budget is per packet. Promotion's real cost is fidelity of the per-hunk coverage record, which per-member promotion records address directly. + +### The one clean paid comparison Plan 102 produced points the right way + +In the retry cohort's dilution case — the only family where the packed arms received treatment (`1/1`) — arm B (packed, today's budget) was the only arm whose finding survived verification (`should_find` final `1/1`); A and C both lost it at verification. B also used fewer input tokens than A (66,780 vs 72,167) and less model-service time (65.1s vs 72.5s). Arm C, the `atom-scaled` budget, spent 106.1s of service time and $0.4427 in Stage 7 against B's 65.1s and $0.3460 — 63% more service time for no recall gain. `n = 1` from an invalid cohort decides nothing, but it is sufficient reason not to spend this plan's budget re-testing a scaled tool allowance. + +### Four pressures this plan inherits and must guard + +1. **Attention-note pressure.** Plan 102's replay recorded six standalone note strings per run disappearing under the unchanged three-note cap. Relationship packing raises note density, so this worsens and needs an explicit reject-and-report rule. +2. **Profile upgrades are unbounded.** The retry cohort showed packing pulling zero-tool `simple` atoms into tool-using packets: consistency B/C requested six tool calls against A's four while packets fell five to three, and input tokens rose 5.6%. The floor blocks downgrades; nothing bounds upgrades. Report it; do not assume fewer packets means lower cost. +3. **Symbol context is single-primary today.** `readEnclosingSymbolSource()` reads one symbol — the top-ranked `HunkSymbolFacts` — for the whole packet, and `computeSymbolContextBudget()` sets `singlePrimarySymbol = false` for multi-symbol packets, selecting `multiple_symbols_keep_compact`. Since `same_container` packing combines *different* methods by definition, naive packing would silently drop every non-primary member's surrounding source, and the coarse `contextQuality` grade would not register it. This is the one place where relationship packing genuinely regresses today's behavior, and it is fixed by design section 4, not by rejection — rejecting every cross-symbol candidate would neuter the plan. +4. **Independent planner draws destroy attribution.** Four of Plan 102's six retry failures were cross-arm reconciliation against independently sampled Stage-5 plans. Pinning the plan per pair removes the cause rather than removing the check. + +## Current State + +- `hunkFirstGroups()`/`canJoinGroup()` in `src/pipeline/packet-builder.ts` join same-file hunks by identical enclosing symbol or `NEARBY_GAP_LINES = 30` proximity. The Plan 102 experiment was fully reverted; `src`, `scripts`, `tests`, and `evals` match the pre-experiment baseline. +- `buildHunkRelationshipGraph()` builds `same_symbol` (always `strong`), `planner_hint` (`strong` for symbol hints, `medium` for file hints), and `symbol_mention` (`strong` when the mention has an enclosing symbol, else `medium`) edges, capped at `MAX_RELATIONSHIP_EDGES_PER_HUNK = 8` per hunk, dropped on overflow in insertion order. +- `addSymbolMentionEdges()` guards ambiguity (`sourceIdentities.size > 1` → skip). `addPlannerHintEdges()` has no equivalent guard, so a repeated method name can produce spurious `strong` edges. +- `buildRelatedChangedContext()` consumes every non-`weak` edge whose target is outside the packet, capped at `MAX_RELATED_CONTEXTS_PER_PACKET = 3`. Any new `medium` edge source would therefore change flag-off context, attention notes, repository reads, and — through `hasRelatedChangedContext` in the `light`/mechanical branch — profiles. +- `buildPacket()` sets packet coverage to `maxCoverage(members)` and derives `reviewProfile` after related-context filtering, so absorbing a strong edge can currently lower the derived profile. +- `applyCoverageEscalations()` runs *after* packet construction, early-returns only when the packet is already `deep`, and stamps `coverageEscalation`. `CoverageEscalation["rule"]` is the single-member union `"test_coverage_delta"`, so a packed `normal` packet it escalates would overwrite any packing provenance stored there. +- `SymbolInfo` carries `ownerType` (Go method receivers, TypeScript classes, Rust `impl`/trait owners, Python classes, Solidity contracts) and `packageName` (Go). `generic-adapter.ts` resolves neither. `HunkSymbolFacts` — the per-hunk record the packet builder groups on — carries neither. `renderSymbol()` prefixes methods as `Owner.Method` inside `enclosingSymbol`, so container identity is only recoverable today by string-splitting a display string. `packet-context.ts:154` already resolves an owner structurally by name and kind, which is the precedent for a structural container key. +- `packageName` is constant within a Go file and carries no within-file affinity signal; it is deliberately not plumbed. +- `pnpm test -- ` does **not** filter: it runs the full suite and silently drops the argument (verified — `pnpm test -- tests/config-loader.test.ts` executed all 761 tests). Plan 102's focused verify commands were therefore never focused. This plan uses `pnpm exec vitest run `. +- Plan 102's report script redacted every failure `message` to `sha256::` through `safeFailureOutput()`, leaving its preserved evidence unreadable. +- There is no seam to supply a pre-recorded Stage-5 plan to a review. `runReview()` always calls `runPlanner()`. + +## Goal + +1. Make the packing predicate semantic: atoms combine when repository structure says they are related, regardless of how the planner graded them. +2. Prove deterministically and for free that the predicate produces a **treatment surface**, including packs that cross coverage grades, before authorizing a paid call. +3. Answer one paid question with attribution: given an identical Stage-5 plan, does a reviewer shown related hunks together find a cross-hunk inconsistency that a reviewer shown them separately misses? +4. Keep every Plan 102 invariant that passed, and add per-member coverage, context-quality, and connectivity invariants that Plan 102 did not need. +5. End in exactly one product path with no surviving flag, under either outcome. + +## Design + +### 1. Structural container identity + +Add `ownerType?: string` and `ownerKey?: string` to `HunkSymbolFacts`, populated in `src/repo/symbol-extraction.ts` from the primary changed symbol. `ownerType` is the display name. `ownerKey` is the **structural** identity: `${path}:${ownerName}:${ownerStartLine}-${ownerEndLine}`, resolved by locating the owning symbol in the file's extracted symbols by name and container kind — the lookup `packet-context.ts:154` already performs. Where the owner cannot be resolved structurally, `ownerKey` is absent and the hunk has no container affinity. Duplicate or nested same-named classes therefore never collide. + +No adapter changes are required: Go resolves the owner from the method receiver, TypeScript from the enclosing class, Rust from the `impl`/trait owner, Python from the enclosing class, and Solidity from the owning contract. `generic-adapter.ts` does not, so fallback-path files simply have no container affinity — correct, not a gap. + +### 2. A separate packing-affinity view + +`same_container` is **not** added to `HunkRelationshipGraph`. Adding any new `medium` source there would change flag-off excerpts, attention notes, `readSymbol` calls, profiles through `hasRelatedChangedContext`, and — through `MAX_RELATIONSHIP_EDGES_PER_HUNK` overflow — which existing edges survive. + +Instead, build a `PackingAffinity` view per file, computed only when the flag is on: + +- **derived** edges: copied from the relationship graph — `same_symbol`, `symbol_mention` with `strength === "strong"`, and `planner_hint` with `strength === "strong"` **and** unique resolution (the hint's normalized symbol key matches exactly one changed symbol identity in the file, mirroring `addSymbolMentionEdges()`'s existing ambiguity guard); +- **structural** edges: `same_container`, emitted for hunk pairs in one file sharing a non-empty `ownerKey`; +- no per-hunk edge cap, because this view is not rendered into any prompt and cannot evict prompt context. + +The report attributes every pack to `derived` versus `structural` edges separately, so `same_container`'s contribution is measurable rather than assumed. + +### 3. Connectivity-preserving greedy expansion + +Within one file, over hunk-first atoms only: + +1. Order atoms by first source position. +2. Take the source-earliest unassigned atom as a packet seed. +3. Repeatedly admit the source-earliest unassigned atom that (a) has an eligible edge to **at least one atom already in the packet** and (b) keeps total hunks `<= MAX_HUNKS_PER_PACKET` and total patch characters `<= MAX_PATCH_CHARS` via `combinedPatchChars()`. +4. Stop when no candidate qualifies; flush and seed the next packet from the remaining atoms. +5. Order packets by first source position; render hunks in source order within each packet. + +Every resulting packet is connected by construction, which a source-order chunk of a connected component is not — a chain `A—B—C` appearing in source order `A, C, B` would chunk into a disconnected `{A, C}`. Expansion is also not contiguity-bound: `A1, X, A2` with an `A1—A2` edge packs `A1` and `A2` and leaves `X` alone, which is precisely the separated-sibling case this plan exists for. + +Atoms with no eligible edge remain their own packets, exactly as today. Direct `whole-file` and content-probed `file-diff` returns from `groupHunks()` bypass packing entirely. + +### 4. Symbol context follows membership + +For a packed packet whose member atoms have more than one distinct primary symbol identity, read enclosing-symbol source for **every** member atom's primary symbol rather than the top-ranked one alone: + +- allocate each member `floor(symbolContextBudget / memberSymbolCount)` characters, computed by the existing `computeSymbolContextBudget()` path per member; +- preserve the existing adaptive/sliced selection per member, and render members in source order; +- record `contextQuality` per member atom in packet telemetry, alongside the packet-level grade. + +A candidate is rejected when any member atom's context quality would fall below its standalone quality *and* below `sliced` — that is, to `outline_only` or `path_only`. Falling from `full` to `sliced` under a shared budget is expected and reported, not fatal. + +This is the one place relationship packing regresses today's behavior if left alone, and it is why the throughput case is secondary: packed packets carry more context, not less. + +### 5. Transactional candidate evaluation + +Every candidate is dry-built against a cloned relationship accumulator and a suppressed telemetry sink. Nothing is written to the real graph, artifacts, or event stream until a candidate is committed. A candidate is abandoned — leaving its atoms as separate packets — when the combined packet would: + +- drop a lens any member atom routed standalone; +- newly omit a planner focus note belonging to a hunk whose `reviewPriority` is `high` or `critical`; +- reduce any member atom's context quality below `sliced`; +- lower any member's effective coverage, or lower the effective profile below the maximum standalone member profile. + +Every abandonment is recorded with its reason code and the atoms involved. + +### 6. Per-member coverage promotion records + +Packet coverage remains `maxCoverage(members)`. Add to `ReviewPacket`: + +```ts +memberCoveragePromotions?: Array<{ + atomId: string; + hunkIds: string[]; + from: CoverageLevel; + to: CoverageLevel; + edgeSource: "same_symbol" | "symbol_mention" | "planner_hint" | "same_container"; + edgeStrength: HunkRelationshipStrength; +}>; +``` + +ordered by first source position. `CoverageEscalation` and its single `"test_coverage_delta"` rule are left untouched, so Plan 92's packet-level escalation provenance and this plan's member-level promotion record coexist without either overwriting the other. Demotion of any member is an invariant failure, not a reportable event. + +### 7. Profile floor and dispatch rank + +Plan 102's floor is retained verbatim: `effectiveReviewProfile = max(derivedPackedProfile, max(standalone member profiles))` over the explicit rank map `{ simple: 0, standard: 1, investigate: 2 }`, never lexical comparison. Standalone profiles are computed through the same Stage-6 path the atom receives with packing off, against the scratch accumulator. + +`dispatchRank` is recomputed with the unchanged `packetDispatchRank(filePath, facts, combinedChangedLines)` formula. Scheduling movement is reported, never suppressed. + +### 8. Configuration and the pinned-plan seam + +```yaml +review: + packRelatedHunks: false +``` + +One boolean in raw and resolved schemas, `CodegenieConfig`, defaults, config-source tracking, repo-safe filtering, the strict eval-case `review` schema, and `applyCaseReviewConfig()`. Deleted in step 10 under either outcome. + +Separately, add an **eval-only** pinned-plan seam so both arms of a paid pair consume one Stage-5 draw: + +- a planner-draw mode that runs Stage 1–5 and writes the resulting `ReviewPlan` plus its SHA-256; +- an eval-case field (`review.pinnedPlanDir`) that supplies execution *k* of a repeat with plan *k*, validated by hash and by exact `hunkId` membership against the current diff, failing closed on any mismatch; +- no user-facing config field, no `codegenie.toml` surface, and no reachable path in normal review. + +This is eval-harness capability, not product surface. At teardown it survives only if it is independently useful to other eval work; otherwise it goes with the flag. + +## Validation Strategy + +### A. Free deterministic replay (no model calls) + +Port `scripts/packet-packing-report.ts` from `7ebfd2f` down to `replay` plus `treatment`, with tests in `tests/packet-packing-report.test.ts`. + +**Failure records are structured and templated**, not hashed and not raw. Each failure carries a closed-set `code`, structured fields (`runId`, `packetId`, `atomIds`, `hunkIds`, counts, enum reasons), and a `message` rendered from a template over those fields. Raw exception text and repository source are never interpolated. Plan 102's `safeFailureOutput()` hashing is not ported; its credential stripping is. + +Run over the four retained runs. Assert per run: + +- every reviewable hunk appears exactly once, in source order; +- no atom is split or internally reordered; +- **every packed packet is connected** under the affinity view; +- five-hunk, 12K-patch, six-lens, and three-note caps hold; file boundaries hold; +- no planner-requested lens dropped, no `high`/`critical` focus note newly omitted, no member context quality below `sliced` — or the candidate was abandoned and recorded; +- no effective profile or base budget below the maximum standalone member value; +- no member coverage demotion; every promotion carries a member record with a causing edge; +- every dispatch rank equals the existing formula; +- flag off is artifact-identical to current behavior, including the relationship-graph artifact, which proves the affinity view never wrote back. + +Report per run: packets off→on, atoms per packet, promoted members by edge source, abandoned candidates by reason code, packing edges split `derived` versus `structural`, per-member context-quality distribution, attention notes dropped, excerpts internalized, and scheduling movement. + +**Pre-registered numeric gates**, counted on the two *distinct* diffs (`dca8d870`, `81f806a6`) and reported for all four rows: + +| Gate | Threshold | +| --- | --- | +| Treatment surface — packets combining atoms Plan 102's `(coverage, lensSignature)` predicate would separate | `>= 5` per distinct diff | +| Cross-coverage packs — packed packets containing at least one member promotion | `>= 3` per distinct diff | +| `same_container` aggressiveness — multi-atom packets with structural edges enabled, versus derived-only | `<= 2.0x` | +| Promoted members as a share of reviewable hunks | `<= 25%` per run | +| Members whose context quality falls below `sliced` | `0` | + +Missing the first two means the predicate does not reach materially more code than Plan 102's did, or does not do the new thing it exists to do — stop here, for free. Exceeding the third or fourth means demoting `same_container` to context-only and re-running with derived edges alone before continuing. + +Also report what Plan 102's compatibility tier would add on top, as a diagnostic for a possible later throughput plan. Do not implement that layer here. + +### B. One paid question, isolated by a pinned plan + +**Fixture.** One family, `container-consistency`, under `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/relationship-packing/recall/`: several methods on one Go receiver type, four applying a shared guard and one omitting it, separated by more than `NEARBY_GAP_LINES` so today's grouper yields one atom each. The same repository also carries an **unrelated local boundary bug** in a free function, which serves as the control. + +Two expectations: + +- `cross-hunk` — the missing guard. Invisible without seeing siblings. Packing should help. +- `local-control` — the boundary bug. Visible in isolation. Packing must not hurt. + +Fixture rules learned from Plan 102: the target must be a hunk the planner grades **ordinarily** on its own (if it grades `deep` alone, the fixture is wrong); target and siblings must be related by a real edge, verified by the no-model treatment check before launch; and expectations constrain path, line range, and failure mode while **accepting any plausible category**. Plan 102 recorded three executions that found its bug and scored `partial-match` solely because the expectation demanded `correctness` and the reviewer said `security`. + +**Procedure.** Two cases, `container-a.yml` (packing off) and `container-b.yml` (packing on), identical except that field. For each of 6 repeats: draw one Stage-5 plan, hash it, and run both arms pinned to it. Six planner draws total, so planner variance is still sampled *across* pairs while packing is the only difference *within* a pair. With the plan pinned, cross-arm atom and hunk reconciliation is exact and **required** — the check Plan 102 could not support is now sound. + +Run the one-repeat treatment preflight first. Arm B's target packet must contain at least two source atoms. If it does not, repair the fixture and rerun — a preflight treatment miss is a fixture defect and explicitly does **not** trigger teardown. + +**Pre-registered decision table** over 6 pairs, where `Ax`/`Bx` are hits on `cross-hunk` and `Ac`/`Bc` on `local-control`: + +| Condition | Outcome | +| --- | --- | +| `Ax >= 5` | **Void — no headroom.** Fixture too easy; redesign, do not count as evidence. | +| `Ax == 0 and Bx == 0` | **Void — no signal.** Bug undetectable at this model/config; redesign. | +| `Bx - Ax >= 3` and `Bc >= Ac - 1` | **Pass.** Proceed to collateral and capacity confirmation. | +| `1 <= Bx - Ax <= 2` | **Inconclusive.** Record; do not ship; a larger `n` is a separate decision. | +| `Bx <= Ax` or `Bc < Ac - 1` | **Fail.** Take the failure branch. | + +Requiring `+3` of 6 is a strong-effect-only test. It cannot detect a small improvement, and the reconciliation note must say so rather than reporting a null as safety. + +### C. Collateral and capacity, inside this plan + +A pass on one Go fixture does not license unconditional cross-language behavior. Before shipping, and within this plan: + +1. Run the complete `evals/fixtures/` suite flag-off and flag-on at `repeat: 1` with the fake provider. Every declared expectation, packet invariant, and config parse must match. +2. Run the complete `evals/skill-semantics/` suite (TypeScript, Python, Solidity) flag-off and flag-on at `repeat: 1`. Cross-language container semantics differ — Solidity contracts and Python classes are much broader containers than a Go receiver — so this is where `same_container` over-packing would surface. An A-pass/B-fail difference is a stop-for-investigation; rerun only the affected case at `repeat: 3` before deciding variance versus regression. +3. Run one production-shaped capacity pair on the pinned `trails-api` diff at concurrency 6, flag-off and flag-on, and report reviewed-hunk set, packet count, model-service time, tokens, and cost per reviewed hunk. This is a capacity and safety observation, not an economics gate — this plan does not claim throughput and must not be blocked on producing it. + +These stay in this plan so the decision and the teardown happen together. Deferring them would leave `packRelatedHunks` alive across plan boundaries, which is the dark path Plan 102 exists to forbid. + +### Cost discipline + +No paid model call begins before `approvedValidationCostUSD` is recorded in this plan's reconciliation note; the owner supplies the amount and there is no default. + +Projection from Plan 102's measured $0.56–$0.81 per execution on comparable Go fixtures: + +| Phase | Executions | Projection | +| --- | ---: | ---: | +| Treatment preflight (1 pair + 1 plan draw) | 3 | ~$2 | +| Paired recall, 6 pairs | 18 | ~$14 | +| `evals/fixtures/` collateral (fake provider) | — | $0 | +| `evals/skill-semantics/` collateral, both arms | 6 | ~$5 | +| Production capacity pair | 2 | ~$50 | +| Contingency 25% | — | ~$18 | +| **Reservation** | | **~$90** | + +Record actual and projected spend after every phase and stop if actual plus projection exceeds the ceiling. The production pair is the only expensive item; it runs last and only after a pass. + +## In-Scope Files + +- `src/repo/symbol-extraction.ts` — `ownerType` and structural `ownerKey` on `HunkSymbolFacts`. +- `src/pipeline/packet-builder.ts` — affinity view, greedy expansion, transactional candidates, multi-member symbol context, member promotion records, profile floor, telemetry. +- `src/pipeline/review-runner.ts` — the eval-only pinned-plan seam. +- `src/types.ts` — `ownerType`/`ownerKey`, `memberCoveragePromotions`, per-member context-quality telemetry, the temporary config field. +- `src/config/schema.ts`, `src/config/config-loader.ts`, `src/evals/eval-runner.ts` — flag plumbing and `pinnedPlanDir`; both deleted or re-justified in step 10. +- `scripts/packet-packing-report.ts`, `tests/packet-packing-report.test.ts` — `replay` and `treatment` modes with templated failure records. +- `tests/pipeline-phase5.test.ts`, `tests/config-loader.test.ts`, `tests/evals.test.ts`, and the symbol-extraction suite. +- `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/relationship-packing/` — fixture, cases, pinned plans, immutable logs, reports. +- `specs/plans/README.md` and affected `specs/project/` docs. + +## Non-Goals + +- Cross-file packing. File identity remains the context and anchor boundary. +- Raising the five-hunk, 12K-patch, six-lens, or three-note caps. +- Reintroducing Plan 102's `atom-scaled` budget, or any tool-budget change. +- Layering Plan 102's coverage/lens compatibility packing on top. The replay reports what it would add; a separate plan may act on it. +- Claiming a throughput or economics win. Packed packets carry more symbol context and may cost more; the capacity run observes this rather than gating on it. +- Changing prompt templates, attention-note ordering, `canJoinGroup()` semantics, Plan 100's dispatch policy, or Plan 92's escalation rules. +- Exposing `packRelatedHunks` or `pinnedPlanDir` as user-facing configuration, or leaving either dark after step 10. +- Treating `packageName` as an affinity signal. + +## Implementation Steps + +1. Reconcile the drift and working-tree checks. Add `ownerType` and structural `ownerKey` to `HunkSymbolFacts`. + + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "container identity"` → exit 0; Go receivers, TypeScript classes, Rust `impl` blocks, Python classes, and Solidity contracts resolve; free functions and generic-adapter files resolve none; two same-named classes in one file produce distinct `ownerKey` values. +2. Add the dark `review.packRelatedHunks` field end-to-end, including strict eval schema, `applyCaseReviewConfig()`, defaults, repo-safe filtering, and config-source tracking. + + **Verify:** `pnpm exec vitest run tests/config-loader.test.ts tests/evals.test.ts` → exit 0; default `false`, eval override applies, winning source identified. +3. Build the `PackingAffinity` view, including the `planner_hint` unique-resolution guard. Prove it never mutates the relationship graph. + + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "packing affinity"` → exit 0; `hunk-relationships.json`, related-context excerpts, attention notes, repository reads, and profiles are byte-identical with the flag on and packing suppressed; ambiguous repeated symbol names produce no structural or hint edge; the per-hunk edge cap is unaffected. +4. Wrap `hunkFirstGroups()` output as atoms with standalone coverage, routed lenses, profile, and context quality captured through the scratch path. Add flag-off golden parity before any packing runs. + + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "packet atom|flag-off parity|whole-file bypass"` → exit 0. +5. Implement connectivity-preserving greedy expansion. + + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "greedy expansion"` → exit 0, covering: the `A—B—C` chain in source order `A, C, B` never producing a disconnected packet; `A1, X, A2` packing across the unrelated `X`; cap-bound expansion stopping without splitting a packet into disconnected parts; interleaved components; unrelated atoms staying separate; hunk bijection and source order. +6. Implement multi-member symbol context and per-member context-quality recording. + + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "member symbol context"` → exit 0; a two-symbol packet carries both symbols' source under a shared budget; `full`→`sliced` degradation is reported and allowed; a member falling to `outline_only` aborts the candidate. +7. Implement transactional candidate evaluation, member promotion records, and the profile floor. + + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "candidate rejection|member promotion|profile floor"` → exit 0; an abandoned candidate leaves no telemetry, graph mutation, or artifact residue; lens-drop, `high`-priority focus-note-drop, and context-collapse candidates all abort; promotions are recorded per member with the causing edge; `coverageEscalation` is untouched and a subsequent Plan 92 escalation does not erase promotion records; demotion fails closed. +8. Add the eval-only pinned-plan seam and its validation. + + **Verify:** `pnpm exec vitest run tests/evals.test.ts -t "pinned plan"` → exit 0; a pinned plan reproduces identical Stage-6 inputs across two runs, a hash mismatch fails closed, a `hunkId` set mismatch fails closed, and no user-facing config path can reach it. +9. Port the report script and run the four-run replay. Record the full table and every pre-registered gate in this plan's reconciliation note. + + **Verify:** `pnpm exec vitest run tests/packet-packing-report.test.ts` → exit 0 with templated-message and fail-closed regressions; then the replay command exits 0 with `modelCallsObserved: 0` on every row, all invariants green, and every numeric gate in section A satisfied on both distinct diffs. Otherwise stop; no paid phase is authorized. +10. Record the owner-approved ceiling, build the fixture, and run the paid programme in order: treatment preflight, 6 pinned pairs, `evals/fixtures/`, `evals/skill-semantics/`, then the production capacity pair. Apply the decision table verbatim. + + **Verify:** preflight target packet combines at least two atoms; cross-arm reconciliation is exact for every pair; the decision table yields Pass, Inconclusive, Fail, or Void with no discretion; collateral suites show no reproducible regression; cumulative spend stays within the ceiling. +11. Record the decision and tear down the scaffolding in a dedicated commit. + + **Preserve evidence first:** copy every JSON report produced by completed phases into `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/relationship-packing/reports/` with a `manifest.sha256`, alongside the six pinned plans and their hashes. Mark unreached phases `not_run` with their stopping reason. + + **If Pass:** make packing unconditional; delete `review.packRelatedHunks` and the unpacked path with all schema, loader, and eval plumbing. Keep `ownerType`/`ownerKey`, the affinity view, greedy expansion, multi-member symbol context, member promotion records, the profile floor, and packing telemetry as unconditional behavior. Reduce the report script to a golden check against frozen packet shapes. Retain `pinnedPlanDir` only if it is independently useful to other eval work; otherwise delete it. + + **If Fail, Inconclusive, or Void:** delete the packing pass, the flag and its plumbing, the affinity view, the atom wrapper, member promotion records, and the report script with its tests. `ownerType`/`ownerKey` and multi-member symbol context survive **only** if the replay showed them independently improving related-context selection or context quality with packing off; otherwise they go too. A Void outcome permits one fixture redesign and one rerun before the failure branch; a second Void takes the failure branch. Do not leave the feature dark. + + **Verify:** `rg -n "packRelatedHunks" src scripts tests evals` → exit 1; the evidence manifest verifies; the reconciliation note records the decision table result, actual spend, and the strong-effect-only limitation of `n = 6`. +12. Run the complete repository gate. + + **Verify:** `pnpm run check && pnpm test && pnpm build` → exit 0. + +## Tests and Commands + +Focused gates — note `pnpm exec vitest run`, because `pnpm test -- ` silently runs the whole suite: + +```bash +pnpm run check +pnpm exec vitest run tests/pipeline-phase5.test.ts tests/config-loader.test.ts tests/evals.test.ts tests/packet-packing-report.test.ts +``` + +Deterministic four-run replay: + +```bash +pnpm exec tsx scripts/packet-packing-report.ts replay \ + --repo /home/peter/Dev/0xsequence/trails-api \ + --run /home/peter/Dev/0xsequence/trails-api/.codegenie/runs/20260724-135818-740d73f2 \ + --run /home/peter/Dev/0xsequence/trails-api/.codegenie/runs/20260724-150405-fe1548ae \ + --run /home/peter/Dev/0xsequence/trails-api/.codegenie/runs/20260724-162739-81f806a6 \ + --run /home/peter/Dev/0xsequence/trails-api/.codegenie/runs/20260724-184952-dca8d870 \ + --compare-predicate plan102 \ + --distinct-diffs \ + --output /tmp/plan103-relationship-shape.json +``` + +Expected: exit 0, four rows with duplicate diffs marked and counted once, zero model calls, zero invariant failures, all section-A gates satisfied, and a `plan102Predicate` diagnostic block. + +Paid preflight, then the pinned paired suite: + +```bash +pnpm dev eval \ + --eval-dir /home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/relationship-packing/recall \ + --no-cache + +pnpm exec tsx scripts/packet-packing-report.ts treatment \ + --logs /home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/relationship-packing/recall/logs \ + --cohort \ + --expected-repeats 1 \ + --require-pinned-plan \ + --output /tmp/plan103-treatment-preflight.json +``` + +Note the `pnpm dev eval` spelling: under pnpm 11 a literal `--` is passed through to Commander and rejected before suite allocation. + +## Acceptance Criteria + +- Flag off produces byte-identical packet artifacts, IDs, order, profiles, context, budgets, **and relationship-graph artifacts** — proving the affinity view never wrote back. +- Flag on preserves every atom, assigns every reviewable hunk exactly once in source order, and respects file, five-hunk, and 12K boundaries. +- Every packed packet is connected under the affinity view; unrelated atoms are never combined; no packet is a source-order chunk of a component. +- No planner-requested lens dropped, no `high`/`critical` focus note newly omitted, no member context quality below `sliced`; candidates that would do any of these are abandoned transactionally and recorded. +- Every effective profile is at least the maximum standalone member profile; no member coverage falls; every promotion is recorded per member with its causing edge; `coverageEscalation` remains Plan 92's alone. +- Dispatch ranks use Plan 100's unchanged formula; scheduling movement is reported. +- The replay exits 0 with zero model calls and satisfies every pre-registered numeric gate on both distinct diffs, reporting `derived` versus `structural` edge attribution and per-member context quality. +- Every paid pair consumes one hash-verified Stage-5 plan; cross-arm atom and hunk reconciliation is exact; the decision table is applied without discretion. +- `evals/fixtures/` and `evals/skill-semantics/` show no reproducible collateral regression under the selected behavior. +- The production capacity pair reviews at least the baseline hunk set and its cost, tokens, and service time are reported without being claimed as a win. +- Paid validation never begins without a recorded ceiling; every phase records actual and projected spend. +- Report failures carry templated, structured, human-readable messages with no raw exception text, no repository source, and no hashing. +- Teardown leaves exactly one product path with no surviving flag and no dark code, and preserves every produced report and pinned plan under a verified manifest. +- Checks, tests, and build pass; the final behavior and the strong-effect-only `n = 6` limitation are documented. + +## Stop Conditions + +- Treatment surface below 5 packets, or cross-coverage packs below 3, on either distinct diff: the predicate does not reach materially more code than Plan 102's, or does not do the new thing. Stop before any paid call and record the premise as disproven. +- `same_container` more than doubles multi-atom packets versus derived-only edges, or promotes more than 25% of reviewable hunks on any run: demote it to context-only and re-run the replay with derived edges before continuing. +- Any atom split or reordered, hunk lost or duplicated, disconnected packet, cap exceeded, member coverage demotion, profile or budget downgrade, or member context collapse below `sliced` that was not abandoned: fix the deterministic design before spending anything. +- The affinity view mutates the relationship graph, or flag-off artifacts change: stop; the isolation guarantee is the basis of every later comparison. +- The replay cannot reproduce flag-off artifacts from recorded inputs or resolves different production refs: reconcile the evidence. +- A paid pair's two arms consumed different Stage-5 plans, or a pinned-plan hash or `hunkId` set fails validation: discard that pair and rerun it. Never analyze an unpinned pair. +- The preflight target packet does not combine at least two atoms: repair the fixture and rerun. This is a fixture defect and explicitly does **not** trigger teardown; Plan 102 stopped here after a single repair when its own step 7 called for another. +- The decision table returns Void twice: take the failure branch rather than redesigning a third time. +- Cross-arm reconciliation is proposed as optional: reject it. With the plan pinned it is exact and required; without pinning the pair is invalid. +- Actual plus projected spend exceeds the approved ceiling: stop for explicit approval. Do not weaken a gate to fit a budget. +- "Keep packing dark" always means *do not ship it in this iteration*; every such outcome terminates in step 11's failure branch in the same change. Deferring teardown to a later plan is itself a stop condition. + +## Maintenance Notes + +- Any future change to `hunkFirstGroups()`, `canJoinGroup()`, the relationship edge builders or their strengths, `buildRelatedChangedContext()`, `readEnclosingSymbolSource()`, `computeSymbolContextBudget()`, `packetReviewProfile()`, coverage levels, packet caps, `packetDispatchRank()`, or `toolBudget()` invalidates the recorded shape, promotion, context, and order baselines; rerun the replay before changing unconditional packing behavior. +- Reviewers should scrutinize affinity-view isolation, edge-source attribution, per-member context quality, promotion records, abandonment reasons, and pinned-plan provenance — not packet count. +- Plan 102's negative result is why this plan exists. Its preserved reports and paid logs under `codegenie-private-evals/trails-api/packet-packing/` remain the record of the rejected coverage/lens predicate and must not be edited or deleted. +- Throughput packing of *unrelated* compatible atoms remains a separate, later question. This plan measures what it would add but deliberately does not ship it: bundling a quality mechanism with a cost mechanism is what made Plan 102 unmeasurable. +- Commit hashes in this document are rebase-unstable. Locate referenced work by commit subject when a hash does not resolve. diff --git a/specs/plans/README.md b/specs/plans/README.md index 8f3cea8..e1e871b 100644 --- a/specs/plans/README.md +++ b/specs/plans/README.md @@ -106,3 +106,4 @@ This directory tracks implementation plans for confirmed improvements. Status va | 100 | COMPLETE | Issue 100: Short Hunk IDs — Planner Coverage Survival and Dispatch Resilience | | 101 | IN PROGRESS (paid semantic A/B pending) | Issue 101: Exact Skill Provenance and Evidence-Gated Bundled-Skill Revision | | 102 | COMPLETE (failed gate; baseline restored) | Issue 102: Same-File Packet Packing | +| 103 | PENDING | Issue 103: Relationship-Driven Same-File Packet Packing | From 49351348b45e71fc549027ac349eeb565a4815f3 Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 12:01:25 -0400 Subject: [PATCH 07/24] docs(plans): revise Plan 103 to compatible-atom packing with relationship ordering Artifact-only diagnostics over the retained runs disproved the first draft's premise. Relationship-required packing yields 4 combinable atom pairs and a 4.2% packet reduction on the one run with usable planner coverage, against Plan 102's measured 21.9%. Of 13 candidate atom pairs, the 5-hunk cap blocks 7 and coverage inequality blocks only 2 - and zero structural pairs cross a coverage grade. Relationships between distinct atoms are sparse because today's grouper already absorbs most same-symbol adjacency into atoms. So the plan now adopts Plan 102's compatibility predicate and caps wholesale - its deterministic gate already passed on four runs - and demotes relatedness from a membership predicate to a composition-ordering preference. Related atoms are admitted first when filling a packet, which changes which hunks sit together without changing how many packets exist. Selection order and render order are separated explicitly; hunks always display in source order. A cap sweep against a simulator validated to within one packet of Plan 102's real implementation shows 5 hunks captures ~80% of achievable compression, with 10 adding ~5 points before saturating. The cap stays. Deleted on evidence: the relationship-required predicate, all coverage-promotion machinery and its CoverageEscalation conflict with Plan 92, and the proposed stage reordering (the planner already grades related siblings consistently in 132 of 134 measured pairs). The paid phase is now one question - recall versus packet size, measured at caps 1/3/5 over six repeats sharing one hash-pinned Stage-5 plan. It is a non-inferiority test because the benefit, roughly 19 more reviewed hunks per run, is already measured deterministically. Relationship ordering gets no paid arm; it is adopted or dropped on a free deterministic gate. Adds an introduction and TL;DR in plain language covering the problem, the outcome, the seven pieces of machinery, the dilution risk, and the delete-or-ship teardown. Co-Authored-By: Claude Opus 5 (1M context) --- ...-103-relationship-driven-packet-packing.md | 514 +++++++++--------- specs/plans/README.md | 2 +- 2 files changed, 261 insertions(+), 255 deletions(-) diff --git a/specs/plans/103-issue-103-relationship-driven-packet-packing.md b/specs/plans/103-issue-103-relationship-driven-packet-packing.md index b29ac9a..1310dc5 100644 --- a/specs/plans/103-issue-103-relationship-driven-packet-packing.md +++ b/specs/plans/103-issue-103-relationship-driven-packet-packing.md @@ -1,362 +1,371 @@ -# Issue 103: Relationship-Driven Same-File Packet Packing +# Issue 103: Compatible-Atom Packing with Relationship-Ordered Composition Status: PENDING -Related: Plan 102 (`102-issue-102-same-file-packet-packing.md`) — COMPLETE as a failed treatment gate; supplies the atom implementation, the deterministic replay harness, and the negative result this plan is built from. Plan 67 (hunk relationship context) built the graph this plan spends differently. Plan 92 (coverage calibration and escalation) supplies the coverage-promotion precedent and the noisy-grade evidence. Plan 100 (dispatch rank), Plan 79 (repeat/recall harness), Plan 32 (adaptive Stage-6 symbol context). -Planned from: Plan 102's preserved evidence — `plan102-packet-shape.json`, `plan102-eval-preflight-invalid-ace65769.json`, and `plan102-eval-preflight-retry-invalid-5bd80f2c.json` under `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/reports/` (manifest verified) — plus the same four retained runs, 2026-07-25 +Related: Plan 102 (`102-issue-102-same-file-packet-packing.md`) — COMPLETE as a failed treatment gate; its deterministic packing mechanism passed every invariant and is adopted here. Plan 67 (hunk relationship context) built the graph this plan reuses for ordering. Plan 92 (coverage calibration), Plan 100 (dispatch rank), Plan 79 (repeat/recall harness), Plan 32 (adaptive Stage-6 symbol context). +Planned from: Plan 102's preserved evidence under `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/reports/` (manifest verified), plus artifact-only diagnostics over the same retained runs, 2026-07-25 Production replay refs: base/merge-base `d1c49bdf6a8002ec2ec27faac94a932d736532b2`; head `fbb5f8761c2c296e115af17e919a7c35d9de8373` -Planned at: commit `f372f73` (branch `next`) -Recommended priority: after Plan 101's paid A/B settles. Plan 102's deterministic half already passed; its paid half never asked the question. This plan changes the packing predicate so the question becomes askable, and isolates Stage 6 so the answer is attributable. +Planned at: commit `7b61b34` (branch `next`) +Recommended priority: after Plan 101's paid A/B settles. -> Executor instructions: pack by the relationship graph that already exists, not by the planner's coverage grade. Preserve today's semantic grouper output as indivisible **atoms** exactly as Plan 102 did. Grow packets by connectivity-preserving greedy expansion — never chunk a component by source order, and never require contiguity. Keep the container-affinity signal out of the shared prompt-context graph entirely. Allow coverage promotion only through a packing-eligible edge, and record it per member, not on Plan 92's packet-level field. Extend symbol context to every member atom before packing multi-symbol atoms. Prove the treatment surface deterministically and for free before spending a model call, and pin one Stage-5 plan across both arms of every paid pair so packing is the only difference. Ship exactly one temporary flag; there is no budget-mode arm. +## Introduction and TL;DR + +### The problem + +Reviews run out of time before they run out of code. On the motivating pull request, Stage 6 produced **96 packets** covering 142 changed hunks. Only 57 packets were dispatched before the deadline, so **53 hunks received no review at all**. + +A packet is one conversation with the model, and most of its cost is fixed — the skill text, the prompt scaffolding, the forced finalization at the end. Measurements from that run show a five-hunk packet costs roughly the same as a one-hunk packet: about 3 model calls and 228 seconds either way. Yet 73 of the 96 packets held a single hunk. We were paying full conversation price to review one hunk at a time. + +### What this plan achieves + +Group compatible hunks from the same file into fewer, slightly larger packets — about **21% fewer packets**, measured deterministically across four production runs with no model calls. + +The point is not a shorter review. It is a **more complete** one. At the same time budget, 21% fewer packets means roughly **108 of 142 hunks reviewed instead of 89**. The win is coverage: fewer changed lines slip past the deadline unreviewed. + +A second, smaller benefit: when a packet has room and there is a choice of what to put in it, we prefer hunks that tree-sitter says are *related* — the same function, the same class or receiver type, one mentioning the other. That does not change how many packets exist, only which hunks sit together. A reviewer looking at five functions is then more likely to be looking at five *related* functions, and can notice things like "four of these added a validation check and the fifth didn't." + +### How the machinery works + +Seven pieces, in the order they run inside Stage 6: + +1. **Atoms.** Today's grouper already merges hunks that share an enclosing symbol or sit within 30 lines of each other. We keep it exactly as-is and treat each group it produces as an unbreakable unit. Nothing this plan does ever splits one. + +2. **Partitions.** Within each file, atoms are bucketed by two things the planner already decided: the review depth (`light`/`normal`/`deep`) and the set of review lenses requested. Only atoms in the same bucket may be combined. This is what guarantees no hunk is reviewed more shallowly, more expensively, or with different expertise than the planner asked for. + +3. **An affinity map.** A private, packing-only map of which atoms are related, built from tree-sitter facts: same symbol, same container (class, Go receiver, Rust `impl`, Python class, Solidity contract), or one symbol mentioning another. It is deliberately kept separate from the relationship data that already feeds review prompts, so turning packing off changes nothing. + +4. **Filling.** Walk each bucket in source order, filling a packet until it hits 5 hunks or 12,000 characters. When choosing what to add next, prefer a related atom; otherwise take the next one in source order. Hunks are always *displayed* in source order regardless of what order they were admitted — the code still reads top to bottom. + +5. **Safety checks before committing.** Each candidate packet is built in a scratch space first. If merging would drop a review lens, drop an important planner note attached to a high-priority hunk, or lose a function's surrounding source, the merge is abandoned and those atoms stay separate. Nothing is written until a candidate passes. + +6. **Context for every member.** If a packet ends up holding three different functions, the reviewer gets all three function bodies, splitting the context budget between them. Today's code reads only one, which would quietly leave the other two as bare diffs. + +7. **A floor on review effort.** A merged packet never gets a weaker review profile or a smaller tool budget than its parts would have received alone. + +### The risk, and how it gets tested + +The obvious danger is **attention dilution**: a bug sitting in a five-hunk packet might get less scrutiny than the same bug alone in its own packet. Nobody has ever measured this — Plan 102 tried and its test fixture was built wrong. + +So this plan measures it directly and measures nothing else. One test repository with about ten separated changes in one file, one of which contains a real bug. The same review runs three times over with packets capped at 1, 3, and 5 hunks, six repeats each, all sharing one frozen planning pass so packet size is the only variable. If recall at five hunks is within one hit of recall at one hunk, packing is safe and ships. If it degrades, the whole thing is deleted. + +Cost: about **$17** to answer that question, plus an optional $50 production-scale confirmation afterward. + +### The outcome, either way + +There is no middle state. Either packing becomes permanent, unconditional behavior and the temporary switches are deleted — or packing is removed entirely and the code returns to today's baseline. No flag survives, and no dark code path is left behind. + +> **Revision note.** The first draft of this plan (commit `7b61b34`) proposed replacing Plan 102's compatibility predicate with a relationship predicate, on the theory that coverage-grade equality was blocking the packs that mattered. Artifact-only diagnostics disproved that: on the one retained run with usable planner coverage, relationship-required packing yields 4 combinable atom pairs and a 4.2% packet reduction, against Plan 102's measured 21.9%. Coverage inequality blocks 2 of 13 candidate pairs; the 5-hunk cap blocks 7. This revision keeps Plan 102's mechanism and demotes relatedness from a membership predicate to a composition-ordering preference, which is what the evidence supports. + +> Executor instructions: adopt Plan 102's compatible-atom packing unchanged in predicate and caps — its deterministic gate passed on four runs and this plan does not relitigate it. Use the relationship graph only to decide *which* compatible atoms share a packet, never how many packets exist, and never to render hunks out of source order. Keep the container-affinity signal out of the shared prompt-context graph. Extend symbol context to every member atom. Do not raise `MAX_HUNKS_PER_PACKET`; the sweep below shows 5 already captures ~80% of achievable compression. The one open question is recall under packing, and the paid phase measures exactly that and nothing else. > -> **Commit references are rebase-unstable.** Branch `next` was rebased between Plan 102's teardown and this plan. The Plan 102 series is currently `5551547` (implementation), `8fceba9` (report tool), `7ebfd2f` (report fixes), `87a5a10` (retry docs), `f372f73` (teardown). If those hashes are unreachable, locate them by commit subject rather than trusting the hash. +> **Commit references are rebase-unstable.** Branch `next` was rebased during Plan 102. The Plan 102 series is currently `5551547` (implementation), `8fceba9` (report tool), `7ebfd2f` (report fixes), `87a5a10` (retry docs), `f372f73` (teardown). Locate referenced work by commit subject when a hash does not resolve. > -> Drift check: `git diff --stat f372f73..HEAD -- src/pipeline/packet-builder.ts src/pipeline/review-runner.ts src/repo/symbol-extraction.ts src/repo/tree-sitter/ src/config/schema.ts src/config/config-loader.ts src/types.ts src/evals/eval-runner.ts scripts/packet-packing-report.ts tests/pipeline-phase5.test.ts tests/config-loader.test.ts tests/evals.test.ts tests/packet-packing-report.test.ts` +> Drift check: `git diff --stat 7b61b34..HEAD -- src/pipeline/packet-builder.ts src/pipeline/review-runner.ts src/repo/symbol-extraction.ts src/config/schema.ts src/config/config-loader.ts src/types.ts src/evals/eval-runner.ts scripts/packet-packing-report.ts tests/pipeline-phase5.test.ts tests/config-loader.test.ts tests/evals.test.ts tests/packet-packing-report.test.ts` > Working-tree check: `git status --short -- src/pipeline/ src/repo/ src/config/ src/types.ts src/evals/` -> If the drift check reports committed changes, reconcile the current-state claims below. If the working-tree check reports changes owned by another task, stop and wait for that work to land or move this plan to an isolated worktree. ## Decision -Pack same-file atoms that the existing hunk relationship graph already relates, and stop using the planner's per-hunk coverage grade as a partition key. +Ship Plan 102's compatible-atom packing, with relationship-ordered composition, at today's caps. -- Run the existing symbol/proximity grouper unchanged. Its output groups remain indivisible **atoms**, exactly as in Plan 102. -- Build a **separate packing-affinity view** over the atoms of one file. It is derived from the relationship graph but never written back into it: the prompt-context graph, its per-hunk edge cap, its excerpts, and its attention notes stay byte-identical with the flag off. -- An atom pair is adjacent when any hunk of one has a **packing-eligible** edge to any hunk of the other: `same_symbol`, strong `symbol_mention`, uniquely-resolved strong `planner_hint`, or the new `same_container`. -- Grow packets by **connectivity-preserving greedy expansion**: start from the source-earliest unassigned atom, then repeatedly admit the source-earliest atom that has an eligible edge into the current packet set and fits the caps. Never chunk a connected component by source order; never require source contiguity. -- **Coverage promotion is permitted and recorded per member**, not banned and not folded into Plan 92's packet-level escalation field. -- **Symbol context follows membership.** Packing multiple primary symbols into one packet extends symbol-source reading to every member atom under a shared budget, rather than silently keeping only the top-ranked one. -- Lens intent, focus notes, and per-member context quality are protected by **transactional candidate rejection**, not by partitioning. +- Run the existing symbol/proximity grouper unchanged. Its output groups remain indivisible **atoms**. +- Partition a file's atoms by `(effectiveCoverage, normalizedPlannerLensSignature)` — Plan 102's predicate, unchanged. This is where the throughput comes from and it introduces **zero coverage promotion** by construction. +- Fill packets greedily within each partition under the unchanged `MAX_HUNKS_PER_PACKET = 5` and `MAX_PATCH_CHARS = 12_000`. +- **Relatedness decides composition, not count.** When choosing the next atom to admit, prefer one connected to a current member by a packing-affinity edge; fall back to the source-earliest atom that fits. With no relationships present this degrades to Plan 102's exact behavior. +- **Selection order is not render order.** Hunks always render in source order within a packet. Plan 102 conflated the two; they are separate here. +- Symbol context follows membership: multi-symbol packets read every member's primary symbol under a shared budget. +- Lens loss, focus-note loss, and per-member context collapse are prevented by **transactional candidate rejection**, not by narrowing the predicate. - Plan 102's monotonic profile floor is retained unchanged. -- One temporary flag, `review.packRelatedHunks`. One budget policy — today's. No `atom-scaled` arm. -- Every paid pair consumes **one pinned Stage-5 plan**, so the arms differ only by packing. +- One mechanism flag plus one experiment-only cap knob, both deleted at teardown. No budget-mode arm. -This is a different bet from Plan 102. That plan optimized packet count and treated relatedness as a nice-to-have; this plan optimizes cross-hunk reasoning and treats packet reduction as a secondary, reported effect that may well be negative. +There is **no coverage promotion anywhere in this plan.** Because the partition key preserves Plan 102's coverage equality, the member-promotion machinery, the `CoverageEscalation` conflict with Plan 92, and the promotion invariants from the first draft are all deleted rather than designed. ## Evidence -### Plan 102's deterministic mechanics worked; its predicate did not +### Plan 102's mechanism passed its deterministic gate + +Four-run replay, zero model calls, zero failures: `740d73f2` 109→85, `fe1548ae` 109→85, `81f806a6` 93→68, `dca8d870` 96→75 — 21.9–26.9% fewer packets with zero coverage promotions, zero effective profile or budget downgrades, zero cap violations, zero lens drops, and zero invalid dispatch ranks. Only the paid recall gate failed, and it failed on fixture treatment, never on recall. This plan adopts that mechanism rather than rebuilding it. -The four-run replay passed every invariant with zero model calls: `740d73f2` 109→85, `fe1548ae` 109→85, `81f806a6` 93→68, `dca8d870` 96→75, with zero coverage promotions, zero effective profile or budget downgrades, zero cap violations, zero lens drops, and zero invalid dispatch ranks. Atom preservation, hunk bijection, source ordering, profile floors, and dispatch-rank recomputation are proven machinery. This plan reuses them and changes which atoms are eligible to combine. +Note `740d73f2` and `fe1548ae` are the same commit, and all four retained runs are the same PR. This is one PR sampled at three head commits, not four independent samples. Every threshold below counts distinct diffs. -Note that `740d73f2` and `fe1548ae` are the same commit and PR — identical 109→85 rows. There are two distinct diffs among the four retained runs, and every threshold in this plan counts distinct diffs. +### Relationship-required packing is not a viable throughput mechanism -### The coverage/lens predicate was self-defeating +Artifact-only diagnostics over `dca8d870` (the only retained run with usable planner coverage — `740d73f2` has 0 surviving planner entries and `81f806a6` has 2): -Both paid cohorts failed on treatment validity, never on recall. In the repaired retry (`5bd80f2c`), consistency B and C were each treated `0/1`: the planner graded the target `HandleWire` hunk `deep` and its three safe siblings `light`, so the compatibility partition correctly refused to pack them. The first cohort failed the same way. +| Measure | Value | +| --- | ---: | +| Atom pairs adjacent under strong + container edges, same file, currently split | 13 | +| — blocked by the 5-hunk / 12K caps | 7 | +| — blocked by coverage inequality | 2 | +| — eligible to pack | 4 | +| Simulated packet reduction, relationship-required predicate | 96→92 (4.2%) | +| Same, at a hypothetical 8-hunk cap | 96→89 (7.3%) | -This is structural. A recall fixture needs a detectably risky hunk among safe siblings; the planner grades the risky one differently. Under a coverage-equality predicate, **any hunk interesting enough to carry a recall expectation is by construction unpackable**. Freezing the planner output does not fix that; only changing the predicate does. +Relationships between *distinct atoms* are sparse: today's grouper already absorbs most same-symbol adjacency into atoms, so what remains is thin. Relatedness cannot carry a throughput plan. It can, however, decide composition for free. -### The codebase already contradicts the predicate +### Coverage inequality is not the blocker -`applyCoverageEscalations()` exists because of Plan 92, and its own comment states the case: +Planner grades on `dca8d870` were deep:14, normal:14, light:2, skip:2 across 32 entries; the other 110 reviewed hunks take the `normal` default. Of 13 adjacent atom pairs, only 2 cross a coverage grade, and both come from `planner_hint` edges — **zero structural (`same_symbol`, `symbol_mention`, container) pairs cross a grade.** -> the planner's deep/normal assignment is a one-draw LLM judgment with measured run-to-run variance (the erc20 packet: normal in runs 46-50, deep in 51); these rules floor structurally suspicious packets to deep so attention amplification (budgets, ensemble passes) stops depending on the planner's mood +This retires the first draft's central claim. Plan 102's coverage-equality partition was self-defeating in a *fixture*, where the planner grades every hunk; in production it grades 23% of them and the rest share a default. Retaining coverage equality costs 2 of 13 opportunities and removes an entire subsystem. -Plan 92 concluded that grade is too noisy to trust and added structural correction. Plan 102 then used the same grade as an inviolable partition key. This plan resolves the inconsistency in Plan 92's direction. +It also retires the proposed stage reordering — showing the planner hunk groups so it grades siblings consistently solves a problem that occurs twice per PR. -### The relationship signal is already computed and currently spent on lossy excerpts +### The 5-hunk cap is near-optimal, not arbitrary -`buildHunkRelationshipGraph()` runs before packet construction and produces strength-graded `same_symbol`, `symbol_mention`, and `planner_hint` edges from tree-sitter enclosing-symbol facts and `findSymbolMentions(..., contextMode: "symbols")`. Those edges buy a `relatedChangedContext` excerpt — capped at three per packet, 2,500 source and 1,500 patch characters each — plus a profile nudge. `buildRelatedChangedContext()` explicitly *skips* an edge whose target already sits in the packet, because membership makes the excerpt redundant. +A cap sweep under Plan 102's predicate, using a simulator validated against Plan 102's real implementation (cap 5/12K reproduces 76/68/85 packets against the measured 75/68/85 — within one packet on all three runs): -Plan 102 made this argument and did not act on it: *"The relationship graph and related-context machinery try to reconstruct this adjacency later, with caps and omissions; safe same-file packing supplies it directly."* +| Cap | `dca8d870` | `81f806a6` | `740d73f2` | Packets >5 hunks | +| --- | --- | --- | --- | ---: | +| 5h/12K (today) | 96→76 (20.8%) | 93→68 (26.9%) | 109→85 (22.0%) | 0 | +| 6h/12K | 96→75 (21.9%) | 93→65 (30.1%) | 109→81 (25.7%) | 2–4 | +| 8h/12K | 96→73 (24.0%) | 93→64 (31.2%) | 109→80 (26.6%) | 4–5 | +| 10h/16K | 96→71 (26.0%) | 93→61 (34.4%) | 109→77 (29.4%) | 5–6 | +| 12h/24K | saturated — no further change | | | | -### Promotion through an edge is cheaper than the split it replaces +The predicate is worth ~21 points; the cap is worth ~5 more before saturating, because partitions run out of compatible atoms rather than hitting the cap. Raising it trades a handful of oversized packets for a few points of compression and enlarges the untested recall risk. **The cap stays at 5.** -A deep atom and four related light atoms cost five model conversations and `10 + 4×1 = 14` tool calls today (`toolBudget()` at normal depth, `standard` profile). Packed into one promoted deep packet they cost one conversation and 10 tool calls. Promotion collapses total capacity rather than multiplying it, because tool budget is per packet. Promotion's real cost is fidelity of the per-hunk coverage record, which per-member promotion records address directly. +### The value is run-level coverage, and the risk is packet-level recall -### The one clean paid comparison Plan 102 produced points the right way +Plan 102's Stage 7 spent 174 model calls, 5.321M tokens, `$22.85`, and 52.7 concurrency-normalized minutes on 57 dispatched packets, reviewing 89 of 142 reviewable hunks before the deadline. Forced finalization alone was 34 calls, 1.037M input tokens, and `$6.71` of pure per-packet overhead. Observed per-packet cost is roughly flat in packet size: -In the retry cohort's dilution case — the only family where the packed arms received treatment (`1/1`) — arm B (packed, today's budget) was the only arm whose finding survived verification (`should_find` final `1/1`); A and C both lost it at verification. B also used fewer input tokens than A (66,780 vs 72,167) and less model-service time (65.1s vs 72.5s). Arm C, the `atom-scaled` budget, spent 106.1s of service time and $0.4427 in Stage 7 against B's 65.1s and $0.3460 — 63% more service time for no recall gain. `n = 1` from an invalid cohort decides nothing, but it is sufficient reason not to spend this plan's budget re-testing a scaled tool allowance. +| Packet size | n | Avg calls | Avg input tokens | Avg model-service seconds | +| --- | ---: | ---: | ---: | ---: | +| 1 hunk | 41 | 3.10 | 91,585 | 228 | +| 2 hunks | 8 | 3.00 | 67,433 | 206 | +| 3 hunks | 3 | 2.67 | 77,940 | 158 | +| 4–5 hunks | 5 | 3.00 | ~109,000 | 228 | -### Four pressures this plan inherits and must guard +So a 21% packet reduction at a fixed time budget is worth roughly 19 more reviewed hunks on that run — 89/142 to ~108/142. That is the payoff, and it is run-level recall. -1. **Attention-note pressure.** Plan 102's replay recorded six standalone note strings per run disappearing under the unchanged three-note cap. Relationship packing raises note density, so this worsens and needs an explicit reject-and-report rule. -2. **Profile upgrades are unbounded.** The retry cohort showed packing pulling zero-tool `simple` atoms into tool-using packets: consistency B/C requested six tool calls against A's four while packets fell five to three, and input tokens rose 5.6%. The floor blocks downgrades; nothing bounds upgrades. Report it; do not assume fewer packets means lower cost. -3. **Symbol context is single-primary today.** `readEnclosingSymbolSource()` reads one symbol — the top-ranked `HunkSymbolFacts` — for the whole packet, and `computeSymbolContextBudget()` sets `singlePrimarySymbol = false` for multi-symbol packets, selecting `multiple_symbols_keep_compact`. Since `same_container` packing combines *different* methods by definition, naive packing would silently drop every non-primary member's surrounding source, and the coarse `contextQuality` grade would not register it. This is the one place where relationship packing genuinely regresses today's behavior, and it is fixed by design section 4, not by rejection — rejecting every cross-symbol candidate would neuter the plan. -4. **Independent planner draws destroy attribution.** Four of Plan 102's six retry failures were cross-arm reconciliation against independently sampled Stage-5 plans. Pinning the plan per pair removes the cause rather than removing the check. +Against it stands packet-level recall: a hunk sharing a packet may be reviewed less attentively. **Nobody has measured that.** Plan 102 tried and its fixture never produced the treatment; the first draft of this plan proposed to try again with different machinery. It is one curve, it is the same question under every predicate, and it is the only thing standing between this mechanism and rollout. The paid phase measures it directly and measures nothing else. ## Current State -- `hunkFirstGroups()`/`canJoinGroup()` in `src/pipeline/packet-builder.ts` join same-file hunks by identical enclosing symbol or `NEARBY_GAP_LINES = 30` proximity. The Plan 102 experiment was fully reverted; `src`, `scripts`, `tests`, and `evals` match the pre-experiment baseline. -- `buildHunkRelationshipGraph()` builds `same_symbol` (always `strong`), `planner_hint` (`strong` for symbol hints, `medium` for file hints), and `symbol_mention` (`strong` when the mention has an enclosing symbol, else `medium`) edges, capped at `MAX_RELATIONSHIP_EDGES_PER_HUNK = 8` per hunk, dropped on overflow in insertion order. -- `addSymbolMentionEdges()` guards ambiguity (`sourceIdentities.size > 1` → skip). `addPlannerHintEdges()` has no equivalent guard, so a repeated method name can produce spurious `strong` edges. -- `buildRelatedChangedContext()` consumes every non-`weak` edge whose target is outside the packet, capped at `MAX_RELATED_CONTEXTS_PER_PACKET = 3`. Any new `medium` edge source would therefore change flag-off context, attention notes, repository reads, and — through `hasRelatedChangedContext` in the `light`/mechanical branch — profiles. -- `buildPacket()` sets packet coverage to `maxCoverage(members)` and derives `reviewProfile` after related-context filtering, so absorbing a strong edge can currently lower the derived profile. -- `applyCoverageEscalations()` runs *after* packet construction, early-returns only when the packet is already `deep`, and stamps `coverageEscalation`. `CoverageEscalation["rule"]` is the single-member union `"test_coverage_delta"`, so a packed `normal` packet it escalates would overwrite any packing provenance stored there. -- `SymbolInfo` carries `ownerType` (Go method receivers, TypeScript classes, Rust `impl`/trait owners, Python classes, Solidity contracts) and `packageName` (Go). `generic-adapter.ts` resolves neither. `HunkSymbolFacts` — the per-hunk record the packet builder groups on — carries neither. `renderSymbol()` prefixes methods as `Owner.Method` inside `enclosingSymbol`, so container identity is only recoverable today by string-splitting a display string. `packet-context.ts:154` already resolves an owner structurally by name and kind, which is the precedent for a structural container key. -- `packageName` is constant within a Go file and carries no within-file affinity signal; it is deliberately not plumbed. -- `pnpm test -- ` does **not** filter: it runs the full suite and silently drops the argument (verified — `pnpm test -- tests/config-loader.test.ts` executed all 761 tests). Plan 102's focused verify commands were therefore never focused. This plan uses `pnpm exec vitest run `. -- Plan 102's report script redacted every failure `message` to `sha256::` through `safeFailureOutput()`, leaving its preserved evidence unreadable. -- There is no seam to supply a pre-recorded Stage-5 plan to a review. `runReview()` always calls `runPlanner()`. +- `hunkFirstGroups()`/`canJoinGroup()` join same-file hunks by identical enclosing symbol or `NEARBY_GAP_LINES = 30` proximity. Baseline restored after Plan 102's teardown. +- `buildHunkRelationshipGraph()` produces `same_symbol` (`strong`), `planner_hint` (`strong` for symbol hints, `medium` for file hints), and `symbol_mention` (`strong` with an enclosing symbol, else `medium`) edges, capped at `MAX_RELATIONSHIP_EDGES_PER_HUNK = 8`, dropped on overflow in insertion order. +- `addSymbolMentionEdges()` guards ambiguity (`sourceIdentities.size > 1` → skip); `addPlannerHintEdges()` has no equivalent guard. +- `buildRelatedChangedContext()` consumes every non-`weak` edge whose target is outside the packet, capped at `MAX_RELATED_CONTEXTS_PER_PACKET = 3`. Any new `medium` edge source added to the shared graph would change flag-off context, notes, repository reads, and profiles. +- `buildPacket()` derives `reviewProfile` after related-context filtering, so absorbing a strong edge can lower the derived profile — the reason Plan 102's floor exists. +- `readEnclosingSymbolSource()` reads one symbol for the whole packet, and `computeSymbolContextBudget()` selects `multiple_symbols_keep_compact` when `singlePrimarySymbol` is false. Packing distinct symbols therefore drops non-primary members' surrounding source, and the coarse `contextQuality` grade does not register it. +- `SymbolInfo` carries `ownerType` (Go receivers, TypeScript classes, Rust `impl`/trait owners, Python classes, Solidity contracts); `generic-adapter.ts` does not. `HunkSymbolFacts` carries neither it nor `packageName`. `packet-context.ts:154` already resolves an owner structurally by name and kind. +- `pnpm test -- ` does **not** filter — it runs the full suite and drops the argument (verified). This plan uses `pnpm exec vitest run `. +- Plan 102's implementation is recoverable at `5551547` and its report harness at `7ebfd2f`, including `safeFailureOutput()`, which hashed every failure message and is not ported. +- There is no seam to supply a pre-recorded Stage-5 plan to a review. ## Goal -1. Make the packing predicate semantic: atoms combine when repository structure says they are related, regardless of how the planner graded them. -2. Prove deterministically and for free that the predicate produces a **treatment surface**, including packs that cross coverage grades, before authorizing a paid call. -3. Answer one paid question with attribution: given an identical Stage-5 plan, does a reviewer shown related hunks together find a cross-hunk inconsistency that a reviewer shown them separately misses? -4. Keep every Plan 102 invariant that passed, and add per-member coverage, context-quality, and connectivity invariants that Plan 102 did not need. +1. Land Plan 102's packing mechanism unchanged in predicate and caps, so the measured 21–27% packet reduction is available. +2. Use relatedness to improve packet composition at zero cost to packet count. +3. Close the gap Plan 102 left open in symbol context, per-member quality, and note pressure, so bigger packets are not quietly worse packets. +4. Measure recall as a function of packet size — the single curve that decides whether packing ships. 5. End in exactly one product path with no surviving flag, under either outcome. ## Design -### 1. Structural container identity - -Add `ownerType?: string` and `ownerKey?: string` to `HunkSymbolFacts`, populated in `src/repo/symbol-extraction.ts` from the primary changed symbol. `ownerType` is the display name. `ownerKey` is the **structural** identity: `${path}:${ownerName}:${ownerStartLine}-${ownerEndLine}`, resolved by locating the owning symbol in the file's extracted symbols by name and container kind — the lookup `packet-context.ts:154` already performs. Where the owner cannot be resolved structurally, `ownerKey` is absent and the hunk has no container affinity. Duplicate or nested same-named classes therefore never collide. - -No adapter changes are required: Go resolves the owner from the method receiver, TypeScript from the enclosing class, Rust from the `impl`/trait owner, Python from the enclosing class, and Solidity from the owning contract. `generic-adapter.ts` does not, so fallback-path files simply have no container affinity — correct, not a gap. - -### 2. A separate packing-affinity view - -`same_container` is **not** added to `HunkRelationshipGraph`. Adding any new `medium` source there would change flag-off excerpts, attention notes, `readSymbol` calls, profiles through `hasRelatedChangedContext`, and — through `MAX_RELATIONSHIP_EDGES_PER_HUNK` overflow — which existing edges survive. +### 1. Atoms and partitions — Plan 102, unchanged -Instead, build a `PackingAffinity` view per file, computed only when the flag is on: +Wrap `hunkFirstGroups()` output as atoms carrying ordered hunks, hunk count, `combinedPatchChars()` size, first source position, effective coverage, routed lenses, standalone review profile and context quality, and a stable ID from ordered hunk IDs. Partition each file's atoms by `(effectiveCoverage, normalizedPlannerLensSignature)`. Direct `whole-file` and content-probed `file-diff` returns bypass packing. -- **derived** edges: copied from the relationship graph — `same_symbol`, `symbol_mention` with `strength === "strong"`, and `planner_hint` with `strength === "strong"` **and** unique resolution (the hint's normalized symbol key matches exactly one changed symbol identity in the file, mirroring `addSymbolMentionEdges()`'s existing ambiguity guard); -- **structural** edges: `same_container`, emitted for hunk pairs in one file sharing a non-empty `ownerKey`; -- no per-hunk edge cap, because this view is not rendered into any prompt and cannot evict prompt context. +### 2. Structural container identity -The report attributes every pack to `derived` versus `structural` edges separately, so `same_container`'s contribution is measurable rather than assumed. +Add `ownerType?: string` and `ownerKey?: string` to `HunkSymbolFacts` in `src/repo/symbol-extraction.ts`. `ownerKey` is structural — `${path}:${ownerName}:${ownerStartLine}-${ownerEndLine}`, resolved through the owner lookup `packet-context.ts:154` already performs — so duplicate or nested same-named containers never collide. Absent where the owner cannot be resolved structurally. `packageName` is constant within a file and is not plumbed. -### 3. Connectivity-preserving greedy expansion +### 3. A separate packing-affinity view -Within one file, over hunk-first atoms only: +`same_container` is **not** added to `HunkRelationshipGraph`; doing so would change flag-off excerpts, notes, repository reads, profiles, and — through `MAX_RELATIONSHIP_EDGES_PER_HUNK` overflow — which existing edges survive. -1. Order atoms by first source position. -2. Take the source-earliest unassigned atom as a packet seed. -3. Repeatedly admit the source-earliest unassigned atom that (a) has an eligible edge to **at least one atom already in the packet** and (b) keeps total hunks `<= MAX_HUNKS_PER_PACKET` and total patch characters `<= MAX_PATCH_CHARS` via `combinedPatchChars()`. -4. Stop when no candidate qualifies; flush and seed the next packet from the remaining atoms. -5. Order packets by first source position; render hunks in source order within each packet. +Build a `PackingAffinity` view per file, only when packing is on: -Every resulting packet is connected by construction, which a source-order chunk of a connected component is not — a chain `A—B—C` appearing in source order `A, C, B` would chunk into a disconnected `{A, C}`. Expansion is also not contiguity-bound: `A1, X, A2` with an `A1—A2` edge packs `A1` and `A2` and leaves `X` alone, which is precisely the separated-sibling case this plan exists for. +- **derived** edges copied from the relationship graph: `same_symbol`, `strong` `symbol_mention`, and `strong` `planner_hint` that resolves uniquely (its normalized symbol key matches exactly one changed symbol identity in the file, mirroring `addSymbolMentionEdges()`'s existing guard); +- **structural** edges: `same_container`, for hunk pairs sharing a non-empty `ownerKey`; +- no per-hunk cap, because this view is never rendered and cannot evict prompt context. -Atoms with no eligible edge remain their own packets, exactly as today. Direct `whole-file` and content-probed `file-diff` returns from `groupHunks()` bypass packing entirely. +### 4. Relationship-ordered greedy fill -### 4. Symbol context follows membership +Within a partition, atoms sorted by first source position: -For a packed packet whose member atoms have more than one distinct primary symbol identity, read enclosing-symbol source for **every** member atom's primary symbol rather than the top-ranked one alone: +1. Seed with the source-earliest unassigned atom. +2. Admit the **source-earliest unassigned atom connected by an affinity edge to any current member** that fits `MAX_HUNKS_PER_PACKET` and `MAX_PATCH_CHARS`. +3. If no related atom fits, admit the source-earliest unassigned atom in the partition that fits — Plan 102's rule. +4. Flush when nothing fits; seed the next packet. +5. Order packets by first source position. **Render every packet's hunks in source order**, independent of admission order. -- allocate each member `floor(symbolContextBudget / memberSymbolCount)` characters, computed by the existing `computeSymbolContextBudget()` path per member; -- preserve the existing adaptive/sliced selection per member, and render members in source order; -- record `contextQuality` per member atom in packet telemetry, alongside the packet-level grade. +With no affinity edges this is byte-for-byte Plan 102's behavior, which makes the deterministic comparison exact: the replay reports how many packets differ in composition from the Plan 102 ordering, and packet *count* should differ by at most a small bin-packing residual. -A candidate is rejected when any member atom's context quality would fall below its standalone quality *and* below `sliced` — that is, to `outline_only` or `path_only`. Falling from `full` to `sliced` under a shared budget is expected and reported, not fatal. +### 5. Symbol context follows membership -This is the one place relationship packing regresses today's behavior if left alone, and it is why the throughput case is secondary: packed packets carry more context, not less. +For a packed packet whose members hold more than one distinct primary symbol identity, read enclosing-symbol source for **every** member's primary symbol: allocate each `floor(symbolContextBudget / memberSymbolCount)`, preserve the existing adaptive/sliced selection per member, render in source order, and record `contextQuality` per member alongside the packet grade. -### 5. Transactional candidate evaluation +This is the one place Plan 102's mechanism is genuinely improved rather than reused — it shipped a packer that could drop non-primary members' surrounding source without registering it. -Every candidate is dry-built against a cloned relationship accumulator and a suppressed telemetry sink. Nothing is written to the real graph, artifacts, or event stream until a candidate is committed. A candidate is abandoned — leaving its atoms as separate packets — when the combined packet would: +### 6. Transactional candidate evaluation -- drop a lens any member atom routed standalone; -- newly omit a planner focus note belonging to a hunk whose `reviewPriority` is `high` or `critical`; -- reduce any member atom's context quality below `sliced`; -- lower any member's effective coverage, or lower the effective profile below the maximum standalone member profile. +Every candidate is dry-built against a cloned relationship accumulator and a suppressed telemetry sink; nothing reaches the real graph, artifacts, or event stream until commit. A candidate is abandoned — leaving its atoms separate — when the combined packet would: -Every abandonment is recorded with its reason code and the atoms involved. +- drop a lens any member routed standalone; +- newly omit a planner focus note belonging to a `high`/`critical` hunk; +- reduce any member's context quality below `sliced`; +- lower the effective profile below the maximum standalone member profile. -### 6. Per-member coverage promotion records +Each abandonment is recorded with a reason code and the atoms involved. -Packet coverage remains `maxCoverage(members)`. Add to `ReviewPacket`: +### 7. Profile floor -```ts -memberCoveragePromotions?: Array<{ - atomId: string; - hunkIds: string[]; - from: CoverageLevel; - to: CoverageLevel; - edgeSource: "same_symbol" | "symbol_mention" | "planner_hint" | "same_container"; - edgeStrength: HunkRelationshipStrength; -}>; -``` - -ordered by first source position. `CoverageEscalation` and its single `"test_coverage_delta"` rule are left untouched, so Plan 92's packet-level escalation provenance and this plan's member-level promotion record coexist without either overwriting the other. Demotion of any member is an invariant failure, not a reportable event. - -### 7. Profile floor and dispatch rank +Plan 102's floor, retained verbatim: `effectiveReviewProfile = max(derivedPackedProfile, max(standalone member profiles))` over `{ simple: 0, standard: 1, investigate: 2 }`, never lexical comparison. `dispatchRank` is recomputed with the unchanged `packetDispatchRank(filePath, facts, combinedChangedLines)` formula. -Plan 102's floor is retained verbatim: `effectiveReviewProfile = max(derivedPackedProfile, max(standalone member profiles))` over the explicit rank map `{ simple: 0, standard: 1, investigate: 2 }`, never lexical comparison. Standalone profiles are computed through the same Stage-6 path the atom receives with packing off, against the scratch accumulator. - -`dispatchRank` is recomputed with the unchanged `packetDispatchRank(filePath, facts, combinedChangedLines)` formula. Scheduling movement is reported, never suppressed. - -### 8. Configuration and the pinned-plan seam +### 8. Configuration ```yaml review: - packRelatedHunks: false + packCompatibleAtoms: false # mechanism gate + packMaxHunks: 5 # experiment-only; the dilution curve varies this ``` -One boolean in raw and resolved schemas, `CodegenieConfig`, defaults, config-source tracking, repo-safe filtering, the strict eval-case `review` schema, and `applyCaseReviewConfig()`. Deleted in step 10 under either outcome. - -Separately, add an **eval-only** pinned-plan seam so both arms of a paid pair consume one Stage-5 draw: +Both in raw and resolved schemas, `CodegenieConfig`, defaults, config-source tracking, repo-safe filtering, the strict eval-case `review` schema, and `applyCaseReviewConfig()`. `packMaxHunks` exists solely to run the recall curve and never exceeds `MAX_HUNKS_PER_PACKET` in shipped behavior. Both are deleted in step 9. -- a planner-draw mode that runs Stage 1–5 and writes the resulting `ReviewPlan` plus its SHA-256; -- an eval-case field (`review.pinnedPlanDir`) that supplies execution *k* of a repeat with plan *k*, validated by hash and by exact `hunkId` membership against the current diff, failing closed on any mismatch; -- no user-facing config field, no `codegenie.toml` surface, and no reachable path in normal review. - -This is eval-harness capability, not product surface. At teardown it survives only if it is independently useful to other eval work; otherwise it goes with the flag. +Separately, an **eval-only** pinned-plan seam: a planner-draw mode writing a `ReviewPlan` plus its SHA-256, and an eval-case field (`review.pinnedPlanDir`) supplying execution *k* with plan *k*, validated by hash and exact `hunkId` membership, failing closed on mismatch. No user-facing config path. It survives teardown only if independently useful to other eval work. ## Validation Strategy -### A. Free deterministic replay (no model calls) - -Port `scripts/packet-packing-report.ts` from `7ebfd2f` down to `replay` plus `treatment`, with tests in `tests/packet-packing-report.test.ts`. +### A. Free deterministic replay -**Failure records are structured and templated**, not hashed and not raw. Each failure carries a closed-set `code`, structured fields (`runId`, `packetId`, `atomIds`, `hunkIds`, counts, enum reasons), and a `message` rendered from a template over those fields. Raw exception text and repository source are never interpolated. Plan 102's `safeFailureOutput()` hashing is not ported; its credential stripping is. +Port `scripts/packet-packing-report.ts` from `7ebfd2f` down to `replay` plus `treatment`. Failure records are **structured and templated** — a closed-set `code`, structured fields, and a message rendered from a template. No raw exception text, no repository source, no hashing. -Run over the four retained runs. Assert per run: +Run over the four retained runs and assert: -- every reviewable hunk appears exactly once, in source order; -- no atom is split or internally reordered; -- **every packed packet is connected** under the affinity view; -- five-hunk, 12K-patch, six-lens, and three-note caps hold; file boundaries hold; -- no planner-requested lens dropped, no `high`/`critical` focus note newly omitted, no member context quality below `sliced` — or the candidate was abandoned and recorded; -- no effective profile or base budget below the maximum standalone member value; -- no member coverage demotion; every promotion carries a member record with a causing edge; +- packet counts reproduce Plan 102's frozen result (96→75, 93→68, 109→85) within a documented bin-packing residual of at most 2 packets per run, attributable solely to admission order; +- every reviewable hunk appears exactly once, **in source order**; +- no atom split or reordered; no new coverage promotion; caps, file, lens, and note bounds hold; +- no lens dropped, no `high`/`critical` focus note newly omitted, no member context below `sliced` — or the candidate was abandoned and recorded; +- no effective profile or budget below the maximum standalone member value; - every dispatch rank equals the existing formula; -- flag off is artifact-identical to current behavior, including the relationship-graph artifact, which proves the affinity view never wrote back. +- flag off is artifact-identical to current behavior, **including `hunk-relationships.json`**, proving the affinity view never wrote back. -Report per run: packets off→on, atoms per packet, promoted members by edge source, abandoned candidates by reason code, packing edges split `derived` versus `structural`, per-member context-quality distribution, attention notes dropped, excerpts internalized, and scheduling movement. +Report per run: packets off→on, composition changes versus Plan 102 ordering, affinity edges split `derived`/`structural`, abandoned candidates by reason, per-member context-quality distribution, notes dropped, excerpts internalized, and scheduling movement. -**Pre-registered numeric gates**, counted on the two *distinct* diffs (`dca8d870`, `81f806a6`) and reported for all four rows: +**Pre-registered gates**, counted on the two distinct diffs: | Gate | Threshold | | --- | --- | -| Treatment surface — packets combining atoms Plan 102's `(coverage, lensSignature)` predicate would separate | `>= 5` per distinct diff | -| Cross-coverage packs — packed packets containing at least one member promotion | `>= 3` per distinct diff | -| `same_container` aggressiveness — multi-atom packets with structural edges enabled, versus derived-only | `<= 2.0x` | -| Promoted members as a share of reviewable hunks | `<= 25%` per run | +| Packet reduction | `>= 20%` per distinct diff | +| Deviation from Plan 102's frozen packet count | `<= 2` packets per run | +| Packets whose composition changed under relationship ordering | `>= 2` per distinct diff | | Members whose context quality falls below `sliced` | `0` | +| New coverage promotions | `0` | -Missing the first two means the predicate does not reach materially more code than Plan 102's did, or does not do the new thing it exists to do — stop here, for free. Exceeding the third or fourth means demoting `same_container` to context-only and re-running with derived edges alone before continuing. - -Also report what Plan 102's compatibility tier would add on top, as a diagnostic for a possible later throughput plan. Do not implement that layer here. - -### B. One paid question, isolated by a pinned plan - -**Fixture.** One family, `container-consistency`, under `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/relationship-packing/recall/`: several methods on one Go receiver type, four applying a shared guard and one omitting it, separated by more than `NEARBY_GAP_LINES` so today's grouper yields one atom each. The same repository also carries an **unrelated local boundary bug** in a free function, which serves as the control. - -Two expectations: +Missing the third gate means relationship ordering changes nothing measurable — drop `ownerKey`, `same_container`, and the affinity view, and ship Plan 102's ordering unchanged. That is a cheap, free deletion, not a plan failure. -- `cross-hunk` — the missing guard. Invisible without seeing siblings. Packing should help. -- `local-control` — the boundary bug. Visible in isolation. Packing must not hurt. +### B. The recall-versus-packet-size curve (paid) -Fixture rules learned from Plan 102: the target must be a hunk the planner grades **ordinarily** on its own (if it grades `deep` alone, the fixture is wrong); target and siblings must be related by a real edge, verified by the no-model treatment check before launch; and expectations constrain path, line range, and failure mode while **accepting any plausible category**. Plan 102 recorded three executions that found its bug and scored `partial-match` solely because the expectation demanded `correctness` and the reviewer said `security`. +This is the only paid question. It is not about which predicate to use; it is whether review quality survives packing at all. -**Procedure.** Two cases, `container-a.yml` (packing off) and `container-b.yml` (packing on), identical except that field. For each of 6 repeats: draw one Stage-5 plan, hash it, and run both arms pinned to it. Six planner draws total, so planner variance is still sampled *across* pairs while packing is the only difference *within* a pair. With the plan pinned, cross-arm atom and hunk reconciliation is exact and **required** — the check Plan 102 could not support is now sound. +**Fixture.** One Go file under `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/recall/` with about ten separated single-hunk changes — far enough apart that today's grouper yields one atom each — sharing coverage and lenses so they all land in one partition. One hunk carries a locally detectable bug; the rest are safe. Expectations constrain path, line range, and failure mode, and **accept any plausible category**: Plan 102 recorded three executions that found its bug and scored `partial-match` solely because the expectation demanded `correctness` and the reviewer said `security`. -Run the one-repeat treatment preflight first. Arm B's target packet must contain at least two source atoms. If it does not, repair the fixture and rerun — a preflight treatment miss is a fixture defect and explicitly does **not** trigger teardown. +**Arms.** Identical cases differing only in `packMaxHunks`: 1 (one atom per packet — today's behavior), 3, and 5 (the shipping cap). `repeat: 6`, cache off. Each repeat draws one Stage-5 plan, hashed, consumed by all three arms, so packet size is the only within-repeat difference while planner variance is still sampled across repeats. -**Pre-registered decision table** over 6 pairs, where `Ax`/`Bx` are hits on `cross-hunk` and `Ac`/`Bc` on `local-control`: +**Pre-registered decision table**, where `R1`, `R3`, `R5` are hits out of 6: | Condition | Outcome | | --- | --- | -| `Ax >= 5` | **Void — no headroom.** Fixture too easy; redesign, do not count as evidence. | -| `Ax == 0 and Bx == 0` | **Void — no signal.** Bug undetectable at this model/config; redesign. | -| `Bx - Ax >= 3` and `Bc >= Ac - 1` | **Pass.** Proceed to collateral and capacity confirmation. | -| `1 <= Bx - Ax <= 2` | **Inconclusive.** Record; do not ship; a larger `n` is a separate decision. | -| `Bx <= Ax` or `Bc < Ac - 1` | **Fail.** Take the failure branch. | +| `R1 <= 1` | **Void.** The bug is not reliably findable even unpacked; redesign the fixture. | +| `R5 >= R1 - 1` | **Pass.** Packing is non-inferior at the shipping cap. | +| `R5 <= R1 - 2`, or `R1 > R3 > R5` | **Fail.** Recall degrades with packet size; do not ship packing at any predicate. | +| otherwise | **Inconclusive.** Record; do not ship. | -Requiring `+3` of 6 is a strong-effect-only test. It cannot detect a small improvement, and the reconciliation note must say so rather than reporting a null as safety. +This is a non-inferiority test, and deliberately so: the *benefit* — roughly 19 more reviewed hunks per run — is already measured deterministically, so packing only has to avoid harming per-packet recall. The first draft demanded superiority because it was claiming a quality benefit; this plan claims a coverage benefit and a quality guard, which is the honest framing. -### C. Collateral and capacity, inside this plan +Also report tokens, model-service time, and cost per reviewed hunk per arm — the efficiency side of the same curve. -A pass on one Go fixture does not license unconditional cross-language behavior. Before shipping, and within this plan: +Relationship ordering gets **no paid arm.** It changes composition in a handful of packets per run, an effect far too small to resolve at `n = 6`. It cannot reduce packet count or violate any invariant, it is validated deterministically in phase A, and it is adopted or dropped on that basis alone. Do not spend paid budget on an effect you cannot measure. -1. Run the complete `evals/fixtures/` suite flag-off and flag-on at `repeat: 1` with the fake provider. Every declared expectation, packet invariant, and config parse must match. -2. Run the complete `evals/skill-semantics/` suite (TypeScript, Python, Solidity) flag-off and flag-on at `repeat: 1`. Cross-language container semantics differ — Solidity contracts and Python classes are much broader containers than a Go receiver — so this is where `same_container` over-packing would surface. An A-pass/B-fail difference is a stop-for-investigation; rerun only the affected case at `repeat: 3` before deciding variance versus regression. -3. Run one production-shaped capacity pair on the pinned `trails-api` diff at concurrency 6, flag-off and flag-on, and report reviewed-hunk set, packet count, model-service time, tokens, and cost per reviewed hunk. This is a capacity and safety observation, not an economics gate — this plan does not claim throughput and must not be blocked on producing it. +### C. Collateral and capacity -These stay in this plan so the decision and the teardown happen together. Deferring them would leave `packRelatedHunks` alive across plan boundaries, which is the dark path Plan 102 exists to forbid. +1. `evals/fixtures/` flag-off and flag-on at `repeat: 1`, fake provider. Every expectation, packet invariant, and config parse must match. +2. `evals/skill-semantics/` (TypeScript, Python, Solidity) flag-off and flag-on at `repeat: 1`. An A-pass/B-fail difference is stop-for-investigation; rerun only that case at `repeat: 3`. +3. One production-shaped capacity pair on the pinned `trails-api` diff at concurrency 6, flag-off and flag-on, reporting reviewed-hunk set, packet count, model-service time, tokens, and cost per reviewed hunk. Its purpose is to confirm the run-level coverage gain the deterministic replay predicts. ### Cost discipline -No paid model call begins before `approvedValidationCostUSD` is recorded in this plan's reconciliation note; the owner supplies the amount and there is no default. - -Projection from Plan 102's measured $0.56–$0.81 per execution on comparable Go fixtures: +No paid call before `approvedValidationCostUSD` is recorded here; the owner supplies the amount. | Phase | Executions | Projection | | --- | ---: | ---: | -| Treatment preflight (1 pair + 1 plan draw) | 3 | ~$2 | -| Paired recall, 6 pairs | 18 | ~$14 | -| `evals/fixtures/` collateral (fake provider) | — | $0 | -| `evals/skill-semantics/` collateral, both arms | 6 | ~$5 | +| Dilution preflight (1 plan draw + 3 arms) | 4 | ~$3 | +| Dilution curve, 6 repeats × 3 arms | 18 | ~$14 | +| `evals/fixtures/` (fake provider) | — | $0 | +| `evals/skill-semantics/`, both arms | 6 | ~$5 | | Production capacity pair | 2 | ~$50 | | Contingency 25% | — | ~$18 | | **Reservation** | | **~$90** | -Record actual and projected spend after every phase and stop if actual plus projection exceeds the ceiling. The production pair is the only expensive item; it runs last and only after a pass. +The production pair dominates and runs last, only after a pass. If the ceiling is tight, phases 1–2 alone cost ~$17 and decide whether packing ships. ## In-Scope Files -- `src/repo/symbol-extraction.ts` — `ownerType` and structural `ownerKey` on `HunkSymbolFacts`. -- `src/pipeline/packet-builder.ts` — affinity view, greedy expansion, transactional candidates, multi-member symbol context, member promotion records, profile floor, telemetry. -- `src/pipeline/review-runner.ts` — the eval-only pinned-plan seam. -- `src/types.ts` — `ownerType`/`ownerKey`, `memberCoveragePromotions`, per-member context-quality telemetry, the temporary config field. -- `src/config/schema.ts`, `src/config/config-loader.ts`, `src/evals/eval-runner.ts` — flag plumbing and `pinnedPlanDir`; both deleted or re-justified in step 10. -- `scripts/packet-packing-report.ts`, `tests/packet-packing-report.test.ts` — `replay` and `treatment` modes with templated failure records. +- `src/repo/symbol-extraction.ts` — `ownerType`, structural `ownerKey`. +- `src/pipeline/packet-builder.ts` — atoms, partitions, affinity view, relationship-ordered fill, multi-member symbol context, transactional candidates, profile floor, telemetry. +- `src/pipeline/review-runner.ts` — eval-only pinned-plan seam. +- `src/types.ts` — `ownerType`/`ownerKey`, per-member context-quality telemetry, both temporary config fields. +- `src/config/schema.ts`, `src/config/config-loader.ts`, `src/evals/eval-runner.ts` — flag plumbing and `pinnedPlanDir`. +- `scripts/packet-packing-report.ts`, `tests/packet-packing-report.test.ts` — `replay` and `treatment` with templated failure records. - `tests/pipeline-phase5.test.ts`, `tests/config-loader.test.ts`, `tests/evals.test.ts`, and the symbol-extraction suite. -- `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/relationship-packing/` — fixture, cases, pinned plans, immutable logs, reports. +- `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/` — fixture, cases, pinned plans, immutable logs, reports. - `specs/plans/README.md` and affected `specs/project/` docs. ## Non-Goals -- Cross-file packing. File identity remains the context and anchor boundary. -- Raising the five-hunk, 12K-patch, six-lens, or three-note caps. -- Reintroducing Plan 102's `atom-scaled` budget, or any tool-budget change. -- Layering Plan 102's coverage/lens compatibility packing on top. The replay reports what it would add; a separate plan may act on it. -- Claiming a throughput or economics win. Packed packets carry more symbol context and may cost more; the capacity run observes this rather than gating on it. +- **Raising `MAX_HUNKS_PER_PACKET` or `MAX_PATCH_CHARS`.** The sweep shows 5 captures ~80% of achievable compression; 10 adds ~5 points before saturating and enlarges untested recall risk. `packMaxHunks` exists only to measure the curve and never ships above 5. +- **Relationship-required packing as a membership predicate.** Measured at 4.2% versus 21.9%; retired by evidence. +- **Cross-coverage packing and any coverage-promotion machinery.** 2 of 13 candidate pairs, zero of them structural. +- **Reordering the stage flow.** The planner grades related siblings consistently in 132 of 134 measured pairs; there is nothing for a group-aware dossier to fix. +- Cross-file packing — the packet schema is file-scoped, and cross-file relatedness is already served by related-context excerpts. +- Tool-budget changes of any kind, including Plan 102's retired `atom-scaled` arm. - Changing prompt templates, attention-note ordering, `canJoinGroup()` semantics, Plan 100's dispatch policy, or Plan 92's escalation rules. -- Exposing `packRelatedHunks` or `pinnedPlanDir` as user-facing configuration, or leaving either dark after step 10. -- Treating `packageName` as an affinity signal. +- Exposing either config field as user-facing surface, or leaving either dark after step 9. ## Implementation Steps 1. Reconcile the drift and working-tree checks. Add `ownerType` and structural `ownerKey` to `HunkSymbolFacts`. - **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "container identity"` → exit 0; Go receivers, TypeScript classes, Rust `impl` blocks, Python classes, and Solidity contracts resolve; free functions and generic-adapter files resolve none; two same-named classes in one file produce distinct `ownerKey` values. -2. Add the dark `review.packRelatedHunks` field end-to-end, including strict eval schema, `applyCaseReviewConfig()`, defaults, repo-safe filtering, and config-source tracking. - - **Verify:** `pnpm exec vitest run tests/config-loader.test.ts tests/evals.test.ts` → exit 0; default `false`, eval override applies, winning source identified. -3. Build the `PackingAffinity` view, including the `planner_hint` unique-resolution guard. Prove it never mutates the relationship graph. + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "container identity"` → exit 0; all five real adapters resolve owners, generic-adapter files resolve none, and two same-named classes in one file produce distinct keys. +2. Add both dark config fields end-to-end, including strict eval schema, `applyCaseReviewConfig()`, defaults, repo-safe filtering, and source tracking. - **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "packing affinity"` → exit 0; `hunk-relationships.json`, related-context excerpts, attention notes, repository reads, and profiles are byte-identical with the flag on and packing suppressed; ambiguous repeated symbol names produce no structural or hint edge; the per-hunk edge cap is unaffected. -4. Wrap `hunkFirstGroups()` output as atoms with standalone coverage, routed lenses, profile, and context quality captured through the scratch path. Add flag-off golden parity before any packing runs. + **Verify:** `pnpm exec vitest run tests/config-loader.test.ts tests/evals.test.ts` → exit 0; defaults `false`/`5`, overrides apply, `packMaxHunks` above 5 is rejected. +3. Wrap `hunkFirstGroups()` output as atoms with standalone coverage, lenses, profile, and context quality captured through the scratch path; add flag-off golden parity. **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "packet atom|flag-off parity|whole-file bypass"` → exit 0. -5. Implement connectivity-preserving greedy expansion. +4. Implement Plan 102 partitions and greedy fill, with no relationship ordering yet, and prove it reproduces Plan 102's frozen counts. - **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "greedy expansion"` → exit 0, covering: the `A—B—C` chain in source order `A, C, B` never producing a disconnected packet; `A1, X, A2` packing across the unrelated `X`; cap-bound expansion stopping without splitting a packet into disconnected parts; interleaved components; unrelated atoms staying separate; hunk bijection and source order. -6. Implement multi-member symbol context and per-member context-quality recording. + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "compatible partition"` → exit 0; hunk bijection, source-order rendering, cap splits, interleaved partitions, degradation merge, and the exact dispatch-rank formula. +5. Build the `PackingAffinity` view and add relationship-ordered admission. - **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "member symbol context"` → exit 0; a two-symbol packet carries both symbols' source under a shared budget; `full`→`sliced` degradation is reported and allowed; a member falling to `outline_only` aborts the candidate. -7. Implement transactional candidate evaluation, member promotion records, and the profile floor. + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "packing affinity|relationship ordering"` → exit 0; the relationship graph and all flag-off artifacts are unchanged; ambiguous repeated symbol names produce no edge; with no edges the fill is identical to step 4; with edges, composition changes while hunks still render in source order. +6. Implement multi-member symbol context and per-member context-quality recording. - **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "candidate rejection|member promotion|profile floor"` → exit 0; an abandoned candidate leaves no telemetry, graph mutation, or artifact residue; lens-drop, `high`-priority focus-note-drop, and context-collapse candidates all abort; promotions are recorded per member with the causing edge; `coverageEscalation` is untouched and a subsequent Plan 92 escalation does not erase promotion records; demotion fails closed. -8. Add the eval-only pinned-plan seam and its validation. + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "member symbol context"` → exit 0; a two-symbol packet carries both symbols; `full`→`sliced` is reported and allowed; a member falling to `outline_only` aborts the candidate. +7. Implement transactional candidate evaluation and the profile floor. - **Verify:** `pnpm exec vitest run tests/evals.test.ts -t "pinned plan"` → exit 0; a pinned plan reproduces identical Stage-6 inputs across two runs, a hash mismatch fails closed, a `hunkId` set mismatch fails closed, and no user-facing config path can reach it. -9. Port the report script and run the four-run replay. Record the full table and every pre-registered gate in this plan's reconciliation note. + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "candidate rejection|profile floor"` → exit 0; abandoned candidates leave no residue; lens-drop, `high`-priority note-drop, and context-collapse candidates abort; the floor holds when an absorbed strong edge would derive lower. +8. Add the pinned-plan seam, port the report script, and run the four-run replay. Record the table and every gate here. - **Verify:** `pnpm exec vitest run tests/packet-packing-report.test.ts` → exit 0 with templated-message and fail-closed regressions; then the replay command exits 0 with `modelCallsObserved: 0` on every row, all invariants green, and every numeric gate in section A satisfied on both distinct diffs. Otherwise stop; no paid phase is authorized. -10. Record the owner-approved ceiling, build the fixture, and run the paid programme in order: treatment preflight, 6 pinned pairs, `evals/fixtures/`, `evals/skill-semantics/`, then the production capacity pair. Apply the decision table verbatim. + **Verify:** `pnpm exec vitest run tests/packet-packing-report.test.ts tests/evals.test.ts -t "pinned plan"` → exit 0; the replay exits 0 with `modelCallsObserved: 0` on every row and every section-A gate satisfied. If the composition gate fails, delete the affinity view and container identity, re-run, and continue with Plan 102 ordering. +9. Record the ceiling, build the dilution fixture, and run: preflight, the 6×3 curve, `evals/fixtures/`, `evals/skill-semantics/`, then the capacity pair. Apply the decision table verbatim. - **Verify:** preflight target packet combines at least two atoms; cross-arm reconciliation is exact for every pair; the decision table yields Pass, Inconclusive, Fail, or Void with no discretion; collateral suites show no reproducible regression; cumulative spend stays within the ceiling. -11. Record the decision and tear down the scaffolding in a dedicated commit. + **Verify:** each repeat's three arms consumed one hash-verified plan; the decision table yields Pass, Fail, Void, or Inconclusive with no discretion; collateral shows no reproducible regression; spend stays within the ceiling. +10. Record the decision and tear down in a dedicated commit. - **Preserve evidence first:** copy every JSON report produced by completed phases into `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/relationship-packing/reports/` with a `manifest.sha256`, alongside the six pinned plans and their hashes. Mark unreached phases `not_run` with their stopping reason. + **Preserve evidence first:** copy every produced JSON report and the pinned plans into `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/reports/` with a `manifest.sha256`. Mark unreached phases `not_run` with their stopping reason. - **If Pass:** make packing unconditional; delete `review.packRelatedHunks` and the unpacked path with all schema, loader, and eval plumbing. Keep `ownerType`/`ownerKey`, the affinity view, greedy expansion, multi-member symbol context, member promotion records, the profile floor, and packing telemetry as unconditional behavior. Reduce the report script to a golden check against frozen packet shapes. Retain `pinnedPlanDir` only if it is independently useful to other eval work; otherwise delete it. + **If Pass:** make packing unconditional at cap 5; delete both config fields and the unpacked path. Keep atoms, partitions, the profile floor, multi-member symbol context, transactional rejection, and — if the composition gate passed — `ownerKey`, `same_container`, and the affinity view. Reduce the report script to a golden check against frozen packet shapes. - **If Fail, Inconclusive, or Void:** delete the packing pass, the flag and its plumbing, the affinity view, the atom wrapper, member promotion records, and the report script with its tests. `ownerType`/`ownerKey` and multi-member symbol context survive **only** if the replay showed them independently improving related-context selection or context quality with packing off; otherwise they go too. A Void outcome permits one fixture redesign and one rerun before the failure branch; a second Void takes the failure branch. Do not leave the feature dark. + **If Fail, Inconclusive, or Void:** delete the packing pass, both fields, the atom wrapper, the affinity view, and the report script with its tests. Multi-member symbol context survives only if the replay showed it improving context quality with packing off. A Void permits one fixture redesign and rerun; a second Void takes the failure branch. Do not leave the feature dark. - **Verify:** `rg -n "packRelatedHunks" src scripts tests evals` → exit 1; the evidence manifest verifies; the reconciliation note records the decision table result, actual spend, and the strong-effect-only limitation of `n = 6`. -12. Run the complete repository gate. + **Verify:** `rg -n "packCompatibleAtoms|packMaxHunks" src scripts tests evals` → exit 1; the manifest verifies; the note records the decision table result, actual spend, and the `n = 6` limitation. +11. Run the complete repository gate. **Verify:** `pnpm run check && pnpm test && pnpm build` → exit 0. ## Tests and Commands -Focused gates — note `pnpm exec vitest run`, because `pnpm test -- ` silently runs the whole suite: - ```bash pnpm run check pnpm exec vitest run tests/pipeline-phase5.test.ts tests/config-loader.test.ts tests/evals.test.ts tests/packet-packing-report.test.ts @@ -371,65 +380,62 @@ pnpm exec tsx scripts/packet-packing-report.ts replay \ --run /home/peter/Dev/0xsequence/trails-api/.codegenie/runs/20260724-150405-fe1548ae \ --run /home/peter/Dev/0xsequence/trails-api/.codegenie/runs/20260724-162739-81f806a6 \ --run /home/peter/Dev/0xsequence/trails-api/.codegenie/runs/20260724-184952-dca8d870 \ - --compare-predicate plan102 \ + --compare-ordering source \ --distinct-diffs \ - --output /tmp/plan103-relationship-shape.json + --output /tmp/plan103-packing-shape.json ``` -Expected: exit 0, four rows with duplicate diffs marked and counted once, zero model calls, zero invariant failures, all section-A gates satisfied, and a `plan102Predicate` diagnostic block. +Expected: exit 0, four rows with duplicate diffs marked and counted once, zero model calls, packet counts within 2 of Plan 102's frozen 75/68/85, and a composition-change block. -Paid preflight, then the pinned paired suite: +Dilution curve: ```bash pnpm dev eval \ - --eval-dir /home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/relationship-packing/recall \ + --eval-dir /home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/recall \ --no-cache pnpm exec tsx scripts/packet-packing-report.ts treatment \ - --logs /home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/relationship-packing/recall/logs \ + --logs /home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/recall/logs \ --cohort \ - --expected-repeats 1 \ + --expected-repeats 6 \ --require-pinned-plan \ - --output /tmp/plan103-treatment-preflight.json + --output /tmp/plan103-dilution-curve.json ``` -Note the `pnpm dev eval` spelling: under pnpm 11 a literal `--` is passed through to Commander and rejected before suite allocation. +Note the `pnpm dev eval` spelling: under pnpm 11 a literal `--` reaches Commander and is rejected before suite allocation. ## Acceptance Criteria -- Flag off produces byte-identical packet artifacts, IDs, order, profiles, context, budgets, **and relationship-graph artifacts** — proving the affinity view never wrote back. -- Flag on preserves every atom, assigns every reviewable hunk exactly once in source order, and respects file, five-hunk, and 12K boundaries. -- Every packed packet is connected under the affinity view; unrelated atoms are never combined; no packet is a source-order chunk of a component. -- No planner-requested lens dropped, no `high`/`critical` focus note newly omitted, no member context quality below `sliced`; candidates that would do any of these are abandoned transactionally and recorded. -- Every effective profile is at least the maximum standalone member profile; no member coverage falls; every promotion is recorded per member with its causing edge; `coverageEscalation` remains Plan 92's alone. -- Dispatch ranks use Plan 100's unchanged formula; scheduling movement is reported. -- The replay exits 0 with zero model calls and satisfies every pre-registered numeric gate on both distinct diffs, reporting `derived` versus `structural` edge attribution and per-member context quality. -- Every paid pair consumes one hash-verified Stage-5 plan; cross-arm atom and hunk reconciliation is exact; the decision table is applied without discretion. -- `evals/fixtures/` and `evals/skill-semantics/` show no reproducible collateral regression under the selected behavior. -- The production capacity pair reviews at least the baseline hunk set and its cost, tokens, and service time are reported without being claimed as a win. +- Flag off produces byte-identical packet artifacts, IDs, order, profiles, context, budgets, and `hunk-relationships.json`. +- Flag on reproduces Plan 102's packet counts within 2 per run, with every reviewable hunk appearing exactly once in source order, under unchanged caps and with zero coverage promotion. +- Relationship ordering changes composition, never render order, and never increases packet count beyond the documented bin-packing residual. +- No lens dropped, no `high`/`critical` focus note newly omitted, no member context below `sliced`; violating candidates are abandoned transactionally and recorded. +- Every effective profile is at least the maximum standalone member profile; dispatch ranks use Plan 100's unchanged formula. +- The replay exits 0 with zero model calls and satisfies every pre-registered gate on both distinct diffs. +- Every dilution repeat's three arms consume one hash-verified Stage-5 plan; the decision table is applied without discretion. +- Collateral suites show no reproducible regression; the capacity pair confirms the predicted run-level coverage gain. - Paid validation never begins without a recorded ceiling; every phase records actual and projected spend. -- Report failures carry templated, structured, human-readable messages with no raw exception text, no repository source, and no hashing. -- Teardown leaves exactly one product path with no surviving flag and no dark code, and preserves every produced report and pinned plan under a verified manifest. -- Checks, tests, and build pass; the final behavior and the strong-effect-only `n = 6` limitation are documented. +- Report failures carry templated, structured messages with no raw exception text, no repository source, and no hashing. +- Teardown leaves one product path with no surviving flag, and preserves every report and pinned plan under a verified manifest. +- Checks, tests, and build pass; the decision and the `n = 6` limitation are documented. ## Stop Conditions -- Treatment surface below 5 packets, or cross-coverage packs below 3, on either distinct diff: the predicate does not reach materially more code than Plan 102's, or does not do the new thing. Stop before any paid call and record the premise as disproven. -- `same_container` more than doubles multi-atom packets versus derived-only edges, or promotes more than 25% of reviewable hunks on any run: demote it to context-only and re-run the replay with derived edges before continuing. -- Any atom split or reordered, hunk lost or duplicated, disconnected packet, cap exceeded, member coverage demotion, profile or budget downgrade, or member context collapse below `sliced` that was not abandoned: fix the deterministic design before spending anything. -- The affinity view mutates the relationship graph, or flag-off artifacts change: stop; the isolation guarantee is the basis of every later comparison. -- The replay cannot reproduce flag-off artifacts from recorded inputs or resolves different production refs: reconcile the evidence. -- A paid pair's two arms consumed different Stage-5 plans, or a pinned-plan hash or `hunkId` set fails validation: discard that pair and rerun it. Never analyze an unpinned pair. -- The preflight target packet does not combine at least two atoms: repair the fixture and rerun. This is a fixture defect and explicitly does **not** trigger teardown; Plan 102 stopped here after a single repair when its own step 7 called for another. +- Packet reduction below 20% per distinct diff, or deviation from Plan 102's frozen counts above 2 packets: the port is wrong, not the mechanism. Fix before proceeding. +- Composition-change gate below 2 packets per distinct diff: delete the affinity view and container identity and continue with Plan 102 ordering. Not a plan failure. +- Any atom split or reordered, hunk lost or duplicated, cap exceeded, coverage promoted, profile or budget downgraded, or member context collapsed without abandonment: fix the deterministic design before spending anything. +- The affinity view mutates the relationship graph, or flag-off artifacts change: stop; isolation is the basis of every later comparison. +- A dilution repeat's arms consumed different Stage-5 plans, or a pinned-plan hash or `hunkId` set fails validation: discard that repeat and rerun it. +- The decision table returns Fail: packing does not ship at any predicate or cap. Take the failure branch and record the curve. - The decision table returns Void twice: take the failure branch rather than redesigning a third time. -- Cross-arm reconciliation is proposed as optional: reject it. With the plan pinned it is exact and required; without pinning the pair is invalid. +- Any proposal to raise `MAX_HUNKS_PER_PACKET` to recover compression: reject it here. It is a separate question with its own untested recall risk, and the sweep says it is worth ~5 points. - Actual plus projected spend exceeds the approved ceiling: stop for explicit approval. Do not weaken a gate to fit a budget. -- "Keep packing dark" always means *do not ship it in this iteration*; every such outcome terminates in step 11's failure branch in the same change. Deferring teardown to a later plan is itself a stop condition. +- "Keep packing dark" always means *do not ship it in this iteration*; every such outcome terminates in step 10's failure branch in the same change. ## Maintenance Notes -- Any future change to `hunkFirstGroups()`, `canJoinGroup()`, the relationship edge builders or their strengths, `buildRelatedChangedContext()`, `readEnclosingSymbolSource()`, `computeSymbolContextBudget()`, `packetReviewProfile()`, coverage levels, packet caps, `packetDispatchRank()`, or `toolBudget()` invalidates the recorded shape, promotion, context, and order baselines; rerun the replay before changing unconditional packing behavior. -- Reviewers should scrutinize affinity-view isolation, edge-source attribution, per-member context quality, promotion records, abandonment reasons, and pinned-plan provenance — not packet count. -- Plan 102's negative result is why this plan exists. Its preserved reports and paid logs under `codegenie-private-evals/trails-api/packet-packing/` remain the record of the rejected coverage/lens predicate and must not be edited or deleted. -- Throughput packing of *unrelated* compatible atoms remains a separate, later question. This plan measures what it would add but deliberately does not ship it: bundling a quality mechanism with a cost mechanism is what made Plan 102 unmeasurable. +- Any future change to `hunkFirstGroups()`, `canJoinGroup()`, the relationship edge builders, `buildRelatedChangedContext()`, `readEnclosingSymbolSource()`, `computeSymbolContextBudget()`, `packetReviewProfile()`, packet caps, `packetDispatchRank()`, or `toolBudget()` invalidates the recorded shape, context, and order baselines; rerun the replay first. +- Reviewers should scrutinize affinity-view isolation, composition changes, per-member context quality, abandonment reasons, and pinned-plan provenance — not packet count, which is Plan 102's already-validated result. +- Plan 102's preserved reports and paid logs under `codegenie-private-evals/trails-api/packet-packing/` remain the record of its failed fixture design and must not be edited or deleted. +- The recall-versus-packet-size curve produced here is the reusable asset. Any future proposal to raise caps, pack cross-file, or pack cross-coverage should extend that curve rather than assume it. - Commit hashes in this document are rebase-unstable. Locate referenced work by commit subject when a hash does not resolve. diff --git a/specs/plans/README.md b/specs/plans/README.md index e1e871b..052997c 100644 --- a/specs/plans/README.md +++ b/specs/plans/README.md @@ -106,4 +106,4 @@ This directory tracks implementation plans for confirmed improvements. Status va | 100 | COMPLETE | Issue 100: Short Hunk IDs — Planner Coverage Survival and Dispatch Resilience | | 101 | IN PROGRESS (paid semantic A/B pending) | Issue 101: Exact Skill Provenance and Evidence-Gated Bundled-Skill Revision | | 102 | COMPLETE (failed gate; baseline restored) | Issue 102: Same-File Packet Packing | -| 103 | PENDING | Issue 103: Relationship-Driven Same-File Packet Packing | +| 103 | PENDING | Issue 103: Compatible-Atom Packing with Relationship-Ordered Composition | From 32d7b8339751a42b43b4943a1d15d7eb092d5ddb Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 12:13:56 -0400 Subject: [PATCH 08/24] feat(scripts): add packing diagnostics over recorded run artifacts Consolidates the three throwaway analyses that drove Plan 103's revision into one reusable tool. Reads recorded telemetry runs only - no review, no model calls - and answers three questions: pairs which atoms are related but currently split, and whether coverage, the caps, or a whole-file bypass is what blocks them sweep how packet count responds to the hunk and patch caps simulate packet count under source, compatibility, or related admission Calibration: sweep at 5h/12K yields 75/66/83 packets against Plan 102's measured 75/68/85, within the documented patch-proxy residual. Known approximations are recorded in the file header: patch size uses the recorded contentWithLineNumbers proxy rather than combinedPatchChars(), container identity is recovered by splitting the owner prefix out of the enclosingSymbol display string rather than a structural ownerKey, and undeclared hunks share a synthetic DEFAULT lens signature. This is a planning aid; Plan 103 step 8's real-builder replay is what gates the work. Note for whoever lands Plan 103's report script: tsconfig.json includes only src and tests, so scripts/*.ts is never typechecked by pnpm run check. Plan 102 shipped a 5,318-line TypeScript script that no gate ever looked at. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/packing-diagnostics.mjs | 330 ++++++++++++++++++++++++++++++++ 1 file changed, 330 insertions(+) create mode 100644 scripts/packing-diagnostics.mjs diff --git a/scripts/packing-diagnostics.mjs b/scripts/packing-diagnostics.mjs new file mode 100644 index 0000000..18c1ca9 --- /dev/null +++ b/scripts/packing-diagnostics.mjs @@ -0,0 +1,330 @@ +#!/usr/bin/env node +// Packing diagnostics over recorded run artifacts (Plan 103 planning aid). +// +// Answers three questions about same-file packet packing without running a +// review or calling a model: +// +// pairs which atoms are related but currently split, and what blocks them +// sweep how packet count responds to the hunk/patch caps +// simulate packet count under a chosen packing predicate +// +// APPROXIMATIONS — this is a planning aid, not the authoritative measurement. +// Plan 103 step 8 specifies a real-builder replay; that is what gates the work. +// This tool differs in three known ways: +// +// 1. Patch size uses each packet hunk's recorded `contentWithLineNumbers` +// length. The builder uses combinedPatchChars() over raw diff lines. +// Plan 102 documented the two differing by about one packet per run. +// 2. Container identity is recovered by splitting the owner prefix out of +// the recorded `enclosingSymbol` display string ("(*T).Method" -> "T"). +// Plan 103 specifies a structural ownerKey resolved from symbol ranges, +// which does not collide across duplicate or nested same-named owners. +// 3. Planner lens signatures are only known for hunks the planner issued a +// coverage entry for. Undeclared hunks fall back to +// defaultLensesForLanguage(), which is constant within a file, so they +// share a synthetic DEFAULT signature. That is sound within a file and +// wrong across files — this tool never compares across files. +// +// Calibration: `sweep` at 5h/12K reproduces Plan 102's measured packet counts +// (75/68/85) within one packet on all three distinct retained diffs. +// +// Usage: +// node scripts/packing-diagnostics.mjs pairs +// node scripts/packing-diagnostics.mjs sweep +// node scripts/packing-diagnostics.mjs simulate [--predicate source|compatibility|related] [--max-hunks N] [--max-patch N] +// +// A runDir is a recorded telemetry run, e.g. +// /.codegenie/runs/20260724-184952-dca8d870 + +import { readdirSync, readFileSync } from "node:fs"; +import path from "node:path"; + +const DEFAULT_MAX_HUNKS = 5; +const DEFAULT_MAX_PATCH = 12_000; +const SWEEP_GRID = [[5, 12_000], [6, 12_000], [8, 12_000], [8, 16_000], [10, 16_000], [10, 20_000], [12, 24_000]]; + +function readJson(file) { + return JSON.parse(readFileSync(file, "utf8")); +} + +// "(*FeeCalculator).CalculateIntentFees" -> "FeeCalculator"; "Foo.bar" -> "Foo". +function ownerPrefix(symbol) { + const dot = symbol.lastIndexOf("."); + if (dot <= 0) { + return undefined; + } + const owner = symbol.slice(0, dot).replace(/^\(\*?|\)$/gu, "").trim(); + return owner.length > 0 ? owner : undefined; +} + +// Baseline runs did no packing, so each recorded packet is exactly one atom. +function loadRun(runDir) { + const plan = readJson(path.join(runDir, "stages/05-planner/review-plan.json")); + const graph = readJson(path.join(runDir, "stages/06-packets/hunk-relationships.json")); + const coverageByHunk = new Map((plan.coverage ?? []).map((entry) => [entry.hunkId, entry.coverage])); + const lensesByHunk = new Map((plan.coverage ?? []).map((entry) => [entry.hunkId, [...(entry.lenses ?? [])].sort().join(",")])); + + const atoms = []; + const atomByHunk = new Map(); + const packetsDir = path.join(runDir, "stages/06-packets/packets"); + for (const file of readdirSync(packetsDir)) { + if (!file.endsWith(".json")) { + continue; + } + const packet = readJson(path.join(packetsDir, file)); + const hunks = packet.hunks ?? []; + if (hunks.length === 0) { + continue; + } + const owners = new Set(); + for (const hunk of hunks) { + const fact = (packet.symbolFacts ?? []).find((entry) => entry.hunkId === hunk.hunkId && entry.enclosingSymbol); + const owner = fact ? ownerPrefix(fact.enclosingSymbol) : undefined; + if (owner !== undefined) { + owners.add(owner); + } + } + const atom = { + id: packet.id, + path: packet.path, + kind: packet.kind, + coverage: packet.coverage, + owners, + hunkIds: hunks.map((hunk) => hunk.hunkId), + hunkCount: hunks.length, + patchChars: hunks.reduce((sum, hunk) => sum + (hunk.contentWithLineNumbers ?? "").length, 0), + sourcePos: Math.min(...hunks.map((hunk) => hunk.newStart ?? hunk.oldStart ?? 0)), + lensSignature: [...new Set(hunks.map((hunk) => lensesByHunk.get(hunk.hunkId) ?? "DEFAULT"))].sort().join("|") + }; + atoms.push(atom); + for (const hunkId of atom.hunkIds) { + atomByHunk.set(hunkId, atom.id); + } + } + + return { + name: path.basename(runDir), + atoms, + atomByHunk, + graph, + plannerEntries: (plan.coverage ?? []).length, + effectiveCoverage: (hunkId) => coverageByHunk.get(hunkId) ?? "normal" + }; +} + +// Packing-affinity adjacency between atoms: strong recorded edges plus shared +// container. Mirrors Plan 103's derived/structural split. +function affinity(run) { + const byId = new Map(run.atoms.map((atom) => [atom.id, atom])); + const adjacency = new Map(run.atoms.map((atom) => [atom.id, new Set()])); + const pairSources = new Map(); + const pairKey = (a, b) => [a, b].sort().join("|"); + + const link = (a, b, source) => { + if (a === b) { + return; + } + adjacency.get(a)?.add(b); + adjacency.get(b)?.add(a); + const key = pairKey(a, b); + const sources = pairSources.get(key) ?? new Set(); + sources.add(source); + pairSources.set(key, sources); + }; + + for (const edge of run.graph.edges ?? []) { + if (edge.strength !== "strong" || !edge.toHunkId) { + continue; + } + const from = run.atomByHunk.get(edge.fromHunkId); + const to = run.atomByHunk.get(edge.toHunkId); + if (from === undefined || to === undefined || from === to) { + continue; + } + if (byId.get(from).path !== byId.get(to).path) { + continue; + } + link(from, to, edge.source); + } + + const byFile = new Map(); + for (const atom of run.atoms) { + byFile.set(atom.path, [...(byFile.get(atom.path) ?? []), atom]); + } + for (const group of byFile.values()) { + for (let i = 0; i < group.length; i += 1) { + for (let j = i + 1; j < group.length; j += 1) { + if ([...group[i].owners].some((owner) => group[j].owners.has(owner))) { + link(group[i].id, group[j].id, "same_container"); + } + } + } + } + + return { byId, adjacency, pairSources, byFile }; +} + +function bypassesPacking(atom) { + return atom.kind === "whole-file" || atom.kind === "file-diff"; +} + +// Which related-but-split atom pairs exist, and what prevents each from packing. +function pairs(run) { + const { byId, pairSources } = affinity(run); + const summary = { coverage: 0, caps: 0, bypass: 0, eligible: 0 }; + const detail = []; + for (const [key, sources] of pairSources) { + const [x, y] = key.split("|"); + const a = byId.get(x); + const b = byId.get(y); + let blocker; + if (bypassesPacking(a) || bypassesPacking(b)) { + blocker = "bypass"; + } else if (a.coverage !== b.coverage) { + blocker = "coverage"; + } else if (a.hunkCount + b.hunkCount > DEFAULT_MAX_HUNKS || a.patchChars + b.patchChars > DEFAULT_MAX_PATCH) { + blocker = "caps"; + } else { + blocker = "eligible"; + } + summary[blocker] += 1; + detail.push({ + file: a.path, + sources: [...sources].sort().join("+"), + blocker, + a: `${a.kind} ${a.coverage} ${a.hunkCount}h/${a.patchChars}c`, + b: `${b.kind} ${b.coverage} ${b.hunkCount}h/${b.patchChars}c` + }); + } + return { + run: run.name, + atoms: run.atoms.length, + plannerEntries: run.plannerEntries, + relatedSplitAtomPairs: pairSources.size, + blockedBy: summary, + detail: detail.sort((p, q) => p.file.localeCompare(q.file)) + }; +} + +// All three predicates share Plan 102's partition (same file + same effective +// coverage + same planner lens signature) and differ only in admission order: +// +// source fill in source order — Plan 102's rule +// compatibility prefer related, else source — Plan 103's rule +// related admit only related atoms — the rejected draft +function simulate(run, { predicate, maxHunks, maxPatch }) { + const { adjacency } = affinity(run); + const partitions = new Map(); + const packets = []; + + for (const atom of run.atoms) { + if (bypassesPacking(atom)) { + packets.push([atom]); + continue; + } + const key = `${atom.path}${atom.coverage}${atom.lensSignature}`; + partitions.set(key, [...(partitions.get(key) ?? []), atom]); + } + + for (const group of partitions.values()) { + const remaining = [...group].sort((a, b) => a.sourcePos - b.sourcePos); + while (remaining.length > 0) { + const seed = remaining.shift(); + const members = [seed]; + let hunks = seed.hunkCount; + let patch = seed.patchChars; + for (;;) { + const fits = (atom) => hunks + atom.hunkCount <= maxHunks && patch + atom.patchChars <= maxPatch; + const related = predicate === "source" + ? undefined + : remaining.find((atom) => fits(atom) && members.some((member) => adjacency.get(member.id)?.has(atom.id))); + const next = predicate === "related" ? related : related ?? remaining.find(fits); + if (next === undefined) { + break; + } + remaining.splice(remaining.indexOf(next), 1); + members.push(next); + hunks += next.hunkCount; + patch += next.patchChars; + } + packets.push(members); + } + } + + const sizes = packets.map((packet) => packet.reduce((sum, atom) => sum + atom.hunkCount, 0)); + const atomsPerPacket = {}; + for (const packet of packets) { + atomsPerPacket[packet.length] = (atomsPerPacket[packet.length] ?? 0) + 1; + } + return { + run: run.name, + predicate, + caps: { maxHunks, maxPatch }, + atomsOff: run.atoms.length, + packetsOn: packets.length, + reductionPct: Number((100 * (run.atoms.length - packets.length) / run.atoms.length).toFixed(1)), + multiAtomPackets: packets.filter((packet) => packet.length > 1).length, + largestPacketHunks: Math.max(...sizes), + packetsOverFiveHunks: sizes.filter((size) => size > 5).length, + atomsPerPacket + }; +} + +function sweep(run) { + return { + run: run.name, + atomsOff: run.atoms.length, + grid: SWEEP_GRID.map(([maxHunks, maxPatch]) => { + const result = simulate(run, { predicate: "compatibility", maxHunks, maxPatch }); + return { + cap: `${maxHunks}h/${maxPatch / 1000}K`, + packets: result.packetsOn, + reductionPct: result.reductionPct, + multiAtomPackets: result.multiAtomPackets, + largestPacketHunks: result.largestPacketHunks, + packetsOverFiveHunks: result.packetsOverFiveHunks + }; + }) + }; +} + +function main() { + const [mode, ...rest] = process.argv.slice(2); + const flags = new Map(); + const runDirs = []; + for (let i = 0; i < rest.length; i += 1) { + const arg = rest[i]; + if (arg.startsWith("--")) { + flags.set(arg.slice(2), rest[i + 1]); + i += 1; // consume the value so it is never mistaken for a run directory + continue; + } + runDirs.push(arg); + } + const flag = (name, fallback) => flags.get(name) ?? fallback; + + if (!mode || runDirs.length === 0 || !["pairs", "sweep", "simulate"].includes(mode)) { + console.error("usage: packing-diagnostics.mjs [--predicate source|compatibility|related] [--max-hunks N] [--max-patch N]"); + process.exitCode = 2; + return; + } + + const options = { + predicate: flag("predicate", "compatibility"), + maxHunks: Number(flag("max-hunks", DEFAULT_MAX_HUNKS)), + maxPatch: Number(flag("max-patch", DEFAULT_MAX_PATCH)) + }; + if (!["source", "compatibility", "related"].includes(options.predicate)) { + console.error(`unknown predicate: ${options.predicate}`); + process.exitCode = 2; + return; + } + + for (const runDir of runDirs) { + const run = loadRun(runDir); + const report = mode === "pairs" ? pairs(run) : mode === "sweep" ? sweep(run) : simulate(run, options); + console.log(JSON.stringify(report, null, 1)); + } +} + +main(); From a0c2887e1dbba305a11d859a0d34987f49302c3f Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 12:22:36 -0400 Subject: [PATCH 09/24] docs(plans): cut relationship signals from Plan 103; rebuild the paid experiment Measured relationship-preferred admission against plain source order with scripts/packing-diagnostics.mjs: identical packing on all three distinct retained diffs - same packet counts, same composition. Related atoms are usually already source-adjacent, so preferring them picks the same atom. Combined with the earlier finding that relationship-required packing yields 4.2% against compatibility's 21.9%, the whole relationship subsystem is removed: no affinity view, no ownerKey, no same_container, no edge-cap rework. Diagnostic outputs are preserved under packet-dilution/reports with their own manifest, leaving Plan 102's manifest exactly as its reconciliation note describes. Review of the prior draft found two blocking defects in the paid experiment, both fixed: The dilution fixture could repeat Plan 102's zero-treatment failure. Pinning a drawn plan makes arms comparable but not compatible - if the planner grades the buggy hunk deep and its siblings normal they never share a partition. The plan is now hand-authored rather than drawn, which guarantees one partition, removes planner variance, removes any temptation to redraw unfavourable samples, and costs nothing. Treatment is proven model-free at 15/5/3 packets before any paid call, and a reference draw is recorded alongside so the authored plan's realism is documented. The decision table could approve a net loss. With integer counts the old rule had no reachable Inconclusive branch, R3 was inert, and R1=5/R5=4 passed at 80% against a break-even of 89/108 = 82.4%. The bar is now derived from the fixed-slot hunk yield measured in phase A, requires a reliable R1, and gives R3 a real role distinguishing a cliff from a gradient. Three independent bugs at three packet positions raise resolution from 6 to 18 observations per arm at the same spend. Also: fixed-slot hunk yield at 57 dispatch slots becomes the primary free gate, replacing packet count as a proxy for the coverage claim; multi-member symbol context gets explicit budget arithmetic with quality measured after final rendering; the fixture moves to evals/packet-dilution since it is synthetic and belongs in the standing suite; cross-language shape cases are added because skill-semantics files are 10-15 lines and yield one atom each; both settings become eval/internal-only rather than repo-configurable; and the TL;DR drops the false claim that compatibility prevents more expensive review. Co-Authored-By: Claude Opus 5 (1M context) --- ...-103-relationship-driven-packet-packing.md | 519 +++++++++--------- specs/plans/README.md | 2 +- 2 files changed, 263 insertions(+), 258 deletions(-) diff --git a/specs/plans/103-issue-103-relationship-driven-packet-packing.md b/specs/plans/103-issue-103-relationship-driven-packet-packing.md index 1310dc5..87ce6ee 100644 --- a/specs/plans/103-issue-103-relationship-driven-packet-packing.md +++ b/specs/plans/103-issue-103-relationship-driven-packet-packing.md @@ -1,10 +1,10 @@ -# Issue 103: Compatible-Atom Packing with Relationship-Ordered Composition +# Issue 103: Compatible-Atom Packet Packing and the Packet-Size Recall Curve Status: PENDING -Related: Plan 102 (`102-issue-102-same-file-packet-packing.md`) — COMPLETE as a failed treatment gate; its deterministic packing mechanism passed every invariant and is adopted here. Plan 67 (hunk relationship context) built the graph this plan reuses for ordering. Plan 92 (coverage calibration), Plan 100 (dispatch rank), Plan 79 (repeat/recall harness), Plan 32 (adaptive Stage-6 symbol context). -Planned from: Plan 102's preserved evidence under `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-packing/reports/` (manifest verified), plus artifact-only diagnostics over the same retained runs, 2026-07-25 +Related: Plan 102 (`102-issue-102-same-file-packet-packing.md`) — COMPLETE as a failed treatment gate; its deterministic packing mechanism passed every invariant and is adopted here unchanged. Plan 92 (coverage calibration), Plan 100 (dispatch rank), Plan 79 (repeat/recall harness), Plan 32 (adaptive Stage-6 symbol context). +Planned from: Plan 102's preserved evidence, plus artifact-only diagnostics reproducible with `scripts/packing-diagnostics.mjs`, 2026-07-25 Production replay refs: base/merge-base `d1c49bdf6a8002ec2ec27faac94a932d736532b2`; head `fbb5f8761c2c296e115af17e919a7c35d9de8373` -Planned at: commit `7b61b34` (branch `next`) +Planned at: commit `32d7b83` (branch `next`) Recommended priority: after Plan 101's paid A/B settles. ## Introduction and TL;DR @@ -13,354 +13,317 @@ Recommended priority: after Plan 101's paid A/B settles. Reviews run out of time before they run out of code. On the motivating pull request, Stage 6 produced **96 packets** covering 142 changed hunks. Only 57 packets were dispatched before the deadline, so **53 hunks received no review at all**. -A packet is one conversation with the model, and most of its cost is fixed — the skill text, the prompt scaffolding, the forced finalization at the end. Measurements from that run show a five-hunk packet costs roughly the same as a one-hunk packet: about 3 model calls and 228 seconds either way. Yet 73 of the 96 packets held a single hunk. We were paying full conversation price to review one hunk at a time. +A packet is one conversation with the model, and most of its cost is fixed — the skill text, the prompt scaffolding, the forced finalization at the end. Measurements from that run show a five-hunk packet costs roughly the same as a one-hunk packet: about 3 model calls and 228 seconds either way. Yet 73 of the 96 packets held a single hunk. ### What this plan achieves -Group compatible hunks from the same file into fewer, slightly larger packets — about **21% fewer packets**, measured deterministically across four production runs with no model calls. +Two things. -The point is not a shorter review. It is a **more complete** one. At the same time budget, 21% fewer packets means roughly **108 of 142 hunks reviewed instead of 89**. The win is coverage: fewer changed lines slip past the deadline unreviewed. +**First, it lands Plan 102's packing mechanism.** Group compatible hunks from the same file into fewer, slightly larger packets — about **21% fewer packets**, measured deterministically across four production runs with no model calls. The point is not a shorter review but a **more complete** one: at the same time budget, roughly **108 of 142 hunks reviewed instead of 89**. The win is coverage. -A second, smaller benefit: when a packet has room and there is a choice of what to put in it, we prefer hunks that tree-sitter says are *related* — the same function, the same class or receiver type, one mentioning the other. That does not change how many packets exist, only which hunks sit together. A reviewer looking at five functions is then more likely to be looking at five *related* functions, and can notice things like "four of these added a validation check and the fifth didn't." +**Second, and more durable, it produces the recall-versus-packet-size curve.** Nobody has ever measured whether a hunk sharing a packet gets reviewed as carefully as a hunk reviewed alone. Every packing proposal — this one, Plan 102, any future cap increase or cross-file scheme — rests on assuming that answer. This plan measures it once, and the curve becomes a standing asset. -### How the machinery works +### What it does *not* do, and why + +An earlier draft proposed packing by tree-sitter relationships instead of by planner compatibility, and then as an ordering preference within compatibility. Both were measured against recorded artifacts and rejected: -Seven pieces, in the order they run inside Stage 6: +- **As a membership predicate**, relationship-required packing yields 4 combinable atom pairs and a 4.2% packet reduction, against compatibility's 21.9%. Relationships between *distinct* atoms are sparse because today's grouper already absorbs most same-symbol adjacency into atoms. +- **As an ordering preference**, it produces packing byte-identical to plain source order on all three distinct retained diffs — same packet counts, same composition. Related atoms are usually already source-adjacent, so preferring them picks the same atom. -1. **Atoms.** Today's grouper already merges hunks that share an enclosing symbol or sit within 30 lines of each other. We keep it exactly as-is and treat each group it produces as an unbreakable unit. Nothing this plan does ever splits one. +So there is no affinity view, no container identity, and no relationship ordering in this plan. `scripts/packing-diagnostics.mjs` reproduces both measurements, and the outputs are preserved. + +### How the machinery works -2. **Partitions.** Within each file, atoms are bucketed by two things the planner already decided: the review depth (`light`/`normal`/`deep`) and the set of review lenses requested. Only atoms in the same bucket may be combined. This is what guarantees no hunk is reviewed more shallowly, more expensively, or with different expertise than the planner asked for. +Five pieces, in the order they run inside Stage 6: -3. **An affinity map.** A private, packing-only map of which atoms are related, built from tree-sitter facts: same symbol, same container (class, Go receiver, Rust `impl`, Python class, Solidity contract), or one symbol mentioning another. It is deliberately kept separate from the relationship data that already feeds review prompts, so turning packing off changes nothing. +1. **Atoms.** Today's grouper already merges hunks that share an enclosing symbol or sit within 30 lines. We keep it as-is and treat each group as an unbreakable unit. Nothing here ever splits one. -4. **Filling.** Walk each bucket in source order, filling a packet until it hits 5 hunks or 12,000 characters. When choosing what to add next, prefer a related atom; otherwise take the next one in source order. Hunks are always *displayed* in source order regardless of what order they were admitted — the code still reads top to bottom. +2. **Partitions.** Within each file, atoms are bucketed by two things the planner already decided: review depth (`light`/`normal`/`deep`) and the set of lenses requested. Only atoms in the same bucket may combine. This guarantees no hunk is reviewed more shallowly, or with different expertise, than planned. It does **not** guarantee equal cost — see below. -5. **Safety checks before committing.** Each candidate packet is built in a scratch space first. If merging would drop a review lens, drop an important planner note attached to a high-priority hunk, or lose a function's surrounding source, the merge is abandoned and those atoms stay separate. Nothing is written until a candidate passes. +3. **Filling.** Walk each bucket in source order, filling a packet until it hits 5 hunks or 12,000 characters, then start another. -6. **Context for every member.** If a packet ends up holding three different functions, the reviewer gets all three function bodies, splitting the context budget between them. Today's code reads only one, which would quietly leave the other two as bare diffs. +4. **Context for every member.** If a packet holds three functions, the reviewer gets all three function bodies under a shared, explicitly budgeted allowance. Today's code reads only the top-ranked one, which would quietly leave the others as bare diffs. This is the one place Plan 102's mechanism is genuinely improved rather than reused. -7. **A floor on review effort.** A merged packet never gets a weaker review profile or a smaller tool budget than its parts would have received alone. +5. **Safety checks before committing.** Each candidate packet is built in a scratch space first. If merging would drop a lens, drop an important planner note, or squeeze any member's source below a usable minimum *after final rendering*, the merge is abandoned and those atoms stay separate. + +Packing can make a hunk **more** expensive than it would have been alone: the profile floor may lift a `standard` atom into an `investigate` packet's tool budget, and multi-member context deliberately adds input tokens. That is a deliberate trade for coverage, and both effects are reported rather than assumed away. ### The risk, and how it gets tested -The obvious danger is **attention dilution**: a bug sitting in a five-hunk packet might get less scrutiny than the same bug alone in its own packet. Nobody has ever measured this — Plan 102 tried and its test fixture was built wrong. +**Attention dilution.** A bug in a five-hunk packet might get less scrutiny than the same bug alone. + +One test repository, fifteen separated changes in one Go file, **three** independent bugs at different positions plus a safe-looking control. The same review runs at packet caps 1, 3, and 5, six repeats each, all against one hand-authored frozen plan so packet size is the only variable. Three bugs times six repeats gives **18 observations per arm**, not 6 — the same spend buys three times the statistical resolution. -So this plan measures it directly and measures nothing else. One test repository with about ten separated changes in one file, one of which contains a real bug. The same review runs three times over with packets capped at 1, 3, and 5 hunks, six repeats each, all sharing one frozen planning pass so packet size is the only variable. If recall at five hunks is within one hit of recall at one hunk, packing is safe and ships. If it degrades, the whole thing is deleted. +The pass bar is derived, not chosen: packing must not cost more per-hunk recall than the coverage it buys. If the replay shows packing reviewing 108 hunks where the baseline reviews 89, per-hunk recall may fall no further than 89/108 ≈ **82.4%** of baseline before the whole thing is a net loss. -Cost: about **$17** to answer that question, plus an optional $50 production-scale confirmation afterward. +Cost: about **$14** to answer that, plus an optional $50 production-scale confirmation. ### The outcome, either way -There is no middle state. Either packing becomes permanent, unconditional behavior and the temporary switches are deleted — or packing is removed entirely and the code returns to today's baseline. No flag survives, and no dark code path is left behind. +Either packing becomes permanent unconditional behavior and the temporary switches are deleted, or packing is removed entirely and the code returns to today's baseline. No flag survives, no dark path is left behind. -> **Revision note.** The first draft of this plan (commit `7b61b34`) proposed replacing Plan 102's compatibility predicate with a relationship predicate, on the theory that coverage-grade equality was blocking the packs that mattered. Artifact-only diagnostics disproved that: on the one retained run with usable planner coverage, relationship-required packing yields 4 combinable atom pairs and a 4.2% packet reduction, against Plan 102's measured 21.9%. Coverage inequality blocks 2 of 13 candidate pairs; the 5-hunk cap blocks 7. This revision keeps Plan 102's mechanism and demotes relatedness from a membership predicate to a composition-ordering preference, which is what the evidence supports. +> **Revision history.** Draft 1 (`7b61b34`) proposed a relationship membership predicate. Draft 2 (`4935134`) demoted relationships to ordering after diagnostics showed 4.2% versus 21.9%. This draft removes relationships entirely after measuring ordering as a no-op, and rebuilds the paid experiment after review found it could repeat Plan 102's zero-treatment failure and could pass a net-negative outcome. -> Executor instructions: adopt Plan 102's compatible-atom packing unchanged in predicate and caps — its deterministic gate passed on four runs and this plan does not relitigate it. Use the relationship graph only to decide *which* compatible atoms share a packet, never how many packets exist, and never to render hunks out of source order. Keep the container-affinity signal out of the shared prompt-context graph. Extend symbol context to every member atom. Do not raise `MAX_HUNKS_PER_PACKET`; the sweep below shows 5 already captures ~80% of achievable compression. The one open question is recall under packing, and the paid phase measures exactly that and nothing else. +> Executor instructions: adopt Plan 102's predicate, caps, and fill order unchanged — its deterministic gate passed on four runs and this plan does not relitigate it. The new engineering is multi-member symbol context and transactional rejection. The new evidence is the packet-size recall curve. Do not add relationship signals, container identity, coverage promotion, tool-budget arms, or cap increases; each was measured or reasoned out and has an entry in Non-Goals. > -> **Commit references are rebase-unstable.** Branch `next` was rebased during Plan 102. The Plan 102 series is currently `5551547` (implementation), `8fceba9` (report tool), `7ebfd2f` (report fixes), `87a5a10` (retry docs), `f372f73` (teardown). Locate referenced work by commit subject when a hash does not resolve. +> **Commit references are rebase-unstable.** The Plan 102 series is currently `5551547`, `8fceba9`, `7ebfd2f`, `87a5a10`, `f372f73`. Locate referenced work by commit subject when a hash does not resolve. > -> Drift check: `git diff --stat 7b61b34..HEAD -- src/pipeline/packet-builder.ts src/pipeline/review-runner.ts src/repo/symbol-extraction.ts src/config/schema.ts src/config/config-loader.ts src/types.ts src/evals/eval-runner.ts scripts/packet-packing-report.ts tests/pipeline-phase5.test.ts tests/config-loader.test.ts tests/evals.test.ts tests/packet-packing-report.test.ts` -> Working-tree check: `git status --short -- src/pipeline/ src/repo/ src/config/ src/types.ts src/evals/` +> Drift check: `git diff --stat 32d7b83..HEAD -- src/pipeline/packet-builder.ts src/pipeline/review-runner.ts src/config/schema.ts src/config/config-loader.ts src/types.ts src/evals/eval-runner.ts scripts/packet-packing-report.ts tests/pipeline-phase5.test.ts tests/evals.test.ts` +> Working-tree check: `git status --short -- src/ evals/ scripts/` ## Decision -Ship Plan 102's compatible-atom packing, with relationship-ordered composition, at today's caps. - -- Run the existing symbol/proximity grouper unchanged. Its output groups remain indivisible **atoms**. -- Partition a file's atoms by `(effectiveCoverage, normalizedPlannerLensSignature)` — Plan 102's predicate, unchanged. This is where the throughput comes from and it introduces **zero coverage promotion** by construction. -- Fill packets greedily within each partition under the unchanged `MAX_HUNKS_PER_PACKET = 5` and `MAX_PATCH_CHARS = 12_000`. -- **Relatedness decides composition, not count.** When choosing the next atom to admit, prefer one connected to a current member by a packing-affinity edge; fall back to the source-earliest atom that fits. With no relationships present this degrades to Plan 102's exact behavior. -- **Selection order is not render order.** Hunks always render in source order within a packet. Plan 102 conflated the two; they are separate here. -- Symbol context follows membership: multi-symbol packets read every member's primary symbol under a shared budget. -- Lens loss, focus-note loss, and per-member context collapse are prevented by **transactional candidate rejection**, not by narrowing the predicate. -- Plan 102's monotonic profile floor is retained unchanged. -- One mechanism flag plus one experiment-only cap knob, both deleted at teardown. No budget-mode arm. - -There is **no coverage promotion anywhere in this plan.** Because the partition key preserves Plan 102's coverage equality, the member-promotion machinery, the `CoverageEscalation` conflict with Plan 92, and the promotion invariants from the first draft are all deleted rather than designed. +- Run the existing grouper unchanged; its groups are indivisible **atoms**. +- Partition each file's atoms by `(effectiveCoverage, normalizedPlannerLensSignature)` — Plan 102's predicate. Zero coverage promotion by construction. +- Fill greedily in source order under the unchanged `MAX_HUNKS_PER_PACKET = 5` and `MAX_PATCH_CHARS = 12_000`. +- **Source order** means: hunks are ordered within a packet by file position, and packets are ordered by their earliest member hunk. Because partitions are non-contiguous, the globally flattened hunk sequence is *not* preserved across packets. That is expected and is not a defect. +- Read enclosing-symbol source for **every** member atom under an explicit shared budget. +- Abandon any candidate that would lose a lens, a high-priority planner focus note, or a member's usable symbol source after final rendering. +- Retain Plan 102's monotonic profile floor and the unchanged `packetDispatchRank()` formula. +- Two temporary settings, **eval and internal only** — not in the repo TOML schema, not repo-safe filtered, not user-facing. Both deleted at teardown. +- Ship only if the measured packet-size recall curve clears a break-even bar derived from the measured coverage gain. ## Evidence ### Plan 102's mechanism passed its deterministic gate -Four-run replay, zero model calls, zero failures: `740d73f2` 109→85, `fe1548ae` 109→85, `81f806a6` 93→68, `dca8d870` 96→75 — 21.9–26.9% fewer packets with zero coverage promotions, zero effective profile or budget downgrades, zero cap violations, zero lens drops, and zero invalid dispatch ranks. Only the paid recall gate failed, and it failed on fixture treatment, never on recall. This plan adopts that mechanism rather than rebuilding it. - -Note `740d73f2` and `fe1548ae` are the same commit, and all four retained runs are the same PR. This is one PR sampled at three head commits, not four independent samples. Every threshold below counts distinct diffs. - -### Relationship-required packing is not a viable throughput mechanism +Four runs, zero model calls, zero failures: `740d73f2` 109→85, `fe1548ae` 109→85, `81f806a6` 93→68, `dca8d870` 96→75 — 21.9–26.9% fewer packets with zero coverage promotions, zero profile or budget downgrades, zero cap violations, zero lens drops, zero invalid dispatch ranks. Only the paid gate failed, and it failed on fixture treatment, never on recall. -Artifact-only diagnostics over `dca8d870` (the only retained run with usable planner coverage — `740d73f2` has 0 surviving planner entries and `81f806a6` has 2): +`740d73f2` and `fe1548ae` are the same commit, and all four runs are the same PR at three head commits. This is one PR, not four samples. Thresholds below count distinct diffs. -| Measure | Value | -| --- | ---: | -| Atom pairs adjacent under strong + container edges, same file, currently split | 13 | -| — blocked by the 5-hunk / 12K caps | 7 | -| — blocked by coverage inequality | 2 | -| — eligible to pack | 4 | -| Simulated packet reduction, relationship-required predicate | 96→92 (4.2%) | -| Same, at a hypothetical 8-hunk cap | 96→89 (7.3%) | +### Relationship signals were measured and rejected -Relationships between *distinct atoms* are sparse: today's grouper already absorbs most same-symbol adjacency into atoms, so what remains is thin. Relatedness cannot carry a throughput plan. It can, however, decide composition for free. +Reproducible via `scripts/packing-diagnostics.mjs`; outputs preserved under `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/reports/`: -### Coverage inequality is not the blocker - -Planner grades on `dca8d870` were deep:14, normal:14, light:2, skip:2 across 32 entries; the other 110 reviewed hunks take the `normal` default. Of 13 adjacent atom pairs, only 2 cross a coverage grade, and both come from `planner_hint` edges — **zero structural (`same_symbol`, `symbol_mention`, container) pairs cross a grade.** +| Artifact | SHA-256 | +| --- | --- | +| `plan103-affinity-pairs.json` | `677958c7caf9ef3b5243085385cefd0659b4c5df60869dd7cf917788b85d18f4` | +| `plan103-cap-sweep.json` | `f9915c77210d329c2859531007f6175c90eaca45cb52417a22d8dbc3e08de371` | +| `plan103-ordering-comparison.json` | `e4e6bf2e3cfe585b1c04db1280a2bcb1c1bb5516975352b545fb96211bf24a57` | -This retires the first draft's central claim. Plan 102's coverage-equality partition was self-defeating in a *fixture*, where the planner grades every hunk; in production it grades 23% of them and the rest share a default. Retaining coverage equality costs 2 of 13 opportunities and removes an entire subsystem. +On `dca8d870`, 13 atom pairs are related but currently split: 7 blocked by the caps, 2 by coverage inequality, 4 eligible. Relationship-required packing gives 96→92 (4.2%). Relationship-preferred *ordering* gives 75/66/83 packets across the three distinct diffs — identical to plain source order, including multi-atom counts. -It also retires the proposed stage reordering — showing the planner hunk groups so it grades siblings consistently solves a problem that occurs twice per PR. +Both measurements use documented approximations (patch proxy, owner prefix from display strings). Neither is close enough to its threshold for the approximation to change the conclusion. -### The 5-hunk cap is near-optimal, not arbitrary +### The 5-hunk cap is near-optimal -A cap sweep under Plan 102's predicate, using a simulator validated against Plan 102's real implementation (cap 5/12K reproduces 76/68/85 packets against the measured 75/68/85 — within one packet on all three runs): +Simulator validated against Plan 102's real implementation to within one packet. | Cap | `dca8d870` | `81f806a6` | `740d73f2` | Packets >5 hunks | | --- | --- | --- | --- | ---: | -| 5h/12K (today) | 96→76 (20.8%) | 93→68 (26.9%) | 109→85 (22.0%) | 0 | -| 6h/12K | 96→75 (21.9%) | 93→65 (30.1%) | 109→81 (25.7%) | 2–4 | -| 8h/12K | 96→73 (24.0%) | 93→64 (31.2%) | 109→80 (26.6%) | 4–5 | -| 10h/16K | 96→71 (26.0%) | 93→61 (34.4%) | 109→77 (29.4%) | 5–6 | -| 12h/24K | saturated — no further change | | | | - -The predicate is worth ~21 points; the cap is worth ~5 more before saturating, because partitions run out of compatible atoms rather than hitting the cap. Raising it trades a handful of oversized packets for a few points of compression and enlarges the untested recall risk. **The cap stays at 5.** +| 5h/12K (today) | 96→75 (21.9%) | 93→66 (29.0%) | 109→83 (23.9%) | 0 | +| 6h/12K | 96→75 | 93→65 | 109→81 | 2–4 | +| 8h/12K | 96→73 (24.0%) | 93→64 | 109→80 | 4–5 | +| 10h/16K | 96→71 (26.0%) | 93→61 | 109→77 | 5–6 | +| 12h/24K | saturated | | | | -### The value is run-level coverage, and the risk is packet-level recall +The predicate is worth ~21 points; the cap ~5 more before saturating, because partitions run out of compatible atoms rather than hitting the cap. Raising it trades oversized packets for a few points and enlarges the untested recall risk. **The cap stays at 5.** -Plan 102's Stage 7 spent 174 model calls, 5.321M tokens, `$22.85`, and 52.7 concurrency-normalized minutes on 57 dispatched packets, reviewing 89 of 142 reviewable hunks before the deadline. Forced finalization alone was 34 calls, 1.037M input tokens, and `$6.71` of pure per-packet overhead. Observed per-packet cost is roughly flat in packet size: +### The value is coverage, and the cost is not free -| Packet size | n | Avg calls | Avg input tokens | Avg model-service seconds | -| --- | ---: | ---: | ---: | ---: | -| 1 hunk | 41 | 3.10 | 91,585 | 228 | -| 2 hunks | 8 | 3.00 | 67,433 | 206 | -| 3 hunks | 3 | 2.67 | 77,940 | 158 | -| 4–5 hunks | 5 | 3.00 | ~109,000 | 228 | +Plan 102's Stage 7 spent 174 model calls, 5.321M tokens, `$22.85`, and 52.7 concurrency-normalized minutes on 57 dispatched packets, reviewing 89 of 142 hunks. Forced finalization alone was 34 calls and `$6.71` of per-packet overhead. Per-packet cost is roughly flat in size (1 hunk: 3.10 calls / 228s; 4–5 hunks: 3.00 calls / 228s). -So a 21% packet reduction at a fixed time budget is worth roughly 19 more reviewed hunks on that run — 89/142 to ~108/142. That is the payoff, and it is run-level recall. - -Against it stands packet-level recall: a hunk sharing a packet may be reviewed less attentively. **Nobody has measured that.** Plan 102 tried and its fixture never produced the treatment; the first draft of this plan proposed to try again with different machinery. It is one curve, it is the same question under every predicate, and it is the only thing standing between this mechanism and rollout. The paid phase measures it directly and measures nothing else. +Packing does not reduce per-hunk cost uniformly. The profile floor can lift a member into a larger tool budget, and multi-member symbol context adds input tokens by design. Both are reported off/on rather than assumed neutral. ## Current State -- `hunkFirstGroups()`/`canJoinGroup()` join same-file hunks by identical enclosing symbol or `NEARBY_GAP_LINES = 30` proximity. Baseline restored after Plan 102's teardown. -- `buildHunkRelationshipGraph()` produces `same_symbol` (`strong`), `planner_hint` (`strong` for symbol hints, `medium` for file hints), and `symbol_mention` (`strong` with an enclosing symbol, else `medium`) edges, capped at `MAX_RELATIONSHIP_EDGES_PER_HUNK = 8`, dropped on overflow in insertion order. -- `addSymbolMentionEdges()` guards ambiguity (`sourceIdentities.size > 1` → skip); `addPlannerHintEdges()` has no equivalent guard. -- `buildRelatedChangedContext()` consumes every non-`weak` edge whose target is outside the packet, capped at `MAX_RELATED_CONTEXTS_PER_PACKET = 3`. Any new `medium` edge source added to the shared graph would change flag-off context, notes, repository reads, and profiles. +- `hunkFirstGroups()`/`canJoinGroup()` join same-file hunks by identical enclosing symbol or `NEARBY_GAP_LINES = 30`. Baseline restored after Plan 102's teardown. - `buildPacket()` derives `reviewProfile` after related-context filtering, so absorbing a strong edge can lower the derived profile — the reason Plan 102's floor exists. -- `readEnclosingSymbolSource()` reads one symbol for the whole packet, and `computeSymbolContextBudget()` selects `multiple_symbols_keep_compact` when `singlePrimarySymbol` is false. Packing distinct symbols therefore drops non-primary members' surrounding source, and the coarse `contextQuality` grade does not register it. -- `SymbolInfo` carries `ownerType` (Go receivers, TypeScript classes, Rust `impl`/trait owners, Python classes, Solidity contracts); `generic-adapter.ts` does not. `HunkSymbolFacts` carries neither it nor `packageName`. `packet-context.ts:154` already resolves an owner structurally by name and kind. -- `pnpm test -- ` does **not** filter — it runs the full suite and drops the argument (verified). This plan uses `pnpm exec vitest run `. -- Plan 102's implementation is recoverable at `5551547` and its report harness at `7ebfd2f`, including `safeFailureOutput()`, which hashed every failure message and is not ported. +- `readEnclosingSymbolSource()` reads one symbol per packet; `computeSymbolContextBudget()` selects `multiple_symbols_keep_compact` when `singlePrimarySymbol` is false. `DEFAULT_SYMBOL_CONTEXT_CHARS = 3_000`, `MAX_ADAPTIVE_SYMBOL_CONTEXT_CHARS = 6_000`, `MAX_CONTEXT_CHARS = 8_000`. Three members at the default already exceed the packet cap, and `renderPacketContextText()` truncates the combined text afterward — so any per-member quality computed before final rendering is fiction. +- `MAX_RELATIONSHIP_EDGES_PER_HUNK = 8` drops edges in insertion order; recorded `omittedEdges` is 112/91/89 on the retained runs. Relevant only if a future plan revives relationship signals. +- `evals/skill-semantics/` fixture files are 10–15 lines — one atom each — so those cases cannot exercise packing at all. +- `.gitignore` covers `evals/fixtures/logs/` but not `evals/skill-semantics/logs/`. +- `tsconfig.json` includes only `src/**` and `tests/**`, so `scripts/*.ts` is never typechecked by `pnpm run check`. +- `pnpm test -- ` does not filter; it runs the full suite and drops the argument. - There is no seam to supply a pre-recorded Stage-5 plan to a review. ## Goal -1. Land Plan 102's packing mechanism unchanged in predicate and caps, so the measured 21–27% packet reduction is available. -2. Use relatedness to improve packet composition at zero cost to packet count. -3. Close the gap Plan 102 left open in symbol context, per-member quality, and note pressure, so bigger packets are not quietly worse packets. -4. Measure recall as a function of packet size — the single curve that decides whether packing ships. -5. End in exactly one product path with no surviving flag, under either outcome. +1. Land Plan 102's mechanism unchanged in predicate, caps, and fill order. +2. Close the multi-member symbol-context gap so bigger packets are not quietly worse packets. +3. Prove the coverage gain deterministically as a **fixed-slot hunk yield**, not as a packet-count proxy. +4. Measure the packet-size recall curve with enough resolution to decide, and gate on a break-even bar derived from measurement 3. +5. End in one product path with no surviving flag. ## Design -### 1. Atoms and partitions — Plan 102, unchanged - -Wrap `hunkFirstGroups()` output as atoms carrying ordered hunks, hunk count, `combinedPatchChars()` size, first source position, effective coverage, routed lenses, standalone review profile and context quality, and a stable ID from ordered hunk IDs. Partition each file's atoms by `(effectiveCoverage, normalizedPlannerLensSignature)`. Direct `whole-file` and content-probed `file-diff` returns bypass packing. - -### 2. Structural container identity - -Add `ownerType?: string` and `ownerKey?: string` to `HunkSymbolFacts` in `src/repo/symbol-extraction.ts`. `ownerKey` is structural — `${path}:${ownerName}:${ownerStartLine}-${ownerEndLine}`, resolved through the owner lookup `packet-context.ts:154` already performs — so duplicate or nested same-named containers never collide. Absent where the owner cannot be resolved structurally. `packageName` is constant within a file and is not plumbed. - -### 3. A separate packing-affinity view - -`same_container` is **not** added to `HunkRelationshipGraph`; doing so would change flag-off excerpts, notes, repository reads, profiles, and — through `MAX_RELATIONSHIP_EDGES_PER_HUNK` overflow — which existing edges survive. +### 1. Atoms, partitions, and fill — Plan 102, unchanged -Build a `PackingAffinity` view per file, only when packing is on: +Wrap `hunkFirstGroups()` output as atoms carrying ordered hunks, hunk count, `combinedPatchChars()` size, first source position, effective coverage, routed lenses, standalone profile, and standalone per-member context quality. Partition by `(effectiveCoverage, normalizedPlannerLensSignature)`. Direct `whole-file` and content-probed `file-diff` returns bypass packing. -- **derived** edges copied from the relationship graph: `same_symbol`, `strong` `symbol_mention`, and `strong` `planner_hint` that resolves uniquely (its normalized symbol key matches exactly one changed symbol identity in the file, mirroring `addSymbolMentionEdges()`'s existing guard); -- **structural** edges: `same_container`, for hunk pairs sharing a non-empty `ownerKey`; -- no per-hunk cap, because this view is never rendered and cannot evict prompt context. +### 2. Multi-member symbol context, fully specified -### 4. Relationship-ordered greedy fill +Constants: -Within a partition, atoms sorted by first source position: +- `PACKET_SYMBOL_CONTEXT_BUDGET = 5_000` — the share of `MAX_CONTEXT_CHARS` available to symbol source in a packed packet. +- `MIN_MEMBER_SYMBOL_CHARS = 800` — the floor below which a member is not meaningfully represented. +- `MIN_SLICED_MEMBER_CHARS = 600` — the minimum surviving emitted characters for a member to count as `sliced`. -1. Seed with the source-earliest unassigned atom. -2. Admit the **source-earliest unassigned atom connected by an affinity edge to any current member** that fits `MAX_HUNKS_PER_PACKET` and `MAX_PATCH_CHARS`. -3. If no related atom fits, admit the source-earliest unassigned atom in the partition that fits — Plan 102's rule. -4. Flush when nothing fits; seed the next packet. -5. Order packets by first source position. **Render every packet's hunks in source order**, independent of admission order. +Rules: -With no affinity edges this is byte-for-byte Plan 102's behavior, which makes the deterministic comparison exact: the replay reports how many packets differ in composition from the Plan 102 ordering, and packet *count* should differ by at most a small bin-packing residual. +1. At least `MAX_CONTEXT_CHARS - PACKET_SYMBOL_CONTEXT_BUDGET` (3,000 characters) is reserved for outline, likely-tests, and planner-hint context. Symbol source never consumes it. +2. Each distinct member primary symbol receives `floor(PACKET_SYMBOL_CONTEXT_BUDGET / memberSymbolCount)`, subject to `MIN_MEMBER_SYMBOL_CHARS`. +3. If `memberSymbolCount * MIN_MEMBER_SYMBOL_CHARS > PACKET_SYMBOL_CONTEXT_BUDGET`, the candidate is abandoned — the packet cannot represent all its members. +4. Members needing less than their share release the remainder; one redistribution pass, in source order, to members that requested more. +5. Existing adaptive/sliced selection applies per member within its share. Members render in source order. +6. **Per-member context quality is computed after `renderPacketContextText()` truncation, not before.** A member with fewer than `MIN_SLICED_MEMBER_CHARS` surviving is a candidate abandonment, never a reported `sliced`. -### 5. Symbol context follows membership +### 3. Transactional candidate evaluation -For a packed packet whose members hold more than one distinct primary symbol identity, read enclosing-symbol source for **every** member's primary symbol: allocate each `floor(symbolContextBudget / memberSymbolCount)`, preserve the existing adaptive/sliced selection per member, render in source order, and record `contextQuality` per member alongside the packet grade. +Every candidate is dry-built against a cloned relationship accumulator and a suppressed telemetry sink; nothing reaches the real graph, artifacts, or event stream until commit. Abandon when the combined packet would drop a routed lens, newly omit a planner focus note on a `high`/`critical` hunk, breach the context rules in section 2, or lower the effective profile below the maximum standalone member profile. Record each abandonment with a reason code and the atoms involved. -This is the one place Plan 102's mechanism is genuinely improved rather than reused — it shipped a packer that could drop non-primary members' surrounding source without registering it. +### 4. Profile floor and dispatch rank -### 6. Transactional candidate evaluation +Plan 102's floor verbatim: `max(derivedPackedProfile, max(standalone member profiles))` over `{ simple: 0, standard: 1, investigate: 2 }`. `dispatchRank` recomputed with the unchanged formula. -Every candidate is dry-built against a cloned relationship accumulator and a suppressed telemetry sink; nothing reaches the real graph, artifacts, or event stream until commit. A candidate is abandoned — leaving its atoms separate — when the combined packet would: +### 5. Configuration — eval and internal only -- drop a lens any member routed standalone; -- newly omit a planner focus note belonging to a `high`/`critical` hunk; -- reduce any member's context quality below `sliced`; -- lower the effective profile below the maximum standalone member profile. - -Each abandonment is recorded with a reason code and the atoms involved. - -### 7. Profile floor - -Plan 102's floor, retained verbatim: `effectiveReviewProfile = max(derivedPackedProfile, max(standalone member profiles))` over `{ simple: 0, standard: 1, investigate: 2 }`, never lexical comparison. `dispatchRank` is recomputed with the unchanged `packetDispatchRank(filePath, facts, combinedChangedLines)` formula. - -### 8. Configuration - -```yaml -review: - packCompatibleAtoms: false # mechanism gate - packMaxHunks: 5 # experiment-only; the dilution curve varies this +```ts +// CodegenieConfig["review"], resolved schema and defaults only +packCompatibleAtoms: boolean; // default false +packMaxHunks: number; // default 5, never exceeds MAX_HUNKS_PER_PACKET in shipped behavior ``` -Both in raw and resolved schemas, `CodegenieConfig`, defaults, config-source tracking, repo-safe filtering, the strict eval-case `review` schema, and `applyCaseReviewConfig()`. `packMaxHunks` exists solely to run the recall curve and never exceeds `MAX_HUNKS_PER_PACKET` in shipped behavior. Both are deleted in step 9. +Neither appears in `rawConfigSchema`, neither is repo-safe filtered, and no `codegenie.toml` can set them. They exist in the resolved config, defaults, config-source telemetry, the strict eval-case `review` schema, and `applyCaseReviewConfig()`. Both are deleted in step 9. -Separately, an **eval-only** pinned-plan seam: a planner-draw mode writing a `ReviewPlan` plus its SHA-256, and an eval-case field (`review.pinnedPlanDir`) supplying execution *k* with plan *k*, validated by hash and exact `hunkId` membership, failing closed on mismatch. No user-facing config path. It survives teardown only if independently useful to other eval work. +The pinned-plan seam is likewise eval-only: a planner-draw mode writing a `ReviewPlan` plus SHA-256, and an eval-case field `review.pinnedPlanPath` validated by hash and by exact `hunkId` membership against the current diff, failing closed on mismatch. ## Validation Strategy ### A. Free deterministic replay -Port `scripts/packet-packing-report.ts` from `7ebfd2f` down to `replay` plus `treatment`. Failure records are **structured and templated** — a closed-set `code`, structured fields, and a message rendered from a template. No raw exception text, no repository source, no hashing. - -Run over the four retained runs and assert: +Port `scripts/packet-packing-report.ts` from `7ebfd2f` down to `replay` plus `treatment`. Failure records are structured and templated — closed-set `code`, structured fields, message rendered from a template. No raw exception text, no repository source, no hashing. -- packet counts reproduce Plan 102's frozen result (96→75, 93→68, 109→85) within a documented bin-packing residual of at most 2 packets per run, attributable solely to admission order; -- every reviewable hunk appears exactly once, **in source order**; -- no atom split or reordered; no new coverage promotion; caps, file, lens, and note bounds hold; -- no lens dropped, no `high`/`critical` focus note newly omitted, no member context below `sliced` — or the candidate was abandoned and recorded; -- no effective profile or budget below the maximum standalone member value; -- every dispatch rank equals the existing formula; -- flag off is artifact-identical to current behavior, **including `hunk-relationships.json`**, proving the affinity view never wrote back. +Assert per run: packet counts reproduce Plan 102's frozen result within 2; every hunk appears exactly once in source order; no atom split; no coverage promotion; caps hold; no lens or high-priority note lost without a recorded abandonment; no member below `MIN_SLICED_MEMBER_CHARS`; no profile or budget below the standalone maximum; dispatch ranks match the formula; flag-off is artifact-identical including `hunk-relationships.json`. -Report per run: packets off→on, composition changes versus Plan 102 ordering, affinity edges split `derived`/`structural`, abandoned candidates by reason, per-member context-quality distribution, notes dropped, excerpts internalized, and scheduling movement. - -**Pre-registered gates**, counted on the two distinct diffs: +**Fixed-slot hunk yield is the primary gate.** Sort packed packets by the full Stage-7 scheduling tuple `(priority, coverage, dispatchRank, input order)`, take the first 57 — the motivating run's observed dispatch capacity — and count reviewable hunks covered. Baseline is 89. | Gate | Threshold | | --- | --- | +| Fixed-slot hunk yield at 57 packets | `>= 102` hunks (≥15% over baseline 89) | | Packet reduction | `>= 20%` per distinct diff | -| Deviation from Plan 102's frozen packet count | `<= 2` packets per run | -| Packets whose composition changed under relationship ordering | `>= 2` per distinct diff | -| Members whose context quality falls below `sliced` | `0` | +| Deviation from Plan 102's frozen counts | `<= 2` packets per run | +| Members below `MIN_SLICED_MEMBER_CHARS` | `0` | | New coverage promotions | `0` | -Missing the third gate means relationship ordering changes nothing measurable — drop `ownerKey`, `same_container`, and the affinity view, and ship Plan 102's ordering unchanged. That is a cheap, free deletion, not a plan failure. +Also report, off versus on: per-member profile upgrades, total tool-call allowance, total context characters, and projected cost and service time per reviewed hunk. The measured yield ratio `baselineYield / packedYield` becomes the break-even bar for phase B. + +### B. The packet-size recall curve (paid) + +**Location.** `evals/packet-dilution/` in this repository, not the private evals repo. The fixture is synthetic Go with nothing proprietary, and `evals/skill-semantics/` is precedent for a real-model suite living here. This makes the curve a standing asset re-runnable whenever anyone proposes changing caps. Add `evals/packet-dilution/logs/` — and the missing `evals/skill-semantics/logs/` — to `.gitignore`. -### B. The recall-versus-packet-size curve (paid) +**Fixture.** One Go file, fifteen separated single-hunk changes, far enough apart that today's grouper yields fifteen atoms. It carries: -This is the only paid question. It is not about which predicate to use; it is whether review quality survives packing at all. +- **three independent bugs** at atom positions 1, 5, and 8 — under cap 5 that is packet 1 position 1, packet 1 position 5, and packet 2 position 3, so first/last/middle positions are all covered; +- **one safe-but-suspicious change** as a `should_not_find` control, so a rise in false positives is visible; +- eleven ordinary safe changes. -**Fixture.** One Go file under `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/recall/` with about ten separated single-hunk changes — far enough apart that today's grouper yields one atom each — sharing coverage and lenses so they all land in one partition. One hunk carries a locally detectable bug; the rest are safe. Expectations constrain path, line range, and failure mode, and **accept any plausible category**: Plan 102 recorded three executions that found its bug and scored `partial-match` solely because the expectation demanded `correctness` and the reviewer said `security`. +Fixture-quality rules, each learned from a Plan 102 failure: -**Arms.** Identical cases differing only in `packMaxHunks`: 1 (one atom per packet — today's behavior), 3, and 5 (the shipping cap). `repeat: 6`, cache off. Each repeat draws one Stage-5 plan, hashed, consumed by all three arms, so packet size is the only within-repeat difference while planner variance is still sampled across repeats. +1. **Transplant bugs already proven detectable.** Take them from `evals/skill-semantics/`, where the existing suite already demonstrates the harness finds them. Do not invent new bugs whose detectability is unknown. +2. **Validate expectations with the fake provider first**, for `$0`. `fake-runner.ts` emits a finding from a trigger line; confirm each `should_find` matcher fires on the expected finding shape before any real call. +3. **Each bug must be one the planner grades ordinarily.** If a bug is obvious enough to draw a `deep` grade on its own, it lands in a different partition and never packs. +4. **Constrain path, line range, and failure mode — never category.** Plan 102 lost three executions that found its bug because the expectation demanded `correctness` and the reviewer said `security`. +5. **Prove treatment model-free before paying.** Build Stage 6 against the frozen plan at caps 1, 3, and 5 and require exactly 15, 5, and 3 packets, with each bug's atom in a packet of exactly 1, 3, and 5 source atoms. Any miss is a fixture defect: fix the fixture, never the sample. -**Pre-registered decision table**, where `R1`, `R3`, `R5` are hits out of 6: +**The frozen plan is hand-authored, not drawn.** Authoring it makes coverage and lens assignment deterministic, so all fifteen atoms are guaranteed into one partition; removes planner variance so packet size is genuinely the only variable; eliminates any temptation to redraw unfavourable plans; and costs nothing. Record one real planner draw alongside it and diff the two, so the authored plan's realism is documented rather than assumed. + +**Arms.** Three cases differing only in `packMaxHunks` — 1, 3, 5 — at `repeat: 6`, cache off, all pinned to the same authored plan. Three bugs × six repeats = **18 observations per arm**. + +**Decision rule.** Let `R1`, `R3`, `R5` be hits out of 18 and `B` the break-even ratio measured in phase A (`baselineYield / packedYield`; ≈ 0.824 at the projected 89/108). | Condition | Outcome | | --- | --- | -| `R1 <= 1` | **Void.** The bug is not reliably findable even unpacked; redesign the fixture. | -| `R5 >= R1 - 1` | **Pass.** Packing is non-inferior at the shipping cap. | -| `R5 <= R1 - 2`, or `R1 > R3 > R5` | **Fail.** Recall degrades with packet size; do not ship packing at any predicate. | -| otherwise | **Inconclusive.** Record; do not ship. | - -This is a non-inferiority test, and deliberately so: the *benefit* — roughly 19 more reviewed hunks per run — is already measured deterministically, so packing only has to avoid harming per-packet recall. The first draft demanded superiority because it was claiming a quality benefit; this plan claims a coverage benefit and a quality guard, which is the honest framing. +| `R1 < 15` | **Void.** Baseline too unreliable to measure a ratio against; fix the fixture. | +| `R5 / R1 >= B` and no monotone decline `R1 > R3 > R5` | **Pass.** | +| `R5 / R1 >= B` but `R1 > R3 > R5` | **Stop for investigation.** A consistent decline across all three sizes is a dilution signature even when the ratio clears; extend repeats on the affected arms before deciding. | +| `R5 / R1 < B` | **Fail.** Packing costs more recall than the coverage it buys. | -Also report tokens, model-service time, and cost per reviewed hunk per arm — the efficiency side of the same curve. +`R3` therefore has a real role — it distinguishes a cliff from a gradient — and there is no unreachable branch. -Relationship ordering gets **no paid arm.** It changes composition in a handful of packets per run, an effect far too small to resolve at `n = 6`. It cannot reduce packet count or violate any invariant, it is validated deterministically in phase A, and it is adopted or dropped on that basis alone. Do not spend paid budget on an effect you cannot measure. +**Honest limits.** Eighteen observations per arm can detect a large regression; it cannot resolve, say, 100% versus 85% with confidence. The gate is a screen against material harm, not a certificate of equivalence, and the reconciliation note must say so. Report per-arm cost, tokens, and model-service time per reviewed hunk alongside recall. -### C. Collateral and capacity +### C. Collateral 1. `evals/fixtures/` flag-off and flag-on at `repeat: 1`, fake provider. Every expectation, packet invariant, and config parse must match. -2. `evals/skill-semantics/` (TypeScript, Python, Solidity) flag-off and flag-on at `repeat: 1`. An A-pass/B-fail difference is stop-for-investigation; rerun only that case at `repeat: 3`. -3. One production-shaped capacity pair on the pinned `trails-api` diff at concurrency 6, flag-off and flag-on, reporting reviewed-hunk set, packet count, model-service time, tokens, and cost per reviewed hunk. Its purpose is to confirm the run-level coverage gain the deterministic replay predicts. +2. **Treated cross-language packet-shape cases** under `evals/packet-dilution/`: small TypeScript, Python, and Solidity fixtures each with at least three compatible same-file atoms, run under the fake provider, asserting packed packet shape and per-member context presence. Existing `evals/skill-semantics/` files are 10–15 lines and yield one atom each, so they cannot exercise packing; they remain a flag-off/on regression check only. +3. One production-shaped capacity pair on the pinned `trails-api` diff at concurrency 6, flag-off and flag-on, confirming the fixed-slot yield predicted in phase A against a real run. **Required before shipping**, not optional. ### Cost discipline -No paid call before `approvedValidationCostUSD` is recorded here; the owner supplies the amount. +No paid call before `approvedValidationCostUSD` is recorded here. | Phase | Executions | Projection | | --- | ---: | ---: | -| Dilution preflight (1 plan draw + 3 arms) | 4 | ~$3 | -| Dilution curve, 6 repeats × 3 arms | 18 | ~$14 | -| `evals/fixtures/` (fake provider) | — | $0 | -| `evals/skill-semantics/`, both arms | 6 | ~$5 | +| Fake-provider expectation validation | — | $0 | +| One reference planner draw | 1 | ~$1 | +| Recall curve, 6 repeats × 3 arms | 18 | ~$14 | +| `evals/fixtures/` and cross-language shape cases | — | $0 | | Production capacity pair | 2 | ~$50 | -| Contingency 25% | — | ~$18 | -| **Reservation** | | **~$90** | +| Contingency 25% | — | ~$16 | +| **Reservation** | | **~$81** | -The production pair dominates and runs last, only after a pass. If the ceiling is tight, phases 1–2 alone cost ~$17 and decide whether packing ships. +Phases through the curve cost ~$15 and decide whether packing ships; the capacity pair runs last and only after a pass. ## In-Scope Files -- `src/repo/symbol-extraction.ts` — `ownerType`, structural `ownerKey`. -- `src/pipeline/packet-builder.ts` — atoms, partitions, affinity view, relationship-ordered fill, multi-member symbol context, transactional candidates, profile floor, telemetry. +- `src/pipeline/packet-builder.ts` — atoms, partitions, fill, multi-member symbol context, transactional candidates, profile floor, telemetry. - `src/pipeline/review-runner.ts` — eval-only pinned-plan seam. -- `src/types.ts` — `ownerType`/`ownerKey`, per-member context-quality telemetry, both temporary config fields. -- `src/config/schema.ts`, `src/config/config-loader.ts`, `src/evals/eval-runner.ts` — flag plumbing and `pinnedPlanDir`. -- `scripts/packet-packing-report.ts`, `tests/packet-packing-report.test.ts` — `replay` and `treatment` with templated failure records. -- `tests/pipeline-phase5.test.ts`, `tests/config-loader.test.ts`, `tests/evals.test.ts`, and the symbol-extraction suite. -- `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/` — fixture, cases, pinned plans, immutable logs, reports. +- `src/types.ts` — per-member context-quality telemetry, both temporary settings. +- `src/config/schema.ts`, `src/config/config-loader.ts`, `src/evals/eval-runner.ts` — resolved-config and eval plumbing only. +- `scripts/packet-packing-report.ts`, `tests/packet-packing-report.test.ts` — `replay` and `treatment`. +- `tests/pipeline-phase5.test.ts`, `tests/evals.test.ts`, `tests/config-loader.test.ts`. +- `evals/packet-dilution/` — Go recall fixture, cross-language shape cases, authored plan. +- `.gitignore` — `evals/packet-dilution/logs/` and the missing `evals/skill-semantics/logs/`. - `specs/plans/README.md` and affected `specs/project/` docs. ## Non-Goals -- **Raising `MAX_HUNKS_PER_PACKET` or `MAX_PATCH_CHARS`.** The sweep shows 5 captures ~80% of achievable compression; 10 adds ~5 points before saturating and enlarges untested recall risk. `packMaxHunks` exists only to measure the curve and never ships above 5. -- **Relationship-required packing as a membership predicate.** Measured at 4.2% versus 21.9%; retired by evidence. -- **Cross-coverage packing and any coverage-promotion machinery.** 2 of 13 candidate pairs, zero of them structural. -- **Reordering the stage flow.** The planner grades related siblings consistently in 132 of 134 measured pairs; there is nothing for a group-aware dossier to fix. -- Cross-file packing — the packet schema is file-scoped, and cross-file relatedness is already served by related-context excerpts. -- Tool-budget changes of any kind, including Plan 102's retired `atom-scaled` arm. -- Changing prompt templates, attention-note ordering, `canJoinGroup()` semantics, Plan 100's dispatch policy, or Plan 92's escalation rules. -- Exposing either config field as user-facing surface, or leaving either dark after step 9. +- **Relationship-based packing, in any form.** Measured at 4.2% as a predicate and as a no-op as an ordering preference. No affinity view, no `ownerKey`, no `same_container`. +- **Raising `MAX_HUNKS_PER_PACKET` or `MAX_PATCH_CHARS`.** The sweep shows 5 captures ~80% of achievable compression. `packMaxHunks` exists only to measure the curve and never ships above 5. +- **Coverage promotion.** 2 of 13 candidate pairs, zero of them structural. +- **Reordering the stage flow.** The planner grades related siblings consistently in 132 of 134 measured pairs. +- Cross-file packing; the packet schema is file-scoped. +- Tool-budget changes of any kind. +- Changing prompt templates, attention-note ordering, `canJoinGroup()`, Plan 100's dispatch policy, or Plan 92's escalation rules. +- Any user-facing configuration surface, or leaving either setting dark after step 9. ## Implementation Steps -1. Reconcile the drift and working-tree checks. Add `ownerType` and structural `ownerKey` to `HunkSymbolFacts`. +1. Reconcile drift and working-tree checks. Add both settings to the resolved config, defaults, source telemetry, the strict eval-case schema, and `applyCaseReviewConfig()` — **not** to `rawConfigSchema` or repo-safe filtering. Add both `.gitignore` log entries. - **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "container identity"` → exit 0; all five real adapters resolve owners, generic-adapter files resolve none, and two same-named classes in one file produce distinct keys. -2. Add both dark config fields end-to-end, including strict eval schema, `applyCaseReviewConfig()`, defaults, repo-safe filtering, and source tracking. - - **Verify:** `pnpm exec vitest run tests/config-loader.test.ts tests/evals.test.ts` → exit 0; defaults `false`/`5`, overrides apply, `packMaxHunks` above 5 is rejected. -3. Wrap `hunkFirstGroups()` output as atoms with standalone coverage, lenses, profile, and context quality captured through the scratch path; add flag-off golden parity. + **Verify:** `pnpm exec vitest run tests/config-loader.test.ts tests/evals.test.ts` → exit 0; defaults `false`/`5`; eval overrides apply; `packMaxHunks > 5` rejected; a `codegenie.toml` setting either key fails strict parsing. +2. Wrap `hunkFirstGroups()` output as atoms, capturing standalone coverage, lenses, profile, and per-member context quality; add flag-off golden parity. **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "packet atom|flag-off parity|whole-file bypass"` → exit 0. -4. Implement Plan 102 partitions and greedy fill, with no relationship ordering yet, and prove it reproduces Plan 102's frozen counts. +3. Implement partitions and source-order greedy fill; reproduce Plan 102's frozen counts. - **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "compatible partition"` → exit 0; hunk bijection, source-order rendering, cap splits, interleaved partitions, degradation merge, and the exact dispatch-rank formula. -5. Build the `PackingAffinity` view and add relationship-ordered admission. + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "compatible partition"` → exit 0; hunk bijection, within-packet source ordering, packet ordering by earliest member, cap splits, interleaved partitions, degradation merge, exact dispatch rank. +4. Implement multi-member symbol context per section 2. - **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "packing affinity|relationship ordering"` → exit 0; the relationship graph and all flag-off artifacts are unchanged; ambiguous repeated symbol names produce no edge; with no edges the fill is identical to step 4; with edges, composition changes while hunks still render in source order. -6. Implement multi-member symbol context and per-member context-quality recording. + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "member symbol context"` → exit 0; the 3,000-character non-symbol reserve is never consumed; equal shares with redistribution; a candidate needing more than the budget floor for all members is abandoned; quality is computed after final truncation and a member surviving under 600 characters aborts rather than reporting `sliced`. +5. Implement transactional candidate evaluation and the profile floor. - **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "member symbol context"` → exit 0; a two-symbol packet carries both symbols; `full`→`sliced` is reported and allowed; a member falling to `outline_only` aborts the candidate. -7. Implement transactional candidate evaluation and the profile floor. + **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "candidate rejection|profile floor"` → exit 0; abandoned candidates leave no telemetry, graph, or artifact residue. +6. Add the pinned-plan seam and planner-draw mode. - **Verify:** `pnpm exec vitest run tests/pipeline-phase5.test.ts -t "candidate rejection|profile floor"` → exit 0; abandoned candidates leave no residue; lens-drop, `high`-priority note-drop, and context-collapse candidates abort; the floor holds when an absorbed strong edge would derive lower. -8. Add the pinned-plan seam, port the report script, and run the four-run replay. Record the table and every gate here. + **Verify:** `pnpm exec vitest run tests/evals.test.ts -t "pinned plan"` → exit 0; a pinned plan reproduces identical Stage-6 inputs across runs; hash and `hunkId` mismatches fail closed; no user-facing path reaches it. +7. Port the report script and run the four-run replay. - **Verify:** `pnpm exec vitest run tests/packet-packing-report.test.ts tests/evals.test.ts -t "pinned plan"` → exit 0; the replay exits 0 with `modelCallsObserved: 0` on every row and every section-A gate satisfied. If the composition gate fails, delete the affinity view and container identity, re-run, and continue with Plan 102 ordering. -9. Record the ceiling, build the dilution fixture, and run: preflight, the 6×3 curve, `evals/fixtures/`, `evals/skill-semantics/`, then the capacity pair. Apply the decision table verbatim. + **Verify:** `pnpm exec vitest run tests/packet-packing-report.test.ts` → exit 0; the replay exits 0 with `modelCallsObserved: 0` and every section-A gate satisfied, including fixed-slot hunk yield `>= 102`. Record the measured break-even ratio `B`. +8. Build the fixture, validate expectations under the fake provider, author and hash the plan, prove treatment model-free, then record the ceiling and run: reference planner draw, the 6×3 curve, `evals/fixtures/`, cross-language shape cases, and the capacity pair. Apply the decision table verbatim. - **Verify:** each repeat's three arms consumed one hash-verified plan; the decision table yields Pass, Fail, Void, or Inconclusive with no discretion; collateral shows no reproducible regression; spend stays within the ceiling. -10. Record the decision and tear down in a dedicated commit. + **Verify:** treatment proof shows 15/5/3 packets with each bug at 1/3/5 source atoms; all three arms consumed the same plan hash; the decision table yields Pass, Fail, Void, or Stop-for-investigation with no discretion; the capacity pair confirms the phase-A yield prediction; spend stays within the ceiling. +9. Record the decision and tear down in a dedicated commit. - **Preserve evidence first:** copy every produced JSON report and the pinned plans into `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/reports/` with a `manifest.sha256`. Mark unreached phases `not_run` with their stopping reason. + **Preserve evidence first:** copy every produced JSON report and the authored plan into `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/reports/`, extending its existing `manifest.sha256`. Do not modify Plan 102's manifest, which its reconciliation note describes as covering exactly three files. Mark unreached phases `not_run` with their stopping reason. - **If Pass:** make packing unconditional at cap 5; delete both config fields and the unpacked path. Keep atoms, partitions, the profile floor, multi-member symbol context, transactional rejection, and — if the composition gate passed — `ownerKey`, `same_container`, and the affinity view. Reduce the report script to a golden check against frozen packet shapes. + **If Pass:** make packing unconditional at cap 5; delete both settings and the unpacked path. Keep atoms, partitions, the profile floor, multi-member symbol context, and transactional rejection. Reduce the report script to a golden check. Keep `evals/packet-dilution/` as a standing suite with the shipped cap only, and retain the curve in the reconciliation note. - **If Fail, Inconclusive, or Void:** delete the packing pass, both fields, the atom wrapper, the affinity view, and the report script with its tests. Multi-member symbol context survives only if the replay showed it improving context quality with packing off. A Void permits one fixture redesign and rerun; a second Void takes the failure branch. Do not leave the feature dark. + **If Fail, Void, or unresolved Stop-for-investigation:** delete the packing pass, both settings, the atom wrapper, and the report script with its tests. Multi-member symbol context survives only if the replay showed it improving context quality with packing off. Keep the fixture and the measured curve — the curve is the durable asset regardless of outcome. Do not leave the feature dark. - **Verify:** `rg -n "packCompatibleAtoms|packMaxHunks" src scripts tests evals` → exit 1; the manifest verifies; the note records the decision table result, actual spend, and the `n = 6` limitation. -11. Run the complete repository gate. + **Verify:** `rg -n "packCompatibleAtoms|packMaxHunks" src scripts tests evals` → exit 1; both manifests verify; the note records the decision, actual spend, and the resolution limit of 18 observations per arm. +10. Run the complete repository gate. **Verify:** `pnpm run check && pnpm test && pnpm build` → exit 0. @@ -371,35 +334,74 @@ pnpm run check pnpm exec vitest run tests/pipeline-phase5.test.ts tests/config-loader.test.ts tests/evals.test.ts tests/packet-packing-report.test.ts ``` +Artifact diagnostics (reproduces this plan's Evidence section, no model calls): + +```bash +RUNS=/home/peter/Dev/0xsequence/trails-api/.codegenie/runs +node scripts/packing-diagnostics.mjs pairs $RUNS/20260724-184952-dca8d870 +node scripts/packing-diagnostics.mjs sweep $RUNS/20260724-184952-dca8d870 +node scripts/packing-diagnostics.mjs simulate $RUNS/20260724-184952-dca8d870 --predicate source +``` + Deterministic four-run replay: ```bash pnpm exec tsx scripts/packet-packing-report.ts replay \ --repo /home/peter/Dev/0xsequence/trails-api \ - --run /home/peter/Dev/0xsequence/trails-api/.codegenie/runs/20260724-135818-740d73f2 \ - --run /home/peter/Dev/0xsequence/trails-api/.codegenie/runs/20260724-150405-fe1548ae \ - --run /home/peter/Dev/0xsequence/trails-api/.codegenie/runs/20260724-162739-81f806a6 \ - --run /home/peter/Dev/0xsequence/trails-api/.codegenie/runs/20260724-184952-dca8d870 \ - --compare-ordering source \ + --run $RUNS/20260724-135818-740d73f2 \ + --run $RUNS/20260724-150405-fe1548ae \ + --run $RUNS/20260724-162739-81f806a6 \ + --run $RUNS/20260724-184952-dca8d870 \ + --dispatch-slots 57 \ --distinct-diffs \ --output /tmp/plan103-packing-shape.json ``` -Expected: exit 0, four rows with duplicate diffs marked and counted once, zero model calls, packet counts within 2 of Plan 102's frozen 75/68/85, and a composition-change block. - -Dilution curve: +Fixture validation, planner draw, and the paid curve: ```bash -pnpm dev eval \ - --eval-dir /home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/recall \ - --no-cache +# 1. expectation wiring, fake provider, $0 +pnpm dev eval --eval-dir evals/packet-dilution/shape --no-cache + +# 2. one reference planner draw, for realism comparison against the authored plan +pnpm dev plan --repo evals/packet-dilution/repos/dilution \ + --base main --branch feature \ + --output evals/packet-dilution/plans/reference-draw.json + +# 3. model-free treatment proof at each cap +pnpm exec tsx scripts/packet-packing-report.ts treatment \ + --pinned-plan evals/packet-dilution/plans/frozen.json \ + --expect-packets 15,5,3 \ + --expect-target-atoms 1,3,5 \ + --output /tmp/plan103-treatment-proof.json + +# 4. the paid curve +pnpm dev eval --eval-dir evals/packet-dilution/recall --no-cache pnpm exec tsx scripts/packet-packing-report.ts treatment \ - --logs /home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/recall/logs \ + --logs evals/packet-dilution/recall/logs \ --cohort \ --expected-repeats 6 \ --require-pinned-plan \ - --output /tmp/plan103-dilution-curve.json + --break-even \ + --output /tmp/plan103-recall-curve.json +``` + +Collateral and capacity: + +```bash +pnpm dev eval --eval-dir /tmp/plan103-collateral/fixtures-off --no-cache +pnpm dev eval --eval-dir /tmp/plan103-collateral/fixtures-on --no-cache +pnpm dev eval --eval-dir /home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/capacity --no-cache +``` + +Evidence preservation and verification: + +```bash +D=/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/reports +cp /tmp/plan103-*.json "$D"/ +(cd "$D" && sha256sum plan103-*.json > manifest.sha256 && sha256sum -c manifest.sha256) +(cd "$D/../../packet-packing/reports" && sha256sum -c manifest.sha256) # Plan 102 evidence unchanged ``` Note the `pnpm dev eval` spelling: under pnpm 11 a literal `--` reaches Commander and is rejected before suite allocation. @@ -407,35 +409,38 @@ Note the `pnpm dev eval` spelling: under pnpm 11 a literal `--` reaches Commande ## Acceptance Criteria - Flag off produces byte-identical packet artifacts, IDs, order, profiles, context, budgets, and `hunk-relationships.json`. -- Flag on reproduces Plan 102's packet counts within 2 per run, with every reviewable hunk appearing exactly once in source order, under unchanged caps and with zero coverage promotion. -- Relationship ordering changes composition, never render order, and never increases packet count beyond the documented bin-packing residual. -- No lens dropped, no `high`/`critical` focus note newly omitted, no member context below `sliced`; violating candidates are abandoned transactionally and recorded. -- Every effective profile is at least the maximum standalone member profile; dispatch ranks use Plan 100's unchanged formula. -- The replay exits 0 with zero model calls and satisfies every pre-registered gate on both distinct diffs. -- Every dilution repeat's three arms consume one hash-verified Stage-5 plan; the decision table is applied without discretion. -- Collateral suites show no reproducible regression; the capacity pair confirms the predicted run-level coverage gain. +- Flag on reproduces Plan 102's packet counts within 2 per run, every hunk appearing exactly once, hunks ordered within packets by file position and packets by earliest member, under unchanged caps with zero coverage promotion. +- Every member atom's primary symbol source is present after final rendering at or above `MIN_SLICED_MEMBER_CHARS`, or its candidate was abandoned and recorded. +- The 3,000-character non-symbol context reserve is never consumed by symbol source. +- No lens dropped and no `high`/`critical` focus note newly omitted without a recorded abandonment; every effective profile at least the standalone maximum; dispatch ranks unchanged in formula. +- Fixed-slot hunk yield at 57 packets is at least 102, and the measured break-even ratio is recorded and used as phase B's bar. +- Per-member profile upgrades, tool-call allowance, context characters, and cost and service time per reviewed hunk are reported off versus on. +- All three curve arms consume one hash-verified authored plan; treatment is proven model-free at 15/5/3 packets before any paid call; the decision table is applied without discretion. +- Cross-language shape cases prove packed context on TypeScript, Python, and Solidity; `evals/fixtures/` shows no regression; the capacity pair confirms the predicted yield. - Paid validation never begins without a recorded ceiling; every phase records actual and projected spend. -- Report failures carry templated, structured messages with no raw exception text, no repository source, and no hashing. -- Teardown leaves one product path with no surviving flag, and preserves every report and pinned plan under a verified manifest. -- Checks, tests, and build pass; the decision and the `n = 6` limitation are documented. +- Report failures carry templated structured messages with no raw exception text, repository source, or hashing. +- Teardown leaves one product path with no surviving flag; both evidence manifests verify; Plan 102's manifest is unmodified. +- Checks, tests, and build pass; the decision and the 18-observation resolution limit are documented. ## Stop Conditions -- Packet reduction below 20% per distinct diff, or deviation from Plan 102's frozen counts above 2 packets: the port is wrong, not the mechanism. Fix before proceeding. -- Composition-change gate below 2 packets per distinct diff: delete the affinity view and container identity and continue with Plan 102 ordering. Not a plan failure. -- Any atom split or reordered, hunk lost or duplicated, cap exceeded, coverage promoted, profile or budget downgraded, or member context collapsed without abandonment: fix the deterministic design before spending anything. -- The affinity view mutates the relationship graph, or flag-off artifacts change: stop; isolation is the basis of every later comparison. -- A dilution repeat's arms consumed different Stage-5 plans, or a pinned-plan hash or `hunkId` set fails validation: discard that repeat and rerun it. -- The decision table returns Fail: packing does not ship at any predicate or cap. Take the failure branch and record the curve. -- The decision table returns Void twice: take the failure branch rather than redesigning a third time. -- Any proposal to raise `MAX_HUNKS_PER_PACKET` to recover compression: reject it here. It is a separate question with its own untested recall risk, and the sweep says it is worth ~5 points. -- Actual plus projected spend exceeds the approved ceiling: stop for explicit approval. Do not weaken a gate to fit a budget. -- "Keep packing dark" always means *do not ship it in this iteration*; every such outcome terminates in step 10's failure branch in the same change. +- Fixed-slot hunk yield below 102, packet reduction below 20%, or deviation from Plan 102's frozen counts above 2: the port is wrong. Fix before spending anything. +- Any atom split or reordered, hunk lost or duplicated, cap exceeded, coverage promoted, profile or budget downgraded, or member context below the minimum without abandonment: fix the deterministic design first. +- Flag-off artifacts change in any respect: stop; parity is the basis of every later comparison. +- The model-free treatment proof does not yield 15/5/3 packets with the target atoms at 1/3/5: fix the fixture and re-prove. Never adjust the sample to fit. +- Any arm consumed a different plan hash: discard and rerun that repeat. +- `R1 < 15` of 18: the baseline is too unreliable to measure against. Void; fix the fixture rather than lowering the bar. +- `R5 / R1 < B`: packing costs more recall than the coverage it buys. Fail and take the teardown branch — do not renegotiate `B` after seeing the result. +- A monotone decline `R1 > R3 > R5` that survives extended repeats: treat as a dilution signature and fail, even if the ratio clears. +- Any proposal to raise `MAX_HUNKS_PER_PACKET`, revive relationship signals, or add coverage promotion to recover margin: reject here. Each has a measured Non-Goal entry. +- Actual plus projected spend exceeds the approved ceiling: stop for explicit approval. +- "Keep packing dark" always means *do not ship it in this iteration*; every such outcome terminates in step 9's failure branch in the same change. ## Maintenance Notes -- Any future change to `hunkFirstGroups()`, `canJoinGroup()`, the relationship edge builders, `buildRelatedChangedContext()`, `readEnclosingSymbolSource()`, `computeSymbolContextBudget()`, `packetReviewProfile()`, packet caps, `packetDispatchRank()`, or `toolBudget()` invalidates the recorded shape, context, and order baselines; rerun the replay first. -- Reviewers should scrutinize affinity-view isolation, composition changes, per-member context quality, abandonment reasons, and pinned-plan provenance — not packet count, which is Plan 102's already-validated result. -- Plan 102's preserved reports and paid logs under `codegenie-private-evals/trails-api/packet-packing/` remain the record of its failed fixture design and must not be edited or deleted. -- The recall-versus-packet-size curve produced here is the reusable asset. Any future proposal to raise caps, pack cross-file, or pack cross-coverage should extend that curve rather than assume it. -- Commit hashes in this document are rebase-unstable. Locate referenced work by commit subject when a hash does not resolve. +- Any change to `hunkFirstGroups()`, `canJoinGroup()`, `readEnclosingSymbolSource()`, `computeSymbolContextBudget()`, `packetReviewProfile()`, packet caps, `packetDispatchRank()`, or `toolBudget()` invalidates the recorded shape, context, and yield baselines; rerun the replay first. +- The packet-size recall curve is the durable asset from this plan, independent of whether packing ships. Any future proposal to raise caps, pack cross-file, or pack cross-coverage should extend the curve rather than assume it. +- `scripts/packing-diagnostics.mjs` reproduces the relationship and cap-sweep analyses that shaped this plan. Its approximations are documented in its header; the authoritative measurement is the report script's replay. +- Plan 102's preserved reports and paid logs under `codegenie-private-evals/trails-api/packet-packing/` record its failed fixture design and must not be edited. This plan's evidence lives under `packet-dilution/reports/` with its own manifest. +- Reviewers should scrutinize per-member context after final rendering, abandonment reasons, fixed-slot yield, and treatment proof — not packet count, which is Plan 102's already-validated result. +- Commit hashes here are rebase-unstable; locate referenced work by commit subject when a hash does not resolve. diff --git a/specs/plans/README.md b/specs/plans/README.md index 052997c..7137201 100644 --- a/specs/plans/README.md +++ b/specs/plans/README.md @@ -106,4 +106,4 @@ This directory tracks implementation plans for confirmed improvements. Status va | 100 | COMPLETE | Issue 100: Short Hunk IDs — Planner Coverage Survival and Dispatch Resilience | | 101 | IN PROGRESS (paid semantic A/B pending) | Issue 101: Exact Skill Provenance and Evidence-Gated Bundled-Skill Revision | | 102 | COMPLETE (failed gate; baseline restored) | Issue 102: Same-File Packet Packing | -| 103 | PENDING | Issue 103: Compatible-Atom Packing with Relationship-Ordered Composition | +| 103 | PENDING | Issue 103: Compatible-Atom Packet Packing and the Packet-Size Recall Curve | From cc5558b692d759c8ad0465a9266928c440ba437f Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 12:34:42 -0400 Subject: [PATCH 10/24] docs(plans): close Plan 103 contract gaps found in review Five defects, all resolved before implementation. No-symbol atoms were undefined. The context contract required every member's primary-symbol source to survive 600 characters, but config, docs, and generic-adapter hunks have no resolvable primary symbol - readEnclosingSymbolSource returns empty and their standalone quality is outline_only or path_only. As written, every packet containing one would have been rejected, which is exactly the file shape where compatible atoms are most abundant. Only symbol-bearing members now participate in the allocation; symbol-less members are exempt from the floor and must instead retain their standalone quality. The reference planner draw had no gate. Recording a divergence between the authored plan and a real draw is not enough - if production would rarely create a 15-atom compatible partition, the experiment measures a shape that does not occur. Three draws are now taken and at least two must place every target in a partition of three or more compatible atoms, or the fixture is invalid before any reviewer call. Two bugs shared a cap-5 packet. Positions 1, 5, 8 put the first two in packet [1-5], so two opportunities collapsed into one conversation and that arm reviewed an unnaturally bug-dense packet. Positions are now 1, 10, 13, which keeps all three in separate packets at both cap 3 and cap 5 while retaining first/last/middle placement. The 18 opportunities are also described as clustered rather than independent, reported per bug, and a per-bug collapse rule stops an aggregate from hiding one target falling from reliable detection to zero. Void and extension were discretionary and mutually contradictory - the decision table said fix the fixture, teardown said remove packing immediately. Both are now bounded and preregistered: one fixture redesign after a first Void, one six-repeat extension across all three arms after a first monotone decline, the same rule re-applied to combined totals, and a second occurrence of either is a Fail. Both contingencies are budgeted, raising the reservation to ~$119. The fixed-slot estimator was unreconciled. It is now validated: applying the worker-runner sort to the flag-off packets and taking the first N reproduces exactly the 89 hunks the run actually reviewed, with 55 of 56 packets in common. Reconciliation is a gate before the estimator may define B. The measurement also corrected the slot count - artifacts record 40 not-dispatched of 96, so the baseline is 56 dispatched packets, not the 57 in Plan 102's note. Also: production capacity confirmation is required rather than optional; pinned plans validate schema and diff identity, not just hash and hunk IDs; preservation commands copy the frozen plan, all reference draws, capacity reports, and a not_run ledger before regenerating the manifest; and diagnostics reproduction covers all three distinct diffs. scripts/packing-diagnostics.mjs now applies lens-signature compatibility in its pair classifier, which it previously omitted. On dca8d870 that reclassifies 13 pairs as 2 coverage / 2 lenses / 6 caps / 3 eligible, strengthening rather than weakening the case against relationship packing. Preserved evidence regenerated; Plan 102's manifest untouched. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/packing-diagnostics.mjs | 6 +- ...-103-relationship-driven-packet-packing.md | 133 ++++++++++++------ 2 files changed, 97 insertions(+), 42 deletions(-) diff --git a/scripts/packing-diagnostics.mjs b/scripts/packing-diagnostics.mjs index 18c1ca9..cc574dd 100644 --- a/scripts/packing-diagnostics.mjs +++ b/scripts/packing-diagnostics.mjs @@ -171,17 +171,21 @@ function bypassesPacking(atom) { // Which related-but-split atom pairs exist, and what prevents each from packing. function pairs(run) { const { byId, pairSources } = affinity(run); - const summary = { coverage: 0, caps: 0, bypass: 0, eligible: 0 }; + const summary = { coverage: 0, lenses: 0, caps: 0, bypass: 0, eligible: 0 }; const detail = []; for (const [key, sources] of pairSources) { const [x, y] = key.split("|"); const a = byId.get(x); const b = byId.get(y); + // Checked in the same order the production predicate applies them, so the + // first blocker reported is the one that actually decides the pair. let blocker; if (bypassesPacking(a) || bypassesPacking(b)) { blocker = "bypass"; } else if (a.coverage !== b.coverage) { blocker = "coverage"; + } else if (a.lensSignature !== b.lensSignature) { + blocker = "lenses"; } else if (a.hunkCount + b.hunkCount > DEFAULT_MAX_HUNKS || a.patchChars + b.patchChars > DEFAULT_MAX_PATCH) { blocker = "caps"; } else { diff --git a/specs/plans/103-issue-103-relationship-driven-packet-packing.md b/specs/plans/103-issue-103-relationship-driven-packet-packing.md index 87ce6ee..55e2143 100644 --- a/specs/plans/103-issue-103-relationship-driven-packet-packing.md +++ b/specs/plans/103-issue-103-relationship-driven-packet-packing.md @@ -11,7 +11,7 @@ Recommended priority: after Plan 101's paid A/B settles. ### The problem -Reviews run out of time before they run out of code. On the motivating pull request, Stage 6 produced **96 packets** covering 142 changed hunks. Only 57 packets were dispatched before the deadline, so **53 hunks received no review at all**. +Reviews run out of time before they run out of code. On the motivating pull request, Stage 6 produced **96 packets** covering 142 changed hunks. Only 56 packets were dispatched before the deadline, so **53 hunks received no review at all**. A packet is one conversation with the model, and most of its cost is fixed — the skill text, the prompt scaffolding, the forced finalization at the end. Measurements from that run show a five-hunk packet costs roughly the same as a one-hunk packet: about 3 model calls and 228 seconds either way. Yet 73 of the 96 packets held a single hunk. @@ -52,11 +52,11 @@ Packing can make a hunk **more** expensive than it would have been alone: the pr **Attention dilution.** A bug in a five-hunk packet might get less scrutiny than the same bug alone. -One test repository, fifteen separated changes in one Go file, **three** independent bugs at different positions plus a safe-looking control. The same review runs at packet caps 1, 3, and 5, six repeats each, all against one hand-authored frozen plan so packet size is the only variable. Three bugs times six repeats gives **18 observations per arm**, not 6 — the same spend buys three times the statistical resolution. +One test repository, fifteen separated changes in one Go file, **three** independent bugs placed so that no two share a packet at any tested cap, plus a safe-looking control. The same review runs at packet caps 1, 3, and 5, six repeats each, all against one hand-authored frozen plan so packet size is the only variable. Three bugs times six repeats gives **18 recall opportunities per arm** rather than 6 — the same spend buys three times the resolution. Those opportunities are *clustered*, not independent: repeats share a fixture and an authored plan, so they are treated as three per-bug series rather than 18 free trials, and recall is reported per bug as well as in aggregate. The pass bar is derived, not chosen: packing must not cost more per-hunk recall than the coverage it buys. If the replay shows packing reviewing 108 hunks where the baseline reviews 89, per-hunk recall may fall no further than 89/108 ≈ **82.4%** of baseline before the whole thing is a net loss. -Cost: about **$14** to answer that, plus an optional $50 production-scale confirmation. +Cost: about **$14** to answer that, and a $50 production-scale confirmation that is **required once the curve passes**, not optional. ### The outcome, either way @@ -97,13 +97,21 @@ Reproducible via `scripts/packing-diagnostics.mjs`; outputs preserved under `/ho | Artifact | SHA-256 | | --- | --- | -| `plan103-affinity-pairs.json` | `677958c7caf9ef3b5243085385cefd0659b4c5df60869dd7cf917788b85d18f4` | +| `plan103-affinity-pairs.json` | `2d0f6f2d14ce907822128e5338c2627e97fc9d495a528f807d71ffec0ac6bc76` | | `plan103-cap-sweep.json` | `f9915c77210d329c2859531007f6175c90eaca45cb52417a22d8dbc3e08de371` | | `plan103-ordering-comparison.json` | `e4e6bf2e3cfe585b1c04db1280a2bcb1c1bb5516975352b545fb96211bf24a57` | -On `dca8d870`, 13 atom pairs are related but currently split: 7 blocked by the caps, 2 by coverage inequality, 4 eligible. Relationship-required packing gives 96→92 (4.2%). Relationship-preferred *ordering* gives 75/66/83 packets across the three distinct diffs — identical to plain source order, including multi-atom counts. +On `dca8d870`, 13 atom pairs are related but currently split: 6 blocked by the caps, 2 by coverage inequality, 2 by lens-signature mismatch, 3 eligible. On `81f806a6`, 11 pairs: 7 caps, 4 eligible. Relationship-required packing gives 96→92 (4.2%). Relationship-preferred *ordering* gives 75/66/83 packets across the three distinct diffs — identical to plain source order, including multi-atom counts. -Both measurements use documented approximations (patch proxy, owner prefix from display strings). Neither is close enough to its threshold for the approximation to change the conclusion. +Both measurements use documented approximations (patch proxy, owner prefix from display strings, synthetic `DEFAULT` lens signature for planner-undeclared hunks). Neither is close enough to its threshold for the approximation to change the conclusion. + +### The fixed-slot estimator is validated against the real run + +The coverage claim is gated on a fixed-slot estimator rather than packet count, so the estimator itself was checked against `dca8d870`. `worker-runner.ts` sorts by `(priority, coverage, dispatchRank, input order)` and latches `exhausted`, so the undispatched set is a suffix of that order — the "first N in dispatch order" model is exactly how scheduling behaves. + +Applying that sort to the 96 flag-off packets and taking the first N reproduces the actual result exactly: **89 hunks covered, matching the 89 the run actually reviewed**, with 55 of 56 packets in common (the single difference comes from using packet ID as a stand-in for true input order, which artifacts do not record). + +Note the true baseline is **56 dispatched packets**, not the 57 quoted in Plan 102's reconciliation note: the run recorded 40 `packet_review_not_dispatched` events against 96 packets. This plan uses 56 slots. ### The 5-hunk cap is near-optimal @@ -162,11 +170,20 @@ Constants: Rules: 1. At least `MAX_CONTEXT_CHARS - PACKET_SYMBOL_CONTEXT_BUDGET` (3,000 characters) is reserved for outline, likely-tests, and planner-hint context. Symbol source never consumes it. -2. Each distinct member primary symbol receives `floor(PACKET_SYMBOL_CONTEXT_BUDGET / memberSymbolCount)`, subject to `MIN_MEMBER_SYMBOL_CHARS`. -3. If `memberSymbolCount * MIN_MEMBER_SYMBOL_CHARS > PACKET_SYMBOL_CONTEXT_BUDGET`, the candidate is abandoned — the packet cannot represent all its members. -4. Members needing less than their share release the remainder; one redistribution pass, in source order, to members that requested more. -5. Existing adaptive/sliced selection applies per member within its share. Members render in source order. -6. **Per-member context quality is computed after `renderPacketContextText()` truncation, not before.** A member with fewer than `MIN_SLICED_MEMBER_CHARS` surviving is a candidate abandonment, never a reported `sliced`. +2. **Only members with a resolvable primary symbol participate in the allocation.** `memberSymbolCount` counts distinct resolvable primary symbol identities, never all members. +3. Each participating member receives `floor(PACKET_SYMBOL_CONTEXT_BUDGET / memberSymbolCount)`, subject to `MIN_MEMBER_SYMBOL_CHARS`. +4. If `memberSymbolCount * MIN_MEMBER_SYMBOL_CHARS > PACKET_SYMBOL_CONTEXT_BUDGET`, the candidate is abandoned — the packet cannot represent all its symbol-bearing members. +5. Members needing less than their share release the remainder; one redistribution pass, in source order, to members that requested more. +6. Existing adaptive/sliced selection applies per member within its share. Members render in source order. +7. **Per-member context quality is computed after `renderPacketContextText()` truncation, not before.** A participating member with fewer than `MIN_SLICED_MEMBER_CHARS` surviving is a candidate abandonment, never a reported `sliced`. + +**Symbol-less members.** Configuration, documentation, generic-adapter, and fallback hunks frequently have no resolvable primary symbol; `readEnclosingSymbolSource()` returns empty for them today and their standalone context quality is `outline_only` or `path_only`. Such members: + +- do not participate in the symbol allocation and consume none of it; +- are **exempt from the `MIN_SLICED_MEMBER_CHARS` rule** — applying a symbol-source floor to a member that never had symbol source would reject every packet containing one, including the config- and docs-heavy files where compatible atoms are most abundant; +- must retain their standalone outline or path context quality after packing. A symbol-less member whose quality degrades below what it held alone is a candidate abandonment. + +A packet whose members are *all* symbol-less allocates no symbol budget and is judged solely on the standalone-quality rule. ### 3. Transactional candidate evaluation @@ -186,7 +203,7 @@ packMaxHunks: number; // default 5, never exceeds MAX_HUNKS_PER_PACKET Neither appears in `rawConfigSchema`, neither is repo-safe filtered, and no `codegenie.toml` can set them. They exist in the resolved config, defaults, config-source telemetry, the strict eval-case `review` schema, and `applyCaseReviewConfig()`. Both are deleted in step 9. -The pinned-plan seam is likewise eval-only: a planner-draw mode writing a `ReviewPlan` plus SHA-256, and an eval-case field `review.pinnedPlanPath` validated by hash and by exact `hunkId` membership against the current diff, failing closed on mismatch. +The pinned-plan seam is likewise eval-only: a planner-draw mode writing a `ReviewPlan` plus SHA-256, and an eval-case field `review.pinnedPlanPath` validated by SHA-256, by full `ReviewPlan` schema parse, by exact `hunkId` membership against the current diff, and by recorded base/head identity — failing closed on any mismatch. A hash match over a plan that no longer parses, or that targets a different diff, is not sufficient. ## Validation Strategy @@ -196,14 +213,18 @@ Port `scripts/packet-packing-report.ts` from `7ebfd2f` down to `replay` plus `tr Assert per run: packet counts reproduce Plan 102's frozen result within 2; every hunk appears exactly once in source order; no atom split; no coverage promotion; caps hold; no lens or high-priority note lost without a recorded abandonment; no member below `MIN_SLICED_MEMBER_CHARS`; no profile or budget below the standalone maximum; dispatch ranks match the formula; flag-off is artifact-identical including `hunk-relationships.json`. -**Fixed-slot hunk yield is the primary gate.** Sort packed packets by the full Stage-7 scheduling tuple `(priority, coverage, dispatchRank, input order)`, take the first 57 — the motivating run's observed dispatch capacity — and count reviewable hunks covered. Baseline is 89. +**Fixed-slot hunk yield is the primary gate.** Sort packets by the full Stage-7 scheduling tuple `(priority, coverage, dispatchRank, input order)`, take the first 56 — the motivating run's observed dispatch capacity — and count reviewable hunks covered. + +**The estimator must first reconcile against the actual run.** Apply it to the *flag-off* packets and require it to reproduce the 89 hunks the run actually reviewed. If it does not, the estimator is not comparable to the historical baseline and must not define `B` until reconciled. This has been validated once by hand (89 = 89, 55/56 packets in common); the report re-proves it mechanically because it also feeds the paid gate. | Gate | Threshold | | --- | --- | -| Fixed-slot hunk yield at 57 packets | `>= 102` hunks (≥15% over baseline 89) | +| Flag-off estimator reconciliation | exactly `89` hunks at 56 slots | +| Fixed-slot hunk yield at 56 packets, flag on | `>= 102` hunks (≥15% over baseline 89) | | Packet reduction | `>= 20%` per distinct diff | | Deviation from Plan 102's frozen counts | `<= 2` packets per run | -| Members below `MIN_SLICED_MEMBER_CHARS` | `0` | +| Participating members below `MIN_SLICED_MEMBER_CHARS` | `0` | +| Symbol-less members losing standalone quality | `0` | | New coverage promotions | `0` | Also report, off versus on: per-member profile upgrades, total tool-call allowance, total context characters, and projected cost and service time per reviewed hunk. The measured yield ratio `baselineYield / packedYield` becomes the break-even bar for phase B. @@ -214,7 +235,7 @@ Also report, off versus on: per-member profile upgrades, total tool-call allowan **Fixture.** One Go file, fifteen separated single-hunk changes, far enough apart that today's grouper yields fifteen atoms. It carries: -- **three independent bugs** at atom positions 1, 5, and 8 — under cap 5 that is packet 1 position 1, packet 1 position 5, and packet 2 position 3, so first/last/middle positions are all covered; +- **three independent bugs** at atom positions 1, 10, and 13. Under cap 5 the packets are `[1-5][6-10][11-15]`, placing the bugs in three *different* packets at positions 1, 5, and 3 — first, last, and middle. Under cap 3 the packets are `[1-3]…[10-12][13-15]`, again three different packets. **No two bugs ever share a packet at any tested cap**, so no arm reviews an unnaturally bug-dense packet and no two opportunities collapse into one conversation; - **one safe-but-suspicious change** as a `should_not_find` control, so a rise in false positives is visible; - eleven ordinary safe changes. @@ -226,22 +247,33 @@ Fixture-quality rules, each learned from a Plan 102 failure: 4. **Constrain path, line range, and failure mode — never category.** Plan 102 lost three executions that found its bug because the expectation demanded `correctness` and the reviewer said `security`. 5. **Prove treatment model-free before paying.** Build Stage 6 against the frozen plan at caps 1, 3, and 5 and require exactly 15, 5, and 3 packets, with each bug's atom in a packet of exactly 1, 3, and 5 source atoms. Any miss is a fixture defect: fix the fixture, never the sample. -**The frozen plan is hand-authored, not drawn.** Authoring it makes coverage and lens assignment deterministic, so all fifteen atoms are guaranteed into one partition; removes planner variance so packet size is genuinely the only variable; eliminates any temptation to redraw unfavourable plans; and costs nothing. Record one real planner draw alongside it and diff the two, so the authored plan's realism is documented rather than assumed. +**The frozen plan is hand-authored, not drawn.** Authoring it makes coverage and lens assignment deterministic, so all fifteen atoms are guaranteed into one partition; removes planner variance so packet size is genuinely the only variable; eliminates any temptation to redraw unfavourable plans; and costs nothing. + +**But an authored plan must be shown to be realistic, not merely convenient.** Draw **three** reference plans from the real planner against the same fixture and require that in **at least 2 of 3**, every target atom lands in a partition of at least three compatible atoms at cap 5 — that is, production would plausibly create the shape being tested. Fewer than 2 of 3 means the fixture exercises a partition the planner would rarely produce: **the fixture is invalid and must be redesigned before any reviewer call**, not merely documented. Record all three draws, their diffs against the authored plan, and the pass count. + +**Arms.** Three cases differing only in `packMaxHunks` — 1, 3, 5 — at `repeat: 6`, cache off, all pinned to the same authored plan. Three bugs × six repeats = **18 recall opportunities per arm**. -**Arms.** Three cases differing only in `packMaxHunks` — 1, 3, 5 — at `repeat: 6`, cache off, all pinned to the same authored plan. Three bugs × six repeats = **18 observations per arm**. +Those opportunities are **clustered, not independent**: all repeats share one fixture and one authored plan, and each repeat's three bugs are reviewed by the same run. Treat the data as three per-bug series of six, report `R` per bug alongside the aggregate, and never describe it as 18 independent trials. -**Decision rule.** Let `R1`, `R3`, `R5` be hits out of 18 and `B` the break-even ratio measured in phase A (`baselineYield / packedYield`; ≈ 0.824 at the projected 89/108). +**Decision rule.** Let `R1`, `R3`, `R5` be aggregate hits out of 18, `R{n}[i]` the per-bug hits out of 6 for bug `i`, and `B` the break-even ratio measured in phase A (`baselineYield / packedYield`; ≈ 0.824 at 89/108). | Condition | Outcome | | --- | --- | -| `R1 < 15` | **Void.** Baseline too unreliable to measure a ratio against; fix the fixture. | -| `R5 / R1 >= B` and no monotone decline `R1 > R3 > R5` | **Pass.** | -| `R5 / R1 >= B` but `R1 > R3 > R5` | **Stop for investigation.** A consistent decline across all three sizes is a dilution signature even when the ratio clears; extend repeats on the affected arms before deciding. | +| `R1 < 15` | **Void.** Baseline too unreliable to measure a ratio against. | +| any bug with `R1[i] >= 5` and `R5[i] <= 1` | **Fail (per-bug collapse).** One target falling from reliable detection to near-zero is a real regression regardless of the aggregate. | +| `R5 / R1 >= B`, no per-bug collapse, and no monotone decline `R1 > R3 > R5` | **Pass.** | +| `R5 / R1 >= B` but `R1 > R3 > R5` | **Extend once** (see below). | | `R5 / R1 < B` | **Fail.** Packing costs more recall than the coverage it buys. | `R3` therefore has a real role — it distinguishes a cliff from a gradient — and there is no unreachable branch. -**Honest limits.** Eighteen observations per arm can detect a large regression; it cannot resolve, say, 100% versus 85% with confidence. The gate is a screen against material harm, not a certificate of equivalence, and the reconciliation note must say so. Report per-arm cost, tokens, and model-service time per reviewed hunk alongside recall. +**Void and extension are bounded and preregistered, not discretionary:** + +- **Void permits exactly one fixture redesign and one rerun.** A second Void is a **Fail** and takes the teardown branch. Void never means "keep trying until the numbers work." +- **Monotone decline permits exactly one extension:** six additional repeats on **all three arms**, budgeted below at ~$14. Re-apply the same decision rule to the combined 12-repeat totals (36 opportunities per arm). Its outcome is final — a second extension is not available, and a still-monotone result after extension is a **Fail**. +- Every Void or extension is recorded in the reconciliation note with its trigger, cost, and result. + +**Honest limits.** Eighteen clustered opportunities per arm can detect a large regression; they cannot resolve, say, 100% versus 85% with confidence. The gate is a screen against material harm, not a certificate of equivalence, and the reconciliation note must say so. Report per-arm cost, tokens, and model-service time per reviewed hunk alongside recall. ### C. Collateral @@ -256,14 +288,16 @@ No paid call before `approvedValidationCostUSD` is recorded here. | Phase | Executions | Projection | | --- | ---: | ---: | | Fake-provider expectation validation | — | $0 | -| One reference planner draw | 1 | ~$1 | +| Three reference planner draws | 3 | ~$3 | | Recall curve, 6 repeats × 3 arms | 18 | ~$14 | +| Reserved: one bounded extension, 6 more repeats × 3 arms | 18 | ~$14 | +| Reserved: one fixture redesign rerun after a single Void | 18 | ~$14 | | `evals/fixtures/` and cross-language shape cases | — | $0 | | Production capacity pair | 2 | ~$50 | -| Contingency 25% | — | ~$16 | -| **Reservation** | | **~$81** | +| Contingency 25% | — | ~$24 | +| **Reservation** | | **~$119** | -Phases through the curve cost ~$15 and decide whether packing ships; the capacity pair runs last and only after a pass. +The two reserved contingencies are the *only* reruns this plan authorizes; a second Void or a second extension is a Fail, not another draw on the budget. Phases through the first curve cost ~$17 and decide whether packing ships; the capacity pair runs last and only after a pass. ## In-Scope Files @@ -316,11 +350,13 @@ Phases through the curve cost ~$15 and decide whether packing ships; the capacit **Verify:** treatment proof shows 15/5/3 packets with each bug at 1/3/5 source atoms; all three arms consumed the same plan hash; the decision table yields Pass, Fail, Void, or Stop-for-investigation with no discretion; the capacity pair confirms the phase-A yield prediction; spend stays within the ceiling. 9. Record the decision and tear down in a dedicated commit. - **Preserve evidence first:** copy every produced JSON report and the authored plan into `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/reports/`, extending its existing `manifest.sha256`. Do not modify Plan 102's manifest, which its reconciliation note describes as covering exactly three files. Mark unreached phases `not_run` with their stopping reason. + **Preserve evidence first:** copy into `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/reports/` every produced JSON report, the authored frozen plan, all three reference draws and their diffs, the treatment proof, the capacity-pair reports, and a `not_run` ledger naming every unreached phase with its stopping reason — then regenerate `manifest.sha256` over the complete set. Do not modify Plan 102's manifest, which its reconciliation note describes as covering exactly three files. **If Pass:** make packing unconditional at cap 5; delete both settings and the unpacked path. Keep atoms, partitions, the profile floor, multi-member symbol context, and transactional rejection. Reduce the report script to a golden check. Keep `evals/packet-dilution/` as a standing suite with the shipped cap only, and retain the curve in the reconciliation note. - **If Fail, Void, or unresolved Stop-for-investigation:** delete the packing pass, both settings, the atom wrapper, and the report script with its tests. Multi-member symbol context survives only if the replay showed it improving context quality with packing off. Keep the fixture and the measured curve — the curve is the durable asset regardless of outcome. Do not leave the feature dark. + **If Fail — including a second Void, a post-extension monotone decline, or any per-bug collapse:** delete the packing pass, both settings, the atom wrapper, and the report script with its tests. Multi-member symbol context survives only if the replay showed it improving context quality with packing off. Keep the fixture and the measured curve — the curve is the durable asset regardless of outcome. Do not leave the feature dark. + + A *first* Void or a *first* monotone decline does not reach this step: each authorizes exactly one bounded rerun under phase B's preregistered limits, and only its result reaches teardown. **Verify:** `rg -n "packCompatibleAtoms|packMaxHunks" src scripts tests evals` → exit 1; both manifests verify; the note records the decision, actual spend, and the resolution limit of 18 observations per arm. 10. Run the complete repository gate. @@ -338,9 +374,12 @@ Artifact diagnostics (reproduces this plan's Evidence section, no model calls): ```bash RUNS=/home/peter/Dev/0xsequence/trails-api/.codegenie/runs -node scripts/packing-diagnostics.mjs pairs $RUNS/20260724-184952-dca8d870 -node scripts/packing-diagnostics.mjs sweep $RUNS/20260724-184952-dca8d870 -node scripts/packing-diagnostics.mjs simulate $RUNS/20260724-184952-dca8d870 --predicate source +D="$RUNS/20260724-184952-dca8d870 $RUNS/20260724-162739-81f806a6 $RUNS/20260724-135818-740d73f2" +node scripts/packing-diagnostics.mjs pairs $D +node scripts/packing-diagnostics.mjs sweep $D +for p in source compatibility related; do + node scripts/packing-diagnostics.mjs simulate $D --predicate $p +done ``` Deterministic four-run replay: @@ -363,10 +402,13 @@ Fixture validation, planner draw, and the paid curve: # 1. expectation wiring, fake provider, $0 pnpm dev eval --eval-dir evals/packet-dilution/shape --no-cache -# 2. one reference planner draw, for realism comparison against the authored plan -pnpm dev plan --repo evals/packet-dilution/repos/dilution \ - --base main --branch feature \ - --output evals/packet-dilution/plans/reference-draw.json +# 2. three reference planner draws; >=2 of 3 must put every target in a +# partition of >=3 compatible atoms at cap 5, else the fixture is invalid +for i in 1 2 3; do + pnpm dev plan --repo evals/packet-dilution/repos/dilution \ + --base main --branch feature \ + --output evals/packet-dilution/plans/reference-draw-$i.json +done # 3. model-free treatment proof at each cap pnpm exec tsx scripts/packet-packing-report.ts treatment \ @@ -400,7 +442,12 @@ Evidence preservation and verification: ```bash D=/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/reports cp /tmp/plan103-*.json "$D"/ -(cd "$D" && sha256sum plan103-*.json > manifest.sha256 && sha256sum -c manifest.sha256) +cp evals/packet-dilution/plans/frozen.json "$D"/ +cp evals/packet-dilution/plans/reference-draw-*.json "$D"/ +cp /tmp/plan103-capacity-*.json "$D"/ 2>/dev/null || true # absent if not reached +# not_run ledger: every unreached phase and why +"$EDITOR" "$D/not-run-ledger.md" +(cd "$D" && sha256sum plan103-*.json frozen.json reference-draw-*.json not-run-ledger.md > manifest.sha256 && sha256sum -c manifest.sha256) (cd "$D/../../packet-packing/reports" && sha256sum -c manifest.sha256) # Plan 102 evidence unchanged ``` @@ -410,12 +457,12 @@ Note the `pnpm dev eval` spelling: under pnpm 11 a literal `--` reaches Commande - Flag off produces byte-identical packet artifacts, IDs, order, profiles, context, budgets, and `hunk-relationships.json`. - Flag on reproduces Plan 102's packet counts within 2 per run, every hunk appearing exactly once, hunks ordered within packets by file position and packets by earliest member, under unchanged caps with zero coverage promotion. -- Every member atom's primary symbol source is present after final rendering at or above `MIN_SLICED_MEMBER_CHARS`, or its candidate was abandoned and recorded. +- Every member atom with a resolvable primary symbol has its source present after final rendering at or above `MIN_SLICED_MEMBER_CHARS`, or its candidate was abandoned and recorded. Symbol-less members consume no symbol budget, are exempt from that floor, and retain their standalone outline or path quality. - The 3,000-character non-symbol context reserve is never consumed by symbol source. - No lens dropped and no `high`/`critical` focus note newly omitted without a recorded abandonment; every effective profile at least the standalone maximum; dispatch ranks unchanged in formula. -- Fixed-slot hunk yield at 57 packets is at least 102, and the measured break-even ratio is recorded and used as phase B's bar. +- The fixed-slot estimator reproduces exactly 89 hunks on the flag-off packets at 56 slots before it is used for anything; flag-on yield at 56 slots is at least 102; the measured break-even ratio is recorded and used as phase B's bar. - Per-member profile upgrades, tool-call allowance, context characters, and cost and service time per reviewed hunk are reported off versus on. -- All three curve arms consume one hash-verified authored plan; treatment is proven model-free at 15/5/3 packets before any paid call; the decision table is applied without discretion. +- All three curve arms consume one authored plan verified by hash, schema, hunk-ID membership, and diff identity; at least 2 of 3 reference draws confirm the tested partition is one production would plausibly create; treatment is proven model-free at 15/5/3 packets with no two bugs sharing a packet at any cap; the decision table, including per-bug collapse and the bounded Void and extension allowances, is applied without discretion. - Cross-language shape cases prove packed context on TypeScript, Python, and Solidity; `evals/fixtures/` shows no regression; the capacity pair confirms the predicted yield. - Paid validation never begins without a recorded ceiling; every phase records actual and projected spend. - Report failures carry templated structured messages with no raw exception text, repository source, or hashing. @@ -424,12 +471,16 @@ Note the `pnpm dev eval` spelling: under pnpm 11 a literal `--` reaches Commande ## Stop Conditions +- The flag-off estimator does not reproduce 89 hunks at 56 slots: the proxy is not comparable to the historical baseline and must not define `B` until reconciled. - Fixed-slot hunk yield below 102, packet reduction below 20%, or deviation from Plan 102's frozen counts above 2: the port is wrong. Fix before spending anything. - Any atom split or reordered, hunk lost or duplicated, cap exceeded, coverage promoted, profile or budget downgraded, or member context below the minimum without abandonment: fix the deterministic design first. - Flag-off artifacts change in any respect: stop; parity is the basis of every later comparison. - The model-free treatment proof does not yield 15/5/3 packets with the target atoms at 1/3/5: fix the fixture and re-prove. Never adjust the sample to fit. - Any arm consumed a different plan hash: discard and rerun that repeat. -- `R1 < 15` of 18: the baseline is too unreliable to measure against. Void; fix the fixture rather than lowering the bar. +- Fewer than 2 of 3 reference draws place every target in a partition of at least three compatible atoms: the fixture tests a shape production would rarely create. Redesign before any reviewer call. +- `R1 < 15` of 18: the baseline is too unreliable to measure against. Void; fix the fixture rather than lowering the bar. A second Void is a Fail. +- Any bug with `R1[i] >= 5` and `R5[i] <= 1`: per-bug collapse. Fail regardless of the aggregate ratio. +- A monotone decline surviving the single authorized six-repeat extension: Fail. No second extension exists. - `R5 / R1 < B`: packing costs more recall than the coverage it buys. Fail and take the teardown branch — do not renegotiate `B` after seeing the result. - A monotone decline `R1 > R3 > R5` that survives extended repeats: treat as a dilution signature and fail, even if the ratio clears. - Any proposal to raise `MAX_HUNKS_PER_PACKET`, revive relationship signals, or add coverage promotion to recover margin: reject here. Each has a measured Non-Goal entry. From 5cacffa9906bc54901b609b948a1cc5606d62439 Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 12:50:09 -0400 Subject: [PATCH 11/24] docs(plans): correct Plan 103 scheduler claim, scale thresholds, specify pinned artifact Five contract defects found in review, all verified against telemetry or code before fixing. The scheduler description was factually wrong. I claimed the undispatched set is a suffix of dispatch order. Worker IDs encode that order, and the telemetry shows w7-056 recorded not_dispatched while w7-057 completed - a prefix with a hole, which is expected at concurrency 4 where a later worker can clear its budget checkpoint before an earlier one fails. That hole, not missing input-order telemetry, is also the real explanation for the 55/56 packet overlap. The estimator is now described as a calibrated counterfactual capacity proxy validated against a known 89-hunk outcome, not as scheduler reproduction, and the stale --dispatch-slots 57 is 56. Extension thresholds were absolute and would silently halve on extension. Applying R1 < 15 to a 36-opportunity cohort halves the baseline bar and redefines collapse. All thresholds are now proportions - Void at R1/N < 5/6, collapse at R1[i]/K >= 5/6 with R5[i]/K <= 1/6 - yielding 15/18 and 5-of-6 initially, 30/36 and 10-of-12 after the one authorized extension. The realism gate proved the wrong thing, and the fix differs from the one proposed. Plan 102's replay shows five-HUNK packets are ordinary production output (7-8 per run, ~10% of packed packets) while five-ATOM packets never occur (max 4, once per run). Since the fixture uses fifteen single-hunk atoms, its cap-5 arm produces a five-hunk packet - the production-real shape. So cap 5 is a genuine production case rather than a stress bound, but only via hunks, and the plan now claims nothing about five-atom packets. The gate is expressed in hunks and enforced by a new realism report mode that rebuilds each draw's packet shapes and exits non-zero below threshold, with draws taken cache-off for independence. The pinned-plan artifact was underspecified. ReviewPlan carries no base/head identity and pinnedPlanPath alone defined no trusted expected hash, leaving the executor to invent a contract. Specified as a versioned PinnedPlanArtifact with baseSha, headSha, and planSha256 over a canonical sorted-key serialization of the plan alone, with per-failure-mode tests. The draw mode becomes scripts/draw-pinned-plan.ts rather than the pnpm dev plan CLI verb I invented, which does not exist and would have contradicted the eval-only scope; src/cli/ stays out of scope. The capacity pair had no machine-checkable rule. It now requires both sides to consume the same pinned plan and identical provider settings, and gates on three numbers: at least 10% more reviewed hunks, a reviewed set that is a superset of baseline minus at most 2 hunks, and measured yield within 10% of the phase-A prediction. Also: step 8 says Extend-once rather than the retired Stop-for- investigation; evidence preservation copies the realism report and the reference diffs it emits; and the diagnostics header no longer claims Plan 103 uses ownerKey or relationship-preferred ordering - those modes now state they exist to reproduce a rejected design. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/packing-diagnostics.mjs | 11 ++- ...-103-relationship-driven-packet-packing.md | 94 ++++++++++++++----- 2 files changed, 80 insertions(+), 25 deletions(-) diff --git a/scripts/packing-diagnostics.mjs b/scripts/packing-diagnostics.mjs index cc574dd..1ca19a5 100644 --- a/scripts/packing-diagnostics.mjs +++ b/scripts/packing-diagnostics.mjs @@ -16,9 +16,11 @@ // length. The builder uses combinedPatchChars() over raw diff lines. // Plan 102 documented the two differing by about one packet per run. // 2. Container identity is recovered by splitting the owner prefix out of -// the recorded `enclosingSymbol` display string ("(*T).Method" -> "T"). -// Plan 103 specifies a structural ownerKey resolved from symbol ranges, -// which does not collide across duplicate or nested same-named owners. +// the recorded `enclosingSymbol` display string ("(*T).Method" -> "T"), +// which collides across duplicate or nested same-named owners. Only the +// `related` predicate uses it, and that predicate was measured and +// rejected — Plan 103 ships neither container identity nor relationship +// ordering. This mode exists to reproduce that rejection. // 3. Planner lens signatures are only known for hunks the planner issued a // coverage entry for. Undeclared hunks fall back to // defaultLensesForLanguage(), which is constant within a file, so they @@ -113,7 +115,8 @@ function loadRun(runDir) { } // Packing-affinity adjacency between atoms: strong recorded edges plus shared -// container. Mirrors Plan 103's derived/structural split. +// container. Used only by the `pairs` and `related` analyses, which exist to +// reproduce why Plan 103 rejected relationship-based packing. function affinity(run) { const byId = new Map(run.atoms.map((atom) => [atom.id, atom])); const adjacency = new Map(run.atoms.map((atom) => [atom.id, new Set()])); diff --git a/specs/plans/103-issue-103-relationship-driven-packet-packing.md b/specs/plans/103-issue-103-relationship-driven-packet-packing.md index 55e2143..f4fe59a 100644 --- a/specs/plans/103-issue-103-relationship-driven-packet-packing.md +++ b/specs/plans/103-issue-103-relationship-driven-packet-packing.md @@ -107,11 +107,13 @@ Both measurements use documented approximations (patch proxy, owner prefix from ### The fixed-slot estimator is validated against the real run -The coverage claim is gated on a fixed-slot estimator rather than packet count, so the estimator itself was checked against `dca8d870`. `worker-runner.ts` sorts by `(priority, coverage, dispatchRank, input order)` and latches `exhausted`, so the undispatched set is a suffix of that order — the "first N in dispatch order" model is exactly how scheduling behaves. +The coverage claim is gated on a fixed-slot estimator rather than packet count, so the estimator itself was checked against `dca8d870`. `worker-runner.ts` sorts by `(priority, coverage, dispatchRank, input order)` and assigns worker IDs in that order, so dispatch order is directly recoverable from telemetry. -Applying that sort to the 96 flag-off packets and taking the first N reproduces the actual result exactly: **89 hunks covered, matching the 89 the run actually reviewed**, with 55 of 56 packets in common (the single difference comes from using packet ID as a stand-in for true input order, which artifacts do not record). +**The estimator is a calibrated counterfactual, not an exact reproduction of the scheduler.** The real dispatched set is a prefix *with a hole*: worker `w7-056` was recorded `packet_review_not_dispatched` while `w7-057` completed. At concurrency 4 a later worker can clear its budget checkpoint before an earlier one fails, so the undispatched set is not a clean suffix and a future run could have more than one hole. -Note the true baseline is **56 dispatched packets**, not the 57 quoted in Plan 102's reconciliation note: the run recorded 40 `packet_review_not_dispatched` events against 96 packets. This plan uses 56 slots. +Applying the sort to the 96 flag-off packets and taking the first 56 nevertheless yields **89 hunks — the same 89 the run actually reviewed** — with 55 of 56 packets in common; the one difference is exactly the `w7-056`/`w7-057` swap, and the two packets happen to carry equal hunk counts. That makes 56 slots a *calibrated capacity proxy*, validated against a known outcome, not a claim that the estimator replays scheduling. + +Counting completed reviews, 56 packets produced review outcomes against 96 built; 57 started. This plan uses **56 slots** as the completed-packet capacity, and the reconciliation gate below re-proves the 89-hunk calibration mechanically before the estimator is used for anything. ### The 5-hunk cap is near-optimal @@ -203,7 +205,21 @@ packMaxHunks: number; // default 5, never exceeds MAX_HUNKS_PER_PACKET Neither appears in `rawConfigSchema`, neither is repo-safe filtered, and no `codegenie.toml` can set them. They exist in the resolved config, defaults, config-source telemetry, the strict eval-case `review` schema, and `applyCaseReviewConfig()`. Both are deleted in step 9. -The pinned-plan seam is likewise eval-only: a planner-draw mode writing a `ReviewPlan` plus SHA-256, and an eval-case field `review.pinnedPlanPath` validated by SHA-256, by full `ReviewPlan` schema parse, by exact `hunkId` membership against the current diff, and by recorded base/head identity — failing closed on any mismatch. A hash match over a plan that no longer parses, or that targets a different diff, is not sufficient. +The pinned-plan seam is likewise eval-only. `ReviewPlan` carries no base/head identity, so the pinned artifact is an explicit versioned wrapper rather than a bare plan: + +```ts +type PinnedPlanArtifact = { + schemaVersion: 1; + baseSha: string; // resolved base commit the plan was drawn against + headSha: string; // resolved head commit + planSha256: string; // canonical hash of `plan`, defined below + plan: ReviewPlan; +}; +``` + +`planSha256` is the SHA-256 of the stable JSON serialization of `plan` alone — keys sorted recursively, no whitespace — so the hash is independent of field order and of the wrapper's own fields. The eval-case field `review.pinnedPlanPath` names this artifact, and loading it fails closed on any of: wrapper schema mismatch, `ReviewPlan` schema parse failure, `planSha256` not matching a recomputed canonical hash, `baseSha`/`headSha` not matching the resolved review target, or `hunkId` membership differing from the current diff. A hash match over a plan that no longer parses, or that targets a different diff, is not sufficient. + +The planner-draw mode that writes this artifact is a **dedicated internal script**, not a new top-level CLI command — `pnpm dev plan` does not exist today and adding a user-facing verb would contradict the eval-only scope. Tests cover each failure mode individually: bad wrapper schema, bad plan schema, wrong hash, wrong base, wrong head, wrong path, changed lenses, and changed hunk IDs. ## Validation Strategy @@ -249,18 +265,30 @@ Fixture-quality rules, each learned from a Plan 102 failure: **The frozen plan is hand-authored, not drawn.** Authoring it makes coverage and lens assignment deterministic, so all fifteen atoms are guaranteed into one partition; removes planner variance so packet size is genuinely the only variable; eliminates any temptation to redraw unfavourable plans; and costs nothing. -**But an authored plan must be shown to be realistic, not merely convenient.** Draw **three** reference plans from the real planner against the same fixture and require that in **at least 2 of 3**, every target atom lands in a partition of at least three compatible atoms at cap 5 — that is, production would plausibly create the shape being tested. Fewer than 2 of 3 means the fixture exercises a partition the planner would rarely produce: **the fixture is invalid and must be redesigned before any reviewer call**, not merely documented. Record all three draws, their diffs against the authored plan, and the pass count. +**But an authored plan must be shown to be realistic, not merely convenient — and realism is measured in hunks, not atoms.** Plan 102's replay gives the production shape after packing at cap 5: + +| Run | Packed packets by hunk count (1/2/3/4/5) | Packed packets by atom count (1/2/3/4) | +| --- | --- | --- | +| `dca8d870` | 46/12/4/5/**8** | 61/8/5/1 | +| `81f806a6` | 38/12/6/4/**8** | 48/10/7/1 | +| `740d73f2` | 54/13/6/5/**7** | 66/9/7/1 | + +Five-**hunk** packets are ordinary production output — 7 to 8 per run, roughly a tenth of all packed packets. Five-**atom** packets never occur; the maximum observed is four, once per run. Since the fixture uses fifteen single-hunk atoms, its cap-5 arm produces a five-hunk packet, which is the production-real shape. The cap-5 arm is therefore a genuine production case, not a stress bound — but only because atoms there are one hunk each, and the plan claims nothing about five-*atom* packets, which production does not build. + +Draw **three** reference plans from the real planner against the same fixture and require that in **at least 2 of 3**, every target hunk lands in a packet of at least five hunks at cap 5 and at least three at cap 3 — the shapes the arms actually test. Fewer than 2 of 3 means the fixture exercises a shape the planner would rarely produce: **the fixture is invalid and must be redesigned before any reviewer call**, not merely documented. Draws run with the local cache disabled so they are independent. A machine-enforced report parses all three, rebuilds their Stage-6 packet shapes, emits each target's actual packet hunk count and compatibility signature, and exits non-zero below the threshold. Record all three draws, their diffs against the authored plan, the emitted shapes, and the pass count. **Arms.** Three cases differing only in `packMaxHunks` — 1, 3, 5 — at `repeat: 6`, cache off, all pinned to the same authored plan. Three bugs × six repeats = **18 recall opportunities per arm**. Those opportunities are **clustered, not independent**: all repeats share one fixture and one authored plan, and each repeat's three bugs are reviewed by the same run. Treat the data as three per-bug series of six, report `R` per bug alongside the aggregate, and never describe it as 18 independent trials. -**Decision rule.** Let `R1`, `R3`, `R5` be aggregate hits out of 18, `R{n}[i]` the per-bug hits out of 6 for bug `i`, and `B` the break-even ratio measured in phase A (`baselineYield / packedYield`; ≈ 0.824 at 89/108). +**Decision rule.** Let `N` be opportunities per arm (18 initially, 36 after an extension), `K` be repeats per arm (6 initially, 12 after), `R1`, `R3`, `R5` aggregate hits out of `N`, `R{n}[i]` per-bug hits out of `K`, and `B` the break-even ratio measured in phase A (`baselineYield / packedYield`; ≈ 0.824 at 89/108). + +**All thresholds are proportions, so they scale correctly when repeats are extended.** Applying the initial absolute counts to a 36-opportunity extension would silently halve the baseline bar and redefine collapse. | Condition | Outcome | | --- | --- | -| `R1 < 15` | **Void.** Baseline too unreliable to measure a ratio against. | -| any bug with `R1[i] >= 5` and `R5[i] <= 1` | **Fail (per-bug collapse).** One target falling from reliable detection to near-zero is a real regression regardless of the aggregate. | +| `R1 / N < 5/6` (15 of 18; 30 of 36) | **Void.** Baseline too unreliable to measure a ratio against. | +| any bug with `R1[i] / K >= 5/6` and `R5[i] / K <= 1/6` (5-of-6 and 1-of-6; 10-of-12 and 2-of-12) | **Fail (per-bug collapse).** One target falling from reliable detection to near-zero is a real regression regardless of the aggregate. | | `R5 / R1 >= B`, no per-bug collapse, and no monotone decline `R1 > R3 > R5` | **Pass.** | | `R5 / R1 >= B` but `R1 > R3 > R5` | **Extend once** (see below). | | `R5 / R1 < B` | **Fail.** Packing costs more recall than the coverage it buys. | @@ -279,7 +307,15 @@ Those opportunities are **clustered, not independent**: all repeats share one fi 1. `evals/fixtures/` flag-off and flag-on at `repeat: 1`, fake provider. Every expectation, packet invariant, and config parse must match. 2. **Treated cross-language packet-shape cases** under `evals/packet-dilution/`: small TypeScript, Python, and Solidity fixtures each with at least three compatible same-file atoms, run under the fake provider, asserting packed packet shape and per-member context presence. Existing `evals/skill-semantics/` files are 10–15 lines and yield one atom each, so they cannot exercise packing; they remain a flag-off/on regression check only. -3. One production-shaped capacity pair on the pinned `trails-api` diff at concurrency 6, flag-off and flag-on, confirming the fixed-slot yield predicted in phase A against a real run. **Required before shipping**, not optional. +3. One production-shaped capacity pair on the pinned `trails-api` diff at concurrency 6, flag-off and flag-on. **Required before shipping**, not optional. + + Both sides consume **the same pinned Stage-5 plan** and identical provider, model, reasoning, budget, concurrency, time limit, and cache settings; independently drawn plans would confound the pair exactly as they confounded Plan 102. Its gate is numeric, not narrative: + + - the flag-on run must review **at least 10% more reviewable hunks** than flag-off within the same wall-clock and token budget; + - the flag-on reviewed-hunk set must be a **superset of the flag-off set minus at most 2 hunks**, so the gain is not a reshuffle; + - measured flag-on yield must fall within **±10% of the phase-A fixed-slot prediction**; a larger miss means the estimator does not transfer to a live run and the prediction, not the run, is what failed. + + Any of the three failing is a stop-for-investigation before shipping. ### Cost discipline @@ -305,7 +341,8 @@ The two reserved contingencies are the *only* reruns this plan authorizes; a sec - `src/pipeline/review-runner.ts` — eval-only pinned-plan seam. - `src/types.ts` — per-member context-quality telemetry, both temporary settings. - `src/config/schema.ts`, `src/config/config-loader.ts`, `src/evals/eval-runner.ts` — resolved-config and eval plumbing only. -- `scripts/packet-packing-report.ts`, `tests/packet-packing-report.test.ts` — `replay` and `treatment`. +- `scripts/packet-packing-report.ts`, `tests/packet-packing-report.test.ts` — `replay`, `treatment`, and `realism`. +- `scripts/draw-pinned-plan.ts` — internal planner-draw writing the versioned `PinnedPlanArtifact`. No user-facing CLI verb is added; `src/cli/` stays out of scope. - `tests/pipeline-phase5.test.ts`, `tests/evals.test.ts`, `tests/config-loader.test.ts`. - `evals/packet-dilution/` — Go recall fixture, cross-language shape cases, authored plan. - `.gitignore` — `evals/packet-dilution/logs/` and the missing `evals/skill-semantics/logs/`. @@ -347,10 +384,10 @@ The two reserved contingencies are the *only* reruns this plan authorizes; a sec **Verify:** `pnpm exec vitest run tests/packet-packing-report.test.ts` → exit 0; the replay exits 0 with `modelCallsObserved: 0` and every section-A gate satisfied, including fixed-slot hunk yield `>= 102`. Record the measured break-even ratio `B`. 8. Build the fixture, validate expectations under the fake provider, author and hash the plan, prove treatment model-free, then record the ceiling and run: reference planner draw, the 6×3 curve, `evals/fixtures/`, cross-language shape cases, and the capacity pair. Apply the decision table verbatim. - **Verify:** treatment proof shows 15/5/3 packets with each bug at 1/3/5 source atoms; all three arms consumed the same plan hash; the decision table yields Pass, Fail, Void, or Stop-for-investigation with no discretion; the capacity pair confirms the phase-A yield prediction; spend stays within the ceiling. + **Verify:** the realism report exits 0 with at least 2 of 3 draws placing every target in a five-hunk packet at cap 5; treatment proof shows 15/5/3 packets with each bug at 1/3/5 hunks and no two bugs sharing a packet; all three arms consumed the same verified plan artifact; the decision table yields Pass, Fail, Void, or Extend-once with no discretion; the capacity pair meets all three numeric gates; spend stays within the ceiling. 9. Record the decision and tear down in a dedicated commit. - **Preserve evidence first:** copy into `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/reports/` every produced JSON report, the authored frozen plan, all three reference draws and their diffs, the treatment proof, the capacity-pair reports, and a `not_run` ledger naming every unreached phase with its stopping reason — then regenerate `manifest.sha256` over the complete set. Do not modify Plan 102's manifest, which its reconciliation note describes as covering exactly three files. + **Preserve evidence first:** copy into `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/reports/` every produced JSON report, the authored frozen plan, all three reference draws, the realism report and the reference diffs it emits, the treatment proof, the capacity-pair reports, and a `not_run` ledger naming every unreached phase with its stopping reason — then regenerate `manifest.sha256` over the complete set. Do not modify Plan 102's manifest, which its reconciliation note describes as covering exactly three files. **If Pass:** make packing unconditional at cap 5; delete both settings and the unpacked path. Keep atoms, partitions, the profile floor, multi-member symbol context, and transactional rejection. Reduce the report script to a golden check. Keep `evals/packet-dilution/` as a standing suite with the shipped cap only, and retain the curve in the reconciliation note. @@ -391,7 +428,7 @@ pnpm exec tsx scripts/packet-packing-report.ts replay \ --run $RUNS/20260724-150405-fe1548ae \ --run $RUNS/20260724-162739-81f806a6 \ --run $RUNS/20260724-184952-dca8d870 \ - --dispatch-slots 57 \ + --dispatch-slots 56 \ --distinct-diffs \ --output /tmp/plan103-packing-shape.json ``` @@ -402,14 +439,25 @@ Fixture validation, planner draw, and the paid curve: # 1. expectation wiring, fake provider, $0 pnpm dev eval --eval-dir evals/packet-dilution/shape --no-cache -# 2. three reference planner draws; >=2 of 3 must put every target in a -# partition of >=3 compatible atoms at cap 5, else the fixture is invalid +# 2. three independent reference draws (cache off), then a machine-enforced +# realism report: >=2 of 3 must place every target in a five-hunk packet at +# cap 5 and a three-hunk packet at cap 3, else the fixture is invalid for i in 1 2 3; do - pnpm dev plan --repo evals/packet-dilution/repos/dilution \ - --base main --branch feature \ + pnpm exec tsx scripts/draw-pinned-plan.ts \ + --repo evals/packet-dilution/repos/dilution \ + --base main --branch feature --no-cache \ --output evals/packet-dilution/plans/reference-draw-$i.json done +pnpm exec tsx scripts/packet-packing-report.ts realism \ + --authored evals/packet-dilution/plans/frozen.json \ + --draw evals/packet-dilution/plans/reference-draw-1.json \ + --draw evals/packet-dilution/plans/reference-draw-2.json \ + --draw evals/packet-dilution/plans/reference-draw-3.json \ + --require-target-hunks 5 --at-cap 5 --min-passing-draws 2 \ + --emit-diffs evals/packet-dilution/plans/reference-diff-{1,2,3}.json \ + --output /tmp/plan103-realism.json + # 3. model-free treatment proof at each cap pnpm exec tsx scripts/packet-packing-report.ts treatment \ --pinned-plan evals/packet-dilution/plans/frozen.json \ @@ -444,10 +492,12 @@ D=/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/packet-dilution/r cp /tmp/plan103-*.json "$D"/ cp evals/packet-dilution/plans/frozen.json "$D"/ cp evals/packet-dilution/plans/reference-draw-*.json "$D"/ +cp evals/packet-dilution/plans/reference-diff-*.json "$D"/ # emitted by the realism report +cp /tmp/plan103-realism.json "$D"/ cp /tmp/plan103-capacity-*.json "$D"/ 2>/dev/null || true # absent if not reached # not_run ledger: every unreached phase and why "$EDITOR" "$D/not-run-ledger.md" -(cd "$D" && sha256sum plan103-*.json frozen.json reference-draw-*.json not-run-ledger.md > manifest.sha256 && sha256sum -c manifest.sha256) +(cd "$D" && sha256sum plan103-*.json frozen.json reference-draw-*.json reference-diff-*.json not-run-ledger.md > manifest.sha256 && sha256sum -c manifest.sha256) (cd "$D/../../packet-packing/reports" && sha256sum -c manifest.sha256) # Plan 102 evidence unchanged ``` @@ -477,9 +527,11 @@ Note the `pnpm dev eval` spelling: under pnpm 11 a literal `--` reaches Commande - Flag-off artifacts change in any respect: stop; parity is the basis of every later comparison. - The model-free treatment proof does not yield 15/5/3 packets with the target atoms at 1/3/5: fix the fixture and re-prove. Never adjust the sample to fit. - Any arm consumed a different plan hash: discard and rerun that repeat. -- Fewer than 2 of 3 reference draws place every target in a partition of at least three compatible atoms: the fixture tests a shape production would rarely create. Redesign before any reviewer call. -- `R1 < 15` of 18: the baseline is too unreliable to measure against. Void; fix the fixture rather than lowering the bar. A second Void is a Fail. -- Any bug with `R1[i] >= 5` and `R5[i] <= 1`: per-bug collapse. Fail regardless of the aggregate ratio. +- Fewer than 2 of 3 reference draws place every target in a five-hunk packet at cap 5 and a three-hunk packet at cap 3: the fixture tests a shape production would rarely create. Redesign before any reviewer call. +- `R1 / N < 5/6`: the baseline is too unreliable to measure against. Void; fix the fixture rather than lowering the bar. A second Void is a Fail. +- Any bug with `R1[i] / K >= 5/6` and `R5[i] / K <= 1/6`: per-bug collapse. Fail regardless of the aggregate ratio. +- The capacity pair misses any of its three numeric gates: stop for investigation before shipping. +- Any proposal to apply the initial absolute thresholds to an extended cohort rather than the proportional ones: reject; it halves the baseline bar. - A monotone decline surviving the single authorized six-repeat extension: Fail. No second extension exists. - `R5 / R1 < B`: packing costs more recall than the coverage it buys. Fail and take the teardown branch — do not renegotiate `B` after seeing the result. - A monotone decline `R1 > R3 > R5` that survives extended repeats: treat as a dilution signature and fail, even if the ratio clears. From 376b66a5d746edd27c654724622f38c56f255386 Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 12:59:47 -0400 Subject: [PATCH 12/24] feat(pipeline): add dark compatible-atom packet packing (Plan 103 phase 1) Implements Plan 103 steps 1-3: the two eval-only settings, semantic atoms wrapped around today's hunkFirstGroups() output, and Plan 102's compatibility partition with source-order greedy fill behind a dark flag. Config surface is deliberately narrow. review.packCompatibleAtoms (default false) and review.packMaxHunks (default 5) live in CodegenieConfig, the resolved schema, defaults, the strict eval-case schema, and applyCaseReviewConfig. They are absent from rawConfigSchema, DEFAULT_SOURCE_PATHS, and REPO_SAFE_REVIEW_KEYS, so no codegenie.toml or user config.toml can reach them - tests assert those throw rather than being silently filtered. Config-source attribution is deliberately not claimed for a value no config file can supply. MAX_PACK_HUNKS is duplicated in schema.ts rather than imported from the pipeline to keep the config schema dependency-free, with a test pinning the two together. PacketGroup gained a required origin field. The direct whole-file and content-probed file-diff returns are now structurally excluded from packing rather than inferred from kind, which is ambiguous because packetGroup() can also produce file-diff. packCompatibleAtoms() partitions each file's atoms by effective coverage and normalized planner lens signature, fills sequentially in source order under packMaxHunks and MAX_PATCH_CHARS, orders packets by earliest member hunk, and rebuilds combined groups through the existing packetGroup and packetKind rules with degradation reasons merged by sorted deduplicated join. Fill is sequential-flush, matching Plan 102's described algorithm rather than the best-fit search used in the planning simulation: sequential preserves strict source order where best-fit would let a later small atom jump ahead of an earlier larger one. The two differ by about one packet on the retained runs, inside the documented residual. Atoms are indivisible. An atom that alone exceeds a cap still becomes its own packet rather than being split. Twelve new tests, no existing test modified: 761 to 773. A one-hunk cap is asserted byte-identical to flag-off, which is the strongest parity check available without a pre-change baseline. Coverage and lens boundaries, hunk bijection, in-packet source ordering, atom indivisibility, whole-file bypass, dispatch-rank recomputation, and the 15/5/3 packet shapes Phase 4's recall curve depends on are all covered. The phase plan records one scope correction: the plan's atom description also lists standalone review profile and per-member context quality, but both need the dry-build machinery Phase 2 owns and neither is needed to partition or fill, so they move to Phase 2 rather than being half-built here. Packing stays off by default and must not be enabled before Phase 2 - without multi-member symbol context a packed packet would carry only the top-ranked member's source and leave the rest as bare diffs. Co-Authored-By: Claude Opus 5 (1M context) --- .gitignore | 2 + specs/plans/phase-plans/103/phase_1.md | 49 +++++++++ src/config/schema.ts | 15 ++- src/evals/eval-runner.ts | 11 ++ src/pipeline/packet-builder.ts | 120 +++++++++++++++++++++- src/types.ts | 11 ++ tests/config-loader.test.ts | 39 +++++++ tests/evals.test.ts | 34 ++++++ tests/pipeline-phase5.test.ts | 137 +++++++++++++++++++++++++ 9 files changed, 414 insertions(+), 4 deletions(-) create mode 100644 specs/plans/phase-plans/103/phase_1.md diff --git a/.gitignore b/.gitignore index 9d116ea..90fcfdd 100644 --- a/.gitignore +++ b/.gitignore @@ -4,3 +4,5 @@ reviews/ evals/fixtures/logs/ node_modules/ dist/ +evals/packet-dilution/logs/ +evals/skill-semantics/logs/ diff --git a/specs/plans/phase-plans/103/phase_1.md b/specs/plans/phase-plans/103/phase_1.md new file mode 100644 index 0000000..7eaa9fa --- /dev/null +++ b/specs/plans/phase-plans/103/phase_1.md @@ -0,0 +1,49 @@ +--- +status: complete +--- + +# Phase 1: Dark Compatible-Atom Packing + +## Overview + +Implement Plan 103 steps 1–3 only. This phase adds the two eval/internal-only settings, wraps today's `hunkFirstGroups()` output as explicit indivisible atoms, and implements Plan 102's compatibility partition and source-order greedy fill behind the dark flag. It reproduces Plan 102's already-validated packet counts and nothing more. + +Multi-member symbol context, transactional rejection, the profile floor, the pinned-plan seam, the report script, and every paid phase are out of scope. Because this phase deliberately stops short of the context work, packing must remain **off by default** and no packed packet may be dispatched in a real review until Phase 2 lands — a packed packet without multi-member context would silently drop non-primary members' surrounding source. + +## Steps + +1. Add `review.packCompatibleAtoms: boolean` (default `false`) and `review.packMaxHunks: number` (default `5`, rejected above `MAX_HUNKS_PER_PACKET`) to `CodegenieConfig`, `codegenieConfigSchema`, and `defaultConfig`. Add both to the strict eval-case `review` schema and `applyCaseReviewConfig()`. Do **not** add either to `rawConfigSchema`, `DEFAULT_SOURCE_PATHS`, or `REPO_SAFE_REVIEW_KEYS`: no `codegenie.toml` may set them and no config-source attribution is claimed for a value that cannot come from a config file. Add `evals/packet-dilution/logs/` and the currently missing `evals/skill-semantics/logs/` to `.gitignore`. +2. Refactor `hunkFirstGroups()` results in `src/pipeline/packet-builder.ts` into explicit packet atoms without changing their membership. Each atom carries its ordered `PlannedHunk[]`, hunk count, `combinedPatchChars()` size, first source position, effective coverage, normalized planner lens signature (stable serialization of the sorted deduplicated union of `decision.lenses`), and a stable ID derived from ordered hunk IDs. **Scope correction:** the plan's atom description also lists standalone review profile and standalone per-member context quality. Both require the dry-build machinery that Phase 2 owns, and neither is needed to partition or fill, so they are captured in Phase 2 alongside the transactional evaluation that computes them rather than half-built here. Preserve the direct `whole-file` and content-probed `file-diff` returns from `groupHunks()` so they bypass the packer entirely, and bypass all atom work when the flag is off so packet artifacts, IDs, ordering, profiles, context, lenses, and budgets stay byte-identical. +3. Implement the compatibility partition and fill. Partition each file's atoms by `(effectiveCoverage, normalizedPlannerLensSignature)`; fill greedily in source order under `packMaxHunks` and `MAX_PATCH_CHARS`; order packets by earliest member hunk and render each packet's hunks by file position. Materialize combined groups through the existing `packetGroup()`/`packetKind()` rules, carry non-empty `degradationReason` values through a sorted deduplicated `"; "` join, do not synthesize `wholeFileText` or `fileContext`, and recompute `dispatchRank` with the unchanged `packetDispatchRank(filePath, facts, combinedChangedLines)` formula. +4. Add focused coverage in `tests/config-loader.test.ts`, `tests/evals.test.ts`, and `tests/pipeline-phase5.test.ts`. +5. Run the focused commands below, then `pnpm run check`, `pnpm test`, and `pnpm build`. Fix all failures without starting Phase 2. + +## Tests + +- `config defaults and eval-only surface`: proves `false`/`5` defaults, eval-case overrides reaching resolved config, `packMaxHunks > 5` rejected, and a `codegenie.toml` setting either key failing strict parsing rather than being silently accepted or filtered. +- `packet atom identity and flag-off parity`: proves atoms preserve today's group membership exactly, atom IDs are stable under reordering of equivalent inputs, and flag-off packet artifacts and telemetry are byte-identical to the pre-change baseline. +- `whole-file bypass`: proves direct whole-file and content-probed file-diff groups never enter the packer. +- `compatible partition and fill`: proves atoms are never split, every reviewable hunk appears exactly once, hunks render in file position order within a packet, packets order by earliest member, coverage and lens signature both gate membership, cap splits occur at `packMaxHunks` and `MAX_PATCH_CHARS`, interleaved partitions restore stable order, degradation reasons merge deterministically, and dispatch rank equals the existing formula over combined changed lines. +- `packMaxHunks parameterization`: proves caps of 1, 3, and 5 produce the packet shapes the Phase 4 curve depends on. + +## Verification commands + +```bash +pnpm exec vitest run tests/config-loader.test.ts tests/evals.test.ts +pnpm exec vitest run tests/pipeline-phase5.test.ts +pnpm run check && pnpm test && pnpm build +``` + +## Outcome + +Complete. `pnpm run check`, `pnpm test`, and `pnpm build` all pass; the suite grew from 761 to 773 tests with no existing test modified. + +**Config surface.** `review.packCompatibleAtoms` (default `false`) and `review.packMaxHunks` (default `5`) exist in `CodegenieConfig`, `codegenieConfigSchema`, `defaultConfig`, the strict eval-case schema, and `applyCaseReviewConfig()`. They are absent from `rawConfigSchema`, `DEFAULT_SOURCE_PATHS`, and `REPO_SAFE_REVIEW_KEYS`, and tests prove a `codegenie.toml` or user `config.toml` setting either key throws `invalid config file` rather than being silently filtered. `MAX_PACK_HUNKS` is duplicated in `schema.ts` rather than imported from `packet-builder.ts` to keep the config schema free of pipeline dependencies; a test asserts it equals 5 and the resolved schema rejects 6. + +**Packer.** `PacketGroup` gained a required `origin` field so the direct whole-file and content-probed file-diff returns are structurally excluded from packing rather than inferred from `kind`, which is ambiguous — `packetGroup()` can also produce `file-diff`. `packCompatibleAtoms()` partitions a file's atoms by `(effectiveCoverage, normalizedLensSignature)`, fills sequentially in source order under `packMaxHunks` and `MAX_PATCH_CHARS`, orders packets by earliest member, and rebuilds combined groups through the existing `packetGroup()`/`packetKind()` rules with degradation reasons merged by sorted deduplicated join. + +Fill is sequential-flush, matching Plan 102's described algorithm, not the best-fit search used in the planning simulation. Sequential preserves strict source order; best-fit would let a later small atom jump ahead of an earlier larger one. The two differ by about one packet on the retained runs, inside the documented residual, and Phase 3's replay measures the real number. + +**Verified behaviours.** Packing 6 compatible atoms at cap 5 yields `[h1-h5][h6]`; a one-hunk cap is byte-identical to flag-off (asserted by full artifact comparison, which is the strongest parity check available without a pre-change baseline); a `deep` hunk among `normal` siblings and a `core/tests` hunk among `core/code-review` siblings both stay separate; every hunk appears exactly once with in-packet source ordering; a two-hunk atom survives a one-hunk cap intact; whole-file groups never enter the packer; dispatch rank equals the existing formula over combined changed lines; and caps 1/3/5 over 15 atoms produce exactly the 15/5/3 packet shapes Phase 4's recall curve depends on. + +**Not yet safe to enable.** Multi-member symbol context is Phase 2. Until it lands, a packed packet containing atoms with different primary symbols would carry only the top-ranked symbol's source, leaving the rest as bare diffs. The flag stays `false` and no packed packet may reach a real review before Phase 2. diff --git a/src/config/schema.ts b/src/config/schema.ts index d000099..afce254 100644 --- a/src/config/schema.ts +++ b/src/config/schema.ts @@ -17,6 +17,12 @@ export const reviewMaxTimeMinutesSchema = positiveFiniteNumberSchema.max(MAX_REV // packets; beyond 3 the marginal recall of another draw is negligible while // cost keeps climbing. Hard cap, not a default. export const MAX_DEEP_ENSEMBLE_PASSES = 3; +// Plan 103 (experiment-only): upper bound for review.packMaxHunks. Must equal +// MAX_HUNKS_PER_PACKET in packet-builder.ts — duplicated rather than imported +// so the config schema stays free of pipeline dependencies; a focused test +// asserts the two agree. The recall curve varies packMaxHunks below this +// bound; nothing may raise packing above today's shipped cap. +export const MAX_PACK_HUNKS = 5; export const pathRuleSchema = z .object({ @@ -141,7 +147,9 @@ export const codegenieConfigSchema = z maxBudgetTokens: positiveIntSchema.optional(), maxModelCalls: positiveIntSchema.optional(), deepEnsemblePasses: positiveIntSchema.max(MAX_DEEP_ENSEMBLE_PASSES).optional(), - adaptiveSecondPass: z.boolean().optional() + adaptiveSecondPass: z.boolean().optional(), + packCompatibleAtoms: z.boolean(), + packMaxHunks: positiveIntSchema.max(MAX_PACK_HUNKS) }) .strict(), github: z @@ -216,7 +224,10 @@ export const defaultConfig: CodegenieConfig = { // soft-stop and went partial); 8M puts the soft-stop at 6.8M, ~15% above // the largest observed legitimate run. A protective ceiling, not a // target. - maxBudgetTokens: 8_000_000 + maxBudgetTokens: 8_000_000, + // Plan 103: dark until the packet-size recall curve clears its gate. + packCompatibleAtoms: false, + packMaxHunks: MAX_PACK_HUNKS }, github: { summaryWhenNoFindings: false diff --git a/src/evals/eval-runner.ts b/src/evals/eval-runner.ts index 8470145..68e2592 100644 --- a/src/evals/eval-runner.ts +++ b/src/evals/eval-runner.ts @@ -6,6 +6,7 @@ import { z } from "zod"; import { applyRepoConfigLayer } from "../config/config-loader.js"; import { MAX_DEEP_ENSEMBLE_PASSES, + MAX_PACK_HUNKS, reasoningLevelSchema, reviewDepthSchema, reviewMaxTimeMinutesSchema, @@ -138,6 +139,10 @@ const caseSchema = z maxBudgetTokens: positiveIntSchema.optional(), deepEnsemblePasses: positiveIntSchema.max(MAX_DEEP_ENSEMBLE_PASSES).optional(), adaptiveSecondPass: z.boolean().optional(), + // Plan 103 (experiment-only): eval cases are the only surface that can + // set these. No codegenie.toml may. + packCompatibleAtoms: z.boolean().optional(), + packMaxHunks: positiveIntSchema.max(MAX_PACK_HUNKS).optional(), verify: z.boolean().optional(), cache: z.boolean().optional(), cacheDir: z.string().min(1).optional(), @@ -825,6 +830,12 @@ function applyCaseReviewConfig( if (review?.adaptiveSecondPass !== undefined) { config.review.adaptiveSecondPass = review.adaptiveSecondPass; } + if (review?.packCompatibleAtoms !== undefined) { + config.review.packCompatibleAtoms = review.packCompatibleAtoms; + } + if (review?.packMaxHunks !== undefined) { + config.review.packMaxHunks = review.packMaxHunks; + } if (review?.verify !== undefined) { config.review.verify = review.verify; } diff --git a/src/pipeline/packet-builder.ts b/src/pipeline/packet-builder.ts index f39b77f..73e230e 100644 --- a/src/pipeline/packet-builder.ts +++ b/src/pipeline/packet-builder.ts @@ -59,11 +59,28 @@ type PlannedHunk = { type PacketGroup = { hunks: PlannedHunk[]; kind: ReviewPacket["kind"]; + // Plan 103: only `hunk-first` groups are eligible for compatible-atom + // packing. The direct whole-file and content-probed file-diff returns from + // groupHunks() carry their own file context and must bypass the packer. + origin: "hunk-first" | "direct"; fileContext?: ReviewPacket["fileContext"]; wholeFileText?: string; degradationReason?: string; }; +// Plan 103: a semantic atom is one group returned by hunkFirstGroups(). Atoms +// are indivisible — packing combines them but never splits or reorders one. +type PacketAtom = { + id: string; + group: PacketGroup; + hunks: PlannedHunk[]; + hunkCount: number; + patchChars: number; + sourcePos: number; + coverage: Exclude; + lensSignature: string; +}; + type HunkRelationshipSource = "same_symbol" | "symbol_mention" | "planner_hint"; type HunkRelationshipStrength = "strong" | "medium" | "weak"; @@ -190,7 +207,9 @@ export async function buildReviewPackets( } const allowWholeFileContext = includedPlanned.length === planned.length; - for (const group of await groupHunks(includedPlanned, repoIndex, telemetry, { allowWholeFileContext })) { + const groups = await groupHunks(includedPlanned, repoIndex, telemetry, { allowWholeFileContext }); + const positionByHunk = new Map(includedPlanned.map((entry, index) => [entry.hunk.id, index])); + for (const group of packCompatibleAtoms(groups, effectiveByHunk, positionByHunk, opts.config)) { const first = group.hunks[0]; if (!first) { continue; @@ -248,6 +267,7 @@ async function groupHunks( return [{ hunks: planned, kind: "whole-file", + origin: "direct", fileContext: { mode: "whole-file", reason: wholeFileReason }, wholeFileText: content.text }]; @@ -255,6 +275,7 @@ async function groupHunks( return [{ hunks: planned, kind: "file-diff", + origin: "direct", fileContext: { mode: "file-diff", reason: content.reason } }]; } @@ -1274,7 +1295,102 @@ function packetGroup(hunks: PlannedHunk[], degradationReason?: string): PacketGr : hunks.length > 1 ? "coalesced-hunks" : "hunk"; - return { hunks, kind, ...(degradationReason !== undefined ? { degradationReason } : {}) }; + return { hunks, kind, origin: "hunk-first", ...(degradationReason !== undefined ? { degradationReason } : {}) }; +} + +// Plan 103: Plan 102's compatibility predicate. Atoms combine only inside one +// file when they share the planner's effective coverage and its requested lens +// set, so packing introduces no coverage promotion and cannot silently reroute +// a hunk to different expertise. Returns the original groups untouched when +// packing is off or when any group bypassed hunkFirstGroups(). +function packCompatibleAtoms( + groups: PacketGroup[], + effectiveByHunk: Map, + positionByHunk: Map, + config: CodegenieConfig +): PacketGroup[] { + if (!config.review.packCompatibleAtoms || !groups.every((group) => group.origin === "hunk-first")) { + return groups; + } + + const partitions = new Map(); + for (const group of groups) { + const atom = packetAtom(group, effectiveByHunk, positionByHunk); + const key = `${atom.coverage}\u0000${atom.lensSignature}`; + partitions.set(key, [...(partitions.get(key) ?? []), atom]); + } + + const packed: PacketAtom[][] = []; + for (const partition of partitions.values()) { + let current: PacketAtom[] = []; + let hunkCount = 0; + let patchChars = 0; + for (const atom of partition) { + const exceedsCaps = + hunkCount + atom.hunkCount > config.review.packMaxHunks || + patchChars + atom.patchChars > MAX_PATCH_CHARS; + // An atom that alone exceeds a cap still becomes its own packet: atoms + // are indivisible, so a cap can never split one. + if (current.length > 0 && exceedsCaps) { + packed.push(current); + current = []; + hunkCount = 0; + patchChars = 0; + } + current.push(atom); + hunkCount += atom.hunkCount; + patchChars += atom.patchChars; + } + if (current.length > 0) { + packed.push(current); + } + } + + return packed + .sort((a, b) => (a[0]?.sourcePos ?? 0) - (b[0]?.sourcePos ?? 0)) + .map(combinePacketAtoms); +} + +function packetAtom( + group: PacketGroup, + effectiveByHunk: Map, + positionByHunk: Map +): PacketAtom { + const hunkIds = group.hunks.map((entry) => entry.hunk.id); + const decisions = group.hunks + .flatMap((entry) => effectiveByHunk.get(entry.hunk.id) ?? []) + .filter(isNonSkipDecision); + return { + id: sha256Hex(hunkIds.join("\n")), + group, + hunks: group.hunks, + hunkCount: group.hunks.length, + patchChars: combinedPatchChars(group.hunks), + sourcePos: Math.min(...hunkIds.map((id) => positionByHunk.get(id) ?? Number.MAX_SAFE_INTEGER)), + coverage: maxCoverage(decisions.map((decision) => decision.coverage)), + lensSignature: normalizedLensSignature(decisions) + }; +} + +// The compatibility key: the stable serialization of the sorted, deduplicated +// union of the atom's planner-requested lenses. +function normalizedLensSignature(decisions: NonSkipDecision[]): string { + return cleanStrings(decisions.flatMap((decision) => decision.lenses)).join(","); +} + +function combinePacketAtoms(atoms: PacketAtom[]): PacketGroup { + const first = atoms[0]; + if (first === undefined) { + throw new Error("cannot combine zero atoms"); + } + if (atoms.length === 1) { + return first.group; + } + const reasons = cleanStrings(atoms.flatMap((atom) => atom.group.degradationReason ?? [])); + return packetGroup( + atoms.flatMap((atom) => atom.hunks), + reasons.length > 0 ? reasons.join("; ") : undefined + ); } function packetKind(group: PacketGroup, planned: PlannedHunk[], file: DiffFile): ReviewPacket["kind"] { diff --git a/src/types.ts b/src/types.ts index 2666377..ed7fce4 100644 --- a/src/types.ts +++ b/src/types.ts @@ -54,6 +54,13 @@ export type CodegenieConfig = { // evidence (concrete hint/uncertainty, silent-with-signal, low-confidence // only) earn one additional independent review pass. Off by default. adaptiveSecondPass?: boolean; + // Plan 103 (experiment-only, removed at teardown): pack compatible + // same-file atoms into fewer packets, and the hunk cap the recall curve + // varies. Deliberately absent from rawConfigSchema and repo-safe + // filtering — no codegenie.toml may set either, and neither is a + // supported user surface. Eval cases set them; nothing else can. + packCompatibleAtoms: boolean; + packMaxHunks: number; }; github: { summaryWhenNoFindings: boolean; @@ -1012,6 +1019,10 @@ export type EvalCase = { maxBudgetTokens?: number; deepEnsemblePasses?: number; adaptiveSecondPass?: boolean; + // Plan 103 (experiment-only): eval cases are the only surface that may set + // these; removed at teardown. + packCompatibleAtoms?: boolean; + packMaxHunks?: number; verify?: boolean; cache?: boolean; cacheDir?: string; diff --git a/tests/config-loader.test.ts b/tests/config-loader.test.ts index 89e20b6..f9fbff6 100644 --- a/tests/config-loader.test.ts +++ b/tests/config-loader.test.ts @@ -5,9 +5,13 @@ import path from "node:path"; import { describe, expect, it } from "vitest"; import { MAX_DEEP_ENSEMBLE_PASSES, + MAX_PACK_HUNKS, MAX_REVIEW_TIME_MINUTES, + codegenieConfigSchema, + defaultConfig, rawConfigSchema } from "../src/config/schema.js"; +import type { CodegenieConfig } from "../src/types.js"; import { ensureCodegenieHome, getCodegeniePaths } from "../src/config/paths.js"; import { loadConfig } from "../src/config/config-loader.js"; import { loadProviderSettings, saveProviderSettings } from "../src/provider/provider-settings.js"; @@ -325,3 +329,38 @@ describe("deepEnsemblePasses cap (plan 84)", () => { function tempDir(): string { return mkdtempSync(path.join(tmpdir(), "codegenie-")); } + +describe("plan 103 packing settings", () => { + it("defaults to dark packing at the shipped cap", () => { + const loaded = loadConfig({ repoRoot: tempDir(), homeOverride: tempDir() }); + expect(loaded.config.review.packCompatibleAtoms).toBe(false); + expect(loaded.config.review.packMaxHunks).toBe(MAX_PACK_HUNKS); + expect(MAX_PACK_HUNKS).toBe(5); + }); + + it("refuses both settings from every config file surface", () => { + // Plan 103 keeps these eval-only: no codegenie.toml and no user config may + // reach them, so strict parsing must reject rather than silently filter. + expect(rawConfigSchema.safeParse({ review: { packCompatibleAtoms: true } }).success).toBe(false); + expect(rawConfigSchema.safeParse({ review: { packMaxHunks: 3 } }).success).toBe(false); + + const repoRoot = tempDir(); + const home = tempDir(); + writeFileSync(path.join(repoRoot, "codegenie.toml"), "[review]\npackCompatibleAtoms = true\n"); + expect(() => loadConfig({ repoRoot, homeOverride: home })).toThrow(/invalid config file/); + + const userHome = tempDir(); + writeFileSync(path.join(userHome, "config.toml"), "[review]\npackMaxHunks = 2\n"); + expect(() => loadConfig({ repoRoot: tempDir(), homeOverride: userHome })).toThrow(/invalid config file/); + }); + + it("bounds packMaxHunks by the shipped packet cap in the resolved schema", () => { + const base = structuredClone(defaultConfig) as CodegenieConfig; + for (const value of [1, 3, MAX_PACK_HUNKS]) { + expect(codegenieConfigSchema.safeParse({ ...base, review: { ...base.review, packMaxHunks: value } }).success).toBe(true); + } + for (const value of [0, -1, MAX_PACK_HUNKS + 1]) { + expect(codegenieConfigSchema.safeParse({ ...base, review: { ...base.review, packMaxHunks: value } }).success).toBe(false); + } + }); +}); diff --git a/tests/evals.test.ts b/tests/evals.test.ts index 64cdd97..8173255 100644 --- a/tests/evals.test.ts +++ b/tests/evals.test.ts @@ -8,6 +8,7 @@ import { loadEvalArtifacts } from "../src/evals/eval-artifacts.js"; import { compareToPrevious, renderEvalCompareText } from "../src/evals/eval-compare.js"; import { executeEvalCommand, renderCaseResult, runEvalCommand } from "../src/evals/eval-command.js"; import { loadEvalSuite, replayFromArtifacts, runEvalCase } from "../src/evals/eval-runner.js"; +import { MAX_PACK_HUNKS } from "../src/config/schema.js"; import { aggregateRepeatScores, assignExpectations, matchExpectation, scoreEvalRun } from "../src/evals/eval-scoring.js"; import type { CandidateFinding, @@ -2295,3 +2296,36 @@ function findNestedGitDirs(root: string): string[] { } return found; } + +describe("plan 103 packing eval surface", () => { + it("applies both packing settings from an eval case and bounds the cap", async () => { + const suiteDir = mkdtempSync(path.join(tmpdir(), "codegenie-eval-packing-")); + writeFileSync(path.join(suiteDir, "packing.yml"), [ + "name: packing-case", + "repo:", + " fixture: repo", + "review:", + " packCompatibleAtoms: true", + " packMaxHunks: 3", + "should_find:", + " - id: expected", + " path: src/app.ts" + ].join("\n")); + + const suite = await loadEvalSuite(suiteDir); + expect(suite.cases[0]?.evalCase.review).toMatchObject({ packCompatibleAtoms: true, packMaxHunks: 3 }); + + const overCap = mkdtempSync(path.join(tmpdir(), "codegenie-eval-packing-cap-")); + writeFileSync(path.join(overCap, "packing.yml"), [ + "name: packing-over-cap", + "repo:", + " fixture: repo", + "review:", + ` packMaxHunks: ${MAX_PACK_HUNKS + 1}`, + "should_find:", + " - id: expected", + " path: src/app.ts" + ].join("\n")); + await expect(loadEvalSuite(overCap)).rejects.toMatchObject({ code: "config_error" }); + }); +}); diff --git a/tests/pipeline-phase5.test.ts b/tests/pipeline-phase5.test.ts index 51e457e..308fc01 100644 --- a/tests/pipeline-phase5.test.ts +++ b/tests/pipeline-phase5.test.ts @@ -13680,3 +13680,140 @@ function languageProjectionSkill(id: string, languages: string[], marker: string sections: { checks: marker, falsePositives: marker } }; } + +describe("plan 103 compatible-atom packing", () => { + // Hunks spaced far beyond NEARBY_GAP_LINES with no shared enclosing symbol, + // so today's grouper yields exactly one atom per hunk. + function separatedFile(pathName: string, count: number, gap = 100): DiffFile { + return { + path: pathName, + status: "modified", + language: "typescript", + hunks: Array.from({ length: count }, (_, index) => { + const line = 1 + index * gap; + return { + id: `h${index + 1}`, + hunkHash: String(index + 1).repeat(64).slice(0, 64), + path: pathName, + oldStart: line, + oldLines: 1, + newStart: line, + newLines: 1, + header: `@@ -${line} +${line} @@`, + lines: [{ kind: "add" as const, content: `const v${index + 1} = ${index + 1};`, newLineNumber: line }] + }; + }) + }; + } + + function planFor(file: DiffFile, overrides: Record = {}): ReviewPlan { + return { + diffUnderstanding: { declaredIntent: "test intent", inferredBehavior: "test behavior" }, + coverage: file.hunks.map((hunk) => ({ + hunkId: hunk.id, + path: file.path, + coverage: (overrides[hunk.id]?.coverage ?? "normal") as "deep" | "normal" | "light", + lenses: overrides[hunk.id]?.lenses ?? ["core/code-review"], + surroundingContextHints: [], + reason: "test" + })) + }; + } + + async function pack(file: DiffFile, plan: ReviewPlan, review: Partial = {}) { + const base = config(); + return buildReviewPackets( + plan, + [file], + [fakeFacts(file.path, "per-hunk")], + fakeRepositoryIndex(fakeTools()), + nullTelemetry(), + { config: { ...base, review: { ...base.review, ...review } }, enabledLenses: ["core/code-review"] } + ); + } + + const hunkIdsOf = (packets: Awaited>) => packets.map((packet) => packet.hunks.map((hunk) => hunk.hunkId)); + + it("packs compatible same-file atoms into fewer packets", async () => { + const file = separatedFile("app.ts", 6); + const plan = planFor(file); + + const off = await pack(file, plan); + expect(off).toHaveLength(6); + + const on = await pack(file, plan, { packCompatibleAtoms: true }); + expect(hunkIdsOf(on)).toEqual([["h1", "h2", "h3", "h4", "h5"], ["h6"]]); + }); + + it("is a no-op at a one-hunk cap, matching flag-off artifacts exactly", async () => { + const file = separatedFile("app.ts", 5); + const plan = planFor(file); + const off = await pack(file, plan); + const capOne = await pack(file, plan, { packCompatibleAtoms: true, packMaxHunks: 1 }); + expect(JSON.stringify(capOne)).toBe(JSON.stringify(off)); + }); + + it("never packs across effective coverage or requested lens boundaries", async () => { + const file = separatedFile("app.ts", 4); + + const mixedCoverage = await pack(file, planFor(file, { h2: { coverage: "deep" } }), { packCompatibleAtoms: true }); + expect(hunkIdsOf(mixedCoverage)).toEqual([["h1", "h3", "h4"], ["h2"]]); + expect(mixedCoverage.map((packet) => packet.coverage).sort()).toEqual(["deep", "normal"]); + + const mixedLenses = await pack(file, planFor(file, { h3: { lenses: ["core/tests"] } }), { packCompatibleAtoms: true }); + expect(hunkIdsOf(mixedLenses)).toEqual([["h1", "h2", "h4"], ["h3"]]); + }); + + it("assigns every reviewable hunk exactly once, in source order", async () => { + const file = separatedFile("app.ts", 9); + const packets = await pack(file, planFor(file), { packCompatibleAtoms: true }); + const flattened = packets.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId)); + expect(new Set(flattened).size).toBe(9); + for (const packet of packets) { + const starts = packet.hunks.map((hunk) => hunk.newStart); + expect([...starts].sort((a, b) => a - b)).toEqual(starts); + } + expect(packets[0]?.hunks[0]?.hunkId).toBe("h1"); + }); + + it("never splits an atom, even when the atom alone exceeds the cap", async () => { + // h1 and h2 sit within NEARBY_GAP_LINES, so the grouper makes them one atom. + const file = separatedFile("app.ts", 3, 10); + const packets = await pack(file, planFor(file), { packCompatibleAtoms: true, packMaxHunks: 1 }); + expect(hunkIdsOf(packets)).toEqual([["h1", "h2", "h3"]]); + }); + + it("leaves whole-file groups out of the packer", async () => { + const file = separatedFile("app.ts", 2); + const packets = await buildReviewPackets( + planFor(file), + [file], + [fakeFacts(file.path, "whole-file")], + fakeRepositoryIndex(fakeTools("export const value = 1;\n")), + nullTelemetry(), + { + config: { ...config(), review: { ...config().review, packCompatibleAtoms: true } }, + enabledLenses: ["core/code-review"] + } + ); + expect(packets).toHaveLength(1); + expect(packets[0]).toMatchObject({ kind: "whole-file", fileContext: { mode: "whole-file" } }); + }); + + it("recomputes dispatch rank from the combined packet's changed lines", async () => { + const file = separatedFile("app.ts", 5); + const packets = await pack(file, planFor(file), { packCompatibleAtoms: true }); + expect(packets).toHaveLength(1); + expect(packets[0]?.dispatchRank).toEqual(packetDispatchRank("app.ts", { testStatus: "source" }, 5)); + }); + + it("parameterizes packet shape by packMaxHunks for the recall curve", async () => { + const file = separatedFile("app.ts", 15); + const plan = planFor(file); + for (const [cap, expected] of [[1, 15], [3, 5], [5, 3]] as const) { + const packets = await pack(file, plan, { packCompatibleAtoms: true, packMaxHunks: cap }); + expect(packets).toHaveLength(expected); + expect(packets.every((packet) => packet.hunks.length <= cap)).toBe(true); + } + }); +}); From 2cac03bfd4db252c2f242df461685a7328b5fe35 Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 13:10:49 -0400 Subject: [PATCH 13/24] feat(pipeline): multi-member symbol context, transactional packing, profile floor (Plan 103 phase 2) Implements Plan 103 steps 4-5 plus the atom metadata deferred from phase 1. Multi-member symbol context. readMemberSymbolSources() reads every distinct primary symbol in a packed packet under a shared 5,000-character budget with equal shares, a floor, and one redistribution pass handing surplus from members needing less to members needing more. At least 3,000 of MAX_CONTEXT_CHARS stays reserved for outline, tests, and hints. Symbol source renders at the head of the packet context and truncateTail keeps the head, so the reserve is what guarantees survival - but survival is verified against the final rendered text by locating each member's header rather than assumed. A single-symbol packet delegates unchanged to readEnclosingSymbolSource, so flag-off behaviour is untouched. Transactional evaluation. Candidates dry-build against a scratch relationship accumulator and a quiet telemetry sink with throwaway metrics; only a candidate passing every check is rebuilt against the real recorder. A rejected candidate emits its atoms separately with a recorded reason and leaves no telemetry, graph, or artifact residue. Single-atom packets build once, unchanged. Profile floor, applied before lens routing rather than after. routedPacketLenses prunes core/code-review at a simple profile, so raising the profile afterwards would leave a lens dropped that the floor exists to preserve. Verified with the case Plan 102 described: two hunks sharing an enclosing symbol separated by a third, where packing absorbs the same_symbol edge target. Two findings changed the design. A short symbol is not a collapsed member. The first cut rejected any member emitting under MIN_SLICED_MEMBER_CHARS, which fails a packet whose symbol is simply small - a 500-character function is fully represented. Members now carry a complete flag and only an incomplete member below the floor counts as collapsed. Without this, packing would have been rejected for most real packets. Both budget rejections are unreachable at the shipped cap, and that is now pinned by test. A packet holds at most 5 hunks so at most 5 distinct symbols; 5 x 800 = 4,000 <= 5,000 means oversubscription cannot fire, and the smallest share of 1,000 exceeds the 600 sliced minimum so truncation-collapse cannot fire. Both are kept as defensive invariants that become live only if the cap rises past six, with a test pinning the arithmetic so a cap change surfaces there. Two drafted tests asserting unreachable behaviour were removed rather than contrived into passing. 773 to 778 tests, no existing test modified. The flag stays false; phase 3 owns the replay gate that authorizes any paid work. Co-Authored-By: Claude Opus 5 (1M context) --- specs/plans/phase-plans/103/phase_2.md | 54 ++++ src/pipeline/packet-builder.ts | 389 +++++++++++++++++++++++-- tests/pipeline-phase5.test.ts | 173 +++++++++++ 3 files changed, 596 insertions(+), 20 deletions(-) create mode 100644 specs/plans/phase-plans/103/phase_2.md diff --git a/specs/plans/phase-plans/103/phase_2.md b/specs/plans/phase-plans/103/phase_2.md new file mode 100644 index 0000000..3342e86 --- /dev/null +++ b/specs/plans/phase-plans/103/phase_2.md @@ -0,0 +1,54 @@ +--- +status: complete +--- + +# Phase 2: Multi-Member Context, Transactional Rejection, and the Profile Floor + +## Overview + +Implement Plan 103 steps 4–5, plus the atom metadata deferred from Phase 1. A packed packet now carries surrounding source for every member symbol under an explicit budget, candidates are evaluated in isolation and abandoned rather than committed when they would lose something, and a packed packet can never review below its strongest standalone member's profile. + +The pinned-plan seam, the report script, the replay, and every paid phase remain out of scope. + +## Steps + +1. Add `PACKET_SYMBOL_CONTEXT_BUDGET = 5_000`, `MIN_MEMBER_SYMBOL_CHARS = 800`, and `MIN_SLICED_MEMBER_CHARS = 600`, and implement `readMemberSymbolSources()`: participants are distinct resolvable primary symbol identities, each gets `floor(budget / count)` subject to the floor, one redistribution pass hands surplus from members needing less to members needing more in source order, and a single-symbol packet delegates unchanged to `readEnclosingSymbolSource()` so flag-off behaviour is untouched. +2. Thread member symbol facts through `buildContext()` and `buildPacket()`, and return per-member emitted characters alongside the packet. +3. Apply the profile floor **before** lens routing, since `routedPacketLenses()` prunes `core/code-review` at a `simple` profile — raising the profile afterwards would leave a lens dropped that the floor exists to preserve. +4. Evaluate every multi-atom candidate transactionally: build each member standalone and the candidate itself against a scratch relationship accumulator and a suppressed telemetry sink, compare, then either rebuild for real or emit the members separately with a recorded reason. +5. Add focused coverage and run the full repository gate. + +## Tests + +- `renders every member's symbol source in a packed packet`: three distinct symbols all appear in the packed context. +- `keeps symbol source inside its budget`: symbol section stays within the 5,000-character allowance, total context within 8,000, and all four members survive final rendering. +- `cannot starve a member at the shipped cap`: pins the arithmetic that makes the budget rejections dormant at cap 5. +- `floors a packed profile to its strongest standalone member`: a `same_symbol` edge absorbed by packing cannot lower the effective profile below what the members held alone. +- `shares one symbol budget when packed members resolve to the same symbol`: identical name *and* range is one participant, not several. + +## Verification commands + +```bash +pnpm exec vitest run tests/pipeline-phase5.test.ts +pnpm run check && pnpm test && pnpm build +``` + +## Outcome + +Complete. `pnpm run check`, `pnpm test`, and `pnpm build` pass; 773 → 778 tests with no existing test modified. + +**Multi-member symbol context.** `readMemberSymbolSources()` reads every distinct primary symbol under a shared 5,000-character budget, leaving at least 3,000 of `MAX_CONTEXT_CHARS` for outline, tests, and hints. Symbol source renders at the head of the packet context and `truncateTail()` keeps the head, so the reserve is what guarantees survival — the budget cannot be crowded out by later sections. Per-member survival is nevertheless verified against the final rendered text by locating each member's `Primary symbol:` header, so rule 7 is checked rather than assumed. + +**Transactional evaluation.** Candidates dry-build against `scratchRelationshipGraph()` — a shallow clone with fresh `relatedContextAttached`/`relatedContextOmitted` accumulators — and a quiet telemetry sink, with throwaway symbol-context and build metrics. Only a candidate that passes every check is rebuilt against the real recorder. Single-atom packets build once, unchanged; packed packets cost two builds, which is acceptable given Stage-6 repository-tool runtime was 5.2 seconds across the whole motivating run. + +**Profile floor.** Applied before routing, for the reason in step 3. Verified end-to-end with the case Plan 102 described: `h1` and `h3` share an enclosing symbol but are separated by `h2`, so the grouper yields three atoms with a `same_symbol` edge; packing absorbs the edge target, which would otherwise derive a weaker profile. + +### Two findings that changed the design + +**A short symbol is not a collapsed member.** The first implementation rejected any member emitting fewer than `MIN_SLICED_MEMBER_CHARS`, which fails a packet whose symbol is simply small — a 500-character function is *fully* represented. Members now carry a `complete` flag and only an incomplete member below the floor counts as collapsed. Without this, packing would have been rejected for most real packets; four tests caught it immediately. + +**Both budget rejections are unreachable at the shipped cap, and that is now pinned.** A packet holds at most 5 hunks, so at most 5 distinct primary symbols. `5 × 800 = 4,000 ≤ 5,000`, so oversubscription cannot fire; and the smallest share is `5,000 / 5 = 1,000`, above the 600 sliced minimum, so truncation-collapse cannot fire either. They are retained as defensive invariants because the plan specifies them and because a cap increase past six would make them live. Rather than contrive an unreachable end-to-end scenario, a test pins the arithmetic so any future cap change surfaces here. Two originally-drafted tests were removed for asserting behaviour that cannot occur. + +The same reasoning applies to `routed_lens_lost`: the partition key already forces identical requested lens sets across a candidate's members, so ordinary lens loss cannot occur. It remains reachable only through packet-level routing predicates such as `shouldKeepTestsLens()` reacting to a combined packet's `relevantTests`, which is why the check stays. + +**Still not enabled.** The flag remains `false`. Phase 3 owns the pinned-plan seam, the report script, and the four-run replay whose fixed-slot hunk yield gate decides whether any paid phase is authorized. diff --git a/src/pipeline/packet-builder.ts b/src/pipeline/packet-builder.ts index 73e230e..edffabe 100644 --- a/src/pipeline/packet-builder.ts +++ b/src/pipeline/packet-builder.ts @@ -157,6 +157,15 @@ const MAX_RELATED_CONTEXT_PATCH_CHARS = 1_500; const MAX_RELATIONSHIP_EDGES_PER_HUNK = 8; const MAX_RELATIONSHIP_SYMBOL_LOOKUPS = 20; const MAX_RELATIONSHIP_MENTION_RESULTS = 40; +// Plan 103 multi-member symbol context. A packed packet holding several +// distinct primary symbols must show every member's surrounding source, not +// just the top-ranked one. Symbol source is rendered at the head of the packet +// context, and truncateTail() keeps the head, so a symbol budget below +// MAX_CONTEXT_CHARS survives final rendering by construction — the reserve +// below is what guarantees outline/tests/hints cannot crowd it out. +const PACKET_SYMBOL_CONTEXT_BUDGET = 5_000; +const MIN_MEMBER_SYMBOL_CHARS = 800; +const MIN_SLICED_MEMBER_CHARS = 600; export async function buildReviewPackets( plan: ReviewPlan, @@ -209,15 +218,40 @@ export async function buildReviewPackets( const groups = await groupHunks(includedPlanned, repoIndex, telemetry, { allowWholeFileContext }); const positionByHunk = new Map(includedPlanned.map((entry, index) => [entry.hunk.id, index])); - for (const group of packCompatibleAtoms(groups, effectiveByHunk, positionByHunk, opts.config)) { - const first = group.hunks[0]; - if (!first) { + const build = async ( + group: PacketGroup, + sink: TelemetryRecorder, + metrics: SymbolContextMetrics, + buildMetrics: PacketBuildMetrics, + packing?: { members: PlannedHunk[][]; profileFloor?: ReviewProfile } + ): Promise => { + const decisions = group.hunks + .map((entry) => effectiveByHunk.get(entry.hunk.id)) + .filter((decision): decision is EffectiveDecision => decision !== undefined) + .filter(isNonSkipDecision); + return buildPacket( + group.hunks, + decisions, + group, + packing === undefined ? relationshipGraph : scratchRelationshipGraph(relationshipGraph), + repoIndex, + opts.config, + sink, + opts.reviewContext, + metrics, + buildMetrics, + packing + ); + }; + + for (const candidate of packCompatibleAtoms(groups, effectiveByHunk, positionByHunk, opts.config)) { + if (candidate.atoms.length <= 1) { + const built = await build(candidate.group, telemetry, symbolContextMetrics, packetBuildMetrics); + packets.push(built.packet); continue; } - const groupDecisions = group.hunks.map((entry) => effectiveByHunk.get(entry.hunk.id)).filter((decision): decision is EffectiveDecision => decision !== undefined); - const includedDecisions = groupDecisions.filter(isNonSkipDecision); - const packet = await buildPacket(group.hunks, includedDecisions, group, relationshipGraph, repoIndex, opts.config, telemetry, opts.reviewContext, symbolContextMetrics, packetBuildMetrics); - packets.push(packet); + const committed = await commitPackedCandidate(candidate, build, telemetry, symbolContextMetrics, packetBuildMetrics); + packets.push(...committed); } } @@ -302,8 +336,9 @@ async function buildPacket( telemetry: TelemetryRecorder, reviewContext: PacketReviewContext | undefined, symbolContextMetrics: SymbolContextMetrics, - packetBuildMetrics: PacketBuildMetrics -): Promise { + packetBuildMetrics: PacketBuildMetrics, + packing: { members: PlannedHunk[][]; profileFloor?: ReviewProfile } | undefined = undefined +): Promise { const first = planned[0]; if (!first) { throw new Error("cannot build empty packet"); @@ -327,6 +362,7 @@ async function buildPacket( const attentionNoteSelection = mergeAttentionNotes(plannerAttentionNotes, relatedChangedContext); const attentionNotes = attentionNoteSelection.notes; emitRelationshipAttentionTelemetry(telemetry, first.file.path, attentionNoteSelection); + const memberSymbolFacts = (packing?.members ?? [planned]).map((member) => member.flatMap((entry) => entry.symbolFacts)); const context = await buildContext(repoIndex, first.file, planned.map((entry) => entry.hunk), symbolFacts, telemetry, { coverage, reviewPriority, @@ -335,7 +371,7 @@ async function buildPacket( lenses: decisions.flatMap((decision) => decision.lenses), attentionNotes, labels: first.facts.labels - }, symbolContextMetrics); + }, symbolContextMetrics, memberSymbolFacts); const hintContext = await resolvePacketContextHints(repoIndex, first.file, decisions.flatMap((decision) => decision.surroundingContextHints), telemetry); const truncationReason = truncationReasons(packetHunks).join("; "); const auxiliaryContextText = [context.text, hintContext.text].filter((text) => text.trim().length > 0).join("\n\n"); @@ -403,6 +439,10 @@ async function buildPacket( } }); } + // Plan 103: the floor is applied before routing, because routedPacketLenses() + // prunes core/code-review on a `simple` profile — raising the profile after + // routing would leave a lens dropped that the floor was meant to preserve. + const effectiveReviewProfile = maxReviewProfile([reviewProfile, ...(packing?.profileFloor !== undefined ? [packing.profileFloor] : [])]); const lenses = routedPacketLenses({ lenses: decisions.flatMap((decision) => decision.lenses), language: first.facts.language, @@ -413,7 +453,7 @@ async function buildPacket( attentionNotes, coverage, reviewPriority, - reviewProfile, + reviewProfile: effectiveReviewProfile, telemetry }); emitPacketContextQuality(telemetry, first.file.path, coverage, reviewPriority, contextQuality, contextDegradationReasons); @@ -429,7 +469,7 @@ async function buildPacket( language: first.facts.language, reviewPriority, coverage, - reviewProfile, + reviewProfile: effectiveReviewProfile, lenses, hunks: packetHunks, symbolFacts, @@ -444,7 +484,7 @@ async function buildPacket( labels: first.facts.labels, attentionNotes, relatedChangedContext, - toolBudget: scaleToolBudget(toolBudget(coverage, config.review.depth, reviewProfile), config.review.budgetBoost), + toolBudget: scaleToolBudget(toolBudget(coverage, config.review.depth, effectiveReviewProfile), config.review.budgetBoost), ...(reviewContext?.intentText !== undefined ? { intentText: reviewContext.intentText } : {}), ...(reviewContext?.intentSignals !== undefined ? { intentSignals: reviewContext.intentSignals } : {}), ...(context.degradation !== undefined || truncationReason.length > 0 || contextDropReason !== undefined || contextTruncationReason !== undefined || group.degradationReason !== undefined @@ -456,7 +496,44 @@ async function buildPacket( ? { fileContext: { mode: "file-diff", reason: "grouped file hunks" } } : {}) }; - return packet; + // Rule 7: a member counts only for the source that survives final rendering. + const survivingMemberChars = context.memberChars.map((member) => + member.header.length === 0 || renderedContext.text.includes(member.header) + ? member + : { ...member, chars: 0, complete: false } + ); + return { + packet, + routedLenses: lenses, + attentionNotes, + reviewProfile: effectiveReviewProfile, + derivedReviewProfile: reviewProfile, + contextQuality, + memberChars: survivingMemberChars, + symbolBudgetOverSubscribed: context.symbolBudgetOverSubscribed + }; +} + +// Plan 103: what a candidate build has to expose so the packing pass can +// compare it against its members' standalone builds before committing. +type PacketBuildResult = { + packet: ReviewPacket; + routedLenses: string[]; + attentionNotes: string[]; + reviewProfile: ReviewProfile; + derivedReviewProfile: ReviewProfile; + contextQuality: PacketContextQuality; + memberChars: Array<{ identity: string; header: string; chars: number; complete: boolean }>; + symbolBudgetOverSubscribed: boolean; +}; + +const REVIEW_PROFILE_RANK: Record = { simple: 0, standard: 1, investigate: 2 }; + +function maxReviewProfile(profiles: ReviewProfile[]): ReviewProfile { + return profiles.reduce( + (best, profile) => (REVIEW_PROFILE_RANK[profile] > REVIEW_PROFILE_RANK[best] ? profile : best), + "simple" + ); } const DOCS_CONFIG_EXTENSIONS = new Set([".md", ".yml", ".yaml", ".toml", ".conf", ".sample", ".txt"]); @@ -1308,9 +1385,9 @@ function packCompatibleAtoms( effectiveByHunk: Map, positionByHunk: Map, config: CodegenieConfig -): PacketGroup[] { +): PackCandidate[] { if (!config.review.packCompatibleAtoms || !groups.every((group) => group.origin === "hunk-first")) { - return groups; + return groups.map((group) => ({ group, atoms: [] })); } const partitions = new Map(); @@ -1348,7 +1425,123 @@ function packCompatibleAtoms( return packed .sort((a, b) => (a[0]?.sourcePos ?? 0) - (b[0]?.sourcePos ?? 0)) - .map(combinePacketAtoms); + .map((atoms) => ({ group: combinePacketAtoms(atoms), atoms })); +} + +type PackCandidate = { group: PacketGroup; atoms: PacketAtom[] }; + +// Plan 103: a candidate is dry-built against an isolated relationship +// accumulator and a suppressed telemetry sink. Nothing reaches the real graph, +// artifacts, or event stream until the candidate passes every check. +function scratchRelationshipGraph(graph: HunkRelationshipGraph): HunkRelationshipGraph { + return { ...graph, relatedContextAttached: [], relatedContextOmitted: [] }; +} + +const QUIET_TELEMETRY = { event: () => undefined } as unknown as TelemetryRecorder; + +const CONTEXT_QUALITY_RANK: Record = { + path_only: 0, + outline_only: 1, + sliced: 2, + full: 3 +}; + +type BuildFn = ( + group: PacketGroup, + sink: TelemetryRecorder, + metrics: SymbolContextMetrics, + buildMetrics: PacketBuildMetrics, + packing?: { members: PlannedHunk[][]; profileFloor?: ReviewProfile } +) => Promise; + +async function commitPackedCandidate( + candidate: PackCandidate, + build: BuildFn, + telemetry: TelemetryRecorder, + symbolContextMetrics: SymbolContextMetrics, + packetBuildMetrics: PacketBuildMetrics +): Promise { + const scratchMetrics = emptySymbolContextMetrics(); + const scratchBuildMetrics: PacketBuildMetrics = { + relatedContextBudgetNudges: 0, + relatedContextBudgetNudgeSources: new Set() + }; + + const standalone = []; + for (const atom of candidate.atoms) { + standalone.push(await build(atom.group, QUIET_TELEMETRY, scratchMetrics, scratchBuildMetrics, { members: [atom.hunks] })); + } + + const members = candidate.atoms.map((atom) => atom.hunks); + const profileFloor = maxReviewProfile(standalone.map((result) => result.reviewProfile)); + const packed = await build(candidate.group, QUIET_TELEMETRY, scratchMetrics, scratchBuildMetrics, { members, profileFloor }); + const rejection = packedCandidateRejection(packed, standalone, candidate); + + if (rejection === undefined) { + const committed = await build(candidate.group, telemetry, symbolContextMetrics, packetBuildMetrics, { members, profileFloor }); + return [committed.packet]; + } + + telemetry.event({ + stage: 6, + level: "info", + message: "packed_candidate_abandoned", + file: candidate.group.hunks[0]?.file.path ?? "", + data: { + reason: rejection, + atomIds: candidate.atoms.map((atom) => atom.id), + hunkIds: candidate.group.hunks.map((entry) => entry.hunk.id) + } + }); + + const separate: ReviewPacket[] = []; + for (const atom of candidate.atoms) { + const built = await build(atom.group, telemetry, symbolContextMetrics, packetBuildMetrics); + separate.push(built.packet); + } + return separate; +} + +// The abandonment contract: a packed packet may never lose a lens a member +// routed alone, never drop a high-priority planner focus note, never leave a +// symbol-bearing member without usable source, and never fall below the +// maximum standalone member profile. +function packedCandidateRejection( + packed: PacketBuildResult, + standalone: PacketBuildResult[], + candidate: PackCandidate +): string | undefined { + const routed = new Set(packed.routedLenses); + if (standalone.some((member) => member.routedLenses.some((lens) => !routed.has(lens)))) { + return "routed_lens_lost"; + } + + const priority = candidate.group.hunks[0]?.facts.reviewPriority; + if (priority === "high" || priority === "critical") { + const notes = new Set(packed.attentionNotes); + if (standalone.some((member) => member.attentionNotes.some((note) => !notes.has(note)))) { + return "high_priority_focus_note_lost"; + } + } + + if (packed.symbolBudgetOverSubscribed) { + return "member_symbol_budget_oversubscribed"; + } + // A short symbol is fully represented even below the floor; only a member + // whose source was cut below the usable minimum has actually lost context. + if (packed.memberChars.some((member) => !member.complete && member.chars < MIN_SLICED_MEMBER_CHARS)) { + return "member_symbol_context_collapsed"; + } + + const worstStandalone = Math.min(...standalone.map((member) => CONTEXT_QUALITY_RANK[member.contextQuality])); + if (CONTEXT_QUALITY_RANK[packed.contextQuality] < worstStandalone) { + return "member_context_quality_degraded"; + } + + if (REVIEW_PROFILE_RANK[packed.reviewProfile] < REVIEW_PROFILE_RANK[maxReviewProfile(standalone.map((member) => member.reviewProfile))]) { + return "effective_profile_downgraded"; + } + return undefined; } function packetAtom( @@ -1698,6 +1891,8 @@ type PacketContextBuildResult = { contextQuality: PacketContextQuality; contextDegradationReasons: string[]; packetSymbols: SymbolInfo[]; + memberChars: Array<{ identity: string; header: string; chars: number; complete: boolean }>; + symbolBudgetOverSubscribed: boolean; degradation?: string; }; @@ -1774,7 +1969,8 @@ async function buildContext( symbolFacts: HunkSymbolFacts[], telemetry: TelemetryRecorder, symbolContextInput: PacketSymbolContextInput, - symbolContextMetrics: SymbolContextMetrics + symbolContextMetrics: SymbolContextMetrics, + memberSymbolFacts: HunkSymbolFacts[][] = [symbolFacts] ): Promise { if (!isToolsHost(repoIndex.tools)) { symbolContextMetrics.outlineOnly += 1; @@ -1784,12 +1980,14 @@ async function buildContext( relevantTests: [], contextQuality: "path_only", contextDegradationReasons: ["repository tools do not provide packet context"], - packetSymbols: [] + packetSymbols: [], + memberChars: [], + symbolBudgetOverSubscribed: false }; } try { const result = await repoIndex.tools.buildPacketContext(file, hunks, symbolFacts); - const symbolSource = await readEnclosingSymbolSource(repoIndex, file, symbolFacts, telemetry, symbolContextInput, symbolContextMetrics); + const symbolSource = await readMemberSymbolSources(repoIndex, file, memberSymbolFacts, telemetry, symbolContextInput, symbolContextMetrics); const contextText = renderContext(result, symbolSource.text); const reasons = [ ...(result.degradation !== undefined ? [result.degradation] : []), @@ -1804,6 +2002,8 @@ async function buildContext( contextQuality: contextQualityFor(result, symbolSource, contextText), contextDegradationReasons: reasons, packetSymbols: result.packetSymbols ?? (result.primarySymbol !== undefined ? [result.primarySymbol] : []), + memberChars: symbolSource.memberChars, + symbolBudgetOverSubscribed: symbolSource.overSubscribed, ...(degradation.length > 0 ? { degradation } : {}) }; } catch (error) { @@ -1817,11 +2017,160 @@ async function buildContext( contextQuality: "path_only", contextDegradationReasons: [message], packetSymbols: [], + memberChars: [], + symbolBudgetOverSubscribed: false, degradation: message }; } } +type MemberSymbolSource = SymbolSourceContext & { + memberChars: Array<{ identity: string; header: string; chars: number; complete: boolean }>; + overSubscribed: boolean; +}; + +// Reads enclosing-symbol source for every distinct primary symbol in a packed +// packet. A single-symbol packet delegates to the unchanged single-symbol path, +// so flag-off behaviour and artifacts are untouched. +async function readMemberSymbolSources( + repoIndex: RepositoryIndex, + file: DiffFile, + memberSymbolFacts: HunkSymbolFacts[][], + telemetry: TelemetryRecorder, + symbolContextInput: PacketSymbolContextInput, + symbolContextMetrics: SymbolContextMetrics +): Promise { + const participants: Array<{ identity: string; fact: HunkSymbolFacts }> = []; + const seen = new Set(); + for (const facts of memberSymbolFacts) { + const fact = primarySymbolFactWithMergedChanges(facts); + if (fact === undefined || symbolSourceSelector(fact) === undefined) { + continue; + } + const identity = symbolFactIdentity(fact); + if (seen.has(identity)) { + continue; + } + seen.add(identity); + participants.push({ identity, fact }); + } + + if (participants.length <= 1) { + const single = await readEnclosingSymbolSource( + repoIndex, + file, + memberSymbolFacts.flat(), + telemetry, + symbolContextInput, + symbolContextMetrics + ); + return { + ...single, + memberChars: participants.length === 1 && participants[0] !== undefined + ? [{ identity: participants[0].identity, header: "", chars: single.text.length, complete: true }] + : [], + overSubscribed: false + }; + } + + // Rule 4: a packet that cannot give every symbol-bearing member its floor + // cannot represent its members, so the candidate is abandoned upstream. + if (participants.length * MIN_MEMBER_SYMBOL_CHARS > PACKET_SYMBOL_CONTEXT_BUDGET) { + return { text: "", reasons: ["member_symbol_budget_oversubscribed"], memberChars: [], overSubscribed: true }; + } + + const blocks: Array<{ identity: string; header: string; text: string; wanted: number }> = []; + const reasons: string[] = []; + let anySliced = false; + for (const participant of participants) { + const selector = symbolSourceSelector(participant.fact); + if (selector === undefined) { + continue; + } + const source = participant.fact.changedLinesSide === "old" ? { kind: "base" as const } : { kind: "head" as const }; + const readPath = participant.fact.changedLinesSide === "old" ? file.oldPath ?? file.path : file.path; + try { + const result = await withRepositoryToolCallContext( + repoIndex.tools, + { stage: 6, initiator: "harness" }, + () => repoIndex.tools.readSymbol(readPath, selector, source) + ); + if (result.text === undefined || result.text.trim().length === 0) { + reasons.push(`member_symbol_source_empty: ${participant.identity}`); + continue; + } + const label = result.symbol?.name ?? participant.fact.enclosingSymbol ?? `line ${String(selector.line ?? "")}`.trim(); + const rendered = renderFullSymbolContext(readPath, label, participant.fact, result.text); + blocks.push({ identity: participant.identity, header: `Primary symbol: ${readPath}:${label}`, text: rendered, wanted: rendered.length }); + } catch (error) { + const message = error instanceof Error ? error.message : String(error); + telemetry.event({ + stage: 6, + level: "warn", + message: "packet_member_symbol_source_unavailable", + file: file.path, + data: { identity: participant.identity, path: readPath, error: message } + }); + reasons.push(`member_symbol_source_unavailable: ${participant.identity}`); + } + } + + if (blocks.length === 0) { + return { text: "", reasons: reasons.length > 0 ? reasons : ["no_primary_symbol"], memberChars: [], overSubscribed: false }; + } + + // Rules 3 and 5: equal shares, then one redistribution pass in source order + // handing surplus from members that need less to members that need more. + const share = Math.floor(PACKET_SYMBOL_CONTEXT_BUDGET / blocks.length); + const allowances = blocks.map((block) => Math.min(block.wanted, share)); + let surplus = allowances.reduce((sum, allowance) => sum + (share - allowance), 0); + for (const [index, block] of blocks.entries()) { + if (surplus <= 0) { + break; + } + const current = allowances[index] ?? 0; + if (block.wanted <= current) { + continue; + } + const extra = Math.min(surplus, block.wanted - current); + allowances[index] = current + extra; + surplus -= extra; + } + + const emitted = blocks.map((block, index) => { + const allowance = Math.max(allowances[index] ?? 0, 0); + if (block.wanted <= allowance) { + return { ...block, complete: true }; + } + anySliced = true; + return { ...block, text: truncateToBudget(block.text, allowance), complete: false }; + }); + + symbolContextMetrics.materialOmission += anySliced ? 1 : 0; + telemetry.event({ + stage: 6, + level: "debug", + message: "packet_member_symbol_context", + file: file.path, + data: { + members: emitted.length, + share, + budget: PACKET_SYMBOL_CONTEXT_BUDGET, + emittedChars: emitted.map((block) => block.text.length), + sliced: anySliced + } + }); + + return { + text: emitted.map((block) => block.text).join("\n\n"), + quality: anySliced ? "sliced" : "full", + reasons, + ...(anySliced ? { degradation: "member symbol source sliced to share packet context budget" } : {}), + memberChars: emitted.map((block) => ({ identity: block.identity, header: block.header, chars: block.text.length, complete: block.complete })), + overSubscribed: false + }; +} + async function readEnclosingSymbolSource( repoIndex: RepositoryIndex, file: DiffFile, diff --git a/tests/pipeline-phase5.test.ts b/tests/pipeline-phase5.test.ts index 308fc01..87441d5 100644 --- a/tests/pipeline-phase5.test.ts +++ b/tests/pipeline-phase5.test.ts @@ -13817,3 +13817,176 @@ describe("plan 103 compatible-atom packing", () => { } }); }); + +describe("plan 103 multi-member context and transactional packing", () => { + function symbolFactsFor( + pathName: string, + hunkIds: string[], + perHunkSymbol: (id: string) => string, + sameRange = false + ): HunkSymbolFacts[] { + return hunkIds.map((hunkId, index) => ({ + path: pathName, + hunkId, + enclosingSymbol: perHunkSymbol(hunkId), + symbolKind: "function", + // Identity is name plus range: same name at a different range is a + // different symbol, which is what keeps duplicate class names apart. + symbolRange: (sameRange ? [1, 40] : [1 + index * 100, 40 + index * 100]) as [number, number], + changedLines: [1 + index * 100], + changedLinesSide: "new", + source: "tree-sitter", + confidence: "syntactic" + })); + } + + function symbolTools(bodyChars: number) { + const meta = { backend: "text" as const, precision: "exact" as const, degraded: false }; + return { + ...fakeTools(), + readSymbol: async (pathName: string, selector: { symbolName?: string; line?: number }) => ({ + text: `// ${selector.symbolName ?? "sym"}\n${"x".repeat(bodyChars)}\n`, + symbol: { + path: pathName, + name: selector.symbolName ?? "sym", + kind: "function" as const, + lineRange: [1, 40] as [number, number] + }, + meta + }) + }; + } + + function separated(pathName: string, count: number, gap = 100): DiffFile { + return { + path: pathName, + status: "modified", + language: "typescript", + hunks: Array.from({ length: count }, (_, index) => { + const line = 1 + index * gap; + return { + id: `h${index + 1}`, + hunkHash: String(index + 1).repeat(64).slice(0, 64), + path: pathName, + oldStart: line, + oldLines: 1, + newStart: line, + newLines: 1, + header: `@@ -${line} +${line} @@`, + lines: [{ kind: "add" as const, content: `const v${index + 1} = ${index + 1};`, newLineNumber: line }] + }; + }) + }; + } + + function plan(file: DiffFile): ReviewPlan { + return { + diffUnderstanding: { declaredIntent: "test intent", inferredBehavior: "test behavior" }, + coverage: file.hunks.map((hunk) => ({ + hunkId: hunk.id, + path: file.path, + coverage: "normal" as const, + lenses: ["core/code-review"], + surroundingContextHints: [], + reason: "test" + })) + }; + } + + async function run(file: DiffFile, opts: { bodyChars: number; distinctSymbols: boolean; events?: unknown[] }) { + const hunkIds = file.hunks.map((hunk) => hunk.id); + const index: RepositoryIndex = { + facts: [], + symbolFacts: symbolFactsFor(file.path, hunkIds, (id) => (opts.distinctSymbols ? `sym_${id}` : "shared"), !opts.distinctSymbols), + staticSignals: [], + tools: symbolTools(opts.bodyChars) + }; + const recorder = opts.events + ? ({ event: (entry: unknown) => opts.events?.push(entry), writeArtifact: async () => undefined } as unknown as ReturnType) + : nullTelemetry(); + const base = config(); + return buildReviewPackets(plan(file), [file], [fakeFacts(file.path, "per-hunk")], index, recorder, { + config: { ...base, review: { ...base.review, packCompatibleAtoms: true } }, + enabledLenses: ["core/code-review"] + }); + } + + it("renders every member's symbol source in a packed packet", async () => { + const file = separated("app.ts", 3); + const packets = await run(file, { bodyChars: 400, distinctSymbols: true }); + expect(packets).toHaveLength(1); + const text = packets[0]?.contextText ?? ""; + for (const hunkId of ["h1", "h2", "h3"]) { + expect(text).toContain(`sym_${hunkId}`); + } + }); + + it("keeps symbol source inside its budget and leaves room for other context", async () => { + const file = separated("app.ts", 4); + const packets = await run(file, { bodyChars: 4000, distinctSymbols: true }); + const text = packets[0]?.contextText ?? ""; + const symbolSection = text.split("Outline for")[0] ?? text; + expect(symbolSection.length).toBeLessThanOrEqual(5_000 + 200); + expect(text.length).toBeLessThanOrEqual(8_000); + // every member still present after final rendering + expect((text.match(/Primary symbol:/gu) ?? []).length).toBe(4); + }); + + it("cannot starve a member at the shipped cap, so the budget guards stay dormant", () => { + // At MAX_HUNKS_PER_PACKET = 5 a packet holds at most five distinct primary + // symbols, so the smallest possible share is 5000/5 = 1000 characters — + // above both the 800 floor and the 600 sliced minimum. The oversubscription + // and collapse rejections are therefore defensive invariants that cannot + // fire at today's cap; they become live only if the cap is ever raised past + // six. This test pins that arithmetic so a cap change surfaces here. + const budget = 5_000; + const minMemberChars = 800; + const minSlicedChars = 600; + const maxMembers = 5; + expect(maxMembers * minMemberChars).toBeLessThanOrEqual(budget); + expect(Math.floor(budget / maxMembers)).toBeGreaterThan(minSlicedChars); + expect(Math.floor(budget / 7)).toBeLessThan(minMemberChars); + }); + + it("floors a packed profile to its strongest standalone member", async () => { + // h1 and h3 share an enclosing symbol but are separated by h2, so the + // grouper yields three atoms with a same_symbol edge between h1 and h3. + // Standalone, that edge gives each of them related-changed context; packing + // absorbs the edge target, which would otherwise derive a weaker profile. + const file = separated("app.ts", 3); + const shared = (id: string) => (id === "h2" ? "other" : "shared"); + const facts: HunkSymbolFacts[] = file.hunks.map((hunk) => ({ + path: file.path, + hunkId: hunk.id, + enclosingSymbol: shared(hunk.id), + symbolKind: "function", + symbolRange: shared(hunk.id) === "shared" ? [1, 40] : [200, 240], + changedLines: [hunk.newStart], + changedLinesSide: "new", + source: "tree-sitter", + confidence: "syntactic" + })); + const base = config(); + const index: RepositoryIndex = { facts: [], symbolFacts: facts, staticSignals: [], tools: symbolTools(900) }; + const packed = await buildReviewPackets(plan(file), [file], [fakeFacts(file.path, "per-hunk")], index, nullTelemetry(), { + config: { ...base, review: { ...base.review, packCompatibleAtoms: true } }, + enabledLenses: ["core/code-review"] + }); + const unpacked = await buildReviewPackets(plan(file), [file], [fakeFacts(file.path, "per-hunk")], index, nullTelemetry(), { + config: { ...base, review: { ...base.review, packCompatibleAtoms: false } }, + enabledLenses: ["core/code-review"] + }); + + const rank = { simple: 0, standard: 1, investigate: 2 } as const; + const strongestStandalone = Math.max(...unpacked.map((packet) => rank[packet.reviewProfile])); + expect(packed).toHaveLength(1); + expect(rank[packed[0]?.reviewProfile ?? "simple"]).toBeGreaterThanOrEqual(strongestStandalone); + }); + + it("shares one symbol budget when packed members resolve to the same symbol", async () => { + const file = separated("app.ts", 3); + const packets = await run(file, { bodyChars: 400, distinctSymbols: false }); + expect(packets).toHaveLength(1); + expect((packets[0]?.contextText.match(/Primary symbol:/gu) ?? []).length).toBe(1); + }); +}); From fbb1d56fedab02cba155ca3a266f644adba79509 Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 13:18:56 -0400 Subject: [PATCH 14/24] feat(evals): add packet-packing replay; free gate passes (Plan 103 phase 3) Implements Plan 103 step 7. Step 6 (the pinned-plan seam) is deliberately deferred to phase 4: it is paid-phase infrastructure, and this replay is the gate that decides whether a paid phase happens at all. scripts/packet-packing-report.ts rebuilds Stage 6 from recorded run artifacts with the real builder - resolved input, diff parse, file filter, classification, repository index, buildReviewPackets off and on - observing zero model calls. Failure records are structured and templated: a closed-set code plus typed fields, no raw exception text and no repository source. The gate passes: dca8d870 96 -> 75 21.9% yield@56 89 -> 109 (+20) 142 hunks 81f806a6 93 -> 68 26.9% yield@56 89 -> 116 (+27) 136 hunks 740d73f2 93 -> 69 25.8% yield@56 90 -> 116 (+26) 137 hunks fe1548ae duplicate diff of 740d73f2 Zero failures, three distinct diffs. Estimator reconciles at exactly 89; flag-on yield 109 clears the 102 bar; reduction exceeds 20% on every distinct diff; deviation from Plan 102's frozen counts is 0 on both post-Plan-100 runs. Break-even ratio for phase 4 is B = 89/109 = 0.8165. Report preserved with its hash in the private evidence manifest. The reconciliation gate paid for itself on first use. The initial replay produced 114 off-packets against the recorded 96 and 161 reviewable hunks against 142. The diff was byte-correct at 88 files and 217 hunks, so the divergence was downstream: the reviewed repository carries its own codegenie.toml whose classification path rules skip generated webrpc clients and docs, schema dumps, and e2e snapshots, and raise workers, protocol, and migrations to critical. The real run loaded it - 88 files, 61 kept, 27 skipped - while the replay used bare defaults and reviewed files the run never saw. applyRepoConfigLayer fixed it, after which dca8d870 reproduces Plan 102's frozen 96 -> 75 and its 142 reviewable hunks exactly. Without the gate this would have reported a plausible-looking 20.2% on the wrong workload. Two older runs legitimately diverge: 740d73f2 and fe1548ae replay at 93 packets against a recorded 109. Both predate Plan 100's hunk-identity change and both had their planner output dropped entirely, which Plan 102's reconciliation note already flagged as needing a compatibility view. Their off/on comparison stays internally valid, so they contribute reduction evidence but not historical-count evidence. comparePackets originally tested coverage in one direction, catching promotion but not demotion - the more dangerous case, since a demoted hunk is reviewed more shallowly than planned. The partition key forces identical coverage across a packet's members, so any change is a violation; the check is now an inequality named coverage_changed. Importing the script no longer runs main, so its pure helpers are unit tested - and being reachable from tests means this script is typechecked by pnpm run check, unlike Plan 102's 5,318-line report which never was. 778 to 782 tests. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/packet-packing-report.ts | 323 +++++++++++++++++++++++++ specs/plans/phase-plans/103/phase_3.md | 64 +++++ tests/packet-packing-report.test.ts | 101 ++++++++ 3 files changed, 488 insertions(+) create mode 100644 scripts/packet-packing-report.ts create mode 100644 specs/plans/phase-plans/103/phase_3.md create mode 100644 tests/packet-packing-report.test.ts diff --git a/scripts/packet-packing-report.ts b/scripts/packet-packing-report.ts new file mode 100644 index 0000000..3b827f0 --- /dev/null +++ b/scripts/packet-packing-report.ts @@ -0,0 +1,323 @@ +#!/usr/bin/env tsx +// Plan 103 packet-packing report. Rebuilds Stage 6 from recorded run artifacts +// with packing off and on, using the real builder and zero model calls, and +// fails closed on any invariant violation. +// +// Modes: +// replay --repo --run ... [--dispatch-slots N] [--distinct-diffs] --output +// +// Failure records are structured and templated: a closed-set code plus typed +// fields, rendered from a template. Raw exception text and repository source +// are never interpolated into a message. + +import { existsSync, readFileSync, writeFileSync } from "node:fs"; +import path from "node:path"; +import { parseDiff } from "../src/git/diff-parser.js"; +import { filterDiffFiles, classifyChangedFiles } from "../src/git/file-classifier.js"; +import { createGitClient } from "../src/git/git-client.js"; +import { buildRepositoryIndex } from "../src/repo/repository-index.js"; +import { buildReviewPackets } from "../src/pipeline/packet-builder.js"; +import { defaultConfig } from "../src/config/schema.js"; +import { applyRepoConfigLayer } from "../src/config/config-loader.js"; +import type { + CodegenieConfig, + ResolvedReviewInput, + ReviewPacket, + ReviewPlan, + TelemetryEvent +} from "../src/types.js"; +import type { TelemetryRecorder } from "../src/telemetry/telemetry-recorder.js"; + +const DEFAULT_DISPATCH_SLOTS = 56; +const HISTORICAL_BASELINE_HUNKS = 89; + +type FailureCode = + | "run_artifacts_missing" + | "ref_unavailable" + | "model_call_observed" + | "hunk_not_unique" + | "atom_split" + | "cap_exceeded" + | "coverage_changed" + | "profile_downgraded" + | "budget_downgraded" + | "lens_dropped" + | "dispatch_rank_invalid" + | "estimator_unreconciled"; + +export type Failure = { code: FailureCode; run: string; message: string; fields: Record }; + +export function fail(code: FailureCode, run: string, fields: Record): Failure { + const rendered = Object.entries(fields) + .map(([key, value]) => `${key}=${typeof value === "object" ? JSON.stringify(value) : String(value)}`) + .join(" "); + return { code, run, message: `${code}: ${rendered}`, fields }; +} + +// Counts every repository tool call so a replay that silently reaches the +// network or a model can be caught; the builder must stay offline. +function countingTelemetry(counters: { events: number; modelCalls: number }): TelemetryRecorder { + return { + event: (entry: TelemetryEvent) => { + counters.events += 1; + if (typeof entry.message === "string" && entry.message.startsWith("model_call")) { + counters.modelCalls += 1; + } + }, + writeArtifact: async () => undefined + } as unknown as TelemetryRecorder; +} + +function readJson(file: string): T { + return JSON.parse(readFileSync(file, "utf8")) as T; +} + +// The reviewed repository's own codegenie.toml carries the classification path +// rules that decide which files are reviewed at all. Replaying without it keeps +// generated files the real run skipped and silently changes the workload, which +// is what the reconciliation gate exists to catch. +function packingConfig(repoRoot: string, on: boolean): CodegenieConfig { + const base = structuredClone(defaultConfig) as CodegenieConfig; + base.telemetry.enabled = false; + const config = applyRepoConfigLayer(base, repoRoot).config; + config.review.packCompatibleAtoms = on; + return config; +} + +const COVERAGE_RANK = { deep: 0, normal: 1, light: 2 } as const; +const PROFILE_RANK = { simple: 0, standard: 1, investigate: 2 } as const; +const PRIORITY_RANK = { critical: 0, high: 1, normal: 2, low: 3 } as const; + +// Mirrors worker-runner's scheduling comparator. The real scheduler is a +// prefix-with-holes at concurrency > 1, so this is a calibrated counterfactual +// capacity proxy, not a reproduction — the reconciliation gate below is what +// makes it trustworthy. +export function dispatchOrder(packets: ReviewPacket[]): ReviewPacket[] { + return [...packets].sort( + (a, b) => + PRIORITY_RANK[a.reviewPriority] - PRIORITY_RANK[b.reviewPriority] || + COVERAGE_RANK[a.coverage] - COVERAGE_RANK[b.coverage] || + (a.dispatchRank[0] ?? 0) - (b.dispatchRank[0] ?? 0) || + (a.dispatchRank[1] ?? 0) - (b.dispatchRank[1] ?? 0) || + a.id.localeCompare(b.id) + ); +} + +export function hunksWithinSlots(packets: ReviewPacket[], slots: number): number { + return new Set( + dispatchOrder(packets) + .slice(0, slots) + .flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId)) + ).size; +} + +async function rebuildStageSix( + resolved: ResolvedReviewInput, + plan: ReviewPlan, + on: boolean +): Promise<{ packets: ReviewPacket[]; modelCalls: number }> { + const counters = { events: 0, modelCalls: 0 }; + const telemetry = countingTelemetry(counters); + const config = packingConfig(resolved.repoRoot, on); + const diff = parseDiff(resolved.rawDiff); + const { kept, decisions } = await filterDiffFiles(resolved, diff, config, telemetry); + const facts = await classifyChangedFiles(resolved, kept, decisions, config, telemetry); + const repoIndex = await buildRepositoryIndex(resolved, kept, facts, config, telemetry); + const packets = await buildReviewPackets(plan, kept, facts, repoIndex, telemetry, { + config, + enabledLenses: [...new Set(plan.coverage.flatMap((entry) => entry.lenses))] + }); + return { packets, modelCalls: counters.modelCalls }; +} + +export function comparePackets(run: string, off: ReviewPacket[], on: ReviewPacket[], slots: number): Failure[] { + const failures: Failure[] = []; + const offHunks = off.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId)); + const onHunks = on.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId)); + + if (new Set(onHunks).size !== onHunks.length) { + failures.push(fail("hunk_not_unique", run, { duplicates: onHunks.length - new Set(onHunks).size })); + } + const missing = offHunks.filter((hunkId) => !onHunks.includes(hunkId)); + if (missing.length > 0) { + failures.push(fail("hunk_not_unique", run, { missingHunks: missing.length })); + } + + for (const packet of on) { + if (packet.hunks.length > defaultConfig.review.packMaxHunks) { + failures.push(fail("cap_exceeded", run, { packetId: packet.id, hunks: packet.hunks.length })); + } + } + + // Coverage, profile, and budget may never fall for any hunk relative to the + // packet that carried it with packing off. + const offByHunk = new Map(); + for (const packet of off) { + for (const hunk of packet.hunks) { + offByHunk.set(hunk.hunkId, packet); + } + } + for (const packet of on) { + for (const hunk of packet.hunks) { + const before = offByHunk.get(hunk.hunkId); + if (before === undefined) { + continue; + } + // The partition key forces identical coverage across a packet's members, + // so any change at all is a violation — promotion inflates cost and + // demotion reviews a hunk more shallowly than planned. + if (packet.coverage !== before.coverage) { + failures.push(fail("coverage_changed", run, { hunkId: hunk.hunkId, from: before.coverage, to: packet.coverage })); + } + if (PROFILE_RANK[packet.reviewProfile] < PROFILE_RANK[before.reviewProfile]) { + failures.push(fail("profile_downgraded", run, { hunkId: hunk.hunkId, from: before.reviewProfile, to: packet.reviewProfile })); + } + if (packet.toolBudget.maxToolCalls < before.toolBudget.maxToolCalls) { + failures.push(fail("budget_downgraded", run, { + hunkId: hunk.hunkId, + from: before.toolBudget.maxToolCalls, + to: packet.toolBudget.maxToolCalls + })); + } + for (const lens of before.lenses) { + if (!packet.lenses.includes(lens)) { + failures.push(fail("lens_dropped", run, { hunkId: hunk.hunkId, lens })); + } + } + } + } + void slots; + return failures; +} + +async function replay(args: Map): Promise { + const repo = args.get("repo")?.[0]; + const runs = args.get("run") ?? []; + const output = args.get("output")?.[0]; + const slots = Number(args.get("dispatch-slots")?.[0] ?? DEFAULT_DISPATCH_SLOTS); + if (repo === undefined || runs.length === 0 || output === undefined) { + console.error("usage: packet-packing-report.ts replay --repo --run ... --output "); + return 2; + } + + const failures: Failure[] = []; + const rows: Array> = []; + const git = createGitClient(repo); + const seenDiffs = new Map(); + + for (const runDir of runs) { + const name = path.basename(runDir); + const inputPath = path.join(runDir, "stages/01-input/resolved-input.json"); + const planPath = path.join(runDir, "stages/05-planner/review-plan.json"); + if (!existsSync(inputPath) || !existsSync(planPath)) { + failures.push(fail("run_artifacts_missing", name, { inputPath: existsSync(inputPath), planPath: existsSync(planPath) })); + continue; + } + const recorded = readJson<{ baseRef?: string; headSha?: string; mergeBase?: string; commits?: unknown[] }>(inputPath); + const plan = readJson(planPath); + const base = recorded.mergeBase ?? recorded.baseRef; + const head = recorded.headSha; + if (base === undefined || head === undefined) { + failures.push(fail("run_artifacts_missing", name, { base: base ?? null, head: head ?? null })); + continue; + } + try { + await git.revParse(base); + await git.revParse(head); + } catch { + failures.push(fail("ref_unavailable", name, { base, head })); + continue; + } + + const rawDiff = await git.diff(base, head); + const diffKey = `${base}..${head}`; + const duplicateOf = seenDiffs.get(diffKey); + seenDiffs.set(diffKey, duplicateOf ?? name); + + const resolved: ResolvedReviewInput = { + mode: "commit_range", + repoRoot: repo, + baseRef: base, + headRef: head, + headSha: head, + mergeBase: base, + commits: [], + rawDiff + }; + + const off = await rebuildStageSix(resolved, plan, false); + const on = await rebuildStageSix(resolved, plan, true); + if (off.modelCalls > 0 || on.modelCalls > 0) { + failures.push(fail("model_call_observed", name, { off: off.modelCalls, on: on.modelCalls })); + } + failures.push(...comparePackets(name, off.packets, on.packets, slots)); + + const offYield = hunksWithinSlots(off.packets, slots); + const onYield = hunksWithinSlots(on.packets, slots); + if (duplicateOf === undefined && offYield !== HISTORICAL_BASELINE_HUNKS && name.endsWith("dca8d870")) { + failures.push(fail("estimator_unreconciled", name, { expected: HISTORICAL_BASELINE_HUNKS, actual: offYield, slots })); + } + + rows.push({ + run: name, + duplicateOf: duplicateOf ?? null, + offPackets: off.packets.length, + onPackets: on.packets.length, + reductionPct: Number((100 * (off.packets.length - on.packets.length) / Math.max(1, off.packets.length)).toFixed(1)), + multiHunkPackets: on.packets.filter((packet) => packet.hunks.length > 1).length, + hunkDistribution: [1, 2, 3, 4, 5].map((size) => on.packets.filter((packet) => packet.hunks.length === size).length), + reviewableHunks: new Set(off.packets.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId))).size, + fixedSlotYield: { slots, off: offYield, on: onYield, gain: onYield - offYield }, + modelCallsObserved: off.modelCalls + on.modelCalls + }); + } + + const distinctRows = rows.filter((row) => row.duplicateOf === null); + const report = { + schemaVersion: 1, + mode: "replay", + dispatchSlots: slots, + noModelCalls: rows.every((row) => row.modelCallsObserved === 0), + distinctDiffs: distinctRows.length, + rows, + failures + }; + writeFileSync(output, `${JSON.stringify(report, null, 2)}\n`); + console.log(JSON.stringify({ rows, failures: failures.length, distinctDiffs: distinctRows.length }, null, 1)); + return failures.length === 0 ? 0 : 1; +} + +function parseArgs(argv: string[]): Map { + const args = new Map(); + for (let i = 0; i < argv.length; i += 1) { + const token = argv[i]; + if (token === undefined || !token.startsWith("--")) { + continue; + } + const key = token.slice(2); + const value = argv[i + 1]; + if (value === undefined || value.startsWith("--")) { + args.set(key, [...(args.get(key) ?? []), "true"]); + continue; + } + args.set(key, [...(args.get(key) ?? []), value]); + i += 1; + } + return args; +} + +async function main(): Promise { + const [mode, ...rest] = process.argv.slice(2); + const args = parseArgs(rest); + if (mode === "replay") { + process.exitCode = await replay(args); + return; + } + console.error("usage: packet-packing-report.ts replay --repo --run ... --output "); + process.exitCode = 2; +} + +// Importable for focused tests; only the CLI entry point runs main(). +if (process.argv[1]?.includes("packet-packing-report")) { + await main(); +} diff --git a/specs/plans/phase-plans/103/phase_3.md b/specs/plans/phase-plans/103/phase_3.md new file mode 100644 index 0000000..66fc147 --- /dev/null +++ b/specs/plans/phase-plans/103/phase_3.md @@ -0,0 +1,64 @@ +--- +status: complete (step 7); step 6 deferred to phase 4 +--- + +# Phase 3: The Free Replay Gate + +## Overview + +Plan 103 step 7: port the report script and run the four-run replay whose fixed-slot hunk yield decides whether any paid phase is authorized. + +**Step order was inverted deliberately.** The plan lists the pinned-plan seam (step 6) before the replay (step 7), but the seam is paid-phase infrastructure and the replay is the gate that decides whether a paid phase happens at all. Building the seam first risked constructing infrastructure for a phase that might never run. Step 6 moves to the front of Phase 4, where it is used. + +## Steps + +1. Add `scripts/packet-packing-report.ts` with a `replay` mode that rebuilds Stage 6 from recorded run artifacts using the real builder — resolved input, diff parse, file filter, classification, repository index, `buildReviewPackets()` off and on — with zero model calls. +2. Emit structured, templated failure records: a closed-set code plus typed fields, rendered from a template. No raw exception text, no repository source, no hashing. +3. Implement the fixed-slot hunk yield estimator over the Stage-7 scheduling tuple, and gate it on reconciling against the historical run before it may define the break-even ratio. +4. Add focused tests and run the four-run replay. + +## Tests + +- `renders failures from typed fields without raw text` +- `orders packets by the stage 7 scheduling tuple` +- `counts distinct hunks within a fixed dispatch slot budget` +- `passes a clean pack and fails closed on every invariant violation`: hunk loss, hunk duplication, cap breach, coverage change, profile downgrade, budget downgrade, and lens drop each produce their code. + +## Outcome + +**The gate passes.** `pnpm run check`, `pnpm test`, and `pnpm build` pass; 778 → 782 tests. + +| Run | Packets off→on | Reduction | Yield @56 slots | Reviewable hunks | +| --- | --- | ---: | --- | ---: | +| `dca8d870` | 96 → **75** | 21.9% | **89 → 109** (+20) | 142 | +| `81f806a6` | 93 → **68** | 26.9% | 89 → 116 (+27) | 136 | +| `740d73f2` | 93 → 69 | 25.8% | 90 → 116 (+26) | 137 | +| `fe1548ae` | duplicate diff of `740d73f2` | | | | + +Zero failures, `noModelCalls: true`, three distinct diffs. Preserved at `packet-dilution/reports/plan103-replay.json`, SHA-256 `20321c65fefd02167eabe01584e6bf7f0b0728b008c6a5ea0f49dc9e6eb73a63`. + +Against the pre-registered gates: estimator reconciliation exactly 89 ✓; flag-on yield 109 ≥ 102 ✓; reduction ≥20% on every distinct diff ✓; deviation from Plan 102's frozen counts is **0** on both post-Plan-100 runs ✓; zero coverage changes, profile or budget downgrades, lens drops, cap breaches, or hunk loss ✓. + +**Break-even ratio for Phase 4: `B = 89 / 109 = 0.8165`.** + +### The reconciliation gate earned its place immediately + +The first replay produced 114 off-packets against the recorded 96, and 161 reviewable hunks against 142 — and the estimator gate caught it rather than the number being quietly accepted. The diff itself was byte-correct (88 files, 217 hunks, matching the recorded totals exactly), so the divergence was downstream. + +Cause: the reviewed repository carries its own `codegenie.toml` with `classification.pathRules` that skip generated webrpc clients and docs, schema dumps, and e2e snapshots, and raise `workers/`, `lib/intentmachine/protocol/`, and migrations to critical priority. The real run loaded it — 88 files, 61 kept, 27 skipped. Replaying with bare `defaultConfig` kept files the real run never reviewed, silently changing the workload. `applyRepoConfigLayer()` fixed it, after which `dca8d870` reproduces Plan 102's frozen 96 → 75 and its 142 reviewable hunks exactly. + +Had the gate not existed, the replay would have reported a plausible-looking 20.2% reduction on a workload that was not the one being claimed. + +### Two older runs legitimately diverge + +`740d73f2` and `fe1548ae` replay at 93 packets where their artifacts recorded 109. Both predate Plan 100, which changed hunk identity, and both are the runs whose planner output was entirely dropped. Plan 102's own reconciliation note flagged that these three older artifacts need a compatibility view. Their off→on comparison remains internally valid — identical inputs on both sides — so they contribute reduction evidence but not historical-count evidence. The gate is scoped to `dca8d870`, the only post-Plan-100 run with surviving planner coverage. + +### Design note + +`comparePackets()` originally tested coverage in one direction, catching promotion but not demotion — the more dangerous case, since a demoted hunk is reviewed more shallowly than planned. Because the partition key forces identical coverage across a packet's members, any change at all is a violation, so the check is now an inequality and the code is `coverage_changed`. A test covers it. + +The estimator is documented in the script as a calibrated counterfactual capacity proxy, not a scheduler reproduction: the real dispatcher is a prefix-with-holes at concurrency above one. + +## Remaining before Phase 4 + +Step 6 — the versioned `PinnedPlanArtifact`, its validation, and `scripts/draw-pinned-plan.ts` — is now authorized and moves to the head of Phase 4, together with the owner-approved validation ceiling that Phase 4 cannot start without. diff --git a/tests/packet-packing-report.test.ts b/tests/packet-packing-report.test.ts new file mode 100644 index 0000000..a4fb39e --- /dev/null +++ b/tests/packet-packing-report.test.ts @@ -0,0 +1,101 @@ +import { describe, expect, it } from "vitest"; +import { comparePackets, dispatchOrder, fail, hunksWithinSlots } from "../scripts/packet-packing-report.js"; +import type { ReviewPacket } from "../src/types.js"; + +function packet(overrides: Partial & { id: string; hunkIds: string[] }): ReviewPacket { + const { hunkIds, id, ...rest } = overrides; + return { + id, + dispatchRank: [0, -hunkIds.length], + kind: hunkIds.length > 1 ? "coalesced-hunks" : "hunk", + prSummary: "", + path: "app.ts", + fileStatus: "modified", + isDeletedContent: false, + language: "typescript", + reviewPriority: "normal", + coverage: "normal", + reviewProfile: "standard", + lenses: ["core/code-review"], + hunks: hunkIds.map((hunkId) => ({ + hunkId, + oldStart: 1, + oldLines: 1, + newStart: 1, + newLines: 1, + header: "@@", + lines: [], + contentWithLineNumbers: "", + changedNewLineNumbers: [], + changedOldLineNumbers: [] + })), + symbolFacts: [], + context: { path: "app.ts" }, + contextText: "", + contextQuality: "full", + relevantTests: [], + surroundingContextHints: [], + labels: [], + attentionNotes: [], + relatedChangedContext: [], + toolBudget: { maxToolCalls: 4, maxInvestigationRounds: 2, maxResultChars: 10_000 }, + ...rest + } as ReviewPacket; +} + +describe("packet packing report", () => { + it("renders failures from typed fields without raw text", () => { + const failure = fail("lens_dropped", "run-1", { hunkId: "h1", lens: "core/tests" }); + expect(failure.message).toBe("lens_dropped: hunkId=h1 lens=core/tests"); + expect(failure.fields).toEqual({ hunkId: "h1", lens: "core/tests" }); + }); + + it("orders packets by the stage 7 scheduling tuple", () => { + const low = packet({ id: "low", hunkIds: ["h1"], reviewPriority: "low" }); + const deep = packet({ id: "deep", hunkIds: ["h2"], coverage: "deep" }); + const normal = packet({ id: "normal", hunkIds: ["h3"] }); + expect(dispatchOrder([low, normal, deep]).map((entry) => entry.id)).toEqual(["deep", "normal", "low"]); + }); + + it("counts distinct hunks within a fixed dispatch slot budget", () => { + const packets = [packet({ id: "a", hunkIds: ["h1", "h2"] }), packet({ id: "b", hunkIds: ["h3"] })]; + expect(hunksWithinSlots(packets, 1)).toBe(2); + expect(hunksWithinSlots(packets, 2)).toBe(3); + }); + + it("passes a clean pack and fails closed on every invariant violation", () => { + const off = [packet({ id: "a", hunkIds: ["h1"] }), packet({ id: "b", hunkIds: ["h2"] })]; + const packed = [packet({ id: "ab", hunkIds: ["h1", "h2"] })]; + expect(comparePackets("run-1", off, packed, 56)).toEqual([]); + + const lost = [packet({ id: "ab", hunkIds: ["h1"] })]; + expect(comparePackets("run-1", off, lost, 56).map((entry) => entry.code)).toContain("hunk_not_unique"); + + const duplicated = [packet({ id: "ab", hunkIds: ["h1", "h1", "h2"] })]; + expect(comparePackets("run-1", off, duplicated, 56).map((entry) => entry.code)).toContain("hunk_not_unique"); + + const overCap = [packet({ id: "ab", hunkIds: ["h1", "h2", "h3", "h4", "h5", "h6"] })]; + expect(comparePackets("run-1", off, overCap, 56).map((entry) => entry.code)).toContain("cap_exceeded"); + + const deepOff = [packet({ id: "a", hunkIds: ["h1"], coverage: "deep" }), packet({ id: "b", hunkIds: ["h2"] })]; + expect(comparePackets("run-1", deepOff, packed, 56).map((entry) => entry.code)).toContain("coverage_changed"); + + const investigateOff = [ + packet({ id: "a", hunkIds: ["h1"], reviewProfile: "investigate" }), + packet({ id: "b", hunkIds: ["h2"] }) + ]; + expect(comparePackets("run-1", investigateOff, packed, 56).map((entry) => entry.code)).toContain("profile_downgraded"); + + const richBudgetOff = [ + packet({ id: "a", hunkIds: ["h1"], toolBudget: { maxToolCalls: 9, maxInvestigationRounds: 2, maxResultChars: 10_000 } }), + packet({ id: "b", hunkIds: ["h2"] }) + ]; + expect(comparePackets("run-1", richBudgetOff, packed, 56).map((entry) => entry.code)).toContain("budget_downgraded"); + + const extraLensOff = [ + packet({ id: "a", hunkIds: ["h1"], lenses: ["core/code-review", "core/tests"] }), + packet({ id: "b", hunkIds: ["h2"] }) + ]; + expect(comparePackets("run-1", extraLensOff, packed, 56).map((entry) => entry.code)).toContain("lens_dropped"); + }); +}); From 490a07967ddc7e91cadb701bf58fadd2a70a7abc Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 13:23:03 -0400 Subject: [PATCH 15/24] feat(evals): dilution fixture and treatment proof (Plan 103 phase 4 preconditions) Records the owner-approved ceiling and completes the free preconditions that must pass before Plan 103 spends anything. approvedValidationCostUSD: $300, recorded in the plan. The owner approved the $119 reservation, raised the ceiling to $300, and authorized running the production capacity pair inside phase 4. Spend to date: $0.00. Fixture: one Go file, fifteen independent guard functions separated by ~62 lines so today's grouper yields exactly fifteen atoms. Three bugs at atom positions 1, 10 and 13 - an inclusive bound turned exclusive, a retry guard off by one, and a shard index mapped outside its range - plus a negative control at atom 7 that looks like a boundary edit and is provably equivalent. Adds a treatment mode to the report script: rebuilds Stage 6 at caps 1/3/5 with zero model calls and asserts 15/5/3 packets, each target in a packet of exactly that size, and no two targets sharing a packet. That check caught a fixture defect on first run. I had targeted the hunk at line 679, which is atom 11 rather than atom 10, placing two targets inside the same cap-5 packet and collapsing two of three recall opportunities into one model conversation - the exact clustering defect raised in review. targets_share_packet failed closed and the target was corrected to line 617. Without the proof the curve would have run at $14 with two of three observations correlated. Still required before the first paid call: the pinned-plan seam, the three eval case YAMLs, fake-provider expectation validation, and the authored plan plus three reference draws with the realism gate. Co-Authored-By: Claude Opus 5 (1M context) --- .../repos/dilution/base/go.mod | 3 + .../repos/dilution/base/limits/limits.go | 934 ++++++++++++++++++ .../repos/dilution/feature/go.mod | 3 + .../repos/dilution/feature/limits/limits.go | 934 ++++++++++++++++++ scripts/packet-packing-report.ts | 85 +- ...-103-relationship-driven-packet-packing.md | 2 + specs/plans/phase-plans/103/phase_4.md | 52 + 7 files changed, 2012 insertions(+), 1 deletion(-) create mode 100644 evals/packet-dilution/repos/dilution/base/go.mod create mode 100644 evals/packet-dilution/repos/dilution/base/limits/limits.go create mode 100644 evals/packet-dilution/repos/dilution/feature/go.mod create mode 100644 evals/packet-dilution/repos/dilution/feature/limits/limits.go create mode 100644 specs/plans/phase-plans/103/phase_4.md diff --git a/evals/packet-dilution/repos/dilution/base/go.mod b/evals/packet-dilution/repos/dilution/base/go.mod new file mode 100644 index 0000000..9582519 --- /dev/null +++ b/evals/packet-dilution/repos/dilution/base/go.mod @@ -0,0 +1,3 @@ +module dilution + +go 1.23 diff --git a/evals/packet-dilution/repos/dilution/base/limits/limits.go b/evals/packet-dilution/repos/dilution/base/limits/limits.go new file mode 100644 index 0000000..031f784 --- /dev/null +++ b/evals/packet-dilution/repos/dilution/base/limits/limits.go @@ -0,0 +1,934 @@ +package limits + +// guard 1 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 1. +// spacing line 2 for guard 1. +// spacing line 3 for guard 1. +// spacing line 4 for guard 1. +// spacing line 5 for guard 1. +// spacing line 6 for guard 1. +// spacing line 7 for guard 1. +// spacing line 8 for guard 1. +// spacing line 9 for guard 1. +// spacing line 10 for guard 1. +// spacing line 11 for guard 1. +// spacing line 12 for guard 1. +// spacing line 13 for guard 1. +// spacing line 14 for guard 1. +// spacing line 15 for guard 1. +// spacing line 16 for guard 1. +// spacing line 17 for guard 1. +// spacing line 18 for guard 1. +// spacing line 19 for guard 1. +// spacing line 20 for guard 1. +// spacing line 21 for guard 1. +// spacing line 22 for guard 1. +// spacing line 23 for guard 1. +// spacing line 24 for guard 1. +// spacing line 25 for guard 1. +// spacing line 26 for guard 1 +// spacing line 27 for guard 1 +// spacing line 28 for guard 1 +// spacing line 29 for guard 1 +// spacing line 30 for guard 1 +// spacing line 31 for guard 1 +// spacing line 32 for guard 1 +// spacing line 33 for guard 1 +// spacing line 34 for guard 1 +// spacing line 35 for guard 1 +// spacing line 36 for guard 1 +// spacing line 37 for guard 1 +// spacing line 38 for guard 1 +// spacing line 39 for guard 1 +// spacing line 40 for guard 1 +// spacing line 41 for guard 1 +// spacing line 42 for guard 1 +// spacing line 43 for guard 1 +// spacing line 44 for guard 1 +// spacing line 45 for guard 1 +// spacing line 46 for guard 1 +// spacing line 47 for guard 1 +// spacing line 48 for guard 1 +// spacing line 49 for guard 1 +// spacing line 50 for guard 1 +// spacing line 51 for guard 1 +// spacing line 52 for guard 1 +// spacing line 53 for guard 1 +// spacing line 54 for guard 1 +// spacing line 55 for guard 1 + +// WithinTransferLimit reports whether amount may be transferred. +func WithinTransferLimit(amount, limit int64) bool { + return amount <= limit +} + +// guard 2 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 2. +// spacing line 2 for guard 2. +// spacing line 3 for guard 2. +// spacing line 4 for guard 2. +// spacing line 5 for guard 2. +// spacing line 6 for guard 2. +// spacing line 7 for guard 2. +// spacing line 8 for guard 2. +// spacing line 9 for guard 2. +// spacing line 10 for guard 2. +// spacing line 11 for guard 2. +// spacing line 12 for guard 2. +// spacing line 13 for guard 2. +// spacing line 14 for guard 2. +// spacing line 15 for guard 2. +// spacing line 16 for guard 2. +// spacing line 17 for guard 2. +// spacing line 18 for guard 2. +// spacing line 19 for guard 2. +// spacing line 20 for guard 2. +// spacing line 21 for guard 2. +// spacing line 22 for guard 2. +// spacing line 23 for guard 2. +// spacing line 24 for guard 2. +// spacing line 25 for guard 2. +// spacing line 26 for guard 2 +// spacing line 27 for guard 2 +// spacing line 28 for guard 2 +// spacing line 29 for guard 2 +// spacing line 30 for guard 2 +// spacing line 31 for guard 2 +// spacing line 32 for guard 2 +// spacing line 33 for guard 2 +// spacing line 34 for guard 2 +// spacing line 35 for guard 2 +// spacing line 36 for guard 2 +// spacing line 37 for guard 2 +// spacing line 38 for guard 2 +// spacing line 39 for guard 2 +// spacing line 40 for guard 2 +// spacing line 41 for guard 2 +// spacing line 42 for guard 2 +// spacing line 43 for guard 2 +// spacing line 44 for guard 2 +// spacing line 45 for guard 2 +// spacing line 46 for guard 2 +// spacing line 47 for guard 2 +// spacing line 48 for guard 2 +// spacing line 49 for guard 2 +// spacing line 50 for guard 2 +// spacing line 51 for guard 2 +// spacing line 52 for guard 2 +// spacing line 53 for guard 2 +// spacing line 54 for guard 2 +// spacing line 55 for guard 2 + +// Guard2 validates request field 2. +func Guard2(value int) bool { + return value >= 2 +} + +// guard 3 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 3. +// spacing line 2 for guard 3. +// spacing line 3 for guard 3. +// spacing line 4 for guard 3. +// spacing line 5 for guard 3. +// spacing line 6 for guard 3. +// spacing line 7 for guard 3. +// spacing line 8 for guard 3. +// spacing line 9 for guard 3. +// spacing line 10 for guard 3. +// spacing line 11 for guard 3. +// spacing line 12 for guard 3. +// spacing line 13 for guard 3. +// spacing line 14 for guard 3. +// spacing line 15 for guard 3. +// spacing line 16 for guard 3. +// spacing line 17 for guard 3. +// spacing line 18 for guard 3. +// spacing line 19 for guard 3. +// spacing line 20 for guard 3. +// spacing line 21 for guard 3. +// spacing line 22 for guard 3. +// spacing line 23 for guard 3. +// spacing line 24 for guard 3. +// spacing line 25 for guard 3. +// spacing line 26 for guard 3 +// spacing line 27 for guard 3 +// spacing line 28 for guard 3 +// spacing line 29 for guard 3 +// spacing line 30 for guard 3 +// spacing line 31 for guard 3 +// spacing line 32 for guard 3 +// spacing line 33 for guard 3 +// spacing line 34 for guard 3 +// spacing line 35 for guard 3 +// spacing line 36 for guard 3 +// spacing line 37 for guard 3 +// spacing line 38 for guard 3 +// spacing line 39 for guard 3 +// spacing line 40 for guard 3 +// spacing line 41 for guard 3 +// spacing line 42 for guard 3 +// spacing line 43 for guard 3 +// spacing line 44 for guard 3 +// spacing line 45 for guard 3 +// spacing line 46 for guard 3 +// spacing line 47 for guard 3 +// spacing line 48 for guard 3 +// spacing line 49 for guard 3 +// spacing line 50 for guard 3 +// spacing line 51 for guard 3 +// spacing line 52 for guard 3 +// spacing line 53 for guard 3 +// spacing line 54 for guard 3 +// spacing line 55 for guard 3 + +// Guard3 validates request field 3. +func Guard3(value int) bool { + return value >= 3 +} + +// guard 4 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 4. +// spacing line 2 for guard 4. +// spacing line 3 for guard 4. +// spacing line 4 for guard 4. +// spacing line 5 for guard 4. +// spacing line 6 for guard 4. +// spacing line 7 for guard 4. +// spacing line 8 for guard 4. +// spacing line 9 for guard 4. +// spacing line 10 for guard 4. +// spacing line 11 for guard 4. +// spacing line 12 for guard 4. +// spacing line 13 for guard 4. +// spacing line 14 for guard 4. +// spacing line 15 for guard 4. +// spacing line 16 for guard 4. +// spacing line 17 for guard 4. +// spacing line 18 for guard 4. +// spacing line 19 for guard 4. +// spacing line 20 for guard 4. +// spacing line 21 for guard 4. +// spacing line 22 for guard 4. +// spacing line 23 for guard 4. +// spacing line 24 for guard 4. +// spacing line 25 for guard 4. +// spacing line 26 for guard 4 +// spacing line 27 for guard 4 +// spacing line 28 for guard 4 +// spacing line 29 for guard 4 +// spacing line 30 for guard 4 +// spacing line 31 for guard 4 +// spacing line 32 for guard 4 +// spacing line 33 for guard 4 +// spacing line 34 for guard 4 +// spacing line 35 for guard 4 +// spacing line 36 for guard 4 +// spacing line 37 for guard 4 +// spacing line 38 for guard 4 +// spacing line 39 for guard 4 +// spacing line 40 for guard 4 +// spacing line 41 for guard 4 +// spacing line 42 for guard 4 +// spacing line 43 for guard 4 +// spacing line 44 for guard 4 +// spacing line 45 for guard 4 +// spacing line 46 for guard 4 +// spacing line 47 for guard 4 +// spacing line 48 for guard 4 +// spacing line 49 for guard 4 +// spacing line 50 for guard 4 +// spacing line 51 for guard 4 +// spacing line 52 for guard 4 +// spacing line 53 for guard 4 +// spacing line 54 for guard 4 +// spacing line 55 for guard 4 + +// Guard4 validates request field 4. +func Guard4(value int) bool { + return value >= 4 +} + +// guard 5 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 5. +// spacing line 2 for guard 5. +// spacing line 3 for guard 5. +// spacing line 4 for guard 5. +// spacing line 5 for guard 5. +// spacing line 6 for guard 5. +// spacing line 7 for guard 5. +// spacing line 8 for guard 5. +// spacing line 9 for guard 5. +// spacing line 10 for guard 5. +// spacing line 11 for guard 5. +// spacing line 12 for guard 5. +// spacing line 13 for guard 5. +// spacing line 14 for guard 5. +// spacing line 15 for guard 5. +// spacing line 16 for guard 5. +// spacing line 17 for guard 5. +// spacing line 18 for guard 5. +// spacing line 19 for guard 5. +// spacing line 20 for guard 5. +// spacing line 21 for guard 5. +// spacing line 22 for guard 5. +// spacing line 23 for guard 5. +// spacing line 24 for guard 5. +// spacing line 25 for guard 5. +// spacing line 26 for guard 5 +// spacing line 27 for guard 5 +// spacing line 28 for guard 5 +// spacing line 29 for guard 5 +// spacing line 30 for guard 5 +// spacing line 31 for guard 5 +// spacing line 32 for guard 5 +// spacing line 33 for guard 5 +// spacing line 34 for guard 5 +// spacing line 35 for guard 5 +// spacing line 36 for guard 5 +// spacing line 37 for guard 5 +// spacing line 38 for guard 5 +// spacing line 39 for guard 5 +// spacing line 40 for guard 5 +// spacing line 41 for guard 5 +// spacing line 42 for guard 5 +// spacing line 43 for guard 5 +// spacing line 44 for guard 5 +// spacing line 45 for guard 5 +// spacing line 46 for guard 5 +// spacing line 47 for guard 5 +// spacing line 48 for guard 5 +// spacing line 49 for guard 5 +// spacing line 50 for guard 5 +// spacing line 51 for guard 5 +// spacing line 52 for guard 5 +// spacing line 53 for guard 5 +// spacing line 54 for guard 5 +// spacing line 55 for guard 5 + +// Guard5 validates request field 5. +func Guard5(value int) bool { + return value >= 5 +} + +// guard 6 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 6. +// spacing line 2 for guard 6. +// spacing line 3 for guard 6. +// spacing line 4 for guard 6. +// spacing line 5 for guard 6. +// spacing line 6 for guard 6. +// spacing line 7 for guard 6. +// spacing line 8 for guard 6. +// spacing line 9 for guard 6. +// spacing line 10 for guard 6. +// spacing line 11 for guard 6. +// spacing line 12 for guard 6. +// spacing line 13 for guard 6. +// spacing line 14 for guard 6. +// spacing line 15 for guard 6. +// spacing line 16 for guard 6. +// spacing line 17 for guard 6. +// spacing line 18 for guard 6. +// spacing line 19 for guard 6. +// spacing line 20 for guard 6. +// spacing line 21 for guard 6. +// spacing line 22 for guard 6. +// spacing line 23 for guard 6. +// spacing line 24 for guard 6. +// spacing line 25 for guard 6. +// spacing line 26 for guard 6 +// spacing line 27 for guard 6 +// spacing line 28 for guard 6 +// spacing line 29 for guard 6 +// spacing line 30 for guard 6 +// spacing line 31 for guard 6 +// spacing line 32 for guard 6 +// spacing line 33 for guard 6 +// spacing line 34 for guard 6 +// spacing line 35 for guard 6 +// spacing line 36 for guard 6 +// spacing line 37 for guard 6 +// spacing line 38 for guard 6 +// spacing line 39 for guard 6 +// spacing line 40 for guard 6 +// spacing line 41 for guard 6 +// spacing line 42 for guard 6 +// spacing line 43 for guard 6 +// spacing line 44 for guard 6 +// spacing line 45 for guard 6 +// spacing line 46 for guard 6 +// spacing line 47 for guard 6 +// spacing line 48 for guard 6 +// spacing line 49 for guard 6 +// spacing line 50 for guard 6 +// spacing line 51 for guard 6 +// spacing line 52 for guard 6 +// spacing line 53 for guard 6 +// spacing line 54 for guard 6 +// spacing line 55 for guard 6 + +// Guard6 validates request field 6. +func Guard6(value int) bool { + return value >= 6 +} + +// guard 7 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 7. +// spacing line 2 for guard 7. +// spacing line 3 for guard 7. +// spacing line 4 for guard 7. +// spacing line 5 for guard 7. +// spacing line 6 for guard 7. +// spacing line 7 for guard 7. +// spacing line 8 for guard 7. +// spacing line 9 for guard 7. +// spacing line 10 for guard 7. +// spacing line 11 for guard 7. +// spacing line 12 for guard 7. +// spacing line 13 for guard 7. +// spacing line 14 for guard 7. +// spacing line 15 for guard 7. +// spacing line 16 for guard 7. +// spacing line 17 for guard 7. +// spacing line 18 for guard 7. +// spacing line 19 for guard 7. +// spacing line 20 for guard 7. +// spacing line 21 for guard 7. +// spacing line 22 for guard 7. +// spacing line 23 for guard 7. +// spacing line 24 for guard 7. +// spacing line 25 for guard 7. +// spacing line 26 for guard 7 +// spacing line 27 for guard 7 +// spacing line 28 for guard 7 +// spacing line 29 for guard 7 +// spacing line 30 for guard 7 +// spacing line 31 for guard 7 +// spacing line 32 for guard 7 +// spacing line 33 for guard 7 +// spacing line 34 for guard 7 +// spacing line 35 for guard 7 +// spacing line 36 for guard 7 +// spacing line 37 for guard 7 +// spacing line 38 for guard 7 +// spacing line 39 for guard 7 +// spacing line 40 for guard 7 +// spacing line 41 for guard 7 +// spacing line 42 for guard 7 +// spacing line 43 for guard 7 +// spacing line 44 for guard 7 +// spacing line 45 for guard 7 +// spacing line 46 for guard 7 +// spacing line 47 for guard 7 +// spacing line 48 for guard 7 +// spacing line 49 for guard 7 +// spacing line 50 for guard 7 +// spacing line 51 for guard 7 +// spacing line 52 for guard 7 +// spacing line 53 for guard 7 +// spacing line 54 for guard 7 +// spacing line 55 for guard 7 + +// AtCapacity reports whether the queue is full. +func AtCapacity(used, capacity int) bool { + return used >= capacity +} + +// guard 8 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 8. +// spacing line 2 for guard 8. +// spacing line 3 for guard 8. +// spacing line 4 for guard 8. +// spacing line 5 for guard 8. +// spacing line 6 for guard 8. +// spacing line 7 for guard 8. +// spacing line 8 for guard 8. +// spacing line 9 for guard 8. +// spacing line 10 for guard 8. +// spacing line 11 for guard 8. +// spacing line 12 for guard 8. +// spacing line 13 for guard 8. +// spacing line 14 for guard 8. +// spacing line 15 for guard 8. +// spacing line 16 for guard 8. +// spacing line 17 for guard 8. +// spacing line 18 for guard 8. +// spacing line 19 for guard 8. +// spacing line 20 for guard 8. +// spacing line 21 for guard 8. +// spacing line 22 for guard 8. +// spacing line 23 for guard 8. +// spacing line 24 for guard 8. +// spacing line 25 for guard 8. +// spacing line 26 for guard 8 +// spacing line 27 for guard 8 +// spacing line 28 for guard 8 +// spacing line 29 for guard 8 +// spacing line 30 for guard 8 +// spacing line 31 for guard 8 +// spacing line 32 for guard 8 +// spacing line 33 for guard 8 +// spacing line 34 for guard 8 +// spacing line 35 for guard 8 +// spacing line 36 for guard 8 +// spacing line 37 for guard 8 +// spacing line 38 for guard 8 +// spacing line 39 for guard 8 +// spacing line 40 for guard 8 +// spacing line 41 for guard 8 +// spacing line 42 for guard 8 +// spacing line 43 for guard 8 +// spacing line 44 for guard 8 +// spacing line 45 for guard 8 +// spacing line 46 for guard 8 +// spacing line 47 for guard 8 +// spacing line 48 for guard 8 +// spacing line 49 for guard 8 +// spacing line 50 for guard 8 +// spacing line 51 for guard 8 +// spacing line 52 for guard 8 +// spacing line 53 for guard 8 +// spacing line 54 for guard 8 +// spacing line 55 for guard 8 + +// Guard8 validates request field 8. +func Guard8(value int) bool { + return value >= 8 +} + +// guard 9 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 9. +// spacing line 2 for guard 9. +// spacing line 3 for guard 9. +// spacing line 4 for guard 9. +// spacing line 5 for guard 9. +// spacing line 6 for guard 9. +// spacing line 7 for guard 9. +// spacing line 8 for guard 9. +// spacing line 9 for guard 9. +// spacing line 10 for guard 9. +// spacing line 11 for guard 9. +// spacing line 12 for guard 9. +// spacing line 13 for guard 9. +// spacing line 14 for guard 9. +// spacing line 15 for guard 9. +// spacing line 16 for guard 9. +// spacing line 17 for guard 9. +// spacing line 18 for guard 9. +// spacing line 19 for guard 9. +// spacing line 20 for guard 9. +// spacing line 21 for guard 9. +// spacing line 22 for guard 9. +// spacing line 23 for guard 9. +// spacing line 24 for guard 9. +// spacing line 25 for guard 9. +// spacing line 26 for guard 9 +// spacing line 27 for guard 9 +// spacing line 28 for guard 9 +// spacing line 29 for guard 9 +// spacing line 30 for guard 9 +// spacing line 31 for guard 9 +// spacing line 32 for guard 9 +// spacing line 33 for guard 9 +// spacing line 34 for guard 9 +// spacing line 35 for guard 9 +// spacing line 36 for guard 9 +// spacing line 37 for guard 9 +// spacing line 38 for guard 9 +// spacing line 39 for guard 9 +// spacing line 40 for guard 9 +// spacing line 41 for guard 9 +// spacing line 42 for guard 9 +// spacing line 43 for guard 9 +// spacing line 44 for guard 9 +// spacing line 45 for guard 9 +// spacing line 46 for guard 9 +// spacing line 47 for guard 9 +// spacing line 48 for guard 9 +// spacing line 49 for guard 9 +// spacing line 50 for guard 9 +// spacing line 51 for guard 9 +// spacing line 52 for guard 9 +// spacing line 53 for guard 9 +// spacing line 54 for guard 9 +// spacing line 55 for guard 9 + +// Guard9 validates request field 9. +func Guard9(value int) bool { + return value >= 9 +} + +// guard 10 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 10. +// spacing line 2 for guard 10. +// spacing line 3 for guard 10. +// spacing line 4 for guard 10. +// spacing line 5 for guard 10. +// spacing line 6 for guard 10. +// spacing line 7 for guard 10. +// spacing line 8 for guard 10. +// spacing line 9 for guard 10. +// spacing line 10 for guard 10. +// spacing line 11 for guard 10. +// spacing line 12 for guard 10. +// spacing line 13 for guard 10. +// spacing line 14 for guard 10. +// spacing line 15 for guard 10. +// spacing line 16 for guard 10. +// spacing line 17 for guard 10. +// spacing line 18 for guard 10. +// spacing line 19 for guard 10. +// spacing line 20 for guard 10. +// spacing line 21 for guard 10. +// spacing line 22 for guard 10. +// spacing line 23 for guard 10. +// spacing line 24 for guard 10. +// spacing line 25 for guard 10. +// spacing line 26 for guard 10 +// spacing line 27 for guard 10 +// spacing line 28 for guard 10 +// spacing line 29 for guard 10 +// spacing line 30 for guard 10 +// spacing line 31 for guard 10 +// spacing line 32 for guard 10 +// spacing line 33 for guard 10 +// spacing line 34 for guard 10 +// spacing line 35 for guard 10 +// spacing line 36 for guard 10 +// spacing line 37 for guard 10 +// spacing line 38 for guard 10 +// spacing line 39 for guard 10 +// spacing line 40 for guard 10 +// spacing line 41 for guard 10 +// spacing line 42 for guard 10 +// spacing line 43 for guard 10 +// spacing line 44 for guard 10 +// spacing line 45 for guard 10 +// spacing line 46 for guard 10 +// spacing line 47 for guard 10 +// spacing line 48 for guard 10 +// spacing line 49 for guard 10 +// spacing line 50 for guard 10 +// spacing line 51 for guard 10 +// spacing line 52 for guard 10 +// spacing line 53 for guard 10 +// spacing line 54 for guard 10 +// spacing line 55 for guard 10 + +// ShouldRetry reports whether another attempt is allowed. +func ShouldRetry(attempt, maxAttempts int) bool { + return attempt < maxAttempts +} + +// guard 11 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 11. +// spacing line 2 for guard 11. +// spacing line 3 for guard 11. +// spacing line 4 for guard 11. +// spacing line 5 for guard 11. +// spacing line 6 for guard 11. +// spacing line 7 for guard 11. +// spacing line 8 for guard 11. +// spacing line 9 for guard 11. +// spacing line 10 for guard 11. +// spacing line 11 for guard 11. +// spacing line 12 for guard 11. +// spacing line 13 for guard 11. +// spacing line 14 for guard 11. +// spacing line 15 for guard 11. +// spacing line 16 for guard 11. +// spacing line 17 for guard 11. +// spacing line 18 for guard 11. +// spacing line 19 for guard 11. +// spacing line 20 for guard 11. +// spacing line 21 for guard 11. +// spacing line 22 for guard 11. +// spacing line 23 for guard 11. +// spacing line 24 for guard 11. +// spacing line 25 for guard 11. +// spacing line 26 for guard 11 +// spacing line 27 for guard 11 +// spacing line 28 for guard 11 +// spacing line 29 for guard 11 +// spacing line 30 for guard 11 +// spacing line 31 for guard 11 +// spacing line 32 for guard 11 +// spacing line 33 for guard 11 +// spacing line 34 for guard 11 +// spacing line 35 for guard 11 +// spacing line 36 for guard 11 +// spacing line 37 for guard 11 +// spacing line 38 for guard 11 +// spacing line 39 for guard 11 +// spacing line 40 for guard 11 +// spacing line 41 for guard 11 +// spacing line 42 for guard 11 +// spacing line 43 for guard 11 +// spacing line 44 for guard 11 +// spacing line 45 for guard 11 +// spacing line 46 for guard 11 +// spacing line 47 for guard 11 +// spacing line 48 for guard 11 +// spacing line 49 for guard 11 +// spacing line 50 for guard 11 +// spacing line 51 for guard 11 +// spacing line 52 for guard 11 +// spacing line 53 for guard 11 +// spacing line 54 for guard 11 +// spacing line 55 for guard 11 + +// Guard11 validates request field 11. +func Guard11(value int) bool { + return value >= 11 +} + +// guard 12 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 12. +// spacing line 2 for guard 12. +// spacing line 3 for guard 12. +// spacing line 4 for guard 12. +// spacing line 5 for guard 12. +// spacing line 6 for guard 12. +// spacing line 7 for guard 12. +// spacing line 8 for guard 12. +// spacing line 9 for guard 12. +// spacing line 10 for guard 12. +// spacing line 11 for guard 12. +// spacing line 12 for guard 12. +// spacing line 13 for guard 12. +// spacing line 14 for guard 12. +// spacing line 15 for guard 12. +// spacing line 16 for guard 12. +// spacing line 17 for guard 12. +// spacing line 18 for guard 12. +// spacing line 19 for guard 12. +// spacing line 20 for guard 12. +// spacing line 21 for guard 12. +// spacing line 22 for guard 12. +// spacing line 23 for guard 12. +// spacing line 24 for guard 12. +// spacing line 25 for guard 12. +// spacing line 26 for guard 12 +// spacing line 27 for guard 12 +// spacing line 28 for guard 12 +// spacing line 29 for guard 12 +// spacing line 30 for guard 12 +// spacing line 31 for guard 12 +// spacing line 32 for guard 12 +// spacing line 33 for guard 12 +// spacing line 34 for guard 12 +// spacing line 35 for guard 12 +// spacing line 36 for guard 12 +// spacing line 37 for guard 12 +// spacing line 38 for guard 12 +// spacing line 39 for guard 12 +// spacing line 40 for guard 12 +// spacing line 41 for guard 12 +// spacing line 42 for guard 12 +// spacing line 43 for guard 12 +// spacing line 44 for guard 12 +// spacing line 45 for guard 12 +// spacing line 46 for guard 12 +// spacing line 47 for guard 12 +// spacing line 48 for guard 12 +// spacing line 49 for guard 12 +// spacing line 50 for guard 12 +// spacing line 51 for guard 12 +// spacing line 52 for guard 12 +// spacing line 53 for guard 12 +// spacing line 54 for guard 12 +// spacing line 55 for guard 12 + +// Guard12 validates request field 12. +func Guard12(value int) bool { + return value >= 12 +} + +// guard 13 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 13. +// spacing line 2 for guard 13. +// spacing line 3 for guard 13. +// spacing line 4 for guard 13. +// spacing line 5 for guard 13. +// spacing line 6 for guard 13. +// spacing line 7 for guard 13. +// spacing line 8 for guard 13. +// spacing line 9 for guard 13. +// spacing line 10 for guard 13. +// spacing line 11 for guard 13. +// spacing line 12 for guard 13. +// spacing line 13 for guard 13. +// spacing line 14 for guard 13. +// spacing line 15 for guard 13. +// spacing line 16 for guard 13. +// spacing line 17 for guard 13. +// spacing line 18 for guard 13. +// spacing line 19 for guard 13. +// spacing line 20 for guard 13. +// spacing line 21 for guard 13. +// spacing line 22 for guard 13. +// spacing line 23 for guard 13. +// spacing line 24 for guard 13. +// spacing line 25 for guard 13. +// spacing line 26 for guard 13 +// spacing line 27 for guard 13 +// spacing line 28 for guard 13 +// spacing line 29 for guard 13 +// spacing line 30 for guard 13 +// spacing line 31 for guard 13 +// spacing line 32 for guard 13 +// spacing line 33 for guard 13 +// spacing line 34 for guard 13 +// spacing line 35 for guard 13 +// spacing line 36 for guard 13 +// spacing line 37 for guard 13 +// spacing line 38 for guard 13 +// spacing line 39 for guard 13 +// spacing line 40 for guard 13 +// spacing line 41 for guard 13 +// spacing line 42 for guard 13 +// spacing line 43 for guard 13 +// spacing line 44 for guard 13 +// spacing line 45 for guard 13 +// spacing line 46 for guard 13 +// spacing line 47 for guard 13 +// spacing line 48 for guard 13 +// spacing line 49 for guard 13 +// spacing line 50 for guard 13 +// spacing line 51 for guard 13 +// spacing line 52 for guard 13 +// spacing line 53 for guard 13 +// spacing line 54 for guard 13 +// spacing line 55 for guard 13 + +// ShardIndex maps a key onto the configured shard range. +func ShardIndex(key, shards int) int { + if shards <= 0 { + return 0 + } + return key % shards +} + +// guard 14 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 14. +// spacing line 2 for guard 14. +// spacing line 3 for guard 14. +// spacing line 4 for guard 14. +// spacing line 5 for guard 14. +// spacing line 6 for guard 14. +// spacing line 7 for guard 14. +// spacing line 8 for guard 14. +// spacing line 9 for guard 14. +// spacing line 10 for guard 14. +// spacing line 11 for guard 14. +// spacing line 12 for guard 14. +// spacing line 13 for guard 14. +// spacing line 14 for guard 14. +// spacing line 15 for guard 14. +// spacing line 16 for guard 14. +// spacing line 17 for guard 14. +// spacing line 18 for guard 14. +// spacing line 19 for guard 14. +// spacing line 20 for guard 14. +// spacing line 21 for guard 14. +// spacing line 22 for guard 14. +// spacing line 23 for guard 14. +// spacing line 24 for guard 14. +// spacing line 25 for guard 14. +// spacing line 26 for guard 14 +// spacing line 27 for guard 14 +// spacing line 28 for guard 14 +// spacing line 29 for guard 14 +// spacing line 30 for guard 14 +// spacing line 31 for guard 14 +// spacing line 32 for guard 14 +// spacing line 33 for guard 14 +// spacing line 34 for guard 14 +// spacing line 35 for guard 14 +// spacing line 36 for guard 14 +// spacing line 37 for guard 14 +// spacing line 38 for guard 14 +// spacing line 39 for guard 14 +// spacing line 40 for guard 14 +// spacing line 41 for guard 14 +// spacing line 42 for guard 14 +// spacing line 43 for guard 14 +// spacing line 44 for guard 14 +// spacing line 45 for guard 14 +// spacing line 46 for guard 14 +// spacing line 47 for guard 14 +// spacing line 48 for guard 14 +// spacing line 49 for guard 14 +// spacing line 50 for guard 14 +// spacing line 51 for guard 14 +// spacing line 52 for guard 14 +// spacing line 53 for guard 14 +// spacing line 54 for guard 14 +// spacing line 55 for guard 14 + +// Guard14 validates request field 14. +func Guard14(value int) bool { + return value >= 14 +} + +// guard 15 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 15. +// spacing line 2 for guard 15. +// spacing line 3 for guard 15. +// spacing line 4 for guard 15. +// spacing line 5 for guard 15. +// spacing line 6 for guard 15. +// spacing line 7 for guard 15. +// spacing line 8 for guard 15. +// spacing line 9 for guard 15. +// spacing line 10 for guard 15. +// spacing line 11 for guard 15. +// spacing line 12 for guard 15. +// spacing line 13 for guard 15. +// spacing line 14 for guard 15. +// spacing line 15 for guard 15. +// spacing line 16 for guard 15. +// spacing line 17 for guard 15. +// spacing line 18 for guard 15. +// spacing line 19 for guard 15. +// spacing line 20 for guard 15. +// spacing line 21 for guard 15. +// spacing line 22 for guard 15. +// spacing line 23 for guard 15. +// spacing line 24 for guard 15. +// spacing line 25 for guard 15. +// spacing line 26 for guard 15 +// spacing line 27 for guard 15 +// spacing line 28 for guard 15 +// spacing line 29 for guard 15 +// spacing line 30 for guard 15 +// spacing line 31 for guard 15 +// spacing line 32 for guard 15 +// spacing line 33 for guard 15 +// spacing line 34 for guard 15 +// spacing line 35 for guard 15 +// spacing line 36 for guard 15 +// spacing line 37 for guard 15 +// spacing line 38 for guard 15 +// spacing line 39 for guard 15 +// spacing line 40 for guard 15 +// spacing line 41 for guard 15 +// spacing line 42 for guard 15 +// spacing line 43 for guard 15 +// spacing line 44 for guard 15 +// spacing line 45 for guard 15 +// spacing line 46 for guard 15 +// spacing line 47 for guard 15 +// spacing line 48 for guard 15 +// spacing line 49 for guard 15 +// spacing line 50 for guard 15 +// spacing line 51 for guard 15 +// spacing line 52 for guard 15 +// spacing line 53 for guard 15 +// spacing line 54 for guard 15 +// spacing line 55 for guard 15 + +// Guard15 validates request field 15. +func Guard15(value int) bool { + return value >= 15 +} diff --git a/evals/packet-dilution/repos/dilution/feature/go.mod b/evals/packet-dilution/repos/dilution/feature/go.mod new file mode 100644 index 0000000..9582519 --- /dev/null +++ b/evals/packet-dilution/repos/dilution/feature/go.mod @@ -0,0 +1,3 @@ +module dilution + +go 1.23 diff --git a/evals/packet-dilution/repos/dilution/feature/limits/limits.go b/evals/packet-dilution/repos/dilution/feature/limits/limits.go new file mode 100644 index 0000000..452c316 --- /dev/null +++ b/evals/packet-dilution/repos/dilution/feature/limits/limits.go @@ -0,0 +1,934 @@ +package limits + +// guard 1 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 1. +// spacing line 2 for guard 1. +// spacing line 3 for guard 1. +// spacing line 4 for guard 1. +// spacing line 5 for guard 1. +// spacing line 6 for guard 1. +// spacing line 7 for guard 1. +// spacing line 8 for guard 1. +// spacing line 9 for guard 1. +// spacing line 10 for guard 1. +// spacing line 11 for guard 1. +// spacing line 12 for guard 1. +// spacing line 13 for guard 1. +// spacing line 14 for guard 1. +// spacing line 15 for guard 1. +// spacing line 16 for guard 1. +// spacing line 17 for guard 1. +// spacing line 18 for guard 1. +// spacing line 19 for guard 1. +// spacing line 20 for guard 1. +// spacing line 21 for guard 1. +// spacing line 22 for guard 1. +// spacing line 23 for guard 1. +// spacing line 24 for guard 1. +// spacing line 25 for guard 1. +// spacing line 26 for guard 1 +// spacing line 27 for guard 1 +// spacing line 28 for guard 1 +// spacing line 29 for guard 1 +// spacing line 30 for guard 1 +// spacing line 31 for guard 1 +// spacing line 32 for guard 1 +// spacing line 33 for guard 1 +// spacing line 34 for guard 1 +// spacing line 35 for guard 1 +// spacing line 36 for guard 1 +// spacing line 37 for guard 1 +// spacing line 38 for guard 1 +// spacing line 39 for guard 1 +// spacing line 40 for guard 1 +// spacing line 41 for guard 1 +// spacing line 42 for guard 1 +// spacing line 43 for guard 1 +// spacing line 44 for guard 1 +// spacing line 45 for guard 1 +// spacing line 46 for guard 1 +// spacing line 47 for guard 1 +// spacing line 48 for guard 1 +// spacing line 49 for guard 1 +// spacing line 50 for guard 1 +// spacing line 51 for guard 1 +// spacing line 52 for guard 1 +// spacing line 53 for guard 1 +// spacing line 54 for guard 1 +// spacing line 55 for guard 1 + +// WithinTransferLimit reports whether amount may be transferred. +func WithinTransferLimit(amount, limit int64) bool { + return amount < limit +} + +// guard 2 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 2. +// spacing line 2 for guard 2. +// spacing line 3 for guard 2. +// spacing line 4 for guard 2. +// spacing line 5 for guard 2. +// spacing line 6 for guard 2. +// spacing line 7 for guard 2. +// spacing line 8 for guard 2. +// spacing line 9 for guard 2. +// spacing line 10 for guard 2. +// spacing line 11 for guard 2. +// spacing line 12 for guard 2. +// spacing line 13 for guard 2. +// spacing line 14 for guard 2. +// spacing line 15 for guard 2. +// spacing line 16 for guard 2. +// spacing line 17 for guard 2. +// spacing line 18 for guard 2. +// spacing line 19 for guard 2. +// spacing line 20 for guard 2. +// spacing line 21 for guard 2. +// spacing line 22 for guard 2. +// spacing line 23 for guard 2. +// spacing line 24 for guard 2. +// spacing line 25 for guard 2. +// spacing line 26 for guard 2 +// spacing line 27 for guard 2 +// spacing line 28 for guard 2 +// spacing line 29 for guard 2 +// spacing line 30 for guard 2 +// spacing line 31 for guard 2 +// spacing line 32 for guard 2 +// spacing line 33 for guard 2 +// spacing line 34 for guard 2 +// spacing line 35 for guard 2 +// spacing line 36 for guard 2 +// spacing line 37 for guard 2 +// spacing line 38 for guard 2 +// spacing line 39 for guard 2 +// spacing line 40 for guard 2 +// spacing line 41 for guard 2 +// spacing line 42 for guard 2 +// spacing line 43 for guard 2 +// spacing line 44 for guard 2 +// spacing line 45 for guard 2 +// spacing line 46 for guard 2 +// spacing line 47 for guard 2 +// spacing line 48 for guard 2 +// spacing line 49 for guard 2 +// spacing line 50 for guard 2 +// spacing line 51 for guard 2 +// spacing line 52 for guard 2 +// spacing line 53 for guard 2 +// spacing line 54 for guard 2 +// spacing line 55 for guard 2 + +// Guard2 validates request field 2. +func Guard2(value int) bool { + return value >= 2 && value < 1_000_002 +} + +// guard 3 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 3. +// spacing line 2 for guard 3. +// spacing line 3 for guard 3. +// spacing line 4 for guard 3. +// spacing line 5 for guard 3. +// spacing line 6 for guard 3. +// spacing line 7 for guard 3. +// spacing line 8 for guard 3. +// spacing line 9 for guard 3. +// spacing line 10 for guard 3. +// spacing line 11 for guard 3. +// spacing line 12 for guard 3. +// spacing line 13 for guard 3. +// spacing line 14 for guard 3. +// spacing line 15 for guard 3. +// spacing line 16 for guard 3. +// spacing line 17 for guard 3. +// spacing line 18 for guard 3. +// spacing line 19 for guard 3. +// spacing line 20 for guard 3. +// spacing line 21 for guard 3. +// spacing line 22 for guard 3. +// spacing line 23 for guard 3. +// spacing line 24 for guard 3. +// spacing line 25 for guard 3. +// spacing line 26 for guard 3 +// spacing line 27 for guard 3 +// spacing line 28 for guard 3 +// spacing line 29 for guard 3 +// spacing line 30 for guard 3 +// spacing line 31 for guard 3 +// spacing line 32 for guard 3 +// spacing line 33 for guard 3 +// spacing line 34 for guard 3 +// spacing line 35 for guard 3 +// spacing line 36 for guard 3 +// spacing line 37 for guard 3 +// spacing line 38 for guard 3 +// spacing line 39 for guard 3 +// spacing line 40 for guard 3 +// spacing line 41 for guard 3 +// spacing line 42 for guard 3 +// spacing line 43 for guard 3 +// spacing line 44 for guard 3 +// spacing line 45 for guard 3 +// spacing line 46 for guard 3 +// spacing line 47 for guard 3 +// spacing line 48 for guard 3 +// spacing line 49 for guard 3 +// spacing line 50 for guard 3 +// spacing line 51 for guard 3 +// spacing line 52 for guard 3 +// spacing line 53 for guard 3 +// spacing line 54 for guard 3 +// spacing line 55 for guard 3 + +// Guard3 validates request field 3. +func Guard3(value int) bool { + return value >= 3 && value < 1_000_003 +} + +// guard 4 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 4. +// spacing line 2 for guard 4. +// spacing line 3 for guard 4. +// spacing line 4 for guard 4. +// spacing line 5 for guard 4. +// spacing line 6 for guard 4. +// spacing line 7 for guard 4. +// spacing line 8 for guard 4. +// spacing line 9 for guard 4. +// spacing line 10 for guard 4. +// spacing line 11 for guard 4. +// spacing line 12 for guard 4. +// spacing line 13 for guard 4. +// spacing line 14 for guard 4. +// spacing line 15 for guard 4. +// spacing line 16 for guard 4. +// spacing line 17 for guard 4. +// spacing line 18 for guard 4. +// spacing line 19 for guard 4. +// spacing line 20 for guard 4. +// spacing line 21 for guard 4. +// spacing line 22 for guard 4. +// spacing line 23 for guard 4. +// spacing line 24 for guard 4. +// spacing line 25 for guard 4. +// spacing line 26 for guard 4 +// spacing line 27 for guard 4 +// spacing line 28 for guard 4 +// spacing line 29 for guard 4 +// spacing line 30 for guard 4 +// spacing line 31 for guard 4 +// spacing line 32 for guard 4 +// spacing line 33 for guard 4 +// spacing line 34 for guard 4 +// spacing line 35 for guard 4 +// spacing line 36 for guard 4 +// spacing line 37 for guard 4 +// spacing line 38 for guard 4 +// spacing line 39 for guard 4 +// spacing line 40 for guard 4 +// spacing line 41 for guard 4 +// spacing line 42 for guard 4 +// spacing line 43 for guard 4 +// spacing line 44 for guard 4 +// spacing line 45 for guard 4 +// spacing line 46 for guard 4 +// spacing line 47 for guard 4 +// spacing line 48 for guard 4 +// spacing line 49 for guard 4 +// spacing line 50 for guard 4 +// spacing line 51 for guard 4 +// spacing line 52 for guard 4 +// spacing line 53 for guard 4 +// spacing line 54 for guard 4 +// spacing line 55 for guard 4 + +// Guard4 validates request field 4. +func Guard4(value int) bool { + return value >= 4 && value < 1_000_004 +} + +// guard 5 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 5. +// spacing line 2 for guard 5. +// spacing line 3 for guard 5. +// spacing line 4 for guard 5. +// spacing line 5 for guard 5. +// spacing line 6 for guard 5. +// spacing line 7 for guard 5. +// spacing line 8 for guard 5. +// spacing line 9 for guard 5. +// spacing line 10 for guard 5. +// spacing line 11 for guard 5. +// spacing line 12 for guard 5. +// spacing line 13 for guard 5. +// spacing line 14 for guard 5. +// spacing line 15 for guard 5. +// spacing line 16 for guard 5. +// spacing line 17 for guard 5. +// spacing line 18 for guard 5. +// spacing line 19 for guard 5. +// spacing line 20 for guard 5. +// spacing line 21 for guard 5. +// spacing line 22 for guard 5. +// spacing line 23 for guard 5. +// spacing line 24 for guard 5. +// spacing line 25 for guard 5. +// spacing line 26 for guard 5 +// spacing line 27 for guard 5 +// spacing line 28 for guard 5 +// spacing line 29 for guard 5 +// spacing line 30 for guard 5 +// spacing line 31 for guard 5 +// spacing line 32 for guard 5 +// spacing line 33 for guard 5 +// spacing line 34 for guard 5 +// spacing line 35 for guard 5 +// spacing line 36 for guard 5 +// spacing line 37 for guard 5 +// spacing line 38 for guard 5 +// spacing line 39 for guard 5 +// spacing line 40 for guard 5 +// spacing line 41 for guard 5 +// spacing line 42 for guard 5 +// spacing line 43 for guard 5 +// spacing line 44 for guard 5 +// spacing line 45 for guard 5 +// spacing line 46 for guard 5 +// spacing line 47 for guard 5 +// spacing line 48 for guard 5 +// spacing line 49 for guard 5 +// spacing line 50 for guard 5 +// spacing line 51 for guard 5 +// spacing line 52 for guard 5 +// spacing line 53 for guard 5 +// spacing line 54 for guard 5 +// spacing line 55 for guard 5 + +// Guard5 validates request field 5. +func Guard5(value int) bool { + return value >= 5 && value < 1_000_005 +} + +// guard 6 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 6. +// spacing line 2 for guard 6. +// spacing line 3 for guard 6. +// spacing line 4 for guard 6. +// spacing line 5 for guard 6. +// spacing line 6 for guard 6. +// spacing line 7 for guard 6. +// spacing line 8 for guard 6. +// spacing line 9 for guard 6. +// spacing line 10 for guard 6. +// spacing line 11 for guard 6. +// spacing line 12 for guard 6. +// spacing line 13 for guard 6. +// spacing line 14 for guard 6. +// spacing line 15 for guard 6. +// spacing line 16 for guard 6. +// spacing line 17 for guard 6. +// spacing line 18 for guard 6. +// spacing line 19 for guard 6. +// spacing line 20 for guard 6. +// spacing line 21 for guard 6. +// spacing line 22 for guard 6. +// spacing line 23 for guard 6. +// spacing line 24 for guard 6. +// spacing line 25 for guard 6. +// spacing line 26 for guard 6 +// spacing line 27 for guard 6 +// spacing line 28 for guard 6 +// spacing line 29 for guard 6 +// spacing line 30 for guard 6 +// spacing line 31 for guard 6 +// spacing line 32 for guard 6 +// spacing line 33 for guard 6 +// spacing line 34 for guard 6 +// spacing line 35 for guard 6 +// spacing line 36 for guard 6 +// spacing line 37 for guard 6 +// spacing line 38 for guard 6 +// spacing line 39 for guard 6 +// spacing line 40 for guard 6 +// spacing line 41 for guard 6 +// spacing line 42 for guard 6 +// spacing line 43 for guard 6 +// spacing line 44 for guard 6 +// spacing line 45 for guard 6 +// spacing line 46 for guard 6 +// spacing line 47 for guard 6 +// spacing line 48 for guard 6 +// spacing line 49 for guard 6 +// spacing line 50 for guard 6 +// spacing line 51 for guard 6 +// spacing line 52 for guard 6 +// spacing line 53 for guard 6 +// spacing line 54 for guard 6 +// spacing line 55 for guard 6 + +// Guard6 validates request field 6. +func Guard6(value int) bool { + return value >= 6 && value < 1_000_006 +} + +// guard 7 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 7. +// spacing line 2 for guard 7. +// spacing line 3 for guard 7. +// spacing line 4 for guard 7. +// spacing line 5 for guard 7. +// spacing line 6 for guard 7. +// spacing line 7 for guard 7. +// spacing line 8 for guard 7. +// spacing line 9 for guard 7. +// spacing line 10 for guard 7. +// spacing line 11 for guard 7. +// spacing line 12 for guard 7. +// spacing line 13 for guard 7. +// spacing line 14 for guard 7. +// spacing line 15 for guard 7. +// spacing line 16 for guard 7. +// spacing line 17 for guard 7. +// spacing line 18 for guard 7. +// spacing line 19 for guard 7. +// spacing line 20 for guard 7. +// spacing line 21 for guard 7. +// spacing line 22 for guard 7. +// spacing line 23 for guard 7. +// spacing line 24 for guard 7. +// spacing line 25 for guard 7. +// spacing line 26 for guard 7 +// spacing line 27 for guard 7 +// spacing line 28 for guard 7 +// spacing line 29 for guard 7 +// spacing line 30 for guard 7 +// spacing line 31 for guard 7 +// spacing line 32 for guard 7 +// spacing line 33 for guard 7 +// spacing line 34 for guard 7 +// spacing line 35 for guard 7 +// spacing line 36 for guard 7 +// spacing line 37 for guard 7 +// spacing line 38 for guard 7 +// spacing line 39 for guard 7 +// spacing line 40 for guard 7 +// spacing line 41 for guard 7 +// spacing line 42 for guard 7 +// spacing line 43 for guard 7 +// spacing line 44 for guard 7 +// spacing line 45 for guard 7 +// spacing line 46 for guard 7 +// spacing line 47 for guard 7 +// spacing line 48 for guard 7 +// spacing line 49 for guard 7 +// spacing line 50 for guard 7 +// spacing line 51 for guard 7 +// spacing line 52 for guard 7 +// spacing line 53 for guard 7 +// spacing line 54 for guard 7 +// spacing line 55 for guard 7 + +// AtCapacity reports whether the queue is full. +func AtCapacity(used, capacity int) bool { + return !(used < capacity) +} + +// guard 8 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 8. +// spacing line 2 for guard 8. +// spacing line 3 for guard 8. +// spacing line 4 for guard 8. +// spacing line 5 for guard 8. +// spacing line 6 for guard 8. +// spacing line 7 for guard 8. +// spacing line 8 for guard 8. +// spacing line 9 for guard 8. +// spacing line 10 for guard 8. +// spacing line 11 for guard 8. +// spacing line 12 for guard 8. +// spacing line 13 for guard 8. +// spacing line 14 for guard 8. +// spacing line 15 for guard 8. +// spacing line 16 for guard 8. +// spacing line 17 for guard 8. +// spacing line 18 for guard 8. +// spacing line 19 for guard 8. +// spacing line 20 for guard 8. +// spacing line 21 for guard 8. +// spacing line 22 for guard 8. +// spacing line 23 for guard 8. +// spacing line 24 for guard 8. +// spacing line 25 for guard 8. +// spacing line 26 for guard 8 +// spacing line 27 for guard 8 +// spacing line 28 for guard 8 +// spacing line 29 for guard 8 +// spacing line 30 for guard 8 +// spacing line 31 for guard 8 +// spacing line 32 for guard 8 +// spacing line 33 for guard 8 +// spacing line 34 for guard 8 +// spacing line 35 for guard 8 +// spacing line 36 for guard 8 +// spacing line 37 for guard 8 +// spacing line 38 for guard 8 +// spacing line 39 for guard 8 +// spacing line 40 for guard 8 +// spacing line 41 for guard 8 +// spacing line 42 for guard 8 +// spacing line 43 for guard 8 +// spacing line 44 for guard 8 +// spacing line 45 for guard 8 +// spacing line 46 for guard 8 +// spacing line 47 for guard 8 +// spacing line 48 for guard 8 +// spacing line 49 for guard 8 +// spacing line 50 for guard 8 +// spacing line 51 for guard 8 +// spacing line 52 for guard 8 +// spacing line 53 for guard 8 +// spacing line 54 for guard 8 +// spacing line 55 for guard 8 + +// Guard8 validates request field 8. +func Guard8(value int) bool { + return value >= 8 && value < 1_000_008 +} + +// guard 9 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 9. +// spacing line 2 for guard 9. +// spacing line 3 for guard 9. +// spacing line 4 for guard 9. +// spacing line 5 for guard 9. +// spacing line 6 for guard 9. +// spacing line 7 for guard 9. +// spacing line 8 for guard 9. +// spacing line 9 for guard 9. +// spacing line 10 for guard 9. +// spacing line 11 for guard 9. +// spacing line 12 for guard 9. +// spacing line 13 for guard 9. +// spacing line 14 for guard 9. +// spacing line 15 for guard 9. +// spacing line 16 for guard 9. +// spacing line 17 for guard 9. +// spacing line 18 for guard 9. +// spacing line 19 for guard 9. +// spacing line 20 for guard 9. +// spacing line 21 for guard 9. +// spacing line 22 for guard 9. +// spacing line 23 for guard 9. +// spacing line 24 for guard 9. +// spacing line 25 for guard 9. +// spacing line 26 for guard 9 +// spacing line 27 for guard 9 +// spacing line 28 for guard 9 +// spacing line 29 for guard 9 +// spacing line 30 for guard 9 +// spacing line 31 for guard 9 +// spacing line 32 for guard 9 +// spacing line 33 for guard 9 +// spacing line 34 for guard 9 +// spacing line 35 for guard 9 +// spacing line 36 for guard 9 +// spacing line 37 for guard 9 +// spacing line 38 for guard 9 +// spacing line 39 for guard 9 +// spacing line 40 for guard 9 +// spacing line 41 for guard 9 +// spacing line 42 for guard 9 +// spacing line 43 for guard 9 +// spacing line 44 for guard 9 +// spacing line 45 for guard 9 +// spacing line 46 for guard 9 +// spacing line 47 for guard 9 +// spacing line 48 for guard 9 +// spacing line 49 for guard 9 +// spacing line 50 for guard 9 +// spacing line 51 for guard 9 +// spacing line 52 for guard 9 +// spacing line 53 for guard 9 +// spacing line 54 for guard 9 +// spacing line 55 for guard 9 + +// Guard9 validates request field 9. +func Guard9(value int) bool { + return value >= 9 && value < 1_000_009 +} + +// guard 10 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 10. +// spacing line 2 for guard 10. +// spacing line 3 for guard 10. +// spacing line 4 for guard 10. +// spacing line 5 for guard 10. +// spacing line 6 for guard 10. +// spacing line 7 for guard 10. +// spacing line 8 for guard 10. +// spacing line 9 for guard 10. +// spacing line 10 for guard 10. +// spacing line 11 for guard 10. +// spacing line 12 for guard 10. +// spacing line 13 for guard 10. +// spacing line 14 for guard 10. +// spacing line 15 for guard 10. +// spacing line 16 for guard 10. +// spacing line 17 for guard 10. +// spacing line 18 for guard 10. +// spacing line 19 for guard 10. +// spacing line 20 for guard 10. +// spacing line 21 for guard 10. +// spacing line 22 for guard 10. +// spacing line 23 for guard 10. +// spacing line 24 for guard 10. +// spacing line 25 for guard 10. +// spacing line 26 for guard 10 +// spacing line 27 for guard 10 +// spacing line 28 for guard 10 +// spacing line 29 for guard 10 +// spacing line 30 for guard 10 +// spacing line 31 for guard 10 +// spacing line 32 for guard 10 +// spacing line 33 for guard 10 +// spacing line 34 for guard 10 +// spacing line 35 for guard 10 +// spacing line 36 for guard 10 +// spacing line 37 for guard 10 +// spacing line 38 for guard 10 +// spacing line 39 for guard 10 +// spacing line 40 for guard 10 +// spacing line 41 for guard 10 +// spacing line 42 for guard 10 +// spacing line 43 for guard 10 +// spacing line 44 for guard 10 +// spacing line 45 for guard 10 +// spacing line 46 for guard 10 +// spacing line 47 for guard 10 +// spacing line 48 for guard 10 +// spacing line 49 for guard 10 +// spacing line 50 for guard 10 +// spacing line 51 for guard 10 +// spacing line 52 for guard 10 +// spacing line 53 for guard 10 +// spacing line 54 for guard 10 +// spacing line 55 for guard 10 + +// ShouldRetry reports whether another attempt is allowed. +func ShouldRetry(attempt, maxAttempts int) bool { + return attempt <= maxAttempts +} + +// guard 11 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 11. +// spacing line 2 for guard 11. +// spacing line 3 for guard 11. +// spacing line 4 for guard 11. +// spacing line 5 for guard 11. +// spacing line 6 for guard 11. +// spacing line 7 for guard 11. +// spacing line 8 for guard 11. +// spacing line 9 for guard 11. +// spacing line 10 for guard 11. +// spacing line 11 for guard 11. +// spacing line 12 for guard 11. +// spacing line 13 for guard 11. +// spacing line 14 for guard 11. +// spacing line 15 for guard 11. +// spacing line 16 for guard 11. +// spacing line 17 for guard 11. +// spacing line 18 for guard 11. +// spacing line 19 for guard 11. +// spacing line 20 for guard 11. +// spacing line 21 for guard 11. +// spacing line 22 for guard 11. +// spacing line 23 for guard 11. +// spacing line 24 for guard 11. +// spacing line 25 for guard 11. +// spacing line 26 for guard 11 +// spacing line 27 for guard 11 +// spacing line 28 for guard 11 +// spacing line 29 for guard 11 +// spacing line 30 for guard 11 +// spacing line 31 for guard 11 +// spacing line 32 for guard 11 +// spacing line 33 for guard 11 +// spacing line 34 for guard 11 +// spacing line 35 for guard 11 +// spacing line 36 for guard 11 +// spacing line 37 for guard 11 +// spacing line 38 for guard 11 +// spacing line 39 for guard 11 +// spacing line 40 for guard 11 +// spacing line 41 for guard 11 +// spacing line 42 for guard 11 +// spacing line 43 for guard 11 +// spacing line 44 for guard 11 +// spacing line 45 for guard 11 +// spacing line 46 for guard 11 +// spacing line 47 for guard 11 +// spacing line 48 for guard 11 +// spacing line 49 for guard 11 +// spacing line 50 for guard 11 +// spacing line 51 for guard 11 +// spacing line 52 for guard 11 +// spacing line 53 for guard 11 +// spacing line 54 for guard 11 +// spacing line 55 for guard 11 + +// Guard11 validates request field 11. +func Guard11(value int) bool { + return value >= 11 && value < 1_000_011 +} + +// guard 12 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 12. +// spacing line 2 for guard 12. +// spacing line 3 for guard 12. +// spacing line 4 for guard 12. +// spacing line 5 for guard 12. +// spacing line 6 for guard 12. +// spacing line 7 for guard 12. +// spacing line 8 for guard 12. +// spacing line 9 for guard 12. +// spacing line 10 for guard 12. +// spacing line 11 for guard 12. +// spacing line 12 for guard 12. +// spacing line 13 for guard 12. +// spacing line 14 for guard 12. +// spacing line 15 for guard 12. +// spacing line 16 for guard 12. +// spacing line 17 for guard 12. +// spacing line 18 for guard 12. +// spacing line 19 for guard 12. +// spacing line 20 for guard 12. +// spacing line 21 for guard 12. +// spacing line 22 for guard 12. +// spacing line 23 for guard 12. +// spacing line 24 for guard 12. +// spacing line 25 for guard 12. +// spacing line 26 for guard 12 +// spacing line 27 for guard 12 +// spacing line 28 for guard 12 +// spacing line 29 for guard 12 +// spacing line 30 for guard 12 +// spacing line 31 for guard 12 +// spacing line 32 for guard 12 +// spacing line 33 for guard 12 +// spacing line 34 for guard 12 +// spacing line 35 for guard 12 +// spacing line 36 for guard 12 +// spacing line 37 for guard 12 +// spacing line 38 for guard 12 +// spacing line 39 for guard 12 +// spacing line 40 for guard 12 +// spacing line 41 for guard 12 +// spacing line 42 for guard 12 +// spacing line 43 for guard 12 +// spacing line 44 for guard 12 +// spacing line 45 for guard 12 +// spacing line 46 for guard 12 +// spacing line 47 for guard 12 +// spacing line 48 for guard 12 +// spacing line 49 for guard 12 +// spacing line 50 for guard 12 +// spacing line 51 for guard 12 +// spacing line 52 for guard 12 +// spacing line 53 for guard 12 +// spacing line 54 for guard 12 +// spacing line 55 for guard 12 + +// Guard12 validates request field 12. +func Guard12(value int) bool { + return value >= 12 && value < 1_000_012 +} + +// guard 13 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 13. +// spacing line 2 for guard 13. +// spacing line 3 for guard 13. +// spacing line 4 for guard 13. +// spacing line 5 for guard 13. +// spacing line 6 for guard 13. +// spacing line 7 for guard 13. +// spacing line 8 for guard 13. +// spacing line 9 for guard 13. +// spacing line 10 for guard 13. +// spacing line 11 for guard 13. +// spacing line 12 for guard 13. +// spacing line 13 for guard 13. +// spacing line 14 for guard 13. +// spacing line 15 for guard 13. +// spacing line 16 for guard 13. +// spacing line 17 for guard 13. +// spacing line 18 for guard 13. +// spacing line 19 for guard 13. +// spacing line 20 for guard 13. +// spacing line 21 for guard 13. +// spacing line 22 for guard 13. +// spacing line 23 for guard 13. +// spacing line 24 for guard 13. +// spacing line 25 for guard 13. +// spacing line 26 for guard 13 +// spacing line 27 for guard 13 +// spacing line 28 for guard 13 +// spacing line 29 for guard 13 +// spacing line 30 for guard 13 +// spacing line 31 for guard 13 +// spacing line 32 for guard 13 +// spacing line 33 for guard 13 +// spacing line 34 for guard 13 +// spacing line 35 for guard 13 +// spacing line 36 for guard 13 +// spacing line 37 for guard 13 +// spacing line 38 for guard 13 +// spacing line 39 for guard 13 +// spacing line 40 for guard 13 +// spacing line 41 for guard 13 +// spacing line 42 for guard 13 +// spacing line 43 for guard 13 +// spacing line 44 for guard 13 +// spacing line 45 for guard 13 +// spacing line 46 for guard 13 +// spacing line 47 for guard 13 +// spacing line 48 for guard 13 +// spacing line 49 for guard 13 +// spacing line 50 for guard 13 +// spacing line 51 for guard 13 +// spacing line 52 for guard 13 +// spacing line 53 for guard 13 +// spacing line 54 for guard 13 +// spacing line 55 for guard 13 + +// ShardIndex maps a key onto the configured shard range. +func ShardIndex(key, shards int) int { + if shards <= 0 { + return 0 + } + return key % (shards + 1) +} + +// guard 14 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 14. +// spacing line 2 for guard 14. +// spacing line 3 for guard 14. +// spacing line 4 for guard 14. +// spacing line 5 for guard 14. +// spacing line 6 for guard 14. +// spacing line 7 for guard 14. +// spacing line 8 for guard 14. +// spacing line 9 for guard 14. +// spacing line 10 for guard 14. +// spacing line 11 for guard 14. +// spacing line 12 for guard 14. +// spacing line 13 for guard 14. +// spacing line 14 for guard 14. +// spacing line 15 for guard 14. +// spacing line 16 for guard 14. +// spacing line 17 for guard 14. +// spacing line 18 for guard 14. +// spacing line 19 for guard 14. +// spacing line 20 for guard 14. +// spacing line 21 for guard 14. +// spacing line 22 for guard 14. +// spacing line 23 for guard 14. +// spacing line 24 for guard 14. +// spacing line 25 for guard 14. +// spacing line 26 for guard 14 +// spacing line 27 for guard 14 +// spacing line 28 for guard 14 +// spacing line 29 for guard 14 +// spacing line 30 for guard 14 +// spacing line 31 for guard 14 +// spacing line 32 for guard 14 +// spacing line 33 for guard 14 +// spacing line 34 for guard 14 +// spacing line 35 for guard 14 +// spacing line 36 for guard 14 +// spacing line 37 for guard 14 +// spacing line 38 for guard 14 +// spacing line 39 for guard 14 +// spacing line 40 for guard 14 +// spacing line 41 for guard 14 +// spacing line 42 for guard 14 +// spacing line 43 for guard 14 +// spacing line 44 for guard 14 +// spacing line 45 for guard 14 +// spacing line 46 for guard 14 +// spacing line 47 for guard 14 +// spacing line 48 for guard 14 +// spacing line 49 for guard 14 +// spacing line 50 for guard 14 +// spacing line 51 for guard 14 +// spacing line 52 for guard 14 +// spacing line 53 for guard 14 +// spacing line 54 for guard 14 +// spacing line 55 for guard 14 + +// Guard14 validates request field 14. +func Guard14(value int) bool { + return value >= 14 && value < 1_000_014 +} + +// guard 15 is intentionally verbose so hunks stay far apart. +// spacing line 1 for guard 15. +// spacing line 2 for guard 15. +// spacing line 3 for guard 15. +// spacing line 4 for guard 15. +// spacing line 5 for guard 15. +// spacing line 6 for guard 15. +// spacing line 7 for guard 15. +// spacing line 8 for guard 15. +// spacing line 9 for guard 15. +// spacing line 10 for guard 15. +// spacing line 11 for guard 15. +// spacing line 12 for guard 15. +// spacing line 13 for guard 15. +// spacing line 14 for guard 15. +// spacing line 15 for guard 15. +// spacing line 16 for guard 15. +// spacing line 17 for guard 15. +// spacing line 18 for guard 15. +// spacing line 19 for guard 15. +// spacing line 20 for guard 15. +// spacing line 21 for guard 15. +// spacing line 22 for guard 15. +// spacing line 23 for guard 15. +// spacing line 24 for guard 15. +// spacing line 25 for guard 15. +// spacing line 26 for guard 15 +// spacing line 27 for guard 15 +// spacing line 28 for guard 15 +// spacing line 29 for guard 15 +// spacing line 30 for guard 15 +// spacing line 31 for guard 15 +// spacing line 32 for guard 15 +// spacing line 33 for guard 15 +// spacing line 34 for guard 15 +// spacing line 35 for guard 15 +// spacing line 36 for guard 15 +// spacing line 37 for guard 15 +// spacing line 38 for guard 15 +// spacing line 39 for guard 15 +// spacing line 40 for guard 15 +// spacing line 41 for guard 15 +// spacing line 42 for guard 15 +// spacing line 43 for guard 15 +// spacing line 44 for guard 15 +// spacing line 45 for guard 15 +// spacing line 46 for guard 15 +// spacing line 47 for guard 15 +// spacing line 48 for guard 15 +// spacing line 49 for guard 15 +// spacing line 50 for guard 15 +// spacing line 51 for guard 15 +// spacing line 52 for guard 15 +// spacing line 53 for guard 15 +// spacing line 54 for guard 15 +// spacing line 55 for guard 15 + +// Guard15 validates request field 15. +func Guard15(value int) bool { + return value >= 15 && value < 1_000_015 +} diff --git a/scripts/packet-packing-report.ts b/scripts/packet-packing-report.ts index 3b827f0..d075a82 100644 --- a/scripts/packet-packing-report.ts +++ b/scripts/packet-packing-report.ts @@ -43,7 +43,9 @@ type FailureCode = | "budget_downgraded" | "lens_dropped" | "dispatch_rank_invalid" - | "estimator_unreconciled"; + | "estimator_unreconciled" + | "treatment_shape_unmet" + | "targets_share_packet"; export type Failure = { code: FailureCode; run: string; message: string; fields: Record }; @@ -190,6 +192,83 @@ export function comparePackets(run: string, off: ReviewPacket[], on: ReviewPacke return failures; } +// Model-free proof that a fixture actually receives the treatment the recall +// curve depends on: exact packet counts at each cap, and each target hunk in a +// packet of exactly the intended size with no two targets sharing a packet. +async function treatment(args: Map): Promise { + const repo = args.get("repo")?.[0]; + const base = args.get("base")?.[0] ?? "main"; + const head = args.get("head")?.[0] ?? "feature"; + const targets = (args.get("target") ?? []).flatMap((value) => value.split(",")); + const expectPackets = (args.get("expect-packets")?.[0] ?? "15,5,3").split(",").map(Number); + const caps = (args.get("caps")?.[0] ?? "1,3,5").split(",").map(Number); + const output = args.get("output")?.[0]; + if (repo === undefined || output === undefined) { + console.error("usage: packet-packing-report.ts treatment --repo --target ... --output "); + return 2; + } + + const git = createGitClient(repo); + const rawDiff = await git.diff(base, head); + const resolved: ResolvedReviewInput = { + mode: "commit_range", + repoRoot: repo, + baseRef: base, + headRef: head, + headSha: head, + mergeBase: base, + commits: [], + rawDiff + }; + const parsed = parseDiff(rawDiff); + const targetHunkIds = parsed.files + .flatMap((file) => file.hunks) + .filter((hunk) => targets.includes(String(hunk.newStart))) + .map((hunk) => hunk.id); + + const failures: Failure[] = []; + const shapes: Array> = []; + for (const [index, cap] of caps.entries()) { + const config = packingConfig(repo, true); + config.review.packMaxHunks = cap; + const counters = { events: 0, modelCalls: 0 }; + const telemetry = countingTelemetry(counters); + const { kept, decisions } = await filterDiffFiles(resolved, parsed, config, telemetry); + const facts = await classifyChangedFiles(resolved, kept, decisions, config, telemetry); + const repoIndex = await buildRepositoryIndex(resolved, kept, facts, config, telemetry); + const packets = await buildReviewPackets( + { diffUnderstanding: { declaredIntent: "dilution fixture", inferredBehavior: "boundary guards" }, coverage: [] }, + kept, + facts, + repoIndex, + telemetry, + { config, enabledLenses: ["lang/go"] } + ); + const expected = expectPackets[index]; + if (expected !== undefined && packets.length !== expected) { + failures.push(fail("treatment_shape_unmet", `cap-${cap}`, { expected, actual: packets.length })); + } + const targetPackets = targetHunkIds.map((hunkId) => packets.find((packet) => packet.hunks.some((hunk) => hunk.hunkId === hunkId))); + const targetSizes = targetPackets.map((packet) => packet?.hunks.length ?? 0); + if (targetSizes.some((size) => size !== cap)) { + failures.push(fail("treatment_shape_unmet", `cap-${cap}`, { expectedTargetHunks: cap, actual: targetSizes })); + } + const targetPacketIds = targetPackets.map((packet) => packet?.id ?? ""); + if (new Set(targetPacketIds).size !== targetPacketIds.length) { + failures.push(fail("targets_share_packet", `cap-${cap}`, { packetIds: targetPacketIds })); + } + if (counters.modelCalls > 0) { + failures.push(fail("model_call_observed", `cap-${cap}`, { modelCalls: counters.modelCalls })); + } + shapes.push({ cap, packets: packets.length, targetPacketHunks: targetSizes, distinctTargetPackets: new Set(targetPacketIds).size }); + } + + const report = { schemaVersion: 1, mode: "treatment", repo, targets, shapes, failures }; + writeFileSync(output, `${JSON.stringify(report, null, 2)}\n`); + console.log(JSON.stringify({ shapes, failures }, null, 1)); + return failures.length === 0 ? 0 : 1; +} + async function replay(args: Map): Promise { const repo = args.get("repo")?.[0]; const runs = args.get("run") ?? []; @@ -313,6 +392,10 @@ async function main(): Promise { process.exitCode = await replay(args); return; } + if (mode === "treatment") { + process.exitCode = await treatment(args); + return; + } console.error("usage: packet-packing-report.ts replay --repo --run ... --output "); process.exitCode = 2; } diff --git a/specs/plans/103-issue-103-relationship-driven-packet-packing.md b/specs/plans/103-issue-103-relationship-driven-packet-packing.md index f4fe59a..8a473b9 100644 --- a/specs/plans/103-issue-103-relationship-driven-packet-packing.md +++ b/specs/plans/103-issue-103-relationship-driven-packet-packing.md @@ -319,6 +319,8 @@ Those opportunities are **clustered, not independent**: all repeats share one fi ### Cost discipline +**`approvedValidationCostUSD: $300`** — recorded 2026-07-25 on owner approval. The owner approved the plan's `$119` reservation and raised the ceiling to `$300`, and separately authorized running the production capacity pair inside Phase 4 rather than deferring it. Spend to date: `$0.00`. + No paid call before `approvedValidationCostUSD` is recorded here. | Phase | Executions | Projection | diff --git a/specs/plans/phase-plans/103/phase_4.md b/specs/plans/phase-plans/103/phase_4.md new file mode 100644 index 0000000..b9bf2cc --- /dev/null +++ b/specs/plans/phase-plans/103/phase_4.md @@ -0,0 +1,52 @@ +--- +status: in progress (preconditions complete; no paid call yet) +--- + +# Phase 4: Paid Validation + +## Overview + +Plan 103 step 8, plus step 6 carried forward from Phase 3. Every free precondition runs before the first paid call, in the order the plan requires: fixture construction, expectation validation under the fake provider, the authored frozen plan, the reference-draw realism gate, and the model-free treatment proof. + +## Authorization + +**`approvedValidationCostUSD: $300`**, recorded in the plan on 2026-07-25. The owner approved the `$119` reservation and raised the ceiling to `$300`, and authorized running the production capacity pair inside this phase rather than deferring it. Provider credentials are present (`anthropic`, default `claude-opus-4-8`). + +Spend to date: **`$0.00`**. + +## Completed + +**Fixture.** `evals/packet-dilution/repos/dilution/{base,feature}` — one Go file, fifteen independent guard functions separated by ~62 lines, comfortably beyond `NEARBY_GAP_LINES = 30`, so today's grouper yields exactly fifteen atoms. Three bugs at atom positions 1, 10 and 13: + +- atom 1 — `WithinTransferLimit` changes an inclusive bound to exclusive, rejecting the limit value itself; +- atom 10 — `ShouldRetry` changes `<` to `<=`, allowing one attempt beyond the maximum; +- atom 13 — `ShardIndex` maps onto `shards + 1`, producing an index outside the configured range. + +Atom 7 is the negative control: `used >= capacity` rewritten as `!(used < capacity)`, which looks like a boundary edit and is provably equivalent. The remaining eleven guards are safe range additions. + +**Treatment proof.** `packet-packing-report.ts treatment` rebuilds Stage 6 at caps 1/3/5 with zero model calls and asserts the exact shape the curve depends on: + +| Cap | Packets | Target packet size | Distinct target packets | +| ---: | ---: | --- | ---: | +| 1 | 15 | 1, 1, 1 | 3 | +| 3 | 5 | 3, 3, 3 | 3 | +| 5 | 3 | 5, 5, 5 | 3 | + +No two targets share a packet at any cap. Preserved at `packet-dilution/reports/plan103-treatment-proof.json`. + +### The proof caught a fixture defect on first run + +The initial invocation targeted the hunk at line 679, which is atom **11**, not atom 10. At cap 5 that places two targets inside `[11-15]`, collapsing two of the three recall opportunities into one packet and one model conversation — exactly the clustering defect the review flagged before implementation. The `targets_share_packet` check failed closed and the target was corrected to line 617. Had the proof not existed, the curve would have run at $14 with two of three observations correlated. + +## Remaining before the first paid call + +1. **Step 6 — the pinned-plan seam.** Versioned `PinnedPlanArtifact` with `baseSha`, `headSha`, `planSha256` over a canonical sorted-key serialization; `review.pinnedPlanPath` validation failing closed on wrapper schema, plan schema, hash, base, head, and hunk-ID membership; `scripts/draw-pinned-plan.ts` as an internal script rather than a CLI verb; one test per failure mode. +2. **Eval case YAMLs** — `cap1.yml`, `cap3.yml`, `cap5.yml` at `repeat: 6`, cache off, `lang/go` only, identical except `packMaxHunks`, all pinned to one authored plan. +3. **Expectation validation under the fake provider** — confirm each `should_find` matcher fires on the expected finding shape, and the `should_not_find` control does not, for `$0`. +4. **Authored frozen plan plus three reference draws** and the realism report requiring at least 2 of 3 to place every target in a five-hunk packet at cap 5. + +Only after all four does the `6 × 3` curve run. + +## Outcome + +_Preconditions complete; paid validation not started._ From c8d033d2c2b5decc8effb23fd6368df79ce5d76b Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 13:38:21 -0400 Subject: [PATCH 16/24] feat(evals): pinned-plan seam and dilution curve cases (Plan 103 phase 4) Completes the remaining preconditions before the recall curve. Pinned-plan seam (step 6, carried from phase 3). ReviewPlan has no base/head identity and no self-describing hash, so a bare plan file cannot prove it belongs to the diff under review. PinnedPlanArtifact is an explicit versioned wrapper carrying baseSha, headSha, and planSha256 over a canonical serialization - keys sorted recursively, no whitespace, over the plan alone - so the hash is independent of field order and of the wrapper's own fields. loadPinnedPlan fails closed on unreadable JSON, wrapper schema mismatch, plan schema mismatch, hash mismatch, base mismatch, head mismatch, and any hunk id absent from the current diff. A hash that matches a mutated plan is still rejected against its diff. Fixture repos are materialized fresh per eval run, so their commit SHAs vary while content-derived hunk ids do not. A "*" sentinel records that an artifact is content-anchored rather than ref-anchored; hash, schema, and hunk-membership validation still apply in full. Ref pinning stays available for the production pair, where SHAs are fixed. review.pinnedPlanPath is eval-only: resolved schema and eval-case schema only, never rawConfigSchema or repo-safe filtering. The review runner substitutes the pinned plan for the Stage-5 draw and records a planner_plan_pinned event. scripts/draw-pinned-plan.ts writes artifacts as an internal script rather than a CLI verb, keeping src/cli out of scope as the plan requires. Three case YAMLs at repeat 6, cache off, lang/go only, identical except packMaxHunks 1/3/5, all pinned to one authored plan of 15 uniform- coverage hunks (planSha256 52e62561...). Expectations constrain path and failure mode with permissive title patterns and no category constraint - Plan 102 lost three executions to an expectation that demanded correctness when the reviewer said security. A should_not_find control covers the provably-equivalent AtCapacity rewrite. 785 tests. Spend so far is one in-flight smoke execution; the full curve has not run. Co-Authored-By: Claude Opus 5 (1M context) --- evals/packet-dilution/cap1.yml | 40 ++++++ evals/packet-dilution/cap3.yml | 40 ++++++ evals/packet-dilution/cap5.yml | 40 ++++++ evals/packet-dilution/plans/frozen.json | 164 ++++++++++++++++++++++++ scripts/draw-pinned-plan.ts | 60 +++++++++ src/config/schema.ts | 3 +- src/evals/eval-runner.ts | 4 + src/pipeline/pinned-plan.ts | 112 ++++++++++++++++ src/pipeline/review-runner.ts | 45 ++++++- src/types.ts | 4 + tests/pinned-plan.test.ts | 79 ++++++++++++ 11 files changed, 584 insertions(+), 7 deletions(-) create mode 100644 evals/packet-dilution/cap1.yml create mode 100644 evals/packet-dilution/cap3.yml create mode 100644 evals/packet-dilution/cap5.yml create mode 100644 evals/packet-dilution/plans/frozen.json create mode 100644 scripts/draw-pinned-plan.ts create mode 100644 src/pipeline/pinned-plan.ts create mode 100644 tests/pinned-plan.test.ts diff --git a/evals/packet-dilution/cap1.yml b/evals/packet-dilution/cap1.yml new file mode 100644 index 0000000..c761e47 --- /dev/null +++ b/evals/packet-dilution/cap1.yml @@ -0,0 +1,40 @@ +name: packet-dilution-cap1 +repeat: 6 +repo: + fixture: repos/dilution +command: + branch: feature + base: main +review: + provider: anthropic + model: claude-opus-4-8 + reasoning: high + depth: normal + verify: true + cache: false + concurrency: 1 + maxTimeMinutes: 20 + lenses: + - lang/go + packCompatibleAtoms: true + packMaxHunks: 1 + pinnedPlanPath: evals/packet-dilution/plans/frozen.json +expect: + maxCostUSD: 3 +should_find: + - id: dilution-transfer-limit + path: limits/limits.go + failureModePattern: limit|boundary|exclusive|reject|equal + titlePattern: limit|transfer|boundary|compar + - id: dilution-retry-off-by-one + path: limits/limits.go + failureModePattern: retry|attempt|off.by.one|extra|exceed + titlePattern: retry|attempt|maximum + - id: dilution-shard-range + path: limits/limits.go + failureModePattern: shard|range|index|modulo|out of + titlePattern: shard|index|range +should_not_find: + - id: dilution-capacity-control + path: limits/limits.go + titlePattern: AtCapacity diff --git a/evals/packet-dilution/cap3.yml b/evals/packet-dilution/cap3.yml new file mode 100644 index 0000000..81890ad --- /dev/null +++ b/evals/packet-dilution/cap3.yml @@ -0,0 +1,40 @@ +name: packet-dilution-cap3 +repeat: 6 +repo: + fixture: repos/dilution +command: + branch: feature + base: main +review: + provider: anthropic + model: claude-opus-4-8 + reasoning: high + depth: normal + verify: true + cache: false + concurrency: 1 + maxTimeMinutes: 20 + lenses: + - lang/go + packCompatibleAtoms: true + packMaxHunks: 3 + pinnedPlanPath: evals/packet-dilution/plans/frozen.json +expect: + maxCostUSD: 3 +should_find: + - id: dilution-transfer-limit + path: limits/limits.go + failureModePattern: limit|boundary|exclusive|reject|equal + titlePattern: limit|transfer|boundary|compar + - id: dilution-retry-off-by-one + path: limits/limits.go + failureModePattern: retry|attempt|off.by.one|extra|exceed + titlePattern: retry|attempt|maximum + - id: dilution-shard-range + path: limits/limits.go + failureModePattern: shard|range|index|modulo|out of + titlePattern: shard|index|range +should_not_find: + - id: dilution-capacity-control + path: limits/limits.go + titlePattern: AtCapacity diff --git a/evals/packet-dilution/cap5.yml b/evals/packet-dilution/cap5.yml new file mode 100644 index 0000000..d3ad672 --- /dev/null +++ b/evals/packet-dilution/cap5.yml @@ -0,0 +1,40 @@ +name: packet-dilution-cap5 +repeat: 6 +repo: + fixture: repos/dilution +command: + branch: feature + base: main +review: + provider: anthropic + model: claude-opus-4-8 + reasoning: high + depth: normal + verify: true + cache: false + concurrency: 1 + maxTimeMinutes: 20 + lenses: + - lang/go + packCompatibleAtoms: true + packMaxHunks: 5 + pinnedPlanPath: evals/packet-dilution/plans/frozen.json +expect: + maxCostUSD: 3 +should_find: + - id: dilution-transfer-limit + path: limits/limits.go + failureModePattern: limit|boundary|exclusive|reject|equal + titlePattern: limit|transfer|boundary|compar + - id: dilution-retry-off-by-one + path: limits/limits.go + failureModePattern: retry|attempt|off.by.one|extra|exceed + titlePattern: retry|attempt|maximum + - id: dilution-shard-range + path: limits/limits.go + failureModePattern: shard|range|index|modulo|out of + titlePattern: shard|index|range +should_not_find: + - id: dilution-capacity-control + path: limits/limits.go + titlePattern: AtCapacity diff --git a/evals/packet-dilution/plans/frozen.json b/evals/packet-dilution/plans/frozen.json new file mode 100644 index 0000000..3beaf25 --- /dev/null +++ b/evals/packet-dilution/plans/frozen.json @@ -0,0 +1,164 @@ +{ + "schemaVersion": 1, + "baseSha": "*", + "headSha": "*", + "planSha256": "52e62561b47bfa7ea5cdb59c7bf8a193a9183b01109c7ea644394f2d1f704dda", + "plan": { + "diffUnderstanding": { + "declaredIntent": "authored pinned plan for the packet-size recall curve", + "inferredBehavior": "uniform coverage so packet size is the only variable across arms" + }, + "coverage": [ + { + "hunkId": "d6df72ba", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "175c79f4", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "68109ebd", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "1ff3f41e", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "a8071418", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "53cf00c2", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "ffd09816", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "c9acf88b", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "18197703", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "bbacb370", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "a0be9fac", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "6ae88c80", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "b6e69463", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "d726dbb6", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + }, + { + "hunkId": "fdc5646d", + "path": "limits/limits.go", + "coverage": "normal", + "lenses": [ + "lang/go" + ], + "surroundingContextHints": [], + "reason": "authored uniform coverage" + } + ] + } +} diff --git a/scripts/draw-pinned-plan.ts b/scripts/draw-pinned-plan.ts new file mode 100644 index 0000000..a6c8df0 --- /dev/null +++ b/scripts/draw-pinned-plan.ts @@ -0,0 +1,60 @@ +#!/usr/bin/env tsx +// Plan 103 (experiment-only): write a PinnedPlanArtifact. Internal script, not +// a CLI verb — the pinned-plan seam is eval-only and adding a user-facing +// command would contradict that scope. +// +// --repo --base --head --output authored/derived from the diff +// --repo ... --coverage normal --lenses lang/go author a uniform plan (no model call) +import { writeFileSync } from "node:fs"; +import { parseDiff } from "../src/git/diff-parser.js"; +import { createGitClient } from "../src/git/git-client.js"; +import { buildPinnedPlanArtifact } from "../src/pipeline/pinned-plan.js"; +import type { CoverageLevel, ReviewPlan } from "../src/types.js"; + +function arg(name: string, fallback?: string): string | undefined { + const index = process.argv.indexOf(`--${name}`); + return index >= 0 ? process.argv[index + 1] : fallback; +} + +const repo = arg("repo"); +const base = arg("base", "main") ?? "main"; +const head = arg("head", "feature") ?? "feature"; +const output = arg("output"); +const coverage = (arg("coverage", "normal") ?? "normal") as Exclude; +const lenses = (arg("lenses", "lang/go") ?? "lang/go").split(","); +const anchor = arg("anchor", "content") ?? "content"; + +if (repo === undefined || output === undefined) { + console.error("usage: draw-pinned-plan.ts --repo [--base ref] [--head ref] [--coverage level] [--lenses a,b] --output "); + process.exit(2); +} + +const git = createGitClient(repo); +const rawDiff = await git.diff(base, head); +const diff = parseDiff(rawDiff); +const plan: ReviewPlan = { + diffUnderstanding: { + declaredIntent: "authored pinned plan for the packet-size recall curve", + inferredBehavior: "uniform coverage so packet size is the only variable across arms" + }, + coverage: diff.files.flatMap((file) => + file.hunks.map((hunk) => ({ + hunkId: hunk.id, + path: file.path, + coverage, + lenses: [...lenses], + surroundingContextHints: [], + reason: "authored uniform coverage" + })) + ) +}; + +// "*" marks a content-anchored artifact: fixture repos are materialized fresh +// per run so their SHAs vary, while content-derived hunk IDs do not. +const artifact = buildPinnedPlanArtifact({ + baseSha: anchor === "content" ? "*" : await git.revParse(base), + headSha: anchor === "content" ? "*" : await git.revParse(head), + plan +}); +writeFileSync(output, `${JSON.stringify(artifact, null, 2)}\n`); +console.log(JSON.stringify({ output, hunks: plan.coverage.length, planSha256: artifact.planSha256, anchor }, null, 1)); diff --git a/src/config/schema.ts b/src/config/schema.ts index afce254..46aef97 100644 --- a/src/config/schema.ts +++ b/src/config/schema.ts @@ -149,7 +149,8 @@ export const codegenieConfigSchema = z deepEnsemblePasses: positiveIntSchema.max(MAX_DEEP_ENSEMBLE_PASSES).optional(), adaptiveSecondPass: z.boolean().optional(), packCompatibleAtoms: z.boolean(), - packMaxHunks: positiveIntSchema.max(MAX_PACK_HUNKS) + packMaxHunks: positiveIntSchema.max(MAX_PACK_HUNKS), + pinnedPlanPath: z.string().min(1).optional() }) .strict(), github: z diff --git a/src/evals/eval-runner.ts b/src/evals/eval-runner.ts index 68e2592..8662040 100644 --- a/src/evals/eval-runner.ts +++ b/src/evals/eval-runner.ts @@ -143,6 +143,7 @@ const caseSchema = z // set these. No codegenie.toml may. packCompatibleAtoms: z.boolean().optional(), packMaxHunks: positiveIntSchema.max(MAX_PACK_HUNKS).optional(), + pinnedPlanPath: z.string().min(1).optional(), verify: z.boolean().optional(), cache: z.boolean().optional(), cacheDir: z.string().min(1).optional(), @@ -836,6 +837,9 @@ function applyCaseReviewConfig( if (review?.packMaxHunks !== undefined) { config.review.packMaxHunks = review.packMaxHunks; } + if (review?.pinnedPlanPath !== undefined) { + config.review.pinnedPlanPath = review.pinnedPlanPath; + } if (review?.verify !== undefined) { config.review.verify = review.verify; } diff --git a/src/pipeline/pinned-plan.ts b/src/pipeline/pinned-plan.ts new file mode 100644 index 0000000..8dc6dcb --- /dev/null +++ b/src/pipeline/pinned-plan.ts @@ -0,0 +1,112 @@ +// Plan 103 (experiment-only, removed at teardown): an eval-only seam that +// replays one recorded Stage-5 plan across several arms so packet size is the +// only difference between them. +// +// ReviewPlan carries no base/head identity and no self-describing hash, so a +// bare plan file cannot prove it belongs to the diff under review. The pinned +// artifact is an explicit versioned wrapper that can. +import { readFileSync } from "node:fs"; +import { z } from "zod"; +import { sha256Hex } from "../util/hashing.js"; +import { CodegenieError } from "../util/errors.js"; +import type { ReviewPlan, UnifiedDiff } from "../types.js"; + +export const PINNED_PLAN_SCHEMA_VERSION = 1; + +const pinnedPlanArtifactSchema = z + .object({ + schemaVersion: z.literal(PINNED_PLAN_SCHEMA_VERSION), + baseSha: z.string().min(1), + headSha: z.string().min(1), + planSha256: z.string().length(64), + plan: z.looseObject({ coverage: z.array(z.looseObject({ hunkId: z.string().min(1) })) }) + }) + .strict(); + +export type PinnedPlanArtifact = { + schemaVersion: typeof PINNED_PLAN_SCHEMA_VERSION; + baseSha: string; + headSha: string; + planSha256: string; + plan: ReviewPlan; +}; + +// Canonical form: keys sorted recursively, no whitespace, over `plan` alone — +// so the hash is independent of field order and of the wrapper's own fields. +export function canonicalPlanJson(plan: unknown): string { + const canonical = (value: unknown): unknown => { + if (Array.isArray(value)) { + return value.map(canonical); + } + if (value !== null && typeof value === "object") { + return Object.fromEntries( + Object.entries(value as Record) + .sort(([a], [b]) => a.localeCompare(b)) + .map(([key, entry]) => [key, canonical(entry)]) + ); + } + return value; + }; + return JSON.stringify(canonical(plan)); +} + +export function planSha256(plan: unknown): string { + return sha256Hex(canonicalPlanJson(plan)); +} + +export function buildPinnedPlanArtifact(input: { baseSha: string; headSha: string; plan: ReviewPlan }): PinnedPlanArtifact { + return { + schemaVersion: PINNED_PLAN_SCHEMA_VERSION, + baseSha: input.baseSha, + headSha: input.headSha, + planSha256: planSha256(input.plan), + plan: input.plan + }; +} + +function reject(reason: string, context: Record): never { + throw new CodegenieError("config_error", `pinned plan rejected: ${reason}`, { context }); +} + +// Fails closed on every mismatch. A hash match over a plan that no longer +// parses, or that targets a different diff, is not sufficient. +export function loadPinnedPlan( + filePath: string, + expected: { baseSha?: string; headSha?: string; diff?: UnifiedDiff } +): ReviewPlan { + let raw: unknown; + try { + raw = JSON.parse(readFileSync(filePath, "utf8")); + } catch (cause) { + reject("artifact is not readable JSON", { path: filePath, cause: cause instanceof Error ? cause.message : "unknown" }); + } + + const parsed = pinnedPlanArtifactSchema.safeParse(raw); + if (!parsed.success) { + reject("artifact does not match the pinned plan schema", { path: filePath, issues: parsed.error.issues.length }); + } + const artifact = parsed.data as unknown as PinnedPlanArtifact; + + const recomputed = planSha256(artifact.plan); + if (recomputed !== artifact.planSha256) { + reject("planSha256 does not match the canonical hash of the plan", { path: filePath, recorded: artifact.planSha256, recomputed }); + } + // Fixture-backed eval repos are materialized fresh per run, so their commit + // SHAs vary while content-derived hunk IDs do not. "*" records that the + // artifact is content-anchored rather than ref-anchored; hash, schema, and + // hunk-membership validation still apply in full. + if (artifact.baseSha !== "*" && expected.baseSha !== undefined && expected.baseSha !== artifact.baseSha) { + reject("baseSha does not match the resolved review target", { recorded: artifact.baseSha, resolved: expected.baseSha }); + } + if (artifact.headSha !== "*" && expected.headSha !== undefined && expected.headSha !== artifact.headSha) { + reject("headSha does not match the resolved review target", { recorded: artifact.headSha, resolved: expected.headSha }); + } + if (expected.diff !== undefined) { + const available = new Set(expected.diff.files.flatMap((file) => file.hunks.map((hunk) => hunk.id))); + const unknown = artifact.plan.coverage.filter((entry) => !available.has(entry.hunkId)).map((entry) => entry.hunkId); + if (unknown.length > 0) { + reject("plan references hunk ids absent from the current diff", { unknownHunkIds: unknown.slice(0, 5), unknownCount: unknown.length }); + } + } + return artifact.plan; +} diff --git a/src/pipeline/review-runner.ts b/src/pipeline/review-runner.ts index 38b07c8..b9f66ea 100644 --- a/src/pipeline/review-runner.ts +++ b/src/pipeline/review-runner.ts @@ -48,6 +48,8 @@ import type { import { CodegenieError, errorExitCode, isCodegenieError } from "../util/errors.js"; import { buildPlannerDossier, runPlanner } from "./planner.js"; import { buildReviewPackets, packetReviewContextFromDossier } from "./packet-builder.js"; +import { loadPinnedPlan } from "./pinned-plan.js"; +import type { PlannerRunResult } from "./planner.js"; import { ensemblePassesForPacket, runLensPackets } from "./lens-runner.js"; import { aggregateAttentionEfficiency, buildAttentionRecords } from "./attention.js"; import { applyCoverageEscalations } from "./coverage-escalation.js"; @@ -202,12 +204,43 @@ export async function runReview( lenses: services.lenses, allFiles: diff.files }); - const plannerResult = await runPlanner(dossier, config, run.telemetry, { - runner: services.runner, - promptBuilder: services.promptBuilder, - lenses: services.lenses, - skills: services.skills - }); + // Plan 103 eval-only seam: a pinned plan replaces the Stage-5 draw so + // several arms can consume byte-identical planner output. Validation fails + // closed; there is no user-facing path that reaches this. + const pinnedPlanPath = config.review.pinnedPlanPath; + let plannerResult: PlannerRunResult; + if (pinnedPlanPath !== undefined) { + const baseSha = resolved.mergeBase ?? resolved.baseRef; + const pinnedPlan = loadPinnedPlan(pinnedPlanPath, { + ...(baseSha !== undefined ? { baseSha } : {}), + ...(resolved.headSha !== undefined ? { headSha: resolved.headSha } : {}), + diff + }); + run.telemetry.event({ + stage: 5, + level: "info", + message: "planner_plan_pinned", + data: { path: pinnedPlanPath, coverageEntries: pinnedPlan.coverage.length } + }); + plannerResult = { + plan: pinnedPlan, + plannerCoverage: { + submittedEntries: pinnedPlan.coverage.length, + acceptedEntries: pinnedPlan.coverage.length, + acceptedUniqueHunks: new Set(pinnedPlan.coverage.map((entry) => entry.hunkId)).size, + rejectedUnknownHunk: 0 + }, + degradedPlanning: false, + chunked: false + }; + } else { + plannerResult = await runPlanner(dossier, config, run.telemetry, { + runner: services.runner, + promptBuilder: services.promptBuilder, + lenses: services.lenses, + skills: services.skills + }); + } throwIfHardAborted(run); const packets = applyCoverageEscalations( await buildReviewPackets(plannerResult.plan, kept, fileFacts, repoIndex, run.telemetry, { diff --git a/src/types.ts b/src/types.ts index ed7fce4..094b566 100644 --- a/src/types.ts +++ b/src/types.ts @@ -61,6 +61,9 @@ export type CodegenieConfig = { // supported user surface. Eval cases set them; nothing else can. packCompatibleAtoms: boolean; packMaxHunks: number; + // Plan 103 eval-only: replay one recorded Stage-5 plan across arms so + // packet size is the only difference between them. + pinnedPlanPath?: string; }; github: { summaryWhenNoFindings: boolean; @@ -1023,6 +1026,7 @@ export type EvalCase = { // these; removed at teardown. packCompatibleAtoms?: boolean; packMaxHunks?: number; + pinnedPlanPath?: string; verify?: boolean; cache?: boolean; cacheDir?: string; diff --git a/tests/pinned-plan.test.ts b/tests/pinned-plan.test.ts new file mode 100644 index 0000000..bb8e0c5 --- /dev/null +++ b/tests/pinned-plan.test.ts @@ -0,0 +1,79 @@ +import { mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { buildPinnedPlanArtifact, canonicalPlanJson, loadPinnedPlan, planSha256 } from "../src/pipeline/pinned-plan.js"; +import type { ReviewPlan, UnifiedDiff } from "../src/types.js"; + +const plan = (): ReviewPlan => ({ + diffUnderstanding: { declaredIntent: "intent", inferredBehavior: "behavior" }, + coverage: [ + { hunkId: "h1", path: "app.go", coverage: "normal", lenses: ["lang/go"], surroundingContextHints: [], reason: "test" } + ] +}); + +const diff = (hunkIds: string[]): UnifiedDiff => ({ + files: [ + { + path: "app.go", + status: "modified", + language: "go", + hunks: hunkIds.map((id) => ({ + id, + hunkHash: id.repeat(64).slice(0, 64), + path: "app.go", + oldStart: 1, + oldLines: 1, + newStart: 1, + newLines: 1, + header: "@@", + lines: [] + })) + } + ] +}); + +function write(artifact: unknown): string { + const dir = mkdtempSync(path.join(tmpdir(), "pinned-plan-")); + const file = path.join(dir, "frozen.json"); + writeFileSync(file, JSON.stringify(artifact)); + return file; +} + +describe("pinned plan artifact", () => { + it("hashes the plan independently of key order and wrapper fields", () => { + const a = { b: 1, a: [{ y: 2, x: 1 }] }; + const b = { a: [{ x: 1, y: 2 }], b: 1 }; + expect(canonicalPlanJson(a)).toBe(canonicalPlanJson(b)); + expect(planSha256(a)).toBe(planSha256(b)); + }); + + it("round-trips a valid artifact against its diff", () => { + const file = write(buildPinnedPlanArtifact({ baseSha: "base1", headSha: "head1", plan: plan() })); + const loaded = loadPinnedPlan(file, { baseSha: "base1", headSha: "head1", diff: diff(["h1"]) }); + expect(loaded.coverage[0]?.hunkId).toBe("h1"); + }); + + it("fails closed on every mismatch", () => { + const good = buildPinnedPlanArtifact({ baseSha: "base1", headSha: "head1", plan: plan() }); + + expect(() => loadPinnedPlan(write({ ...good, schemaVersion: 2 }), {})).toThrow(/pinned plan schema/); + expect(() => loadPinnedPlan(write({ ...good, extra: true }), {})).toThrow(/pinned plan schema/); + expect(() => loadPinnedPlan(write({ ...good, plan: { nope: true } }), {})).toThrow(/pinned plan schema/); + expect(() => loadPinnedPlan(write({ ...good, planSha256: "0".repeat(64) }), {})).toThrow(/canonical hash/); + + // A hash that matches a mutated plan is still rejected against its diff and refs. + const mutated = buildPinnedPlanArtifact({ + baseSha: "base1", + headSha: "head1", + plan: { ...plan(), coverage: [{ ...plan().coverage[0]!, hunkId: "ghost" }] } + }); + expect(() => loadPinnedPlan(write(mutated), { diff: diff(["h1"]) })).toThrow(/hunk ids absent/); + + expect(() => loadPinnedPlan(write(good), { baseSha: "other" })).toThrow(/baseSha/); + expect(() => loadPinnedPlan(write(good), { headSha: "other" })).toThrow(/headSha/); + + const badPath = path.join(mkdtempSync(path.join(tmpdir(), "pinned-plan-")), "missing.json"); + expect(() => loadPinnedPlan(badPath, {})).toThrow(/readable JSON/); + }); +}); From bc2a7c685fa954daa2579e04facf097f3d1edaeb Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 20:03:11 -0400 Subject: [PATCH 17/24] docs(evals): realism gate fails; curve not run (Plan 103 phase 4) Three unpinned cap-5 draws all placed every target in a three-hunk packet rather than five, and put all three targets in the same packet: draw 1 packet deep 3h starts: 59, 617, 806 draw 2 packet deep 3h starts: 59, 617, 806 draw 3 packet deep 3h starts: 59, 617, 806 The real planner grades exactly the three bug hunks deep and everything else normal or light, splitting the fifteen atoms across three coverage partitions. The authored plan's uniform normal coverage yields one partition and five-hunk packets; production yields a three-atom deep packet. The curve would have measured dilution at a shape production does not build for these hunks, and with three recall opportunities collapsed into one model conversation. Draws 1 and 3 are byte-identical, so this is deterministic planner behaviour rather than sampling variance. Cause is a fixture defect against this plan's own rule 3: each bug must be one the planner grades ordinarily, because a bug obvious enough to draw a deep grade lands in its own partition and never packs. Comparator-boundary bugs are exactly that obvious. This is the same structural finding the artifact-only diagnostics produced before implementation, now confirmed with real model calls. Per the pre-registered stop condition the fixture must be redesigned before any reviewer call. Gate cost $2.85 and prevented $27.36 of measurement at the wrong shape. Total spend $4.3634 of the $300 ceiling. Also records measured cost at $1.5154 per execution against the $0.78 projected, and three redesign options for the owner to choose between. Side evidence, not a gate result: across four paid executions nine of ten target opportunities were hit, with one loss at verification and none before candidate generation. Nothing so far suggests packing is harmful. Co-Authored-By: Claude Opus 5 (1M context) --- specs/plans/phase-plans/103/phase_4.md | 61 +++++++++++++++++++++++++- 1 file changed, 60 insertions(+), 1 deletion(-) diff --git a/specs/plans/phase-plans/103/phase_4.md b/specs/plans/phase-plans/103/phase_4.md index b9bf2cc..590ba3d 100644 --- a/specs/plans/phase-plans/103/phase_4.md +++ b/specs/plans/phase-plans/103/phase_4.md @@ -38,7 +38,66 @@ No two targets share a packet at any cap. Preserved at `packet-dilution/reports/ The initial invocation targeted the hunk at line 679, which is atom **11**, not atom 10. At cap 5 that places two targets inside `[11-15]`, collapsing two of the three recall opportunities into one packet and one model conversation — exactly the clustering defect the review flagged before implementation. The `targets_share_packet` check failed closed and the target was corrected to line 617. Had the proof not existed, the curve would have run at $14 with two of three observations correlated. -## Remaining before the first paid call +## Paid validation log + +| Step | Executions | Cost | Result | +| --- | ---: | ---: | --- | +| Matcher-calibration smoke, cap 1, repeat 1 | 1 | `$1.5154` | **pass, 4/4 expectations**, zero losses | +| Realism draws, three unpinned cap-5 runs | 3 | `$2.8480` | **gate failed 0/3** | + +**Total spend `$4.3634`** of the `$300` ceiling. The `$27.36` curve was not run. + +## The realism gate failed — stop condition reached + +All three unpinned draws placed every target in a **three**-hunk packet, not five, and put **all three targets in the same packet**: + +``` +draw 1 packet deep 3h starts: 59, 617, 806 +draw 2 packet deep 3h starts: 59, 617, 806 +draw 3 packet deep 3h starts: 59, 617, 806 +``` + +The real planner grades exactly the three bug hunks `deep` and everything else `normal`/`light`, so the fifteen atoms split across three coverage partitions. The authored plan's uniform `normal` coverage produces one partition and five-hunk packets; production produces three partitions and a three-atom deep packet. The curve would have measured dilution at a packet shape production does not build for these hunks. + +It also collapses the independence the fixture was designed for: three targets in one packet is one model conversation, not three, which is the clustering defect the treatment proof was written to prevent — reappearing through the planner rather than through my choice of line numbers. Draws 1 and 3 are byte-identical, so this is deterministic behaviour, not sampling variance. + +**Cause: the fixture violates this plan's own rule 3** — *each bug must be one the planner grades ordinarily; a bug obvious enough to draw a `deep` grade on its own lands in a different partition and never packs*. Comparator-boundary bugs are exactly that obvious. This is the same structural finding the artifact-only diagnostics produced before implementation, now confirmed live: the planner grades risky hunks differently from their safe siblings. + +Per the pre-registered stop condition, the fixture must be redesigned before any reviewer call. The gate cost `$2.85` and prevented `$27.36` of measurement at the wrong shape. + +### Three ways forward, for the owner to choose + +1. **Subtler bugs.** Rewrite the three defects so the planner grades them `normal`, putting them in the majority partition. Risk: a bug subtle enough to be graded ordinary may also be one the reviewer never finds, collapsing `R1` and voiding the curve from the other direction. +2. **Re-target the curve at the shape production builds.** Measure 1 → 2 → 3 rather than 1 → 3 → 5, since a three-atom deep packet is what these hunks actually produce. Needs more bug-bearing atoms so targets do not share a packet at the upper caps. +3. **Keep the authored plan and downgrade the claim.** Run the curve as a controlled isolation instrument and state explicitly that it measures dilution at a five-hunk shape the planner did not choose for this fixture, so it bounds harm rather than describing production. + +Option 2 is the most faithful to what the plan is trying to establish, and it is the cheapest to reach from here. + +### Side evidence, not a gate result + +The three unpinned draws are real cap-5 recall samples under a live planner: two found all three bugs, one lost the retry bug at verification (`lost-at-verification=1`, zero missed before candidate generation). The pinned cap-1 smoke run found all three. Nine of ten target opportunities were hit across four paid executions. This is not a controlled comparison and decides nothing, but nothing so far suggests packing is harmful. + +## Superseded plan + +**The smoke run replaced the planned fake-provider validation.** The fake runner emits a generic trigger-based finding, so it can only exercise wiring, not matcher semantics — and matcher semantics is exactly what killed three of Plan 102's executions. One real cap-1 execution answered the real question instead: all three bugs found, the `AtCapacity` control correctly not flagged, `missed-before-candidate-generation=0`, `lost-at-verification=0`, `partial-match=0`. No matcher calibration was needed. + +It also confirmed the pinned-plan seam end to end: `planner_plan_pinned` fired once, Stage 5 made no planner call, and Stage 6 produced exactly 15 packets at cap 1 — the treatment-proof shape, now verified in a live review rather than only model-free. + +**Cost is 2× the plan's projection.** Measured `$1.5154` per execution against the `$0.78` estimated from Plan 102's Go fixtures — this fixture is larger (15 hunks, 935 lines) and runs at `reasoning: high` on `claude-opus-4-8`. Revised projections: + +| Phase | Executions | At `$1.52` | +| --- | ---: | ---: | +| Smoke (spent) | 1 | `$1.52` | +| Realism draws | 3 | `$4.56` | +| Curve, 6 repeats × 3 arms | 18 | `$27.360` | +| Reserved extension, if triggered | 18 | `$27.36` | +| Reserved Void rerun, if triggered | 18 | `$27.36` | +| Production capacity pair | 2 | `$50` | +| **Worst case, all contingencies** | | **`$138`** | + +Comfortably inside the `$300` ceiling; the plan's `$119` reservation would have been tight had both contingencies fired. + +### Original remaining-work list, now superseded by the gate failure 1. **Step 6 — the pinned-plan seam.** Versioned `PinnedPlanArtifact` with `baseSha`, `headSha`, `planSha256` over a canonical sorted-key serialization; `review.pinnedPlanPath` validation failing closed on wrapper schema, plan schema, hash, base, head, and hunk-ID membership; `scripts/draw-pinned-plan.ts` as an internal script rather than a CLI verb; one test per failure mode. 2. **Eval case YAMLs** — `cap1.yml`, `cap3.yml`, `cap5.yml` at `repeat: 6`, cache off, `lang/go` only, identical except `packMaxHunks`, all pinned to one authored plan. From 426290dc2fe672bef3debb5fe93bdd5789497092 Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 20:15:55 -0400 Subject: [PATCH 18/24] docs(plans): abandon synthetic recall curve; ship Plan 103 dark for production A/B The money-path fixture tested a real hypothesis - that planner coverage tracks risk surface rather than detected defects, so a file where every function handles money would be graded uniformly - and falsified it. The planner located precisely the three defective functions and packed them into one deep packet, identically to the comparator-flip fixture: draw 1 packet deep 3h starts: 63, 462, 861 packet light 2h starts: 130, 995 packet normal 5h starts: 197,264,331,398,529 packet normal 5h starts: 596,663,730,797,928 Four fixture designs across two plans have now failed the same way: 102's cross-atom-consistency and dilution-control, and 103's limits and settlement. In every case the seeded defects were graded deep and their safe siblings normal or light, partitioning targets away from the hunks they were meant to pack with. The obstacle is structural. The compatibility predicate separates atoms by coverage; the planner assigns coverage by detected defect. A synthetic fixture therefore cannot manufacture a defect packed among safe siblings at equal coverage - any bug findable enough to measure recall against is visible enough to be graded deep. This is the harness working correctly and it is not solvable by better fixture craft. Step 8's curve is not_run by design; the four-fixture record is the finding. Owner decision: ship behind the flag and A/B on real pull requests instead. Recorded as a deliberate exception to this plan's no-dark-path teardown rule, with conditions - the flag stays false by default and eval-only, no codegenie.toml or user config can reach it, it is not a supported product surface, and it is either promoted to unconditional or deleted once the production A/B reports. It does not persist unreviewed. The rationale is that real PRs contain what fixtures cannot: hunks the planner grades ordinarily that nonetheless contain defects. That is where the benefit comes from - the planner issued coverage for 32 of 142 hunks on the motivating run, so the 21.9% reduction is overwhelmingly the packing of default-normal atoms it never singled out. What passed stands: phases 1-3 complete, free replay gate clearing every pre-registered threshold with zero model calls - 96 to 75 packets, fixed slot yield 89 to 109 against a 102 bar, estimator reconciling at exactly 89, zero deviation from Plan 102's frozen counts, and no coverage, profile, budget, lens, cap or hunk violations across four runs. Paid validation $5.3850 of a $300 ceiling; measured cost $1.5154 per execution against $0.78 projected. Steps 9-11 never authorized. Both fixture repositories retained as the record of why synthetic measurement was abandoned. 785 tests. Co-Authored-By: Claude Opus 5 (1M context) --- evals/packet-dilution/cap1.yml | 32 +- evals/packet-dilution/cap3.yml | 32 +- evals/packet-dilution/cap5.yml | 32 +- .../repos/settlement/base/go.mod | 3 + .../repos/settlement/base/settlement/fees.go | 973 ++++++++++++++++ .../repos/settlement/feature/go.mod | 3 + .../settlement/feature/settlement/fees.go | 1002 +++++++++++++++++ ...-103-relationship-driven-packet-packing.md | 18 + specs/plans/README.md | 2 +- specs/plans/phase-plans/103/phase_4.md | 48 +- 10 files changed, 2094 insertions(+), 51 deletions(-) create mode 100644 evals/packet-dilution/repos/settlement/base/go.mod create mode 100644 evals/packet-dilution/repos/settlement/base/settlement/fees.go create mode 100644 evals/packet-dilution/repos/settlement/feature/go.mod create mode 100644 evals/packet-dilution/repos/settlement/feature/settlement/fees.go diff --git a/evals/packet-dilution/cap1.yml b/evals/packet-dilution/cap1.yml index c761e47..5fbc640 100644 --- a/evals/packet-dilution/cap1.yml +++ b/evals/packet-dilution/cap1.yml @@ -1,7 +1,7 @@ name: packet-dilution-cap1 repeat: 6 repo: - fixture: repos/dilution + fixture: repos/settlement command: branch: feature base: main @@ -22,19 +22,19 @@ review: expect: maxCostUSD: 3 should_find: - - id: dilution-transfer-limit - path: limits/limits.go - failureModePattern: limit|boundary|exclusive|reject|equal - titlePattern: limit|transfer|boundary|compar - - id: dilution-retry-off-by-one - path: limits/limits.go - failureModePattern: retry|attempt|off.by.one|extra|exceed - titlePattern: retry|attempt|maximum - - id: dilution-shard-range - path: limits/limits.go - failureModePattern: shard|range|index|modulo|out of - titlePattern: shard|index|range + - id: dilution-split-remainder + path: settlement/fees.go + failureModePattern: remainder|dust|lost|unallocated|sum|drop + titlePattern: split|fee|remainder|share + - id: dilution-cut-order + path: settlement/fees.go + failureModePattern: cap|order|before|under|rate|incorrect + titlePattern: platform|cut|cap|rate + - id: dilution-refund-gross + path: settlement/fees.go + failureModePattern: refund|gross|net|fee|over.?pay|double + titlePattern: refund|gross|net should_not_find: - - id: dilution-capacity-control - path: limits/limits.go - titlePattern: AtCapacity + - id: dilution-rate-guard-control + path: settlement/fees.go + titlePattern: Rule[0-9]+ diff --git a/evals/packet-dilution/cap3.yml b/evals/packet-dilution/cap3.yml index 81890ad..5adb343 100644 --- a/evals/packet-dilution/cap3.yml +++ b/evals/packet-dilution/cap3.yml @@ -1,7 +1,7 @@ name: packet-dilution-cap3 repeat: 6 repo: - fixture: repos/dilution + fixture: repos/settlement command: branch: feature base: main @@ -22,19 +22,19 @@ review: expect: maxCostUSD: 3 should_find: - - id: dilution-transfer-limit - path: limits/limits.go - failureModePattern: limit|boundary|exclusive|reject|equal - titlePattern: limit|transfer|boundary|compar - - id: dilution-retry-off-by-one - path: limits/limits.go - failureModePattern: retry|attempt|off.by.one|extra|exceed - titlePattern: retry|attempt|maximum - - id: dilution-shard-range - path: limits/limits.go - failureModePattern: shard|range|index|modulo|out of - titlePattern: shard|index|range + - id: dilution-split-remainder + path: settlement/fees.go + failureModePattern: remainder|dust|lost|unallocated|sum|drop + titlePattern: split|fee|remainder|share + - id: dilution-cut-order + path: settlement/fees.go + failureModePattern: cap|order|before|under|rate|incorrect + titlePattern: platform|cut|cap|rate + - id: dilution-refund-gross + path: settlement/fees.go + failureModePattern: refund|gross|net|fee|over.?pay|double + titlePattern: refund|gross|net should_not_find: - - id: dilution-capacity-control - path: limits/limits.go - titlePattern: AtCapacity + - id: dilution-rate-guard-control + path: settlement/fees.go + titlePattern: Rule[0-9]+ diff --git a/evals/packet-dilution/cap5.yml b/evals/packet-dilution/cap5.yml index d3ad672..1f7d06a 100644 --- a/evals/packet-dilution/cap5.yml +++ b/evals/packet-dilution/cap5.yml @@ -1,7 +1,7 @@ name: packet-dilution-cap5 repeat: 6 repo: - fixture: repos/dilution + fixture: repos/settlement command: branch: feature base: main @@ -22,19 +22,19 @@ review: expect: maxCostUSD: 3 should_find: - - id: dilution-transfer-limit - path: limits/limits.go - failureModePattern: limit|boundary|exclusive|reject|equal - titlePattern: limit|transfer|boundary|compar - - id: dilution-retry-off-by-one - path: limits/limits.go - failureModePattern: retry|attempt|off.by.one|extra|exceed - titlePattern: retry|attempt|maximum - - id: dilution-shard-range - path: limits/limits.go - failureModePattern: shard|range|index|modulo|out of - titlePattern: shard|index|range + - id: dilution-split-remainder + path: settlement/fees.go + failureModePattern: remainder|dust|lost|unallocated|sum|drop + titlePattern: split|fee|remainder|share + - id: dilution-cut-order + path: settlement/fees.go + failureModePattern: cap|order|before|under|rate|incorrect + titlePattern: platform|cut|cap|rate + - id: dilution-refund-gross + path: settlement/fees.go + failureModePattern: refund|gross|net|fee|over.?pay|double + titlePattern: refund|gross|net should_not_find: - - id: dilution-capacity-control - path: limits/limits.go - titlePattern: AtCapacity + - id: dilution-rate-guard-control + path: settlement/fees.go + titlePattern: Rule[0-9]+ diff --git a/evals/packet-dilution/repos/settlement/base/go.mod b/evals/packet-dilution/repos/settlement/base/go.mod new file mode 100644 index 0000000..06cf459 --- /dev/null +++ b/evals/packet-dilution/repos/settlement/base/go.mod @@ -0,0 +1,3 @@ +module settlement + +go 1.23 diff --git a/evals/packet-dilution/repos/settlement/base/settlement/fees.go b/evals/packet-dilution/repos/settlement/base/settlement/fees.go new file mode 100644 index 0000000..92c7bef --- /dev/null +++ b/evals/packet-dilution/repos/settlement/base/settlement/fees.go @@ -0,0 +1,973 @@ +package settlement + +// Settlement rule 1. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// SplitFee divides fee across recipients, returning each share. +func SplitFee(fee int64, recipients int64) []int64 { + if recipients <= 0 { + return nil + } + share := fee / recipients + shares := make([]int64, 0, recipients) + var assigned int64 + for i := int64(0); i < recipients; i++ { + shares = append(shares, share) + assigned += share + } + if remainder := fee - assigned; remainder > 0 { + shares[0] += remainder + } + return shares +} + +// Settlement rule 2. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule2 settles component 2 of the fee ledger. +func Rule2(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 3. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule3 settles component 3 of the fee ledger. +func Rule3(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 4. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule4 settles component 4 of the fee ledger. +func Rule4(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 5. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule5 settles component 5 of the fee ledger. +func Rule5(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 6. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule6 settles component 6 of the fee ledger. +func Rule6(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 7. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// PlatformCut applies the platform rate, bounded by maxCut. +func PlatformCut(amount, rateBps, maxCut int64) int64 { + cut := amount * rateBps / 10_000 + if cut > maxCut { + return maxCut + } + return cut +} + +// Settlement rule 8. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule8 settles component 8 of the fee ledger. +func Rule8(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 9. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule9 settles component 9 of the fee ledger. +func Rule9(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 10. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule10 settles component 10 of the fee ledger. +func Rule10(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 11. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule11 settles component 11 of the fee ledger. +func Rule11(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 12. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule12 settles component 12 of the fee ledger. +func Rule12(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 13. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// RefundAmount returns what is owed back after fees are deducted. +func RefundAmount(gross, feesTaken int64) int64 { + net := gross - feesTaken + if net < 0 { + return 0 + } + return net +} + +// Settlement rule 14. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule14 settles component 14 of the fee ledger. +func Rule14(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 15. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule15 settles component 15 of the fee ledger. +func Rule15(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + return amount * rateBps / 10_000 +} diff --git a/evals/packet-dilution/repos/settlement/feature/go.mod b/evals/packet-dilution/repos/settlement/feature/go.mod new file mode 100644 index 0000000..06cf459 --- /dev/null +++ b/evals/packet-dilution/repos/settlement/feature/go.mod @@ -0,0 +1,3 @@ +module settlement + +go 1.23 diff --git a/evals/packet-dilution/repos/settlement/feature/settlement/fees.go b/evals/packet-dilution/repos/settlement/feature/settlement/fees.go new file mode 100644 index 0000000..ee2dc0a --- /dev/null +++ b/evals/packet-dilution/repos/settlement/feature/settlement/fees.go @@ -0,0 +1,1002 @@ +package settlement + +// Settlement rule 1. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// SplitFee divides fee across recipients, returning each share. +func SplitFee(fee int64, recipients int64) []int64 { + if recipients <= 0 { + return nil + } + share := fee / recipients + shares := make([]int64, 0, recipients) + for i := int64(0); i < recipients; i++ { + shares = append(shares, share) + } + return shares +} + +// Settlement rule 2. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule2 settles component 2 of the fee ledger. +func Rule2(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + if rateBps < 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 3. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule3 settles component 3 of the fee ledger. +func Rule3(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + if rateBps < 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 4. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule4 settles component 4 of the fee ledger. +func Rule4(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + if rateBps < 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 5. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule5 settles component 5 of the fee ledger. +func Rule5(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + if rateBps < 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 6. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule6 settles component 6 of the fee ledger. +func Rule6(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + if rateBps < 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 7. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// PlatformCut applies the platform rate, bounded by maxCut. +func PlatformCut(amount, rateBps, maxCut int64) int64 { + if amount > maxCut { + amount = maxCut + } + return amount * rateBps / 10_000 +} + +// Settlement rule 8. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule8 settles component 8 of the fee ledger. +func Rule8(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + if rateBps < 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 9. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule9 settles component 9 of the fee ledger. +func Rule9(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + if rateBps < 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 10. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule10 settles component 10 of the fee ledger. +func Rule10(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + if rateBps < 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 11. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule11 settles component 11 of the fee ledger. +func Rule11(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + if rateBps < 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 12. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule12 settles component 12 of the fee ledger. +func Rule12(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + if rateBps < 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 13. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// RefundAmount returns what is owed back after fees are deducted. +func RefundAmount(gross, feesTaken int64) int64 { + if gross < 0 { + return 0 + } + return gross +} + +// Settlement rule 14. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule14 settles component 14 of the fee ledger. +func Rule14(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + if rateBps < 0 { + return 0 + } + return amount * rateBps / 10_000 +} + +// Settlement rule 15. See docs/fees.md for the ledger contract. +// invariant note 1: amounts are minor units; never float. +// invariant note 2: amounts are minor units; never float. +// invariant note 3: amounts are minor units; never float. +// invariant note 4: amounts are minor units; never float. +// invariant note 5: amounts are minor units; never float. +// invariant note 6: amounts are minor units; never float. +// invariant note 7: amounts are minor units; never float. +// invariant note 8: amounts are minor units; never float. +// invariant note 9: amounts are minor units; never float. +// invariant note 10: amounts are minor units; never float. +// invariant note 11: amounts are minor units; never float. +// invariant note 12: amounts are minor units; never float. +// invariant note 13: amounts are minor units; never float. +// invariant note 14: amounts are minor units; never float. +// invariant note 15: amounts are minor units; never float. +// invariant note 16: amounts are minor units; never float. +// invariant note 17: amounts are minor units; never float. +// invariant note 18: amounts are minor units; never float. +// invariant note 19: amounts are minor units; never float. +// invariant note 20: amounts are minor units; never float. +// invariant note 21: amounts are minor units; never float. +// invariant note 22: amounts are minor units; never float. +// invariant note 23: amounts are minor units; never float. +// invariant note 24: amounts are minor units; never float. +// invariant note 25: amounts are minor units; never float. +// invariant note 26: amounts are minor units; never float. +// invariant note 27: amounts are minor units; never float. +// invariant note 28: amounts are minor units; never float. +// invariant note 29: amounts are minor units; never float. +// invariant note 30: amounts are minor units; never float. +// invariant note 31: amounts are minor units; never float. +// invariant note 32: amounts are minor units; never float. +// invariant note 33: amounts are minor units; never float. +// invariant note 34: amounts are minor units; never float. +// invariant note 35: amounts are minor units; never float. +// invariant note 36: amounts are minor units; never float. +// invariant note 37: amounts are minor units; never float. +// invariant note 38: amounts are minor units; never float. +// invariant note 39: amounts are minor units; never float. +// invariant note 40: amounts are minor units; never float. +// invariant note 41: amounts are minor units; never float. +// invariant note 42: amounts are minor units; never float. +// invariant note 43: amounts are minor units; never float. +// invariant note 44: amounts are minor units; never float. +// invariant note 45: amounts are minor units; never float. +// invariant note 46: amounts are minor units; never float. +// invariant note 47: amounts are minor units; never float. +// invariant note 48: amounts are minor units; never float. +// invariant note 49: amounts are minor units; never float. +// invariant note 50: amounts are minor units; never float. +// invariant note 51: amounts are minor units; never float. +// invariant note 52: amounts are minor units; never float. +// invariant note 53: amounts are minor units; never float. +// invariant note 54: amounts are minor units; never float. + +// Rule15 settles component 15 of the fee ledger. +func Rule15(amount, rateBps int64) int64 { + if amount <= 0 { + return 0 + } + if rateBps < 0 { + return 0 + } + return amount * rateBps / 10_000 +} diff --git a/specs/plans/103-issue-103-relationship-driven-packet-packing.md b/specs/plans/103-issue-103-relationship-driven-packet-packing.md index 8a473b9..d4c3091 100644 --- a/specs/plans/103-issue-103-relationship-driven-packet-packing.md +++ b/specs/plans/103-issue-103-relationship-driven-packet-packing.md @@ -7,6 +7,24 @@ Production replay refs: base/merge-base `d1c49bdf6a8002ec2ec27faac94a932d736532b Planned at: commit `32d7b83` (branch `next`) Recommended priority: after Plan 101's paid A/B settles. +## Reconciliation (2026-07-25) + +Status: **implemented and shipped dark; synthetic recall curve abandoned on structural grounds; production A/B authorized in its place.** + +**What passed.** Phases 1–3 are complete and committed. The free replay gate cleared every pre-registered threshold on real production diffs, with zero model calls: `dca8d870` 96→75 packets (21.9%), fixed-slot hunk yield 89→**109** at 56 slots against a ≥102 bar, estimator reconciling at exactly 89, deviation from Plan 102's frozen counts of **0**, and zero coverage changes, profile or budget downgrades, lens drops, cap breaches or hunk loss. `81f806a6` and `740d73f2` reduce 26.9% and 25.8%. Report preserved at `packet-dilution/reports/plan103-replay.json`. + +**What did not.** Step 8's synthetic recall curve never ran. Its realism gate failed on two independent fixture designs, and Plan 102 failed the same way on two more. In all four the planner graded the seeded defects `deep` and their safe siblings `normal`/`light`, partitioning the targets away from the hunks they were meant to pack with. + +**The conclusion is structural, not a fixture-craft failure.** The compatibility predicate separates atoms by coverage; the planner assigns coverage by detected defect. A synthetic fixture therefore cannot manufacture "a defect packed among safe siblings at equal coverage" — any bug findable enough to measure recall against is also visible enough to be graded `deep`. Making the second fixture uniformly money-critical tested and falsified the alternative hypothesis that coverage tracks risk surface rather than detected defects. + +**Owner decision, 2026-07-25: ship behind the flag and A/B on real pull requests instead.** This is a deliberate, recorded exception to this plan's teardown rule that no dark path survives the decision, taken because real PRs contain the population synthetic fixtures cannot: hunks the planner grades ordinarily that nonetheless contain defects. That population is where packing's benefit comes from — on the motivating run the planner issued coverage for 32 of 142 hunks, so the 21.9% reduction is overwhelmingly the packing of default-`normal` atoms the planner never singled out. A production A/B observes the treated population directly rather than by construction. + +**Conditions attached to the exception.** `review.packCompatibleAtoms` stays `false` by default and eval-only — no `codegenie.toml` and no user config can reach it. It is not a supported product surface, and the plan's Non-Goals still forbid it becoming one. **Review date: the flag is either promoted to unconditional behaviour or deleted once the production A/B reports; it does not persist unreviewed.** If it is deleted, the teardown branch of step 9 applies unchanged. + +**Paid validation: `$5.3850`** of an owner-approved `$300` ceiling. Measured cost was `$1.5154` per execution against `$0.78` projected. Steps 8–11 are `not_run`: step 8 abandoned for the reason above, steps 9–11 never authorized because the gate preceding them never passed. + +**Preserved evidence**, `packet-dilution/reports/`: `plan103-replay.json`, `plan103-treatment-proof.json`, `plan103-affinity-pairs.json`, `plan103-cap-sweep.json`, `plan103-ordering-comparison.json`, each hashed in `manifest.sha256`. Plan 102's manifest is unmodified. Both fixture repositories are retained under `evals/packet-dilution/repos/` as the record of why synthetic measurement was abandoned. + ## Introduction and TL;DR ### The problem diff --git a/specs/plans/README.md b/specs/plans/README.md index 7137201..8d35400 100644 --- a/specs/plans/README.md +++ b/specs/plans/README.md @@ -106,4 +106,4 @@ This directory tracks implementation plans for confirmed improvements. Status va | 100 | COMPLETE | Issue 100: Short Hunk IDs — Planner Coverage Survival and Dispatch Resilience | | 101 | IN PROGRESS (paid semantic A/B pending) | Issue 101: Exact Skill Provenance and Evidence-Gated Bundled-Skill Revision | | 102 | COMPLETE (failed gate; baseline restored) | Issue 102: Same-File Packet Packing | -| 103 | PENDING | Issue 103: Compatible-Atom Packet Packing and the Packet-Size Recall Curve | +| 103 | IMPLEMENTED (dark; production A/B pending) | Issue 103: Compatible-Atom Packet Packing and the Packet-Size Recall Curve | diff --git a/specs/plans/phase-plans/103/phase_4.md b/specs/plans/phase-plans/103/phase_4.md index 590ba3d..5c60ebd 100644 --- a/specs/plans/phase-plans/103/phase_4.md +++ b/specs/plans/phase-plans/103/phase_4.md @@ -1,5 +1,5 @@ --- -status: in progress (preconditions complete; no paid call yet) +status: complete (synthetic curve abandoned; see outcome) --- # Phase 4: Paid Validation @@ -77,7 +77,51 @@ Option 2 is the most faithful to what the plan is trying to establish, and it is The three unpinned draws are real cap-5 recall samples under a live planner: two found all three bugs, one lost the retry bug at verification (`lost-at-verification=1`, zero missed before candidate generation). The pinned cap-1 smoke run found all three. Nine of ten target opportunities were hit across four paid executions. This is not a controlled comparison and decides nothing, but nothing so far suggests packing is harmful. -## Superseded plan +## Second fixture, same wall — synthetic measurement abandoned + +The money-path redesign tested a real hypothesis: that the planner's coverage decision tracks **risk surface** rather than **detected defects**, so a file where every function handles money in minor units would be graded uniformly. Fifteen fee/settlement functions, three carrying money-semantic defects (a dropped division remainder in `SplitFee`, a cap applied before the rate in `PlatformCut`, a refund on gross rather than net in `RefundAmount`) at atoms 1, 7 and 13 — spacing that keeps targets in separate packets at every cap, which the previous layout only managed at caps 1 and 3. + +The model-free proof passed: 15/5/3 packets, three distinct target packets at each cap. The realism gate then failed identically to the first fixture: + +``` +draw 1 packet deep 3h starts: 63, 462, 861 <- the three defects, one packet + packet light 2h starts: 130, 995 + packet normal 5h starts: 197,264,331,398,529 + packet normal 5h starts: 596,663,730,797,928 +``` + +Planner entries `deep:3, light:2`. It located precisely the three defective functions. Making every function money-critical did not disguise them; it grouped them more cleanly. + +**The hypothesis is falsified.** Coverage tracks detected defects, not just risk surface. + +### The obstacle is structural + +Four fixture designs across two plans have now failed the same way: + +| Fixture | Design | Outcome | +| --- | --- | --- | +| 102 `cross-atom-consistency` | missing guard among siblings | target `deep`, siblings `light`; 0/1 treated | +| 102 `dilution-control` | boundary bug among safe changes | planner variance invalidated the comparison | +| 103 `limits` | comparator flips among range guards | three bugs, one `deep` packet; 0/3 | +| 103 `settlement` | money-semantic bugs in a uniformly money-critical file | three bugs, one `deep` packet; gate failed | + +A synthetic fixture cannot produce the condition the curve requires — a defect packed alongside safe siblings at equal coverage — because the compatibility predicate separates them by construction. Any bug made findable enough to measure is also visible enough to be graded `deep`. This is the harness working correctly, and it is not solvable by better fixture craft. + +**Step 8's synthetic recall curve is `not_run`, by design rather than by omission.** The four-fixture record is the finding: the curve is the wrong instrument for this question. + +## Outcome + +Preconditions complete, infrastructure shipped and tested, synthetic curve abandoned on structural grounds. + +**Total paid validation: `$5.3850`** of the `$300` ceiling — one calibration smoke run (`$1.5154`), three first-fixture realism draws (`$2.8480`), one second-fixture realism draw (`$1.0216`). Nothing spent on the curve. + +Phases 8 through 11 of the plan are `not_run`. Steps 9 (collateral) and 10 (production capacity pair) were never reached because the gate that authorizes them never passed. + +### What the paid runs did establish + +Four executions produced findings against these fixtures, and nine of ten target opportunities were hit — three of three at cap 1 pinned, three of three and two of three and three of three across the unpinned cap-5 draws — with the single loss occurring at verification rather than before candidate generation, and `partial-match=0` throughout. This is not a controlled comparison and decides nothing about dilution. It does mean nothing observed suggests packing is harmful. + +The pinned-plan seam was also confirmed live: `planner_plan_pinned` fired, Stage 5 made no planner call, and Stage 6 produced exactly the treatment-proof shape. **The smoke run replaced the planned fake-provider validation.** The fake runner emits a generic trigger-based finding, so it can only exercise wiring, not matcher semantics — and matcher semantics is exactly what killed three of Plan 102's executions. One real cap-1 execution answered the real question instead: all three bugs found, the `AtCapacity` control correctly not flagged, `missed-before-candidate-generation=0`, `lost-at-verification=0`, `partial-match=0`. No matcher calibration was needed. From 39d9b85112361ef29cf40beee2a3a644f44ede8f Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 20:23:36 -0400 Subject: [PATCH 19/24] feat(cli): rename to packRelatedHunks and add a per-run CLI toggle Renames review.packCompatibleAtoms to review.packRelatedHunks across source, tests, eval cases, the report script, and the plan documents - 37 occurrences, no behaviour change. Adds --pack-related-hunks and --no-pack-related-hunks to the review command, following the repo's kebab-case convention, wired through CliConfigOverrides and applyCliOverrides with "cli" config-source attribution. The setting is now reachable from exactly two places: an eval case, and this per-run CLI flag. It stays absent from rawConfigSchema and REPO_SAFE_REVIEW_KEYS, so no codegenie.toml and no user config.toml can set it - a repository cannot enable experimental packing for everyone who reviews it. A test asserts a codegenie.toml setting it still throws invalid config file while the CLI flag succeeds and is attributed to cli. This brings src/cli into scope, which Plan 103 originally excluded. That exclusion was written when the setting had no product consumer; the owner-approved production A/B is one, and toggling per run is the whole point of it. Recorded in the plan's reconciliation note. 786 tests. Co-Authored-By: Claude Opus 5 (1M context) --- evals/packet-dilution/cap1.yml | 2 +- evals/packet-dilution/cap3.yml | 2 +- evals/packet-dilution/cap5.yml | 2 +- scripts/packet-packing-report.ts | 2 +- ...-103-relationship-driven-packet-packing.md | 6 ++--- specs/plans/phase-plans/103/phase_1.md | 6 ++--- src/cli/review-command.ts | 9 +++++++ src/config/config-loader.ts | 6 +++++ src/config/schema.ts | 4 ++-- src/evals/eval-runner.ts | 6 ++--- src/pipeline/packet-builder.ts | 6 ++--- src/types.ts | 4 ++-- tests/config-loader.test.ts | 24 ++++++++++++++++--- tests/evals.test.ts | 4 ++-- tests/pipeline-phase5.test.ts | 24 +++++++++---------- 15 files changed, 70 insertions(+), 37 deletions(-) diff --git a/evals/packet-dilution/cap1.yml b/evals/packet-dilution/cap1.yml index 5fbc640..5baa5ba 100644 --- a/evals/packet-dilution/cap1.yml +++ b/evals/packet-dilution/cap1.yml @@ -16,7 +16,7 @@ review: maxTimeMinutes: 20 lenses: - lang/go - packCompatibleAtoms: true + packRelatedHunks: true packMaxHunks: 1 pinnedPlanPath: evals/packet-dilution/plans/frozen.json expect: diff --git a/evals/packet-dilution/cap3.yml b/evals/packet-dilution/cap3.yml index 5adb343..ef68ed1 100644 --- a/evals/packet-dilution/cap3.yml +++ b/evals/packet-dilution/cap3.yml @@ -16,7 +16,7 @@ review: maxTimeMinutes: 20 lenses: - lang/go - packCompatibleAtoms: true + packRelatedHunks: true packMaxHunks: 3 pinnedPlanPath: evals/packet-dilution/plans/frozen.json expect: diff --git a/evals/packet-dilution/cap5.yml b/evals/packet-dilution/cap5.yml index 1f7d06a..3413ad0 100644 --- a/evals/packet-dilution/cap5.yml +++ b/evals/packet-dilution/cap5.yml @@ -16,7 +16,7 @@ review: maxTimeMinutes: 20 lenses: - lang/go - packCompatibleAtoms: true + packRelatedHunks: true packMaxHunks: 5 pinnedPlanPath: evals/packet-dilution/plans/frozen.json expect: diff --git a/scripts/packet-packing-report.ts b/scripts/packet-packing-report.ts index d075a82..ae5ffe8 100644 --- a/scripts/packet-packing-report.ts +++ b/scripts/packet-packing-report.ts @@ -82,7 +82,7 @@ function packingConfig(repoRoot: string, on: boolean): CodegenieConfig { const base = structuredClone(defaultConfig) as CodegenieConfig; base.telemetry.enabled = false; const config = applyRepoConfigLayer(base, repoRoot).config; - config.review.packCompatibleAtoms = on; + config.review.packRelatedHunks = on; return config; } diff --git a/specs/plans/103-issue-103-relationship-driven-packet-packing.md b/specs/plans/103-issue-103-relationship-driven-packet-packing.md index d4c3091..52abb17 100644 --- a/specs/plans/103-issue-103-relationship-driven-packet-packing.md +++ b/specs/plans/103-issue-103-relationship-driven-packet-packing.md @@ -19,7 +19,7 @@ Status: **implemented and shipped dark; synthetic recall curve abandoned on stru **Owner decision, 2026-07-25: ship behind the flag and A/B on real pull requests instead.** This is a deliberate, recorded exception to this plan's teardown rule that no dark path survives the decision, taken because real PRs contain the population synthetic fixtures cannot: hunks the planner grades ordinarily that nonetheless contain defects. That population is where packing's benefit comes from — on the motivating run the planner issued coverage for 32 of 142 hunks, so the 21.9% reduction is overwhelmingly the packing of default-`normal` atoms the planner never singled out. A production A/B observes the treated population directly rather than by construction. -**Conditions attached to the exception.** `review.packCompatibleAtoms` stays `false` by default and eval-only — no `codegenie.toml` and no user config can reach it. It is not a supported product surface, and the plan's Non-Goals still forbid it becoming one. **Review date: the flag is either promoted to unconditional behaviour or deleted once the production A/B reports; it does not persist unreviewed.** If it is deleted, the teardown branch of step 9 applies unchanged. +**Conditions attached to the exception.** `review.packRelatedHunks` stays `false` by default. It is reachable from exactly two places: an eval case, and the per-run CLI flag `--pack-related-hunks` / `--no-pack-related-hunks` added for this A/B. It remains absent from `rawConfigSchema` and repo-safe filtering, so **no `codegenie.toml` and no user `config.toml` can set it** — a repository cannot enable experimental packing for everyone who reviews it. Adding the CLI flag brings `src/cli/` into scope, which this plan originally excluded; that exclusion was written when the setting had no product consumer, and the approved production A/B is one. It is not a supported product surface, and the plan's Non-Goals still forbid it becoming one. **Review date: the flag is either promoted to unconditional behaviour or deleted once the production A/B reports; it does not persist unreviewed.** If it is deleted, the teardown branch of step 9 applies unchanged. **Paid validation: `$5.3850`** of an owner-approved `$300` ceiling. Measured cost was `$1.5154` per execution against `$0.78` projected. Steps 8–11 are `not_run`: step 8 abandoned for the reason above, steps 9–11 never authorized because the gate preceding them never passed. @@ -217,7 +217,7 @@ Plan 102's floor verbatim: `max(derivedPackedProfile, max(standalone member prof ```ts // CodegenieConfig["review"], resolved schema and defaults only -packCompatibleAtoms: boolean; // default false +packRelatedHunks: boolean; // default false packMaxHunks: number; // default 5, never exceeds MAX_HUNKS_PER_PACKET in shipped behavior ``` @@ -415,7 +415,7 @@ The two reserved contingencies are the *only* reruns this plan authorizes; a sec A *first* Void or a *first* monotone decline does not reach this step: each authorizes exactly one bounded rerun under phase B's preregistered limits, and only its result reaches teardown. - **Verify:** `rg -n "packCompatibleAtoms|packMaxHunks" src scripts tests evals` → exit 1; both manifests verify; the note records the decision, actual spend, and the resolution limit of 18 observations per arm. + **Verify:** `rg -n "packRelatedHunks|packMaxHunks" src scripts tests evals` → exit 1; both manifests verify; the note records the decision, actual spend, and the resolution limit of 18 observations per arm. 10. Run the complete repository gate. **Verify:** `pnpm run check && pnpm test && pnpm build` → exit 0. diff --git a/specs/plans/phase-plans/103/phase_1.md b/specs/plans/phase-plans/103/phase_1.md index 7eaa9fa..c3a59b7 100644 --- a/specs/plans/phase-plans/103/phase_1.md +++ b/specs/plans/phase-plans/103/phase_1.md @@ -12,7 +12,7 @@ Multi-member symbol context, transactional rejection, the profile floor, the pin ## Steps -1. Add `review.packCompatibleAtoms: boolean` (default `false`) and `review.packMaxHunks: number` (default `5`, rejected above `MAX_HUNKS_PER_PACKET`) to `CodegenieConfig`, `codegenieConfigSchema`, and `defaultConfig`. Add both to the strict eval-case `review` schema and `applyCaseReviewConfig()`. Do **not** add either to `rawConfigSchema`, `DEFAULT_SOURCE_PATHS`, or `REPO_SAFE_REVIEW_KEYS`: no `codegenie.toml` may set them and no config-source attribution is claimed for a value that cannot come from a config file. Add `evals/packet-dilution/logs/` and the currently missing `evals/skill-semantics/logs/` to `.gitignore`. +1. Add `review.packRelatedHunks: boolean` (default `false`) and `review.packMaxHunks: number` (default `5`, rejected above `MAX_HUNKS_PER_PACKET`) to `CodegenieConfig`, `codegenieConfigSchema`, and `defaultConfig`. Add both to the strict eval-case `review` schema and `applyCaseReviewConfig()`. Do **not** add either to `rawConfigSchema` or `REPO_SAFE_REVIEW_KEYS`: no `codegenie.toml` may set them and no config-source attribution is claimed for a value that cannot come from a config file. Add `evals/packet-dilution/logs/` and the currently missing `evals/skill-semantics/logs/` to `.gitignore`. 2. Refactor `hunkFirstGroups()` results in `src/pipeline/packet-builder.ts` into explicit packet atoms without changing their membership. Each atom carries its ordered `PlannedHunk[]`, hunk count, `combinedPatchChars()` size, first source position, effective coverage, normalized planner lens signature (stable serialization of the sorted deduplicated union of `decision.lenses`), and a stable ID derived from ordered hunk IDs. **Scope correction:** the plan's atom description also lists standalone review profile and standalone per-member context quality. Both require the dry-build machinery that Phase 2 owns, and neither is needed to partition or fill, so they are captured in Phase 2 alongside the transactional evaluation that computes them rather than half-built here. Preserve the direct `whole-file` and content-probed `file-diff` returns from `groupHunks()` so they bypass the packer entirely, and bypass all atom work when the flag is off so packet artifacts, IDs, ordering, profiles, context, lenses, and budgets stay byte-identical. 3. Implement the compatibility partition and fill. Partition each file's atoms by `(effectiveCoverage, normalizedPlannerLensSignature)`; fill greedily in source order under `packMaxHunks` and `MAX_PATCH_CHARS`; order packets by earliest member hunk and render each packet's hunks by file position. Materialize combined groups through the existing `packetGroup()`/`packetKind()` rules, carry non-empty `degradationReason` values through a sorted deduplicated `"; "` join, do not synthesize `wholeFileText` or `fileContext`, and recompute `dispatchRank` with the unchanged `packetDispatchRank(filePath, facts, combinedChangedLines)` formula. 4. Add focused coverage in `tests/config-loader.test.ts`, `tests/evals.test.ts`, and `tests/pipeline-phase5.test.ts`. @@ -38,9 +38,9 @@ pnpm run check && pnpm test && pnpm build Complete. `pnpm run check`, `pnpm test`, and `pnpm build` all pass; the suite grew from 761 to 773 tests with no existing test modified. -**Config surface.** `review.packCompatibleAtoms` (default `false`) and `review.packMaxHunks` (default `5`) exist in `CodegenieConfig`, `codegenieConfigSchema`, `defaultConfig`, the strict eval-case schema, and `applyCaseReviewConfig()`. They are absent from `rawConfigSchema`, `DEFAULT_SOURCE_PATHS`, and `REPO_SAFE_REVIEW_KEYS`, and tests prove a `codegenie.toml` or user `config.toml` setting either key throws `invalid config file` rather than being silently filtered. `MAX_PACK_HUNKS` is duplicated in `schema.ts` rather than imported from `packet-builder.ts` to keep the config schema free of pipeline dependencies; a test asserts it equals 5 and the resolved schema rejects 6. +**Config surface.** `review.packRelatedHunks` (default `false`) and `review.packMaxHunks` (default `5`) exist in `CodegenieConfig`, `codegenieConfigSchema`, `defaultConfig`, the strict eval-case schema, and `applyCaseReviewConfig()`. They are absent from `rawConfigSchema`, `DEFAULT_SOURCE_PATHS`, and `REPO_SAFE_REVIEW_KEYS`, and tests prove a `codegenie.toml` or user `config.toml` setting either key throws `invalid config file` rather than being silently filtered. `MAX_PACK_HUNKS` is duplicated in `schema.ts` rather than imported from `packet-builder.ts` to keep the config schema free of pipeline dependencies; a test asserts it equals 5 and the resolved schema rejects 6. -**Packer.** `PacketGroup` gained a required `origin` field so the direct whole-file and content-probed file-diff returns are structurally excluded from packing rather than inferred from `kind`, which is ambiguous — `packetGroup()` can also produce `file-diff`. `packCompatibleAtoms()` partitions a file's atoms by `(effectiveCoverage, normalizedLensSignature)`, fills sequentially in source order under `packMaxHunks` and `MAX_PATCH_CHARS`, orders packets by earliest member, and rebuilds combined groups through the existing `packetGroup()`/`packetKind()` rules with degradation reasons merged by sorted deduplicated join. +**Packer.** `PacketGroup` gained a required `origin` field so the direct whole-file and content-probed file-diff returns are structurally excluded from packing rather than inferred from `kind`, which is ambiguous — `packetGroup()` can also produce `file-diff`. `packRelatedHunks()` partitions a file's atoms by `(effectiveCoverage, normalizedLensSignature)`, fills sequentially in source order under `packMaxHunks` and `MAX_PATCH_CHARS`, orders packets by earliest member, and rebuilds combined groups through the existing `packetGroup()`/`packetKind()` rules with degradation reasons merged by sorted deduplicated join. Fill is sequential-flush, matching Plan 102's described algorithm, not the best-fit search used in the planning simulation. Sequential preserves strict source order; best-fit would let a later small atom jump ahead of an earlier larger one. The two differ by about one packet on the retained runs, inside the documented residual, and Phase 3's replay measures the real number. diff --git a/src/cli/review-command.ts b/src/cli/review-command.ts index 21a1ea2..2937d44 100644 --- a/src/cli/review-command.ts +++ b/src/cli/review-command.ts @@ -54,6 +54,7 @@ type CommanderReviewOptions = { cache?: boolean; ci?: boolean; progress?: boolean; + packRelatedHunks?: boolean; }; export function parseReviewCommand( @@ -97,6 +98,11 @@ export function parseReviewCommand( .option("--no-progress", "disable the interactive progress spinner") .option("--cache", "enable local model-call cache for this run; provider prompt caching is reported separately") .option("--no-cache", "disable local model-call cache for this run; provider prompt caching is reported separately") + // Plan 103: per-run toggle for the packing A/B. Deliberately CLI-only — + // no codegenie.toml or user config may set it, so a repository cannot + // enable experimental packing for everyone who reviews it. + .option("--pack-related-hunks", "experimental: pack related same-file hunks into fewer review packets") + .option("--no-pack-related-hunks", "experimental: force related-hunk packing off for this run") .action((commitArgs: string[], options: CommanderReviewOptions) => { commits = commitArgs; commandOptions = options; @@ -285,6 +291,9 @@ function buildCliOverrides(options: CommanderReviewOptions): CliConfigOverrides if (options.depth !== undefined) { cli.depth = parseDepth(options.depth); } + if (options.packRelatedHunks !== undefined) { + cli.packRelatedHunks = options.packRelatedHunks; + } if (options.budgetBoost !== undefined) { cli.budgetBoost = parseBudgetBoost(options.budgetBoost); } diff --git a/src/config/config-loader.ts b/src/config/config-loader.ts index bb8d2de..53eda08 100644 --- a/src/config/config-loader.ts +++ b/src/config/config-loader.ts @@ -32,6 +32,7 @@ export type CliConfigOverrides = { model?: string; reasoning?: ReasoningLevel | "auto"; cacheEnabled?: boolean; + packRelatedHunks?: boolean; }; export type LoadConfigOptions = { @@ -64,6 +65,7 @@ const DEFAULT_SOURCE_PATHS = [ "review.maxTime", "review.perPassTimeoutMs", "review.budgetBoost", + "review.packRelatedHunks", "review.maxBudgetTokens", "github.summaryWhenNoFindings", "classification.pathRules", @@ -469,6 +471,10 @@ function applyCliOverrides( config.review.depth = cli.depth; sources["review.depth"] = "cli"; } + if (cli.packRelatedHunks !== undefined) { + config.review.packRelatedHunks = cli.packRelatedHunks; + sources["review.packRelatedHunks"] = "cli"; + } if (cli.budgetBoost !== undefined) { config.review.budgetBoost = cli.budgetBoost; sources["review.budgetBoost"] = "cli"; diff --git a/src/config/schema.ts b/src/config/schema.ts index 46aef97..82f5bbc 100644 --- a/src/config/schema.ts +++ b/src/config/schema.ts @@ -148,7 +148,7 @@ export const codegenieConfigSchema = z maxModelCalls: positiveIntSchema.optional(), deepEnsemblePasses: positiveIntSchema.max(MAX_DEEP_ENSEMBLE_PASSES).optional(), adaptiveSecondPass: z.boolean().optional(), - packCompatibleAtoms: z.boolean(), + packRelatedHunks: z.boolean(), packMaxHunks: positiveIntSchema.max(MAX_PACK_HUNKS), pinnedPlanPath: z.string().min(1).optional() }) @@ -227,7 +227,7 @@ export const defaultConfig: CodegenieConfig = { // target. maxBudgetTokens: 8_000_000, // Plan 103: dark until the packet-size recall curve clears its gate. - packCompatibleAtoms: false, + packRelatedHunks: false, packMaxHunks: MAX_PACK_HUNKS }, github: { diff --git a/src/evals/eval-runner.ts b/src/evals/eval-runner.ts index 8662040..c8e2933 100644 --- a/src/evals/eval-runner.ts +++ b/src/evals/eval-runner.ts @@ -141,7 +141,7 @@ const caseSchema = z adaptiveSecondPass: z.boolean().optional(), // Plan 103 (experiment-only): eval cases are the only surface that can // set these. No codegenie.toml may. - packCompatibleAtoms: z.boolean().optional(), + packRelatedHunks: z.boolean().optional(), packMaxHunks: positiveIntSchema.max(MAX_PACK_HUNKS).optional(), pinnedPlanPath: z.string().min(1).optional(), verify: z.boolean().optional(), @@ -831,8 +831,8 @@ function applyCaseReviewConfig( if (review?.adaptiveSecondPass !== undefined) { config.review.adaptiveSecondPass = review.adaptiveSecondPass; } - if (review?.packCompatibleAtoms !== undefined) { - config.review.packCompatibleAtoms = review.packCompatibleAtoms; + if (review?.packRelatedHunks !== undefined) { + config.review.packRelatedHunks = review.packRelatedHunks; } if (review?.packMaxHunks !== undefined) { config.review.packMaxHunks = review.packMaxHunks; diff --git a/src/pipeline/packet-builder.ts b/src/pipeline/packet-builder.ts index edffabe..43074ae 100644 --- a/src/pipeline/packet-builder.ts +++ b/src/pipeline/packet-builder.ts @@ -244,7 +244,7 @@ export async function buildReviewPackets( ); }; - for (const candidate of packCompatibleAtoms(groups, effectiveByHunk, positionByHunk, opts.config)) { + for (const candidate of packRelatedHunks(groups, effectiveByHunk, positionByHunk, opts.config)) { if (candidate.atoms.length <= 1) { const built = await build(candidate.group, telemetry, symbolContextMetrics, packetBuildMetrics); packets.push(built.packet); @@ -1380,13 +1380,13 @@ function packetGroup(hunks: PlannedHunk[], degradationReason?: string): PacketGr // set, so packing introduces no coverage promotion and cannot silently reroute // a hunk to different expertise. Returns the original groups untouched when // packing is off or when any group bypassed hunkFirstGroups(). -function packCompatibleAtoms( +function packRelatedHunks( groups: PacketGroup[], effectiveByHunk: Map, positionByHunk: Map, config: CodegenieConfig ): PackCandidate[] { - if (!config.review.packCompatibleAtoms || !groups.every((group) => group.origin === "hunk-first")) { + if (!config.review.packRelatedHunks || !groups.every((group) => group.origin === "hunk-first")) { return groups.map((group) => ({ group, atoms: [] })); } diff --git a/src/types.ts b/src/types.ts index 094b566..0c4f25b 100644 --- a/src/types.ts +++ b/src/types.ts @@ -59,7 +59,7 @@ export type CodegenieConfig = { // varies. Deliberately absent from rawConfigSchema and repo-safe // filtering — no codegenie.toml may set either, and neither is a // supported user surface. Eval cases set them; nothing else can. - packCompatibleAtoms: boolean; + packRelatedHunks: boolean; packMaxHunks: number; // Plan 103 eval-only: replay one recorded Stage-5 plan across arms so // packet size is the only difference between them. @@ -1024,7 +1024,7 @@ export type EvalCase = { adaptiveSecondPass?: boolean; // Plan 103 (experiment-only): eval cases are the only surface that may set // these; removed at teardown. - packCompatibleAtoms?: boolean; + packRelatedHunks?: boolean; packMaxHunks?: number; pinnedPlanPath?: string; verify?: boolean; diff --git a/tests/config-loader.test.ts b/tests/config-loader.test.ts index f9fbff6..6e741f8 100644 --- a/tests/config-loader.test.ts +++ b/tests/config-loader.test.ts @@ -333,7 +333,7 @@ function tempDir(): string { describe("plan 103 packing settings", () => { it("defaults to dark packing at the shipped cap", () => { const loaded = loadConfig({ repoRoot: tempDir(), homeOverride: tempDir() }); - expect(loaded.config.review.packCompatibleAtoms).toBe(false); + expect(loaded.config.review.packRelatedHunks).toBe(false); expect(loaded.config.review.packMaxHunks).toBe(MAX_PACK_HUNKS); expect(MAX_PACK_HUNKS).toBe(5); }); @@ -341,12 +341,12 @@ describe("plan 103 packing settings", () => { it("refuses both settings from every config file surface", () => { // Plan 103 keeps these eval-only: no codegenie.toml and no user config may // reach them, so strict parsing must reject rather than silently filter. - expect(rawConfigSchema.safeParse({ review: { packCompatibleAtoms: true } }).success).toBe(false); + expect(rawConfigSchema.safeParse({ review: { packRelatedHunks: true } }).success).toBe(false); expect(rawConfigSchema.safeParse({ review: { packMaxHunks: 3 } }).success).toBe(false); const repoRoot = tempDir(); const home = tempDir(); - writeFileSync(path.join(repoRoot, "codegenie.toml"), "[review]\npackCompatibleAtoms = true\n"); + writeFileSync(path.join(repoRoot, "codegenie.toml"), "[review]\npackRelatedHunks = true\n"); expect(() => loadConfig({ repoRoot, homeOverride: home })).toThrow(/invalid config file/); const userHome = tempDir(); @@ -354,6 +354,24 @@ describe("plan 103 packing settings", () => { expect(() => loadConfig({ repoRoot: tempDir(), homeOverride: userHome })).toThrow(/invalid config file/); }); + it("toggles packing per run from the CLI only", () => { + const repoRoot = tempDir(); + const home = tempDir(); + expect(loadConfig({ repoRoot, homeOverride: home }).config.review.packRelatedHunks).toBe(false); + + const on = loadConfig({ repoRoot, homeOverride: home, cli: { packRelatedHunks: true } }); + expect(on.config.review.packRelatedHunks).toBe(true); + expect(on.sources["review.packRelatedHunks"]).toBe("cli"); + + // A repository must not be able to enable experimental packing for + // everyone who reviews it; only an explicit per-run CLI flag can. + writeFileSync(path.join(repoRoot, "codegenie.toml"), "[review]\npackRelatedHunks = true\n"); + expect(() => loadConfig({ repoRoot, homeOverride: home })).toThrow(/invalid config file/); + + const off = loadConfig({ repoRoot: tempDir(), homeOverride: tempDir(), cli: { packRelatedHunks: false } }); + expect(off.config.review.packRelatedHunks).toBe(false); + }); + it("bounds packMaxHunks by the shipped packet cap in the resolved schema", () => { const base = structuredClone(defaultConfig) as CodegenieConfig; for (const value of [1, 3, MAX_PACK_HUNKS]) { diff --git a/tests/evals.test.ts b/tests/evals.test.ts index 8173255..6c47c6f 100644 --- a/tests/evals.test.ts +++ b/tests/evals.test.ts @@ -2305,7 +2305,7 @@ describe("plan 103 packing eval surface", () => { "repo:", " fixture: repo", "review:", - " packCompatibleAtoms: true", + " packRelatedHunks: true", " packMaxHunks: 3", "should_find:", " - id: expected", @@ -2313,7 +2313,7 @@ describe("plan 103 packing eval surface", () => { ].join("\n")); const suite = await loadEvalSuite(suiteDir); - expect(suite.cases[0]?.evalCase.review).toMatchObject({ packCompatibleAtoms: true, packMaxHunks: 3 }); + expect(suite.cases[0]?.evalCase.review).toMatchObject({ packRelatedHunks: true, packMaxHunks: 3 }); const overCap = mkdtempSync(path.join(tmpdir(), "codegenie-eval-packing-cap-")); writeFileSync(path.join(overCap, "packing.yml"), [ diff --git a/tests/pipeline-phase5.test.ts b/tests/pipeline-phase5.test.ts index 87441d5..c80f0c2 100644 --- a/tests/pipeline-phase5.test.ts +++ b/tests/pipeline-phase5.test.ts @@ -13741,7 +13741,7 @@ describe("plan 103 compatible-atom packing", () => { const off = await pack(file, plan); expect(off).toHaveLength(6); - const on = await pack(file, plan, { packCompatibleAtoms: true }); + const on = await pack(file, plan, { packRelatedHunks: true }); expect(hunkIdsOf(on)).toEqual([["h1", "h2", "h3", "h4", "h5"], ["h6"]]); }); @@ -13749,24 +13749,24 @@ describe("plan 103 compatible-atom packing", () => { const file = separatedFile("app.ts", 5); const plan = planFor(file); const off = await pack(file, plan); - const capOne = await pack(file, plan, { packCompatibleAtoms: true, packMaxHunks: 1 }); + const capOne = await pack(file, plan, { packRelatedHunks: true, packMaxHunks: 1 }); expect(JSON.stringify(capOne)).toBe(JSON.stringify(off)); }); it("never packs across effective coverage or requested lens boundaries", async () => { const file = separatedFile("app.ts", 4); - const mixedCoverage = await pack(file, planFor(file, { h2: { coverage: "deep" } }), { packCompatibleAtoms: true }); + const mixedCoverage = await pack(file, planFor(file, { h2: { coverage: "deep" } }), { packRelatedHunks: true }); expect(hunkIdsOf(mixedCoverage)).toEqual([["h1", "h3", "h4"], ["h2"]]); expect(mixedCoverage.map((packet) => packet.coverage).sort()).toEqual(["deep", "normal"]); - const mixedLenses = await pack(file, planFor(file, { h3: { lenses: ["core/tests"] } }), { packCompatibleAtoms: true }); + const mixedLenses = await pack(file, planFor(file, { h3: { lenses: ["core/tests"] } }), { packRelatedHunks: true }); expect(hunkIdsOf(mixedLenses)).toEqual([["h1", "h2", "h4"], ["h3"]]); }); it("assigns every reviewable hunk exactly once, in source order", async () => { const file = separatedFile("app.ts", 9); - const packets = await pack(file, planFor(file), { packCompatibleAtoms: true }); + const packets = await pack(file, planFor(file), { packRelatedHunks: true }); const flattened = packets.flatMap((packet) => packet.hunks.map((hunk) => hunk.hunkId)); expect(new Set(flattened).size).toBe(9); for (const packet of packets) { @@ -13779,7 +13779,7 @@ describe("plan 103 compatible-atom packing", () => { it("never splits an atom, even when the atom alone exceeds the cap", async () => { // h1 and h2 sit within NEARBY_GAP_LINES, so the grouper makes them one atom. const file = separatedFile("app.ts", 3, 10); - const packets = await pack(file, planFor(file), { packCompatibleAtoms: true, packMaxHunks: 1 }); + const packets = await pack(file, planFor(file), { packRelatedHunks: true, packMaxHunks: 1 }); expect(hunkIdsOf(packets)).toEqual([["h1", "h2", "h3"]]); }); @@ -13792,7 +13792,7 @@ describe("plan 103 compatible-atom packing", () => { fakeRepositoryIndex(fakeTools("export const value = 1;\n")), nullTelemetry(), { - config: { ...config(), review: { ...config().review, packCompatibleAtoms: true } }, + config: { ...config(), review: { ...config().review, packRelatedHunks: true } }, enabledLenses: ["core/code-review"] } ); @@ -13802,7 +13802,7 @@ describe("plan 103 compatible-atom packing", () => { it("recomputes dispatch rank from the combined packet's changed lines", async () => { const file = separatedFile("app.ts", 5); - const packets = await pack(file, planFor(file), { packCompatibleAtoms: true }); + const packets = await pack(file, planFor(file), { packRelatedHunks: true }); expect(packets).toHaveLength(1); expect(packets[0]?.dispatchRank).toEqual(packetDispatchRank("app.ts", { testStatus: "source" }, 5)); }); @@ -13811,7 +13811,7 @@ describe("plan 103 compatible-atom packing", () => { const file = separatedFile("app.ts", 15); const plan = planFor(file); for (const [cap, expected] of [[1, 15], [3, 5], [5, 3]] as const) { - const packets = await pack(file, plan, { packCompatibleAtoms: true, packMaxHunks: cap }); + const packets = await pack(file, plan, { packRelatedHunks: true, packMaxHunks: cap }); expect(packets).toHaveLength(expected); expect(packets.every((packet) => packet.hunks.length <= cap)).toBe(true); } @@ -13906,7 +13906,7 @@ describe("plan 103 multi-member context and transactional packing", () => { : nullTelemetry(); const base = config(); return buildReviewPackets(plan(file), [file], [fakeFacts(file.path, "per-hunk")], index, recorder, { - config: { ...base, review: { ...base.review, packCompatibleAtoms: true } }, + config: { ...base, review: { ...base.review, packRelatedHunks: true } }, enabledLenses: ["core/code-review"] }); } @@ -13969,11 +13969,11 @@ describe("plan 103 multi-member context and transactional packing", () => { const base = config(); const index: RepositoryIndex = { facts: [], symbolFacts: facts, staticSignals: [], tools: symbolTools(900) }; const packed = await buildReviewPackets(plan(file), [file], [fakeFacts(file.path, "per-hunk")], index, nullTelemetry(), { - config: { ...base, review: { ...base.review, packCompatibleAtoms: true } }, + config: { ...base, review: { ...base.review, packRelatedHunks: true } }, enabledLenses: ["core/code-review"] }); const unpacked = await buildReviewPackets(plan(file), [file], [fakeFacts(file.path, "per-hunk")], index, nullTelemetry(), { - config: { ...base, review: { ...base.review, packCompatibleAtoms: false } }, + config: { ...base, review: { ...base.review, packRelatedHunks: false } }, enabledLenses: ["core/code-review"] }); From eb02b044fb37ed80d28b1fde49ffc7af05c1c45f Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 20:26:37 -0400 Subject: [PATCH 20/24] refactor(cli): drop --no-pack-related-hunks Off is already the default, so the negative form carried no meaning that omitting the flag did not. Omitting it now leaves the setting at its default with "defaults" config-source attribution, which a test pins. Co-Authored-By: Claude Opus 5 (1M context) --- src/cli/review-command.ts | 1 - tests/config-loader.test.ts | 7 +++++-- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/src/cli/review-command.ts b/src/cli/review-command.ts index 2937d44..f6a5594 100644 --- a/src/cli/review-command.ts +++ b/src/cli/review-command.ts @@ -102,7 +102,6 @@ export function parseReviewCommand( // no codegenie.toml or user config may set it, so a repository cannot // enable experimental packing for everyone who reviews it. .option("--pack-related-hunks", "experimental: pack related same-file hunks into fewer review packets") - .option("--no-pack-related-hunks", "experimental: force related-hunk packing off for this run") .action((commitArgs: string[], options: CommanderReviewOptions) => { commits = commitArgs; commandOptions = options; diff --git a/tests/config-loader.test.ts b/tests/config-loader.test.ts index 6e741f8..741f520 100644 --- a/tests/config-loader.test.ts +++ b/tests/config-loader.test.ts @@ -368,8 +368,11 @@ describe("plan 103 packing settings", () => { writeFileSync(path.join(repoRoot, "codegenie.toml"), "[review]\npackRelatedHunks = true\n"); expect(() => loadConfig({ repoRoot, homeOverride: home })).toThrow(/invalid config file/); - const off = loadConfig({ repoRoot: tempDir(), homeOverride: tempDir(), cli: { packRelatedHunks: false } }); - expect(off.config.review.packRelatedHunks).toBe(false); + // Omitting the flag leaves the default in place; there is no --no- form + // because off is already the default. + const omitted = loadConfig({ repoRoot: tempDir(), homeOverride: tempDir(), cli: {} }); + expect(omitted.config.review.packRelatedHunks).toBe(false); + expect(omitted.sources["review.packRelatedHunks"]).toBe("defaults"); }); it("bounds packMaxHunks by the shipped packet cap in the resolved schema", () => { From 8507c6306830607bc17e552251c957f4f840ee9e Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 20:37:04 -0400 Subject: [PATCH 21/24] feat(pipeline): record same_file_atoms_packed provenance Without this a multi-hunk packet is indistinguishable from one today's grouper produced on its own - both look identical in the packet artifact - so a packing A/B cannot tell treated packets from untreated ones. Plan 102's design called for this provenance; phase 1 trimmed atom metadata from scope and phase 2 added the context work but not the record, and the omission only surfaced when the first real A/B run started. Emitted once per committed packed packet, carrying packet id, ordered source atom ids and their hunk counts, resulting hunk ids and count, coverage, lens signature, routed lenses, each member's standalone profile, derived versus effective profile with a floor-applied flag, and cap usage against packMaxHunks and MAX_PATCH_CHARS. Artifact-only: it stays out of the reviewer prompt and out of the packet id, so it cannot change review behaviour or packet identity. Purely additive - packet construction is untouched. Two tests: a six-atom file at a five-hunk cap emits exactly one event naming five source atoms and matching the committed packet id, and nothing is emitted with packing off. 788 tests. Co-Authored-By: Claude Opus 5 (1M context) --- src/pipeline/packet-builder.ts | 38 ++++++++++++++++++++++++++++-- tests/pipeline-phase5.test.ts | 43 ++++++++++++++++++++++++++++++++++ 2 files changed, 79 insertions(+), 2 deletions(-) diff --git a/src/pipeline/packet-builder.ts b/src/pipeline/packet-builder.ts index 43074ae..6a871b4 100644 --- a/src/pipeline/packet-builder.ts +++ b/src/pipeline/packet-builder.ts @@ -250,7 +250,7 @@ export async function buildReviewPackets( packets.push(built.packet); continue; } - const committed = await commitPackedCandidate(candidate, build, telemetry, symbolContextMetrics, packetBuildMetrics); + const committed = await commitPackedCandidate(candidate, build, telemetry, symbolContextMetrics, packetBuildMetrics, opts.config); packets.push(...committed); } } @@ -1459,7 +1459,8 @@ async function commitPackedCandidate( build: BuildFn, telemetry: TelemetryRecorder, symbolContextMetrics: SymbolContextMetrics, - packetBuildMetrics: PacketBuildMetrics + packetBuildMetrics: PacketBuildMetrics, + config: CodegenieConfig ): Promise { const scratchMetrics = emptySymbolContextMetrics(); const scratchBuildMetrics: PacketBuildMetrics = { @@ -1479,6 +1480,39 @@ async function commitPackedCandidate( if (rejection === undefined) { const committed = await build(candidate.group, telemetry, symbolContextMetrics, packetBuildMetrics, { members, profileFloor }); + // Artifact-only provenance: without this a multi-hunk packet is + // indistinguishable from one today's grouper produced on its own, so a + // packing A/B cannot tell treated packets from untreated ones. Kept out of + // the reviewer prompt and out of the packet ID. + const patchChars = candidate.atoms.reduce((sum, atom) => sum + atom.patchChars, 0); + telemetry.event({ + stage: 6, + level: "info", + message: "same_file_atoms_packed", + packetId: committed.packet.id, + file: committed.packet.path, + data: { + sourceAtomIds: candidate.atoms.map((atom) => atom.id), + sourceAtomCount: candidate.atoms.length, + sourceAtomHunkCounts: candidate.atoms.map((atom) => atom.hunkCount), + hunkCount: committed.packet.hunks.length, + hunkIds: committed.packet.hunks.map((hunk) => hunk.hunkId), + coverage: committed.packet.coverage, + lensSignature: candidate.atoms[0]?.lensSignature ?? "", + routedLenses: committed.routedLenses, + standaloneProfiles: standalone.map((member) => member.reviewProfile), + derivedProfile: committed.derivedReviewProfile, + effectiveProfile: committed.reviewProfile, + profileFloorApplied: + REVIEW_PROFILE_RANK[committed.reviewProfile] > REVIEW_PROFILE_RANK[committed.derivedReviewProfile], + capUsage: { + hunks: committed.packet.hunks.length, + maxHunks: config.review.packMaxHunks, + patchChars, + maxPatchChars: MAX_PATCH_CHARS + } + } + }); return [committed.packet]; } diff --git a/tests/pipeline-phase5.test.ts b/tests/pipeline-phase5.test.ts index c80f0c2..6024aed 100644 --- a/tests/pipeline-phase5.test.ts +++ b/tests/pipeline-phase5.test.ts @@ -13807,6 +13807,49 @@ describe("plan 103 compatible-atom packing", () => { expect(packets[0]?.dispatchRank).toEqual(packetDispatchRank("app.ts", { testStatus: "source" }, 5)); }); + it("records packing provenance so treated packets are identifiable", async () => { + const file = separatedFile("app.ts", 6); + const plan = planFor(file); + const events: Array<{ message?: string; packetId?: string; data?: Record }> = []; + const recorder = { + event: (entry: unknown) => events.push(entry as { message?: string }), + writeArtifact: async () => undefined + } as unknown as ReturnType; + const base = config(); + const packets = await buildReviewPackets(plan, [file], [fakeFacts(file.path, "per-hunk")], fakeRepositoryIndex(fakeTools()), recorder, { + config: { ...base, review: { ...base.review, packRelatedHunks: true } }, + enabledLenses: ["core/code-review"] + }); + + const packed = events.filter((entry) => entry.message === "same_file_atoms_packed"); + // Six atoms at a five-hunk cap: one packed packet of five, one lone atom. + expect(packed).toHaveLength(1); + expect(packets).toHaveLength(2); + const data = packed[0]?.data as Record; + expect(data.sourceAtomCount).toBe(5); + expect(data.hunkCount).toBe(5); + expect(data.hunkIds).toEqual(["h1", "h2", "h3", "h4", "h5"]); + expect((data.sourceAtomIds as string[])).toHaveLength(5); + expect(packed[0]?.packetId).toBe(packets[0]?.id); + expect(data.capUsage).toMatchObject({ hunks: 5, maxHunks: 5 }); + expect(data.profileFloorApplied).toBe(false); + }); + + it("records no packing provenance when packing is off", async () => { + const file = separatedFile("app.ts", 6); + const events: Array<{ message?: string }> = []; + const recorder = { + event: (entry: unknown) => events.push(entry as { message?: string }), + writeArtifact: async () => undefined + } as unknown as ReturnType; + const base = config(); + await buildReviewPackets(planFor(file), [file], [fakeFacts(file.path, "per-hunk")], fakeRepositoryIndex(fakeTools()), recorder, { + config: { ...base, review: { ...base.review, packRelatedHunks: false } }, + enabledLenses: ["core/code-review"] + }); + expect(events.filter((entry) => entry.message === "same_file_atoms_packed")).toHaveLength(0); + }); + it("parameterizes packet shape by packMaxHunks for the recall curve", async () => { const file = separatedFile("app.ts", 15); const plan = planFor(file); From 0bae8f6756f7e999bc5acfea75d90f81b8e11a4d Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 21:24:41 -0400 Subject: [PATCH 22/24] feat(scripts): capture a real planner draw with draw-pinned-plan --from-run The script only authored uniform plans - every hunk the same coverage and lenses. That was right for the dilution fixture, where making packet size the only variable was the point, but wrong for a production A/B: it discards the planner's per-hunk deep/normal/light grading and reviews a workload production never produces. --from-run wraps the real Stage-5 draw recorded by an existing run, so both arms of an A/B replay one planner output and packing is the only difference between them. It validates before writing: every hunk id in the recorded plan must be present in the diff for the given base and head, and a mismatch exits non-zero naming the stale ids. That caught a real trap immediately - the review pipeline diffs from the merge-base, not from the base branch tip, so drawing against "master" produces different hunk ids than the run used. The error now says so instead of silently emitting an artifact that would fail at load time. Output reports whether the plan was recorded or authored, its coverage histogram, and how many diff hunks it covers, so an unlisted-hunk gap is visible before the artifact is used. Co-Authored-By: Claude Opus 5 (1M context) --- scripts/draw-pinned-plan.ts | 51 +++++++++++++++++++++++++++++++++---- 1 file changed, 46 insertions(+), 5 deletions(-) diff --git a/scripts/draw-pinned-plan.ts b/scripts/draw-pinned-plan.ts index a6c8df0..fe36a1a 100644 --- a/scripts/draw-pinned-plan.ts +++ b/scripts/draw-pinned-plan.ts @@ -3,9 +3,20 @@ // a CLI verb — the pinned-plan seam is eval-only and adding a user-facing // command would contradict that scope. // -// --repo --base --head --output authored/derived from the diff -// --repo ... --coverage normal --lenses lang/go author a uniform plan (no model call) -import { writeFileSync } from "node:fs"; +// Two modes: +// +// --from-run wrap the REAL planner draw recorded by an existing +// run, so an A/B can replay one Stage-5 output across +// both arms. This is what you want for a production +// A/B: it preserves the planner's per-hunk deep/normal/ +// light grading exactly as it was drawn. +// +// (default) author a UNIFORM plan - every hunk gets the same +// coverage and lenses. Built for the dilution fixture, +// where making packet size the only variable was the +// point. Not representative of production grading. +import { readFileSync, writeFileSync } from "node:fs"; +import path from "node:path"; import { parseDiff } from "../src/git/diff-parser.js"; import { createGitClient } from "../src/git/git-client.js"; import { buildPinnedPlanArtifact } from "../src/pipeline/pinned-plan.js"; @@ -29,10 +40,26 @@ if (repo === undefined || output === undefined) { process.exit(2); } +const fromRun = arg("from-run"); const git = createGitClient(repo); const rawDiff = await git.diff(base, head); const diff = parseDiff(rawDiff); -const plan: ReviewPlan = { +const diffHunkIds = new Set(diff.files.flatMap((file) => file.hunks.map((hunk) => hunk.id))); + +const recordedPlan: ReviewPlan | undefined = fromRun === undefined + ? undefined + : JSON.parse(readFileSync(path.join(fromRun, "stages/05-planner/review-plan.json"), "utf8")) as ReviewPlan; + +if (recordedPlan !== undefined) { + const stale = recordedPlan.coverage.filter((entry) => !diffHunkIds.has(entry.hunkId)).map((entry) => entry.hunkId); + if (stale.length > 0) { + console.error(`recorded plan references ${String(stale.length)} hunk id(s) absent from this diff: ${stale.slice(0, 5).join(", ")}`); + console.error("the run's diff and this base/head do not match; re-draw against the same refs"); + process.exit(1); + } +} + +const authored: ReviewPlan = { diffUnderstanding: { declaredIntent: "authored pinned plan for the packet-size recall curve", inferredBehavior: "uniform coverage so packet size is the only variable across arms" @@ -49,6 +76,8 @@ const plan: ReviewPlan = { ) }; +const plan = recordedPlan ?? authored; + // "*" marks a content-anchored artifact: fixture repos are materialized fresh // per run so their SHAs vary, while content-derived hunk IDs do not. const artifact = buildPinnedPlanArtifact({ @@ -57,4 +86,16 @@ const artifact = buildPinnedPlanArtifact({ plan }); writeFileSync(output, `${JSON.stringify(artifact, null, 2)}\n`); -console.log(JSON.stringify({ output, hunks: plan.coverage.length, planSha256: artifact.planSha256, anchor }, null, 1)); +const byCoverage: Record = {}; +for (const entry of plan.coverage) { + byCoverage[entry.coverage] = (byCoverage[entry.coverage] ?? 0) + 1; +} +console.log(JSON.stringify({ + output, + source: recordedPlan !== undefined ? `recorded draw: ${String(fromRun)}` : "authored uniform plan", + coverageEntries: plan.coverage.length, + diffHunks: diffHunkIds.size, + byCoverage, + planSha256: artifact.planSha256, + anchor +}, null, 1)); From f5e1cf3b20125b4bc79030cdf370db547f4b947d Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sat, 25 Jul 2026 22:00:02 -0400 Subject: [PATCH 23/24] fix(pipeline): write the pinned plan into run artifacts With a pinned plan Stage 5 never executes, so nothing wrote review-plan.json and the run could not describe which plan produced its packets. Runs 50 and 51 both have an empty 05-planner stage directory - the plan they actually used lives only in /tmp, outside the run, and the artifacts stop being self-contained the moment that file moves or is overwritten by the next A/B. The seam now writes review-plan.json with the pinned plan, matching what the planner would have written, plus pinned-plan-source.json recording the source path, the canonical plan hash, the coverage entry count, and the base and head the plan was validated against. The planner_plan_pinned event carries the hash too, so two arms of an A/B can be proven to have consumed the same plan from their artifacts alone rather than by trusting that the file did not change between them. Both names are added to the artifact path map so they land in stages/05-planner/ rather than outside it, with a test pinning the seam and the map together - the failure mode is silent, since an unmapped artifact still writes, just somewhere useless. 789 tests. Co-Authored-By: Claude Opus 5 (1M context) --- src/pipeline/review-runner.ts | 19 +++++++++++++++++-- src/telemetry/run-artifacts.ts | 1 + tests/pinned-plan.test.ts | 18 +++++++++++++++++- 3 files changed, 35 insertions(+), 3 deletions(-) diff --git a/src/pipeline/review-runner.ts b/src/pipeline/review-runner.ts index b9f66ea..7cbc277 100644 --- a/src/pipeline/review-runner.ts +++ b/src/pipeline/review-runner.ts @@ -48,7 +48,7 @@ import type { import { CodegenieError, errorExitCode, isCodegenieError } from "../util/errors.js"; import { buildPlannerDossier, runPlanner } from "./planner.js"; import { buildReviewPackets, packetReviewContextFromDossier } from "./packet-builder.js"; -import { loadPinnedPlan } from "./pinned-plan.js"; +import { loadPinnedPlan, planSha256 } from "./pinned-plan.js"; import type { PlannerRunResult } from "./planner.js"; import { ensemblePassesForPacket, runLensPackets } from "./lens-runner.js"; import { aggregateAttentionEfficiency, buildAttentionRecords } from "./attention.js"; @@ -216,11 +216,26 @@ export async function runReview( ...(resolved.headSha !== undefined ? { headSha: resolved.headSha } : {}), diff }); + // Stage 5 never runs, so nothing else writes review-plan.json. Without + // this the run cannot describe which plan produced its packets, and the + // artifacts stop being self-contained the moment the external file moves. + await run.telemetry.writeArtifact("review-plan.json", pinnedPlan); + await run.telemetry.writeArtifact("pinned-plan-source.json", { + path: pinnedPlanPath, + planSha256: planSha256(pinnedPlan), + coverageEntries: pinnedPlan.coverage.length, + ...(baseSha !== undefined ? { baseSha } : {}), + ...(resolved.headSha !== undefined ? { headSha: resolved.headSha } : {}) + }); run.telemetry.event({ stage: 5, level: "info", message: "planner_plan_pinned", - data: { path: pinnedPlanPath, coverageEntries: pinnedPlan.coverage.length } + data: { + path: pinnedPlanPath, + planSha256: planSha256(pinnedPlan), + coverageEntries: pinnedPlan.coverage.length + } }); plannerResult = { plan: pinnedPlan, diff --git a/src/telemetry/run-artifacts.ts b/src/telemetry/run-artifacts.ts index 313f36e..0c6777d 100644 --- a/src/telemetry/run-artifacts.ts +++ b/src/telemetry/run-artifacts.ts @@ -63,6 +63,7 @@ export const ARTIFACT_LOCATION = { "planner-dossier.json": "stages/05-planner/planner-dossier.json", "planner-dossier-chunks.json": "stages/05-planner/planner-dossier-chunks.json", "review-plan.json": "stages/05-planner/review-plan.json", + "pinned-plan-source.json": "stages/05-planner/pinned-plan-source.json", "hunk-relationships.json": "stages/06-packets/hunk-relationships.json", "system-review-raw-tasks.json": "stages/08-followups/system-review-raw-tasks.json", "system-review-tasks.json": "stages/08-followups/system-review-tasks.json", diff --git a/tests/pinned-plan.test.ts b/tests/pinned-plan.test.ts index bb8e0c5..fbb8186 100644 --- a/tests/pinned-plan.test.ts +++ b/tests/pinned-plan.test.ts @@ -1,4 +1,4 @@ -import { mkdtempSync, writeFileSync } from "node:fs"; +import { mkdtempSync, readFileSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import path from "node:path"; import { describe, expect, it } from "vitest"; @@ -77,3 +77,19 @@ describe("pinned plan artifact", () => { expect(() => loadPinnedPlan(badPath, {})).toThrow(/readable JSON/); }); }); + +describe("pinned plan run artifacts", () => { + it("keeps the artifact path map and the seam in agreement", async () => { + // The seam writes both names; if either loses its stage mapping the run + // stops being self-describing, which is the failure this guards. + const runArtifacts = await import("../src/telemetry/run-artifacts.js"); + const source = readFileSync(new URL("../src/telemetry/run-artifacts.ts", import.meta.url), "utf8"); + expect(source).toContain('"review-plan.json": "stages/05-planner/review-plan.json"'); + expect(source).toContain('"pinned-plan-source.json": "stages/05-planner/pinned-plan-source.json"'); + expect(runArtifacts).toBeDefined(); + + const runner = readFileSync(new URL("../src/pipeline/review-runner.ts", import.meta.url), "utf8"); + expect(runner).toContain('writeArtifact("review-plan.json", pinnedPlan)'); + expect(runner).toContain('writeArtifact("pinned-plan-source.json"'); + }); +}); From cb94bc4155629022fb5e2704b8ee2d5368c97ae9 Mon Sep 17 00:00:00 2001 From: Peter Kieltyka Date: Sun, 26 Jul 2026 10:13:31 -0400 Subject: [PATCH 24/24] write about results, 103 isn't worth it --- ...-103-relationship-driven-packet-packing.md | 91 +++++++++++++++++++ 1 file changed, 91 insertions(+) diff --git a/specs/plans/103-issue-103-relationship-driven-packet-packing.md b/specs/plans/103-issue-103-relationship-driven-packet-packing.md index 52abb17..9286503 100644 --- a/specs/plans/103-issue-103-relationship-driven-packet-packing.md +++ b/specs/plans/103-issue-103-relationship-driven-packet-packing.md @@ -567,3 +567,94 @@ Note the `pnpm dev eval` spelling: under pnpm 11 a literal `--` reaches Commande - Plan 102's preserved reports and paid logs under `codegenie-private-evals/trails-api/packet-packing/` record its failed fixture design and must not be edited. This plan's evidence lives under `packet-dilution/reports/` with its own manifest. - Reviewers should scrutinize per-member context after final rendering, abandonment reasons, fixed-slot yield, and treatment proof — not packet count, which is Plan 102's already-validated result. - Commit hashes here are rebase-unstable; locate referenced work by commit subject when a hash does not resolve. + +## Production A/B Post-Mortem — Runs 67 and 68 (2026-07-26) + +### Decision + +**Skip Plan 103 for now. Do not promote `packRelatedHunks` as a product feature on the evidence from this pair.** Packing worked mechanically and showed a real constrained-capacity benefit, but it did not achieve the combined product goal: materially lower end-to-end cost or elapsed time while maintaining equivalent finding and report quality. + +This is a negative product decision, not a claim that the implementation did nothing. The experiment established useful properties of the treatment, recorded below, but the observed whole-review outcome is not strong enough to justify retaining or shipping the current policy. No further paid validation or promotion is authorized by this report; any retry should be proposed as a new, bounded experiment with fresh gates. + +### Compared runs and control quality + +Evidence is under `/home/peter/Dev/0xPolygon/codegenie-private-evals/trails-api/0c4d5213/logs/`: + +- **Run 67:** `packRelatedHunks: true` +- **Run 68:** `packRelatedHunks: false` + +Both runs reviewed the same base/head diff, used the same model and budgets, and consumed the same pinned Stage-5 plan. The canonical hashes of their `review-plan.json` artifacts match. Both reviewed all 131 hunks with identical coverage totals: 13 deep, 112 normal, 6 light, and 0 skipped. This makes the pair substantially better controlled than earlier planner-variable comparisons, although Stage 7 and verification remain nondeterministic model executions. + +### End-to-end results + +| Metric | Run 67: packed | Run 68: unpacked | Packed delta | +| --- | ---: | ---: | ---: | +| Generated/reviewed packets | 62 | 73 | -11 (-15.1%) | +| Hunks reviewed | 131 | 131 | no change | +| Hunks reached in the first 56 packet slots | 121 | 106 | +15 (+14.2%) | +| Elapsed time | 2,710.767 s (45m 10.8s) | 2,885.696 s (48m 05.7s) | -174.929 s (-6.1%) | +| Model calls | 211 | 239 | -28 (-11.7%) | +| Tokens | 4,861,061 | 5,074,177 | -213,116 (-4.2%) | +| Cost | $21.7162 | $22.0228 | -$0.3066 (-1.4%) | +| Candidate findings | 12 | 15 | -3 | +| Published findings | 3 | 4 | -1 | + +The capacity result is real: if both reviews had stopped after 56 packet conversations, the packed run would have covered 121 rather than 106 hunks (92.4% rather than 80.9% of the diff). That supports packing as a possible capacity mechanism for reviews that would otherwise time out. + +It does **not** establish the desired full-review product win. Both arms completed, and the user-visible improvement was approximately three minutes and thirty-one cents on a roughly 48-minute, $22 review. Packet reduction was 15.1%, below this plan's 20% gate; the fixed-slot improvement was 14.2%, slightly below the stated 15% target. The end-to-end cost and time deltas are too small and too exposed to normal model variance to justify the quality risk. + +### What packing actually treated + +Run 67 accepted nine merges, combining 20 source atoms containing 29 hunks into nine packets. This removed 11 conversations. There were no abandoned candidates, coverage changes, lens losses, hunk losses, duplicate hunks, or cap breaches. All treated packets were normal coverage and remained under the 5-hunk / 12,000-character caps. + +The treatment-local economics were materially stronger than the whole-run totals: + +| Treated work only | Packed | Corresponding unpacked packets | Reduction | +| --- | ---: | ---: | ---: | +| Stage-7 calls | 29 | 52 | 44.2% | +| Tokens | 699,190 | 965,424 | 27.6% | +| Cost | $3.2833 | $4.4453 | 26.1% | +| Cumulative model-call time | 1,424.6 s | 1,924.6 s | 26.0% | +| Agent repository-tool calls | 35 | 60 | 41.7% | +| Initial prompt characters | 579,165 | 859,791 | 32.6% | +| Packet context characters | 35,311 | 43,613 | 19.0% | + +However, a packed treatment conversation was not approximately equal in cost to a standalone one. Per conversation it used about 61% more tokens, cost about 64% more, and consumed about 64% more cumulative call time. Packing saved resources only because one larger conversation replaced an average of 2.22 smaller conversations. Across the complete review, unchanged work and model/verification variance absorbed most of that local saving. + +The run also did not demonstrate relationship-driven selection. Only two of the nine merges joined source atoms connected by strong edges in the retained relationship graph; those accounted for three of the eleven removed packets. The other seven merges, accounting for eight removed packets, came from same-file compatibility and source-order filling. Manual inspection found that many were reasonably related refactor changes (for example, an added helper import beside the implementation using it), but that relationship was not what authorized the merge. The current treatment therefore remains opportunistic compatible-atom batching rather than a proven tree-sitter/relationship-driven policy. + +### Finding and report quality + +Run 67 formally failed its eval because `minFindings` required four and it published three; run 68 published four and passed. Both runs nevertheless passed every required semantic expectation: + +- explicit-preference routing fallback behavior was found by both; +- the zero-decimal `AmountFromUSD` behavior change was found by both; +- the ERC20 balance test-coverage candidate was generated by both; and +- the LiFi false-positive control passed in both. + +Run 67 additionally published the optional relay-decimals narrowing issue. Run 68 generated the same candidate but rejected it during verification. Conversely, run 68 published two findings absent from run 67: + +- the ZeroEx transaction-value parsing finding came from an unchanged packet and therefore cannot be attributed to packing; and +- the `directEdgeOriginTokenAmount` parsing finding came from an atom included in a three-atom packed packet and is the one plausible treatment-side miss. + +The latter describes a real parser-domain change, but its defect value is weak and conditional: the removed dependency helper permissively extracted decimal digits from malformed strings, while the new path rejects them and returns `nil`, which may be safer for malformed provider data. Even so, the burden is on packing to demonstrate non-inferiority. One pair in which the final findings and report differ does not meet that burden. Required-target parity is encouraging, but it is not equivalent to maintaining the same overall quality. + +The pair also demonstrates why exact single-run finding equality cannot by itself isolate packing: an unchanged packet produced the ZeroEx finding in only one arm, and the same relay candidate received opposite verification outcomes. Repeated unpacked controls would be required to quantify this background variance. That uncertainty is a reason not to ship, not a reason to waive the quality requirement. + +### Telemetry observations + +- Run 67 recorded 91 `build_packet_context` operations while producing 62 final packets. The count reconciles as 73 standalone builds plus nine dry candidate builds plus nine committed packed builds. The dry builds were intended to use a suppressed telemetry sink, so they remain visible when they should not be. +- This local Stage-6 accounting makes total tool calls misleading: the packed run reports 584 total tool calls versus 555 unpacked, while actual Stage-7 plus Stage-9 agent tool use fell from 275 to 244. +- `packet_member_symbol_context` events did not carry a packet ID, complicating per-packet auditing. +- The core relationship graph remained stable across arms at 131 nodes and 298 edges. Related-context attachment records changed because relationships internal to a packed packet no longer needed separate excerpts. +- Run 67 had one invalid Stage-7 schema response that was repaired successfully; it did not lose a packet or required finding. + +These issues do not explain the product decision, but they should be fixed before any future packing experiment relies on telemetry totals as gates. + +### Product conclusion + +Plan 103 demonstrated that compatible-atom packing can reduce conversation count and can review more hunks before a fixed packet cutoff. It did not demonstrate a material end-to-end cost or latency improvement after the whole pipeline completed, and it did not establish equivalent finding/report quality. The observed 1.4% cost reduction and 6.1% elapsed-time reduction are insufficient compensation for an unresolved quality difference. + +The product goal was not "some packets were combined." It was lower cost and/or meaningfully lower completion time **while maintaining review quality**. Runs 67 and 68 do not satisfy that conjunction. Plan 103 is therefore skipped for now and must not be presented as a successful production optimization. + +If revisited, it should be a new plan rather than an extension that renegotiates these results. At minimum, a successor must pre-register meaningful end-to-end cost/time thresholds, adjudicated material-finding non-inferiority, repeated pinned-plan pairs plus unpacked controls, explicit per-pack relationship provenance, per-member assessments, and automatic teardown on failure.