Outcome 5 (1/2): replication statistics aggregator#175
Merged
joshuawheelock merged 2 commits intoJun 24, 2026
Conversation
Pure aggregator for K replicated A/B comparisons of the same scenario/harness,
the foundation for reporting lift with statistical confidence (GOAL.md
Outcome 5) instead of single-point estimates.
- domain/replication.ts: ReplicationReport + DimensionLiftStat
{ dimension, k, applicableReplications, meanJumbo, meanBaseline, meanLift,
sdLift, tStatistic, isSignal }, re-exported from the domain barrel.
- analysis/replication-stats.ts: aggregateReplications(comparisons) ->
ReplicationReport. Per dimension present in every replication: meanLift is the
mean of (jumbo - baseline) per replication, sdLift the sample (n-1) SD;
isSignal only when |meanLift| > sdLift (K<2 -> 0 SD, not a signal); tStatistic
reported against the K=5 one-tailed alpha=0.05 threshold (t>2.13, df=4).
token-efficiency replications that were N/A (maxScore 0) are excluded and
counted in applicableReplications. Pure, deterministic, no I/O.
7 new tests; full unit suite 361/361 green; tsc --noEmit clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
joshuawheelock
approved these changes
Jun 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
First building block of Outcome 5 (statistical confidence). A pure aggregator that turns K replicated A/B comparisons of the same scenario/harness into per-dimension lift statistics, so lift can be reported as mean ± SD instead of single-point estimates. Branches off
main— independent of the open Outcome 4 PRs (#170, #174).What's here
src/domain/replication.ts—ReplicationReport+DimensionLiftStat{ dimension, k, applicableReplications, meanJumbo, meanBaseline, meanLift, sdLift, tStatistic, isSignal }, re-exported from the domain barrel.src/analysis/replication-stats.ts—aggregateReplications(comparisons): ReplicationReport(pure, deterministic, no I/O). For each dimension present in every replication:meanLift= mean of(jumboScore − baselineScore)per replication;sdLift= sample (n−1) SD.isSignalonly when|meanLift| > sdLift(GOAL.md: a lift is a signal only when it exceeds one SD);K<2→ SD 0, never a signal.tStatisticreported against the K=5 one-tailed α=0.05 threshold (t > 2.13, df=4).maxScore 0) are excluded and counted inapplicableReplications.Scope
Aggregation only. The
eval run --replicate Kcommand that drives, persists, and displays this is the next building block (2/2), which will stack on this branch.Verification
isSignalboundary above/below 1 SD, K=1, N/A exclusion + applicable count, multi-dimension, "present in every replication", threshold note).tsc --noEmitclean.🤖 Generated with Claude Code