Skip to content

feat(pi): eval reproducibility and comparative reporting - #229

Merged
Patel230 merged 1 commit into
mainfrom
feat/pi-eval-reporting
Aug 22, 2026
Merged

feat(pi): eval reproducibility and comparative reporting#229
Patel230 merged 1 commit into
mainfrom
feat/pi-eval-reporting

Conversation

@Patel230

Copy link
Copy Markdown
Contributor

Summary

Implements the remaining feasible item from the Pi adoption follow-up plan (docs/plans/pi-renderer-and-eval-reporting.md):

  • Reproducibility hash: evalloop.Result.ReproHash — deterministic SHA-256 over model/provider/prompt/config-version and the transcript.
  • Comparative report: evalloop.Compare aggregates runs (token/cost/duration ranges, unique repro hashes) with a stable sort; FormatComparison renders it.
  • CLI: hawk eval loop --report prints the comparative summary and includes repro_hash in the JSON output.
  • Plan doc records why the Bubble Tea renderer swap and Kitty render-loop integration are NOT adopted: Bubble Tea v2 exposes no public Renderer interface (unexported renderer), so a clean differential-renderer swap is not feasible without forking renderer internals.

Test plan

  • Full go test ./... green (twice + via pre-push hook)
  • New tests: hash determinism + input-sensitivity, Compare aggregation/sort, FormatComparison
  • internal/feature/evalloop race-tested
  • make vet, make lint, hawk verify green
  • Boundary guards, submodule drift, govulncheck passed in pre-push hooks
  • markdownlint-cli2 clean on the new plan doc

@Patel230
Patel230 merged commit 605ebac into main Aug 22, 2026
26 checks passed
@Patel230
Patel230 deleted the feat/pi-eval-reporting branch August 22, 2026 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant