test: add end-to-end rewrite-retention regression check (#200)

[lizhuojunx86]

As offered in #200 — a self-contained e2e guard for the #204 fix.

What it does: generates a synthetic Claude Code corpus (streaming-duplicate
lines, skip-cases) under an isolated $HOME, scans it with the built binary,
simulates a resume/compact rewrite (drops the last N assistant
message-groups), rescans, and asserts:

• R1 parse+dedup sanity (scan == generator manifest)
• R2 rewrite retention (post-rewrite scan == pre-rewrite scan) — the Data-integrity note for JSONL-based trackers: session files rewrite on resume/compact #200 regression
• R3 restart stability (3 scans byte-identical)
• R4 (optional, SPLITRAIL_BASELINE_BIN=…3.5.9) demonstrates the pre-fix drift

Verified: passes on 3.6.0; R2 fails on 3.5.9 as expected (that's the bug).
On my real ~50-day corpus the same protocol passed all four assertions
(details in #200). Touches nothing outside scripts/rewrite-retention-check/;
no real ~/.claude data, no config, no upload path. Unix + python3 stdlib only.

Happy to adjust layout/conventions or wire it into CI if you want it there.

Summary by CodeRabbit

• New Features
  • Added an end-to-end regression harness to validate session resume/compact/retention JSONL totals.
  • Generates a synthetic Claude Code corpus with filtering and token-metric expectations.
  • Verifies restart stability via byte-identical consecutive scan outputs.
  • Supports optional baseline comparison to detect prior retention drift.
• Documentation
  • Documented how to run the isolated checks under a temporary synthetic home, including CI commands and validation assertions.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e35e05cd-8350-45c2-9faf-c31f282ad072

📥 Commits

Reviewing files that changed from the base of the PR and between 5c24840 and 1ac6a80.

📒 Files selected for processing (2)

• scripts/rewrite-retention-check/check_retention.py
• scripts/rewrite-retention-check/run_check.sh

🚧 Files skipped from review as they are similar to previous changes (2)

• scripts/rewrite-retention-check/run_check.sh
• scripts/rewrite-retention-check/check_retention.py

---

📝 Walkthrough

Walkthrough

Added an isolated end-to-end regression harness that generates synthetic Claude Code transcripts, simulates retention rewrites, scans results, and validates parity, restart stability, and optional baseline drift.

Changes

Rewrite-retention regression check

Layer / File(s)	Summary
Synthetic corpus and expected totals scripts/rewrite-retention-check/gen_corpus.py	Generates duplicated assistant records, skip-case lines, session transcripts, and per-model token manifests.
Transcript rewrite simulation scripts/rewrite-retention-check/simulate_rewrite.py	Removes the final assistant message groups from the largest transcript and records expected per-model deltas.
Scan orchestration and retention assertions scripts/rewrite-retention-check/run_check.sh, scripts/rewrite-retention-check/check_retention.py, scripts/rewrite-retention-check/README.md	Runs isolated pre- and post-rewrite scans with restart checks, validates R1–R4 assertions, and documents usage and CI commands.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

• Piebald-AI/splitrail#204: Implements the session-history preservation and deduplication behavior validated by this harness.

Poem

A rabbit checks each transcript line,
Counts the tokens, makes them shine.
Rewrite, restart—no totals stray,
Baseline drift is shown the way.
“PASS!” I thump beneath the moon.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding an end-to-end rewrite-retention regression check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/rewrite-retention-check/check_retention.py`:
- Around line 91-96: Update the R2 result construction around the same check so
its failure detail also reports cost-only drift, not just the existing diff()
output for integer totals. Preserve the current success message and integer-diff
reporting, while adding the relevant base/post cost differences when the cost
tolerance check is what makes same false.

In `@scripts/rewrite-retention-check/run_check.sh`:
- Around line 41-44: Update the scan function to set XDG_STATE_HOME and
XDG_DATA_HOME to the fake home alongside HOME when invoking the stats command,
ensuring the retention check uses only isolated fake state and data directories.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: a576cdd1-dfa5-44e1-83b5-61f3c254ee9c

📥 Commits

Reviewing files that changed from the base of the PR and between e2f1959 and 5c24840.

📒 Files selected for processing (5)

• scripts/rewrite-retention-check/README.md
• scripts/rewrite-retention-check/check_retention.py
• scripts/rewrite-retention-check/gen_corpus.py
• scripts/rewrite-retention-check/run_check.sh
• scripts/rewrite-retention-check/simulate_rewrite.py

[mike1858]

Great, thank you!