Skip to content

test(mcp): replace exact context-bytes pins with tolerance bands - #6603

Merged
bokelley merged 1 commit into
mainfrom
fix-6571-snapshot-tolerance-band
Aug 16, 2026
Merged

test(mcp): replace exact context-bytes pins with tolerance bands#6603
bokelley merged 1 commit into
mainfrom
fix-6571-snapshot-tolerance-band

Conversation

@bokelley

Copy link
Copy Markdown
Contributor

The exact-equality pins on standalone.context_bytes and the dictionary-definition counts in tests/mcp-schema-analysis.test.cjs force every schema-touching PR to re-pin the constants — guaranteeing merge conflicts between any two in-flight schema PRs. PR #6279 was knocked from MERGEABLE back to CONFLICTING four times in two days by exactly this line.

This replaces the pins with the invariants that matter:

  • standalone context within [200 KiB, 400 KiB] (large enough for the experiment comparisons to mean something, below the projection budget)
  • shared dictionary stays substantial (>100 definitions)
  • prompt cleanup strictly reduces definitions

The ratio comparisons (cleanup < 0.82×, shared dictionary < 0.37×) and the payload-consistency equality are unchanged — they carry the experiment's real signal.

Closes the snapshot-pin half of #6571; the budget-headroom half stays open pending the shared-dictionary graduation.

🤖 Generated with Claude Code

The exact-equality pins on standalone.context_bytes and the dictionary
definition counts forced every schema-touching PR to re-pin the
constants, guaranteeing merge conflicts between any two in-flight schema
PRs (#6571). Replace them with the invariants that matter: the standalone
context stays within [200 KiB, 400 KiB], the shared dictionary stays
substantial, and prompt cleanup strictly reduces definitions. The ratio
comparisons and the payload-consistency equality are unchanged and carry
the experiment's real signal.

Closes the snapshot-pin half of #6571; the budget-headroom half stays
open pending the shared-dictionary graduation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@aao-secretariat aao-secretariat Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ladon verdict: Approve

Approve — clean test-hygiene change.

This PR touches only tests/mcp-schema-analysis.test.cjs (+20/-5), replacing three brittle exact-equality snapshot pins (context_bytes == 287_357, dictionary_definitions == 148 and == 127) with a tolerance band plus strict-reduction inequalities. Ratio comparisons and the payload-consistency self-check are untouched, so context_bytes is still validated against its computed source rather than a frozen literal. No published protocol surface (schemas, docs/reference, dist artifacts, task defs) is touched, so no changeset is required. Well-justified fix for real merge-conflict churn (#6279, #6571).

Decision-table walk: no critical/high findings (row 1 no); gated_paths false (row 2 no); high_risk false, no deletions/modifications-with-medium (rows 3–5 no); no prior decision (row 6 no); no no-auto-approve team match (row 7 no); zero medium findings (row 8 no). Falls through to row 9 → approve.

Nit (non-blocking): line 18 lower-bound literal 200_000 is mislabeled as "200 KiB" in the error message.

@bokelley
bokelley merged commit c4947d3 into main Aug 16, 2026
30 checks passed
@bokelley
bokelley deleted the fix-6571-snapshot-tolerance-band branch August 16, 2026 18:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant