test: consolidate multipass differential suites into one target#539
Merged
Conversation
Move the interpreter-vs-multipass differential tests into a dedicated gtest target, evmDifferentialTests, in its own self-contained source file. It holds 40 fixture-based cases (RangeNarrowing, ConstShiftPruning, SubWrapU64) plus the 4-test adversarial-matrix harness, all sharing one executor that forces synchronous multipass compilation so the JIT run is non-vacuous. Three in-flight optimization PRs each appended a near-identical differential suite to src/tests/evm_interp_tests.cpp, guaranteeing merge conflicts; a fourth added the matrix harness there too. A single dedicated home removes both the duplication and the conflicts. The differential invariant holds on plain main, so the suite passes independently of those optimizations. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This was referenced Jun 11, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Adds a consolidated interpreter-vs-multipass EVM differential test target and a set of adversarial fixtures to catch value-range narrowing and limb-folding miscompilations.
Changes:
- Adds a new gtest executable
evmDifferentialTestswith both fixture-driven and programmatic adversarial-matrix differential tests. - Introduces 40 new EVM-assembly fixtures (
.easm+.expected) covering range-narrowing, shift pruning, and u64 SUB wrap/borrow edge cases. - Registers the new test target in
src/tests/CMakeLists.txtand documents the change indocs/changes/....
Reviewed changes
Copilot reviewed 83 out of 83 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/evm_asm/xor_dyn_u64_wide.expected | Adds expected output for XOR u64-vs-wide fixture |
| tests/evm_asm/xor_dyn_u64_wide.easm | Adds XOR u64-vs-wide adversarial fixture bytecode |
| tests/evm_asm/xor_dyn_u64_u64.expected | Adds expected output for XOR u64-vs-u64 fixture |
| tests/evm_asm/xor_dyn_u64_u64.easm | Adds XOR u64-vs-u64 fixture bytecode |
| tests/evm_asm/sub_wide_u64_control.expected | Adds expected output for wide-minus-u64 control fixture |
| tests/evm_asm/sub_wide_u64_control.easm | Adds wide-minus-u64 control fixture bytecode |
| tests/evm_asm/sub_u64_pair_zero_rhs_dyn.expected | Adds expected output for (u64 - dyn(0)) fixture |
| tests/evm_asm/sub_u64_pair_zero_rhs_dyn.easm | Adds (u64 - dyn(0)) fixture bytecode |
| tests/evm_asm/sub_u64_pair_wrap_boundary.expected | Adds expected output for u64 borrow/sign-fill boundary fixture |
| tests/evm_asm/sub_u64_pair_wrap_boundary.easm | Adds u64 borrow/sign-fill boundary fixture bytecode |
| tests/evm_asm/sub_u64_pair_underflow.expected | Adds expected output for u64 underflow fixture |
| tests/evm_asm/sub_u64_pair_underflow.easm | Adds u64 underflow fixture bytecode |
| tests/evm_asm/sub_u64_pair_nounderflow.expected | Adds expected output for u64 non-underflow fixture |
| tests/evm_asm/sub_u64_pair_nounderflow.easm | Adds u64 non-underflow fixture bytecode |
| tests/evm_asm/sub_u64_pair_equal.expected | Adds expected output for equal-u64 SUB fixture |
| tests/evm_asm/sub_u64_pair_equal.easm | Adds equal-u64 SUB fixture bytecode |
| tests/evm_asm/slt_dyn_neg_vs_const.expected | Adds expected output for SLT dyn-negative vs const fixture |
| tests/evm_asm/slt_dyn_neg_vs_const.easm | Adds SLT dyn-negative vs const fixture bytecode |
| tests/evm_asm/slt_dyn_msb64_vs_const.expected | Adds expected output for SLT msb64 edge fixture |
| tests/evm_asm/slt_dyn_msb64_vs_const.easm | Adds SLT msb64 edge fixture bytecode |
| tests/evm_asm/slt_dyn_highsparse_vs_const.expected | Adds expected output for SLT high-sparse limb fixture |
| tests/evm_asm/slt_dyn_highsparse_vs_const.easm | Adds SLT high-sparse limb fixture bytecode |
| tests/evm_asm/slt_dyn_eq_const.expected | Adds expected output for SLT equal-value boundary fixture |
| tests/evm_asm/slt_dyn_eq_const.easm | Adds SLT equal-value boundary fixture bytecode |
| tests/evm_asm/slt_const_vs_dyn.expected | Adds expected output for SLT const-vs-dyn swapped-dispatch fixture |
| tests/evm_asm/slt_const_vs_dyn.easm | Adds SLT const-vs-dyn swapped-dispatch fixture bytecode |
| tests/evm_asm/shr_const8_u64val.expected | Adds expected output for SHR const over u64-tagged value |
| tests/evm_asm/shr_const8_u64val.easm | Adds SHR const over u64-tagged value fixture bytecode |
| tests/evm_asm/shr_const72_dyn.expected | Adds expected output for SHR cross-limb const shift fixture |
| tests/evm_asm/shr_const72_dyn.easm | Adds SHR cross-limb const shift fixture bytecode |
| tests/evm_asm/shr_const4_dyn.expected | Adds expected output for SHR small const shift fixture |
| tests/evm_asm/shr_const4_dyn.easm | Adds SHR small const shift fixture bytecode |
| tests/evm_asm/shr_const256_dyn.expected | Adds expected output for SHR >=256 fold fixture |
| tests/evm_asm/shr_const256_dyn.easm | Adds SHR >=256 fold fixture bytecode |
| tests/evm_asm/shl_dyn_amount.expected | Adds expected output for SHL dynamic-amount path fixture |
| tests/evm_asm/shl_dyn_amount.easm | Adds SHL dynamic-amount path fixture bytecode |
| tests/evm_asm/shl_const_highlimb_dyn.expected | Adds expected output for SHL “high-limb shift amount” fold fixture |
| tests/evm_asm/shl_const_highlimb_dyn.easm | Adds SHL “high-limb shift amount” fold fixture bytecode |
| tests/evm_asm/shl_const96_dyn.expected | Adds expected output for SHL cross-limb const shift fixture |
| tests/evm_asm/shl_const96_dyn.easm | Adds SHL cross-limb const shift fixture bytecode |
| tests/evm_asm/shl_const4_dyn.expected | Adds expected output for SHL small const shift fixture |
| tests/evm_asm/shl_const4_dyn.easm | Adds SHL small const shift fixture bytecode |
| tests/evm_asm/shl_const256_dyn.expected | Adds expected output for SHL >=256 fold fixture |
| tests/evm_asm/shl_const256_dyn.easm | Adds SHL >=256 fold fixture bytecode |
| tests/evm_asm/shl_const200_u64val.expected | Adds expected output for SHL const over u64-tagged value (large shift) |
| tests/evm_asm/shl_const200_u64val.easm | Adds SHL const over u64-tagged value (large shift) fixture bytecode |
| tests/evm_asm/shl_const136_u64val.expected | Adds expected output for SHL const over u64-tagged value (carry-only limb) |
| tests/evm_asm/shl_const136_u64val.easm | Adds SHL const over u64-tagged value (carry-only limb) fixture bytecode |
| tests/evm_asm/sgt_dyn_neg_vs_const.expected | Adds expected output for SGT dyn-negative vs const fixture |
| tests/evm_asm/sgt_dyn_neg_vs_const.easm | Adds SGT dyn-negative vs const fixture bytecode |
| tests/evm_asm/sgt_dyn_msb64_vs_const.expected | Adds expected output for SGT msb64 edge fixture |
| tests/evm_asm/sgt_dyn_msb64_vs_const.easm | Adds SGT msb64 edge fixture bytecode |
| tests/evm_asm/sgt_dyn_highsparse_vs_const.expected | Adds expected output for SGT high-sparse limb fixture |
| tests/evm_asm/sgt_dyn_highsparse_vs_const.easm | Adds SGT high-sparse limb fixture bytecode |
| tests/evm_asm/sgt_const_vs_dyn.expected | Adds expected output for SGT const-vs-dyn swapped-dispatch fixture |
| tests/evm_asm/sgt_const_vs_dyn.easm | Adds SGT const-vs-dyn swapped-dispatch fixture bytecode |
| tests/evm_asm/sar_const8_neg.expected | Adds expected output for SAR const shift on negative value fixture |
| tests/evm_asm/sar_const8_neg.easm | Adds SAR const shift on negative value fixture bytecode |
| tests/evm_asm/sar_const64_pos.expected | Adds expected output for SAR const shift on positive value fixture |
| tests/evm_asm/sar_const64_pos.easm | Adds SAR const shift on positive value fixture bytecode |
| tests/evm_asm/or_dyn_u64_wide.expected | Adds expected output for OR u64-vs-wide fixture |
| tests/evm_asm/or_dyn_u64_wide.easm | Adds OR u64-vs-wide adversarial fixture bytecode |
| tests/evm_asm/or_dyn_u64_u64.expected | Adds expected output for OR u64-vs-u64 fixture |
| tests/evm_asm/or_dyn_u64_u64.easm | Adds OR u64-vs-u64 fixture bytecode |
| tests/evm_asm/jumpi_u64_cond_taken.expected | Adds expected output for JUMPI on u64-tagged nonzero condition |
| tests/evm_asm/jumpi_u64_cond_taken.easm | Adds JUMPI on u64-tagged nonzero condition fixture bytecode |
| tests/evm_asm/jumpi_u64_cond_nottaken.expected | Adds expected output for JUMPI on u64-tagged zero condition |
| tests/evm_asm/jumpi_u64_cond_nottaken.easm | Adds JUMPI on u64-tagged zero condition fixture bytecode |
| tests/evm_asm/jumpi_iszero_iszero_fused.expected | Adds expected output for ISZERO/ISZERO fusion + JUMPI fixture |
| tests/evm_asm/jumpi_iszero_iszero_fused.easm | Adds ISZERO/ISZERO fusion + JUMPI fixture bytecode |
| tests/evm_asm/jumpi_iszero_fused_taken.expected | Adds expected output for deferred-ISZERO fusion (taken) fixture |
| tests/evm_asm/jumpi_iszero_fused_taken.easm | Adds deferred-ISZERO fusion (taken) fixture bytecode |
| tests/evm_asm/jumpi_iszero_fused_nottaken_highsparse.expected | Adds expected output for deferred-ISZERO fusion (not taken, high-sparse) |
| tests/evm_asm/jumpi_iszero_fused_nottaken_highsparse.easm | Adds deferred-ISZERO fusion (not taken, high-sparse) fixture bytecode |
| tests/evm_asm/iszero_dyn_u64_nonzero.expected | Adds expected output for ISZERO on u64-tagged nonzero dynamic |
| tests/evm_asm/iszero_dyn_u64_nonzero.easm | Adds ISZERO on u64-tagged nonzero dynamic fixture bytecode |
| tests/evm_asm/iszero_dyn_highsparse.expected | Adds expected output for ISZERO on high-sparse dynamic value |
| tests/evm_asm/iszero_dyn_highsparse.easm | Adds ISZERO on high-sparse dynamic fixture bytecode |
| tests/evm_asm/iszero_calldatasize.expected | Adds expected output for ISZERO(CALLDATASIZE) fixture |
| tests/evm_asm/iszero_calldatasize.easm | Adds ISZERO(CALLDATASIZE) fixture bytecode |
| src/tests/evm_differential_tests.cpp | New consolidated differential test suite (fixtures + matrix harness) |
| src/tests/CMakeLists.txt | Adds and wires evmDifferentialTests into build + ctest |
| docs/changes/2026-06-11-evm-differential-suite/README.md | Documents the consolidated differential suite and verification |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
⚡ Performance Regression Check Results✅ Performance Check Passed (interpreter)Performance Benchmark Results (threshold: 25%)
Summary: 194 benchmarks, 0 regressions ✅ Performance Check Passed (multipass)Performance Benchmark Results (threshold: 25%)
Summary: 194 benchmarks, 0 regressions |
Add explicit standard-library includes instead of relying on transitive ones, and make the matrix harness assert the interpreter run succeeds so a shared failure (e.g. out-of-gas in both engines) cannot pass the differential check vacuously. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nts) Align the doc with the technical-writing rule: describe internal labels by behavior, remove who-reviewed and process narrative, and keep every count, flag, code anchor, and measurement. No code or runtime change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cen5bPpPEgkSkcxWWTSY7d
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Consolidate the interpreter-vs-multipass differential tests into one gtest target.
What
Adds a gtest target,
evmDifferentialTests(src/tests/evm_differential_tests.cpp), consolidating the interpreter-vs-multipass differential tests into one self-contained file:.easm/.expectedpairs undertests/evm_asm/) in three groups —RangeNarrowing(21),ConstShiftPruning(13),SubWrapU64(6) — each loading a hex fixture, running both engines, and asserting identical status and output.EVMRangeDifferential) that build bytecode programmatically and sweep an 11-value operand matrix (value-range boundaries plus high-sparse words) across 20 binary opcodes, the three shifts on limb-boundary amounts, and the AND-u64 narrow fast path.One shared executor underpins both parts. It forces synchronous multipass compilation (
DisableMultipassMultithread) and assertsJITCompiled, so a multipass run that silently fell back to the interpreter cannot pass vacuously.Why
Differential JIT tests do not belong in the interpreter test file. Three in-flight optimization PRs (#534, #535, #536) each appended a near-identical copy of the differential suite plus fixtures to
src/tests/evm_interp_tests.cpp, and draft #533 added the matrix harness to the same file; on merge each copy would conflict with the others. This PR places all of that coverage in one dedicated target. The optimization PRs have been trimmed to code-only, and #533 is superseded by this PR.The differential invariant — interpreter output equals multipass output, and multipass actually JIT-compiles — holds on plain
main, so this suite passes independently of the optimization PRs and can merge in any order relative to them.Verification
evmDifferentialTests: 44/44 pass on plainmain(no optimization PR applied).ctest(SPEC_TESTS_ARGS="-m multipass --format evm --enable-evm-gas"): 12/12 pass;evmInterpTestsadditionally runs the 40 new fixtures through its golden sample test (198/198).tools/format.sh check: pass. No new compiler warnings.Change doc:
docs/changes/2026-06-11-evm-differential-suite/README.md.🤖 Generated with Claude Code