diff --git a/.claude/commands/edit-workflow.md b/.claude/commands/edit-workflow.md index f576b8572fb..9c906e5ebfa 100644 --- a/.claude/commands/edit-workflow.md +++ b/.claude/commands/edit-workflow.md @@ -15,4 +15,4 @@ uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2 ## Validation -Run `uvx tox -e static` before committing — this runs `actionlint` to validate YAML syntax and structure. +Run `just lint-actions` before committing to validate YAML syntax and structure. diff --git a/.claude/commands/lint.md b/.claude/commands/lint.md index 9d6f54ddac0..bc5f3b12f84 100644 --- a/.claude/commands/lint.md +++ b/.claude/commands/lint.md @@ -5,18 +5,17 @@ Run the full static analysis suite and fix issues. This matches the CI check on ## Step 1: Run the Full Check ```bash -uvx tox -e static +just static ``` This runs ruff, mypy, codespell, ethereum-spec-lint, and actionlint in one pass. If everything passes, you're done. ## Step 2: Auto-Fix Formatting and Lint Issues -If tox reports ruff errors, run these first — they resolve most issues automatically: +If static checks report ruff errors, run the fix recipe first — it resolves most issues automatically: ```bash -uv run ruff format -uv run ruff check --fix +just fix ``` ## Step 3: Resolve Remaining Issues Manually @@ -24,12 +23,12 @@ uv run ruff check --fix After auto-fix, re-run to see what's left: ```bash -uvx tox -e static +just static ``` - **Remaining ruff issues**: fix manually (auto-fix can't handle all rules) - **mypy errors**: fix type annotations, add missing types, correct signatures -- **codespell errors**: fix typos, or add intentional words to `whitelist.txt` +- **codespell errors**: fix typos, or add intentional words via `just whitelist ` - **ethereum-spec-lint errors**: fix import isolation violations (see `/implement-eip` for import rules) - **actionlint errors**: fix workflow YAML issues (see `/edit-workflow`) @@ -38,5 +37,5 @@ uvx tox -e static Re-run until clean: ```bash -uvx tox -e static +just static ``` diff --git a/.claude/commands/pytester.md b/.claude/commands/pytester.md new file mode 100644 index 00000000000..a1d23d78303 --- /dev/null +++ b/.claude/commands/pytester.md @@ -0,0 +1,36 @@ +# Pytester + +Guide for pytester-based plugin/CLI tests. Run before writing or modifying these tests. + +## Which execution mode to use + +- **`runpytest()`** — default. In-process, fast, full `RunResult` API (`assert_outcomes()`, `fnmatch_lines()`). +- **`runpytest_subprocess()`** — use only when in-process causes state leakage (Pydantic cache pollution, global mutation in `pytest_configure`). Same `RunResult` API. +- **Raw `subprocess.run()`** — never use alongside pytester. Use `runpytest_subprocess()` instead. + +Subprocess isolation masks bugs rather than fixing them. Prefer fixing the root cause and use subprocess as defense-in-depth. + +## Expected inner failures + +`runpytest_subprocess()` replays inner output to outer stdout (by design). Suppress with `capsys.readouterr()`: + +```python +def test_expected_failure(pytester: Any, capsys: Any, pytestconfig: Any) -> None: + result = pytester.runpytest_subprocess(...) + capsys.readouterr() # suppress inner failure bleed + assert result.ret != 0 + output = "\n".join(result.outlines + result.errlines) + # conditional print for -s debugging + if pytestconfig.getoption("capture") == "no": + with capsys.disabled(): + print(output) +``` + +## RunResult API + +Prefer `assert_outcomes()` and `fnmatch_lines()` over manual `any(... in line ...)` — better failure messages. + +- `result.ret` — exit code +- `result.outlines` / `result.errlines` — line lists +- `result.assert_outcomes(passed=N, failed=N)` +- `result.stdout.fnmatch_lines(["*pattern*"])` diff --git a/.claude/commands/write-docstring.md b/.claude/commands/write-docstring.md new file mode 100644 index 00000000000..2492cb0b6ff --- /dev/null +++ b/.claude/commands/write-docstring.md @@ -0,0 +1,290 @@ +# Write Docstring + +Conventions for writing docstrings in `src/ethereum/`. Docstrings are the primary prose of the specification — they read as a narrative explaining how Ethereum works, not as traditional Python API documentation. They are rendered into HTML by docc, which parses them as **Markdown** (via mistletoe). Run this skill before writing or modifying docstrings. + +## General Rules + +- **Markdown only** — no reStructuredText (`.. directives::`, `:param:`, RST section underlines) +- 79-character line limit (same as code) +- Imperative mood for summaries ("Obtain" not "Obtains", "Return" not "Returns") +- Summary on the line after opening `"""` +- Blank line after the summary for multi-line docstrings +- For multi-line docstrings, the closing `"""` should be on its own line +- No `__init__` docstrings (D107 is disabled), only the class itself is documented +- Reference link definitions go at the end of the docstring, after a blank line +- Do not include constants/numeric values in the docstring (values in docstrings can easily desync with the code, and no tool will detect it) +- Avoid restating what the code is doing (the code should speak for itself) +- Avoid mentioning the current fork unnecessarily (creates noisy diffs between forks) +- Docstrings should be reserved for meaningful specification, while comments (`# ...`) can be used to explain particulars of the Python reference implementation + +## Module Docstrings + +Module docstrings introduce the concepts in the module. They should read as narrative prose — imagine a textbook chapter opening. + +Start with a one-line summary, then expand with paragraphs that explain what the module contains and why. Use cross-references to link to the key types and functions defined in the module. + +```python +""" +Ethash is a proof-of-work algorithm designed to be [ASIC] resistant through +[memory hardness][mem-hard]. + +To achieve memory hardness, computing Ethash requires access to subsets of a +large structure. The particular subsets chosen are based on the nonce and block +header, while the set itself is changed every [`epoch`]. + +At a high level, the Ethash algorithm is as follows: + +1. Create a **seed** value, generated with [`generate_seed`] and based on the + preceding block numbers. +1. From the seed, compute a pseudorandom **cache** with [`generate_cache`]. +1. From the cache, generate a **dataset** with [`generate_dataset`]. The + dataset grows over time based on [`DATASET_EPOCH_GROWTH_SIZE`]. +1. Miners hash slices of the dataset together, which is where the memory + hardness is introduced. Verification of the proof-of-work only requires the + cache to be able to recompute a much smaller subset of the full dataset. + +[`DATASET_EPOCH_GROWTH_SIZE`]: ref:ethereum.ethash.DATASET_EPOCH_GROWTH_SIZE +[`generate_dataset`]: ref:ethereum.ethash.generate_dataset +[`generate_cache`]: ref:ethereum.ethash.generate_cache +[`generate_seed`]: ref:ethereum.ethash.generate_seed +[`epoch`]: ref:ethereum.ethash.epoch +[ASIC]: https://en.wikipedia.org/wiki/Application-specific_integrated_circuit +[mem-hard]: https://en.wikipedia.org/wiki/Memory-hard_function +""" +``` + +Short modules that need no narrative can use a single-line summary: + +```python +""" +Utility functions used in this specification. +""" +``` + +## Function Docstrings + +Function docstrings describe what the function does and why, as part of the specification narrative. Reference parameters inline with backticks — do **not** use formal `Parameters`, `Returns`, or `Raises` sections. + +### Short (summary only) + +```python +def convert(balance: str) -> U256: + """ + Convert a string in either hexadecimal or base-10 to a `U256`. + """ +``` + +### Multi-paragraph (with context) + +```python +def add_genesis_block( + hardfork: GenesisFork, chain: Any, genesis: GenesisConfiguration +) -> None: + """ + Add the genesis block to an empty blockchain. + + The genesis block is an entirely sui generis block (unique) that is not + governed by the general rules applying to all other Ethereum blocks. + Instead, the only consensus requirement is that it must be identical to + the block added by this function. + + The initial state is populated with balances based on the Ethereum presale + that happened on the Bitcoin blockchain. Additional ether worth 1.98% of + the presale was given to the foundation. + + The `nonce` field is `0x42` referencing Douglas Adams' "HitchHiker's Guide + to the Galaxy". + + On testnets the genesis configuration usually allocates 1 wei to addresses + `0x00` to `0xFF` to avoid edge cases around precompiles being created or + cleared (by [EIP-161]). + + [EIP-161]: https://eips.ethereum.org/EIPS/eip-161 + """ +``` + +### With cross-references + +```python +def cache_size(block_number: Uint) -> Uint: + """ + Obtain the cache size (in bytes) of the epoch to which `block_number` + belongs. + + See [`INITIAL_CACHE_SIZE`] and [`CACHE_EPOCH_GROWTH_SIZE`] for the initial + size and linear growth rate, respectively. The cache is generated in + [`generate_cache`]. + + The actual cache size is smaller than simply multiplying + `CACHE_EPOCH_GROWTH_SIZE` by the epoch number to minimize the risk of + unintended cyclic behavior. It is defined as the highest prime number below + what linear growth would calculate. + + [`INITIAL_CACHE_SIZE`]: ref:ethereum.ethash.INITIAL_CACHE_SIZE + [`CACHE_EPOCH_GROWTH_SIZE`]: ref:ethereum.ethash.CACHE_EPOCH_GROWTH_SIZE + [`generate_cache`]: ref:ethereum.ethash.generate_cache + """ +``` + +## Class Docstrings + +Brief summary of what the class represents, with optional narrative and cross-references. + +```python +class GenesisConfiguration: + """ + Configuration for the first block of an Ethereum chain. + + Specifies the allocation of ether set out in the pre-sale, and some of + the fields of the genesis block. + """ +``` + +```python +class EvmTracer(Protocol): + """ + [`Protocol`] that describes tracer functions. + + See [`ethereum.trace`] for details about tracing in general, and + [`__call__`] for more on how to implement a tracer. + + [`Protocol`]: https://docs.python.org/3/library/typing.html#typing.Protocol + [`ethereum.trace`]: ref:ethereum.trace + [`__call__`]: ref:ethereum.trace.EvmTracer.__call__ + """ +``` + +## Attribute Docstrings + +docc documents any assignment that is followed by a bare string literal. This is non-standard Python — normally only modules, classes, and functions can have docstrings. Place a triple-quoted string immediately after the assignment. + +This works for **constants**, **class fields**, **module-level variables**, and **type aliases**. + +### Constants + +```python +EPOCH_SIZE = Uint(30000) +""" +Number of blocks before a dataset needs to be regenerated (known as an +"epoch".) See [`epoch`]. + +[`epoch`]: ref:ethereum.ethash.epoch +""" +``` + +### Class fields + +```python +class Example: + chain_id: U64 + """ + Discriminant between diverged blockchains; `1` for Ethereum's main network. + """ +``` + +### Module-level variables + +```python +_evm_trace: EvmTracer = discard_evm_trace +""" +Active [`EvmTracer`] that is used for generating traces. + +[`EvmTracer`]: ref:ethereum.trace.EvmTracer +""" +``` + +### Type aliases + +```python +TraceEvent = ( + TransactionStart + | TransactionEnd + | PrecompileStart + | PrecompileEnd + | OpStart + | OpEnd + | OpException + | EvmStop + | GasAndRefund +) +""" +All possible types of events that an [`EvmTracer`] is expected to handle. + +[`EvmTracer`]: ref:ethereum.trace.EvmTracer +""" +``` + +## Cross-References + +docc resolves Markdown reference links with the `ref:` scheme into hyperlinks in the generated documentation. + +### Internal (to other Python objects) + +Use backtick-wrapped names as the link text, with `ref:` pointing to the fully-qualified path: + +``` +[`ForkCriteria`]: ref:ethereum.fork_criteria.ForkCriteria +[`generate_cache`]: ref:ethereum.ethash.generate_cache +``` + +Short aliases work when the full name is unwieldy: + +``` +[ds]: ref:ethereum.ethash.DATASET_EPOCH_GROWTH_SIZE +``` + +### External URLs + +Standard Markdown reference links: + +``` +[ASIC]: https://en.wikipedia.org/wiki/Application-specific_integrated_circuit +[EIP-3155]: https://eips.ethereum.org/EIPS/eip-3155 +``` + +Bare URLs in angle brackets for inline use: + +``` +Available at . +``` + +If a URL is too long to include because of the line length limit, you can add `# noqa: E501` after the trailing `"""` to squelch the warning (but this should be a last resort). + +### Usage in text + +Reference links are used inline with brackets: + +``` +For these intentional forks to succeed, all participants need to agree on +exactly when to switch rules. The agreed upon criteria are represented by +subclasses of [`ForkCriteria`], like [`ByBlockNumber`] and [`ByTimestamp`]. +``` + +## Markdown Formatting + +- `_italic_` to introduce domain terms: `_Genesis_ is the term for...` +- `**bold**` to highlight key concepts: `Create a **seed** value` +- Backticks for code references: `` `block_number` ``, `` `0x42` `` +- Numbered lists (`1.`) for sequential steps +- Bullet lists (`-`) for unordered items +- Markdown headings are rarely needed inside docstrings; use paragraphs instead + +## Anti-Patterns + +- **No RST directives**: `.. contents::`, `.. note::`, `:param:`, `:returns:` — these are outdated +- **No NumPy/Google sections**: no `Parameters\n----------` or `Args:` blocks +- **No RST section underlines**: `Introduction\n------------` is RST, not Markdown +- **No type repetition in docstrings**: types come from annotations, not prose +- **No empty boilerplate**: don't write `"""Ethereum Specification."""` with a `.. contents::` block — write real narrative or a concise summary +- **Don't skip attribute docstrings**: constants and fields deserve explanations + +## Reference Files + +For examples of well-written docstrings, see: + +- `src/ethereum/ethash.py` — narrative module + function docstrings +- `src/ethereum/genesis.py` — class, attribute, and multi-paragraph function docstrings +- `src/ethereum/trace.py` — class, attribute, and protocol docstrings +- `src/ethereum/fork_criteria.py` — narrative module docstring with Markdown formatting + +If these files no longer exist or are no longer good examples, abort with an appropriate error message. diff --git a/.claude/commands/write-test.md b/.claude/commands/write-test.md index 5fc15c6c31d..c3b654cdc7e 100644 --- a/.claude/commands/write-test.md +++ b/.claude/commands/write-test.md @@ -61,6 +61,10 @@ Conventions and patterns for writing consensus tests. Run this skill before writ - `@pytest.mark.parametrize("name", [pytest.param(val, id="label"), ...])` with descriptive `id=` strings - Stack parametrize decorators for multiple dimensions +## Unit Tests (execution_testing package) + +Plain pytest. Tests are co-located with each module under `packages/testing/src/execution_testing/` in a sibling `tests/` directory. When adding a guardrail or validation, verify the tests fail without the change and pass with it. + ## After Writing Tests After writing or modifying tests, ask the user: "Would you like me to load the `/fill-tests` skill to verify the new tests fill correctly? (This loads an additional skill into context.)" If they agree, run `/fill-tests`, fill the new tests, then inspect the generated fixture JSON to verify the fixture contents match what the test intends. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 0d5b8b038dc..d0e5e1a21b7 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -9,9 +9,9 @@ N/A. ## ✅ Checklist -- [ ] All: Ran fast `tox` checks to avoid unnecessary CI fails, see also [Code Standards](https://eest.ethereum.org/main/getting_started/code_standards/) and [Enabling Pre-commit Checks](https://eest.ethereum.org/main/dev/precommit/): +- [ ] All: Ran fast static checks to avoid unnecessary CI fails, see also [Code Standards](https://eest.ethereum.org/main/getting_started/code_standards/) and [Enabling Pre-commit Checks](https://eest.ethereum.org/main/dev/precommit/): ```console - uvx tox -e static + just static ``` - [ ] All: PR title adheres to the [repo standard](https://eest.ethereum.org/main/getting_started/contributing/?h=contri#commit-messages-issue-and-pr-titles) - it will be used as the squash commit message and should start `type(scope):`. - [ ] All: Considered updating the online docs in the [./docs/](/ethereum/execution-specs/blob/HEAD/docs/) directory. diff --git a/.github/actions/build-fixtures/action.yaml b/.github/actions/build-fixtures/action.yaml index 683c6cebace..71800e90667 100644 --- a/.github/actions/build-fixtures/action.yaml +++ b/.github/actions/build-fixtures/action.yaml @@ -4,6 +4,15 @@ inputs: release_name: description: "Name of the fixture release" required: true + from_fork: + description: "Fill from this fork (inclusive). Empty for unsplit builds." + default: "" + until_fork: + description: "Fill until this fork (inclusive). Empty for unsplit builds." + default: "" + split_label: + description: "Label for this fork-range split. Empty for unsplit builds." + default: "" runs: using: "composite" steps: @@ -25,17 +34,41 @@ runs: with: type: ${{ steps.properties.outputs.evm-type }} - name: Install pigz for parallel tarball compression + if: inputs.split_label == '' shell: bash run: sudo apt-get install -y pigz - name: Generate fixtures using fill shell: bash run: | + IS_SPLIT="${{ inputs.split_label }}" + + if [ -n "$IS_SPLIT" ]; then + OUTPUT_ARG="--output=fixtures_${{ inputs.release_name }}" + FORK_ARGS="--from=${{ inputs.from_fork }} --until=${{ inputs.until_fork }}" + else + OUTPUT_ARG="--output=fixtures_${{ inputs.release_name }}.tar.gz" + FORK_ARGS="" + fi + + # Allow exit code 5 (NO_TESTS_COLLECTED) for fork ranges with no tests. + EXIT_CODE=0 if [ "${{ steps.evm-builder.outputs.impl }}" = "eels" ]; then - uv run fill -n ${{ steps.evm-builder.outputs.x-dist }} ${{ steps.properties.outputs.fill-params }} --output=fixtures_${{ inputs.release_name }}.tar.gz --build-name ${{ inputs.release_name }} --no-html --durations=100 --log-level=DEBUG + uv run fill -n ${{ steps.evm-builder.outputs.x-dist }} ${{ steps.properties.outputs.fill-params }} $FORK_ARGS $OUTPUT_ARG --build-name ${{ inputs.release_name }} --no-html --durations=100 --log-level=DEBUG || EXIT_CODE=$? else - uv run fill -n ${{ steps.evm-builder.outputs.x-dist }} --evm-bin=${{ steps.evm-builder.outputs.evm-bin }} ${{ steps.properties.outputs.fill-params }} --output=fixtures_${{ inputs.release_name }}.tar.gz --build-name ${{ inputs.release_name }} --no-html --durations=100 --log-level=DEBUG + uv run fill -n ${{ steps.evm-builder.outputs.x-dist }} --evm-bin=${{ steps.evm-builder.outputs.evm-bin }} ${{ steps.properties.outputs.fill-params }} $FORK_ARGS $OUTPUT_ARG --build-name ${{ inputs.release_name }} --no-html --durations=100 --log-level=DEBUG || EXIT_CODE=$? + fi + if [ "$EXIT_CODE" -ne 0 ] && [ "$EXIT_CODE" -ne 5 ]; then + exit "$EXIT_CODE" fi - uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0 with: name: fixtures_${{ inputs.release_name }} path: fixtures_${{ inputs.release_name }}.tar.gz + - name: Upload fixture directory (split) + if: inputs.split_label != '' + uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0 + with: + name: fixtures__${{ inputs.split_label }} + include-hidden-files: true + path: fixtures_${{ inputs.release_name }}/ + if-no-files-found: ignore diff --git a/.github/actions/setup-uv/action.yaml b/.github/actions/setup-uv/action.yaml index 578d7fde3b0..cf251242a69 100644 --- a/.github/actions/setup-uv/action.yaml +++ b/.github/actions/setup-uv/action.yaml @@ -1,5 +1,5 @@ -name: Setup uv and tox -description: Install uv, Python, and tox for CI jobs +name: Setup uv and just +description: Install uv, Python, and just for CI jobs inputs: python-version: description: Python version to install (e.g. "3.14", "pypy3.11") @@ -14,7 +14,7 @@ runs: using: "composite" steps: - name: Install uv and python ${{ inputs.python-version }} - uses: astral-sh/setup-uv@e06108dd0aef18192324c70427afc47652e63a82 # v7.5.0 + uses: astral-sh/setup-uv@e06108dd0aef18192324c70427afc47652e63a82 # v7.5.0 with: enable-cache: ${{ inputs.enable-cache }} cache-dependency-glob: ${{ inputs.cache-dependency-glob }} @@ -23,6 +23,7 @@ runs: - name: Prefer uv-managed Python over system Python shell: bash run: echo "UV_PYTHON_PREFERENCE=only-managed" >> "$GITHUB_ENV" - - name: Install tox - shell: bash - run: uv tool install tox + - name: Install just + uses: taiki-e/install-action@e24b8b7a939c6a537188f34a4163cb153dd85cf6 # v2.69.1 + with: + tool: just@1.46 diff --git a/.github/configs/evm.yaml b/.github/configs/evm.yaml index d8ec22827a5..66df506394a 100644 --- a/.github/configs/evm.yaml +++ b/.github/configs/evm.yaml @@ -1,8 +1,4 @@ -stable: - impl: eels - repo: null - ref: null -develop: +eels: impl: eels repo: null ref: null diff --git a/.github/configs/feature.yaml b/.github/configs/feature.yaml index 4fd98921c2c..4e4822c1579 100644 --- a/.github/configs/feature.yaml +++ b/.github/configs/feature.yaml @@ -1,11 +1,7 @@ # Unless filling for special features, all features should fill for previous forks (starting from Frontier) too -stable: - evm-type: stable - fill-params: --until=Prague --fill-static-tests --ignore=tests/static/state_tests/stQuadraticComplexityTest - -develop: - evm-type: develop - fill-params: --until=BPO4 --fill-static-tests --ignore=tests/static/state_tests/stQuadraticComplexityTest +mainnet: + evm-type: eels + fill-params: --until=BPO2 --generate-all-formats monad: evm-type: develop diff --git a/.github/configs/fork-ranges.yaml b/.github/configs/fork-ranges.yaml new file mode 100644 index 00000000000..feb5ed48e38 --- /dev/null +++ b/.github/configs/fork-ranges.yaml @@ -0,0 +1,21 @@ +# Shared fork ranges for splitting multi-fork releases across parallel runners. +# Features using --until are automatically split using applicable ranges. +# Features using --fork (single fork) are never split. +- label: pre-cancun + from: Frontier + until: Shanghai +- label: cancun + from: Cancun + until: Cancun +- label: prague + from: Prague + until: Prague +- label: osaka + from: Osaka + until: Osaka +- label: bpo + from: BPO1 + until: BPO2 +- label: amsterdam + from: Amsterdam + until: Amsterdam diff --git a/.github/scripts/create_release_tarball.py b/.github/scripts/create_release_tarball.py new file mode 100644 index 00000000000..1f4e7d47a2f --- /dev/null +++ b/.github/scripts/create_release_tarball.py @@ -0,0 +1,84 @@ +#!/usr/bin/env -S uv run --script +# +# /// script +# requires-python = ">=3.12" +# dependencies = [] +# /// +""" +Create a release tarball from a merged fixture directory. + +Archive all ``.json`` and ``.ini`` files under a ``fixtures/`` prefix, +matching the structure produced by +``execution_testing.cli.pytest_commands.plugins.shared.fixture_output``. + +Use ``pigz`` for parallel compression when available, otherwise fall +back to Python's built-in gzip. +""" + +import shutil +import subprocess +import sys +import tarfile +import warnings +from pathlib import Path + + +def create_tarball_with_pigz(source_dir: Path, output_path: Path) -> None: + """Create tarball using Python tarfile + pigz for parallel compression.""" + temp_tar = output_path.with_suffix("") # strip .gz + + with tarfile.open(temp_tar, "w") as tar: + for file in sorted(source_dir.rglob("*")): + if file.is_file() and file.suffix in {".json", ".ini"}: + arcname = Path("fixtures") / file.relative_to(source_dir) + tar.add(file, arcname=str(arcname)) + + subprocess.run( + ["pigz", "-f", str(temp_tar)], + check=True, + capture_output=True, + ) + + +def create_tarball_standard(source_dir: Path, output_path: Path) -> None: + """Create tarball using Python's tarfile module (single-threaded).""" + with tarfile.open(output_path, "w:gz") as tar: + for file in sorted(source_dir.rglob("*")): + if file.is_file() and file.suffix in {".json", ".ini"}: + arcname = Path("fixtures") / file.relative_to(source_dir) + tar.add(file, arcname=str(arcname)) + + +def main() -> None: + """Entry point.""" + if len(sys.argv) != 3: + print( + "Usage: create_release_tarball.py ", + file=sys.stderr, + ) + sys.exit(1) + + source_dir = Path(sys.argv[1]) + output_path = Path(sys.argv[2]) + + if not source_dir.is_dir(): + print(f"Error: '{source_dir}' is not a directory.", file=sys.stderr) + sys.exit(1) + + if shutil.which("pigz"): + try: + create_tarball_with_pigz(source_dir, output_path) + except (subprocess.CalledProcessError, OSError) as e: + warnings.warn( + f"pigz failed ({type(e).__name__}: {e}), falling back to gzip", + stacklevel=2, + ) + create_tarball_standard(source_dir, output_path) + else: + create_tarball_standard(source_dir, output_path) + + print(f"Created {output_path}") + + +if __name__ == "__main__": + main() diff --git a/.github/scripts/generate_build_matrix.py b/.github/scripts/generate_build_matrix.py new file mode 100644 index 00000000000..de8b3b5144a --- /dev/null +++ b/.github/scripts/generate_build_matrix.py @@ -0,0 +1,160 @@ +#!/usr/bin/env -S uv run --script +# +# /// script +# requires-python = ">=3.12" +# dependencies = [ +# "pyyaml", +# ] +# /// +""" +Generate the build matrix for release fixture workflows. + +Read `.github/configs/feature.yaml` and emit a flat JSON build matrix +suitable for ``strategy.matrix`` in GitHub Actions. + +Features whose ``fill-params`` contain ``--until`` are split across the +shared fork ranges defined in `.github/configs/fork-ranges.yaml`. +Features using ``--fork`` (single fork) produce a single unsplit entry. +""" + +import json +import re +import sys +from pathlib import Path + +import yaml + +FEATURE_CONFIG = Path(".github/configs/feature.yaml") +FORK_RANGES_CONFIG = Path(".github/configs/fork-ranges.yaml") + +# Canonical fork ordering used to filter fork ranges per feature. +FORK_ORDER = [ + "Frontier", + "Homestead", + "DAOFork", + "TangerineWhistle", + "SpuriousDragon", + "Byzantium", + "Constantinople", + "Istanbul", + "MuirGlacier", + "Berlin", + "London", + "ArrowGlacier", + "GrayGlacier", + "Paris", + "Shanghai", + "Cancun", + "Prague", + "Osaka", + "BPO1", + "BPO2", + "Amsterdam", +] + +FORK_INDEX = {name: i for i, name in enumerate(FORK_ORDER)} + + +def load_config(path: Path) -> dict: + """Load and return the feature configuration.""" + with open(path) as f: + return yaml.safe_load(f) + + +def parse_until_fork(fill_params: str) -> str | None: + """ + Extract the ``--until`` value from fill-params. + + Return ``None`` when ``--fork`` is used instead (single-fork + feature that should not be split). + """ + if re.search(r"--fork\b", fill_params): + return None + m = re.search(r"--until[=\s]+(\S+)", fill_params) + return m.group(1) if m else None + + +def applicable_ranges(fork_ranges: list[dict], until_fork: str) -> list[dict]: + """ + Return fork ranges whose ``from`` is at or before *until_fork*. + + Clamp the last applicable range's ``until`` to *until_fork* so we + never fill beyond the feature's declared boundary. + """ + limit = FORK_INDEX[until_fork] + result = [] + for r in fork_ranges: + if FORK_INDEX[r["from"]] <= limit: + entry = dict(r) + if FORK_INDEX[r["until"]] > limit: + entry["until"] = until_fork + result.append(entry) + return result + + +def build_matrix( + feature: dict, name: str, fork_ranges: list[dict] +) -> tuple[list[dict], str]: + """ + Build the matrix for a single feature. + + Return (build_entries, combine_labels). Split features produce + one entry per fork range and a space-separated label string for + the combine step. Unsplit features produce a single entry with + empty labels. + """ + until = parse_until_fork(feature["fill-params"]) + if until and fork_ranges: + ranges = applicable_ranges(fork_ranges, until) + if len(ranges) > 1: + build = [ + { + "feature": name, + "label": r["label"], + "from_fork": r["from"], + "until_fork": r["until"], + } + for r in ranges + ] + labels = " ".join(r["label"] for r in ranges) + return build, labels + + return [ + { + "feature": name, + "label": "", + "from_fork": "", + "until_fork": "", + } + ], "" + + +def main() -> None: + """Entry point.""" + if len(sys.argv) != 2: + print( + "Usage: generate_build_matrix.py ", + file=sys.stderr, + ) + sys.exit(1) + + config = load_config(FEATURE_CONFIG) + fork_ranges = load_config(FORK_RANGES_CONFIG) or [] + name = sys.argv[1] + + if name not in config or not isinstance(config[name], dict): + print( + f"Error: feature '{name}' not found in {FEATURE_CONFIG}.", + file=sys.stderr, + ) + sys.exit(1) + + build, labels = build_matrix(config[name], name, fork_ranges) + + print(f"build_matrix={json.dumps(build)}") + print(f"feature_name={name}") + print(f"combine_labels={labels}") + + +if __name__ == "__main__": + main() diff --git a/.github/scripts/merge_index_files.py b/.github/scripts/merge_index_files.py new file mode 100644 index 00000000000..ea41b299a10 --- /dev/null +++ b/.github/scripts/merge_index_files.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python3 +""" +Merge multiple .meta/index.json files from split fixture builds. + +Accept fixture directories as arguments, load each directory's +``.meta/index.json``, merge them via ``IndexFile.merge()``, and write +the result to the specified output path. +""" + +import sys +from pathlib import Path + +from execution_testing.fixtures.consume import IndexFile + + +def main() -> None: + """Entry point.""" + if len(sys.argv) < 3: + print( + "Usage: merge_index_files.py " + " [ ...]", + file=sys.stderr, + ) + sys.exit(1) + + output_path = Path(sys.argv[1]) + fixture_dirs = [Path(d) for d in sys.argv[2:]] + + indexes: list[IndexFile] = [] + for d in fixture_dirs: + index_path = d / ".meta" / "index.json" + if not index_path.exists(): + print(f"Skipping {d} (no .meta/index.json)") + continue + indexes.append(IndexFile.model_validate_json(index_path.read_text())) + + if not indexes: + print("No index files found, nothing to merge.") + sys.exit(0) + + merged = IndexFile.merge(indexes) + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text(merged.model_dump_json(indent=2)) + print(f"Merged {len(indexes)} index files ({merged.test_count} tests)") + + +if __name__ == "__main__": + main() diff --git a/.github/workflows/release_fixture_feature.yaml b/.github/workflows/release_fixture_feature.yaml index 72688ea26dd..dc959731263 100644 --- a/.github/workflows/release_fixture_feature.yaml +++ b/.github/workflows/release_fixture_feature.yaml @@ -1,4 +1,4 @@ -name: Create Fixture Feature Release +name: Create Fixture Release on: push: @@ -7,32 +7,35 @@ on: workflow_dispatch: jobs: - feature-names: + setup: runs-on: ubuntu-24.04 outputs: - names: ${{ steps.feature-name.outputs.names }} + build_matrix: ${{ steps.matrix.outputs.build_matrix }} + feature_name: ${{ steps.matrix.outputs.feature_name }} + combine_labels: ${{ steps.matrix.outputs.combine_labels }} steps: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: submodules: false + - uses: ./.github/actions/setup-uv - - name: Get feature names - id: feature-name + - name: Generate build matrix + id: matrix shell: bash run: | FEATURE_PREFIX="${GITHUB_REF_NAME//@*/}" FEATURE_NAME="${FEATURE_PREFIX#tests-}" - names=$(grep -Po "^${FEATURE_NAME}(?=:)" .github/configs/feature.yaml | jq --raw-input . | jq -c --slurp .) - echo "names=${names}" - echo "names=${names}" >> "$GITHUB_OUTPUT" + uv run -q .github/scripts/generate_build_matrix.py "$FEATURE_NAME" >> "$GITHUB_OUTPUT" build: - needs: feature-names + name: fill (${{ matrix.label || matrix.feature }}) + needs: setup runs-on: ubuntu-24.04 timeout-minutes: 1440 strategy: + fail-fast: true matrix: - feature: ${{ fromJSON(needs.feature-names.outputs.names) }} + include: ${{ fromJson(needs.setup.outputs.build_matrix) }} steps: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: @@ -41,19 +44,80 @@ jobs: - uses: ./.github/actions/build-fixtures with: release_name: ${{ matrix.feature }} + from_fork: ${{ matrix.from_fork }} + until_fork: ${{ matrix.until_fork }} + split_label: ${{ matrix.label }} + + combine: + name: combine (${{ needs.setup.outputs.feature_name }}) + needs: [setup, build] + if: needs.setup.outputs.combine_labels != '' + runs-on: ubuntu-24.04 + steps: + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 + with: + submodules: false + - uses: ./.github/actions/setup-uv + - name: Install pigz + run: sudo apt-get install -y pigz + - name: Download and merge split artifacts + shell: bash + run: | + mkdir -p combined + for label in ${{ needs.setup.outputs.combine_labels }}; do + echo "Downloading: fixtures__${label}" + if gh run download ${{ github.run_id }} -n "fixtures__${label}" --dir "split_artifacts/fixtures__${label}"; then + cp -r "split_artifacts/fixtures__${label}"/* combined/ + else + echo "No artifact for ${label} (no tests collected, skipping)" + fi + done + echo "Combined directory contents:" + find combined -maxdepth 3 -type d | head -30 || true + env: + GH_TOKEN: ${{ github.token }} + - name: Merge split index files + shell: bash + run: | + SPLIT_DIRS=() + for label in ${{ needs.setup.outputs.combine_labels }}; do + dir="split_artifacts/fixtures__${label}" + if [ -d "$dir" ]; then + SPLIT_DIRS+=("$dir") + fi + done + if [ ${#SPLIT_DIRS[@]} -gt 0 ]; then + uv run python .github/scripts/merge_index_files.py combined/.meta/index.json "${SPLIT_DIRS[@]}" + fi + - name: Free disk space + run: rm -rf split_artifacts/ + - name: Create release tarball + shell: bash + run: | + uv run -q .github/scripts/create_release_tarball.py combined fixtures_${{ needs.setup.outputs.feature_name }}.tar.gz + - name: Upload combined fixture tarball + uses: actions/upload-artifact@bbbca2ddaa5d8feaa63e36b76fdaad77386f024f # v7.0.0 + with: + name: fixtures_${{ needs.setup.outputs.feature_name }} + path: fixtures_${{ needs.setup.outputs.feature_name }}.tar.gz release: runs-on: ubuntu-24.04 - needs: build - if: startsWith(github.ref, 'refs/tags/tests-') + needs: [setup, build, combine] + if: always() && needs.build.result == 'success' && (needs.combine.result == 'success' || needs.combine.result == 'skipped') && startsWith(github.ref, 'refs/tags/tests-') + permissions: + contents: write steps: - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2 with: submodules: false + fetch-depth: 0 - - name: Download all artifacts + - name: Download release artifacts + shell: bash run: | - gh run download ${{ github.run_id }} --dir ./artifacts + gh run download ${{ github.run_id }} -p "fixtures_*" --dir ./artifacts + rm -rf ./artifacts/fixtures__*/ env: GH_TOKEN: ${{ github.token }} diff --git a/.github/workflows/stale.yml b/.github/workflows/stale.yml deleted file mode 100644 index c59d637e499..00000000000 --- a/.github/workflows/stale.yml +++ /dev/null @@ -1,30 +0,0 @@ -name: Manage stale issues and PRs -on: - schedule: - - cron: '0 0 * * *' # runs daily - -jobs: - stale: - runs-on: ubuntu-latest - steps: - - uses: actions/stale@b5d41d4e1d5dceea10e7104786b73624c18a190f # v10.2.0 - with: - # --- Stale settings --- - days-before-stale: 60 - - # raise this from the default 30 - operations-per-run: 200 - - # process oldest first - ascending: true - - # --- PR settings --- - days-before-pr-close: -1 # Never auto-close PRs - stale-pr-label: 'stale' - stale-pr-message: 'This PR has had no recent activity and has been marked as stale.' - close-pr-message: 'Closing this PR due to inactivity.' - - # --- Issue settings --- - days-before-issue-close: -1 # Never auto-close issues - stale-issue-label: 'stale' - stale-issue-message: 'This issue has had no recent activity and has been marked as stale.' diff --git a/.github/workflows/test.yaml b/.github/workflows/test.yaml index a0b07af8ea9..613a4f8000d 100644 --- a/.github/workflows/test.yaml +++ b/.github/workflows/test.yaml @@ -12,7 +12,7 @@ on: - ".gitignore" - ".vscode/**" - "whitelist.txt" - - "docs/**" + - "docs/**" - "mkdocs.yml" workflow_dispatch: pull_request: @@ -22,7 +22,7 @@ on: - ".gitignore" - ".vscode/**" - "whitelist.txt" - - "docs/**" + - "docs/**" - "mkdocs.yml" concurrency: @@ -42,8 +42,26 @@ jobs: - name: Install build dependencies shell: bash run: sudo DEBIAN_FRONTEND=noninteractive apt-get install --yes --force-yes build-essential pkg-config + - name: Detect Python version + id: python + shell: bash + run: echo "version=$(python3 -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')" >> "$GITHUB_OUTPUT" + - name: Restore mypy cache + uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3 + with: + path: .mypy_cache + key: mypy-${{ runner.os }}-py${{ steps.python.outputs.version }}-${{ hashFiles('uv.lock') }}-${{ github.sha }} + restore-keys: | + mypy-${{ runner.os }}-py${{ steps.python.outputs.version }}-${{ hashFiles('uv.lock') }}- + mypy-${{ runner.os }}-py${{ steps.python.outputs.version }}- - name: Run static checks - run: tox -e static + run: just static + - name: Save mypy cache + if: always() + uses: actions/cache/save@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3 + with: + path: .mypy_cache + key: mypy-${{ runner.os }}-py${{ steps.python.outputs.version }}-${{ hashFiles('uv.lock') }}-${{ github.sha }} - name: Validate workflow config variables run: | cat >> .github/actionlint.yaml << 'EOF' diff --git a/.gitignore b/.gitignore index 827d04d6a69..12f73d03081 100644 --- a/.gitignore +++ b/.gitignore @@ -8,6 +8,9 @@ # AI .claude/settings.local.json +CLAUDE.local.md +.mb/ +mb/ .devcontainer # Benchmark fixed opcode counts @@ -19,6 +22,7 @@ # Cache cached_downloads/ .cache +.mypy_cache .mypy_cache/ .ruff_cache/ __pycache__/ @@ -47,6 +51,8 @@ MANIFEST # Python virtual environment .venv/ venv/ +# User config; not imposed +.python-version # pycharm .idea/ @@ -65,6 +71,8 @@ pip-log.txt pip-delete-this-directory.txt .tox/ +.just/ +tests/json_loader/bench_gas_fixtures /doc/_autosummary @@ -89,3 +97,6 @@ logs/ env.yaml site/ + +# Temporary data; used for generated checklists +tmp/ diff --git a/CLAUDE.md b/CLAUDE.md index eb3a2e04fd0..c2aeb8fb7df 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ Ethereum Execution Layer Specification written in Python. This is a **specificat ## Tooling -- **uv** is the package manager. **tox** orchestrates test environments (`uvx tox -al`). +- **uv** is the package manager. **just** is the command runner (`just --list`). - The `execution_testing` package under `packages/testing/` is a UV workspace member. ## Linting @@ -42,17 +42,20 @@ When reviewing PRs that implement or test EIPs: ## When to Use Skills - Writing or modifying tests → run `/write-test` first +- Writing or modifying pytester-based plugin tests → run `/pytester` first - Filling test fixtures → run `/fill-tests` first - Implementing an EIP or modifying fork code in `src/` → run `/implement-eip` first - Modifying GitHub Actions workflows → run `/edit-workflow` first - Assessing EIP complexity or scope → run `/assess-eip` - Working on EIP test coverage or checklists → run `/eip-checklist` first - Checking if config/skills are stale → run `/audit-config` +- Writing or modifying docstrings in `src/ethereum/` → run `/write-docstring` first - Done with changes and ready to lint → run `/lint` ## Available Skills - `/write-test` — test writing patterns, fixtures, markers, bytecode helpers +- `/pytester` — pytester execution modes, isolation, output handling for plugin tests - `/fill-tests` — `fill` CLI reference, flags, debugging, benchmark tests - `/implement-eip` — fork structure, import rules, adding opcodes/precompiles/tx types - `/edit-workflow` — GitHub Actions conventions and version pinning @@ -60,4 +63,5 @@ When reviewing PRs that implement or test EIPs: - `/eip-checklist` — EIP testing checklist system for tracking coverage - `/lint` — full static analysis suite with auto-fix workflow - `/audit-config` — verify CLAUDE.md and skills are still accurate +- `/write-docstring` — narrative Markdown docstring conventions for the spec - `/grammar-check` — audit grammar in documentation and code comments diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bff6f1bd808..13dd3058c30 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,274 +1,46 @@ # Contribution Guidelines -Help is always welcome and there are plenty of options to contribute to the Ethereum Execution Layer Specifications (EELS). +Help is always welcome. The Ethereum Execution Layer Specifications (EELS) are a community effort and we appreciate support in the following areas: -In particular, we appreciate support in the following areas: - -- Reporting issues -- Fixing and responding to [issues](https://github.com/ethereum/execution-specs/issues), especially those tagged as [E-easy](https://github.com/ethereum/execution-specs/labels/E-easy) which are meant as introductory issues for external contributors. +- Reporting issues. +- Fixing and responding to [issues](https://github.com/ethereum/execution-specs/issues), especially those tagged [E-easy](https://github.com/ethereum/execution-specs/labels/E-easy), which are intended as introductory issues for external contributors. - Improving the documentation. > [!IMPORTANT] -> Generally, we do not assign issues to external contributors. If you want to work on an issue, you are very welcome to go ahead and make a pull request. We would, however, be happy to answer questions you may have before you start implementing. - -For details about EELS usage and building, please refer to the [README](https://github.com/ethereum/execution-specs/blob/master/README.md#usage) - -## Contribution Guidelines - -This specification aims to be: - -1. **Correct** - Describe the _intended_ behavior of the Ethereum blockchain, and any deviation from that is a bug. -2. **Complete** - Capture the entirety of _consensus critical_ parts of Ethereum. -3. **Accessible** - Prioritize readability, clarity, and plain language over performance and brevity. - -### Style - -#### Spelling and Naming - -- Attempt to use descriptive English words (or _very common_ abbreviations) in documentation and identifiers. -- Avoid using EIP numbers in identifiers, and prefer descriptive text instead (eg. `FeeMarketTransaction` instead of `Eip1559Transaction`). -- If necessary, there is a custom dictionary `whitelist.txt`. -- Avoid uninformative prefixes in identifiers (like `get_` or `compute_`). They don't add useful meaning and take up valuable real estate. - -#### Comments - -- Don't repeat what is obvious from the code. --
- (expand) Consider how future changes will interleave with yours, especially when creating semantic blocks. - -
Consider: - - - - - - - - - - - - - - - -
Fork TFork T+1
- - - - ```python - # EIP-1234: The dingus is the rate of fleep - dingus = a + b - dingus += c ^ d - dingus /= fleep(e) - ``` - - - - ```python - # EIP-1234: The dingus is the rate of fleep - dingus = a + b - - # EIP-4567: Frobulate the dingus - dingus = frobulate(dingus) - - dingus += c ^ d # <- - dingus /= fleep(e) # <- - ``` - -
- - The marked lines (`<-`) are now incorrectly attributed to EIP-4567 in Fork+1. Instead, omit the EIP identifier in the comments, and describe the changes introduced by the EIP in the function's docstrings. The rendered diffs will make it pretty obvious what's changed. -
- -#### Docstrings - -- Write in complete sentences, providing necessary background and context for the associated code. -- Function and method docstrings must use the imperative mood in the summary line. - - **Good:** Build the house using the provided lumber. - - **Bad:** Builds the house using the provided lumber. -- Always start with a single-line summary. When more detail is needed, use a multi-line docstring with a blank line after the summary line. - - **One-line summary:** - - ```python - """Return the pathname of the KOS root directory.""" - ``` - - - **Multi-line:** - - ```python - """ - Add a bloom entry to the bloom filter. - - The number of hash functions used is 3. They are calculated by - taking the least significant 11 bits from the first 3 16-bit - words of the `keccak_256()` hash of `bloom_entry`. - """ - ``` - -- Format using markdown. -- Links to relevant standards and EIPs may be specified using reference-style links. - - ```python - """ - Minimum gas cost per byte of calldata as per [EIP-7976]. - - [EIP-7976]: https://eips.ethereum.org/EIPS/eip-7976 - """ - ``` - -- Avoid beginning docstrings with an article ("the"/"a") or a pronoun ("it", "they", etc.). -- Don't include the function's signature. - -##### Constants - -- Do not include constant values in docstrings, neither as literals nor as expressions. It's too easy to change a constant's value and forget to update its docstring. -- Construct the constant's value from other constants or meaningful expressions in order to provide meaningful context. - - **Great:** `TARGET_BLOB_GAS_PER_BLOCK = GAS_PER_BLOB * BLOB_SCHEDULE_TARGET` - - Composed from named constants; the reader immediately understands what the value represents. - - **Acceptable:** `TX_MAX_GAS = Uint(2 ** 24)` - - More readable than a raw number, but still a literal expression that doesn't convey _why_ this value was chosen. - - **Bad:** `TX_MAX_GAS = Uint(16_777_216)` - - A magic number with no context. - -### Changes across various Forks - -Many contributions require changes across multiple forks, organized under `src/ethereum/forks/*`. When making such changes, please ensure that differences between the forks are minimal and consist only of necessary differences. This will help with getting cleaner [diff outputs](https://ethereum.github.io/execution-specs/diffs/index.html). - -When creating pull requests affecting multiple forks, we recommended submitting your PR in two steps: - -1. Apply the changes on a single fork, open a _draft_ pull request, and get feedback; then -2. Apply the changes across the other forks, push them, and mark the pull request as ready for review. - -This saves you having to apply code review feedback repeatedly for each fork. - -### Development - -Running the tests necessary to merge into the repository requires: - -- [`uv`](https://docs.astral.sh/uv/) package manager, -- Python 3.11.x, -- [PyPy](https://www.pypy.org/) [7.3.19](https://downloads.python.org/pypy/) or later. -- `geth` installed and present in `$PATH`. - -`execution-specs` depends on a submodule that contains common tests that are run across all clients, so we need to clone the repo with the --recursive flag. Example: - -```bash -git clone --recursive https://github.com/ethereum/execution-specs.git -``` - -Or, if you've already cloned the repository, you can fetch the submodules with: - -```bash -git submodule update --init --recursive -``` - -The tests can be run with: - -```bash -tox -``` - -The development tools can also be run outside of `tox`, and can automatically reformat the code: - -```bash -uv run ruff check # Detects code issues and produces a report to STDOUT. -uv run ruff check --fix # Fixes minor code issues (like unsorted imports). -uv run ruff format # Formats code. -uv run mypy # Verifies type annotations. -``` - -It is recommended to use a [virtual environment](https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/#creating-a-virtual-environment) to keep your system Python installation clean. - -A trace of the EVM execution for any test case can be obtained by providing the `--evm-trace` argument to pytest. -Note: Make sure to run the EVM trace on a small number of tests at a time. The log might otherwise get very big. -Below is an example. - -```bash -uv run pytest \ - 'tests/json_loader/test_state_tests.py::test_state_tests_frontier[stAttackTest - ContractCreationSpam - 0]' \ - --evm_trace -``` - -## CLI Utilities `ethereum_spec_tools` - -The EELS repository has various CLI utilities that can help in the development process. - -### New Fork Tool - -This tool can be used to create the base code for a new fork by using the existing code from a given fork. - -The command takes 4 arguments, 2 of which are optional - -- from_fork: The fork name from which the code is to be duplicated. Eg. - "Tangerine Whistle" -- to_fork: The fork name of the new fork Eg - "Spurious Dragon" -- from_test (Optional): Name of the from fork within the test fixtures in case it is different from fork name. Eg. - "EIP150" -- to_test (Optional): Name of the to fork within the test fixtures in case it is different from fork name Eg - "EIP158" - -As an example, if one wants to create baseline code for the `Spurious Dragon` fork from the `Tangerine Whistle` one - -```bash -ethereum-spec-new-fork --from_fork="Tangerine Whistle" --to_fork="Spurious Dragon" --from_test=EIP150 --to_test=EIP158 -``` - -The following will have to however, be updated manually - -1. The fork number and `MAINNET_FORK_BLOCK` in `__init__.py`. If you are proposing a new EIP, please set `MAINNET_FORK_BLOCK` to `None`. -2. Any absolute package imports from other forks eg. in `trie.py` -3. Package names under `setup.cfg` -4. Add the new fork to the `monkey_patch()` function in `src/ethereum_optimized/__init__.py` -5. Adjust the underline in `fork/__init__.py` +> Generally, we do not assign issues to external contributors. If you want to work on an issue, you are welcome to go ahead and make a pull request. We are happy to answer questions before you start implementing. -### Sync Tool +## Contributions we don't accept -The sync tool allows one to use an RPC provider to fetch and validate blocks against EELS. -The state can also be stored in a local DB after validation. Since syncing directly with the specs can be -very slow, one can also leverage the optimized module. This contains alternative implementations of routines -in EELS that have been optimized for speed rather than clarity/readability. +Pull requests should have reasonable substance and context. In particular, we do not accept: -The tool can be called using the `ethereum-spec-sync` command which takes the following arguments +- Contributions that only fix spelling or grammatical errors in documentation, code, or elsewhere. +- Drive-by or vibe-coded contributions without proper engagement or context. -- rpc-url: Endpoint providing the Ethereum RPC API. Defaults to `http://localhost:8545/` -- unoptimized: Don't use the optimized state/ethash (this can be extremely slow) -- persist: Store the state in a db in this file -- geth: Use geth specific RPC endpoints while fetching blocks -- reset: Delete the db and start from scratch -- gas-per-commit: Commit to db each time this much gas is consumed. Defaults to 1_000_000_000 -- initial-state: Start from the state in this db, rather than genesis -- stop-at: After syncing this block, exit successfully +## Code of Conduct -- The following options are not supported WITH `--unoptimized` -> `--persist`, `--initial-state`, `--reset` -- The following options are not supported WITHOUT `--persist` -> `--initial_state`, `--reset` +All contributors are expected to be excellent to each other; other behavior is not tolerated. To report a concern, contact one of the [STEEL team members](https://steel.ethereum.foundation/team/). -### Patch Tool +## Principles -This tool can be used to apply the unstaged changes in `SOURCE_FORK` to each of the `TARGET_FORKS`. If some -of the change didn't apply, '.rej' files listing the unapplied changes will be left in the `TARGET_FORK`. +The specification aims to be: -The tool takes the following command line arguments +1. **Correct.** Describe the *intended* behavior of the Ethereum blockchain. Any deviation from that is a bug. +2. **Complete.** Capture the entirety of *consensus-critical* parts of Ethereum. +3. **Accessible.** Prioritize readability, clarity, and plain language over performance and brevity. -- The fork name where the changes have been made. Eg:- `frontier` (only a single fork name) -- The fork names where the changes have to be applied. Eg:- `homestead` (multiple values can be provided separated by space) -- optimized: Patch the optimized code instead -- tests: Patch the tests instead +## Getting set up -As an example, if one wants to apply changes made in `Frontier` fork to `Homestead` and `Tangerine Whistle` +Environment setup (cloning the repository, installing `uv` and `just`, Python requirements) is documented in the [Installation guide](docs/getting_started/installation.md). -```bash -python src/ethereum_spec_tools/patch_tool.py frontier homestead tangerine_whistle -``` +Before opening a PR, run the checks relevant to your change; see [Verifying Changes](docs/getting_started/verifying_changes.md). -### Lint Tool +## Changes that affect multiple forks -This tool checks for style and formatting issues specific to EELS and emits diagnostics -when issues are found +When creating pull requests that touch several forks under `src/ethereum/forks/`, we recommend a two-step workflow: -The tool currently performs the following checks +1. Apply the changes on a single fork, open a *draft* PR, and get feedback. +2. Apply the changes across the other forks, push them, and mark the PR as ready for review. -- The order of the identifiers between each hardfork is consistent. -- Import statements follow the relevant import rules in modules. +This saves you from applying code review feedback repeatedly for each fork. -The command to run the tool is `ethereum-spec-lint` +See [Writing Specs](docs/specs/writing_specs.md) for the technical style rules (naming, comments, docstrings, constants, cross-fork discipline) and for the `ethereum_spec_tools` CLI utilities that help with these workflows. diff --git a/EIP_AUTHORS_MANUAL.md b/EIP_AUTHORS_MANUAL.md deleted file mode 100644 index 2d549e23cfb..00000000000 --- a/EIP_AUTHORS_MANUAL.md +++ /dev/null @@ -1,69 +0,0 @@ -# EIP Author's Manual - -## Introduction - -This document outlines the process of specifying and testing EIPs for the Ethereum execution layer. It is intended for EIP authors, researchers and implementers. An EIP will typically go through the following stages: - -| Stage | Activities | Outputs | -| ------------------ | ----------- | ------- | -| _Pre‑Draft_ | Prospective EIP author conceives of an idea for an improvement to Ethereum, and discusses with the community. |
  • Vague Consensus on [Ethereum Magicians][0]
| -| **Draft** |

EIP author writes a technical human-language document describing the improvement, initially in broad strokes and becoming more specific over time.

Concurrently, they develop a Python reference implementation to make the EIP executable and identify any immediate/obvious implementation issues. For example, the EIP may not be compatible with some detail of the current Ethereum Virtual Machine.

Finally for this stage, the author begins to write test schemes for the EIP. Having the reference implementation should help identify the various logical flows to test and thus feed into more robust testing. Once the test schemes are written, the reference implementation can then be used to fill the tests and generate the test vectors.

|
  • Complete (but not final) document in [EIPs Repository][1]
  • Reference Implementation in [EELS][2]
  • Initial Tests in [EEST][3]
| -| **Review** |

The broader Ethereum community discusses and provides input on the proposal.

Although the feedback from the community can be sought at all lifecycle stages, having a reference implementation and tests act as a good bridge between research and client implementation. It also helps core developers (who have limited time and resources) to understand the EIP better and provide more informed feedback.

|
  • Complete & final document in the [EIPs Repository][1]
  • Comprehensive tests in [EEST][3]
| -| **Last Call** | Usually after being nominated for inclusion in a fork, the EIP author signals that the proposal is effectively done and begins the last period for comments/discussion. |
  • Complete reference implementation in [EELS][2]
  • Complete tests in [EEST][3]
  • Immutable proposal in [EIPs Repository][1]
| -| **Final** | The proposal is now immutable (cannot be changed) and exists for reference. |
  • Mainnet client implementations
| - -[0]: https://ethereum-magicians.org/ -[1]: https://github.com/ethereum/EIPs/ -[2]: https://github.com/ethereum/execution-specs -[3]: https://github.com/ethereum/execution-spec-tests - -This document will focus on stages 3 and 4 of the above lifecycle. - -## Executable Specifications - -This repository contains the executable specifications for the Ethereum execution layer. - -### Folder Structure - -#### Forks live on mainnet - -The folder `src/ethereum` contains the specifications for the different execution layer forks. Each fork has its own folder. For example, the folder `src/ethereum/frontier` contains the specifications for the Frontier hardfork. The `state_transition` function which is available in the `src/ethereum//fork.py` is the transition function for each fork. - -#### Fork under development - -At any given time, there is a single fork under development. Any new EIP has to be implemented in the folder that is meant for that fork (`src/ethereum/` folder). - -For example, at the time of writing, the Prague Fork is still under development and the previous fork is Cancun, which is live on Mainnet. So the prague folder under `src/ethereum` is essentially just the Cancun fork with the values of variables updated to reflect Prague and its under-development status. This folder (`src/ethereum/prague`) serves as the baseline for further development and all new EIPs are to be implemented in this folder. - -### Branch Structure - -#### Forks live on mainnet - -The final stable specification for all forks that are currently live on mainnet are in the `mainnet` branch. - -#### Fork under development - -At any given time, there can only be one fork under active development. The branch structure for the fork under development is as follows: - -- `forks/`: The main branch for the fork under development. For example, `forks/prague` is the branch for the Prague fork. This branch will be merged into `mainnet` after the fork has gone live on mainnet. -- `eips//`: Branches for each EIP within the fork under development. For example, `eips/prague/eip-7702` is the branch for EIP-7702 for the Prague fork. This branch will be merged into `forks/prague` after the EIP has been confirmed for release in the fork. - -## Writing New EIPS - -Implementing a new EIP in the `execution-specs` repository involves the following steps: - -1. **Create a new branch**: Create a new branch for the EIP under the appropriate fork. For example, if you are implementing an EIP for the Prague fork, create a new branch under `eips//eip-`. -2. **Implement the EIP**: Implement the EIP in the `src/ethereum/` folder. -3. **Basic sanity checks**: Run `tox -e static` to run basic formatting and linting checks. -4. **Raise a PR**: Raise a PR against the appropriate branch. For example, if you are implementing an EIP for the Prague fork, raise a PR against the `forks/prague` branch. - -An EIP can only be CFI'd (Considered For Inclusion) if it has a reference `execution-specs` implementation. The EIP author is responsible for maintaining their EIP up-to-date with the latest changes. For example, if an author had written their EIP for Cancun under `eips/cancun/eip-x`, but for some reason it didn't make it into Cancun, they would need to rebase their EIP to reflect the changes in Prague under `eips/prague/eip-x`. - -Please refer the following tutorial for writing new EIP. It takes you through a sample EIP for adding a new opcode to the specs. -[Tutorial](https://www.youtube.com/watch?v=QIcw_DGSy3s&t) - -## Writing Tests with `execution-spec-tests` - -In addition to having a reference implementation, it is also very useful for the community and core development if the EIP author concieves and writes test vectors for the EIP. There is a very user friendly framework for writing ethereum tests in the `execution-spec-tests` repository. Please refer to the following guide for writing tests to your EIP. - -[Writing Tests with `execution-spec-tests`](https://eest.ethereum.org/v2.0.0/getting_started/quick_start/) diff --git a/Justfile b/Justfile new file mode 100644 index 00000000000..350ba08ab61 --- /dev/null +++ b/Justfile @@ -0,0 +1,370 @@ +set positional-arguments := true + +alias help := list + +# List available recipes (default) +[default, private] +list: + @just --list + +root := justfile_directory() +output_dir := root / ".just" +xdist_workers := env("PYTEST_XDIST_AUTO_NUM_WORKERS", "6") +evm_bin := env("EVM_BIN", "evm") +latest_fork := "Amsterdam" + +# --- Static Analysis --- + +# Auto-fix formatting and lint issues +[group('static analysis')] +fix: + uv run ruff format + uv run ruff check --fix + +# Run all static checks (spellcheck, lint, format, mypy, ...) +[group('static analysis'), parallel] +static: typecheck lint-spec spellcheck deadcode lint-actions lock-check format-check lint + +# Check spelling +[group('static analysis')] +spellcheck: + #!/usr/bin/env bash + if ! uv run codespell; then + echo "" + echo "If false positive, add to whitelist:" + echo " just whitelist " + echo "" + echo "To auto-fix interactively:" + echo " uv run codespell -i 3" + exit 1 + else + echo "uv run codespell # passed!" + fi + +# Add a word to the spellcheck whitelist +[group('static analysis')] +whitelist *words: + uv run whitelist "$@" + +# Lint with ruff +[group('static analysis')] +lint *args: + uv run ruff check "$@" + +# Check for dead code with vulture +[group('static analysis')] +deadcode: + uv run vulture src/ vulture_whitelist.py + +# Check formatting with ruff +[group('static analysis')] +format-check *args: + uv run ruff format --check "$@" + +# Run type checking with mypy +[group('static analysis')] +typecheck *args: + uv run mypy "$@" + +# Check EELS import isolation +[group('static analysis')] +lint-spec: + uv run ethereum-spec-lint + +# Verify uv.lock is up to date +[group('static analysis')] +lock-check: + #!/usr/bin/env bash + if ! uv lock --check; then + echo "" + echo "To sync the lock file:" + echo " uv lock" + echo "" + echo "Then commit the updated uv.lock." + exit 1 + fi + +# Lint GitHub Actions workflows +[group('static analysis')] +lint-actions: + uv run actionlint -pyflakes pyflakes -shellcheck "shellcheck -S warning" + +# --- Consensus Tests --- + +# Generate HTML coverage report from last just fill run +[group('consensus tests')] +coverage: + uv run coverage html -d "{{ output_dir }}/fill/coverage-html" + +# Generate EIP test checklists from eip_checklist markers +[group('consensus tests')] +checklist *args: + uv run checklist --output tmp/checklist "$@" + +# Fill the consensus tests using EELS (with Python) +[group('consensus tests')] +fill *args: + @mkdir -p "{{ output_dir }}/fill/tmp" "{{ output_dir }}/fill/logs" + uv run fill \ + -m "not slow" \ + -n {{ xdist_workers }} --dist=loadgroup \ + --skip-index \ + --output="{{ output_dir }}/fill/fixtures" \ + --cov-config=pyproject.toml \ + --cov=ethereum \ + --cov-report=term \ + --cov-report "xml:{{ output_dir }}/fill/coverage.xml" \ + --no-cov-on-fail \ + --cov-branch \ + --basetemp="{{ output_dir }}/fill/tmp" \ + --log-to "{{ output_dir }}/fill/logs" \ + --clean \ + --until "{{ latest_fork }}" \ + --durations=50 \ + "$@" \ + tests + +# --- Integration Tests --- + +# Fill the base coverage consensus tests using EELS with PyPy +[group('integration tests')] +fill-pypy *args: + @mkdir -p "{{ output_dir }}/fill-pypy/tmp" "{{ output_dir }}/fill-pypy/logs" + uv run --python pypy3.11 fill \ + --skip-index \ + --output="{{ output_dir }}/fill-pypy/fixtures" \ + --no-html \ + --tb=long \ + -ra \ + --show-capture=no \ + --disable-warnings \ + -m "eels_base_coverage and not derived_test" \ + -n auto --maxprocesses 7 \ + --dist=loadgroup \ + --basetemp="{{ output_dir }}/fill-pypy/tmp" \ + --log-to "{{ output_dir }}/fill-pypy/logs" \ + --clean \ + --until "{{ latest_fork }}" \ + --ignore=tests/ported_static \ + "$@" \ + tests + +# Fill the base coverage consensus tests and run EELS against the fixtures +[group('integration tests')] +json-loader *args: + @mkdir -p "{{ output_dir }}/json-loader/tmp" + uv run fill \ + -m "eels_base_coverage and not derived_test" \ + --until "{{ latest_fork }}" \ + -n {{ xdist_workers }} --dist=loadgroup \ + --skip-index \ + --clean \ + --ignore=tests/ported_static \ + --output="tests/json_loader/fixtures" \ + --cov-config=pyproject.toml \ + --cov=ethereum \ + --cov-fail-under=85 + uv run pytest \ + -m "not slow" \ + -n auto --maxprocesses 6 --dist=loadfile \ + --basetemp="{{ output_dir }}/json-loader/tmp" \ + "$@" \ + tests/json_loader + +# --- Unit Tests --- + +# Run the testing package unit tests (with Python) +[group('unit tests')] +test-tests *args: + @mkdir -p "{{ output_dir }}/test-tests/tmp" + cd packages/testing && uv run pytest \ + -n {{ xdist_workers }} \ + --basetemp="{{ output_dir }}/test-tests/tmp" \ + --ignore=src/execution_testing/cli/pytest_commands/plugins/filler/tests/test_benchmarking.py \ + "$@" \ + src + +# Run the testing package unit tests (with PyPy) +[group('unit tests')] +test-tests-pypy *args: + @mkdir -p "{{ output_dir }}/test-tests-pypy/tmp" + cd packages/testing && uv run --python pypy3.11 pytest \ + -n auto --maxprocesses 6 \ + --basetemp="{{ output_dir }}/test-tests-pypy/tmp" \ + --ignore=src/execution_testing/cli/pytest_commands/plugins/filler/tests/test_benchmarking.py \ + "$@" \ + src + +# Run benchmark framework unit tests (with Python) +[group('unit tests')] +[group('benchmark tests')] +test-tests-bench *args: + @mkdir -p "{{ output_dir }}/test-tests-bench/tmp" + uv run pytest \ + --basetemp="{{ output_dir }}/test-tests-bench/tmp" \ + "$@" \ + packages/testing/src/execution_testing/cli/pytest_commands/plugins/filler/tests/test_benchmarking.py + +# Run CI release script integration tests +[group('unit tests')] +test-ci-scripts *args: + uv run pytest "$@" .github/scripts/tests/ + +# --- Benchmarks --- + +# Fill benchmark tests with --gas-benchmark-values, then verify with EELS +[group('benchmark tests')] +bench-gas *args: + @mkdir -p "{{ output_dir }}/bench-gas/tmp" "{{ output_dir }}/bench-gas/logs" + @echo "==> Step 1/2: Filling benchmark fixtures with configured EVM (EVM_BIN={{ evm_bin }})" + uv run fill \ + --evm-bin="{{ evm_bin }}" \ + --gas-benchmark-values 1 \ + --generate-all-formats \ + --fork Osaka \ + -m "not slow" \ + -n auto --maxprocesses 10 --dist=loadgroup \ + --output="{{ output_dir }}/bench-gas/fixtures" \ + --basetemp="{{ output_dir }}/bench-gas/tmp" \ + --log-to "{{ output_dir }}/bench-gas/logs" \ + --clean \ + "$@" \ + tests/benchmark/compute + @echo "==> Step 2/2: Running filled fixtures against EELS via json_loader" + @rm -rf tests/json_loader/bench_gas_fixtures + ln -sfn "{{ output_dir }}/bench-gas/fixtures" tests/json_loader/bench_gas_fixtures + cd tests/json_loader && uv run --python pypy3.11 pytest \ + --fork Osaka \ + --allow-post-state-hash \ + -n auto --maxprocesses 10 --dist=loadfile \ + --basetemp="{{ output_dir }}/bench-gas/json-loader-tmp" \ + bench_gas_fixtures + +# Fill benchmark tests with --fixed-opcode-count 1 +[group('benchmark tests')] +bench-opcode *args: + @mkdir -p "{{ output_dir }}/bench-opcode/tmp" "{{ output_dir }}/bench-opcode/logs" + uv run fill \ + --evm-bin="{{ evm_bin }}" \ + --fixed-opcode-count 1 \ + --fork Osaka \ + -m repricing \ + -n auto --maxprocesses 10 --dist=loadgroup \ + -k "not test_alt_bn128 and not test_bls12_381 and not test_modexp and not uncachable" \ + --output="{{ output_dir }}/bench-opcode/fixtures" \ + --basetemp="{{ output_dir }}/bench-opcode/tmp" \ + --log-to "{{ output_dir }}/bench-opcode/logs" \ + --clean \ + "$@" \ + tests/benchmark/compute + +# Run benchmark_parser, then fill benchmark tests using its config +[group('benchmark tests')] +bench-opcode-config *args: + @mkdir -p "{{ output_dir }}/bench-opcode-config/tmp" "{{ output_dir }}/bench-opcode-config/logs" + uv run benchmark_parser + uv run fill \ + --evm-bin="{{ evm_bin }}" \ + --fixed-opcode-count \ + --fork Osaka \ + -m repricing \ + -n auto --maxprocesses 10 --dist=loadgroup \ + -k "not test_alt_bn128 and not test_bls12_381 and not test_modexp and not test_point_evaluation_uncachable" \ + --output="{{ output_dir }}/bench-opcode-config/fixtures" \ + --basetemp="{{ output_dir }}/bench-opcode-config/tmp" \ + --log-to "{{ output_dir }}/bench-opcode-config/logs" \ + --clean \ + "$@" \ + tests/benchmark/compute + +# --- Docs --- + +export GEN_TEST_DOC_VERSION := "local" +export DYLD_FALLBACK_LIBRARY_PATH := if os() == "macos" { "/opt/homebrew/lib" } else { "" } + +# Generate documentation for EELS using docc +[group('docs')] +docs-spec $DOCC_SKIP_DIFFS="": + uv run docc --output "{{ output_dir }}/docs-spec" + uv run python -c 'import pathlib; print("documentation available under file://{0}".format(pathlib.Path(r"{{ output_dir }}") / "docs-spec" / "index.html"))' + +# Generate documentation for EELS using docc, skipping the slow per-fork diff render +[group('docs')] +docs-spec-fast: (docs-spec "1") + +# Build HTML site documentation with mkdocs +[group('docs')] +docs *args: + uv run mkdocs build --strict -d "{{ output_dir }}/docs/site" "$@" + +# Build HTML site documentation with mkdocs (skip test case reference) +[group('docs')] +docs-fast *args: + FAST_DOCS=True uv run mkdocs build --strict -d "{{ output_dir }}/docs/site" "$@" + +# Serve site documentation locally with mkdocs (live reload) +[group('docs')] +docs-serve *args: + uv run mkdocs serve "$@" + +# Serve site documentation locally with mkdocs (skip test case reference) +[group('docs')] +docs-serve-fast *args: + FAST_DOCS=True uv run mkdocs serve "$@" + +# Lint markdown files (markdownlint) +[group('docs')] +lint-md: + uv run markdownlintcli2_soft_fail + +[private] +crops: + @uvx pycowsay==0.0.0.2 "ethereum is good" + +# --- Housekeeping --- + +# Remove caches and build artifacts (.pytest_cache, .mypy_cache, __pycache__, ...) +[group('housekeeping')] +clean *args: + uv run eest clean "$@" + +# Remove caches, build artifacts, .just, and .venv +[group('housekeeping')] +clean-all *args: + uv run eest clean --all "$@" + +# Print the command to install shell completions for just recipes +[group('housekeeping')] +shell-completions: + #!/usr/bin/env bash + case "$(basename "$SHELL")" in + bash) + echo "Run the following commands to install just completions for bash:" + echo "" + echo " mkdir -p ~/.local/share/bash-completion/completions" + echo " just --completions bash > ~/.local/share/bash-completion/completions/just" + ;; + zsh) + echo "Run the following commands to install just completions for zsh:" + echo "" + echo " mkdir -p ~/.zsh/completions" + echo " just --completions zsh > ~/.zsh/completions/_just" + echo "" + echo "Then add to your .zshrc:" + echo "" + echo " fpath=(~/.zsh/completions \$fpath)" + echo " autoload -U compinit" + echo " compinit" + ;; + fish) + echo "Run the following commands to install just completions for fish:" + echo "" + echo " mkdir -p ~/.config/fish/completions" + echo " just --completions fish > ~/.config/fish/completions/just.fish" + ;; + *) + echo "See the link below for instructions for your shell." + ;; + esac + echo "" + echo "For more details, see https://just.systems/man/en/shell-completion-scripts.html" diff --git a/README.md b/README.md index c6f7c548993..8cb484326f2 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,14 @@ -# Ethereum Execution Client Specifications +# Ethereum Execution Layer Specifications -[![GitPOAP Badge](https://public-api.gitpoap.io/v1/repo/ethereum/execution-specs/badge)](https://www.gitpoap.io/gh/ethereum/execution-specs) -[![codecov](https://codecov.io/gh/ethereum/execution-specs/graph/badge.svg?token=0LQZO56RTM)](https://codecov.io/gh/ethereum/execution-specs) +[![latest version](https://img.shields.io/github/v/release/ethereum/execution-specs)](https://github.com/ethereum/execution-specs/releases/latest) +[![PyPI version](https://img.shields.io/pypi/v/ethereum-execution)](https://pypi.org/project/ethereum-execution/) +[![License](https://img.shields.io/github/license/ethereum/execution-specs)](https://github.com/ethereum/execution-specs/blob/main/LICENSE) [![Python Specification](https://github.com/ethereum/execution-specs/actions/workflows/test.yaml/badge.svg)](https://github.com/ethereum/execution-specs/actions/workflows/test.yaml) +[![codecov](https://codecov.io/gh/ethereum/execution-specs/graph/badge.svg?token=0LQZO56RTM)](https://codecov.io/gh/ethereum/execution-specs) +![Python Versions](https://img.shields.io/badge/python-3.11%20%7C%203.12%20%7C%203.13%20%7C%203.14-blue) +[![ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff) +[![uv](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/uv/main/assets/badge/v0.json)](https://github.com/astral-sh/uv) +[![GitPOAP Badge](https://public-api.gitpoap.io/v1/repo/ethereum/execution-specs/badge)](https://www.gitpoap.io/gh/ethereum/execution-specs) ## Monadized `execution-specs` and spec tests @@ -19,18 +25,19 @@ It mainly serves as an alternative implementation of https://github.com/category 1. Install `uv`: `curl -LsSf https://astral.sh/uv/install.sh | sh` 2. Clone and setup: -```bash -git clone --depth 1 https://github.com/monad-developers/execution-specs -cd execution-specs -uv python install 3.12 -uv python pin 3.12 -uv sync --all-extras -``` + ```bash + git clone --depth 1 https://github.com/monad-developers/execution-specs + cd execution-specs + uv python install 3.12 + uv python pin 3.12 + uv sync --all-extras + ``` + 3. Fill all monadized spec tests as of writing this (see below for explanation of flags): -```bash -uv run fill --clean -m "blockchain_test or transaction_test" --from MONAD_EIGHT --until MONAD_NINE --chain-id 143 -n auto tests -``` + ```bash + uv run fill --clean -m "blockchain_test or transaction_test" --from MONAD_EIGHT --until MONAD_NINE --chain-id 143 -n auto tests + ``` 4. The test fixtures to be found in `fixtures/` directory under repo root. @@ -43,14 +50,14 @@ Filling tests is the process of running Python generators of spec tests [like on #### `uv run fill` flags - **`-m blockchain_test or transaction_test`**: causes these two flavors of fixtures to be generated - - `blockchain_test` is the currently supported by `monad` flavor of a spec test checking correctness of the state transition. Note this includes `blockchain_test_from_state_test`, meaning that all `state_test(...)` fillers are included - - `transaction_test` is also supported, tests only correctness of static transaction checks + - `blockchain_test` is the currently supported by `monad` flavor of a spec test checking correctness of the state transition. Note this includes `blockchain_test_from_state_test`, meaning that all `state_test(...)` fillers are included + - `transaction_test` is also supported, tests only correctness of static transaction checks - **`--from MONAD_EIGHT --until MONAD_NINE`**: hardforks for which to generate fixtures. Must match with those defined in `monad`, inclusive - **`--chain-id 143`**: must be specified for signatures and EIP-7702 to work correctly - **`-n auto`**: from `pytest`, parallel execution of tests - **`tests`**: root directory to traverse to discover tests - - inside the tests are organized by EIP/MIP and the hardfork when they **became relevant**. For Monad these are `monad_eight` and `monad_nine` as of writing this. Here hardfork is informative only, doesn't need to match with `monad`. - - note that tests relevant for previous hardforks, e.g. `tests/prague/eip7702_set_code_tx`, are filled for hardforks requested with `--from`, `--until` flags, respecting any constraints the test itself might define. In other words, in the invocation above, EIP-7702 tests **will be filled** + - inside the tests are organized by EIP/MIP and the hardfork when they **became relevant**. For Monad these are `monad_eight` and `monad_nine` as of writing this. Here hardfork is informative only, doesn't need to match with `monad`. + - note that tests relevant for previous hardforks, e.g. `tests/prague/eip7702_set_code_tx`, are filled for hardforks requested with `--from`, `--until` flags, respecting any constraints the test itself might define. In other words, in the invocation above, EIP-7702 tests **will be filled** `pytest` flags are in general supported, while `--traces` will give you detailed EVM step traces. Refer to https://eest.ethereum.org/main/filling_tests/ for more details. @@ -64,101 +71,68 @@ Filling tests is the process of running Python generators of spec tests [like on ## Description -This repository contains the specifications related to the Ethereum execution client, specifically the [pyspec](/src/ethereum/__init__.py) and specifications for [network upgrades](/src/ethereum/__init__.py). The [JSON-RPC API specification](https://github.com/ethereum/execution-apis) can be found in a separate repository. - -### Ethereum Protocol Releases - -| Version and Code Name | Block No. | Released | Incl EIPs | Fork Specifications | Blog | -|-----------------------|-----------|----------|-----------|-------|-------| -| Osaka | 23935694 | 2025-12-03 | [EIP-7594]
[EIP-7642]
[EIP-7823]
[EIP-7825]
[EIP-7883]
[EIP-7892]
[EIP-7910]
[EIP-7917]
[EIP-7918]
[EIP-7934]
[EIP-7935]
[EIP-7939]
[EIP-7951] | [Hardfork Meta EIP-7607](https://eips.ethereum.org/EIPS/eip-7607)
[Fork Manifest](/src/ethereum/forks/osaka/__init__.py) | [Blog](https://blog.ethereum.org/2025/11/06/fusaka-mainnet-announcement) | -| Prague | 22431084 | 2025-05-07 | [EIP-2537]
[EIP-2935]
[EIP-6110]
[EIP-7002]
[EIP-7251]
[EIP-7549]
[EIP-7623]
[EIP-7685]
[EIP-7691]
[EIP-7702] | [Hardfork Meta EIP-7600](https://eips.ethereum.org/EIPS/eip-7600)
[Fork Manifest](/src/ethereum/forks/prague/__init__.py)| [Blog](https://blog.ethereum.org/2025/04/23/pectra-mainnet) | -| Cancun | 19426587 | 2024-03-13
(1710338135) | [EIP-1153](https://eips.ethereum.org/EIPS/eip-1153)
[EIP-4788](https://eips.ethereum.org/EIPS/eip-4788)
[EIP-4844](https://eips.ethereum.org/EIPS/eip-4844)
[EIP-5656](https://eips.ethereum.org/EIPS/eip-5656)
[EIP-6780](https://eips.ethereum.org/EIPS/eip-6780)
[EIP-7044](https://eips.ethereum.org/EIPS/eip-7044)
[EIP-7045](https://eips.ethereum.org/EIPS/eip-7045)
[EIP-7514](https://eips.ethereum.org/EIPS/eip-7514)
[EIP-7516](https://eips.ethereum.org/EIPS/eip-7516)| [Hardfork Meta EIP-7569](https://eips.ethereum.org/EIPS/eip-7569)
[Fork Manifest](/src/ethereum/forks/cancun/__init__.py) | [Blog](https://blog.ethereum.org/2024/02/27/dencun-mainnet-announcement) | -| Shanghai | 17034870 | 2023-04-12
(1681338455) | [EIP-3651](https://eips.ethereum.org/EIPS/eip-3651)
[EIP-3855](https://eips.ethereum.org/EIPS/eip-3855)
[EIP-3860](https://eips.ethereum.org/EIPS/eip-3860)
[EIP-4895](https://eips.ethereum.org/EIPS/eip-4895) | [(Backfill) Meta EIP-7568](https://eips.ethereum.org/EIPS/eip-7568)
[Fork Manifest](/src/ethereum/forks/shanghai/__init__.py) | [Blog](https://blog.ethereum.org/2023/03/28/shapella-mainnet-announcement) | -| Paris | 15537394 | 2022-09-15 | [EIP-3675](https://eips.ethereum.org/EIPS/eip-3675)
[EIP-4399](https://eips.ethereum.org/EIPS/eip-4399) | [(Backfill) Meta EIP-7568](https://eips.ethereum.org/EIPS/eip-7568)
[Fork Manifest](/src/ethereum/forks/paris/__init__.py) | [Blog](https://blog.ethereum.org/2022/08/24/mainnet-merge-announcement) | -| Gray Glacier | 15050000 | 2022-06-30 | [EIP-5133](https://eips.ethereum.org/EIPS/eip-5133) | [(Backfill) Meta EIP-7568](https://eips.ethereum.org/EIPS/eip-7568)
[Fork Manifest](/src/ethereum/forks/gray_glacier/__init__.py) | [Blog](https://blog.ethereum.org/2022/06/16/gray-glacier-announcement/) | -| Arrow Glacier | 13773000 | 2021-12-09 | [EIP-4345](https://eips.ethereum.org/EIPS/eip-4345) | [(Backfill) Meta EIP-7568](https://eips.ethereum.org/EIPS/eip-7568)
[Fork Manifest](/src/ethereum/forks/arrow_glacier/__init__.py) | [Blog](https://blog.ethereum.org/2021/11/10/arrow-glacier-announcement/) | -| London | 12965000 | 2021-08-05 | [EIP-1559](https://eips.ethereum.org/EIPS/eip-1559)
[EIP-3198](https://eips.ethereum.org/EIPS/eip-3198)
[EIP-3529](https://eips.ethereum.org/EIPS/eip-3529)
[EIP-3541](https://eips.ethereum.org/EIPS/eip-3541)
[EIP-3554](https://eips.ethereum.org/EIPS/eip-3554)| [(Backfill) Meta EIP-7568](https://eips.ethereum.org/EIPS/eip-7568)
[Fork Manifest](/src/ethereum/forks/london/__init__.py) | [Blog](https://blog.ethereum.org/2021/07/15/london-mainnet-announcement/) | -| Berlin | 12244000 | 2021-04-15 | [EIP-2565](https://eips.ethereum.org/EIPS/eip-2565)
[EIP-2929](https://eips.ethereum.org/EIPS/eip-2929)
[EIP-2718](https://eips.ethereum.org/EIPS/eip-2718)
[EIP-2930](https://eips.ethereum.org/EIPS/eip-2930) | ~[Hardfork Meta EIP-2070](https://eips.ethereum.org/EIPS/eip-2070)~
[(Backfill) Meta EIP-7568](https://eips.ethereum.org/EIPS/eip-7568)
[Fork Manifest](/src/ethereum/forks/berlin/__init__.py) | [Blog](https://blog.ethereum.org/2021/03/08/ethereum-berlin-upgrade-announcement/) | -| Muir Glacier | 9200000 | 2020-01-02 | [EIP-2384](https://eips.ethereum.org/EIPS/eip-2384) | [Hardfork Meta EIP-2387](https://eips.ethereum.org/EIPS/eip-2387)
[Fork Manifest](/src/ethereum/forks/muir_glacier/__init__.py) | [Blog](https://blog.ethereum.org/2019/12/23/ethereum-muir-glacier-upgrade-announcement/) | -| Istanbul | 9069000 | 2019-12-07 | [EIP-152](https://eips.ethereum.org/EIPS/eip-152)
[EIP-1108](https://eips.ethereum.org/EIPS/eip-1108)
[EIP-1344](https://eips.ethereum.org/EIPS/eip-1344)
[EIP-1884](https://eips.ethereum.org/EIPS/eip-1884)
[EIP-2028](https://eips.ethereum.org/EIPS/eip-2028)
[EIP-2200](https://eips.ethereum.org/EIPS/eip-2200) | [Hardfork Meta EIP-1679](https://eips.ethereum.org/EIPS/eip-1679)
[Fork Manifest](/src/ethereum/forks/istanbul/__init__.py) | [Blog](https://blog.ethereum.org/2019/11/20/ethereum-istanbul-upgrade-announcement/) | -| Petersburg | 7280000 | 2019-02-28 | [EIP-145](https://eips.ethereum.org/EIPS/eip-145)
[EIP-1014](https://eips.ethereum.org/EIPS/eip-1014)
[EIP-1052](https://eips.ethereum.org/EIPS/eip-1052)
[EIP-1234](https://eips.ethereum.org/EIPS/eip-1234) | [Hardfork Meta EIP-1716](https://eips.ethereum.org/EIPS/eip-1716) | [Blog](https://blog.ethereum.org/2019/02/22/ethereum-constantinople-st-petersburg-upgrade-announcement/) | -| Constantinople | 7280000 | 2019-02-28 | [EIP-145](https://eips.ethereum.org/EIPS/eip-145)
[EIP-1014](https://eips.ethereum.org/EIPS/eip-1014)
[EIP-1052](https://eips.ethereum.org/EIPS/eip-1052)
[EIP-1234](https://eips.ethereum.org/EIPS/eip-1234)
[EIP-1283](https://eips.ethereum.org/EIPS/eip-1283) | [Hardfork Meta EIP-1013](https://eips.ethereum.org/EIPS/eip-1013)
[Fork Manifest](/src/ethereum/forks/constantinople/__init__.py) | [Blog](https://blog.ethereum.org/2019/02/22/ethereum-constantinople-st-petersburg-upgrade-announcement/) | -| Byzantium | 4370000 | 2017-10-16 | [EIP-100](https://eips.ethereum.org/EIPS/eip-100)
[EIP-140](https://eips.ethereum.org/EIPS/eip-140)
[EIP-196](https://eips.ethereum.org/EIPS/eip-196)
[EIP-197](https://eips.ethereum.org/EIPS/eip-197)
[EIP-198](https://eips.ethereum.org/EIPS/eip-198)
[EIP-211](https://eips.ethereum.org/EIPS/eip-211)
[EIP-214](https://eips.ethereum.org/EIPS/eip-214)
[EIP-649](https://eips.ethereum.org/EIPS/eip-649)
[EIP-658](https://eips.ethereum.org/EIPS/eip-658) | [Hardfork Meta EIP-609](https://eips.ethereum.org/EIPS/eip-609)
[Fork Manifest](/src/ethereum/forks/byzantium/__init__.py) | [Blog](https://blog.ethereum.org/2017/10/12/byzantium-hf-announcement/) | -| Spurious Dragon | 2675000 | 2016-11-22 | [EIP-155](https://eips.ethereum.org/EIPS/eip-155)
[EIP-160](https://eips.ethereum.org/EIPS/eip-160)
[EIP-161](https://eips.ethereum.org/EIPS/eip-161)
[EIP-170](https://eips.ethereum.org/EIPS/eip-170) | [Hardfork Meta EIP-607](https://eips.ethereum.org/EIPS/eip-607)
[Fork Manifest](/src/ethereum/forks/spurious_dragon/__init__.py) | [Blog](https://blog.ethereum.org/2016/11/18/hard-fork-no-4-spurious-dragon/) | -| Tangerine Whistle | 2463000 | 2016-10-18 | [EIP-150](https://eips.ethereum.org/EIPS/eip-150) | [Hardfork Meta EIP-608](https://eips.ethereum.org/EIPS/eip-608)
[Fork Manifest](/src/ethereum/forks/tangerine_whistle/__init__.py) | [Blog](https://blog.ethereum.org/2016/10/13/announcement-imminent-hard-fork-eip150-gas-cost-changes/) | -| DAO Fork | 1920000 | 2016-07-20 | | [Hardfork Meta EIP-779](https://eips.ethereum.org/EIPS/eip-779)
[Fork Manifest](/src/ethereum/forks/dao_fork/__init__.py) | [Blog](https://blog.ethereum.org/2016/07/15/to-fork-or-not-to-fork/) | -| DAO Wars | aborted | aborted | | | [Blog](https://blog.ethereum.org/2016/06/24/dao-wars-youre-voice-soft-fork-dilemma/) | -| Homestead | 1150000 | 2016-03-14 | [EIP-2](https://eips.ethereum.org/EIPS/eip-2)
[EIP-7](https://eips.ethereum.org/EIPS/eip-7)
[EIP-8](https://eips.ethereum.org/EIPS/eip-8) | [Hardfork Meta EIP-606](https://eips.ethereum.org/EIPS/eip-606)
[Fork Manifest](/src/ethereum/forks/homestead/__init__.py) | [Blog](https://blog.ethereum.org/2016/02/29/homestead-release/) | -| Frontier Thawing | 200000 | 2015-09-07 | | | [Blog](https://blog.ethereum.org/2015/08/04/the-thawing-frontier/) | -| Frontier | 1 | 2015-07-30 | | [Fork Manifest](/src/ethereum/forks/frontier/__init__.py) | [Blog](https://blog.ethereum.org/2015/07/22/frontier-is-coming-what-to-expect-and-how-to-prepare/) | - -*Note:* Starting with Paris, updates are no longer rolled out based on block numbers. Paris was enabled once proof-of-work Total Difficulty reached 58750000000000000000000. As of Shanghai (at 1681338455), upgrade activation is based on timestamps. - -[EIP-2537]: https://eips.ethereum.org/EIPS/eip-2537 -[EIP-2935]: https://eips.ethereum.org/EIPS/eip-2935 -[EIP-6110]: https://eips.ethereum.org/EIPS/eip-6110 -[EIP-7002]: https://eips.ethereum.org/EIPS/eip-7002 -[EIP-7251]: https://eips.ethereum.org/EIPS/eip-7251 -[EIP-7549]: https://eips.ethereum.org/EIPS/eip-7549 -[EIP-7623]: https://eips.ethereum.org/EIPS/eip-7623 -[EIP-7685]: https://eips.ethereum.org/EIPS/eip-7685 -[EIP-7691]: https://eips.ethereum.org/EIPS/eip-7691 -[EIP-7702]: https://eips.ethereum.org/EIPS/eip-7702 -[EIP-7594]: https://eips.ethereum.org/EIPS/eip-7594 -[EIP-7642]: https://eips.ethereum.org/EIPS/eip-7642 -[EIP-7823]: https://eips.ethereum.org/EIPS/eip-7823 -[EIP-7825]: https://eips.ethereum.org/EIPS/eip-7825 -[EIP-7883]: https://eips.ethereum.org/EIPS/eip-7883 -[EIP-7892]: https://eips.ethereum.org/EIPS/eip-7892 -[EIP-7910]: https://eips.ethereum.org/EIPS/eip-7910 -[EIP-7917]: https://eips.ethereum.org/EIPS/eip-7917 -[EIP-7918]: https://eips.ethereum.org/EIPS/eip-7918 -[EIP-7934]: https://eips.ethereum.org/EIPS/eip-7934 -[EIP-7935]: https://eips.ethereum.org/EIPS/eip-7935 -[EIP-7939]: https://eips.ethereum.org/EIPS/eip-7939 -[EIP-7951]: https://eips.ethereum.org/EIPS/eip-7951 - -Some clarifications were enabled without protocol releases: - -| EIP | Block No. | -|-----|-----------| -| [EIP-2681](https://eips.ethereum.org/EIPS/eip-2681) | 0 | -| [EIP-3607](https://eips.ethereum.org/EIPS/eip-3607) | 0 | -| [EIP-7523](https://eips.ethereum.org/EIPS/eip-7523) | 15537394 | -| [EIP-7610](https://eips.ethereum.org/EIPS/eip-7610) | 0 | - -## Execution Specification (work-in-progress) - -The execution specification is a python implementation of Ethereum that prioritizes readability and simplicity. It will be accompanied by both narrative and API level documentation of the various components written in markdown and rendered using docc... - -* [Rendered specification](https://ethereum.github.io/execution-specs/) - -## Usage - -The Ethereum specification is maintained as a Python library, for better integration with tooling and testing. - -Requires Python 3.11+ - -### Building Specification Documentation - -Building the spec documentation is most easily done through [`tox`](https://tox.readthedocs.io/en/latest/): - -```bash -uvx --with=tox-uv tox -e spec-docs -``` +The Ethereum Execution Layer Specifications (EELS) are an executable Python reference implementation of Ethereum's execution layer, along with the test cases that verify it. It provides a shared, runnable description of consensus-critical behaviour, and the accompanying tests generate fixtures that can be used to validate execution client implementations. -The path to the generated HTML will be printed to the console. +## Quick Start -### Browsing Updated Documentation +execution-specs uses [`uv`](https://docs.astral.sh/uv/) to manage the Python environment and dependencies, and [`just`](https://just.systems/) as a task runner for common commands (linting, building docs, generating fixtures). The commands below install both and set up the repo from scratch. -To view the updated local documentation, run: +Requires a Unix-like shell. All platforms: -```bash -uv run mkdocs serve +```console +git clone https://github.com/ethereum/execution-specs +cd execution-specs +curl -LsSf https://astral.sh/uv/install.sh | sh +uv python install 3.12 +uv python pin 3.12 +uv sync +uv tool install --exclude-newer "10 days" rust-just +just shell-completions ``` -then connect to `localhost:8000` in a browser +Python 3.11–3.14 are supported; 3.12 tends to be the smoothest for local setup (pre-built wheels are available across the dependency set). For alternative `just` installation paths, macOS-specific installation notes, and troubleshooting, see [Installation](docs/getting_started/installation.md). + +## Documentation + +- **Repo documentation (default branch/fork)**: +- **Protocol history**: [docs/specs/protocol_history.md](docs/specs/protocol_history.md) +- **Versioning scheme**: [docs/specs/spec_releases.md](docs/specs/spec_releases.md) (PEP 440 compatible; hardfork encoded in the minor version, `rcN` marks devnets). + +## Contributing + +Earnest contributions are welcome; drive-by contributions are not. See [CONTRIBUTING.md](CONTRIBUTING.md) for how to raise issues and pull requests. Further reading: + +- [Code Standards](docs/getting_started/code_standards.md): Python coding preferences enforced in CI. +- [Verifying Changes](docs/getting_started/verifying_changes.md): Which local checks to run before opening a PR. +- [Writing Specs](docs/specs/writing_specs.md): Style rules and `ethereum_spec_tools` utilities for changes under `src/ethereum/`. +- [Writing Tests](docs/writing_tests/index.md): For guidance on adding consensus tests under `./tests/`. + +This repository is maintained by the [STEEL Team](https://steel.ethereum.foundation/) at the Ethereum Foundation. + +## Community and Support + +Discussion around the initial specification of protocol changes happens on [Ethereum Magicians](https://ethereum-magicians.org/), in pull requests on [ethereum/EIPs](https://github.com/ethereum/EIPs), on the [Ethereum R&D Discord](https://discord.com/invite/qGpsxSA) (one of the channels in the *Execution R&D* category; for testing use `#el-testing`), and in the AllCoreDevs calls. + +For tracking the status of upcoming Ethereum upgrades, see [Forkcast](https://forkcast.org/): EIP inclusion, client implementation progress, and ACD call summaries. + +For other help, see the [Documentation](#documentation) section above, or reach out to one of the [STEEL team members](https://steel.ethereum.foundation/team/) in the Ethereum R&D Discord. + +### Related projects + +- [ethereum/EIPs](https://github.com/ethereum/EIPs): The prose EIP documents that EELS implements. +- [ethereum/execution-apis](https://github.com/ethereum/execution-apis): The JSON-RPC API specification, which lives in a separate repository. +- [ethereum/consensus-specs](https://github.com/ethereum/consensus-specs): The consensus-layer counterpart to this repository. + +Production execution clients that implement the spec include [besu](https://github.com/besu-eth/besu), [erigon](https://github.com/erigontech/erigon), [ethrex](https://github.com/lambdaclass/ethrex), [geth](https://github.com/ethereum/go-ethereum), [nethermind](https://github.com/NethermindEth/nethermind), and [reth](https://github.com/paradigmxyz/reth). + +## Responsible Disclosure of Vulnerabilities + +> [!CAUTION] +> Care is required when filing issues or PRs for functionality that is live on Ethereum mainnet. Please report vulnerabilities and verify bounty eligibility via the [bug bounty program](https://bounty.ethereum.org); see [SECURITY.md](SECURITY.md) for details. +> +> - **Please do not create a PR with a vulnerability visible.** +> - **Please do not file a public ticket mentioning the vulnerability.** ## License -The Ethereum Execution Layer Specification code is licensed under the [Creative Commons Zero v1.0 Universal](LICENSE.md). +The Ethereum Execution Layer Specification is licensed under the [Creative Commons Zero v1.0 Universal](LICENSE.md). diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md deleted file mode 100644 index 4286cf29063..00000000000 --- a/docs/CHANGELOG.md +++ /dev/null @@ -1,1130 +0,0 @@ -# Changelog - -Test fixtures for use by clients are available for each release on the [Github releases page](https://github.com/ethereum/execution-spec-tests/releases). - -**Key:** ✨ = New, 🐞 = Fixed, 🔀 = Changed. 💥 = Breaking - -## 🔜 [Unreleased] - -### 💥 Breaking Change - -### 🛠️ Framework - -- 🐞 Remove `Op.CLZ` from `UndefinedOpcodes` list ([#1970](https://github.com/ethereum/execution-specs/pull/1970)). -- 🐞 Make `TransactionTraces` `CamelModel` less lestrictive ([#2081](https://github.com/ethereum/execution-specs/pull/2081)). - -#### `fill` - -- ✨ Allow command to customize `--chain-id` used for filling ([#2016](https://github.com/ethereum/execution-specs/pull/2016)). - -#### `consume` - -- ✨ Add Besu `evmtool` support for `consume direct` via `state-test` and `block-test` subcommands ([#2219](https://github.com/ethereum/execution-specs/pull/2219)). - -#### `execute` - -- ✨ Add transaction batching to avoid RPC overload when executing tests with many transactions. Transactions are now sent in configurable batches (default: 750) with progress logging. Use `--max-tx-per-batch` to configure the batch size ([#1907](https://github.com/ethereum/execution-specs/pull/1907)). -- ✨ `execute hive` and `execute remote` now defer funding of accounts until the minimum amount required to send the test transactions is calculated, in order to optimize the amount of Eth used to execute the tests ([#1822](https://github.com/ethereum/execution-specs/pull/1822)). -- ✨ Dynamically fetch gas prices from the network and update all transactions to use 1.5x the current values ([#1822](https://github.com/ethereum/execution-specs/pull/1822)). -- ✨ New `--dry-run` flag to calculate the amount of Eth that will be spent executing a test given the current network gas prices ([#1822](https://github.com/ethereum/execution-specs/pull/1822)). -- 🔀 Load balancing mode of `execute` for xdist was updated from `loadscope` to `load` ([#1822](https://github.com/ethereum/execution-specs/pull/1822)). -- 💥 `--eoa-fund-amount-default` has been deprecated since the command now automatically calculates the funding amount ([#1822](https://github.com/ethereum/execution-specs/pull/1822)). -- 💥 `--sender-key-initial-balance` flag of `execute hive` has been renamed to `--seed-key-initial-balance` ([#1822](https://github.com/ethereum/execution-specs/pull/1822)). -- 🔀 Flags --default-gas-price, --default-max-fee-per-gas and --default-max-priority-fee-per-gas now default to None and ideally should be omitted because, when unset, the command now defaults to fetch the value from the network, which is a more reliable behavior ([#1822](https://github.com/ethereum/execution-specs/pull/1822)). - -### 📋 Misc - -- ✨ Implement EIP-7928 Block-Level Access Lists ([#1719](https://github.com/ethereum/execution-specs/pull/1719)). - -### 🧪 Test Cases - -- ✨ Add missing fuzzy-compute benchmark configurations for `KECCAK256`, `CODECOPY`, `CALLDATACOPY`, `RETURNDATACOPY`, `MLOAD`, `MSTORE`, `MSTORE8`, `MCOPY`, `LOG*`, `CALLDATASIZE`, `CALLDATALOAD`, and `RETURNDATASIZE` opcodes ([#1956](https://github.com/ethereum/execution-specs/pull/1956)). -- ✨ Add precompile benchmark configurations for `ecPairing`, `blake2f`, `BLS12_G1_MSM`, `BLS12_G2_MSM` and `BLS12_PAIRING` to unblock repricing analysis ([#2003](https://github.com/ethereum/execution-specs/pull/2003)). -- 🔀 Relabel `@pytest.mark.repricing` markers in benchmark tests to reflect configurations requested for gas repricing analysis ([#1971](https://github.com/ethereum/execution-specs/pull/1971)). -- ✨ New EIP-7702 test cases added ([#1974](https://github.com/ethereum/execution-specs/pull/1974)). -- ✨ Add missing benchmark configurations / opcode to benchmark tests for repricing analysis([#2006](https://github.com/ethereum/execution-specs/pull/2006)). -- ✨ Port STATICCALL to CALL tests with zero and non-zero value transfer from `tests/static`, extending coverage with `pytest.mark.with_all_precompiles` ([#1960](https://github.com/ethereum/execution-specs/pull/1960)). -- ✨ Add BAL tests that dequeue EIP-7251 consolidation requests. ([#2076](https://github.com/ethereum/execution-specs/pull/2076)). -- ✨ Add BAL tests for handling 7702 delegation reset and delegated create. ([#2097](https://github.com/ethereum/execution-specs/pull/2097)). -- ✨ Add benchmark scenarios for ether transfers to precompiles, warm access list transfers, and max-size contract creation transactions ([#2171](https://github.com/ethereum/execution-specs/pull/2171)). - -## [v5.4.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v5.4.0) - 2025-12-07 - -### 🛠️ Framework - -#### General - -- 🔀 Updated default block gas limit from 45M to 60M to align with [EIP-7935](https://eips.ethereum.org/EIPS/eip-7935) for the Osaka fork. - -#### `fill` - -- 🐞 Allow `evmone` to fill Prague and Osaka blockchain tests (mainly modified deposit contract tests) ([#1689](https://github.com/ethereum/execution-specs/pull/1689)). -- 🐞 Turn off Block-Level Access List related checks when filling tests for Amsterdam ([#1737](https://github.com/ethereum/execution-specs/pull/1737)). - -#### `consume` - -- 🐞 Fix a bug with `consume sync` tests where some clients don't have JSON-RPC immediately available after syncing and can't yet serve the synced block ([#1670](https://github.com/ethereum/execution-specs/pull/1670)). - -### 📋 Misc - -- 🐞 WELDed the EEST tox environments relevant to producing documentation into EELS, and added a tool to cleanly add codespell whitelist entries. ([#1695](https://github.com/ethereum/execution-specs/pull/1659)). -- 🐞 Fix duplicate storage write issues for block access lists EIP-7928 implementation ([#1743](https://github.com/ethereum/execution-specs/pull/1743)). - -### 🧪 Test Cases - -- 🐞 Fix BALs opcode OOG test vectors by updating the Amsterdam commit hash in specs and validating appropriately on the testing side ([#2293](https://github.com/ethereum/execution-spec-tests/pull/2293)). -- ✨ Fix test vector for BALs SSTORE with OOG by pointing to updated specs; add new boundary conditions cases for SSTORE w/ OOG ([#2297](https://github.com/ethereum/execution-spec-tests/pull/2297)). -- ✨ Expand EIP-6110 modified contract tests, where the extra event log has no topics at all ([#1693](https://github.com/ethereum/execution-specs/pull/1693)). -- ✨ Add a CREATE/2 test cases for when it runs OOG on code deposit ([#1705](https://github.com/ethereum/execution-specs/pull/1705)). -- ✨ Expand cases to test *CALL opcodes causing OOG ([#1703](https://github.com/ethereum/execution-specs/pull/1703)). -- ✨ Add a test case for base fee in block check after London ([#1702](https://github.com/ethereum/execution-specs/pull/1702)). -- ✨ Add tests for `modexp` and `ripemd` precompiled contracts ([#1691](https://github.com/ethereum/execution-specs/pull/1691), [#1781](https://github.com/ethereum/execution-specs/pull/1781)). -- ✨ Add `ecrecover` precompile tests originating form `evmone` unittests ([#1685](https://github.com/ethereum/execution-specs/pull/1685)). -- ✨ Add test to validate withdarawls root ([#1746](https://github.com/ethereum/execution-specs/pull/1746)). -- ✨ Add test for old behavior of zero gasprice txs ([#1736](https://github.com/ethereum/execution-specs/pull/1736)). -- ✨ Add stack overflow tests and expand `BLOCKHASH` tests ([#1728](https://github.com/ethereum/execution-specs/pull/1728)). -- ✨ Ported tests for `ripemd` precompile going OOG for all forks ([#1732](https://github.com/ethereum/execution-specs/pull/1732)). -- ✨ Add tests for ecadd/ecmul/ecpairing constant gas repricing ([#1738](https://github.com/ethereum/execution-specs/pull/1738)). -- ✨ Add tests that EIP-1559 and EIP-2930 typed txs are invalid and void before their fork ([#1754](https://github.com/ethereum/execution-specs/pull/1754)). -- ✨ Add tests for an old validation rule for gas limit above 5000 ([#1731](https://github.com/ethereum/execution-specs/pull/1731)). -- ✨ Add tests for OOG in EXP, LOG and others ([#1686](https://github.com/ethereum/execution-specs/pull/1686)). -- ✨ Make EIP-7934 tests more dynamic and able to handle new header fields added in future forks ([#2022](https://github.com/ethereum/execution-specs/pull/2022)). - -## [v5.3.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v5.3.0) - 2025-10-09 - -## 🇯🇵 Summary - -EEST v5.3.0 is a follow-up from our main v5.0.0 [release](https://github.com/ethereum/execution-spec-tests/releases/tag/v5.0.0), with updated BPO1 and BPO2 values aligning with the testnet parameters. - -This release additionally includes fixes for tests in hive, as well as new test cases for EIP-7883, EIP-7934 and critical cases for EIP-7951 (added to EEST by @chfast following a coverage review of the test suite). - -## 🔑 Key Changes - -### 🛠️ Framework - -- ✨ Add benchmark-specific test wrapper (`benchmark_test`) that supports **EIP-7825** and create a benchmark code generator for common test pattern ([#1945](https://github.com/ethereum/execution-spec-tests/pull/1945)). - -#### `fill` - -- ✨ Added `--optimize-gas`, `--optimize-gas-output` and `--optimize-gas-post-processing` flags that allow to binary search the minimum gas limit value for a transaction in a test that still yields the same test result ([#1979](https://github.com/ethereum/execution-spec-tests/pull/1979)). -- ✨ Added `--watch` flag that monitors test files for changes and automatically re-runs the fill command when developing tests ([#2173](https://github.com/ethereum/execution-spec-tests/pull/2173)). -- 🔀 Upgraded ckzg version to 2.1.3 or newer for correct handling of points at infinity ([#2171](https://github.com/ethereum/execution-spec-tests/pull/2171)). -- 🔀 Move pytest marker registration for `fill` and `execute-*` from their respective ini files to the shared `pytest_plugins.shared.execute_fill` pytest plugin ([#2110](https://github.com/ethereum/execution-spec-tests/pull/2110)). -- ✨ Added an `--input.blobParams` CLI argument to the transition tool (`t8n`) invocation ([#2264](https://github.com/ethereum/execution-spec-tests/pull/2264)). - -#### `consume` - -- ✨ Add retry logic to RPC requests to fix flaky connection issues in Hive ([#2205](https://github.com/ethereum/execution-spec-tests/pull/2205)). -- 🛠️ Mark `consume sync` tests as `flaky` with 3 retires due to client sync inconsistencies ([#2252](https://github.com/ethereum/execution-spec-tests/pull/2252)). -- ✨ Add `consume direct` using `evmone-statetest` and `evmone-blockchaintest` ([#2243](https://github.com/ethereum/execution-spec-tests/pull/2243)). - -### 📋 Misc - -- ✨ Add tighter validation for EIP-7928 model coming from t8n when filling ([#2138](https://github.com/ethereum/execution-spec-tests/pull/2138)). -- ✨ Add flexible API for absence checks for EIP-7928 (BAL) tests ([#2124](https://github.com/ethereum/execution-spec-tests/pull/2124)). -- 🐞 Use `engine_newPayloadV5` for `>=Amsterdam` forks in `consume engine` ([#2170](https://github.com/ethereum/execution-spec-tests/pull/2170)). -- 🔀 Refactor EIP-7928 (BAL) absence checks into a friendlier class-based DevEx ([#2175](https://github.com/ethereum/execution-spec-tests/pull/2175)). -- 🐞 Tighten up validation for empty lists on Block-Level Access List tests ([#2118](https://github.com/ethereum/execution-spec-tests/pull/2118)). -- ✨ Added the `MemoryVariable` EVM abstraction to generate more readable bytecode when there's heavy use of variables that are stored in memory ([#1609](https://github.com/ethereum/execution-spec-tests/pull/1609)). -- 🐞 Fix an issue with `test_bal_block_rewards` where the block base fee was wrongfully overridden ([#2262](https://github.com/ethereum/execution-spec-tests/pull/2262)). -- ✨ Complete EIP checklist for EIP-7934 and update the checklist template to include block-level constraint checks ([#2282](https://github.com/ethereum/execution-spec-tests/pull/2282)). - -### 🧪 Test Cases - -- ✨ Add safe EIP-6110 workaround to allow Geth/Reth to pass invalid deposit request tests even thought they are out of spec ([#2177](https://github.com/ethereum/execution-spec-tests/pull/2177), [#2233](https://github.com/ethereum/execution-spec-tests/pull/2233)). -- ✨ Add an EIP-7928 test case targeting the `SELFDESTRUCT` opcode. ([#2159](https://github.com/ethereum/execution-spec-tests/pull/2159)). -- ✨ Add essential tests for coverage gaps in EIP-7951 (`p256verify` precompile) ([#2179](https://github.com/ethereum/execution-spec-tests/pull/2159), [#2203](https://github.com/ethereum/execution-spec-tests/pull/2203), [#2215](https://github.com/ethereum/execution-spec-tests/pull/2215), [#2216](https://github.com/ethereum/execution-spec-tests/pull/2216), [#2217](https://github.com/ethereum/execution-spec-tests/pull/2217), [#2218](https://github.com/ethereum/execution-spec-tests/pull/2218), [#2221](https://github.com/ethereum/execution-spec-tests/pull/2221), [#2229](https://github.com/ethereum/execution-spec-tests/pull/2229), [#2230](https://github.com/ethereum/execution-spec-tests/pull/2230), [#2237](https://github.com/ethereum/execution-spec-tests/pull/2237), [#2238](https://github.com/ethereum/execution-spec-tests/pull/2238)). -- ✨ Add EIP-7928 successful and OOG single-opcode tests ([#2118](https://github.com/ethereum/execution-spec-tests/pull/2118)). -- ✨ Add EIP-7928 tests for EIP-2930 interactions ([#2167](https://github.com/ethereum/execution-spec-tests/pull/2167)). -- ✨ Add EIP-7928 tests for NOOP operations ([#2178](https://github.com/ethereum/execution-spec-tests/pull/2178)). -- ✨ Add EIP-7928 tests for net-zero balance transfers ([#2280](https://github.com/ethereum/execution-spec-tests/pull/2280)). -- ✨ Add fork transition test cases for EIP-7934 ([#2282](https://github.com/ethereum/execution-spec-tests/pull/2282)). - -## [v5.0.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v5.0.0) - 2025-09-05 - -## 🇯🇵 Summary - -EEST Fujisan is our first full release for Osaka, the first full release since Pectra! - -In addition to the latest Osaka specific test cases, it includes re-filled `GeneralStateTests` from `ethereum/tests` (now fully maintained within EEST under `tests/static`) for Osaka adhering to the transaction gas limit cap from [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825). Further framework changes include new simulators, test formats and test types. - -## ⚔️ Future Weld with EELS - -EEST will merge with [EELS](https://github.com/ethereum/execution-specs) during **Q4 2025**, after which EEST becomes read-only for external contributors. - -**What this means?** - -- All EEST code moves to the EELS repository. -- New EEST framework location: `execution-specs/src/ethereum_spec_tests/`. -- New EEST tests location: `execution-specs/tests/eest/`. -- Future PRs go to EELS instead of EEST. - -**Important Notes:** - -- All PRs for tests and framework changes should still be directed at EEST until further notice. -- There will be a brief freeze on EEST contributions during Q4 "The Switch", after which contributors can continue as before, but in EELS. -- Test releases will continue from EEST as normal before, during, and after this transition. - -More information will be communicated accordingly through the normal communication channels. - -## ❗Current Status Quo - -### Test Fixtures Overview - -- `fixtures_static.tar.gz` has been deprecated. -- `fixtures_stable.tar.gz` & `fixtures_develop.tar.gz` now both contain re-filled static tests, `GeneralStateTests` from `ethereum/tests`, filled **from Cancun**. -- `fixtures_stable.tar.gz` contains tests filled for forks until Prague. -- `fixtures_develop.tar.gz` contains tests filled for forks until Osaka. -- `fixtures_benchmark.tar.gz` contains benchmark tests filled for only Prague. - -### EL Client Test Requirements - -**Prague Coverage (Mainnet):** - -- Run all `state_test`'s & `blockchain_test`'s from `fixtures_stable.tar.gz`. -- Run only `BlockchainTests` from the latest `ethereum/tests` [release](https://github.com/ethereum/tests/releases/tag/v17.2), filled **until Prague**. - -**Fusaka Coverage (Including Mainnet):** - -- Run all `state_test`'s & `blockchain_test`'s from `fixtures_develop.tar.gz`. -- Run only `BlockchainTests` from the latest `ethereum/tests` [release](https://github.com/ethereum/tests/releases/tag/v17.2), filled **until Prague**. - -**Note**: If you require `GeneralStateTests` from `ethereum/tests` **filled for forks before Cancun** then you must get these from the latest `ethereum/tests` [release](https://github.com/ethereum/tests/releases/tag/v17.2). - -### Benchmark Tests - -For the most up-to-date benchmark tests, use `fixtures_benchmark.tar.gz`. - -> **Note**: Benchmark tests for Osaka (compliant with EIP-7825 transaction gas limit cap) will be added in a future feature release. - -### Test Fixture Formats - -This release includes 2 new test formats designed primarily for [Hive](https://github.com/ethereum/hive) simulators: - -- [**`blockchain_tests_engine_x`**](https://eest.ethereum.org/v5.0.0/running_tests/test_formats/blockchain_test_engine_x/?h=#blockchain-engine-x-tests): An optimized version of `blockchain_tests_engine` where multiple tests share the same genesis state, allowing multiple tests to run on a single client instantiation within Hive's `consume-engine`. The standard format requires a fresh client startup for each test. Due its combined genesis state, this is additionally the primary format used by the Nethermind team for benchmarking. - -- [**`blockchain_tests_sync`**](https://eest.ethereum.org/main/running_tests/test_formats/blockchain_test_sync/?h=sync#blockchain-engine-sync-tests): A new format adjacent to the existing `blockchain_tests_engine` format. Used specifically for the upcoming `consume-sync` simulator, which delivers engine payloads from test fixtures to the client under test, then sync's a separate client to it. This test fixture is only marked to be filled for the [EIP-7934](https://eips.ethereum.org/EIPS/eip-7934) block RLP limit tests in Osaka. - -### Tooling & Simulators - -Improved tooling and new Hive simulators are additionally included in this release: - -- [**`execute remote`**](https://eest.ethereum.org/main/running_tests/execute/remote/#running-test-on-a-live-remote-network): this command now supports optional [Engine RPC endpoints](https://eest.ethereum.org/main/running_tests/execute/remote/#engine-rpc-endpoint-optional) (`--engine-endpoint`) with JWT authentication #2070. - - This allows manual control over block creation and transaction inclusion for more deterministic test execution on live networks. Previously, `execute remote` could only submit transactions and rely on the network's automatic block production, but now it can actively drive chain progression by creating blocks on-demand via the Engine API. -- [**`consume sync`**](https://eest.ethereum.org/main/running_tests/running/#sync): Adjacent to `consume-engine`, designed to work with the `blockchain_tests_sync` format for testing client sync scenarios. -- [**`execute blobs`**](https://eest.ethereum.org/main/running_tests/execute/hive/#the-eestexecute-blobs-simulator): A new Hive specific simulator that uses the EEST execute pytest plugin. Sends blob transactions to the client under test and verifies its `engine_getBlobsVX` endpoint. Requires tests to be written with a new python test format `blob_transaction_test`. Primarily used to test PeerDAS from the EL perspective. -- [**`execute eth config`**](https://eest.ethereum.org/main/running_tests/execute/eth_config/): A command used to test the `eth_config` endpoint from [EIP-7910](https://eips.ethereum.org/EIPS/eip-7910). Can be ran [remotely](https://github.com/ethereum/execution-spec-tests/issues/2049) or within Hive. - -### Filling For Stateless Clients - -A `witness-filler` extension is included in this release, allowing for tests to be filled that include an `executionWitness` for each fixture #2066. This essentially calls an external executable written in rust, and hence must be installed for usage within `fill` using the `--witness` flag. The current approach is below: - -```sh -cargo install --git https://github.com/kevaundray/reth.git --branch jsign-witness-filler witness-filler -uv run fill ... --output=fixtures-witness --witness --clean -``` - -> **Note**: The `witness-filler` executable is not maintained by EEST so we cannot help with any issues. - -## 💥 Breaking Changes - -### Important changes for EEST superusers - -- EEST now requires `uv>=0.7.0` ([#1904](https://github.com/ethereum/execution-spec-tests/pull/1904)). If your version of `uv` is too old. -- When filling fixtures transition forks are included within there respective "to" fork, where `--fork Osaka` will now include `PragueToOsakaAtTime15k`. Previously transitions fork would only be included when filling with `--from Prague --until Osaka` flags. -- Python 3.10 support was removed in this release ([#1808](https://github.com/ethereum/execution-spec-tests/pull/1808)). -- EEST no longer allows usage of Yul code in Python tests. From now on, please make use of our opcode wrapper. Yul code is now only allowed in the "static tests" located in `./tests/static/` (these are test cases defined by JSON and YAML files instead of Python test functions that were originally maintained in [ethereum/tests](https://github.com/ethereum/tests)). -- In order to fill the static tests (which is not the case by default), please ensure that `solc` is located in your `PATH`. -- The output behavior of `fill` has changed ([#1608](https://github.com/ethereum/execution-spec-tests/pull/1608)): - - Before: `fill` wrote fixtures into the directory specified by the `--output` flag (default: `fixtures`). This could have many unintended consequences, including unexpected errors if old or invalid fixtures existed in the directory (for details see [#1030](https://github.com/ethereum/execution-spec-tests/issues/1030)). - - Now: `fill` will exit without filling any tests if the specified directory exists and is not-empty. This may be overridden by adding the `--clean` flag, which will first remove the specified directory. -- Writing debugging information to the EVM "dump directory" by default has been disabled. To obtain debug output, the `--evm-dump-dir` flag must now be explicitly set. As a consequence, the now redundant `--skip-evm-dump` option was removed ([#1874](https://github.com/ethereum/execution-spec-tests/pull/1874)). This undoes functionality originally introduced in [#999](https://github.com/ethereum/execution-spec-tests/pull/999) and [#1150](https://github.com/ethereum/execution-spec-tests/pull/1150). - -### Feature `zkevm` updated to `benchmark` - -Due to the crossover between `zkevm` and `benchmark` tests, all instances of the former have been replaced with the latter nomenclature. Repository PR labels and titles are additionally updated to reflect this change. - -This update renames the `zkevm` feature release to `benchmark` and further expands the latter for 1M, 10M, 30M, 45M, 60M, 90M, and 120M block gas limits in `fixtures_benchmark.tar.gz`. - -To select a test for a given gas limit, the IDs of the tests have been expanded to contain `benchmark-gas-value_XM`, where `X` can be any of the aforementioned values. - -The benchmark release also now includes BlockchainEngineX format that combines most of the tests into a minimal amount of genesis files. For more info see [Blockchain Engine X Tests](https://eest.ethereum.org/main/running_tests/test_formats/blockchain_test_engine_x/) in the EEST documentation. - -Users can select any of the artifacts depending on their benchmarking or testing needs for their provers. - -## 🔑 Other Key Changes - -### 🛠️ Framework - -#### 🔀 Refactoring - -- 🔀 Move `TransactionType` enum from test file to proper module location in `ethereum_test_types.transaction_types` for better code organization and reusability. -- ✨ Opcode classes now validate keyword arguments and raise `ValueError` with clear error messages. -- 🔀 This PR removes the `solc` requirement to fill Python test cases. Regular test contributors no longer need to concern themselves with `solc` and, as such, the `solc-select` dependency has been removed. The remaining tests that used Yul have been ported to the EEST opcode wrapper mini-lang and the use of Yul in Python tests is no longer supported. Maintainers only: To fill the "static" JSON and YAML tests (`./tests/static/`) locally, `solc` (ideally v0.8.24) must be available in your PATH. -- 🔀 Updated default block gas limit from 36M to 45M to match mainnet environment. -- 🔀 Refactor fork logic to include transition forks within there "to" fork ([#2051](https://github.com/ethereum/execution-spec-tests/pull/2051)). - -#### `fill` - -- ✨ Add the `ported_from` test marker to track Python test cases that were converted from static fillers in [ethereum/tests](https://github.com/ethereum/tests) repository ([#1590](https://github.com/ethereum/execution-spec-tests/pull/1590)). -- ✨ Add a new pytest plugin, `ported_tests`, that lists the static fillers and PRs from `ported_from` markers for use in the coverage Github Workflow ([#1634](https://github.com/ethereum/execution-spec-tests/pull/1634)). -- ✨ Enable two-phase filling of fixtures with pre-allocation groups and add a `BlockchainEngineXFixture` format ([#1706](https://github.com/ethereum/execution-spec-tests/pull/1706), [#1760](https://github.com/ethereum/execution-spec-tests/pull/1760)). -- ✨ Add `--generate-all-formats` flag to enable generation of all fixture formats including `BlockchainEngineXFixture` in a single command; enable `--generate-all-formats` automatically for tarball output, `--output=fixtures.tar.gz`, [#1855](https://github.com/ethereum/execution-spec-tests/pull/1855). -- 🔀 Refactor: Encapsulate `fill`'s fixture output options (`--output`, `--flat-output`, `--single-fixture-per-file`) into a `FixtureOutput` class ([#1471](https://github.com/ethereum/execution-spec-tests/pull/1471),[#1612](https://github.com/ethereum/execution-spec-tests/pull/1612)). -- ✨ Don't warn about a "high Transaction gas_limit" for `zkevm` tests ([#1598](https://github.com/ethereum/execution-spec-tests/pull/1598)). -- 🐞 `fill` no longer writes generated fixtures into an existing, non-empty output directory; it must now be empty or `--clean` must be used to delete it first ([#1608](https://github.com/ethereum/execution-spec-tests/pull/1608)). -- 🐞 `zkevm` marked tests have been removed from `tests-deployed` tox environment into its own separate workflow `tests-deployed-zkevm` and are filled by `evmone-t8n` ([#1617](https://github.com/ethereum/execution-spec-tests/pull/1617)). -- ✨ Field `postStateHash` is now added to all `blockchain_test` and `blockchain_test_engine` tests that use `exclude_full_post_state_in_output` in place of `postState`. Fixes `evmone-blockchaintest` test consumption and indirectly fixes coverage runs for these tests ([#1667](https://github.com/ethereum/execution-spec-tests/pull/1667)). -- 🔀 Changed INVALID_DEPOSIT_EVENT_LAYOUT to a BlockException instead of a TransactionException ([#1773](https://github.com/ethereum/execution-spec-tests/pull/1773)). -- 🔀 Disabled writing debugging information to the EVM "dump directory" to improve performance. To obtain debug output, the `--evm-dump-dir` flag must now be explicitly set. As a consequence, the now redundant `--skip-evm-dump` option was removed ([#1874](https://github.com/ethereum/execution-spec-tests/pull/1874)). -- ✨ Generate unique addresses with Python for compatible static tests, instead of using hard-coded addresses from legacy static test fillers ([#1781](https://github.com/ethereum/execution-spec-tests/pull/1781)). -- ✨ Added support for the `--benchmark-gas-values` flag in the `fill` command, allowing a single genesis file to be used across different gas limit settings when generating fixtures. ([#1895](https://github.com/ethereum/execution-spec-tests/pull/1895)). -- ✨ Static tests can now specify a maximum fork where they should be filled for ([#1977](https://github.com/ethereum/execution-spec-tests/pull/1977)). -- ✨ Static tests can now be filled in every format using `--generate-all-formats` ([#2006](https://github.com/ethereum/execution-spec-tests/pull/2006)). -- 💥 Flag `--flat-output` has been removed due to having been unneeded for an extended period of time ([#2018](https://github.com/ethereum/execution-spec-tests/pull/2018)). -- ✨ Add support for `BlockchainEngineSyncFixture` format for tests marked with `pytest.mark.verify_sync` to enable client synchronization testing via `consume sync` command ([#2007](https://github.com/ethereum/execution-spec-tests/pull/2007)). -- ✨ Framework is updated to include BPO ([EIP-7892](https://eips.ethereum.org/EIPS/eip-7892)) fork markers to enable the filling of BPO tests ([#2050](https://github.com/ethereum/execution-spec-tests/pull/2050)). -- ✨ Generate and include execution witness data in blockchain fixtures if `--witness` is specified ([#2066](https://github.com/ethereum/execution-spec-tests/pull/2066)). - -#### `consume` - -- ✨ Add `--extract-to` parameter to `consume cache` command for direct fixture extraction to specified directory, replacing the need for separate download scripts. ([#1861](https://github.com/ethereum/execution-spec-tests/pull/1861)). -- 🐞 Fix `consume cache --cache-folder` parameter being ignored, now properly caches fixtures in the specified directory instead of always using the default system cache location. -- 🐞 Fix the `consume_direct.sh` script generated by `consume` in the `--evm-dump` dir by quoting test IDs [#1987](https://github.com/ethereum/execution-spec-tests/pull/1987). -- 🔀 `consume` now automatically avoids GitHub API calls when using direct release URLs (better for CI environments), while release specifiers like `stable@latest` continue to use the API for version resolution ([#1788](https://github.com/ethereum/execution-spec-tests/pull/1788)). -- 🔀 Refactor consume simulator architecture to use explicit pytest plugin structure with forward-looking architecture ([#1801](https://github.com/ethereum/execution-spec-tests/pull/1801)). -- 🔀 Add exponential retry logic to initial fcu within consume engine ([#1815](https://github.com/ethereum/execution-spec-tests/pull/1815)). -- ✨ Add `consume sync` command to test client synchronization capabilities by having one client sync from another via Engine API and P2P networking ([#2007](https://github.com/ethereum/execution-spec-tests/pull/2007)). -- 💥 Removed the `consume hive` command, this was a convenience command that ran `consume rlp` and `consume engine` in one pytest session; the individual commands should now be used instead ([#2008](https://github.com/ethereum/execution-spec-tests/pull/2008)). -- ✨ Update the hive ruleset to include BPO ([EIP-7892](https://eips.ethereum.org/EIPS/eip-7892)) forks ([#2050](https://github.com/ethereum/execution-spec-tests/pull/2050)). - -#### `execute` - -- ✨ Added new `Blob` class which can use the ckzg library to generate valid blobs at runtime ([#1614](https://github.com/ethereum/execution-spec-tests/pull/1614)). -- ✨ Added `blob_transaction_test` execute test spec, which allows tests that send blob transactions to a running client and verifying its `engine_getBlobsVX` endpoint behavior ([#1644](https://github.com/ethereum/execution-spec-tests/pull/1644)). -- ✨ Added `execute eth-config` command to test the `eth_config` RPC endpoint of a client, and includes configurations by default for Mainnet, Sepolia, Holesky, and Hoodi ([#1863](https://github.com/ethereum/execution-spec-tests/pull/1863)). -- ✨ Command `execute remote` now allows specification of an Engine API endpoint to drive the chain via `--engine-endpoint` and either `--engine-jwt-secret` or `--engine-jwt-secret-file`. This mode is useful when there's no consensus client connected to the execution client so `execute` will automatically request blocks via the Engine API when it sends transactions ([#2070](https://github.com/ethereum/execution-spec-tests/pull/2070)). -- ✨ Added `--address-stubs` flag to the `execute` command which allows to specify a JSON-formatted string, JSON file or YAML file which contains label-to-address of specific pre-deployed contracts already existing in the network where the tests are executed ([#2073](https://github.com/ethereum/execution-spec-tests/pull/2073)). - -### 📋 Misc - -- ✨ Add pypy3.11 support ([#1854](https://github.com/ethereum/execution-spec-tests/pull/1854)). -- 🔀 Use only relative imports in `tests/` directory ([#1848](https://github.com/ethereum/execution-spec-tests/pull/1848)). -- 🔀 Convert absolute imports to relative imports in `src/` directory for better code maintainability ([#1907](https://github.com/ethereum/execution-spec-tests/pull/1907)). -- 🔀 Misc. doc updates, including a navigation footer ([#1846](https://github.com/ethereum/execution-spec-tests/pull/1846)). -- 🔀 Remove Python 3.10 support ([#1808](https://github.com/ethereum/execution-spec-tests/pull/1808)). -- 🔀 Modernize codebase with Python 3.11 language features ([#1812](https://github.com/ethereum/execution-spec-tests/pull/1812)). -- ✨ Add changelog formatting validation to CI to ensure consistent punctuation in bullet points [#1691](https://github.com/ethereum/execution-spec-tests/pull/1691). -- ✨ Added the [EIP checklist template](https://eest.ethereum.org/main/writing_tests/checklist_templates/eip_testing_checklist_template/) that serves as a reference to achieve better coverage when implementing tests for new EIPs ([#1327](https://github.com/ethereum/execution-spec-tests/pull/1327)). -- ✨ Added [Post-Mortems of Missed Test Scenarios](https://eest.ethereum.org/main/writing_tests/post_mortems/) to the documentation that serves as a reference list of all cases that were missed during the test implementation phase of a new EIP, and includes the steps taken in order to prevent similar test cases to be missed in the future ([#1327](https://github.com/ethereum/execution-spec-tests/pull/1327)). -- ✨ Add documentation "Running Tests" that explains the different methods available to run EEST tests and reference guides for running `consume` and `hive`: ([#1172](https://github.com/ethereum/execution-spec-tests/pull/1172)). -- ✨ Added a new `eest` sub-command, `eest info`, to easily print a cloned EEST repository's version and the versions of relevant tools, e.g., `python`, `uv` ([#1621](https://github.com/ethereum/execution-spec-tests/pull/1621)). -- ✨ Add `CONTRIBUTING.md` for execution-spec-tests and improve coding standards documentation ([#1604](https://github.com/ethereum/execution-spec-tests/pull/1604)). -- ✨ Add `CLAUDE.md` to help working in @ethereum/execution-spec-tests with [Claude Code](https://docs.anthropic.com/en/docs/claude-code/overview) ([#1749](https://github.com/ethereum/execution-spec-tests/pull/1749)). -- ✨ Use `codespell` instead of `pyspelling` to spell-check python and markdown sources ([#1715](https://github.com/ethereum/execution-spec-tests/pull/1715)). -- 🔀 Updated from pytest 7 to [pytest 8](https://docs.pytest.org/en/stable/changelog.html#features-and-improvements), benefits include improved type hinting and hook typing, stricter mark handling, and clearer error messages for plugin and metadata development ([#1433](https://github.com/ethereum/execution-spec-tests/pull/1433)). -- 🐞 Fix bug in ported-from plugin and coverage script that made PRs fail with modified tests that contained no ported tests ([#1661](https://github.com/ethereum/execution-spec-tests/pull/1661)). -- 🔀 Refactor the `click`-based CLI interface used for pytest-based commands (`fill`, `execute`, `consume`) to make them more extensible ([#1654](https://github.com/ethereum/execution-spec-tests/pull/1654)). -- 🔀 Split `src/ethereum_test_types/types.py` into several files to improve code organization ([#1665](https://github.com/ethereum/execution-spec-tests/pull/1665)). -- ✨ Added `extract_config` command to extract genesis files used to launch clients in hive ([#1740](https://github.com/ethereum/execution-spec-tests/pull/1740)). -- ✨ Added automatic checklist generation for every EIP inside of the `tests` folder. The checklist is appended to each EIP in the documentation in the "Test Case Reference" section ([#1679](https://github.com/ethereum/execution-spec-tests/pull/1679), [#1718](https://github.com/ethereum/execution-spec-tests/pull/1718)). -- 🔀 Add macOS hive development mode workaround to the docs [#1786](https://github.com/ethereum/execution-spec-tests/pull/1786). -- 🔀 Refactor and clean up of exceptions including EOF exceptions within client specific mappers [#1803](https://github.com/ethereum/execution-spec-tests/pull/1803). -- 🔀 Rename `tests/zkevm/` to `tests/benchmark/` and replace the `zkevm` pytest mark with `benchmark` [#1804](https://github.com/ethereum/execution-spec-tests/pull/1804). -- 🔀 Add fixture comparison check to optimize coverage workflow in CI ([#1833](https://github.com/ethereum/execution-spec-tests/pull/1833)). -- 🔀 Move `TransactionType` enum from test file to proper module location in `ethereum_test_types.transaction_types` for better code organization and reusability ([#1763](https://github.com/ethereum/execution-spec-tests/pull/1673)). -- ✨ Opcode classes now validate keyword arguments and raise `ValueError` with clear error messages ([#1739](https://github.com/ethereum/execution-spec-tests/pull/1739), [#1856](https://github.com/ethereum/execution-spec-tests/pull/1856)). -- ✨ All commands (`fill`, `consume`, `execute`) now work without having to clone the repository, e.g. `uv run --with git+https://github.com/ethereum/execution-spec-tests.git consume` now works from any folder ([#1863](https://github.com/ethereum/execution-spec-tests/pull/1863)). -- 🔀 Move Prague to stable and Osaka to develop ([#1573](https://github.com/ethereum/execution-spec-tests/pull/1573)). -- ✨ Add a `pytest.mark.with_all_typed_transactions` marker that creates default typed transactions for each `tx_type` supported by the current `fork` ([#1890](https://github.com/ethereum/execution-spec-tests/pull/1890)). -- ✨ Add basic support for ``Amsterdam`` fork in order to begin testing Glamsterdam ([#2069](https://github.com/ethereum/execution-spec-tests/pull/2069)). -- ✨ [EIP-7928](https://eips.ethereum.org/EIPS/eip-7928): Add initial framework support for `Block Level Access Lists (BAL)` testing for Amsterdam ([#2067](https://github.com/ethereum/execution-spec-tests/pull/2067)). - -### 🧪 Test Cases - -- ✨ [BloatNet](https://bloatnet.info)/Multidimensional Metering: Add benchmarks to be used as part of the BloatNet project and also for Multidimensional Metering. -- ✨ [EIP-7951](https://eips.ethereum.org/EIPS/eip-7951): Add additional test cases for modular comparison. -- 🔀 Refactored `BLOBHASH` opcode context tests to use the `pre_alloc` plugin in order to avoid contract and EOA address collisions ([#1637](https://github.com/ethereum/execution-spec-tests/pull/1637)). -- 🔀 Refactored `SELFDESTRUCT` opcode collision tests to use the `pre_alloc` plugin in order to avoid contract and EOA address collisions ([#1643](https://github.com/ethereum/execution-spec-tests/pull/1643)). -- ✨ EIP-7594: Sanity test cases to send blob transactions and verify `engine_getBlobsVX` using the `execute` command ([#1644](https://github.com/ethereum/execution-spec-tests/pull/1644),[#1884](https://github.com/ethereum/execution-spec-tests/pull/1884)). -- 🔀 Refactored EIP-145 static tests into python ([#1683](https://github.com/ethereum/execution-spec-tests/pull/1683)). -- ✨ EIP-7823, EIP-7883: Add test cases for ModExp precompile gas-cost updates and input limits on Osaka ([#1579](https://github.com/ethereum/execution-spec-tests/pull/1579), [#1729](https://github.com/ethereum/execution-spec-tests/pull/1729), [#1881](https://github.com/ethereum/execution-spec-tests/pull/1881)). -- ✨ [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825): Add test cases for the transaction gas limit of 2^24 gas ([#1711](https://github.com/ethereum/execution-spec-tests/pull/1711), [#1882](https://github.com/ethereum/execution-spec-tests/pull/1882)). -- ✨ [EIP-7951](https://eips.ethereum.org/EIPS/eip-7951): add test cases for `P256VERIFY` precompile to support secp256r1 curve [#1670](https://github.com/ethereum/execution-spec-tests/pull/1670). -- ✨ Introduce blockchain tests for benchmark to cover the scenario of pure ether transfers [#1742](https://github.com/ethereum/execution-spec-tests/pull/1742). -- ✨ [EIP-7934](https://eips.ethereum.org/EIPS/eip-7934): Add test cases for the block RLP max limit of 10MiB ([#1730](https://github.com/ethereum/execution-spec-tests/pull/1730)). -- ✨ [EIP-7939](https://eips.ethereum.org/EIPS/eip-7939): Add count leading zeros (CLZ) opcode tests for Osaka ([#1733](https://github.com/ethereum/execution-spec-tests/pull/1733)). -- ✨ [EIP-7934](https://eips.ethereum.org/EIPS/eip-7934): Add additional test cases for block RLP max limit with all typed transactions and for a log-creating transactions ([#1890](https://github.com/ethereum/execution-spec-tests/pull/1890)). -- ✨ [EIP-7825](https://eips.ethereum.org/EIPS/eip-7825): Pre-Osaka tests have been updated to either (1) dynamically adapt to the transaction gas limit cap, or (2) reduce overall gas consumption to fit the new limit ([#1924](https://github.com/ethereum/EIPs/pull/1924), [#1928](https://github.com/ethereum/EIPs/pull/1928), [#1980](https://github.com/ethereum/EIPs/pull/1980)). -- ✨ [EIP-7918](https://eips.ethereum.org/EIPS/eip-7918): Blob base fee bounded by execution cost test cases (initial), includes some adjustments to [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844) tests ([#1685](https://github.com/ethereum/execution-spec-tests/pull/1685)). -- 🔀 Adds the max blob transaction limit to the tests including updates to [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844) for Osaka ([#1884](https://github.com/ethereum/execution-spec-tests/pull/1884)). -- 🐞 Fix issues when filling block rlp size limit tests with ``--generate-pre-alloc-groups`` ([#1989](https://github.com/ethereum/execution-spec-tests/pull/1989)). -- ✨ [EIP-7928](https://eips.ethereum.org/EIPS/eip-7928): Add test cases for `Block Level Access Lists (BAL)` to Amsterdam ([#2067](https://github.com/ethereum/execution-spec-tests/pull/2067)). -- 🐞 Fix issues with `Block Level Access Lists (BAL)` tests for Amsterdam ([#2121](https://github.com/ethereum/execution-spec-tests/pull/2121)). - -## [v4.5.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v4.5.0) - 2025-05-14 - -### 💥 Breaking Change - -#### EOF removed from Osaka - -Following ["Interop Testing Call 34"](https://github.com/ethereum/pm/issues/1499) and the procedural EIPs [PR](https://github.com/ethereum/EIPs/pull/9703) the decision to remove EOF from Osaka was made. - -To accommodate EOF testing for the interim within EEST, its tests have migrated to a new `tests/unscheduled` folder. This folder will now contain tests for features that are not yet CFI'd in any fork. When EOF is CFI'd for a fork in the future, all tests will be moved from unscheduled to the respective future fork folder. - -A new fork `EOFv1` has additionally been created to fill and consume EOF related fixtures. Client tests fillers such as `evmone` (and client consumers) will now need to use this fork name. - -### 🛠️ Framework - -- ✨ Add an empty account function for usage within fill and execute ([#1482](https://github.com/ethereum/execution-spec-tests/pull/1482)). -- ✨ Added `TransactionException.INTRINSIC_GAS_BELOW_FLOOR_GAS_COST` exception to specifically catch the case where the intrinsic gas cost is insufficient due to the data floor gas cost ([#1582](https://github.com/ethereum/execution-spec-tests/pull/1582)). - -### 📋 Misc - -- ✨ Engine API updates for Osaka, add `get_blobs` rpc method ([#1510](https://github.com/ethereum/execution-spec-tests/pull/1510)). -- ✨ The EIP Version checker has been moved from `fill` and `execute` to it's own command-line tool `check_eip_versions` that gets ran daily as a Github Action ([#1537](https://github.com/ethereum/execution-spec-tests/pull/1537)). -- 🔀 Add new `tests/unscheduled` folder, move EOF from Osaka to unscheduled, add `EOFv1` fork name for EOF tests ([#1507](https://github.com/ethereum/execution-spec-tests/pull/1507)). -- ✨ CI features now contain an optional field to skip them from EEST full releases, `benchmark` and EOF features are now feature only ([#1596](https://github.com/ethereum/execution-spec-tests/pull/1596)). -- 🐞 Don't attempt to install `solc` via `solc-select` on ARM (official Linux ARM builds of `solc` are not available at the time of writing, cf [ethereum/solidity#11351](https://github.com/ethereum/solidity/issues/11351)) and add a version sanity check ([#1556](https://github.com/ethereum/execution-spec-tests/pull/1556)). - -### 🧪 Test Cases - -- 🔀 Automatically apply the `benchmark` marker to all tests under `./tests/benchmark/` and `./tests/prague/eip2537_bls_12_381_precompiles/` via conftest configuration ([#1534](https://github.com/ethereum/execution-spec-tests/pull/1534)). -- ✨ Port [calldataload](https://github.com/ethereum/tests/blob/ae4791077e8fcf716136e70fe8392f1a1f1495fb/src/GeneralStateTestsFiller/VMTests/vmTests/calldatacopyFiller.yml) and [calldatasize](https://github.com/ethereum/tests/blob/81862e4848585a438d64f911a19b3825f0f4cd95/src/GeneralStateTestsFiller/VMTests/vmTests/calldatasizeFiller.yml) tests ([#1236](https://github.com/ethereum/execution-spec-tests/pull/1236)). - -## [v4.4.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v4.4.0) - 2025-04-29 - -### 💥 Breaking Change - -#### `fixtures_static` - -A new fixture tarball has been included in this release: `fixtures_static.tar.gz`. - -This tarball contains all tests inside of [`./tests/static`](https://github.com/ethereum/execution-spec-tests/tree/main/tests/static), which at this point only contains all tests copied from [`GeneralStateTests` in `ethereum/tests@7dc757ec132e372b6178a016b91f4c639f366c02`](https://github.com/ethereum/tests/tree/7dc757ec132e372b6178a016b91f4c639f366c02/src/GeneralStateTestsFiller). - -The tests have been filled using the new static test filler introduced in [#1336](https://github.com/ethereum/execution-spec-tests/pull/1336), and enhanced in [#1362](https://github.com/ethereum/execution-spec-tests/pull/1362) and [#1439](https://github.com/ethereum/execution-spec-tests/pull/1439). - -Users can expect that all tests currently living in [ethereum/tests](https://github.com/ethereum/tests/tree/develop/src) should eventually make its way into [`./tests/static`](https://github.com/ethereum/execution-spec-tests/tree/main/tests/static) and can rely that these tests, filled for new forks even, will be included in `fixtures_static.tar.gz`. - -#### `fixtures_benchmark` - -Another new fixture tarball has been included in this release: `fixtures_benchmark.tar.gz`. - -Includes tests that are tailored specifically to test the execution layer proof generators. - -### 🛠️ Framework - -#### `fill` - -- 🐞 Fix the reported fixture source URLs for the case of auto-generated tests ([#1488](https://github.com/ethereum/execution-spec-tests/pull/1488)). - -#### `consume` - -- 🐞 Fix the Hive commands used to reproduce test executions that are displayed in test descriptions in the Hive UI ([#1494](https://github.com/ethereum/execution-spec-tests/pull/1494)). -- 🐞 Fix consume direct fails for geth blockchain tests ([#1502](https://github.com/ethereum/execution-spec-tests/pull/1502)). - -### 📋 Misc - -### 🧪 Test Cases - -- ✨ [EIP-7702](https://eips.ethereum.org/EIPS/eip-7702): Test that DELEGATECALL to a 7702 target works as intended ([#1485](https://github.com/ethereum/execution-spec-tests/pull/1485)). -- ✨ [EIP-2573](https://eips.ethereum.org/EIPS/eip-2537): Includes a BLS12 point generator, alongside additional coverage many of the precompiles ([#1350](https://github.com/ethereum/execution-spec-tests/pull/1350)), ([#1505](https://github.com/ethereum/execution-spec-tests/pull/1505)). -- ✨ Add all [`GeneralStateTests` from `ethereum/tests`](https://github.com/ethereum/tests/tree/7dc757ec132e372b6178a016b91f4c639f366c02/src/GeneralStateTestsFiller) to `execution-spec-tests` located now at [tests/static/state_tests](https://github.com/ethereum/execution-spec-tests/tree/main/tests/static/state_tests) ([#1442](https://github.com/ethereum/execution-spec-tests/pull/1442)). -- ✨ [EIP-2929](https://eips.ethereum.org/EIPS/eip-2929): Test that precompile addresses are cold/warm depending on the fork they are activated ([#1495](https://github.com/ethereum/execution-spec-tests/pull/1495)). - -## [v4.3.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v4.3.0) - 2025-04-18 - -### 💥 Breaking Change - -#### Consume engine strict exception checking - -`consume engine` now checks exceptions returned by the execution clients in their Engine API responses, specifically in the `validationError`field of the `engine_newPayloadVX` method. - -While not strictly a breaking change since tests will continue to run normally, failures are expected if a client modifies their exception messages. - -This feature can be disabled by using `--disable-strict-exception-matching` for specific clients or forks. - -### 🛠️ Framework - -#### `fill` - -- ✨ The `static_filler` plug-in now has support for static state tests (from [GeneralStateTests](https://github.com/ethereum/tests/tree/develop/src/GeneralStateTestsFiller)) ([#1362](https://github.com/ethereum/execution-spec-tests/pull/1362)). -- ✨ Introduce `pytest.mark.exception_test` to mark tests that contain an invalid transaction or block ([#1436](https://github.com/ethereum/execution-spec-tests/pull/1436)). -- 🐞 Fix `DeprecationWarning: Pickle, copy, and deepcopy support will be removed from itertools in Python 3.14.` by avoiding use `itertools` object in the spec `BaseTest` pydantic model ([#1414](https://github.com/ethereum/execution-spec-tests/pull/1414)). -- ✨ An optional configuration flag to override the maximum gas limit in the environment for filling or executing tests is now available. The `--block-gas-limit` flag overrides the default block gas limit during filling. The `--transaction-gas-limit` flag overrides the maximum for transactions during execution. ([#1470](https://github.com/ethereum/execution-spec-tests/pull/1470)). - -#### `consume` - -- 🐞 Fix fixture tarball downloading with regular, non-Github release URLS and with numerical versions in regular release specs, e.g., `stable@v4.2.0` ([#1437](https://github.com/ethereum/execution-spec-tests/pull/1437)). -- ✨ `consume engine` now has strict exception mapping enabled by default ([#1416](https://github.com/ethereum/execution-spec-tests/pull/1416)). - -#### Tools - -- 🔀 `generate_system_contract_deploy_test` test generator has been updated to handle system contracts that are not allowed to be absent when the fork happens ([#1394](https://github.com/ethereum/execution-spec-tests/pull/1394)). -- ✨ Add `generate_system_contract_error_test` to generate tests on system contracts that invalidate a block in case of error ([#1394](https://github.com/ethereum/execution-spec-tests/pull/1394)). - -#### Exceptions - -- ✨ `BlockException.SYSTEM_CONTRACT_EMPTY`: Raised when a required system contract was not found in the state by the time it was due to execution with a system transaction call ([#1394](https://github.com/ethereum/execution-spec-tests/pull/1394)). -- ✨ `BlockException.SYSTEM_CONTRACT_CALL_FAILED`: Raised when a system contract call made by a system transaction fails ([#1394](https://github.com/ethereum/execution-spec-tests/pull/1394)). -- ✨ `BlockException.INVALID_BLOCK_HASH`: Raised when the calculated block hash does not match the expectation (Currently only during Engine API calls) ([#1416](https://github.com/ethereum/execution-spec-tests/pull/1416)). -- ✨ `BlockException.INVALID_VERSIONED_HASHES`: Raised when a discrepancy is found between versioned hashes in the payload and the ones found in the transactions ([#1416](https://github.com/ethereum/execution-spec-tests/pull/1416)). - -### 🧪 Test Cases - -- ✨ [EIP-7702](https://eips.ethereum.org/EIPS/eip-7702): Test precompile case in same transaction as delegation without extra gas in case of precompile code execution; parametrize all call opcodes in existing precompile test ([#1431](https://github.com/ethereum/execution-spec-tests/pull/1431)). -- ✨ [EIP-7702](https://eips.ethereum.org/EIPS/eip-7702): Add invalid nonce authorizations tests for the case of multiple signers when the sender's nonce gets increased ([#1441](https://github.com/ethereum/execution-spec-tests/pull/1441)). -- ✨ [EIP-7702](https://eips.ethereum.org/EIPS/eip-7702): Add a test that verifies that set code transactions are correctly rejected before Prague activation ([#1463](https://github.com/ethereum/execution-spec-tests/pull/1463)). -- ✨ [EIP-7623](https://eips.ethereum.org/EIPS/eip-7623): Additionally parametrize transaction validity tests with the `to` set to an EOA account (previously only contracts) ([#1422](https://github.com/ethereum/execution-spec-tests/pull/1422)). -- ✨ [EIP-7251](https://eips.ethereum.org/EIPS/eip-7251): Add EIP-7251 test cases for modified consolidations contract that allows more consolidations ([#1465](https://github.com/ethereum/execution-spec-tests/pull/1465)). -- ✨ [EIP-6110](https://eips.ethereum.org/EIPS/eip-6110): Add extra deposit request edge cases, sending eth to the deposit contract while sending a deposit request ([#1467](https://github.com/ethereum/execution-spec-tests/pull/1467)). -- ✨ [EIP-6110](https://eips.ethereum.org/EIPS/eip-6110): Add cases for deposit log layout and other topics (ERC-20) transfer ([#1371](https://github.com/ethereum/execution-spec-tests/pull/1371)). -- ✨ [EIP-7251](https://eips.ethereum.org/EIPS/eip-7251): Remove pytest skips for consolidation request cases ([#1449](https://github.com/ethereum/execution-spec-tests/pull/1449)). -- ✨ [EIP-7002](https://eips.ethereum.org/EIPS/eip-7002), [EIP-7251](https://eips.ethereum.org/EIPS/eip-7251): Add cases to verify behavior of contracts missing at fork ([#1394](https://github.com/ethereum/execution-spec-tests/pull/1394)). -- ✨ [EIP-7002](https://eips.ethereum.org/EIPS/eip-7002), [EIP-7251](https://eips.ethereum.org/EIPS/eip-7251): Add cases to verify behavior of system contract errors invalidating a block ([#1394](https://github.com/ethereum/execution-spec-tests/pull/1394)). -- 🔀 Remove [EIP-7698](https://eips.ethereum.org/EIPS/eip-7698): EIP has been removed and the tests related to it have also been removed, while preserving a subset of the tests to verify that functionality is removed in clients ([#1451](https://github.com/ethereum/execution-spec-tests/pull/1451)). - -### 📋 Misc - -- 🐞 Configure `markdownlint` to expect an indent of 4 with unordered lists (otherwise HTML documentation is rendered incorrectly, [#1460](https://github.com/ethereum/execution-spec-tests/pull/1460)). -- 🔀 Update `eels_resolutions.json` to point to temporary commit `bb0eb750d643ced0ebf5dec732cdd23558d0b7f2`, which is based on `forks/prague` branch, commit `d9a7ee24db359aacecd636349b4f3ac95a4a6e71`, with PRs , and merged ([#1394](https://github.com/ethereum/execution-spec-tests/pull/1394)). - -## [v4.2.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v4.2.0) - 2025-04-08 - -**Note**: Although not a breaking change, `consume` users should delete the cache directory (typically located at `~/.cache/ethereum-execution-spec-tests`) used to store downloaded fixture release tarballs. This release adds support for [ethereum/tests](https://github.com/ethereum/tests) and [ethereum/legacytests](https://github.com/ethereum/legacytests) fixture release downloads and the structure of the cache directory has been updated to accommodate this change. - -To try this feature: - -```shell -consume direct --input=https://github.com/ethereum/tests/releases/download/v17.0/fixtures_blockchain_tests.tgz -``` - -To determine the cache directory location, see the `--cache-folder` entry from the command: - -```shell -consume cache --help -``` - -### 💥 Breaking Change - -### 🛠️ Framework - -#### `consume` - -- ✨ Add support for [ethereum/tests](https://github.com/ethereum/tests) and [ethereum/legacytests](https://github.com/ethereum/legacytests) release tarball download via URL to the `--input` flag of `consume` commands ([#1306](https://github.com/ethereum/execution-spec-tests/pull/1306)). -- ✨ Add support for Nethermind's `nethtest` command to `consume direct` ([#1250](https://github.com/ethereum/execution-spec-tests/pull/1250)). -- ✨ Allow filtering of test cases by fork via pytest marks (e.g., `-m "Cancun or Prague"`) ([#1304](https://github.com/ethereum/execution-spec-tests/pull/1304), [#1318](https://github.com/ethereum/execution-spec-tests/pull/1318)). -- ✨ Allow filtering of test cases by fixture format via pytest marks (e.g., `-m blockchain_test`) ([#1314](https://github.com/ethereum/execution-spec-tests/pull/1314)). -- ✨ Add top-level entries `forks` and `fixture_formats` to the index file that list all the forks and fixture formats used in the indexed fixtures ([#1318](https://github.com/ethereum/execution-spec-tests/pull/1318)). -- ✨ Enable logging from `consume` commands ([#1361](https://github.com/ethereum/execution-spec-tests/pull/1361)). -- ✨ Propagate stdout and stderr (including logs) captured during test execution to the Hive test result ([#1361](https://github.com/ethereum/execution-spec-tests/pull/1361)). -- 🐞 Don't parametrize tests for unsupported fixture formats; improve `consume` test collection ([#1315](https://github.com/ethereum/execution-spec-tests/pull/1315)). -- 🐞 Fix the the hive command printed in test reports to reproduce tests in isolation by prefixing the `--sim.limit` flag value with `id:` ([#1333](https://github.com/ethereum/execution-spec-tests/pull/1333)). -- 🐞 Improve index generation of [ethereum/tests](https://github.com/ethereum/tests) fixtures: Allow generation at any directory level and include `generatedTestHash` in the index file for the `fixture_hash` ([#1303](https://github.com/ethereum/execution-spec-tests/pull/1303)). -- 🐞 Fix loading of [ethereum/tests](https://github.com/ethereum/tests) and [ethereum/legacytests](https://github.com/ethereum/legacytests) fixtures for the case of mixed `0x0` and `0x1` transaction types in multi-index (`data`, `gas`, `value`) state test fixtures ([#1330](https://github.com/ethereum/execution-spec-tests/pull/1330)). -- ✨ Add Osaka to the hive ruleset, includes a small ruleset refactor ([#1355](https://github.com/ethereum/execution-spec-tests/pull/1355)). - -#### `fill` - -- 🐞 Fix `--fork/from/until` for transition forks when using `fill` [#1311](https://github.com/ethereum/execution-spec-tests/pull/1311). -- 🐞 Fix the node id for state tests marked by transition forks ([#1313](https://github.com/ethereum/execution-spec-tests/pull/1313)). -- ✨ Add `static_filler` plug-in which allows to fill static YAML and JSON tests (from [ethereum/tests](https://github.com/ethereum/tests)) by adding flag `--fill-static-tests` to `uv run fill` ([#1336](https://github.com/ethereum/execution-spec-tests/pull/1336)). - -#### `execute` - -- 🔀 Test IDs have changed to include the name of the test spec where the test came from (e.g. `state_test`, `blockchain_test`, etc) ([#1367](https://github.com/ethereum/execution-spec-tests/pull/1367)). -- ✨ Markers can now be used to execute only tests from a specific test spec type (e.g. `-m state_test`, `-m blockchain_test`, etc) ([#1367](https://github.com/ethereum/execution-spec-tests/pull/1367)). - -### 📋 Misc - -- 🔀 Bump the version of `execution-specs` used by the framework to the package [`ethereum-execution==1.17.0rc6.dev1`](https://pypi.org/project/ethereum-execution/1.17.0rc6.dev1/); bump the version used for test fixture generation for forks < Prague to current `execution-specs` master, [fa847a0](https://github.com/ethereum/execution-specs/commit/fa847a0e48309debee8edc510ceddb2fd5db2f2e) ([#1310](https://github.com/ethereum/execution-spec-tests/pull/1310)). -- 🐞 Init `TransitionTool` in `GethTransitionTool` ([#1276](https://github.com/ethereum/execution-spec-tests/pull/1276)). -- 🔀 Refactored RLP encoding of test objects to allow automatic generation of tests ([#1359](https://github.com/ethereum/execution-spec-tests/pull/1359)). -- ✨ Document how to manage `execution-spec-tests` package dependencies ([#1388](https://github.com/ethereum/execution-spec-tests/pull/1388)). - -#### Packaging - -- 🐞 Fix `eest make test` when `ethereum-execution-spec-tests` is installed as a package ([#1342](https://github.com/ethereum/execution-spec-tests/pull/1342)). -- 🔀 Pin `setuptools` and `wheel` in `[build-system]`, bump `trie>=3.1` and remove `setuptools` from package dependencies ([#1345](https://github.com/ethereum/execution-spec-tests/pull/1345), [#1351](https://github.com/ethereum/execution-spec-tests/pull/1351)). - -### 🧪 Test Cases - -- ✨ Add additional test coverage for EIP-152 Blake2 precompiles ([#1244](https://github.com/ethereum/execution-spec-tests/pull/1244)). Refactor to add variables for spec constants and common fixture code. ([#1395](https://github.com/ethereum/execution-spec-tests/pull/1395)), ([#1405](https://github.com/ethereum/execution-spec-tests/pull/1405)). -- ✨ Add EIP-7702 incorrect-rlp-encoding tests ([#1347](https://github.com/ethereum/execution-spec-tests/pull/1347)). -- ✨ Add EIP-2935 tests for all call opcodes ([#1379](https://github.com/ethereum/execution-spec-tests/pull/1379)). -- ✨ Add more tests for EIP-7702: max-fee-per-gas verification, delegation-designation as initcode tests ([#1372](https://github.com/ethereum/execution-spec-tests/pull/1372)). -- ✨ Add converted Identity precompile tests ([#1344](https://github.com/ethereum/execution-spec-tests/pull/1344)). - -## [v4.1.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v4.1.0) - 2025-03-11 - -### 💥 Breaking Changes - -The following changes may be potentially breaking (all clients were tested with these changes with the state test format, but not the blockchain test format): - -- 💥 Add a `yParity` field (that duplicates `v`) to transaction authorization tuples in fixture formats to have fields that conform to EIP-7702 spec, resolves [erigontech/erigon#14073](https://github.com/erigontech/erigon/issues/14073) ([#1286](https://github.com/ethereum/execution-spec-tests/pull/1286)). -- 💥 Rename the recently introduced `_info` field `fixture_format` to `fixture-format` for consistency [#1295](https://github.com/ethereum/execution-spec-tests/pull/1295). - -### 🛠️ Framework - -- 🔀 Make `BaseFixture` able to parse any fixture format such as `BlockchainFixture` ([#1210](https://github.com/ethereum/execution-spec-tests/pull/1210)). -- ✨ Blockchain and Blockchain-Engine tests now have a marker to specify that they were generated from a state test, which can be used with `-m blockchain_test_from_state_test` and `-m blockchain_test_engine_from_state_test` respectively ([#1220](https://github.com/ethereum/execution-spec-tests/pull/1220)). -- ✨ Blockchain and Blockchain-Engine tests that were generated from a state test now have `blockchain_test_from_state_test` or `blockchain_test_engine_from_state_test` as part of their test IDs ([#1220](https://github.com/ethereum/execution-spec-tests/pull/1220)). -- 🔀 Refactor `ethereum_test_fixtures` and `ethereum_clis` to create `FixtureConsumer` and `FixtureConsumerTool` classes which abstract away the consumption process used by `consume direct` ([#935](https://github.com/ethereum/execution-spec-tests/pull/935)). -- ✨ Allow `consume direct --collect-only` without specifying a fixture consumer binary on the command-line ([#1237](https://github.com/ethereum/execution-spec-tests/pull/1237)). -- ✨ Allow `fill --collect-only` without the need for existence of the folder `./fixtures`. -- ✨ Report the (resolved) fixture tarball URL and local fixture cache directory when `consume`'s `--input` flag is a release spec or URL ([#1239](https://github.com/ethereum/execution-spec-tests/pull/1239)). -- ✨ EOF Container validation tests (`eof_test`) now generate container deployment state tests, by wrapping the EOF container in an init-container and sending a deploy transaction ([#783](https://github.com/ethereum/execution-spec-tests/pull/783), [#1233](https://github.com/ethereum/execution-spec-tests/pull/1233)). -- ✨ Use regexes for Hive's `--sim.limit` argument and don't use xdist if `--sim.parallelism==1` in the `eest/consume-rlp` and `eest/consume-rlp` simulators ([#1220](https://github.com/ethereum/execution-spec-tests/pull/1220)). -- 🐞 Register generated test markers, e.g., `blockchain_test_from_state_test`, to prevent test session warnings ([#1238](https://github.com/ethereum/execution-spec-tests/pull/1238), [#1245](https://github.com/ethereum/execution-spec-tests/pull/1245)). -- 🐞 Zero-pad `Environment` fields passed to `t8n` tools as required by `evmone-t8n` ([#1268](https://github.com/ethereum/execution-spec-tests/pull/1268)). - -### 📋 Misc - -- ✨ Add a guide to the docs for porting tests from `ethereum/tests` to EEST ([#1165](https://github.com/ethereum/execution-spec-tests/pull/1165)). -- ✨ Improve the `uv run eest make test` interactive CLI to enable creation of new test modules within existing test sub-folders ([#1241](https://github.com/ethereum/execution-spec-tests/pull/1241)). -- ✨ Update `mypy` to latest release `>=1.15.0,<1.16` ([#1209](https://github.com/ethereum/execution-spec-tests/pull/1209)). -- 🐞 Bug fix for filling with EELS for certain Python versions due to an issue with CPython ([#1231](https://github.com/ethereum/execution-spec-tests/pull/1231)). -- 🐞 Fix HTML site deployment due to the site's index file exceeding Github's max file size limit ([#1292](https://github.com/ethereum/execution-spec-tests/pull/1292)). -- ✨ Update the build fixtures workflow to use multiple self-hosted runners, remove `pectra-devnet-6` feature build ([#1296](https://github.com/ethereum/execution-spec-tests/pull/1296)). - -### 🧪 Test Cases - -- ✨ Add gas cost of delegation access in CALL opcode ([#1208](https://github.com/ethereum/execution-spec-tests/pull/1208)). -- ✨ Add EIP-7698 failed nonce and short data tests ([#1211](https://github.com/ethereum/execution-spec-tests/pull/1211)). -- ✨ Add EIP-2537 additional pairing precompile tests cases, and then update all BLS12 test vectors ([#1275](https://github.com/ethereum/execution-spec-tests/pull/1275), [#1289](https://github.com/ethereum/execution-spec-tests/pull/1289)). -- ✨ Add EIP-7685 and EIP-7002 test cases for additional request type combinations and modified withdrawal contract that allows more withdrawals ([#1340](https://github.com/ethereum/execution-spec-tests/pull/1340)). -- ✨ Add test cases for EIP-152 Blake2 and Identity precompiles ([#1244](https://github.com/ethereum/execution-spec-tests/pull/1244)). - -## [v4.0.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v4.0.0) - 2025-02-14 - 💕 - -### 📁 Fixture Releases - -- 🔀 Initially we moved old fork configured tests within stable and develop fixture releases to a separate legacy release ([#788](https://github.com/ethereum/execution-spec-tests/pull/788)). -- 🔀 This was later reverted after some client teams preferred to keep them in all in the same releases ([#1053](https://github.com/ethereum/execution-spec-tests/pull/1053)). - -### 💥 Breaking Change - -- ✨ Use uv for package management replacing pip ([#777](https://github.com/ethereum/execution-spec-tests/pull/777)). -- ✨ Ruff now replaces Flake8, Isort and Black resulting in significant changes to the entire code base including its usage ([#922](https://github.com/ethereum/execution-spec-tests/pull/922)). -- 🔀 Fill test fixtures using EELS by default. EEST now uses the [`ethereum-specs-evm-resolver`](https://github.com/petertdavies/ethereum-spec-evm-resolver) with the EELS daemon ([#792](https://github.com/ethereum/execution-spec-tests/pull/792)). -- 🔀 The EOF fixture format contained in `eof_tests` may now contain multiple exceptions in the `"exception"` field in the form of a pipe (`|`) separated string ([#759](https://github.com/ethereum/execution-spec-tests/pull/759)). -- 🔀 `state_test`, `blockchain_test` and `blockchain_test_engine` fixtures now contain a `config` field, which contains an object that contains a `blobSchedule` field. On the `blockchain_test` and `blockchain_test_engine` fixtures, the object also contains a duplicate of the `network` root field. The root's `network` field will be eventually deprecated ([#1040](https://github.com/ethereum/execution-spec-tests/pull/1040)). -- 🔀 `latest-stable-release` and `latest-develop-release` keywords for the `--input` flag in consume commands have been replaced with `stable@latest` and `develop@latest` respectively ([#1044](https://github.com/ethereum/execution-spec-tests/pull/1044)). - -### 🛠️ Framework - -- ✨ Execute command added to run existing tests in live networks ([#](https://github.com/ethereum/execution-spec-tests/pull/1157)). -- 🐞 Fixed consume hive commands from spawning different hive test suites during the same test execution when using xdist ([#712](https://github.com/ethereum/execution-spec-tests/pull/712)). -- ✨ `consume hive` command is now available to run all types of hive tests ([#712](https://github.com/ethereum/execution-spec-tests/pull/712)). -- ✨ Generated fixtures now contain the test index `index.json` by default ([#716](https://github.com/ethereum/execution-spec-tests/pull/716)). -- ✨ A metadata folder `.meta/` now stores all fixture metadata files by default ([#721](https://github.com/ethereum/execution-spec-tests/pull/721)). -- 🐞 Fixed `fill` command index generation issue due to concurrency ([#725](https://github.com/ethereum/execution-spec-tests/pull/725)). -- ✨ Added `with_all_evm_code_types`, `with_all_call_opcodes` and `with_all_create_opcodes` markers, which allow automatic parametrization of tests to EOF ([#610](https://github.com/ethereum/execution-spec-tests/pull/610), [#739](https://github.com/ethereum/execution-spec-tests/pull/739)). -- ✨ Added `with_all_system_contracts` marker, which helps parametrize tests with all contracts that affect the chain on a system level ([#739](https://github.com/ethereum/execution-spec-tests/pull/739)). -- ✨ Code generators `Conditional` and `Switch` now support EOF by adding parameter `evm_code_type` ([#610](https://github.com/ethereum/execution-spec-tests/pull/610)). -- ✨ `fill` command now supports parameter `--evm-code-type` that can be (currently) set to `legacy` or `eof_v1` to force all test smart contracts to deployed in normal or in EOF containers ([#610](https://github.com/ethereum/execution-spec-tests/pull/610)). -- 🐞 Fixed fixture index generation on EOF tests ([#728](https://github.com/ethereum/execution-spec-tests/pull/728)). -- 🐞 Fixes consume genesis mismatch exception for hive based simulators ([#734](https://github.com/ethereum/execution-spec-tests/pull/734)). -- ✨ Adds reproducible consume commands to hiveview ([#717](https://github.com/ethereum/execution-spec-tests/pull/717)). -- 💥 Added multiple exceptions to the EOF fixture format ([#759](https://github.com/ethereum/execution-spec-tests/pull/759)). -- ✨ Added optional parameter to all `with_all_*` markers to specify a lambda function that filters the parametrized values ([#739](https://github.com/ethereum/execution-spec-tests/pull/739)). -- ✨ Added [`extend_with_defaults` utility function](https://eest.ethereum.org/main/writing_tests/writing_a_new_test/#ethereum_test_tools.utility.pytest.extend_with_defaults), which helps extend test case parameter sets with default values. `@pytest.mark.parametrize` ([#739](https://github.com/ethereum/execution-spec-tests/pull/739)). -- ✨ Added `Container.Init` to `ethereum_test_types.EOF.V1` package, which allows generation of an EOF init container more easily ([#739](https://github.com/ethereum/execution-spec-tests/pull/739)). -- ✨ Introduce method valid_opcodes() to the fork class ([#748](https://github.com/ethereum/execution-spec-tests/pull/748)). -- 🐞 Fixed `consume` exit code return values, ensuring that pytest's return value is correctly propagated to the shell. This allows the shell to accurately reflect the test results (e.g., failures) based on the pytest exit code ([#765](https://github.com/ethereum/execution-spec-tests/pull/765)). -- ✨ Added a new flag `--solc-version` to the `fill` command, which allows the user to specify the version of the Solidity compiler to use when compiling Yul source code; this version will now be automatically downloaded by `fill` via [`solc-select`](https://github.com/crytic/solc-select) ([#772](https://github.com/ethereum/execution-spec-tests/pull/772)). -- 🐞 Fix usage of multiple `@pytest.mark.with_all*` markers which shared parameters, such as `with_all_call_opcodes` and `with_all_create_opcodes` which shared `evm_code_type`, and now only parametrize compatible values ([#762](https://github.com/ethereum/execution-spec-tests/pull/762)). -- ✨ Added `selector` and `marks` fields to all `@pytest.mark.with_all*` markers, which allows passing lambda functions to select or mark specific parametrized values (see [documentation](https://eest.ethereum.org/main/writing_tests/test_markers/#covariant-marker-keyword-arguments) for more information) ([#762](https://github.com/ethereum/execution-spec-tests/pull/762)). -- ✨ Improves consume input flags for develop and stable fixture releases, fixes `--help` flag for consume ([#745](https://github.com/ethereum/execution-spec-tests/pull/745)). -- 🔀 Return exit-code from `consume` commands ([#766](https://github.com/ethereum/execution-spec-tests/pull/766)). -- 🔀 Remove duplicate EOF container tests, automatically check for duplicates ([#800](https://github.com/ethereum/execution-spec-tests/pull/800)). -- 🔀 Fix DATALOAD `pushed_stack_items` calculation ([#784](https://github.com/ethereum/execution-spec-tests/pull/784)). -- ✨ Add `evm_bytes` rename and print asm ([#844](https://github.com/ethereum/execution-spec-tests/pull/844)). -- 🔀 Use the `session_temp_folder` introduced in #824 ([#845](https://github.com/ethereum/execution-spec-tests/pull/845)). -- 🐞 Don't treat eels resolutions as a fixture ([#878](https://github.com/ethereum/execution-spec-tests/pull/878)). -- ✨ Emphasize that no tests were executed during a fill pytest session ([#887](https://github.com/ethereum/execution-spec-tests/pull/887)). -- 🔀 Move generic code from TransitionTool to a new generic base class EthereumCLI ([#894](https://github.com/ethereum/execution-spec-tests/pull/894)). -- 🐞 Fix erroneous fork mes- 🔀 Fix max stack height calculation ([#810](https://github.com/ethereum/execution-spec-tests/pull/810)). -- ✨ Add storage key hint in Storage class ([#917](https://github.com/ethereum/execution-spec-tests/pull/917)). -- ✨ Add system to verify exception strings ([#795](https://github.com/ethereum/execution-spec-tests/pull/795)). -- 🔀 Fix `FixedBytes` assignment rules ([#1010](https://github.com/ethereum/execution-spec-tests/pull/1010)). -- 🔀 Fix `Address` padding options ([#1113](https://github.com/ethereum/execution-spec-tests/pull/1113)). -- 🔀 Add `Container.expected_bytecode` optional parameter ([#737](https://github.com/ethereum/execution-spec-tests/pull/737)). -- ✨ Add support for `initcode_prefix` on EOF `Container.Init` ([#819](https://github.com/ethereum/execution-spec-tests/pull/819)).sage in pytest session header with development forks ([#806](https://github.com/ethereum/execution-spec-tests/pull/806)). -- 🐞 Fix `Conditional` code generator in EOF mode ([#821](https://github.com/ethereum/execution-spec-tests/pull/821)). -- 🔀 Ensure that `Block` objects keep track of their `fork`, for example, when the block's `rlp_modifier` is not `None` ([#854](https://github.com/ethereum/execution-spec-tests/pull/854)). -- 🔀 `ethereum_test_rpc` library has been created with what was previously `ethereum_test_tools.rpc` ([#822](https://github.com/ethereum/execution-spec-tests/pull/822)). -- ✨ Add `Wei` type to `ethereum_test_base_types` which allows parsing wei amounts from strings like "1 ether", "1000 wei", "10**2 gwei", etc ([#825](https://github.com/ethereum/execution-spec-tests/pull/825)). -- ✨ Pin EELS versions in `eels_resolutions.json` and include this file in fixture releases ([#872](https://github.com/ethereum/execution-spec-tests/pull/872)). -- 🔀 Replace `ethereum.base_types` with `ethereum-types` ([#850](https://github.com/ethereum/execution-spec-tests/pull/850)). -- 💥 Rename the `PragueEIP7692` fork to `Osaka` ([#869](https://github.com/ethereum/execution-spec-tests/pull/869)). -- ✨ Improve `fill` terminal output to emphasize that filling tests is not actually testing a client ([#807](https://github.com/ethereum/execution-spec-tests/pull/887)). -- ✨ Add the `BlockchainTestEngine` test spec type that only generates a fixture in the `EngineFixture` (`blockchain_test_engine`) format ([#888](https://github.com/ethereum/execution-spec-tests/pull/888)). -- 🔀 Move the `evm_transition_tool` package to `ethereum_clis` and derive the transition tool CL interfaces from a shared `EthereumCLI` class that can be reused for other sub-commands ([#894](https://github.com/ethereum/execution-spec-tests/pull/894)). -- ✨ Pass `state_test` property to T8N tools that support it (Only EELS at the time of merge) ([#943](https://github.com/ethereum/execution-spec-tests/pull/943)). -- ✨ Add the `eofwrap` cli used to wrap tests from `ethereum/tests` in an EOF container ([#896](https://github.com/ethereum/execution-spec-tests/pull/896)). -- 🔀 Improve gentest architecture with `EthereumTestBaseModel` and `EthereumTestRootModel` ([#901](https://github.com/ethereum/execution-spec-tests/pull/901)). -- 🔀 Migrate transaction test to `state_test` for `gentest` ([#903](https://github.com/ethereum/execution-spec-tests/pull/903)). -- 🔀 `ethereum_test_forks` forks now contain gas-calculating functions, which return the appropriate function to calculate the gas used by a transaction or memory function for the given fork ([#779](https://github.com/ethereum/execution-spec-tests/pull/779)). -- 🐞 Fix `Bytecode` class `__eq__` method ([#939](https://github.com/ethereum/execution-spec-tests/pull/939)). -- 🔀 Update `pydantic` from 2.8.2 to 2.9.2 ([#960](https://github.com/ethereum/execution-spec-tests/pull/960)). -- ✨ Add the `eest make test` command, an interactive CLI that helps users create a new test module and function ([#950](https://github.com/ethereum/execution-spec-tests/pull/950)). -- ✨ Add the `eest clean` command that helps delete generated files and directories from the repository ([#980](https://github.com/ethereum/execution-spec-tests/pull/980)). -- ✨ Add framework changes for EIP-7742, required for Prague devnet-5 ([#931](https://github.com/ethereum/execution-spec-tests/pull/931)). -- ✨ Add the `eest make env` command that generates a default env file (`env.yaml`)([#996](https://github.com/ethereum/execution-spec-tests/pull/996)). -- ✨ Generate Transaction Test type ([#933](https://github.com/ethereum/execution-spec-tests/pull/933)). -- ✨ Add a default location for evm logs (`--evm-dump-dir`) when filling tests ([#999](https://github.com/ethereum/execution-spec-tests/pull/999)). -- ✨ Slow tests now have greater timeout when making a request to the T8N server ([#1037](https://github.com/ethereum/execution-spec-tests/pull/1037)). -- ✨ Introduce [`pytest.mark.parametrize_by_fork`](https://eest.ethereum.org/main/writing_tests/test_markers/#pytestmarkfork_parametrize) helper marker ([#1019](https://github.com/ethereum/execution-spec-tests/pull/1019), [#1057](https://github.com/ethereum/execution-spec-tests/pull/1057)). -- 🐞 fix(consume): allow absolute paths with `--evm-bin` ([#1052](https://github.com/ethereum/execution-spec-tests/pull/1052)). -- ✨ Disable EIP-7742 framework changes for Prague ([#1023](https://github.com/ethereum/execution-spec-tests/pull/1023)). -- ✨ Allow verification of the transaction receipt on executed test transactions ([#1068](https://github.com/ethereum/execution-spec-tests/pull/1068)). -- ✨ Modify `valid_at_transition_to` marker to add keyword arguments `subsequent_transitions` and `until` to fill a test using multiple transition forks ([#1081](https://github.com/ethereum/execution-spec-tests/pull/1081)). -- 🐞 fix(consume): use `"HIVE_CHECK_LIVE_PORT"` to signal hive to wait for port 8551 (Engine API port) instead of the 8545 port when running `consume engine` ([#1095](https://github.com/ethereum/execution-spec-tests/pull/1095)). -- ✨ `state_test`, `blockchain_test` and `blockchain_test_engine` fixtures now contain the `blobSchedule` from [EIP-7840](https://github.com/ethereum/EIPs/blob/master/EIPS/eip-7840.md), only for tests filled for Cancun and Prague forks ([#1040](https://github.com/ethereum/execution-spec-tests/pull/1040)). -- 🔀 Change `--dist` flag to the default value, `load`, for better parallelism handling during test filling ([#1118](https://github.com/ethereum/execution-spec-tests/pull/1118)). -- 🔀 Refactor framework code to use the [`ethereum-rlp`](https://pypi.org/project/ethereum-rlp/) package instead of `ethereum.rlp`, previously available in ethereum/execution-specs ([#1180](https://github.com/ethereum/execution-spec-tests/pull/1180)). -- 🔀 Update EELS / execution-specs EEST dependency to [99238233](https://github.com/ethereum/execution-specs/commit/9923823367b5586228e590537d47aa9cc4c6a206) for EEST framework libraries and test case generation ([#1181](https://github.com/ethereum/execution-spec-tests/pull/1181)). -- ✨ Add the `consume cache` command to cache fixtures before running consume commands ([#1044](https://github.com/ethereum/execution-spec-tests/pull/1044)). -- ✨ The `--input` flag of the consume commands now supports parsing of tagged release names in the format `@` ([#1044](https://github.com/ethereum/execution-spec-tests/pull/1044)). -- 🐞 Fix stdout output when using the `fill` command ([#1188](https://github.com/ethereum/execution-spec-tests/pull/1188)). -- ✨ Add tests for blockchain intermediate state verification ([#1075](https://github.com/ethereum/execution-spec-tests/pull/1075)). -- ✨ Add Interactive CLI input functionality ([#947](https://github.com/ethereum/execution-spec-tests/pull/947)). -- 🔀 Rename `EOFTest.data` to `EOFTest.container` with rebase of `EOFStateTest` ([#1145](https://github.com/ethereum/execution-spec-tests/pull/1145)). -- ✨ Turn on `--traces` for EELS + `ethereum-specs-evm-resolver` ([#1174](https://github.com/ethereum/execution-spec-tests/pull/1174)). - -### 📋 Misc - -- ✨ Feature releases can now include multiple types of fixture tarball files from different releases that start with the same prefix ([#736](https://github.com/ethereum/execution-spec-tests/pull/736)). -- ✨ Releases for feature eip7692 now include both Cancun and Prague based tests in the same release, in files `fixtures_eip7692.tar.gz` and `fixtures_eip7692-prague.tar.gz` respectively ([#743](https://github.com/ethereum/execution-spec-tests/pull/743)). -✨ Re-write the test case reference doc flow as a pytest plugin and add pages for test functions with a table providing an overview of their parametrized test cases ([#801](https://github.com/ethereum/execution-spec-tests/pull/801), [#842](https://github.com/ethereum/execution-spec-tests/pull/842)). -- 🔀 Simplify Python project configuration and consolidate it into `pyproject.toml` ([#764](https://github.com/ethereum/execution-spec-tests/pull/764)). -- ✨ Add dev docs to help using nectos/act ([#776](https://github.com/ethereum/execution-spec-tests/pull/776)). -- 🔀 Update `uv.lock` for updated solc deps ([#782](https://github.com/ethereum/execution-spec-tests/pull/782)). -- ✨ Enable coverage on any test change ([#790](https://github.com/ethereum/execution-spec-tests/pull/790)). -- 🔀 Created `pytest_plugins.concurrency` plugin to sync multiple `xdist` processes without using a command flag to specify the temporary working folder ([#824](https://github.com/ethereum/execution-spec-tests/pull/824)). -- 🔀 Move pytest plugin `pytest_plugins.filler.solc` to `pytest_plugins.solc.solc` ([#823](https://github.com/ethereum/execution-spec-tests/pull/823)). -- 🔀 Remove `formats.py`, embed properties as class vars ([#826](https://github.com/ethereum/execution-spec-tests/pull/826)). -- ✨ Add `build-evm-base` to docs deploy workflows ([#829](https://github.com/ethereum/execution-spec-tests/pull/829)). -- 🔀 Add links to the online test case docs in the EOF tracker ([#838](https://github.com/ethereum/execution-spec-tests/pull/838)). -- 🔀 Fix miscellaneous improvements to troubleshooting, navigation, styling ([#840](https://github.com/ethereum/execution-spec-tests/pull/840)). -- ✨ Include all parameters in test parameter datatables ([#842](https://github.com/ethereum/execution-spec-tests/pull/842)). -- ✨ Add info about ripemd160 & update running actions locally ([#847](https://github.com/ethereum/execution-spec-tests/pull/847)). -- ✨ Add `SECURITY.md` describing how to report vulnerabilities ([#848](https://github.com/ethereum/execution-spec-tests/pull/848)). -- 🔀 Change image from ubuntu-24.04 to ubuntu-latest in CI ([#855](https://github.com/ethereum/execution-spec-tests/pull/855)). -- 🐞 Asserts that the deploy docs tags workflow is only triggered for full releases ([#857](https://github.com/ethereum/execution-spec-tests/pull/857)). -- 🐞 Fix deploy docs tags workflow trigger ([#858](https://github.com/ethereum/execution-spec-tests/pull/858)). -- ✨ A new application-wide configuration manager provides access to environment and application configurations. ([#892](https://github.com/ethereum/execution-spec-tests/pull/892)). -- 🔀 Update the developer docs navigation ([#898](https://github.com/ethereum/execution-spec-tests/pull/898)). -- 🔀 Use jinja2 templating in `gentest` ([#900](https://github.com/ethereum/execution-spec-tests/pull/900)). -- ✨ Fix/add test github actions locally page ([#909](https://github.com/ethereum/execution-spec-tests/pull/909)). -- 🐞 Fix print fill output in coverage workflow on errors ([#919](https://github.com/ethereum/execution-spec-tests/pull/919)). -- 🐞 Use a local version of ethereum/execution-specs (EELS) when running the framework tests in CI ([#997](https://github.com/ethereum/execution-spec-tests/pull/997)). -- ✨ Use self-hosted runners for fixture building in CI ([#1051](https://github.com/ethereum/execution-spec-tests/pull/1051)). -- ✨ Release tarballs now contain fixtures filled for all forks, not only the fork under active development and the fork currently deployed on mainnet ([#1053](https://github.com/ethereum/execution-spec-tests/pull/1053)). -- ✨ `StateTest` fixture format now contains `state` field in each network post result, containing the decoded post allocation that results from the transaction execution ([#1064](https://github.com/ethereum/execution-spec-tests/pull/1064)). -- ✨ Include EELS fork resolution information in filled json test fixtures ([#1123](https://github.com/ethereum/execution-spec-tests/pull/1123)). -- 🔀 Updates ruff from version 0.8.2 to 0.9.4 ([#1168](https://github.com/ethereum/execution-spec-tests/pull/1168)). -- 🔀 Update `uv.lock` to be compatible with `uv>=0.5.22` ([#1178](https://github.com/ethereum/execution-spec-tests/pull/1178)). -- 🔀 Update `mypy` from version `0.991` to `1.15` ([#1209](https://github.com/ethereum/execution-spec-tests/pull/1209)). - -### 🧪 Test Cases - -- ✨ Migrate validation tests `EIP3540/validInvalidFiller.yml` ([#598](https://github.com/ethereum/execution-spec-tests/pull/598)). -- ✨ EIP-4844 test `tests/cancun/eip4844_blobs/test_point_evaluation_precompile.py` includes an EOF test case ([#610](https://github.com/ethereum/execution-spec-tests/pull/610)). -- ✨ Example test `tests/frontier/opcodes/test_dup.py` now includes EOF parametrization ([#610](https://github.com/ethereum/execution-spec-tests/pull/610)). -- ✨ Add EOFv1 function test - Call simple contract test ([#695](https://github.com/ethereum/execution-spec-tests/pull/695)). -- ✨ Add section size validation tests ([#705](https://github.com/ethereum/execution-spec-tests/pull/705)). -- ✨ Add deep and wide EOF subcontainers tests ([#718](https://github.com/ethereum/execution-spec-tests/pull/718)). -- ✨ Update [EIP-7702](https://eips.ethereum.org/EIPS/eip-7702) tests for Devnet-3 ([#733](https://github.com/ethereum/execution-spec-tests/pull/733)). -- ✨ Migrate "valid" EOFCREATE validation ([#738](https://github.com/ethereum/execution-spec-tests/pull/738)). -- ✨ Migrate tests for truncated sections ([#740](https://github.com/ethereum/execution-spec-tests/pull/740)). -- ✨ Add out of order container section test ([#741](https://github.com/ethereum/execution-spec-tests/pull/741)). -- ✨ Convert all opcodes validation test `tests/frontier/opcodes/test_all_opcodes.py` ([#748](https://github.com/ethereum/execution-spec-tests/pull/748)). -- 🔀 Add Test types with 128 inputs ([#749](https://github.com/ethereum/execution-spec-tests/pull/749)). -- 🔀 Use new marker to EOF-ize MCOPY test ([#754](https://github.com/ethereum/execution-spec-tests/pull/754)). -- ✨ Add EOF Tests from Fuzzing ([#756](https://github.com/ethereum/execution-spec-tests/pull/756)). -- ✨ Add embedded container tests ([#763](https://github.com/ethereum/execution-spec-tests/pull/763)). -- ✨ Validate EOF only opcodes are invalid in legacy ([#768](https://github.com/ethereum/execution-spec-tests/pull/768)). -- ✨ Update EOF tracker, add unimplemented tests ([#773](https://github.com/ethereum/execution-spec-tests/pull/773)). -- ✨ Add EIP-7620 EOFCREATE gas tests ([#785](https://github.com/ethereum/execution-spec-tests/pull/785)). -- ✨ Add more fuzzing discovered EOF tests ([#789](https://github.com/ethereum/execution-spec-tests/pull/789)). -- ✨ Add EOF tests for invalid non-returning sections ([#794](https://github.com/ethereum/execution-spec-tests/pull/794)). -- ✨ Test to ensure transient storage is cleared after transactions ([#798](https://github.com/ethereum/execution-spec-tests/pull/798)). -- ✨ Test that transient storage stays at correct address ([#799](https://github.com/ethereum/execution-spec-tests/pull/799)). -- ✨ Add EOFCREATE referencing the same subcontainer twice test ([#809](https://github.com/ethereum/execution-spec-tests/pull/809)). -- ✨ Add dangling data in subcontainer test ([#812](https://github.com/ethereum/execution-spec-tests/pull/812)). -- 🐞 Fix [EIP-7702](https://eips.ethereum.org/EIPS/eip-7702)+EOF tests due to incorrect test expectations and faulty `Conditional` test generator in EOF mode ([#821](https://github.com/ethereum/execution-spec-tests/pull/821)). -- 🐞 Fix TSTORE EOF variant test ([#831](https://github.com/ethereum/execution-spec-tests/pull/831)). -- 🔀 Unify EOF return code constants ([#834](https://github.com/ethereum/execution-spec-tests/pull/834)). -- ✨ Add RJUMP* vs CALLF tests ([#833](https://github.com/ethereum/execution-spec-tests/pull/833)). -- ✨ Add tests to clarify "non-returning instruction" ([#837](https://github.com/ethereum/execution-spec-tests/pull/837)). -- ✨ Add double RJUMPI stack validation tests ([#851](https://github.com/ethereum/execution-spec-tests/pull/851)). -- ✨ Add unreachable code sections tests ([#856](https://github.com/ethereum/execution-spec-tests/pull/856)). -- 💥 `PragueEIP7692` fork in tests has been updated to `Osaka` ([#869](https://github.com/ethereum/execution-spec-tests/pull/869)). -- ✨ Update [EIP-6110](https://eips.ethereum.org/EIPS/eip-6110), [EIP-7002](https://eips.ethereum.org/EIPS/eip-7002), [EIP-7251](https://eips.ethereum.org/EIPS/eip-7251), [EIP-7685](https://eips.ethereum.org/EIPS/eip-7685), and [EIP-7702](https://eips.ethereum.org/EIPS/eip-7702) tests for Devnet-4 ([#832](https://github.com/ethereum/execution-spec-tests/pull/832)). -- ✨ Add EIP-7069 and EIP-7620 failures and context vars tests ([#836](https://github.com/ethereum/execution-spec-tests/pull/836)). -- ✨ Add EOF EIP-4750 Stack validation in CALLF test ([#889](https://github.com/ethereum/execution-spec-tests/pull/889)). -- ✨ Add stack overflow by rule check to JUMPF tests ([#902](https://github.com/ethereum/execution-spec-tests/pull/902)). -- 🐞 Fix erroneous test with`CALLF` rule bug ([#907](https://github.com/ethereum/execution-spec-tests/pull/907)). -- ✨ Add test for EXTDELEGATECALL value cost ([#911](https://github.com/ethereum/execution-spec-tests/pull/911)). -- ✨ Add basic EOF execution tests ([#912](https://github.com/ethereum/execution-spec-tests/pull/912)). -- ✨ Add parametrized CALLF execution tests ([#913](https://github.com/ethereum/execution-spec-tests/pull/913)). -- ✨ Add CALLF execution tests ([#914](https://github.com/ethereum/execution-spec-tests/pull/914)). -- ✨ Add fibonacci and factorial CALLF tests ([#915](https://github.com/ethereum/execution-spec-tests/pull/915)). -- ✨ Add RJUMP* execution tests ([#916](https://github.com/ethereum/execution-spec-tests/pull/916)). -- ✨ [EIP-7702](https://eips.ethereum.org/EIPS/eip-7702) many delegations test ([#923](https://github.com/ethereum/execution-spec-tests/pull/923)). -- ✨ Add opcode validation tests ([#932](https://github.com/ethereum/execution-spec-tests/pull/932)). -- ✨ Add RJUMPI with JUMPF tests ([#928](https://github.com/ethereum/execution-spec-tests/pull/928)). -- ✨ [EIP-7702](https://eips.ethereum.org/EIPS/eip-7702) set code of non-empty-storage account test ([#948](https://github.com/ethereum/execution-spec-tests/pull/948)). -- ✨ Add PUSH* opcode tests ([#975](https://github.com/ethereum/execution-spec-tests/pull/975)). -- ✨ [EIP-7702](https://eips.ethereum.org/EIPS/eip-7702) implement 7702 test ideas ([#981](https://github.com/ethereum/execution-spec-tests/pull/981)). -- ✨ [EIP-7702](https://eips.ethereum.org/EIPS/eip-7702) Remove delegation behavior of EXTCODE* ([#984](https://github.com/ethereum/execution-spec-tests/pull/984)). -- ✨ Add EIP-7620 RETURNCONTRACT behavior verification test ([#1109](https://github.com/ethereum/execution-spec-tests/pull/1109)). -- ✨ Add EIP-7069 p256verify EOF calls tests ([#1021](https://github.com/ethereum/execution-spec-tests/pull/1021)). -- ✨ Add EIP-7480 DATACOPY edge cases tests ([#1020](https://github.com/ethereum/execution-spec-tests/pull/1020)). -- ✨ Add EIP-7069 EXTCALL creation gas charge tests ([#1025](https://github.com/ethereum/execution-spec-tests/pull/1025)). -- ✨ Add generic precompile-absence test ([#1036](https://github.com/ethereum/execution-spec-tests/pull/1036)). -- ✨ Add test for [EIP-2537](https://eips.ethereum.org/EIPS/eip-2537) which uses the full discount table of G2 MSM ([#1038](https://github.com/ethereum/execution-spec-tests/pull/1038)). -- ✨ [EIP-7691](https://eips.ethereum.org/EIPS/eip-7691) Blob throughput increase tests by parametrization of existing EIP-4844 tests ([#1023](https://github.com/ethereum/execution-spec-tests/pull/1023), [#1082](https://github.com/ethereum/execution-spec-tests/pull/1082)). -- ✨ Port [calldatacopy test](https://github.com/ethereum/tests/blob/ae4791077e8fcf716136e70fe8392f1a1f1495fb/src/GeneralStateTestsFiller/VMTests/vmTests/calldatacopyFiller.yml) ([#1056](https://github.com/ethereum/execution-spec-tests/pull/1056)). -- ✨ [EIP-7623](https://eips.ethereum.org/EIPS/eip-7623) Increase calldata cost ([#1004](https://github.com/ethereum/execution-spec-tests/pull/1004), [#1071](https://github.com/ethereum/execution-spec-tests/pull/1071)). -- ✨ Add CALLF invalid section index tests ([#1111](https://github.com/ethereum/execution-spec-tests/pull/1111)). -- ✨ Add JUMPF invalid section index tests ([#1112](https://github.com/ethereum/execution-spec-tests/pull/1112)). -- ✨ Add CALLF truncated immediate bytes tests ([#1114](https://github.com/ethereum/execution-spec-tests/pull/1114)). -- ✨ [EIP-152](https://eips.ethereum.org/EIPS/eip-152) Add tests for Blake2 compression function `F` precompile ([#1067](https://github.com/ethereum/execution-spec-tests/pull/1067)). -- ✨ Add CALLF non-returning section tests ([#1126](https://github.com/ethereum/execution-spec-tests/pull/1126)). -- ✨ Add DATALOADN truncated immediate bytes tests ([#1127](https://github.com/ethereum/execution-spec-tests/pull/1127)). -- 🔀 Update EIP-7702 test expectations according to [spec updates](https://github.com/ethereum/EIPs/pull/9248) ([#1129](https://github.com/ethereum/execution-spec-tests/pull/1129)). -- ✨ Add tests for CALLF and non-returning ([#1140](https://github.com/ethereum/execution-spec-tests/pull/1140)). -- 🔀 Update EIP-7251 according to spec updates [#9127](https://github.com/ethereum/EIPs/pull/9127), [#9289](https://github.com/ethereum/EIPs/pull/9289) ([#1024](https://github.com/ethereum/execution-spec-tests/pull/1024), [#1155](https://github.com/ethereum/execution-spec-tests/pull/1155)). -- 🔀 Update EIP-7002 according to spec updates [#9119](https://github.com/ethereum/EIPs/pull/9119), [#9288](https://github.com/ethereum/EIPs/pull/9288) ([#1024](https://github.com/ethereum/execution-spec-tests/pull/1024), [#1155](https://github.com/ethereum/execution-spec-tests/pull/1155)). -- 🔀 Update EIP-2935 according to spec updates [#9144](https://github.com/ethereum/EIPs/pull/9144), [#9287](https://github.com/ethereum/EIPs/pull/9287) ([#1046](https://github.com/ethereum/execution-spec-tests/pull/1046), [#1155](https://github.com/ethereum/execution-spec-tests/pull/1155)). -- ✨ Add DATALOADN validation and execution tests ([#1162](https://github.com/ethereum/execution-spec-tests/pull/1162)). -- ✨ Add EOF prefix tests ([#1187](https://github.com/ethereum/execution-spec-tests/pull/1187)). -- ✨ Add tests for EOF code header missing ([#1193](https://github.com/ethereum/execution-spec-tests/pull/1193)). -- ✨ Add tests for empty EOF type section ([#1194](https://github.com/ethereum/execution-spec-tests/pull/1194)). -- ✨ Add tests for multiple EOF type sections ([#1195](https://github.com/ethereum/execution-spec-tests/pull/1195)). -- ✨ Add EIP-7698 legacy EOF creation prevention tests ([#1206](https://github.com/ethereum/execution-spec-tests/pull/1206)). - -## [v3.0.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v3.0.0) - 2024-07-22 - -### 🧪 Test Cases - -- ✨ Port create2 return data test ([#497](https://github.com/ethereum/execution-spec-tests/pull/497)). -- ✨ Add tests for eof container's section bytes position smart fuzzing ([#592](https://github.com/ethereum/execution-spec-tests/pull/592)). -- ✨ Add `test_create_selfdestruct_same_tx_increased_nonce` which tests self-destructing a contract with a nonce > 1 ([#478](https://github.com/ethereum/execution-spec-tests/pull/478)). -- ✨ Add `test_double_kill` and `test_recreate` which test resurrection of accounts killed with `SELFDESTRUCT` ([#488](https://github.com/ethereum/execution-spec-tests/pull/488)). -- ✨ Add eof example valid invalid tests from ori, fetch EOF Container implementation ([#535](https://github.com/ethereum/execution-spec-tests/pull/535)). -- ✨ Add tests for [EIP-2537: Precompile for BLS12-381 curve operations](https://eips.ethereum.org/EIPS/eip-2537) ([#499](https://github.com/ethereum/execution-spec-tests/pull/499)). -- ✨ [EIP-663](https://eips.ethereum.org/EIPS/eip-663): Add `test_dupn.py` and `test_swapn.py` ([#502](https://github.com/ethereum/execution-spec-tests/pull/502)). -- ✨ Add tests for [EIP-6110: Supply validator deposits on chain](https://eips.ethereum.org/EIPS/eip-6110) ([#530](https://github.com/ethereum/execution-spec-tests/pull/530)). -- ✨ Add tests for [EIP-7002: Execution layer triggerable withdrawals](https://eips.ethereum.org/EIPS/eip-7002) ([#530](https://github.com/ethereum/execution-spec-tests/pull/530)). -- ✨ Add tests for [EIP-7685: General purpose execution layer requests](https://eips.ethereum.org/EIPS/eip-7685) ([#530](https://github.com/ethereum/execution-spec-tests/pull/530)). -- ✨ Add tests for [EIP-2935: Serve historical block hashes from state](https://eips.ethereum.org/EIPS/eip-2935) ([#564](https://github.com/ethereum/execution-spec-tests/pull/564), [#585](https://github.com/ethereum/execution-spec-tests/pull/585)). -- ✨ Add tests for [EIP-4200: EOF - Static relative jumps](https://eips.ethereum.org/EIPS/eip-4200) ([#581](https://github.com/ethereum/execution-spec-tests/pull/581), [#666](https://github.com/ethereum/execution-spec-tests/pull/666)). -- ✨ Add tests for [EIP-7069: EOF - Revamped CALL instructions](https://eips.ethereum.org/EIPS/eip-7069) ([#595](https://github.com/ethereum/execution-spec-tests/pull/595)). -- 🐞 Fix typos in self-destruct collision test from erroneous pytest parametrization ([#608](https://github.com/ethereum/execution-spec-tests/pull/608)). -- ✨ Add tests for [EIP-3540: EOF - EVM Object Format v1](https://eips.ethereum.org/EIPS/eip-3540) ([#634](https://github.com/ethereum/execution-spec-tests/pull/634), [#668](https://github.com/ethereum/execution-spec-tests/pull/668)). -- 🔀 Update EIP-7002 tests to match spec changes in [ethereum/execution-apis#549](https://github.com/ethereum/execution-apis/pull/549) ([#600](https://github.com/ethereum/execution-spec-tests/pull/600)). -- ✨ Convert a few eip1153 tests from ethereum/tests repo into .py ([#440](https://github.com/ethereum/execution-spec-tests/pull/440)). -- ✨ Add tests for [EIP-7480: EOF - Data section access instructions](https://eips.ethereum.org/EIPS/eip-7480) ([#518](https://github.com/ethereum/execution-spec-tests/pull/518), [#664](https://github.com/ethereum/execution-spec-tests/pull/664)). -- ✨ Add tests for subcontainer kind validation from [EIP-7620: EOF Contract Creation](https://eips.ethereum.org/EIPS/eip-7620) for the cases with deeply nested containers and non-first code sections ([#676](https://github.com/ethereum/execution-spec-tests/pull/676)). -- ✨ Add tests for runtime stack overflow at CALLF instruction from [EIP-4750: EOF - Functions](https://eips.ethereum.org/EIPS/eip-4750) ([#678](https://github.com/ethereum/execution-spec-tests/pull/678)). -- ✨ Add tests for runtime stack overflow at JUMPF instruction from [EIP-6206: EOF - JUMPF and non-returning functions](https://eips.ethereum.org/EIPS/eip-6206) ([#690](https://github.com/ethereum/execution-spec-tests/pull/690)). -- ✨ Add tests for Devnet-1 version of [EIP-7702: Set EOA account code](https://eips.ethereum.org/EIPS/eip-7702) ([#621](https://github.com/ethereum/execution-spec-tests/pull/621)). - -### 🛠️ Framework - -- 🐞 Fix incorrect `!=` operator for `FixedSizeBytes` ([#477](https://github.com/ethereum/execution-spec-tests/pull/477)). -- ✨ Add Macro enum that represents byte sequence of Op instructions ([#457](https://github.com/ethereum/execution-spec-tests/pull/457)). -- ✨ Number of parameters used to call opcodes (to generate bytecode) is now checked ([#492](https://github.com/ethereum/execution-spec-tests/pull/492)). -- ✨ Libraries have been refactored to use `pydantic` for type checking in most test types ([#486](https://github.com/ethereum/execution-spec-tests/pull/486), [#501](https://github.com/ethereum/execution-spec-tests/pull/501), [#508](https://github.com/ethereum/execution-spec-tests/pull/508)). -- ✨ Opcodes are now subscriptable and it's used to define the data portion of the opcode: `Op.PUSH1(1) == Op.PUSH1[1] == b"\x60\x01"` ([#513](https://github.com/ethereum/execution-spec-tests/pull/513)). -- ✨ Added EOF fixture format ([#512](https://github.com/ethereum/execution-spec-tests/pull/512)). -- ✨ Verify filled EOF fixtures using `evmone-eofparse` during `fill` execution ([#519](https://github.com/ethereum/execution-spec-tests/pull/519)). -- ✨ Added `--traces` support when running with Hyperledger Besu ([#511](https://github.com/ethereum/execution-spec-tests/pull/511)). -- ✨ Use pytest's "short" traceback style (`--tb=short`) for failure summaries in the test report for more compact terminal output ([#542](https://github.com/ethereum/execution-spec-tests/pull/542)). -- ✨ The `fill` command now generates HTML test reports with links to the JSON fixtures and debug information ([#537](https://github.com/ethereum/execution-spec-tests/pull/537)). -- ✨ Add an Ethereum RPC client class for use with consume commands ([#556](https://github.com/ethereum/execution-spec-tests/pull/556)). -- ✨ Add a "slow" pytest marker, in order to be able to limit the filled tests until release ([#562](https://github.com/ethereum/execution-spec-tests/pull/562)). -- ✨ Add a CLI tool that generates blockchain tests as Python from a transaction hash ([#470](https://github.com/ethereum/execution-spec-tests/pull/470), [#576](https://github.com/ethereum/execution-spec-tests/pull/576)). -- ✨ Add more Transaction and Block exceptions from existing ethereum/tests repo ([#572](https://github.com/ethereum/execution-spec-tests/pull/572)). -- ✨ Add "description" and "url" fields containing test case documentation and a source code permalink to fixtures during `fill` and use them in `consume`-generated Hive test reports ([#579](https://github.com/ethereum/execution-spec-tests/pull/579)). -- ✨ Add git workflow evmone coverage script for any new lines mentioned in converted_ethereum_tests.txt ([#503](https://github.com/ethereum/execution-spec-tests/pull/503)). -- ✨ Add a new covariant marker `with_all_contract_creating_tx_types` that allows automatic parametrization of a test with all contract-creating transaction types at the current executing fork ([#602](https://github.com/ethereum/execution-spec-tests/pull/602)). -- ✨ Tests are now encouraged to declare a `pre: Alloc` parameter to get the pre-allocation object for the test, and use `pre.deploy_contract` and `pre.fund_eoa` to deploy contracts and fund accounts respectively, instead of declaring the `pre` as a dictionary or modifying its contents directly (see the [state test tutorial](https://eest.ethereum.org/v4.1.0/writing_tests/tutorials/state_transition/) for an updated example) ([#584](https://github.com/ethereum/execution-spec-tests/pull/584)). -- ✨ Enable loading of [ethereum/tests/BlockchainTests](https://github.com/ethereum/tests/tree/develop/BlockchainTests) ([#596](https://github.com/ethereum/execution-spec-tests/pull/596)). -- 🔀 Refactor `gentest` to use `ethereum_test_tools.rpc.rpc` by adding to `get_transaction_by_hash`, `debug_trace_call` to `EthRPC` ([#568](https://github.com/ethereum/execution-spec-tests/pull/568)). -- ✨ Write a properties file to the output directory and enable direct generation of a fixture tarball from `fill` via `--output=fixtures.tgz`([#627](https://github.com/ethereum/execution-spec-tests/pull/627)). -- 🔀 `ethereum_test_tools` library has been split into multiple libraries ([#645](https://github.com/ethereum/execution-spec-tests/pull/645)). -- ✨ Add the consume engine simulator and refactor the consume simulator suite ([#691](https://github.com/ethereum/execution-spec-tests/pull/691)). -- 🐞 Prevents forcing consume to use stdin as an input when running from hive ([#701](https://github.com/ethereum/execution-spec-tests/pull/701)). - -### 📋 Misc - -- 🐞 Fix CI by using Golang 1.21 in Github Actions to build geth ([#484](https://github.com/ethereum/execution-spec-tests/pull/484)). -- 💥 "Merge" has been renamed to "Paris" in the "network" field of the Blockchain tests, and in the "post" field of the State tests ([#480](https://github.com/ethereum/execution-spec-tests/pull/480)). -- ✨ Port entry point scripts to use [click](https://click.palletsprojects.com) and add tests ([#483](https://github.com/ethereum/execution-spec-tests/pull/483)). -- 💥 As part of the pydantic conversion, the fixtures have the following (possibly breaking) changes ([#486](https://github.com/ethereum/execution-spec-tests/pull/486)): - - State test field `transaction` now uses the proper zero-padded hex number format for fields `maxPriorityFeePerGas`, `maxFeePerGas`, and `maxFeePerBlobGas`. - - Fixtures' hashes (in the `_info` field) are now calculated by removing the "_info" field entirely instead of it being set to an empty dict. -- 🐞 Relax minor and patch dependency requirements to avoid conflicting package dependencies ([#510](https://github.com/ethereum/execution-spec-tests/pull/510)). -- 🔀 Update all CI actions to use their respective Node.js 20 versions, ahead of their Node.js 16 version deprecations ([#527](https://github.com/ethereum/execution-spec-tests/pull/527)). -- ✨ Releases now contain a `fixtures_eip7692.tar.gz` which contains all EOF fixtures ([#573](https://github.com/ethereum/execution-spec-tests/pull/573)). -- ✨ Use `solc-select` for tox when running locally and within CI ([#604](https://github.com/ethereum/execution-spec-tests/pull/604)). - -### 💥 Breaking Change - -- Cancun is now the latest deployed fork, and the development fork is now Prague ([#489](https://github.com/ethereum/execution-spec-tests/pull/489)). -- Stable fixtures artifact `fixtures.tar.gz` has been renamed to `fixtures_stable.tar.gz` ([#573](https://github.com/ethereum/execution-spec-tests/pull/573)). -- The "Blockchain Test Hive" fixture format has been renamed to "Blockchain Test Engine" and updated to more closely resemble the `engine_newPayload` format in the `execution-apis` specification () and now contains a single `"params"` field instead of multiple fields for each parameter ([#687](https://github.com/ethereum/execution-spec-tests/pull/687)). -- Output folder for fixtures has been renamed from "blockchain_tests_hive" to "blockchain_tests_engine" ([#687](https://github.com/ethereum/execution-spec-tests/pull/687)). - -## [v2.1.1](https://github.com/ethereum/execution-spec-tests/releases/tag/v2.1.1) - 2024-03-09 - -### 🧪 Test Cases - -- 🐞 Dynamic create2 collision from different transactions same block ([#430](https://github.com/ethereum/execution-spec-tests/pull/430)). -- 🐞 Fix beacon root contract deployment tests so the account in the pre-alloc is not empty ([#425](https://github.com/ethereum/execution-spec-tests/pull/425)). -- 🔀 All beacon root contract tests are now contained in tests/cancun/eip4788_beacon_root/test_beacon_root_contract.py, and all state tests have been converted back to blockchain tests format ([#449](https://github.com/ethereum/execution-spec-tests/pull/449)). - -### 🛠️ Framework - -- ✨ Adds two `consume` commands [#339](https://github.com/ethereum/execution-spec-tests/pull/339): - - 1. `consume direct` - Execute a test fixture directly against a client using a `blocktest`-like command (currently only geth supported). - 2. `consume rlp` - Execute a test fixture in a hive simulator against a client that imports the test's genesis config and blocks as RLP upon startup. This is a re-write of the [ethereum/consensus](https://github.com/ethereum/hive/tree/master/simulators/ethereum/consensus) Golang simulator. - -- ✨ Add Prague to forks ([#419](https://github.com/ethereum/execution-spec-tests/pull/419)). -- ✨ Improve handling of the argument passed to `solc --evm-version` when compiling Yul code ([#418](https://github.com/ethereum/execution-spec-tests/pull/418)). -- 🐞 Fix `fill -m yul_test` which failed to filter tests that are (dynamically) marked as a yul test ([#418](https://github.com/ethereum/execution-spec-tests/pull/418)). -- 🔀 Helper methods `to_address`, `to_hash` and `to_hash_bytes` have been deprecated in favor of `Address` and `Hash`, which are automatically detected as opcode parameters and pushed to the stack in the resulting bytecode ([#422](https://github.com/ethereum/execution-spec-tests/pull/422)). -- ✨ `Opcodes` enum now contains docstrings with each opcode description, including parameters and return values, which show up in many development environments ([#424](https://github.com/ethereum/execution-spec-tests/pull/424)) @ThreeHrSleep. -- 🔀 Locally calculate state root for the genesis blocks in the blockchain tests instead of calling t8n ([#450](https://github.com/ethereum/execution-spec-tests/pull/450)). -- 🐞 Fix bug that causes an exception during test collection because the fork parameter contains `None` ([#452](https://github.com/ethereum/execution-spec-tests/pull/452)). -- ✨ The `_info` field in the test fixtures now contains a `hash` field, which is the hash of the test fixture, and a `hasher` script has been added which prints and performs calculations on top of the hashes of all fixtures (see `hasher -h`) ([#454](https://github.com/ethereum/execution-spec-tests/pull/454)). -- ✨ Adds an optional `verify_sync` field to hive blockchain tests (EngineAPI). When set to true a second client attempts to sync to the first client that executed the tests ([#431](https://github.com/ethereum/execution-spec-tests/pull/431)). -- 🐞 Fix manually setting the gas limit in the genesis test env for post genesis blocks in blockchain tests ([#472](https://github.com/ethereum/execution-spec-tests/pull/472)). - -### 📋 Misc - -- 🐞 Fix deprecation warnings due to outdated config in recommended VS Code project settings ([#420](https://github.com/ethereum/execution-spec-tests/pull/420)). -- 🐞 Fix typo in the selfdestruct revert tests module ([#421](https://github.com/ethereum/execution-spec-tests/pull/421)). - -## [v2.1.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v2.1.0) - 2024-01-29: 🐍🏖️ Cancun - -Release [v2.1.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v2.1.0) primarily fixes a small bug introduced within the previous release where transition forks are used within the new `StateTest` format. This was highlighted by @chfast within #405 (), where the fork name `ShanghaiToCancunAtTime15k` was found within state tests. - -### 🧪 Test Cases - -- ✨ [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844): Adds `test_blob_gas_subtraction_tx()` verifying the blob gas fee is subtracted from the sender before executing the blob tx ([#407](https://github.com/ethereum/execution-spec-tests/pull/407)). - -### 🛠️ Framework - -- 🐞 State tests generated with transition forks no longer use the transition fork name in the fixture output, instead they use the actual enabled fork according to the state test's block number and timestamp ([#406](https://github.com/ethereum/execution-spec-tests/pull/406)). - -### 📋 Misc - -- ✨ Use `run-parallel` and shared wheel packages for `tox` ([#408](https://github.com/ethereum/execution-spec-tests/pull/408)). - -## [v2.0.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v2.0.0) - 2024-01-25: 🐍🏖️ Cancun - -Release [v2.0.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v2.0.0) contains many important framework changes, including introduction of the `StateTest` format, and some additional Cancun and other test coverage. - -Due to changes in the framework, there is a breaking change in the directory structure in the release tarball, please see the dedicated "💥 Breaking Changes" section below for more information. - -### 🧪 Test Cases - -- ✨ [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844): Add `test_sufficient_balance_blob_tx()` and `test_sufficient_balance_blob_tx_pre_fund_tx()` ([#379](https://github.com/ethereum/execution-spec-tests/pull/379)). -- ✨ [EIP-6780](https://eips.ethereum.org/EIPS/eip-6780): Add a reentrancy suicide revert test ([#372](https://github.com/ethereum/execution-spec-tests/pull/372)). -- ✨ [EIP-1153](https://eips.ethereum.org/EIPS/eip-1153): Add `test_run_until_out_of_gas()` for transient storage opcodes ([#401](https://github.com/ethereum/execution-spec-tests/pull/401)). -- ✨ [EIP-198](https://eips.ethereum.org/EIPS/eip-198): Add tests for the MODEXP precompile ([#364](https://github.com/ethereum/execution-spec-tests/pull/364)). -- ✨ Tests for nested `CALL` and `CALLCODE` gas consumption with a positive value transfer (previously lacking coverage) ([#371](https://github.com/ethereum/execution-spec-tests/pull/371)). -- 🐞 [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844): Fixed `test_invalid_tx_max_fee_per_blob_gas()` to account for extra gas required in the case where the account is incorrectly deduced the balance as if it had the correct block blob gas fee ([#370](https://github.com/ethereum/execution-spec-tests/pull/370)). -- 🐞 [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844): Fixed `test_insufficient_balance_blob_tx()` to correctly calculate the minimum balance required for the accounts ([#379](https://github.com/ethereum/execution-spec-tests/pull/379)). -- 🐞 [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844): Fix and enable `test_invalid_blob_tx_contract_creation` ([#379](https://github.com/ethereum/execution-spec-tests/pull/379)). -- 🔀 Convert all eligible `BlockchainTest`s to `StateTest`s (and additionally generate corresponding `BlockchainTest`s) ([#368](https://github.com/ethereum/execution-spec-tests/pull/368), [#370](https://github.com/ethereum/execution-spec-tests/pull/370)). - -### 🛠️ Framework - -- ✨ Add `StateTest` fixture format generation; `StateTests` now generate a `StateTest` and a corresponding `BlockchainTest` test fixture, previously only `BlockchainTest` fixtures were generated ([#368](https://github.com/ethereum/execution-spec-tests/pull/368)). -- ✨ Add `StateTestOnly` fixture format is now available and its only difference with `StateTest` is that it does not produce a `BlockchainTest` ([#368](https://github.com/ethereum/execution-spec-tests/pull/368)). -- ✨ Add `evm_bytes_to_python` command-line utility which converts EVM bytecode to Python Opcodes ([#357](https://github.com/ethereum/execution-spec-tests/pull/357)). -- ✨ Fork objects used to write tests can now be compared using the `>`, `>=`, `<`, `<=` operators, to check for a fork being newer than, newer than or equal, older than, older than or equal, respectively when compared against other fork ([#367](https://github.com/ethereum/execution-spec-tests/pull/367)). -- ✨ Add [solc 0.8.23](https://github.com/ethereum/solidity/releases/tag/v0.8.23) support ([#373](https://github.com/ethereum/execution-spec-tests/pull/373)). -- ✨ Add framework unit tests for post state exception verification ([#350](https://github.com/ethereum/execution-spec-tests/pull/350)). -- ✨ Add a helper class `ethereum_test_tools.TestParameterGroup` to define test parameters as dataclasses and auto-generate test IDs ([#364](https://github.com/ethereum/execution-spec-tests/pull/364)). -- ✨ Add a `--single-fixture-per-file` flag to generate one fixture JSON file per test case ([#331](https://github.com/ethereum/execution-spec-tests/pull/331)). -- 🐞 Storage type iterator is now fixed ([#369](https://github.com/ethereum/execution-spec-tests/pull/369)). -- 🐞 Fix type coercion in `FixtureHeader.join()` ([#398](https://github.com/ethereum/execution-spec-tests/pull/398)). -- 🔀 Locally calculate the transactions list's root instead of using the one returned by t8n when producing BlockchainTests ([#353](https://github.com/ethereum/execution-spec-tests/pull/353)). -- 🔀 Change custom exception classes to dataclasses to improve testability ([#386](https://github.com/ethereum/execution-spec-tests/pull/386)). -- 🔀 Update fork name from "Merge" to "Paris" used within the framework and tests ([#363](https://github.com/ethereum/execution-spec-tests/pull/363)). -- 💥 Replace `=` with `_` in pytest node ids and test fixture names ([#342](https://github.com/ethereum/execution-spec-tests/pull/342)). -- 💥 The `StateTest`, spec format used to write tests, is now limited to a single transaction per test ([#361](https://github.com/ethereum/execution-spec-tests/pull/361)). -- 💥 Tests must now use `BlockException` and `TransactionException` to define the expected exception of a given test, which can be used to test whether the client is hitting the proper exception when processing the block or transaction ([#384](https://github.com/ethereum/execution-spec-tests/pull/384)). -- 💥 `fill`: Remove the `--enable-hive` flag; now all test types are generated by default ([#358](https://github.com/ethereum/execution-spec-tests/pull/358)). -- 💥 Rename test fixtures names to match the corresponding pytest node ID as generated using `fill` ([#342](https://github.com/ethereum/execution-spec-tests/pull/342)). - -### 📋 Misc - -- ✨ Docs: Add a ["Consuming Tests"](https://eest.ethereum.org/main/consuming_tests/) section to the docs, where each test fixture format is described, along with the steps to consume them, and the description of the structures used in each format ([#375](https://github.com/ethereum/execution-spec-tests/pull/375)). -- 🔀 Docs: Update `t8n` tool branch to fill tests for development features in the [readme](https://github.com/ethereum/execution-spec-tests) ([#338](https://github.com/ethereum/execution-spec-tests/pull/338)). -- 🔀 Filling tool: Updated the default filling tool (`t8n`) to go-ethereum@master ([#368](https://github.com/ethereum/execution-spec-tests/pull/368)). -- 🐞 Docs: Fix error banner in online docs due to mermaid syntax error ([#398](https://github.com/ethereum/execution-spec-tests/pull/398)). -- 🐞 Docs: Fix incorrectly formatted nested lists in online doc ([#403](https://github.com/ethereum/execution-spec-tests/pull/403)). -- 🔀 CLI: `evm_bytes_to_python` is renamed to `evm_bytes` and now accepts flag `--assembly` to output the code in assembly format ([#844](https://github.com/ethereum/execution-spec-tests/pull/844)). - -### 💥 Breaking Changes - -A concrete example of the test name renaming and change in directory structure is provided below. - -1. Fixture output, including release tarballs, now contain subdirectories for different test types: - - 1. `blockchain_tests`: Contains `BlockchainTest` formatted tests - 2. `blockchain_tests_hive`: Contains `BlockchainTest` with Engine API call directives for use in hive - 3. `state_tests`: Contains `StateTest` formatted tests - -2. `StateTest`, spec format used to write tests, is now limited to a single transaction per test. -3. In this release the pytest node ID is now used for fixture names (previously only the test parameters were used), this should not be breaking. However, `=` in both node IDs (and therefore fixture names) have been replaced with `_`, which may break tooling that depends on the `=` character. -4. Produced `blockchain_tests` fixtures and their corresponding `blockchain_tests_hive` fixtures now contain the named exceptions `BlockException` and `TransactionException` as strings in the `expectException` and `validationError` fields, respectively. These exceptions can be used to test whether the client is hitting the proper exception when processing an invalid block. - - Blockchain test: - - ```json - "blocks": [ - { - ... - "expectException": "TransactionException.INSUFFICIENT_ACCOUNT_FUNDS", - ... - } - ... - ] - ``` - - Blockchain hive test: - - ```json - "engineNewPayloads": [ - { - ... - "validationError": "TransactionException.INSUFFICIENT_ACCOUNT_FUNDS", - ... - } - ... - ] - ``` - -#### Renaming and Release Tarball Directory Structure Change Example - -The fixture renaming provides a more consistent naming scheme between the pytest node ID and fixture name and allows the fixture name to be provided directly to pytest 5on the command line to execute individual tests in isolation, e.g. `pytest tests/frontier/opcodes/test_dup.py::test_dup[fork_Frontier]`. - -1. Pytest node ID example: - - 1. Previous node ID: `tests/frontier/opcodes/test_dup.py::test_dup[fork=Frontier]`. - 2. New node ID: `tests/frontier/opcodes/test_dup.py::test_dup[fork_Frontier]`. - -2. Fixture name example: - - 1. Previous fixture name: `000-fork=Frontier` - 2. New fixture name: `tests/frontier/opcodes/test_dup.py::test_dup[fork_Frontier]` (now the same as the pytest node ID). - -3. Fixture JSON file name example (within the release tarball): - - 1. Previous fixture file name: `fixtures/frontier/opcodes/dup/dup.json` (`BlockChainTest` format). - 2. New fixture file names (all present within the release tarball): - - - `fixtures/state_tests/frontier/opcodes/dup/dup.json` (`StateTest` format). - - `fixtures/blockchain_tests/frontier/opcodes/dup/dup.json` (`BlockChainTest` format). - - `fixtures/blockchain_tests_hive/frontier/opcodes/dup/dup.json` (a blockchain test in `HiveFixture` format). - -## [v1.0.6](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.6) - 2023-10-19: 🐍🏖️ Cancun Devnet 10 - -### 🧪 Test Cases - -- 🔀 [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844): Update KZG point evaluation test vectors to use data from the official KZG setup and Mainnet Trusted Setup ([#336](https://github.com/ethereum/execution-spec-tests/pull/336)). - -### 🛠️ Framework - -- 🔀 Fixtures: Add a non-RLP format field (`rlp_decoded`) to invalid blocks ([#322](https://github.com/ethereum/execution-spec-tests/pull/322)). -- 🔀 Spec: Refactor state and blockchain spec ([#307](https://github.com/ethereum/execution-spec-tests/pull/307)). - -### 🔧 EVM Tools - -- ✨ Run geth's `evm blocktest` command to verify JSON fixtures after test case execution (`--verify-fixtures`) ([#325](https://github.com/ethereum/execution-spec-tests/pull/325)). -- ✨ Enable tracing support for `ethereum-spec-evm` ([#289](https://github.com/ethereum/execution-spec-tests/pull/289)). - -### 📋 Misc - -- ✨ Tooling: Add Python 3.12 support ([#309](https://github.com/ethereum/execution-spec-tests/pull/309)). -- ✨ Process: Added a Github pull request template ([#308](https://github.com/ethereum/execution-spec-tests/pull/308)). -- ✨ Docs: Changelog updated post release ([#321](https://github.com/ethereum/execution-spec-tests/pull/321)). -- ✨ Docs: Add [a section explaining execution-spec-tests release artifacts](https://eest.ethereum.org/v4.1.0/consuming_tests/) ([#334](https://github.com/ethereum/execution-spec-tests/pull/334)). -- 🔀 T8N Tool: Branch used to generate the tests for Cancun is now [lightclient/go-ethereum@devnet-10](https://github.com/lightclient/go-ethereum/tree/devnet-10) ([#336](https://github.com/ethereum/execution-spec-tests/pull/336)). - -### 💥 Breaking Change - -- Fixtures now use the Mainnet Trusted Setup merged on [consensus-specs#3521](https://github.com/ethereum/consensus-specs/pull/3521) ([#336](https://github.com/ethereum/execution-spec-tests/pull/336)). - -## [v1.0.5](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.5) - 2023-09-26: 🐍🏖️ Cancun Devnet 9 Release 3 - -This release mainly serves to update the EIP-4788 beacon roots address to `0x000F3df6D732807Ef1319fB7B8bB8522d0Beac02`, as updated in [ethereum/EIPs/pull/7672](https://github.com/ethereum/EIPs/pull/7672). - -### 🧪 Test Cases - -- 🐞 [EIP-4844](https://eips.ethereum.org/EIPS/eip-4844): Fix invalid blob txs pre-Cancun engine response ([#306](https://github.com/ethereum/execution-spec-tests/pull/306)). -- ✨ [EIP-4788](https://eips.ethereum.org/EIPS/eip-4788): Final update to the beacon root address ([#312](https://github.com/ethereum/execution-spec-tests/pull/312)). - -### 📋 Misc - -- ✨ Docs: Changelog added ([#305](https://github.com/ethereum/execution-spec-tests/pull/305)). -- ✨ CI/CD: Run development fork tests in Github Actions ([#302](https://github.com/ethereum/execution-spec-tests/pull/302)). -- ✨ CI/CD: Generate test JSON fixtures on push ([#303](https://github.com/ethereum/execution-spec-tests/pull/303)). - -### 💥 Breaking Change - -Please use development fixtures from now on when testing Cancun. These refer to changes that are currently under development within clients: - -- fixtures: All tests until the last stable fork (Shanghai). -- fixtures_develop: All tests until the last development fork (Cancun). -- fixtures_hive: All tests until the last stable fork (Shanghai) in hive format (Engine API directives instead of the usual BlockchainTest format). -- fixtures_develop_hive: All tests until the last development fork (Cancun) in hive format. - -## [v1.0.4](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.4) - 2023-09-21: 🐍 Cancun Devnet 9 Release 2 - -This release adds additional coverage to the current set of Cancun tests, up to the [Devnet-9 Cancun specification](https://notes.ethereum.org/@ethpandaops/dencun-devnet-9). - -**Note:** Additional EIP-4788 updates from [ethereum/EIPs/pull/7672](https://github.com/ethereum/EIPs/pull/7672) will be included in the next release. - -### 🧪 Test Cases - -- ✨ [EIP-7516: BLOBBASEFEE opcode](https://eips.ethereum.org/EIPS/eip-7516): Add first and comprehensive tests (@marioevz in [#294](https://github.com/ethereum/execution-spec-tests/pull/294)). -- ✨ [EIP-4788: Beacon block root in the EVM](https://eips.ethereum.org/EIPS/eip-4788): Increase coverage (@spencer-tb in [#297](https://github.com/ethereum/execution-spec-tests/pull/297)). -- 🐞 [EIP-1153: Transient storage opcodes](https://eips.ethereum.org/EIPS/eip-1153): Remove conftest '+1153' in network field (@spencer-tb in [#299](https://github.com/ethereum/execution-spec-tests/pull/299)). - -### 🛠️ Framework - -- 🔀 [EIP-4788](https://eips.ethereum.org/EIPS/eip-4788): Beacon root contract is pre-deployed at `0xbEAC020008aFF7331c0A389CB2AAb67597567d7a` (@spencer-tb in [#297](https://github.com/ethereum/execution-spec-tests/pull/297)). -- ✨ Deprecate empty accounts within framework (@spencer-tb in [#300](https://github.com/ethereum/execution-spec-tests/pull/300)). -- ✨ Fixture generation split based on hive specificity (@spencer-tb in [#301](https://github.com/ethereum/execution-spec-tests/pull/301)). -- 💥 `fill`: `--disable-hive` flag removed; replaced by `--enable-hive` (@spencer-tb in [#301](https://github.com/ethereum/execution-spec-tests/pull/301)). -- ✨ Add engine API forkchoice updated information in fixtures (@spencer-tb in [#256](https://github.com/ethereum/execution-spec-tests/pull/256)). - -## [v1.0.3](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.3) - 2023-09-14: 🐍 Cancun Devnet 9 Release - -See [v1.0.3](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.3). - -## [v1.0.2](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.2) - 2023-08-11: 🐍 Cancun Devnet 8 + 4788 v2 Pre-Release - -See [v1.0.2](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.2). - -## [v1.0.1](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.1) - 2023-08-03: 🐍 Cancun Devnet-8 Pre-Release - -See [v1.0.1](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.1). - -## [v1.0.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.0) - 2023-06-27: 🧪 Welcome to the Pytest Era - -See [v1.0.0](https://github.com/ethereum/execution-spec-tests/releases/tag/v1.0.0). - -Older releases can be found on [the releases page](https://github.com/ethereum/execution-spec-tests/releases). diff --git a/docs/changelog_section_template.md b/docs/changelog_section_template.md deleted file mode 100644 index dfa30a4ec83..00000000000 --- a/docs/changelog_section_template.md +++ /dev/null @@ -1,19 +0,0 @@ -# Changelog Section Template - -The following can be copy-pasted into the `CHANGELOG.md` file for a new release. - -## 🔜 [Unreleased] - -### 💥 Breaking Change - -### 🛠️ Framework - -#### `fill` - -#### `consume` - -#### `execute` - -### 📋 Misc - -### 🧪 Test Cases diff --git a/docs/consensus_tests/index.md b/docs/consensus_tests/index.md new file mode 100644 index 00000000000..0971d147a44 --- /dev/null +++ b/docs/consensus_tests/index.md @@ -0,0 +1,7 @@ +# Consensus Tests + +The consensus tests in `./tests/` verify that Ethereum execution clients implement the protocol correctly and agree on the resulting state transitions. This section covers the full workflow: + +- [Writing Tests](../writing_tests/index.md): Authoring new Python test cases using the execution-testing framework. +- [Filling Tests](../filling_tests/index.md): Generating JSON test fixtures (vectors) from the Python test cases using the `fill` command. +- [Running Tests](../running_tests/index.md): Executing the generated fixtures against clients, either directly or via Hive. diff --git a/docs/dev/coding_style.md b/docs/dev/coding_style.md deleted file mode 100644 index 6772746ca0b..00000000000 --- a/docs/dev/coding_style.md +++ /dev/null @@ -1,4 +0,0 @@ -# Coding Style - -!!! warning "Documentation Moved" - This documentation has been relocated to [Getting Started -> Code Standards](../getting_started/code_standards.md). Please use the new location for the most up-to-date information. diff --git a/docs/dev/deps_and_packaging.md b/docs/dev/deps_and_packaging.md index ab52eec197b..a0d6997fe9e 100644 --- a/docs/dev/deps_and_packaging.md +++ b/docs/dev/deps_and_packaging.md @@ -1,8 +1,17 @@ -# EEST Dependency Management and Packaging +# Dependency Management and Packaging -EEST uses [`uv`](https://docs.astral.sh/uv/) to manage and pin its dependencies. +EELS uses [`uv`](https://docs.astral.sh/uv/) to manage and pin its dependencies, and a minimum `uv>=0.7.0` is required. -A minimum version of `uv>=0.7.0` is required to ensure `uv` writes `uv.lock` files with consistent fields and formatting (see [ethereum/execution-spec-tests#1597](https://github.com/ethereum/execution-spec-tests/pull/1597)). +## Workspace Layout + +The repo is a `uv` workspace with two members, each defined by its own `pyproject.toml`: + +| Package | `pyproject.toml` | Contents | +| ---------------------------- | --------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------- | +| `ethereum-execution` | [`pyproject.toml`](https://github.com/ethereum/execution-specs/blob/a830dab6f130151ab9023a473b7543120aa21961/pyproject.toml) | The Python specs (`src/ethereum/`) and associated tools. | +| `ethereum-execution-testing` | [`packages/testing/pyproject.toml`](https://github.com/ethereum/execution-specs/blob/a830dab6f130151ab9023a473b7543120aa21961/packages/testing/pyproject.toml) | The EEST test framework under `packages/testing/`. | + +A single [`uv.lock`](https://github.com/ethereum/execution-specs/blob/a830dab6f130151ab9023a473b7543120aa21961/uv.lock) at the repo root pins dependencies for both packages. ## Managing Dependencies @@ -10,120 +19,101 @@ We aim to provide specific [version specifiers](https://peps.python.org/pep-0440 !!! note "Packages should be managed via `uv`" - Dependencies should be managed using `uv` on the command-line to ensure that version compatibility is ensured across all dependencies and that `uv.lock` is updated as required. + Dependencies should be managed using `uv` on the command-line to ensure that version compatibility is maintained across all dependencies and that `uv.lock` is updated as required. The docs below cover common operations, see the `uv` [documentation on managing dependencies](https://docs.astral.sh/uv/concepts/projects/dependencies/#multiple-sources) for more information. -!!! info "Separate PRs are preferred when managing dependencies" +!!! info "Target the right workspace member" - An upgrade of all pinned dependencies in `uv.lock` must be performed in a dedicated PR! - - For other dependency changes, they can be included in the PR that removed use of the library, for example. But if a version bump is made, without related source code changes, it should be done in a dedicated PR. This makes the change: + Run `uv` commands from the repo root. By default they target `ethereum-execution` (the specs package). To target the test framework, pass `--package ethereum-execution-testing` (or equivalently, `cd packages/testing/` first and run `uv` from there). - - Easier to track. - - Trivial to revert. + Either way, the single `uv.lock` at the repo root is updated and should be committed alongside the `pyproject.toml` change. + +!!! info "Separate PRs are preferred when managing dependencies" -### Adding/modifying direct dependencies + An upgrade of all pinned dependencies in `uv.lock` must be performed in a dedicated PR. -These are packages listed in the project's direct dependencies, i.e., in `pyproject.toml` `[project]` section: + For other dependency changes, they can be included in the PR that adds or removes use of the library. But if a version bump is made without related source code changes, it should be done in a dedicated PR. This makes the change: -```toml -[project] -... -dependencies = [ - "click>=8.1.0,<9", - ... - "pytest-regex>=0.2.0,<0.3", -] -``` + - Easier to track. + - Trivial to revert. -or, for source package dependencies (directly installed via a `git+` specifier from Github), in the `[tool.uv.sources]` section: +### Adding or modifying direct dependencies -```toml -[tool.uv.sources] -ethereum-spec-evm-resolver = { git = "https://github.com/petertdavies/ethereum-spec-evm-resolver", rev = \ -... -``` +Direct dependencies are the packages listed in each package's `[project] dependencies` table. -!!! example "Example: Updating direct dependencies" +!!! example "Adding a direct dependency to the specs package" - Example of a package dependency update: ```console uv add "requests>=2.31,<2.33" ``` - Example of a source dependency update: +!!! example "Adding a direct dependency to the testing package" + ```console - uv add "ethereum-spec-evm-resolver @ git+https://github.com/petertdavies/ethereum-spec-evm-resolver@623ac4565025e72b65f45b926da2a3552041b469" + uv add --package ethereum-execution-testing "requests>=2.31,<2.33" ``` -### Adding/modifying development dependencies - -Development dependencies are managed in dependency groups: `lint`, `doc`, `test`, and `mkdocs` defined in the `pyproject.toml`: - -```toml -[dependency-groups] -test = [ - "pytest>=8,<9", - "pytest-cov>=4.1.0,<5", - ... -] -lint = [ - "ruff==0.13.2", - "mypy==1.17.0", - "types-requests>=2.31,<2.33", - ... -] -``` +### Adding or modifying development dependencies -These can be modified via `uv`on the command-line or edited by hand. If editing manually, you must run `uv lock` afterwards to update the lockfile. +Development dependencies are grouped into `[dependency-groups]`, one group per concern, plus a `dev` meta-group that includes them all. -!!! example "Example: Updating a development dependency" +Groups defined by the specs package: + +- `test`, `lint`, `actionlint`, `doc`, `mkdocs`. +- `dev` includes all of the above plus the `optimized` extra. + +Groups defined by the testing package: + +- `test`, `lint`. +- `dev` includes both. + +!!! example "Adding a dev dependency to the specs `lint` group" - Using uv: ```console uv add --group lint "types-requests>=2.31,<2.33" ``` - Or edit `pyproject.toml` manually and then run: +!!! example "Adding a dev dependency to the testing package `test` group" + ```console - uv lock + uv add --package ethereum-execution-testing --group test "pytest-timeout>=2.3,<3" ``` -### Adding/modifying optional dependencies - -The `optimized` optional extra provides performance enhancements and is the only remaining optional dependency group: +### Adding or modifying optional dependencies -```toml -[project.optional-dependencies] -optimized = [ - "rust-pyspec-glue>=0.0.9,<0.1.0", - "ethash>=1.1.0,<2", -] -``` +The specs package defines a single optional extra, `optimized`, which pulls in `rust-pyspec-glue` and `ethash` for EVM performance. -!!! example "Example: Updating an optional dependency" +!!! example "Updating an optional dependency" ```console uv add --optional optimized "ethash>=1.1.0,<2" ``` - Or edit `pyproject.toml` by hand and run `uv lock`. - ## Upgrading Pinned Dependencies in `uv.lock` -To upgrade all pinned dependencies in `uv.lock` to the latest version permitted by EEST's `project.toml` version specifiers run: +To upgrade all pinned dependencies in `uv.lock` to the latest versions permitted by both packages' version specifiers, run: ```console uv lock --upgrade ``` -Project-wide dependency upgrades must be made via a dedicated PR! +Project-wide dependency upgrades must be made via a dedicated PR. -To upgrade a single package run: +To upgrade a single package, run: ```console uv lock --upgrade-package ``` See [Locking and Syncing](https://docs.astral.sh/uv/concepts/projects/sync/#upgrading-locked-package-versions) in the `uv` docs for more information. + +## Verifying `uv.lock` + +After any dependency change, verify that `uv.lock` is consistent with both `pyproject.toml` files: + +```console +just lock-check +``` + +This recipe also runs as part of `just static` and must be clean before committing. diff --git a/docs/dev/docs.md b/docs/dev/docs.md index 8673217ae51..3be9a658e6b 100644 --- a/docs/dev/docs.md +++ b/docs/dev/docs.md @@ -1,6 +1,6 @@ # Documentation -The `execution-spec-tests` documentation is generated via [`mkdocs`](https://www.mkdocs.org/) and hosted remotely on Github Pages at [eest.ethereum.org](https://eest.ethereum.org/). +The `execution-specs` documentation is generated via [`mkdocs`](https://www.mkdocs.org/) and hosted at [steel.ethereum.foundation/docs/execution-specs/](https://steel.ethereum.foundation/docs/execution-specs/). ## Prerequisites @@ -13,21 +13,24 @@ uv sync One time build in strict mode: ```console -uv run mkdocs build --strict +just docs ``` -Perform all docs related checks via `tox` in parallel: +Run all docs related checks: ```console -uvx tox -e spellcheck,markdownlint,mkdocs --parallel +just spellcheck +just lint-md +just docs ``` ### Local Deployment and Test -This runs continually: Deploys the site locally and re-generates the site upon modifications to `docs/**/*.md` or `tests/**/*.py`: +Serve the site locally with live reload on changes to `docs/**/*.md` or `tests/**/*.py`: ```console -uv run mkdocs serve +just docs-serve # full build +just docs-serve-fast # skips the Test Case Reference, faster iteration ``` For more help (including ensuring a clean build), see the `gen_test_doc` pytest plugin's documentation: @@ -36,96 +39,6 @@ For more help (including ensuring a clean build), see the `gen_test_doc` pytest options: members: no -## Remote Deployment and Versioning - -The execution-specs-test docs are hosted using Github pages at [eest.ethereum.org](https://eest.ethereum.org/). Versions are updated/deployed automatically as part of Github Actions, but this can also be performed on the command-line. - -Our mkdocs configuration uses [mike](https://github.com/jimporter/mike) as a version provider. All deployments should be made via `mike` (whether as part of CI/CD or executed locally). - -The deployed versions of the docs managed via `mike` are kept in the [gh-pages](https://github.com/ethereum/execution-spec-tests/tree/gh-pages) branch. When you run `mike` it commits to this branch and optionally pushes the changes directly to remote. - -### Aliases - -We currently have two aliases (which both point to `main` as of [#998](https://github.com/ethereum/execution-spec-tests/pull/998)): - -- [`latest`](https://eest.ethereum.org/latest): the current state of the main branch. -- [`development`](https://eest.ethereum.org/development): the current state of the main branch. - -These aliases point to specific versions, as configured below. It's possible to share links containing either of these aliases or to specific versions, i.e, the following are all valid links: - -- https://eest.ethereum.org/ (redirects to latest/main) -- https://eest.ethereum.org/latest (redirects to main) -- https://eest.ethereum.org/development (redirects main) -- https://eest.ethereum.org/main -- https://eest.ethereum.org/v1.0.0 - -### CI/CD: Doc Deployment via Github Actions - -There are two workflows that automatically deploy updated/new versions of the docs: - -| Workflow `yaml` File | What | When | -|----------------------|------|------| -| `docs_main.yaml` | Update "main" version of docs | Push to 'main' branch, (e.g., on PR merge) | -| `docs_tags.yaml` | Deploy new version of docs; tag is used as version name | Upon creating a tag matching `v*` | - -### Build and Deployment (without alias update) - -Build a new version and deploy it to remote (this version will then show up in the version selector list): - -```console -uv run mike deploy -rebase --push v1.2.3 -``` - -!!! note "Local deployment" - If you deploy locally, the documentation will be built with any changes made in your local repository. Check out the tag to deploy tagged versions. - -### Build, Deploy and Update the Alias - -Build, deploy and update the version an alias points to with: - -```console -uv run mike deploy --rebase --push --update-aliases v1.2.3 latest -``` - -where `v1.2.3` indicates the version's name and `development` is the alias. This will overwrite the version if it already exists. - -!!! note "Updating the 'main' version locally" - "main" is just a version name (intended to reflect that it is build from the main branch). However, `mike` will build the docs site from the current local repository state (including local modifications). Therefore, make sure you're on the HEAD of the main branch before executing (unless you know what you're doing :wink:)! - - ```console - uv run mike deploy --rebase --push main - ``` - - If the alias accidentally go change: - - ```console - uv run mike deploy --rebase --push --update-aliases main development - ``` - -### Viewing and Deleting Versions - -List versions: - -```console -uv run mike list -``` - -Delete a version: - -```console -uv run mike delete v1.2.3a1 -``` - -### Set Default Version - -Set the default version of the docs to open upon loading the page: - -```console -uv run mike set-default --push latest -``` - -Typically, this must only be executed once for a repo. - ## Implementation ### Plugins @@ -171,7 +84,3 @@ All pages that are to be included in the documentation and the navigation bar mu - `navigation-post-test-case-reference.md`, and create an arbitrary ordering in the Test Case Reference doc gen script. But this is untested. - -## Read the Docs - -Originally, before June 2023, documentation was hosted at readthedocs.io. Currently, this page no longer exists (execution-spec-tests.readthedocs.io). diff --git a/docs/dev/documenting_clis.md b/docs/dev/documenting_clis.md index f7d3e382d71..18fe63ab7e4 100644 --- a/docs/dev/documenting_clis.md +++ b/docs/dev/documenting_clis.md @@ -7,4 +7,4 @@ Current limitations: 1. `mkdocs serve` does not automatically update the CLI documentation when the source code changes. You must restart the server to see the changes. 2. `mkdocs-click` does not automatically generate a short help string from sub-command docstrings. You must provide a short help string for each sub-command in the source code with `@click.command(short_help="...")`. -See the [markdown](https://github.com/ethereum/execution-spec-tests/blob/main/docs/library/cli/evm_bytes.md) and corresponding [Python docstrings](https://github.com/ethereum/execution-spec-tests/blob/main/src/cli/evm_bytes.py) for the [`evm_bytes` CLI documentation](../library/cli/evm_bytes.md) as an example of how to document a CLI using `mkdocs-click`. +See the [markdown](https://github.com/ethereum/execution-specs/blob/a48e0b381d5225a6c3de2d06cd9ee7ae0b6ca9bb/docs/library/cli/evm_bytes.md) and corresponding [Python docstrings](https://github.com/danceratopz/execution-specs/blob/ca2b3b18a5d4058b2e2fd517ba7db31e86919a09/packages/testing/src/execution_testing/cli/evm_bytes.py) for the [`evm_bytes` CLI documentation](../library/cli/evm_bytes.md) as an example of how to document a CLI using `mkdocs-click`. diff --git a/docs/dev/index.md b/docs/dev/index.md index d7fe93286d6..f347d317b10 100644 --- a/docs/dev/index.md +++ b/docs/dev/index.md @@ -1,16 +1,14 @@ # Developer Documentation -This documentation is aimed at `execution-spec-tests` developers: +This documentation is aimed at `execution-specs` developers: - [Managing configurations](./configurations.md): Instructions for setting up and modifying test configurations. - [Interactive usage](./interactive_usage.md): Guide on interactive use of EEST packages using `ipython`. - [Documenting CLI commands](./documenting_clis.md): Instructions for documenting command line interfaces (CLIs). -- [Coding style](./coding_style.md): Standards and best practices for code formatting and to maintain consistency across the repository. - [Logging](./logging.md): Documentation on using the custom logging system with enhanced features. -- [Enabling pre-commit checks](./precommit.md): A guide for setting up pre-commit hooks to enforce code quality before commits. - [Running github actions locally](./test_actions_locally.md): Instructions for testing GitHub Actions workflows on your local machine to streamline development and debugging. -These sections are primarily aimed at `execution-spec-tests` maintainers: +These sections are primarily aimed at `execution-specs` maintainers: - [Generating documentation](./docs.md): Steps to create and build documentation and manage documentation versions. -- [Managing dependencies and packaging](./deps_and_packaging.md): How to update EEST dependencies. +- [Managing dependencies and packaging](./deps_and_packaging.md): How to manage the specs and testing package dependencies. diff --git a/docs/dev/precommit.md b/docs/dev/precommit.md deleted file mode 100644 index 3b8af55539d..00000000000 --- a/docs/dev/precommit.md +++ /dev/null @@ -1,16 +0,0 @@ -# Enabling Pre-Commit Checks - -There's a [pre-commit](https://pre-commit.com/) config file available in the repository root (`.pre-commit-config.yaml`) that can be used to enable automatic checks upon commit - the commit will not go through if the checks don't pass. - -To enable pre-commit, the following must be run once: - -```console -uvx pre-commit install -``` - -!!! note "Bypassing pre-commit checks" - Enabling of pre-commit checks is not mandatory (it cannot be enforced) and even if it is enabled, it can always be bypassed with: - - ```console - git commit --no-verify - ``` diff --git a/docs/dev/releasing.md b/docs/dev/releasing.md new file mode 100644 index 00000000000..137cf9dc702 --- /dev/null +++ b/docs/dev/releasing.md @@ -0,0 +1,82 @@ +# Releasing + +This is the maintainer runbook for cutting an EELS release. For the +contributor-facing explanation of the versioning scheme, see +[Spec Releases](../specs/spec_releases.md). + +## Overview + +1. Choose a version number (see + [Spec Releases](../specs/spec_releases.md)). +2. Update the version in source code. +3. Create a pull request. +4. Wait for it to get merged. +5. Create a tag. +6. Create a GitHub release. +7. Publish to PyPI. + +## Updating the version in source code + +The version number is set in `src/ethereum/__init__.py`. Change it +there. For example: + +```patch +diff --git a/src/ethereum/__init__.py b/src/ethereum/__init__.py +index 252f2f317..8cdd89a55 100644 +--- a/src/ethereum/__init__.py ++++ b/src/ethereum/__init__.py +@@ -18,7 +18,7 @@ possible, to aid in defining the behavior of Ethereum clients. + """ + import sys + +-__version__ = "1.15.0" ++__version__ = "1.16.0rc1" + + # + # Ensure we can reach 1024 frames of recursion +``` + +## Creating the pull request + +The usual: `git checkout -b release-vX.Y.Z`, `git commit -a`, and +`git push`. + +## Creating the tag + +> [!WARNING] +> Do not create the tag from the `HEAD` branch of the pull request. +> +> GitHub can rewrite commits when merging pull requests, and tagging the +> original commit will make the git history messier than necessary. + +The tag name should be the letter `v` followed by the version number +(e.g. `1.15.0rc5.post3` becomes `v1.15.0rc5.post3`). + +To create and push the tag: + +```bash +git checkout master # Replace `master` with the pull request's base branch. +git pull +git tag -a -s v1.15.0 # Replace `v1.15.0` with the tag name from earlier. +git push origin v1.15.0 # Replace the tag name here too. +``` + +> [!IMPORTANT] +> If `git tag` complains about a missing GPG/PGP key, follow +> [this guide][keygen] to generate one. It's best to add the key to +> your GitHub account as well. + +[keygen]: https://docs.github.com/en/authentication/managing-commit-signature-verification/generating-a-new-gpg-key + +## Creating the GitHub release + +Go to the [release page][release], choose the newly created tag, and +generate release notes. + +[release]: https://github.com/ethereum/execution-specs/releases/new + +## Publishing to PyPI + +See the [Python Packaging User Guide][ppug]. + +[ppug]: https://packaging.python.org/en/latest/tutorials/packaging-projects/#generating-distribution-archives diff --git a/docs/dev/test_actions_locally.md b/docs/dev/test_actions_locally.md index 4653c9b9392..6a9e890cf02 100644 --- a/docs/dev/test_actions_locally.md +++ b/docs/dev/test_actions_locally.md @@ -44,22 +44,18 @@ will output something similar to: ```bash INFO[0000] Using docker host 'unix:///var/run/docker.sock', and daemon socket 'unix:///var/run/docker.sock' -Stage Job ID Job name Workflow name Workflow file Events -0 evmone-coverage-diff evmone-coverage-diff Evmone Coverage Report coverage.yaml pull_request -0 deploy deploy Deploy Docs Main docs_main.yaml push -0 deploy deploy Deploy Docs Tags docs_tags.yaml push -0 features features Build and Package Fixtures fixtures.yaml push,workflow_dispatch -0 feature-names feature-names Build and Package Fixtures for a feature fixtures_feature.yaml push,workflow_dispatch -0 lint Lint python sources with ruff Tox tox_verify.yaml push,pull_request,workflow_dispatch -0 typecheck Typecheck python sources with mypy Tox tox_verify.yaml push,pull_request,workflow_dispatch -0 spellcheck Spellcheck sources with pyspelling Tox tox_verify.yaml push,pull_request,workflow_dispatch -0 markdownlint Lint markdown files with markdownlint Tox tox_verify.yaml push,pull_request,workflow_dispatch -0 mkdocs Build html documentation with mkdocs Tox tox_verify.yaml push,pull_request,workflow_dispatch -0 pytest_framework Run unit tests, ${{ matrix.os }}, ${{ matrix.python }} Tox tox_verify.yaml push,pull_request,workflow_dispatch -0 tests_deployed Fill tests, deployed, ${{ matrix.os }}, ${{ matrix.python }} Tox tox_verify.yaml push,pull_request,workflow_dispatch -1 build build Build and Package Fixtures fixtures.yaml push,workflow_dispatch -1 build build Build and Package Fixtures for a feature fixtures_feature.yaml push,workflow_dispatch -2 release release Build and Package Fixtures fixtures.yaml push,workflow_dispatch +Stage Job ID Job name Workflow name Workflow file Events +0 evmone-coverage-diff evmone-coverage-diff Evmone Coverage Report coverage.yaml pull_request +0 deploy deploy Deploy Docs Main docs_main.yaml push +0 deploy deploy Deploy Docs Tags docs_tags.yaml push +0 features features Build and Package Fixtures fixtures.yaml push,workflow_dispatch +0 feature-names feature-names Build and Package Fixtures for a feature fixtures_feature.yaml push,workflow_dispatch +0 static Run static checks Test test.yaml push,pull_request,workflow_dispatch +0 py3 Fill tests (Python) Test test.yaml push,pull_request,workflow_dispatch +0 docs Build html documentation with mkdocs Test Docs test-docs.yaml push,pull_request,workflow_dispatch +1 build build Build and Package Fixtures fixtures.yaml push,workflow_dispatch +1 build build Build and Package Fixtures for a feature fixtures_feature.yaml push,workflow_dispatch +2 release release Build and Package Fixtures fixtures.yaml push,workflow_dispatch 2 release release Build and Package Fixtures for a feature fixtures_feature.yaml push,workflow_dispatch ``` @@ -77,7 +73,7 @@ DEFAULT_PYTHON_VERSION=3.12 and use the `--var-file` option to specify the file: ```bash -gh act --workflows .github/workflows/tox_verify.yaml -s GITHUB_TOKEN=$(gh auth token) --var-file=gh_vars.txt -j lint +gh act --workflows .github/workflows/test.yaml -s GITHUB_TOKEN=$(gh auth token) --var-file=gh_vars.txt -j static ``` ### Running Workflows that use a Matrix Strategy @@ -85,7 +81,7 @@ gh act --workflows .github/workflows/tox_verify.yaml -s GITHUB_TOKEN=$(gh auth t This is optional, recent versions will automatically detect the matrix strategy and run supported values. To run a specific matrix value, use the `--matrix` option: ```bash -gh act --workflows .github/workflows/tox_verify.yaml -s GITHUB_TOKEN=$(gh auth token) --matrix python:3.12 -j pytest_framework +gh act --workflows .github/workflows/test.yaml -s GITHUB_TOKEN=$(gh auth token) --matrix python:3.12 -j test-tests ``` ### Running Release Workflows @@ -115,3 +111,13 @@ It's possible to specify the Docker image used by the `act` tool for a specific ``` This can be added to any `gh act` command. + +## Debugging CI Failures + +The pytest-based CLIs (`fill`, `execute`, ...) print the resolved `pytest ...` invocation at startup whenever the `CI` environment variable is set: + +```text +Executing: pytest ... +``` + +When run under Github Actions, the same command is also appended to the job summary (via `$GITHUB_STEP_SUMMARY`), so it can be copied directly from the "Summary" panel of the failing run without scrolling through the log. diff --git a/docs/filling_tests/debugging_t8n_tools.md b/docs/filling_tests/debugging_t8n_tools.md index 76960b45909..2352bec641d 100644 --- a/docs/filling_tests/debugging_t8n_tools.md +++ b/docs/filling_tests/debugging_t8n_tools.md @@ -1,6 +1,6 @@ # Debugging Transition Tools -There are two flags that can help debugging `t8n` tools or the execution-spec-tests framework: +There are two flags that can help debugging `t8n` tools or the execution-testing framework: 1. `--evm-dump-dir`: Write debug information from `t8n` tool calls to the specified directory. 2. `--traces`: Collect traces of the execution from the transition tool. diff --git a/docs/filling_tests/fill_stateful.md b/docs/filling_tests/fill_stateful.md new file mode 100644 index 00000000000..ada7c290d13 --- /dev/null +++ b/docs/filling_tests/fill_stateful.md @@ -0,0 +1,246 @@ +# Filling Stateful Benchmark Fixtures + +The `fill-stateful` command produces `BlockchainEngineStatefulFixture` JSON for benchmark tests by driving block construction on a live EL client via `testing_buildBlockV1` against a pre-loaded network snapshot. The fixtures are replayed by [`benchmarkoor`](https://github.com/ethpandaops/benchmarkoor) against the same snapshot on any EL client. This replaces the gas-benchmarks MITMProxy approach. + +!!! note "When to use `fill-stateful`" + Use the standard `fill` for t8n-based fixture generation. Use `fill-stateful` to run benchmarks in a stateful environment (e.g., perfnet, Kurtosis, or other snapshots) to observe how state size affects performance. Any test can be run using this command, but some benchmarks – like the ones under `tests/benchmark/stateful/` – only produce meaningful results in such environments. + +`fill-stateful` does not manage datadirs — it expects the target client to already be running with the snapshot mounted. Snapshot management (overlayfs / ZFS / copy) is `benchmarkoor`'s responsibility on the replay side. + +## Prerequisites + +The target client must expose: + +- `testing` (`testing_buildBlockV1`) — block construction with explicit transaction ordering. +- `engine` — `engine_newPayloadVX`, `engine_forkchoiceUpdatedVX`. +- `eth`, `debug` — chain queries and `debug_setHead` for between-test rewind. +- `web3` (optional) — `web3_clientVersion` is recorded into the fixture's `_info.filling-transition-tool` for traceability. + +The production-ready filler is `ethpandaops/geth:master`. + +## End-to-end flow + +### 1. Bootstrap a snapshot + +For local development, a kurtosis enclave produces a synthetic snapshot: + +```yaml +# /tmp/fillst-kurtosis-args.yaml +participants: + - el_type: geth + el_image: ethpandaops/geth:master + el_extra_params: + - "--http.api=admin,debug,eth,miner,net,txpool,web3,testing,engine" + - "--miner.gaslimit=1000000000000" + cl_type: lodestar +network_params: + preset: minimal + genesis_delay: 30 + fulu_fork_epoch: 0 + gas_limit: 1000000000000 +ethereum_genesis_generator_params: + image: "ethpandaops/ethereum-genesis-generator:5.3.5" +``` + +```bash +kurtosis run --enclave fillst /path/to/ethereum-package \ + --args-file /tmp/fillst-kurtosis-args.yaml +``` + +Drain a few blocks, stop CL then EL cleanly, extract the datadir + genesis bundle: + +```bash +docker stop -t 30 vc-1-geth-lodestar--... cl-1-lodestar-geth--... +docker stop -t 60 el-1-geth-lodestar--... +docker cp el-1-geth-lodestar--...:/data/geth/execution-data /tmp/multi-snap/geth/ +rm -f /tmp/multi-snap/geth/execution-data/geth/{LOCK,nodes/LOCK,chaindata/LOCK} +kurtosis files download fillst el_cl_genesis_data /tmp/multi-snap/genesis +kurtosis files download fillst jwt_file /tmp/fillst-out/jwt +``` + +For production benchmarking, use a perfnet / bloatnet snapshot instead. + +### 2. Start a standalone client on a copy of the snapshot + +Keep the original snapshot pristine so `benchmarkoor` can reuse it on the replay side: + +```bash +cp -a /tmp/multi-snap/geth/execution-data /tmp/multi-snap/geth-fillcopy + +docker run -d --name geth-fillcopy \ + -p 18545:8545 -p 18551:8551 \ + -v /tmp/multi-snap/geth-fillcopy:/datadir \ + -v /tmp/multi-snap/genesis:/genesis:ro \ + -v /tmp/fillst-out/jwt:/jwt:ro \ + ethpandaops/geth:master \ + --datadir=/datadir --override.genesis=/genesis/genesis.json \ + --http --http.addr=0.0.0.0 --http.port=8545 \ + --http.api=admin,debug,eth,miner,net,txpool,web3,testing,engine \ + --authrpc.port=8551 --authrpc.addr=0.0.0.0 \ + --authrpc.jwtsecret=/jwt/jwtsecret \ + --syncmode=full --gcmode=archive \ + --miner.gaslimit=1000000000000 \ + --nodiscover --maxpeers=0 +``` + +### 3. Fill + +```bash +uv run fill-stateful \ + --clean \ + --rpc-endpoint=http://127.0.0.1:18545 \ + --engine-endpoint=http://127.0.0.1:18551 \ + --engine-jwt-secret-file=/tmp/fillst-out/jwt/jwtsecret \ + --fork=Osaka \ + --output=/tmp/fillst-out/fixtures \ + --snapshot-block=0x<32-byte-hash> \ + --gas-benchmark-values=10,30 \ + tests/benchmark/stateful/bloatnet/test_transient_storage.py +``` + +### 4. Replay + +Point `benchmarkoor`'s `datadirs.geth.source_dir` at the pristine snapshot (`/tmp/multi-snap/geth/execution-data`) — never at the fillcopy — and `tests.source.eest_fixtures.local_fixtures_dir` at the fill output. See the [benchmarkoor docs](https://github.com/ethpandaops/benchmarkoor) for the full config shape. + +## CLI options + +Required: + +- `--engine-jwt-secret-file PATH` — JWT secret for engine API auth. +- `--fork NAME` — fork to fill against, e.g. `Osaka`. + +Optional: + +- `--rpc-endpoint URL` — default `http://localhost:8545`. +- `--engine-endpoint URL` — derived from `--rpc-endpoint` with port `8551`. +- `--chain-id INT` — auto-detected from the client. +- `--snapshot-block HASH_OR_NUMBER` — anchor to a specific block; accepts a 32-byte hash (recommended) or an integer block number (hex `0x...` or decimal). Defaults to the client's `latest`, recorded by hash. +- `--rpc-seed-key 0x<64hex>` — pin the seed account for reproducible fills. When omitted, a random key is generated and funded via CL withdrawal each session. +- `--address-stubs PATH` — JSON map of label → on-chain address (and optional pkey). Required by stub-dependent tests; see [Stub-dependent tests](#stub-dependent-tests) below. +- `--max-gas-per-test INT` — overrides the fork's `transaction_gas_limit_cap()` (EIP-7825). +- `--gas-benchmark-values 10,30,...` — gas budgets in millions to parametrize against. +- `--default-{gas-price,max-fee-per-gas,max-priority-fee-per-gas,max-fee-per-blob-gas}` — pin per-session fees; defaults bump live-query values by `1.5×`. +- `--output PATH` — default `./fixtures`. +- `--clean` — wipe the output dir before filling. + +## Output layout + +```text +/ +└── blockchain_tests_stateful_engine/ + ├── pre_run/ + │ └── .json # session bootstrap (factory deploy + seed funding) + └── for__at_M/ + └── / + └── .json # per-test setup + execution payloads +``` + +Each `pre_run/.json` (a `StatefulPreRunFixture`) is replayed once per `benchmarkoor` run. Per-test fixtures (`BlockchainEngineStatefulFixture`) reference their setup file by hash: a fixture with `startBlockHash = 0xabc...` is preceded by `pre_run/0xabc....json`. Each per-test fixture carries `snapshotBlockNumber`/`Hash`, `startBlockNumber`/`Hash`, `setupEngineNewPayloads`, `engineNewPayloads`, plus a `benchmarkGasUsed` field and the EL build in `_info.filling-transition-tool`. The hash-based filename leaves room for multiple pre-run files (e.g. different setup variants off one snapshot) without coordinating names. + +!!! warning "Snapshot anchoring" + `--snapshot-block` accepts a hash on purpose. Anchoring to `latest` works against a quiescent client, but a live reorg between session start and fixture write would silently re-anchor the fixture to a different block. The hash form rejects that. + +!!! warning "State pollution across fills" + Re-running `fill-stateful` against the same datadir progresses the chain past previous fills. Always start from a fresh copy of the snapshot. + +!!! note "Single-worker" + `fill-stateful` forces `-n 0` — pytest-xdist is not used; the chain advances sequentially. + +## Stub-dependent tests + +Some stateful tests (e.g. `test_single_opcode.py`, `test_multi_opcode.py`) target on-chain accounts the snapshot already contains. They reach them two ways: + +- `@pytest.mark.stub_parametrize("name", "prefix_")` — parametrize values pulled from `--address-stubs` matching `prefix_`. +- `pre.deploy_contract(stub="