Skip to content

Conversation

@unnawut
Copy link
Collaborator

@unnawut unnawut commented Jan 30, 2026

🗒️ Description

Currently the API specs rely on implementers reading the python implementation and opiniately decide what is the spec and what is the leanSpec's own implementation flavor.

This PR creates a new test suite specifically for the API specs so that API conformance test can be run against a target server by running uv run apitest <server_url>:<server_port>. Implementers can see exactly what is expected by reading & running the tests, e.g. what url structure, what Accept headers, what should be encoded, etc. This way we can reduce the tiring chats and use these tests as reference for discussion and implementation instead.

Also as part of the standard uv run pytest, it will spawn its own server from leanSpec implementation and run it against these same API tests so these API tests that can be used with client implementations are guaranteed to be conformant with the spec (similar to how uv run fill ... also tests against the spec before producing the test vectors).

Example

Run ream (I'll try with other clients as well but want to get the base proposal reviewed & merged first) in ephemeral mode with:

cargo run --release -- --ephemeral lean_node \
    --network ephemery \
    --validator-registry-path ./bin/ream/assets/lean/validators.yaml

Then run the tests in leanSpec with:

uv run apitest http://localhost:5052

Results:

$ uv run apitest http://localhost:5052
=================================================================================== test session starts ===================================================================================
platform darwin -- Python 3.13.3, pytest-8.4.2, pluggy-1.6.0
rootdir: /Users/unnawut/Development/repo/leanSpec-2
configfile: packages/testing/src/framework/cli/pytest_ini_files/pytest-apitest.ini
plugins: anyio-4.12.1, xdist-3.8.0, cov-6.3.0, hypothesis-6.148.7
collected 14 items

tests/api_conformance/test_finalized_state.py .....
tests/api_conformance/test_health.py ...
tests/api_conformance/test_justified_checkpoint.py ....
tests/api_conformance/test_unknown_routes.py ..

=================================================================================== 14 passed in 0.45s ====================================================================================

🔗 Related Issues or PRs

N/A

✅ Checklist

  • Ran tox checks to avoid unnecessary CI fails:
    uvx tox
  • Considered adding appropriate tests for the changes.
  • Considered updating the online docs in the ./docs/ directory.

@unnawut unnawut marked this pull request as ready for review January 30, 2026 11:02
@unnawut unnawut added framework Scope: Changes to the testing framework tests Scope: Changes to the spec tests labels Jan 30, 2026
@unnawut unnawut added this to the pq-devnet-3 milestone Jan 30, 2026
Copy link
Collaborator

@tcoratger tcoratger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool this sounds good.

Could we integrate these tests to the CI so that they run on each PR and merge as well?

@unnawut
Copy link
Collaborator Author

unnawut commented Jan 30, 2026

Cool this sounds good.

Could we integrate these tests to the CI so that they run on each PR and merge as well?

yep it’s part of the default uv run pytest

@tcoratger tcoratger merged commit 6c9fcfe into leanEthereum:main Jan 30, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

framework Scope: Changes to the testing framework tests Scope: Changes to the spec tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants