[NOGIL] Add CI verification jobs for free-threaded Python 3.14t#2304
Conversation
- Add a Semaphore block running source package verification and
integration tests on CPython 3.14t (classic and consumer group
protocols). cimpl does not declare free-threading support yet, so
importing it re-enables the GIL: these jobs validate the 3.14t
toolchain and packaging until that declaration ships.
- Skip the CI-only orjson install on free-threaded interpreters: no
free-threaded orjson wheels exist and the source build would fail;
the stdlib JSON fallback path stays covered.
- Add a module-scoped autouse fixture (defined on free-threaded builds
only) that warns when the GIL is re-enabled around a test module.
Hard asserts are staged behind TODO FTS markers, to be enabled in the
same PR that declares Py_MOD_GIL_NOT_USED.
Interpreter detection follows the free-threading HOWTO:
https://docs.python.org/3/howto/free-threading-python.html
Ojasva Jain (ojasvajain)
requested review from
a team and
Matthew Seal (MSeal)
as code owners
|
🎉 All Contributor License Agreements have been signed. Ready to merge. |
On free-threaded (no-GIL) builds, the rules and json-fast extras'
compiled deps (tink, google-re2, grpcio; orjson) ship no free-threaded
wheels and fail to build from source, so:
- Add requirements-tests-install-nogil.txt, a variant of
requirements-tests-install.txt without those extras, and install it
from source-package-verification.sh when the interpreter is
free-threaded (detected via Py_GIL_DISABLED).
- Exclude the schema_registry test modules that import tink/celpy/orjson
at the top of the file from collection on free-threaded builds only;
on regular builds a missing dep stays a loud collection error rather
than a silent skip. Plain serdes coverage recovery is marked as a
TODO NOGIL follow-up.
|
Robert Yokota (rayokota)
left a comment
Robert Yokota (rayokota)
left a comment
There was a problem hiding this comment.
Thanks Ojasva Jain (@ojasvajain) , LGTM
Kaushik Raina (k-raina)
left a comment
Kaushik Raina (k-raina)
left a comment
There was a problem hiding this comment.
Thanks for raising PR. Provided first pass of comments
| - cp test-report.xml test-output | ||
| - test-results publish test-output | ||
| - artifact push workflow coverage.xml | ||
| - name: "Source package verification and Integration tests with Python 3.14t free-threaded (Linux x64)" |
There was a problem hiding this comment.
Any reason why we are adding new jobs and not changing current ones?
Adding new jobs and running old ones too means additional costs.
There was a problem hiding this comment.
The existing job runs on 3.11 which is a GIL based Python. We need this new job because there are some differences in how the GIL free Python behaves:
-
There are some dependencies that do not provide free threaded wheels. If such deps are imported in the code, the job will fail. The code author should then take steps to exclude it and possibly document the behaviour saying that XYZ functionality is not supported on free threaded Python builds.
-
There is a good difference in the underlying CPython 3.14t vs any GIL based CPython. It is important we ensure all our existing test cases pass. Also I have added a fixture that will ensure GIL does not get enabled during a test run. That fixture will be applicable only when we run test cases on 3.14t. Currently the fixture is commented out, but I have a TODO to enable it once I make the core changes.
| jobs: | ||
| - name: Build and Tests with 'classic' group protocol | ||
| commands: | ||
| - sem-version python 3.11 |
There was a problem hiding this comment.
Is using 3.11 version intentional?
There was a problem hiding this comment.
Yes. sem-version does not provide 3.14t. So I am setting 3.11 as the host Python interpretor and then I am using uv and venv to set the version of the virtual environment to 3.14t
uv venv _venv --python 3.14t && source _venv/bin/activate.
There was a problem hiding this comment.
Considering that this PR merged into feature beanch, why are we not making changes to requirements/requirements-tests-install.txt ?
There was a problem hiding this comment.
I have added a branch in source-package-verification.sh to use requirements-tests-install when running on GIL based Python. There are certain deps in requirements-tests-install that are not supported on GIL free python so I had to exclude them and create a new file called requirements-tests-install-nogil.
Kaushik Raina (k-raina)
left a comment
Kaushik Raina (k-raina)
left a comment
There was a problem hiding this comment.
Thanks for PR Ojasva Jain (@ojasvajain) . LGTM!
Ojasva Jain (ojasvajain)
merged commit 9412d74
into
dev_thread_free_support
3 participants
First step toward free-threading (no-GIL) support: run source-package verification and integration tests against CPython 3.14t on every commit.
requirements-tests-install-nogil.txt— test deps without the rules/json-fast extras, whose compiled deps (tink, google-re2, grpcio; orjson) ship no free-threaded wheels; selected by the verification script on free-threaded builds.Note: cimpl doesn't yet declare Py_MOD_GIL_NOT_USED, so these jobs run with the GIL re-enabled — they validate the 3.14t toolchain and packaging, not lock-free execution. cp314t wheels and the thread-safety work will be added in a separate PR.
What
Checklist
References
JIRA:
Test & Review
Open questions / Follow-ups