Regenerate Cython outputs with 3.2.4 for CPython 3.14 support#69
Draft
Regenerate Cython outputs with 3.2.4 for CPython 3.14 support#69
Conversation
CPython 3.14 changed the signature of _PyLong_AsByteArray (added an endianness argument). The .c files checked into this repo were generated with Cython 3.0.11 and no longer compile against 3.14 headers: bufferedreader.c:15818: error: too few arguments to function '_PyLong_AsByteArray' Regenerating the .pyx -> .c outputs with Cython 3.2.4 emits code that handles the new signature. No .pyx changes, no runtime behavior change for Python <= 3.13. Free-threaded (cp314t) compatibility: the resulting extensions build and import on cp314t, but the runtime auto-re-enables the GIL for each .so because the modules do not yet declare Py_MOD_GIL_NOT_USED. That's a separate audit + opt-in change; functionality is preserved. Verified locally: python3.14t -m pip wheel . --no-deps # builds python3.14t -c "import proton_driver; ..." # imports, functional Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cython 3.1+ default-emits a compressed module string table that's decompressed at import time via `zlib` / `bz2` / `compression.zstd`. On minimal/embedded CPython builds that omit those stdlib modules, the regenerated extensions now fail to import — a regression relative to the pre-Cython-3.2 checked-in C. The generated C keeps an uncompressed `#else /* compression: none */` fallback guarded by the C macro CYTHON_COMPRESS_STRINGS (default 1). Pass -DCYTHON_COMPRESS_STRINGS=0 via each Extension's define_macros so the fallback branch is selected and no compression module is imported. Verified on CPython 3.12 by artificially blocking zlib/bz2/ compression.zstd before import: all four extensions load cleanly and no blocked module leaks into sys.modules. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Bump pypa/cibuildwheel from v2.21.3 to v3.4.1. v2.21.3 predates Python 3.14 support; v3.x adds cp314 across macOS/Windows/manylinux/ musllinux and cp314t on every Tier-1 platform, so without this pin the release job silently skipped 3.14 and would never produce free-threaded wheels. - Bump runner images (ubuntu-20.04 → ubuntu-22.04, windows-2019 → windows-2022, macos-12 → macos-13). ubuntu-20.04 and macos-12 are retired on GitHub Actions and would refuse to schedule. - Upgrade actions/upload-artifact and actions/download-artifact from v3 (deprecated) to v4. v4 needs unique artifact names per matrix job, so uploads are now scoped by matrix.os and the publish job merges them via pattern + merge-multiple. - Add a cibuildwheel test-command that installs each built wheel into a fresh env and imports every compiled extension. Catches runtime ABI or stdlib-dependency regressions at release time instead of on user machines (the recent Cython 3.2 zlib/bz2/zstd import-time dependency would have been caught here). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test.yml: add "3.14" to the main driver test matrix. - test.yml: add a dedicated smoke-ft job that builds the Cython extensions under free-threaded 3.14 (setup-python "3.14t") and asserts every compiled extension imports. The full driver test matrix is not run under 3.14t on purpose: the generated module init still emits Py_MOD_GIL_USED, so a real FT run would just prove the driver falls back to GIL mode. Running a real free-threading audit is out of scope for this PR; the smoke job is there to lock in "still importable" and catch regressions from future Cython regens. - README.rst: include 3.14 (+ note 3.14t) in the supported list. - setup.py: add the 3.14 trove classifier. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The earlier smoke checks only did plain imports on runners that already
ship zlib / bz2 / compression.zstd / lzma, so a future regression to
Cython's default compressed string tables (CYTHON_COMPRESS_STRINGS != 0
in setup.py) would have slipped through silently.
Move the actual invariant check into tests/smoke_no_compression.py:
install a sys.meta_path finder that raises ImportError for the four
stdlib compression modules, then import proton_driver and every
compiled extension. Verified locally to round-trip:
- setup.py CYTHON_COMPRESS_STRINGS=1 (the reviewer-flagged
regression): smoke rc=1, error "Failed to import 'zlib.decompress'
- cannot initialise module strings. String compression was
configured with the C macro 'CYTHON_COMPRESS_STRINGS=1'."
- setup.py CYTHON_COMPRESS_STRINGS=0 (current): smoke rc=0,
"proton_driver imports cleanly on Python 3.x with {zlib, bz2,
compression.zstd, lzma} blocked".
Wire it into both invocation paths:
- pyproject.toml test-command runs the script on every built wheel
inside cibuildwheel, against the freshly-installed wheel in a
clean env.
- .github/workflows/test.yml smoke-ft job installs the driver into
site-packages (pip install .) under free-threaded 3.14t and runs
the same script, so the resolution is unambiguous regardless of
cwd.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The previous wording read "3.14 (including the free-threaded 3.14t build)", which implies the driver runs without the GIL on 3.14t. It doesn't — the generated module init still calls PyUnstable_Module_SetGIL(..., Py_MOD_GIL_USED), so importing the driver into a Py_GIL_DISABLED interpreter flips that interpreter back to GIL-enabled at import time. Describe the posture accurately: extensions build and import cleanly on cp314t; the driver re-enables the GIL; a real Py_MOD_GIL_NOT_USED audit is a separate track. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CPython 3.14 changed the signature of _PyLong_AsByteArray (added an endianness argument). The .c files checked into this repo were generated with Cython 3.0.11 and no longer compile against 3.14 headers:
bufferedreader.c:15818: error: too few arguments to function
'_PyLong_AsByteArray'
Regenerating the .pyx -> .c outputs with Cython 3.2.4 emits code that handles the new signature. No .pyx changes, no runtime behavior change for Python <= 3.13.
Free-threaded (cp314t) compatibility: the resulting extensions build and import on cp314t, but the runtime auto-re-enables the GIL for each .so because the modules do not yet declare Py_MOD_GIL_NOT_USED. That's a separate audit + opt-in change; functionality is preserved.
Verified locally:
python3.14t -m pip wheel . --no-deps # builds
python3.14t -c "import proton_driver; ..." # imports, functional