diff --git a/peps/pep-0803.rst b/peps/pep-0803.rst index 292737a1733..d8641170131 100644 --- a/peps/pep-0803.rst +++ b/peps/pep-0803.rst @@ -1,6 +1,6 @@ PEP: 803 Title: "abi3t": Stable ABI for Free-Threaded Builds -Author: Petr Viktorin +Author: Petr Viktorin , Nathan Goldbaum Discussions-To: https://discuss.python.org/t/106181 Status: Draft Type: Standards Track @@ -81,6 +81,172 @@ This PEP proposes additional API limitations, as required to be compatible with both GIL-enabled and free-threaded builds of CPython. +Ecosystem maintainers want decreased maintenance burden +------------------------------------------------------- + +A major advantage of the limited API and stable ABI wheels is that new +Python versions are supported on the day of release. Without stable ABI +wheels, maintainers are left with a choice between closely following CPython +releases and producing wheels during CPython beta periods or dealing with +inevitable user requests for support for new CPython versions. + + +Cryptography +^^^^^^^^^^^^ + +The cryptography project shipped 48 wheel files with their `most recent release +`_. Cryptography is +somewhat unusual in that they ship 14 wheels each for both the ``cp38`` and +``cp311`` stable ABIs to enable optimizations available in newer limited API +versions. They also ship 14 additional ``cp314t`` wheels and 6 wheels for +pypy. If there is no free-threaded stable ABI, then with Python 3.15, +cryptography will be using roughly the same amount of space on PyPI to support +two versions of the free-threaded build as *all* non-EOL versions of the +GIL-enabled build. + +Cryptography maintainer Alex Gaynor `expressed a desire +`_ +on Discourse for a free-threaded stable ABI: + + Just to state this explicitly from the PyCA maintainers perspective, as + long as we have O(1) builds, that’s ok. What we can’t/won’t do is O(n) + where we need new builds for every Python release. + +When one of the PEP authors asked Alex in the ``#pyca`` Libera IRC channel for +his current opinion, he said: + + One other thing I'll note that's *really* valuable about ``abi3`` is that it + means our old wheels keep working for new Python versions. If we have + per-Python release wheels, we have to do a bunch of work at various points + in the python release cycle (including potentially backport releases to add + new wheels, if we're not otherwise planning a release at that time). + + As maintainers, we *really* like to structure our work to avoid being "on + the clock" like that. + + +moocore +^^^^^^^ + +The `moocore project ships `_ +seven ``abi3`` wheels. When the topic of adding support for +the free-threaded build `came up on the moocore issue tracker +`_, maintainer Manuel +López-Ibáñez let the person reporting the issue know that: + + I don't want to build for 3.14 free-threading unless you really need it. + +Later, after discovering the tracking issue for supporting the limited API on the +free-threaded build, he commented: + + By the way, python/cpython#111506 is about extending the stable ABI to + support free-threaded Python. If they do that, then the builds of moocore + will work in both classical and free-threaded Python versions, without + needing to build new wheels for each Python version. + + [...] + + I will revisit this again once Python 3.15 is released. Hopefully the ABI + will be stable (or even better, free-threading will be the default). + +Pydantic +^^^^^^^^ + +Pydantic maintainer David Hewitt `observed +`_: + + Pydantic distributes wheels for a native core built using Rust & PyO3. The + latest release of `pydantic-core distributed + `_ 112 wheels and + this number is set to grow as more environments are to be added + (e.g. Android, iOS, wasm). Pydantic has historically not distributed using + the stable ABI because the feature set was too immature. Much Pydantic + functionality interacts with Python objects via the C API in hot loops so + performance is key concern. As the stable ABI matures it will be ideal for + Pydantic to switch tier 2 platforms to the stable ABI (and perhaps + eventually tier 1 platforms too), which will significantly reduce the + number of wheels to build, test, and distribute. + + I would like to highlight that if free-threading does not adopt a stable + ABI, all the benefits above will be lost when free-threading becomes the + default and only build (which seems the expected long-term plan). + + +SciPy +^^^^^ + +The SciPy project uploaded 60 version-specific wheel files for its `last release +`_ to support four different +CPython versions. They do not upload wheels for PyPy. + +When asked about this proposal, SciPy steering council chair `Ralf Gommers said +`_: + + SciPy and a number of other projects in the scientific Python ecosystem + are quite interested in starting to use the Stable ABI, in particular to + reduce the maintenance load of `providing more wheels + `_. + With recent CPython, Cython and NumPy releases, this now seems + possible. The performance costs seem acceptable and small, although we'll + only really build confidence in that assessment after having made the + switch. + +By providing a new free-threaded stable ABI in Python 3.15, SciPy will not have +to consider the lack of a stable ABI on the free-threaded build as the project +considers switching to stable ABI wheels. + + +Bindings generators +------------------- + +Both moocore and cryptography use bindings generators to interface with the C +API. Cryptography uses PyO3 and CFFI while moocore uses only CFFI. Both CFFI and +PyO3 already handle all the details of abstracting over the C API to enable +different build configurations and there is no need to laboriously port +extension types to use APIs that are only available on one build or another. + +Using bindings generators will enable these projects to quickly adopt the new +stable ABI. Initial testing using the experimental ``_Py_OPAQUE_PYOBJECT`` flag +defined in CPython's ``main`` branch, indicates that PyO3, CFFI, and Cython will +all work with PEP 803 using packaging tools that have been patched to account +for an ``abi3.abi3t`` tag. + +PyO3 maintainer `David Hewitt said +`_ in +support of this proposal: + + PyO3 greatly benefits from having a stable ABI - one of the biggest + challenges for the framework is the need to abstract over a wide range of + Python / OS / CPU / environment combinations. We also offer the + possibility to build with the stable ABI for each of these environments + (targeting a given minimum Python version's stable ABI). The goal is + always that all functionality PyO3 offers works the same on all these + combinations (sometimes with a Python version floor to access certain + features). We currently support Python 3.7+. All functionality added to + the stable ABI is a very welcome promise that PyO3 will not need to + introduce further conditional code to support a given feature. In the long + run this makes it possible for PyO3 to simplify code paths as support for + older Python versions is dropped, helping to keep maintenance burden under + control. + +When asked to comment about this proposal, Cython +maintainer `David Woods said +`_: + + Cython doesn't have huge problems with the number of wheels we distribute + because ultimately it works fine as pure-Python. We do distribute wheels for + a few of the smaller platforms as Stable ABI wheels but that's more + "dogfooding" than because we actually need to. So I'm adding this in + anticipation that other people will find it useful rather than because I + will. + + I do remain slightly concerned that the performance trade-offs for this will + turn out to be too much for many Cython users (it's possible that the + trade-off may be different for other binding tools). That's not a huge + disaster since we're not getting rid of the regular compilation mode so + people are free to pick their own personal trade-offs. + + Rationale ========= @@ -500,10 +666,10 @@ built with a given interpreter and ``Py_LIMITED_API`` macro: | ``cp315-abi3`` | 3.15+ (GIL) | 3.15 | --- | continued | +-----------------------+-------------+--------------------+---------------------+-----------+ | ``cp315-abi3t`` | 3.15+ | --- | 3.15 | new | -+ +-------------+--------------------+---------------------+ + -| | 3.15+ (FT) | 3.15 | --- | | +-----------------------+-------------+--------------------+---------------------+-----------+ -| ``cp315-abi3.abi3t`` | 3.15+ | 3.15 | 3.15 | new | +| ``cp315-abi3.abi3t`` | 3.15+ (FT) | 3.15 | --- | new | ++ +-------------+--------------------+---------------------+ + +| | 3.15+ | 3.15 | 3.15 | | +-----------------------+-------------+--------------------+---------------------+-----------+