Add fallback implementation of PyCriticalSection_BeginMutex for Python 3.13t (#5981)

* Add failback implementation of `PyCriticalSection_BeginMutex` for Python 3.13t

* Add comment for Python version

* Use `_PyCriticalSection_BeginSlow`

* Add forward declaration

* Fix forward declaration

* Remove always true condition `defined(PY_VERSION_HEX)`

* Detect musllinux

* Add manylinux test

* Use direct mutex locking for Python 3.13t

`_PyCriticalSection_BeginSlow` is a private CPython function not exported
on Linux. For Python < 3.14.0rc1, use direct `mutex.lock()`/`mutex.unlock()`
instead of critical section APIs.

* Empty commit to trigger CI

* Empty commit to trigger CI

* Empty commit to trigger CI

* Run apt update before apt install

* Remove unnecessary prefix

* Add manylinux test with Python 3.13t

* Simplify pycritical_section with std::unique_lock fallback for Python < 3.14

* Fix potential deadlock in make_iterator_impl for Python 3.13t

Refactor pycritical_section into a unified class with internal version
checks instead of using a type alias fallback. Skip locking in
make_iterator_impl for Python < 3.14.0rc1 to avoid deadlock during
type registration, as pycritical_section cannot release the mutex
during Python callbacks without PyCriticalSection_BeginMutex.

* Add reference for xfail message

Author: XuehaiPan

.github/workflows/ci.yml

@@ -237,11 +237,23 @@ jobs:
 
 
   manylinux:
-    name: Manylinux on 🐍 3.14t
+    name: Manylinux on 🐍 ${{ matrix.python-version }} (${{ matrix.container }})
     if: github.event.pull_request.draft == false
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - container: quay.io/pypa/manylinux_2_28_x86_64:latest
+            python-version: '3.13t'
+          - container: quay.io/pypa/musllinux_1_2_x86_64:latest
+            python-version: '3.13t'
+          - container: quay.io/pypa/manylinux_2_28_x86_64:latest
+            python-version: '3.14t'
+          - container: quay.io/pypa/musllinux_1_2_x86_64:latest
+            python-version: '3.14t'
     runs-on: ubuntu-latest
     timeout-minutes: 40
-    container: quay.io/pypa/musllinux_1_2_x86_64:latest
+    container: ${{ matrix.container }}
     steps:
       - uses: actions/checkout@v6
         with:
@@ -254,7 +266,7 @@ jobs:
         run: uv tool install ninja
 
       - name: Configure via preset
-        run: cmake --preset venv -DPYBIND11_CREATE_WITH_UV=python3.14t
+        run: cmake --preset venv -DPYBIND11_CREATE_WITH_UV="${{ matrix.python-version }}"
 
       - name: Build C++11
         run: cmake --build --preset venv

.github/workflows/reusable-standard.yml

@@ -44,7 +44,7 @@ jobs:
 
       - name: Setup Boost (Linux)
         if: runner.os == 'Linux'
-        run: sudo apt-get install libboost-dev
+        run: sudo apt-get update && sudo apt-get install -y libboost-dev
 
       - name: Setup Boost (macOS)
         if: runner.os == 'macOS'

include/pybind11/detail/internals.h

@@ -241,13 +241,27 @@ class pymutex {
 
 class pycritical_section {
     pymutex &mutex;
+#    if PY_VERSION_HEX >= 0x030E00C1 // 3.14.0rc1
     PyCriticalSection cs;
+#    endif
 
 public:
     explicit pycritical_section(pymutex &m) : mutex(m) {
+        // PyCriticalSection_BeginMutex was added in Python 3.15.0a1 and backported to 3.14.0rc1
+#    if PY_VERSION_HEX >= 0x030E00C1 // 3.14.0rc1
         PyCriticalSection_BeginMutex(&cs, &mutex.mutex);
+#    else
+        // Fall back to direct mutex locking for older free-threaded Python versions
+        mutex.lock();
+#    endif
+    }
+    ~pycritical_section() {
+#    if PY_VERSION_HEX >= 0x030E00C1 // 3.14.0rc1
+        PyCriticalSection_End(&cs);
+#    else
+        mutex.unlock();
+#    endif
     }
-    ~pycritical_section() { PyCriticalSection_End(&cs); }
 
     // Non-copyable and non-movable to prevent double-unlock
     pycritical_section(const pycritical_section &) = delete;

include/pybind11/pybind11.h

@@ -3173,7 +3173,11 @@ iterator make_iterator_impl(Iterator first, Sentinel last, Extra &&...extra) {
     using state = detail::iterator_state<Access, Policy, Iterator, Sentinel, ValueType, Extra...>;
     // TODO: state captures only the types of Extra, not the values
 
+    // For Python < 3.14.0rc1, pycritical_section uses direct mutex locking (same as a unique
+    // lock), which may deadlock during type registration. See detail/internals.h for details.
+#if PY_VERSION_HEX >= 0x030E00C1 // 3.14.0rc1
     PYBIND11_LOCK_INTERNALS(get_internals());
+#endif
     if (!detail::get_type_info(typeid(state), false)) {
         class_<state>(handle(), "iterator", pybind11::module_local())
             .def(

tests/env.py

@@ -11,6 +11,18 @@
 WIN = sys.platform.startswith("win32") or sys.platform.startswith("cygwin")
 FREEBSD = sys.platform.startswith("freebsd")
 
+MUSLLINUX = False
+MANYLINUX = False
+if LINUX:
+
+    def _is_musl() -> bool:
+        libc, _ = platform.libc_ver()
+        return libc == "musl" or (libc != "glibc" and libc != "")
+
+    MUSLLINUX = _is_musl()
+    MANYLINUX = not MUSLLINUX
+    del _is_musl
+
 CPYTHON = platform.python_implementation() == "CPython"
 PYPY = platform.python_implementation() == "PyPy"
 GRAALPY = sys.implementation.name == "graalpy"

tests/test_multiple_interpreters.py

@@ -408,6 +408,11 @@ def test_import_in_subinterpreter_before_main():
 @pytest.mark.skipif(
     sys.platform.startswith("emscripten"), reason="Requires loadable modules"
 )
+@pytest.mark.xfail(
+    env.MUSLLINUX,
+    reason="Flaky on musllinux, see also: https://github.com/pybind/pybind11/pull/5972#discussion_r2755283335",
+    strict=False,
+)
 @pytest.mark.skipif(not CONCURRENT_INTERPRETERS_SUPPORT, reason="Requires 3.14.0b3+")
 def test_import_in_subinterpreter_concurrently():
     """Tests that importing a module in multiple subinterpreters concurrently works correctly"""