Python free-threading support

[frankie567]

Now that free-threading (aka no-GIL) is a thing in Python 3.14, it could be interesting to explore if we can migrate to that. Not sure what kind of performance gain we could get, but it's worth to check.

The main blockers right now are libraries with C extensions that have not been adapted yet to this new mode.

Dependency blockers

1. sqlalchemy (ready, to release in 2.1): Implement support for free-threaded Python sqlalchemy/sqlalchemy#12881
   • but... asyncio features rely on greenlet, which doesn't seem to aim at supporting free-threading: Prevent greenlet from building on free-threaded Python python-greenlet/greenlet#423
2. asyncpg: Add Python 3.14 support, experimental subinterpreter/freethreading support MagicStack/asyncpg#1279
3. httptools (dependency of Uvicorn): Support for free-threaded Python MagicStack/httptools#118

Experiments

2025-10-18

Tried to run the API with:

• env PYTHON_GIL=0 to avoid greenlet warning
• Installed asyncpg from Add Python 3.14 support, experimental subinterpreter/freethreading support MagicStack/asyncpg#1279
• --http h11 option of Uvicorn to avoid 3

Server is starting, can serve a few requests, but then blocks and doesn't answer new requests. Probably a concurrency problem with greenlet.

2026-01-26

• Python 3.14.2
• Upgraded all dependencies to their latest (in particular, asyncpg and greenlet)
• env PYTHON_GIL=0 to avoid warnings with some libraries
• --http h11 option of Uvicorn to avoid 3

Server is starting, can serve requests, looks super fine.

Worker is also starting, can process a few tasks, but then crashes:

[2026-01-26 11:14:39,242] [PID 67689] [MainThread] [dramatiq.MainProcess] [CRITICAL] Worker with PID 67702 exited unexpectedly (code -10). Shutting down...

Running with env PYTHONFAULTHANDLER=1 to get the full thread stack trace:

Full crash stack trace

Fatal Python error: Bus error
<Cannot show all threads while the GIL is disabled>                                                        Stack (most recent call first):
  File "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/python3.14t/asyncio/selector_events.py", line 1071 in write
  File "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/asyncpg/connection.py", line 443 in _get_statement
  File "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/asyncpg/connection.py", line 657 in _prepare
  File "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/asyncpg/connection.py", line 638 in prepare
  File "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 773 in _prepare
  File "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 526 in _prepare_and_execute
  File "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196 in greenlet_spawn                                                                    File "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/sqlalchemy/ext/asyncio/session.py", line 449 in execute                                                                              File "/Users/fvoron/Development/polar/server/polar/kit/repository/base.py", line 71 in get_one_or_none     File "/Users/fvoron/Development/polar/server/polar/kit/repository/base.py", line 234 in get_by_id          File "/Users/fvoron/Development/polar/server/polar/subscription/tasks.py", line 55 in subscription_cycle   File "/Users/fvoron/Development/polar/server/polar/worker/__init__.py", line 90 in _wrapped_fn             File "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/dramatiq/asyncio.py", line 143 in wrapped_coro                                                                                       File "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/python3.14t/asyncio/events.py", line 94 in _run                                                                         File "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/python3.14t/asyncio/base_events.py", line 2046 in _run_once
  File "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/python3.14t/asyncio/base_events.py", line 677 in run_forever
  File "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/dramatiq/asyncio.py", line 92 in run
  File "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/sentry_sdk/integrations/threading.py", line 133 in _run_old_run_func                                                                 File "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/sentry_sdk/integrations/threading.py", line 140 in run                                                                               File "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/python3.14t/threading.py", line 1082 in _bootstrap_inner                                                                File "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/python3.14t/threading.py", line 1044 in _bootstrap

Current thread's C stack trace (most recent call first):                                                     Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at _Py_DumpStack+0x44 [0x1028fc2cc]                                                       Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at faulthandler_dump_c_stack+0x54 [0x10297cde4]                                           Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at faulthandler_fatal_error+0x168 [0x10297cc40]
  Binary file "/usr/lib/system/libsystem_platform.dylib", at _sigtramp+0x38 [0x1983a3744]
  Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at PyEval_SaveThread+0x74 [0x102165030]
  Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at PyEval_SaveThread+0x74 [0x102165030]
  Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at sock_call_ex+0xb0 [0x10294b4e4]
  Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at sock_send+0x74 [0x10294a564]
  Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at method_vectorcall_VARARGS.llvm.17970578127269467053+0x78 [0x1022f3db4]
  Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at _TAIL_CALL_CALL_NON_PY_GENERAL.llvm.2430637514300127925+0x108 [0x102342870]
  Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at _PyFunction_Vectorcall+0x2e4 [0x10228cb2c]
  Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at _PyObject_VectorcallTstate.llvm.16746352739323006973+0x54 [0x1020792b0]                Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at PyObject_VectorcallMethod+0x78 [0x102026610]                                           Binary file "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/asyncpg/protocol/protocol.cpython-314t-darwin.so", at __pyx_f_7asyncpg_8protocol_8protocol_12BaseProtocol__write+0x90 [0x108243800]                                                                                                      Binary file "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/asyncpg/protocol/protocol.cpython-314t-darwin.so", at __pyx_f_7asyncpg_8protocol_8protocol_12CoreProtocol__prepare_and_describe+0x394 [0x108238e68]                                                                                      Binary file "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/asyncpg/protocol/protocol.cpython-314t-darwin.so", at __pyx_gb_7asyncpg_8protocol_8protocol_12BaseProtocol_12generator+0x5d4 [0x10827ba20]                                                                                               Binary file "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/asyncpg/protocol/protocol.cpython-314t-darwin.so", at __Pyx_Coroutine_SendEx+0x78 [0x108212acc]                               Binary file "/Users/fvoron/Development/polar/server/.venv/lib/python3.14t/site-packages/asyncpg/protocol/protocol.cpython-314t-darwin.so", at __Pyx_Generator_Next+0x104 [0x108213d60]                                Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at _TAIL_CALL_SEND.llvm.2430637514300127925+0xb4 [0x1021b2250]                            Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at _PyEval_EvalFrameDefault+0xc0 [0x10215e5bc]
  Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at gen_send_ex2+0xbc [0x10215e2b0]                                                        Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at task_step_impl+0x174 [0x102903054]                                                     Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at task_step+0x38 [0x102902e84]                                                           Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at task_wakeup_lock_held+0x130 [0x1029040f8]                                              Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at task_wakeup+0x6c [0x102903fa8]                                                         Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at context_run+0x88 [0x102210fe8]                                                         Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at _TAIL_CALL_CALL_FUNCTION_EX.llvm.2430637514300127925+0x100 [0x1020aebe8]               Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at _PyFunction_Vectorcall+0x2e4 [0x10228cb2c]                                             Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at context_run+0x88 [0x102210fe8]                                                         Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at _TAIL_CALL_CALL.llvm.2430637514300127925+0x12c [0x10208a434]                           Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at _PyFunction_Vectorcall+0x2e4 [0x10228cb2c]                                             Binary file "/Users/fvoron/.local/share/uv/python/cpython-3.14.2+freethreaded-macos-aarch64-none/lib/libpython3.14t.dylib", at method_vectorcall.llvm.2242813384965064185+0x90 [0x1022f0108]                          <truncated rest of calls>
                                                                                                           Extension modules: greenlet._greenlet, zope.interface._zope_interface_coptimizations, gevent.libev.corecext, gevent._gevent_c_greenlet_primitives, gevent._gevent_c_hub_local, gevent._gevent_c_waiter, gevent._gevent_c_hub_primitives, gevent._gevent_c_ident, gevent._gevent_cgreenlet, gevent._gevent_c_abstract_linkable, gevent._gevent_cevent, _watchdog_fsevents, gevent._gevent_cqueue, gevent._gevent_c_semaphore, gevent._gevent_c_imap, gevent._gevent_clocal, psutil._psutil_osx, psutil._psutil_posix, _brotli, _cffi_backend, markupsafe._speedups, yaml._yaml, fontTools.misc.bezierTools, fontTools.varLib.iup, PIL._imaging, PIL._imagingcms, maxminddb.extension, psycopg2._psycopg, asyncpg.pgproto.pgproto, asyncpg.protocol.record, asyncpg.protocol.protocol (total: 31)

So there is apparently still stuff to solve on asyncpg side.

[IlluvatarEru]

What is needed to priorise this @birkjernstrom ?
There might not be any performance gain for polars specific opreations but there are systems that benefit massively from the removing the GIL and also use polars. Right now it is simply not compatible unfortunately. Please don't make us revert to pandas.

[frankie567]

@IlluvatarEru This is Polar, the billing platform and not Polars, the DataFrame engine.

[pieterbeulque]

Do you want to keep this open?

[frankie567]

Yes, please, I think it'll be useful someday :D