docs(python-udf): CPython 3.14 runtime, init + flush hooks, package install grammar, and UDF runtime gates

[yokofly]

What

Documents the Python UDF and Python external stream features that shipped without docs. The bulk is Timeplus Enterprise 3.3.1 — which upgraded the embedded interpreter from CPython 3.10 to 3.14 free-threaded (cp314t) and stopped bundling third-party packages — plus two older features (3.1.2, 3.2.1) that were never written up and are backfilled here.

Four commits, each independently reviewable:

Commit	Feature	Shipped in
83048e8	CPython 3.14 free-threaded runtime, removal of bundled packages	3.3.1
c82bdca	Python UDF init hooks + named collections	3.3.1
0cfeed5	Python external stream sink flush hook	3.3.1
e4afca6	Package install grammar; UDF runtime gates	3.1.2 / 3.2.1

---

1. CPython 3.14 free-threaded runtime (83048e8)

The docs still described the 3.10 environment and listed ~40 bundled packages that no longer exist.

docs/py-udf.md

• Built-in Libraries — replaced the ~40-entry JSON list (numpy, requests, openai, pydantic, proton-driver, timeplus-neutrino, …) with the clean environment it actually is, plus a warning that any UDF importing one of the old bundled packages now fails with ModuleNotFoundError.
• New: python_requirements — the declarative S3 requirements.txt package sync (poll interval, credential chain, pip index overrides, install-only reconcile semantics). This is the path for ephemeral compute nodes, where a local SYSTEM INSTALL does not survive a reschedule.
• New: Concurrency and the free-threaded runtime — the GIL is genuinely gone, so it no longer accidentally protects Python state. Module-level state in the UDF body is per-query and safe; module-level state in an imported helper module is shared through sys.modules across threads and races. Includes the measured loss (8 concurrent queries × 5000 increments finished near 18,000/40,000 unprotected, exactly 40,000 with a threading.Lock) and a locking example.
• New: Upgrading to the Python 3.14 runtime — replace bin/ and lib/ (the startup ABI guard rejects a GIL runtime), reinstall the packages your UDFs import, check for cp314t wheels, audit shared mutable state, and roll back via a data-folder copy.
• Corrected the runtime version in the intro, the SQL notes and the Limitations list.

docs/sql-system-python-packages.md

• Corrected the runtime version and added a "clean environment since 3.3.1" note.
• Added cp314t wheel-compatibility and install-durability caveats.
• Added a python_requirements section as the declarative alternative, cross-linked to py-udf.

docs/sql-create-function.md

• Corrected the runtime version in the Python UDF intro.

docs/named-collection.md

• Listed Python UDF initialization parameters under Supported Usage, and added a worked example.

---

2. Python UDF init hooks + named collections (c82bdca)

Init hooks (init_function_name / init_function_parameters / named_collection, ceaea40d939d, #12198) shipped in the same v3.3.1 release but were documented nowhere — sql-create-function.md showed a bare SETTINGS ... with no list of what is accepted.

• py-udf.md — new Initialization hook section: the hook runs once at module load, before the first call and before a UDAF class is constructed; parameters are passed as a single string (JSON by convention); named_collection reads the collection's init_function_parameters key so the value stays out of SHOW CREATE FUNCTION, unlike inline init_function_parameters. Documents the rules — the two parameter sources are mutually exclusive, both require init_function_name, creation needs the NAMED COLLECTION privilege (and returns ACCESS_DENIED whether or not the collection exists, so it does not leak which collections are defined), a collection without the key means a zero-arg call, values resolve at module-load time so rotating a collection does not reach running materialized views, and a failed hook discards the module so a later call retries.
• sql-create-function.md — a settings table for the three Python-only settings, plus an example.

---

3. Python external stream sink flush hook (0cfeed5)

flush_function_name shipped in v3.3.1 (4a7f2b6a, #12183) alongside the CPython migration, and was documented nowhere. Python external streams had a documented ctor (init) and dtor (deinit), but nothing about flushing — so a batching sink had no discoverable way to avoid losing buffered rows.

• shared/python-external-stream.md — added flush_fn to the DDL skeleton and flush_function_name to the SETTINGS block, a settings bullet, and a new lifecycle step. The lifecycle sequence previously went straight from the entry function to deinit.
• shared/python-external-stream-write.md — new Flushing buffered writes section with a batching-sink example (init opens the buffer → write collects → flush posts → deinit clears), plus the contract: zero-arg, return value ignored, sink-only (never called on a read), invoked on every checkpoint and once before deinit on close, and a missing function name fails the INSERT with UDF_INTERNAL_ERROR at runtime — unlike init_function_parameters, it is not validated at CREATE time.
• sql-create-external-stream.md — added the setting to the Python syntax block with its own version note, since the page's blanket "3.2.2+" line does not hold for it.

---

4. Package install grammar + UDF runtime gates (e4afca6)

Two features that shipped before 3.3.1 and were never documented.

SYSTEM INSTALL PYTHON PACKAGE grammar — v3.1.2 (aa38eeb0, #11494)

Only the 'pkg' and 'pkg' 'version' forms were documented, so the REQUIREMENTS / INDEX_URL / EXTRA_INDEX_URL clauses looked nonexistent. index_url appeared only as a YAML comment inside the python_requirements block, which made it look like a config-file-only capability.

• sql-system-python-packages.md — new Installing from requirements text and Private package indexes sections: one package spec per line, blank/# lines skipped, pip options inside the text rejected (-r nested.txt, --index-url …) with a pointer to the clauses, the 1024-line cap, REQUIREMENTS cannot be combined with a version literal, http(s)-only URL validation that runs before any node invokes pip, and the non-obvious bit — a REQUIREMENTS batch is a single row in system.python_package_tasks under package_name = 'requirements.txt', not one row per package. Plus a compatibility note: these statements are dispatched with a newer cluster request format, so every node must be on 3.1.2+.
• py-udf.md — the two most useful forms inline, pointing at the reference page.

UDF runtime gates — v3.2.1 (fb17e0b9, #11752)

• py-udf.md — new Turning the runtime off section for enable_python_udf: applied by SYSTEM RELOAD CONFIG with no restart, exact error text, enforced on the REST endpoint too, and the two things that trip people up — it gates creation only (existing UDFs keep running, package management is unaffected), and it is read from each node's own config rather than replicated cluster-wide.
• server_config.md — enable_python_udf and enable_javascript_udf as config items, so they are findable from the config reference.
• js-udf.md — one line for enable_javascript_udf, since the flag is symmetric.

One deliberate divergence from the source: programs/server/config.yaml describes these flags as rejecting creation and execution, but ensureUDFIsEnabled is only reached from the two creation paths (InterpreterCreateFunctionQuery, UDFHandler), which is also all 99145_udf_runtime_enable_flags.sh asserts. The docs say creation-only. The server-side comment looks like the thing that needs fixing.

---

Verification

• yarn build passes with no broken-link or broken-anchor warnings.
• Init hooks verified against 99150_python_udf_init_named_collection.sh and 99121_python_udf_init_named_collection_acl.sh.
• Flush hook verified against 99177_external_python_stream_flush.sql, src/CPython/PythonModuleSession.cpp and src/Storages/ExternalStream/Python/PythonSink.cpp.
• Install grammar verified against 99124_python_package_requirements_syntax.sh, src/Parsers/ParserSystemQuery.cpp and src/CPython/PythonPackage.cpp.
• Runtime gates verified against 99145_udf_runtime_enable_flags.sh and src/Functions/UserDefined/UserDefinedFunctionFactory.cpp.
• Runtime details verified against the source: contrib/cpython at 3.14.6, --enable-python-free-threaded in release_build.yml, the pip+truststore-only dist-packages staging, PythonRequirementsReconciler and programs/server/config.yaml.

Not included

• No enterprise-v3.3.md release-notes page — the repo currently stops at enterprise-v3.2.md, and writing 3.3 release notes needs the full changelog. Nothing here links to /enterprise-v3.3, so the build stays clean. Worth a follow-up.
• The python() / python_query() table function (TableFunctionPythonQuery) is registered in the engine but has no test coverage, and the only related test has its RETURNS TABLE UDF case commented out as serverError UNSUPPORTED. It looks unfinished, so it is left undocumented rather than guessed at.

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for elastic-elion-a958b6 ready!

Name	Link
🔨 Latest commit	e4afca6
🔍 Latest deploy log	https://app.netlify.com/projects/elastic-elion-a958b6/deploys/6a7400dcf90e7600082b2382
😎 Deploy Preview	https://deploy-preview-664--elastic-elion-a958b6.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.