Add private Windows ETW thread spawn tracer prototype#221
Draft
ogrisel wants to merge 17 commits into
Draft
Conversation
ogrisel
commented
Jul 10, 2026
Introduce threadpoolctl._thread_tracer as an experimental private package with a ctypes-based WindowsThreadSpawnTracer that counts kernel Thread Start events for a target PID via the NT Kernel Logger. Includes pure-Python event payload parsing, mocked ETW lifecycle tests that run on all platforms, and keeps the tracer out of the public threadpoolctl API. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Replace mocked ETW lifecycle tests with subprocess integration checks that spawn real Python, OpenMP, and BLAS threads while the kernel tracer is attached. Keep lightweight payload parsing unit tests for all platforms. Add windows_etw_admin_available() and a short flush delay on tracer stop to improve reliability on CI runners. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Warm up BLAS/OpenMP pools in the parent process, then use threadpool_info (or get_threadpool_limits on older releases) to set the minimum expected ETW thread spawn count instead of hard-coded guesses. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Remove trivial availability helpers from the private API and inline the Windows/admin checks in integration test skipif predicates. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
cursor
Bot
force-pushed
the
cursor/windows-thread-tracer-88ad
branch
from
cb5a29c to
3a8b0ed
Compare
Move the private _thread_tracer prototype and its tests out of the threadpoolctl/ directory, which conflicted with threadpoolctl.py during flit install. Update imports for the modern tests layout and OpenMP helper module names. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Use correctly-sized EVENT_TRACE_LOGFILE nested structures so OpenTraceW does not corrupt adjacent memory. Route ETW callbacks through a module- level function keyed by UserContext, and pass ProcessTrace a TRACEHANDLE array with stable logger-name references. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Drop the threading.Barrier child synchronization that could break under load, derive native pool spawn expectations as team_size minus one, and capture child stderr to simplify future CI debugging. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
…ests _run_traced_child now only runs the subprocess and tracer, returning ThreadSpawnStats. Each integration test asserts the exact expected spawn count explicitly instead of checking a minimum threshold. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
CI shows Windows traced children emit a stable startup overhead on top of work-specific spawns. Measure that baseline once per module and assert total counts as startup + expected work spawns instead of bare totals. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Remove the hardcoded PYTHON_THREAD_WORK_SPAWN_COUNT constant. Each test asserts spawn counts relative to a traced noop child subprocess, with extra spawns defined by the test itself (thread count, OpenMP, or BLAS). Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Delay tracer attach until worker threads are already alive and enable include_existing_threads so ETW rundown is observable. Measure a noop baseline with the same attach timing and assert extra existing counts relative to that baseline. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
The delayed tracer attach must find the child still running; a bare pass body finishes before attach and yielded a zero existing-thread baseline. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
ogrisel
commented
Jul 10, 2026
Use WINFUNCTYPE unconditionally in the Windows-only ETW module instead of a cross-platform callback fallback. Restore the macOS and Linux flexiblas matrix entries in the CI workflow. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Defer WINFUNCTYPE and the ETW callback registration to win32 only so importing _thread_tracer succeeds on Linux and macOS CI jobs. Usage still raises ThreadTracerError when starting a trace off Windows. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Contributor
Author
|
Note: I plan to leave this PR in draft mode for now but the results of the integration tests run on the windows CI seem to indicate that it works as expected. Unfortunately, I don't think it's possible to do the same for macOS and Linux without relying on a helper compiled extension or on a helper system tracing tool that might require extra privileges. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context:
The purpose of this cursor-driven PR is to study the feasibility of using only ctypes to trace thread start events on Windows from a Python program.
Introduce threadpoolctl._thread_tracer as an experimental private package with a ctypes-based WindowsThreadSpawnTracer that counts kernel Thread Start events for a target PID via the NT Kernel Logger.
Includes pure-Python event payload parsing, mocked ETW lifecycle tests that run on all platforms, and keeps the tracer out of the public threadpoolctl API.
I do not have a local windows setup, so I will let the agent leverage the Windows CI of this repo to iterate.