Prototype private Windows ETW thread spawn tracer#7
Draft
ogrisel wants to merge 17 commits into
Draft
Conversation
Introduce threadpoolctl._thread_tracer as an experimental private package with a ctypes-based WindowsThreadSpawnTracer that counts kernel Thread Start events for a target PID via the NT Kernel Logger. Includes pure-Python event payload parsing, mocked ETW lifecycle tests that run on all platforms, and keeps the tracer out of the public threadpoolctl API. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Replace mocked ETW lifecycle tests with subprocess integration checks that spawn real Python, OpenMP, and BLAS threads while the kernel tracer is attached. Keep lightweight payload parsing unit tests for all platforms. Add windows_etw_admin_available() and a short flush delay on tracer stop to improve reliability on CI runners. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Warm up BLAS/OpenMP pools in the parent process, then use threadpool_info (or get_threadpool_limits on older releases) to set the minimum expected ETW thread spawn count instead of hard-coded guesses. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Remove trivial availability helpers from the private API and inline the Windows/admin checks in integration test skipif predicates. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
cursor
Bot
force-pushed
the
cursor/windows-thread-tracer-88ad
branch
from
cb5a29c to
3a8b0ed
Compare
Move the private _thread_tracer prototype and its tests out of the threadpoolctl/ directory, which conflicted with threadpoolctl.py during flit install. Update imports for the modern tests layout and OpenMP helper module names. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Use correctly-sized EVENT_TRACE_LOGFILE nested structures so OpenTraceW does not corrupt adjacent memory. Route ETW callbacks through a module- level function keyed by UserContext, and pass ProcessTrace a TRACEHANDLE array with stable logger-name references. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Drop the threading.Barrier child synchronization that could break under load, derive native pool spawn expectations as team_size minus one, and capture child stderr to simplify future CI debugging. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
…ests _run_traced_child now only runs the subprocess and tracer, returning ThreadSpawnStats. Each integration test asserts the exact expected spawn count explicitly instead of checking a minimum threshold. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
CI shows Windows traced children emit a stable startup overhead on top of work-specific spawns. Measure that baseline once per module and assert total counts as startup + expected work spawns instead of bare totals. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Remove the hardcoded PYTHON_THREAD_WORK_SPAWN_COUNT constant. Each test asserts spawn counts relative to a traced noop child subprocess, with extra spawns defined by the test itself (thread count, OpenMP, or BLAS). Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Delay tracer attach until worker threads are already alive and enable include_existing_threads so ETW rundown is observable. Measure a noop baseline with the same attach timing and assert extra existing counts relative to that baseline. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
The delayed tracer attach must find the child still running; a bare pass body finishes before attach and yielded a zero existing-thread baseline. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Use WINFUNCTYPE unconditionally in the Windows-only ETW module instead of a cross-platform callback fallback. Restore the macOS and Linux flexiblas matrix entries in the CI workflow. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Defer WINFUNCTYPE and the ETW callback registration to win32 only so importing _thread_tracer succeeds on Linux and macOS CI jobs. Usage still raises ThreadTracerError when starting a trace off Windows. Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a private
_thread_tracerpackage (repo root, not public API) with a ctypes-based Windows kernel ETW consumer for counting thread spawn events, plus parsing unit tests and Windows-only integration tests.Review feedback addressed
windows_tracer_available()/windows_etw_admin_available()helpers; platform/admin checks are inlined in testskipifpredicates.WINFUNCTYPEunconditionally in_windows_etw.py(Windows-only module; no cross-platform callback fallback).CI
master(757e83b).Integration tests
test_tracer_counts_existing_python_threadswith delayed attach.