FIX intermittent Windows module enumeration failure (#217) by ogrisel · Pull Request #219 · joblib/threadpoolctl

ogrisel · 2026-07-09T08:09:09Z

Summary

Fix intermittent OSError("GetModuleFileNameEx failed") on Windows when DLLs are loaded or unloaded during library discovery (reported with conda-forge OpenCV).
Replace the two-phase EnumProcessModulesEx + GetModuleFileNameExW scan with a snapshot-first approach using CreateToolhelp32Snapshot, keeping long-path support via GetModuleFileNameExW and graceful per-module fallbacks.
Add Windows integration tests for library paths longer than MAX_PATH, inspired by the local repro documented in ENH Support path length > MAX_PATH=260 on Windows #189 (comment).

Closes #217

Test plan

Windows CI (py311_conda_forge_openblas): new tests test_windows_library_path_longer_than_max_path and test_windows_library_path_exceeds_internal_limit pass
Windows CI: existing BLAS/OpenMP discovery tests still pass
Manual repro from issue ThreadpoolController() raises an error intermittently with conda-forge OpenCV on Windows #217 with conda-forge OpenCV:
```
import cv2
from threadpoolctl import ThreadpoolController

for _ in range(1000):
    ThreadpoolController()
```

AI assistance disclosure

This pull request was prepared with assistance from an AI coding agent (Cursor / Claude).

Made with Cursor

Use CreateToolhelp32Snapshot for atomic module discovery on Windows, with GetModuleFileNameExW for long paths and graceful per-module fallbacks when handles go stale. Add Windows integration tests for paths longer than MAX_PATH. Co-authored-by: Cursor <cursoragent@cursor.com>

- Run repeated-init stress test on all platforms - Drop redundant OpenBLAS glob pattern and win32 guard in utils - Remove MAX_PATH tests from SAFE_SKIPPED_TESTS whitelist Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Dugal Harris <dugal@leftfield.online> Co-authored-by: Cursor <cursoragent@cursor.com>

Correct numpy.libs glob to use the site-packages sibling directory, add conda-forge Library/bin patterns, and fall back to the loaded OpenBLAS filepath from ThreadpoolController when globs find nothing. Co-authored-by: Cursor <cursoragent@cursor.com>

Use extended-length paths when copying and loading test DLLs so LoadLibrary works beyond MAX_PATH, and pass the module buffer directly to EnumProcessModulesEx to match its ctypes signature. Co-authored-by: Cursor <cursoragent@cursor.com>

Use libopenblas-prefixed destination DLL names so conda builds are recognized, normalize extended-length paths for comparisons, prefer shipped libscipy_openblas sources, and detect truncation when the resolved path fills the GetModuleFileNameExW buffer. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

Load cv2 before stressing ThreadpoolController() to match the Windows conda-forge repro from issue joblib#217, and install OpenCV in CI jobs that exercise this test. Co-authored-by: Cursor <cursoragent@cursor.com>

Use GetCurrentProcess(), trust snapshot szExePath on the happy path, and resolve fallback module paths with GetModuleFileNameW before the slower GetModuleFileNameExW long-path lookup. Co-authored-by: Cursor <cursoragent@cursor.com>

ogrisel

Code style review.

Move ThreadpoolController usage to module scope in utils, hoist conda OpenBLAS globs out of the numpy import block, and inline Windows MAX_PATH test setup in the tests instead of small utils helpers. Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

ogrisel

I think this LGTM. The licensing issue is still there but will be addressed in a follow-up PR to decouple the concerns.

cc @jeremiedbb @tomMoral.

ogrisel commented Jul 9, 2026

View reviewed changes

Comment thread continuous_integration/check_no_test_skipped.py Outdated

Comment thread tests/test_threadpoolctl.py Outdated

Comment thread tests/utils.py Outdated

Comment thread tests/utils.py Outdated

ogrisel and others added 2 commits July 9, 2026 10:18

Address review feedback and fix black formatting

943acae

- Run repeated-init stress test on all platforms - Drop redundant OpenBLAS glob pattern and win32 guard in utils - Remove MAX_PATH tests from SAFE_SKIPPED_TESTS whitelist Co-authored-by: Cursor <cursoragent@cursor.com>

Grant co-authorship to the original issue reporter.

8bb742f

Co-authored-by: Dugal Harris <dugal@leftfield.online> Co-authored-by: Cursor <cursoragent@cursor.com>

ogrisel mentioned this pull request Jul 9, 2026

ThreadpoolController() raises an error intermittently with conda-forge OpenCV on Windows #217

Open

ogrisel and others added 4 commits July 9, 2026 14:08

STYLE fix black formatting in tests/utils.py

68576ad

Co-authored-by: Cursor <cursoragent@cursor.com>

ogrisel commented Jul 9, 2026

View reviewed changes

Comment thread CHANGES.md Outdated

ogrisel added 2 commits July 9, 2026 17:46

Apply suggestion from @ogrisel

2411562

Add missing openblas architecture

aaf547c

ogrisel marked this pull request as ready for review July 9, 2026 16:34

dugalh reviewed Jul 9, 2026

View reviewed changes

Comment thread tests/test_threadpoolctl.py

Comment thread threadpoolctl.py Outdated

Comment thread threadpoolctl.py Outdated

Comment thread threadpoolctl.py

TEST import OpenCV in repeated-init regression test

b2efac3

Load cv2 before stressing ThreadpoolController() to match the Windows conda-forge repro from issue joblib#217, and install OpenCV in CI jobs that exercise this test. Co-authored-by: Cursor <cursoragent@cursor.com>

ogrisel mentioned this pull request Jul 10, 2026

_find_libraries_with_enum_process_module_ex and Stackoverflow licensing #206

Open

REF Windows module discovery per review feedback

9b325f3

Use GetCurrentProcess(), trust snapshot szExePath on the happy path, and resolve fallback module paths with GetModuleFileNameW before the slower GetModuleFileNameExW long-path lookup. Co-authored-by: Cursor <cursoragent@cursor.com>

ogrisel commented Jul 10, 2026

View reviewed changes

Comment thread tests/utils.py Outdated

Comment thread tests/utils.py Outdated

Comment thread tests/utils.py Outdated

Comment thread tests/utils.py Outdated

Comment thread tests/test_threadpoolctl.py Outdated

ogrisel and others added 3 commits July 10, 2026 14:43

Merge branch 'master' into fix-windows-module-enumeration-217

2df80a6

STYLE fix black formatting in tests/utils.py

94f0af4

Co-authored-by: Cursor <cursoragent@cursor.com>

ogrisel commented Jul 10, 2026

View reviewed changes

Comment thread threadpoolctl.py Outdated

ogrisel added 2 commits July 10, 2026 16:36

Update threadpoolctl.py

35a6874

Merge branch 'master' into fix-windows-module-enumeration-217

675e910

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

FIX intermittent Windows module enumeration failure (#217)#219

FIX intermittent Windows module enumeration failure (#217)#219
ogrisel wants to merge 16 commits into
joblib:masterfrom
ogrisel:fix-windows-module-enumeration-217

ogrisel commented Jul 9, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ogrisel left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ogrisel left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ogrisel commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

AI assistance disclosure

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ogrisel commented Jul 9, 2026 •

edited

Loading