Thread-local limits for OpenBLAS with pthreads

[itamarst]

The problem

OpenBLAS with pthreads is widely used: a billion downloads of NumPy from PyPI a month; 400 million for SciPy. Not all are Linux, but most are. Not all of them use BLAS, of course, but some do. It adds up!

The API limit for threads is process-wide. This is annoying, but not the worst part.

The worst part is that it has a process-wide shared pool. Consider a CPU with 32 cores, and a Python thread pool with 4 threads, each of which wants to do some OpenBLAS work. A user whose code needs to run with a variety of BLAS libraries (e.g. sklearn committer) sets a limit 32 // 4 == 8 cores. For many setups this is fine; for OpenBLAS with pthreads, it means 24 cores sit idle.

Potential solutions

• Change this upstream in OpenBLAS.
• Get PyPI packages to stop shipping pthreads version and instead use OpenMP version. So far all OpenMP libraries I've seen don't have process-wide shared thread pool, so this will make things better in most cases.
• Turning https://github.com/itamarst/openblas-tod/ into a real project, and having its own thread-local-scoped limiting API.
• Rethink scope limiting in threadpoolctl in this case.

To expand on the last point—

Maybe threadpoolctl should do something different for OpenBLAS with pthreads (or more broadly APIs affecting a process-wide thread pool)

A user might set three different settings:

1. All cores. "All" means... all cores if it's a top-level process, or whatever share of cores were assigned to this process if it's a worker in a process pool.
2. Run inline/sequentially, without a thread pool, typically set via 1 or 0.
3. A number in between.

Typically the first case will be used when there are no Python threads, so it should be exactly the right number. The second case will be used when there is a Python thread per-core, so again exactly the right number.

The only problems occur, then, in the third case. As we saw in the example above, you may end up with a bunch of cores sitting idle. So maybe in the third case it's worth changing the user limit to something better.

To make this concrete, imagine 16 cores, and a Python thread pool that calls BLAS but also does other work, perhaps in a single thread, perhaps in additional thread pools (Python thread pools, OpenMP, etc, who can say...).

Python thread pool size	Naive OpenBLAS limit	Better OpenBLAS limit
2	8	All cores might be better, 16 instead of 8, but risks oversaturation
3	5	All cores might be better, 16 instead of 5, but risks oversaturation
4	4	All cores might be better, 16 instead of 4, but risks oversaturation
5	3	1 is somewhat better, since then it runs inline and uses 5 cores instead of 3. since it is inline there is no risk of oversaturation
6	2 or 3	1 is much better, since then it runs inline and uses 6 cores instead of 2 or 3. since it is inline there is no risk of oversaturation
7	2	1 is much better
8	2	1 is much better
9-16	1	1 is fine

With 64 cores:

Python thread pool size	Naive OpenBLAS limit	Better OpenBLAS limit
2	32	Maybe better unchanged? Depends if there are other nested thread pools
4	16	?
8	8	?
9	7	?
10	6	1 might be better, depending on if there are other nested thread pools there are that might end up running in parallel to BLAS and thus utilizing more cores.
16-31	3 or 4	1 is vastly better. alternatively, all cores... maybe viable for smaller number of python threads?
32-63	2	1 is vastly better

Conclusions:

• If you limit the number of BLAS threads below a certain threshold, you typically want to just set it to 1. E.g. if you set it to 2, better to just run inline. Will have to think about heuristic thresholds; one potential one is BLAS limit being lower than square root of available cores for the process, since that means parent Python thread pool has higher number of threads than the BLAS limit so running inline will use more cores.
• There may be some cases where it's probably better to use all cores instead of the set limit?

[ogrisel]

BTW, I think we should add a new dynamic inspection tool to threadpoolctl to inspect all the unique threads spawn either by Python or by threadpool libraries using monitoring thread managed via a context manager. This would allow use to write tests that set expectations on the maximum total number of live threads running while executing a particular Python code snippet in a test.

This would allow us to check our assumptions about the default behavior of OpenBLAS+pthreads, OpenBLAS+OpenMP and openblas-tod.

[itamarst]

I've been thinking about how to do this, yeah. Introspecting running threads is easily available via e.g. psutil, so that's one option, and may well suffice. Thread starts are more deterministic but trickier to implement for OS threads (Python is pretty easy: https://docs.python.org/3/library/audit_events.html#audit-events). Some mechanisms I know of:

• On Linux or macOS you can use LD_PRELOAD or the Mac equivalent to catch all thread starts (I have Rust code lying around for this pretty sure).
• memray does something else (ELF patching on Linux, for example).

[itamarst]

I guess the introspection is probably what we'd want for #214

[ogrisel]

For windows, it seems to be possible to use the Win32 API via ctypes as explored in #221.