Documentation for proposed API to cover Python thread pool use case#218
Documentation for proposed API to cover Python thread pool use case#218itamarst wants to merge 4 commits into
Conversation
ogrisel
left a comment
ogrisel
left a comment
There was a problem hiding this comment.
Thanks for the PR. Here is some feedback.
| with ThreadPool(2, initializer=limiter.limit_in_pythread) as pool: | ||
| # ... run some BLAS-using code in the thread pool ... | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Maybe we could warn users that changing the thread pool sizes is expected to have some overhead and should not be done from within an otherwise fast iterating loop.
There was a problem hiding this comment.
It's probably not that expensive compared to anything you're likely to run that uses BLAS or OpenMP...
| decorators are accessible through their `wrap` method. | ||
|
|
||
| This should only be used for functions you expect to be called from the main | ||
| thread of a process, without the use of any Python thread pools. |
There was a problem hiding this comment.
Wouldn't this approach also work if you passed such a decorated callable to the POOL.map as an alternative to what is presented in "Switching Back And Forth Between Main Thread and Python Threads"?
There was a problem hiding this comment.
If it's called from a thread, then it's not equivalent to the per-thread API, e.g. it won't know to call MKL's thread-specific limiting API and will instead want to use MKL's process-wide limiting API.
Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
|
Since it sounded like the API design itself is probably OK, I will probably close this draft PR for now and re-open when there's code to show. |
|
@itamarst Actually, I thought a bit more about this and I think we don't necessarily need to change the existing code but rather document how to use it correctly in a Python-level threaded app as explained in #215 (comment). |
2 participants
@ogrisel some questions:
Actual coding depends on #213 and #216.