Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,18 +185,22 @@ function calls.
* [HPC Cluster Executor](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html)
* [SLURM](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#slurm)
* [Flux](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#flux)
* [Disconnecting and Reconnecting](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#disconnecting-and-reconnecting)
* [Combine both](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#combine-both)
* [Cleaning Cache](https://executorlib.readthedocs.io/en/latest/2-hpc-cluster.html#cleaning-cache)
* [HPC Job Executor](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html)
* [SLURM](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm)
* [SLURM with Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#slurm-with-flux)
* [Flux](https://executorlib.readthedocs.io/en/latest/3-hpc-job.html#flux)
* [Application](https://executorlib.readthedocs.io/en/latest/application.html)
* [GPAW](https://executorlib.readthedocs.io/en/latest/4-1-gpaw.html)
* [Quantum Espresso](https://executorlib.readthedocs.io/en/latest/4-2-quantum-espresso.html)
* [Coupling with other Libraries](https://executorlib.readthedocs.io/en/latest/coupling.html)
* [emcee](https://executorlib.readthedocs.io/en/latest/coupling.html#emcee-markov-chain-monte-carlo)
* [pipefunc](https://executorlib.readthedocs.io/en/latest/coupling.html#pipefunc-function-pipelines)
* [omp4py](https://executorlib.readthedocs.io/en/latest/coupling.html#omp4py-openmp-for-python)
* [pylammpsmpi](https://executorlib.readthedocs.io/en/latest/coupling.html#pylammpsmpi-mpi-parallel-lammps)
* [Integration](https://executorlib.readthedocs.io/en/latest/coupling.html)
* [emcee](https://executorlib.readthedocs.io/en/latest/coupling.html#emcee)
* [pipefunc](https://executorlib.readthedocs.io/en/latest/coupling.html#pipefunc)
* [omp4py](https://executorlib.readthedocs.io/en/latest/coupling.html#omp4py)
* [pylammpsmpi](https://executorlib.readthedocs.io/en/latest/coupling.html#pylammpsmpi)
* [General Pattern](https://executorlib.readthedocs.io/en/latest/coupling.html#general-pattern)
* [Trouble Shooting](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html)
* [Filesystem Usage](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#filesystem-usage)
* [Firewall Issues](https://executorlib.readthedocs.io/en/latest/trouble_shooting.html#firewall-issues)
Expand Down
10 changes: 5 additions & 5 deletions docs/coupling.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Coupling with other Libraries
# Integration
A lot of scientific Python packages already know how to distribute work over many processes - they just need to be
handed an object that behaves like an executor or a worker pool. Because executorlib implements the
[Executor interface](https://docs.python.org/3/library/concurrent.futures.html#executor-objects) of the Python standard
Expand All @@ -18,7 +18,7 @@ not part of the executorlib test environment. In every example the `SingleNodeEx
HPC cluster.
```

## emcee (Markov Chain Monte Carlo)
## emcee
[emcee](https://emcee.readthedocs.io) is a widely used Python package for Markov Chain Monte Carlo (MCMC) sampling, for
example to estimate the posterior distribution of model parameters from experimental data. The likelihood function has
to be evaluated many times per sampling step, and these evaluations are independent of each other, so they can be
Expand Down Expand Up @@ -50,7 +50,7 @@ more expensive likelihood functions the parallel evaluation provides a substanti
`SingleNodeExecutor` with a `FluxJobExecutor` the very same sampling can be distributed over multiple compute nodes of an
HPC cluster.

## pipefunc (Function Pipelines)
## pipefunc
[pipefunc](https://pipefunc.readthedocs.io) is a library to build function pipelines, where the output of one function
is used as the input for the next function, including map-reduce patterns over many parameters. pipefunc takes care of
the book keeping of the pipeline, while the actual execution of the individual functions is delegated to an executor.
Expand Down Expand Up @@ -92,7 +92,7 @@ results = pipeline.map(inputs, executor=executor)
The combination of pipefunc and executorlib is explained in more detail in the
[pipefunc documentation on execution and parallelism](https://pipefunc.readthedocs.io/en/latest/concepts/execution-and-parallelism/).

## omp4py (OpenMP for Python)
## omp4py
The [thread based parallelism](https://executorlib.readthedocs.io/en/latest/1-single-node.html#thread-parallel-functions)
of executorlib is most commonly used to control the number of threads in linked libraries like NumPy. With
[omp4py](https://omp4py.readthedocs.io) - a Python implementation of [OpenMP](https://www.openmp.org) - it is also
Expand Down Expand Up @@ -124,7 +124,7 @@ with SingleNodeExecutor() as exe:
The `threads_per_core` parameter sets the environment variables which control the number of threads, so the requested
number of cores is reserved for the threads created by omp4py inside the `calc_pi()` function.

## pylammpsmpi (MPI-parallel LAMMPS)
## pylammpsmpi
[pylammpsmpi](https://pylammpsmpi.readthedocs.io) provides a Python interface to the molecular dynamics code
[LAMMPS](https://www.lammps.org) which distributes the simulation over multiple MPI ranks while the Python process
itself remains serial. Internally pylammpsmpi uses an executor to start the MPI-parallel LAMMPS processes, so an
Expand Down
Loading