[Documentation] Consolidate workshop tutorials into the documentation

[jan-janssen]

Motivation

The documentation for executorlib is currently scattered across a number of separate workshop and tutorial repositories. This PR reviews the Jupyter notebooks in those repositories, compares them to the documentation in this repository, and folds the content that was only available in the workshops into the main documentation.

Repositories reviewed:

• lanl-executorlib-tutorial
• executorlib-with-slurm
• pylammpsmpi-with-executorlib
• executor-without-context
• atomistics-with-executorlib
• pipefunc-executorlib
• executorlib-emcee
• executorlib-omp4py
• remote-executor (archived; functionality already covered by the HPC Cluster Executor)

What was missing and is now added

The following topics were present in the workshops but absent from the documentation (verified by searching docs/ and notebooks/):

• Coupling with other libraries (docs/coupling.md, new page): executorlib implements the standard concurrent.futures.Executor interface, so it can be handed to scientific packages that accept an executor or worker pool. New reference examples for emcee, pipefunc, omp4py and pylammpsmpi.
• "Which Executor should I use?" (README.md): an overview table of the five Executor classes (where each runs, scheduler command, when to use it) plus links to the new page.
• Understanding Future objects (1-single-node.ipynb): a gentle, executable introduction to result() / done() / cancel() and the submit-then-collect pattern for users without a strong HPC/CS background.
• Disconnecting and reconnecting (2-hpc-cluster.ipynb): how to submit functions, close the Python process, and reload the results later via shutdown(wait=..., cancel_futures=...), with a summary table of the behaviour.
• Verifying the resource assignment (2-hpc-cluster.ipynb): retrieving the SLURM queue_id via get_cache_data() and inspecting it with sacct.
• Starting Flux from the cluster executor (3-hpc-job.ipynb): a submission template that boots a Flux instance per sbatch job for nested execution.

Notes

• SLURM examples are kept as non-executed reference code (markdown code blocks), consistent with the existing notebooks, because the notebooks are executed in CI where no SLURM scheduler is available. The only new executable cell uses the SingleNodeExecutor and was verified to run.
• API usage was checked against the source: the resource-dictionary keys (gpus_per_core, memory_max), split_future(future, n=...) and shutdown(wait, cancel_futures) follow the actual implementation.
• The documentation aims to stay approachable for scientists who use executorlib as a tool rather than HPC specialists.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Documentation
  • Added "Which Executor should I use?" guidance covering executor types and selection criteria.
  • Added comprehensive documentation for integrating executorlib with popular scientific libraries (emcee, pipefunc, omp4py, pylammpsmpi) with reference code examples.
  • Enhanced HPC notebook with Flux/SLURM integration guidance and submission templates.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: f84c6d37-5f57-440b-b789-9a71b802f341

📥 Commits

Reviewing files that changed from the base of the PR and between ba290e1 and 71a7569.

📒 Files selected for processing (6)

• README.md
• docs/_toc.yml
• docs/coupling.md
• notebooks/1-single-node.ipynb
• notebooks/2-hpc-cluster.ipynb
• notebooks/3-hpc-job.ipynb

---

📝 Walkthrough

Walkthrough

This PR expands executorlib documentation by introducing executor selection guidance in the README, a new comprehensive guide on integrating executorlib with external scientific Python libraries (emcee, pipefunc, omp4py, pylammpsmpi), and enhanced notebook examples for HPC SLURM/Flux workflows. All changes are documentation-only.

Changes

Documentation expansion for executor selection and library coupling

Layer / File(s)	Summary
Executor selection guide in README README.md	README adds a table comparing five executor classes by execution location, scheduler command, and best-use scenario, plus explanation of Cluster vs. Job executor communication patterns. Documentation navigation is extended with a new "Coupling with other Libraries" section and sub-links for emcee, pipefunc, omp4py, and pylammpsmpi.
Library coupling documentation docs/_toc.yml, docs/coupling.md	New coupling.md document demonstrates using executorlib as a drop-in executor replacement for external libraries: emcee's EnsembleSampler (with block_allocation and HPC swap examples), pipefunc pipelines (per-output executor mapping), omp4py (thread parallelism via resource_dict["threads_per_core"]), and pylammpsmpi (MPI-parallel LAMMPS). Concludes with a general pattern for scaling from laptop to HPC by changing executor class names. Document is registered in the Jupyter Book TOC.
HPC notebook improvements notebooks/2-hpc-cluster.ipynb, notebooks/3-hpc-job.ipynb	2-hpc-cluster.ipynb notebook metadata is updated. 3-hpc-job.ipynb "SLURM with Flux" section is enhanced with a submission_template bash snippet showing automatic per-job flux start via srun, and guidance on using a nested FluxJobExecutor to distribute many short tasks within the flux allocation.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~8 minutes

Possibly related PRs

• pyiron/executorlib#686: Modifies notebooks/3-hpc-job.ipynb in the SLURM+Flux section, directly overlapping with the same notebook content in this PR's flux executor guidance.
• pyiron/executorlib#507: Updates README executor usage and SLURM/flux documentation, overlapping with this PR's README executor selection additions and HPC context clarifications.
• pyiron/executorlib#560: Modifies both notebooks/3-hpc-job.ipynb and README descriptions for Cluster/Job Executors, directly overlapping with HPC executor terminology and notebook content in this PR.

Poem

🐰✨ Coupling executors with labs outside,
Five choices guide the way far wide,
From emcee's dance to LAMMPS's might,
One humble pool orchestrates the flight—
Single to cluster, a class-swap's delight!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Title check	⚠️ Warning	The PR title claims to 'consolidate workshop tutorials into the documentation,' but the primary changes are adding new documentation content (coupling.md, executor overview) rather than consolidating existing tutorials into docs.	Revise the title to accurately reflect that the PR adds new documentation pages demonstrating library integration patterns and executor guidance, such as: '[Documentation] Add executor usage guide and library coupling examples' or '[Documentation] Document executor selection and third-party library integration'.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch consolidate-workshop-docs

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Ruff (0.15.15)

notebooks/2-hpc-cluster.ipynb

Unexpected end of JSON input

notebooks/3-hpc-job.ipynb

Unexpected end of JSON input

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.24%. Comparing base (ba290e1) to head (71a7569).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1017   +/-   ##
=======================================
  Coverage   94.24%   94.24%           
=======================================
  Files          39       39           
  Lines        2119     2119           
=======================================
  Hits         1997     1997           
  Misses        122      122           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.