CI: add mpi/distributed module to free CI pipe

[iarejula-bsc]

Adds three free CI jobs, all using minidmr, an open-source tool built by our team that allows creating a reproducible HPC cluster on Docker to simulate a multi-node MPI cluster . minidmr is already integrated into the CI pipelines of two of our other projects, and is also used to run MPI tests locally.

By using it on QuEST, this lets your CI pipeline run MPI tests on a single free ubuntu-latest runner. test_paid.yml is untouched for now; however, after a careful review, it could be removed in a follow-up to save money.

This PR adds:

1. test_free.yml: distributed-minidmr-test: full functional tests over real MPI (minidmr, 4 simulated nodes).
2. audit.yml: sanitisation-test-mpi: ASan over comm/* (the MPI code), minidmr with 2 nodes : the first time this code has run under a sanitiser.
3. audit.yml: coverage-test-mpi: lcov over comm/* via minidmr, 2 nodes: complements the existing serial coverage job, which can't exercise MPI.

Impact:

• Cost: replaces the paid equivalent (mpi=ON, cuda=OFF in test_paid.yml, ~$4.80/run, only triggerable manually via workflow_dispatch) with free jobs on every push/PR, saving ~$5/run.
• Coverage: comm/* goes from 0% measured (explicitly excluded from the serial coverage job, which doesn't compile with MPI) to comm/comm_config.cpp at 77.4% and comm/comm_routines.cpp at 4.3% line coverage.

[iarejula-bsc]

added author info in the header per convention, not sure if this is correct, might be better placed in the headers of the new test instead. Happy to change if needed.

[iarejula-bsc]

i run the tests on mare nostrum 5 and on minidmr local and gh actions, same results on all of them

[otbrown]

Hi @iarejula-bsc, thanks for this!

We're currently trying to (finally) get the 4.3 release ready, as well as buried under proposal writing, so will probably park this at least until 4.3 is out, but I like it!

Note that I currently don't expect the tests to pass when run on multiple processes, but that is one of the items on my list for post-4.3. Your contribution will certainly help with working towards that.

[iarejula-bsc]

We're currently trying to (finally) get the 4.3 release ready, as well as buried under proposal writing, so will probably park this at least until 4.3 is out, but I like it!

Glad to hear that!

Your contribution will certainly help with working towards that.

I woudl like to say that minidmr can we also used in the local machine as a local shell to compile/run/test/debug code, if you feel this will help u to develop faster the MPI code dont hestiate on pinging me and will try to explain how to use it :D

[TysonRayJones]

Howdy!
Could I clarify what the benefit of minidmr is? We can already run multiprocess jobs in CI using oversubscription, though choose not to since painfully slow. Minidmr will of course have the same disadvantage, since ultimately running on the same single Github runner. It sounds like using it will shrink our possible tested platforms to Ubuntu. What's the benefit? 😅

[iarejula-bsc]

Let me split this into two things: what minidmr actually is, and what it specifically gets us here.

minidmr tries to replicate a small HPC environment - MPI, Slurm, PRRTE, the whole scheduler stack. For this PR that side is not relevant, since these jobs don't touch anything Slurm-specific.

The actual difference between --oversubscribe and minidmr: minidmr runs the MPI ranks in two separate containers, each with its own network namespace, which forces OpenMPI onto the tcp BTL instead of the shared-memory one used when ranks are co-located:

• oversubscribed, same container: mca: bml: Using vader btl for send to [...] on node (null) (never tcp).
• two real containers (minidmr): mca: bml: Using tcp btl for send to [...] on node mc-slurmd-2, with an actual btl:tcp: connect() to 172.19.0.4:1024 completed.

That said, there's no impact on the comm/*.cpp coverage numbers in the PR description either way.

It sounds like using it will shrink our possible tested platforms to Ubuntu.

I don't think that's quite right, independent of the above. Nothing existing gets removed, this only adds new Ubuntu-only jobs, and the macOS/Windows serial jobs in test_free.yml stay untouched. Also, the distributed case isn't something I'd expect to run on a non-HPC-like platform anyway - a real multi-node MPI deployment is basically always Linux.

Given all this, there's no single right answer, and I don't want to oversell minidmr:

• If you don't need the network path, --oversubscribe gets you the same comm/*.cpp coverage.
• If you'd rather not depend on minidmr specifically, the same two-container setup can be done directly with docker-compose (minidmr is really just a small Slurm cluster on Docker under the hood). minidmr just adds convenience on top, minidmr exec, automatic directory bind-mounting, cluster bootstrap, etc. All possible by hand with plain Docker, just more manual work.

One advantage worth mentioning separately from all this: minidmr also gives any contributor a reproducible environment, one command gets you the full MPI stack locally, without installing or configuring anything on your own machine. I think this is really interesting and one of the reason we created minidmr. Worth mention it supports custim images, so we can create a reduced image for Quest (without slurm) and still provide other features like exec and mounting the home dir.

minidmr start --nodes 2

minidmr exec -- bash -lc '
cmake -B build -DENABLE_TESTING=ON -DFLOAT_PRECISION=2 \
  -DENABLE_MULTITHREADING=OFF -DENABLE_DISTRIBUTION=ON
cmake --build build --target tests --parallel
'

HOSTS=mc-slurmd-1:1,mc-slurmd-2:1
minidmr exec -e TEST_MAX_NUM_QUBIT_PERMUTATIONS=5 -e TEST_ALL_DEPLOYMENTS=0 -- \
  mpirun -x TEST_ALL_DEPLOYMENTS -x TEST_MAX_NUM_QUBIT_PERMUTATIONS \
    --host "$HOSTS" -np 2 \
    "build/tests/tests"

Happy to go either way, whatever's easier for you to maintain long-term.

For what it's worth, my own preference would still be to keep some form of distributed test in the free pipeline, even if it runs longer than the rest. QuEST doesn't have continuous deployment, releases happen per-version rather than on every merge, so pipeline wall-clock time isn't that critical here.