Abort undersized auto-distributed Qureg instead of replicating it

[iarejula-bsc]

In a multi-node environment, createQureg() can currently return a Qureg which is replicated across every node rather than distributed between them. Every node then redundantly simulates the same state, so all but one of the allocated nodes do no useful work, and nothing warns the user. The auto-deployer reaches that outcome by two routes:

• the state has too few amplitudes to divide evenly between the nodes (numQubits < log2(numNodes)), so it cannot be distributed at all
• the state would divide evenly, but distribution is judged not worthwhile, because each node would be left with fewer than MIN_NUM_LOCAL_QUBITS_FOR_AUTO_QUREG_DISTRIBUTION (26) local qubits.

The solution provided is to require that an automatic deployment utilises every node of the environment, and to abort with a clear error otherwise, the same way already done in validation (e.g. the "node count must be a power of 2" check), so callers can react instead of QuEST silently proceeding. Each route reports its own reason: how many qubits would be needed to divide the state between the nodes, or how many amplitudes each node would be left holding.

Explicitly requesting a non-distributed Qureg (via createCustomQureg() or createQuregFromEnvDeployments()) remains permitted, since that replication is deliberate; only deployments left to the auto-deployer are affected.

reportQuESTEnv() now reports the sizes which can no longer be automatically deployed, rather than advertising a deployment which createQureg() would reject.

I would like to discuss the use of Malleability instead of aborting. Where the runtime supports malleability (DMR library), a better response is available, shrink the job to a node count at which the Qureg does distribute, and carry on, which would turn this error into a resize and leave the abort as the fallback for whatever malleability cannot cover. This would not affect the core of Quest and would be low intrusive, as it would only modify the autodeployer, which i think it fits the use of malleability.

Disclosure: this contribution was developed with support of Opus.

[TysonRayJones]

Could you elaborate in words the decision made? It sounds like you propose to remove the automatic disabling of distribution. The existing functionality is very useful to ensure that the chosen parallelisations/deployments are performant for the user's Qureg. Forcing distribution, when the environment is distributed, will pointlessly cripple performance. Keep in mind that a user may be creating multiple Qureg of varying sizes within the same execution, and it is senseless to force the small ones to distribute their data. Please see instead createForcedQureg

[iarejula-bsc]

I am not removing the distribution, but the replication (same qureg on all nodes).

This comes because I have been doing benchmarks on Quest for the last months. I realized that after the 26 local qubits limit, the autodeployer just wastes resources. As you can see in the image, with 32 nodes it uses all resources but only 1 node is useful, as the rest are replicas with the same result. Personally, I do not see why this is worth it, nor was I aware of this behavior until I saw the benchmarks and analyzed the code to understand the issue of 32 nodes being the worst performant (which is not a good ux, cuz if i did not run the benchmark i wouldnt be aware of 32 is worse than 16 and if i did not read the code i would not know the reason of this behaviour)

The idea of this PR is to notify the user and avoid wasting resources, but still allow the user to use this functionality in the "no auto" mode, if they really want it.

I understand that this limits the scenario where a user allocates 16 nodes for a circuit of 30 qubits and another of 5 qubits: the first one distributes, and the second one fails. I believe that not wasting resources is the better default, with wasting them being opt-in, I am open to discussing it if you disagree and think it is worth to discuss it.

However, I feel the best option here is to use malleability to reallocate the allocated nodes, but that is another discussion.