Server panic in scheduler/solver.rs:222 (unwrap on None) when workers register into a deep WAITING queue of compact-resource tasks

[trap-fleet-ci]

Summary

HQ v0.26.2 server crashes with a Rust panic in tako/src/internal/scheduler/solver.rs:222 when a large number of tasks with compact resource constraints are in the WAITING state at the moment new workers register (or reconnect) to the server.

Version

HyperQueue v0.26.2

Panic message

thread 'main' panicked at 'called `Option::unwrap()` on a `None` value',
tako/src/internal/scheduler/solver.rs:222:65

The server process exits immediately; all in-flight and queued work is lost unless the server was started with --journal.

Reproduction conditions

1. Submit a large batch of tasks (~50+) that all require compact CPU and memory resources (e.g. --resource cpus=N:compact --resource mem=sum(M):compact).
2. Allow the tasks to accumulate in WAITING state (few or no workers connected).
3. Register one or more new workers simultaneously (e.g. several SLURM jobs starting at once).
4. The server panics during the scheduling pass triggered by the new worker registration.

We observed the panic reliably when ≥48 compact-resource tasks were queued and multiple workers connected at the same time. With fewer waiting tasks or non-compact resource assignments the scheduler did not panic.

Impact

• Server crashes immediately on the scheduling cycle triggered by worker connection.
• All queued tasks remain in the journal but the running server is gone.
• Workers that were mid-task lose their connection; tasks may be lost depending on on-server-lost policy.

Recovery workaround

1. Cancel or reduce the WAITING task count below the threshold (we used ~35 as a safe cap).
2. Restart the server from journal: hq server start --journal <journal-path>
3. Allow workers to reconnect (retry loop on the worker side helps here).
4. Once the queue is drained to a safe level, resume normal submission rate.

System context

• Running on a SLURM-managed HPC cluster (Linux x86_64)
• Workers submitted as SLURM batch jobs, multiple workers registering simultaneously when SLURM releases the allocation
• Tasks use cpus:<N>:compact and mem:sum(<M>):compact resource specs

Request

It would help to have either:

• A bounds check / graceful error in solver.rs:222 instead of unwrap(), or
• A --wait or --queue-tasks-before-scheduling worker startup mode that defers the first scheduling pass until the initial task batch is settled

Happy to provide more detail or test a patch. Thank you for the great tool!