fix: hold cbc to a wall clock so a stuck solve cannot take the replica

[andig]

What happens today

OPTIMIZER_TIME_LIMIT is 10 s, but CBC only tests -sec between branch and bound nodes. Presolve and the root LP are not interrupted, so a solve that grinds there runs unbounded.

With --workers 1, that worker is the replica. gunicorn kills it at --timeout 60, the in-flight request dies, everything queued behind it on that replica dies with it, and the solver is orphaned onto the master.

Eight times in the ten hours after the 26 Jul 11:00 UTC rollout of optimizer--0000029, on four different replicas:

13:11:48 x78wl  reaping orphaned solver 10297 left by worker 9879
13:11:48 x78wl  [CRITICAL] WORKER TIMEOUT (pid:9879)
19:49:11 h2t27  Booting worker with pid: 40095
19:50:40 h2t27  [CRITICAL] WORKER TIMEOUT (pid:40095)     <- wedged 89 s after booting

Every one of them left a solver behind, which is what puts the stall inside the solve rather than in Python.

The core leak this used to cause is already gone — #117's reaper fires in the same second, 7 of the 8 events show the reaping orphaned solver line. What is left is the replica: up to 60 s of a replica serving nothing, plus its queue.

What this changes

• The solve runs under its own wall at 2 × time_limit + 5 — 25 s in production. PuLP raises when its solver dies (cbc.wait() != 0), and app.py already turns that into a 500, so the request fails in 25 s with the worker intact.
• The /proc scan the master used to find orphans is the same scan a worker needs for its own solver, so it moves to optimizer.solvers and takes the parent pid as an argument. solvers_of(1) is the master's reaper, solvers_of(os.getpid()) is the worker's wall. Tests move with it.
• --timeout 60 → 40, the backstop for a request stuck outside the solve. Still above the worst elapsed time seen in production, 30 s.

Not changed: -timeMode elapsed was the first thing I reached for, but PuLP 3.3 already passes it whenever timeLimit is set. -sec is wall clock here — it is simply not checked often enough.

Verification

27 tests pass, ruff clean, az bicep build clean. The new tests drive solver_wall against a fake /proc: a solve past the wall loses its own solver and not somebody else's, a solve inside the wall keeps it, and no configured limit means no wall. The scan itself is a no-op off procfs, so the wall does nothing on macOS — the same scan has been running in production for the reaper since #117.

Untested outside CI: that a real cbc dies at the wall under load. Worth watching WORKER TIMEOUT after the rollout — the count should go to zero and be replaced by 500s at ~25 s.

🤖 Generated with Claude Code