fix: Reap JobRunner subprocesses before os._exit to prevent orphans and semaphore leaks

[hongquanli]

Summary

Fixes two user-visible symptoms of the same bug: orphaned multiprocessing spawn_main processes (PPID 1) accumulating after GUI sessions, and the shutdown warning resource_tracker: There appear to be N leaked semaphore objects to clean up at shutdown.

Root cause

main_hcs.py exits via os._exit() (deliberate — PyQt5's C++ destructor order conflicts with Python's GC). But os._exit() also skips multiprocessing's atexit hook, the mechanism that normally terminates daemon child processes. Any JobRunner subprocess still alive at that moment is orphaned to PPID 1 and its queue/event semaphores leak.

The per-acquisition cleanup (_finish_jobs) shuts runners down in non-blocking daemon threads, with a comment claiming "if app exits before threads complete, OS will terminate subprocesses anyway" — untrue under os._exit(). Runners stay alive whenever their shutdown event was never set: the exit-time abort join times out (15 s cap), the worker dies before _finish_jobs, or the shutdown threads lose the race to os._exit().

Fix

• shutdown_all_job_runners() (job_processing.py): finds live JobRunner children via multiprocessing.active_children(), runs their full shutdown() in parallel (bounded), then terminates any survivor. Called in main_hcs.py immediately before os._exit().
• JobRunner.shutdown() now also clears _ready_event, releasing its semaphore like the other primitives.
• Corrected the _finish_jobs comment.

Testing

• New tests/control/core/test_job_runner_teardown.py: a runner whose shutdown event was never set is reaped (deterministic reproduction of the leak condition); no-op with no runners; safe after a prior clean shutdown.
• All 65 existing JobRunner-related tests pass (test_job_runner_shutdown/pending, backpressure, zarr writer, OME-TIFF saving).
• GUI driver verification in simulation (programmatic Qt driver replicating main_hcs.py's exact exit path): normal close after an acquisition and close mid-acquisition — zero orphaned children, no resource_tracker warning.
• Note: the common GUI close paths already cleaned up correctly in control runs (the existing non-blocking shutdown usually wins its race); this fix closes the timing/crash windows, which matches how the leaks were observed in practice — occasional, and hours old.

🤖 Generated with Claude Code