fix(camera): prevent intermittent all-zero ToupCam frames from read-buffer realloc race

[Alpaca233]

Summary

Fixes intermittent all-zero saved frames in multi-channel acquisitions (a scattered ~1–2% of FOVs across channels — first noticed as "dim"/dark 405 tiles in a mosaic, but the raw TIFFs are exactly zero). A real exposure is never all-zero, so this is a data-path defect, not a light/laser/firmware issue.

Root cause — a thread race on the camera read buffer

ToupcamCamera._on_frame_callback (SDK callback thread) reads self._internal_read_buffer twice, non-atomically:

1. PullImageV2(self._internal_read_buffer, …) fills it, then
2. np.frombuffer(self._internal_read_buffer, …) re-reads it.

Meanwhile _update_internal_settings — invoked by set_exposure_time on every channel switch (set_microscope_mode → set_exposure_time) from the acquisition thread — reassigns self._internal_read_buffer = bytes(buffer_size) (a fresh zero-filled buffer) without holding _raw_frame_callback_lock. In MultiPointWorker.acquire_camera_image, the channel setup (_select_config → the realloc) runs before the wait on the previous frame's callback, so the next channel's reallocation is not gated by the in-flight callback.

If that reassignment lands between the callback's two reads, PullImageV2 filled the old buffer but np.frombuffer views the new all-zero one → the delivered/saved frame is entirely zeros. It's channel-agnostic (the buffer is re-zeroed before every frame) and intermittent (the store must land in the pull window — observed 0.5–45.8 ms per callback in production logs).

Fix

Snapshot the buffer reference once inside the callback lock and use it for both the fill and the read, so a concurrent reassignment can no longer split them. The local reference also keeps that buffer alive for the SDK's C-level write. Minimal, localized, and correct regardless of the async triggering pipeline.

Test

tests/control/test_camera_toupcam_buffer_race.py drives the real _on_frame_callback/_process_raw_frame (faking only the SDK boundary + ROI getters) and deterministically forces the reallocation-during-pull interleaving:

• Before the fix: the delivered frame is all zeros (the exact production symptom).
• After the fix: the frame reflects the buffer that was actually filled.

Plus a no-realloc sanity case. black-clean; camera-layer tests pass locally.

Notes / follow-ups

• Verified against a faithful unit-level reproduction, not against physical hardware — worth confirming on a scope with a real multi-channel run.
• Optional follow-up (not in this PR): guard the realloc in _update_internal_settings to only run when buffer_size actually changes. Exposure changes don't change the buffer size, so today it needlessly allocates+zeroes the full frame buffer on every channel switch — a perf win and defense-in-depth that narrows the race window further. Deserves its own test.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR addresses an intermittent Toupcam frame corruption issue where saved/delivered frames can become entirely zero due to a race between the SDK callback thread reading self._internal_read_buffer and the acquisition thread reallocating that buffer during exposure/channel switches.

Changes:

• Snapshot self._internal_read_buffer once inside ToupcamCamera._on_frame_callback and use that reference for both PullImageV2 and np.frombuffer, preventing split-buffer reads.
• Add a regression test that deterministically forces the “realloc during pull” interleaving and asserts the delivered frame is not all-zero.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File	Description
software/control/camera_toupcam.py	Prevents all-zero frames by ensuring the callback uses a consistent read-buffer object for both SDK fill and NumPy view.
software/tests/control/test_camera_toupcam_buffer_race.py	Adds a targeted regression test to reproduce and guard against the buffer-realloc race.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.