add ImageAddedToBuffer notification

[jacopoabramo]

I originally asked this question and the reply was to follow a pattern of constantly polling the core. Initially this worked fine but now I'm in the situation where I would like to leverage the core in an asyncio event loop and a callback would be quite useful.

The change is simply to emit a callback with the label of the camera acquiring the image so that an user may provide its own callback (or pymmcore plus can provide a signal for this).

I added a couple of smoke tests to verify the correct behavior, I'd be happy to add other tests to ensure situations that might cause problems.

EDIT: just to be clear I did run these tests locally on Windows and they pass, not sure if the CI triggers a testing run

[jacopoabramo]

I'm sure that given the very small scope of the change you guys might have better arguments of why this is a bad idea, but it could be helpful also for the integration of the storage API (see #313 and #770)

[marktsuchida]

@jacopoabramo Sorry for the late response. Thanks for the PR!

I am all for a mechanism to avoid polling but I think this is a bit of a stretch for notifications, which are primarily a UI-refreshing mechanism. They are not intended for more precise sequencing of actions.

We probably need a callback-based interface that is more integrated into the sequence acquisition API, in which a callback is registered per acquisition. Otherwise it will add an obstacle as we move toward better support of multi-channel cameras and multiple simultaneous acquisitions.

Note that the camera label that is passed to onImageAddedToBuffer() would (at the moment) be the physical camera in the case of Multi Camera, but also could be an intrinsic multi-channel camera. Handling these callbacks correctly on the Python side requires the Python side having a complete model of which of those cases is currently active and the physical cameras involved -- and keeping how all that works unchanged makes it hard to clean up our multi-camera support.

So if we really need this right now, I might be more comfortable with an onImageAddedToBuffer() without any parameter. That way, the Python side just uses it as a hint, and goes and checks (all of) the currently running acquisition(s) (even if there could be more than one in the future). This behavior we could preserve when future improvements are made, even if the notification becomes deprecated.

But one more thing we absolutely need before adding such a notification is the ability to coalesce multiple notifications. We don't want 1000s of these to accumulate in the queue for a high-frame-rate acquisition.

[jacopoabramo]

Thanks for the reply - no need to apologize for the delay!

But one more thing we absolutely need before adding such a notification is the ability to coalesce multiple notifications. We don't want 1000s of these to accumulate in the queue for a high-frame-rate acquisition.

Yes I was also concerned about this. To be honest I'm not using multi-camera at all so it skipped from my mind, so I understand your concerns - thanks for pointing them out.

So if we really need this right now, I might be more comfortable with an onImageAddedToBuffer() without any parameter.

That's easy to do, but the question is at this point if it's worthwhile to apply this. I can say that I'm in no rush of having this notification at this time - it was just something I started investigating and to get feedback on since it really looked too blatantly simple and there had to be a catch somewhere.

We probably need a callback-based interface that is more integrated into the sequence acquisition API, in which a callback is registered per acquisition. Otherwise it will add an obstacle as we move toward better support of #838 and multiple simultaneous acquisitions.

That makes more sense, but I imagine it would imply adding a new API to the core and expose it to pymmcore, right? Something like addImageCallback or something...?

To be fair a real game changer - imho at least - would be (again, only for the python side - in particular pymmcore-nano) using a PyCapsule to setup a jump from pymmcore to a separate writer without having to pass from python at all and embedding the notification there - but at a glance it's not trivial and it might be in contrast with the storage API currently being designed.

If you feel that it's more worthwhile to pursue a notification more embedded into the sequence API I can probably start having a look - although I'd love if you could elaborate on what you were imagining

[marktsuchida]

It won't be simple -- I think we're still a few nontrivial steps away from a new acquisition API. I'm planning a series of internal changes first: model an "acquisition" as an explicit object (needed just to be correctly thread-safe inside MMCore!), manage multi-channel cameras better (with MMCore having knowledge of the logical-physical camera associations, at least for a given acquisition), have a per-channel sequence buffer, etc. These can all be done without (yet) adding a new API, it turns out, and it's best done that way because we do need to preserve the existing API anyway.

So I wouldn't want to rush into a new API any time soon, which is why I proposed a minimal notification if avoiding polling is a high priority for you at the moment (though perhaps it's not).

I guess the rest of the discussion is about avoiding image copy at the C++ → Python boundary, which I think is best thought of as an orthogonal issue to the polling vs callback API. They are both important.

To be fair a real game changer - imho at least - would be (again, only for the python side - in particular pymmcore-nano) using a PyCapsule to setup a jump from pymmcore to a separate writer without having to pass from python at all and embedding the notification there - but at a glance it's not trivial and it might be in contrast with the storage API currently being designed.

One thing I do envision is the ability to use Python-allocated memory for the sequence buffer. So you would "enqueue" NumPy arrays (or any buffer protocol object) into MMCore and start the acquisition. On every callback you receive one of these arrays that has been filled, which you can pass to a file writer with no copying; when done, you re-enqueue the empty buffer. I think this will be doable, but only after a few rounds of internal cleanup in MMCore, including the ones mentioned above. (In particular, exposing MMCore-allocated buffers to Python is a can of worms that we probably should avoid.)

Using PyCapsule to eliminate the per-frame call through Python would be a step beyond that and may make sense for high frame rates (especially until everything supports free threading). But wouldn't that require a custom capsule interface that the writer would need to implement in C? At that point the storage device might be easier to manage because we will have full control over buffer lifetime management. Alternatively, we could keep the call through Python but batch multiple frames together to reduce its frequency — makes sense for small and fast images.

I think the most likely order in which these will happen is storage device → application-allocated buffers → maybe capsules.