Deprecate ezmsg.util.generator

[cboulay]

AI coding assistants will sometimes grab on to these old patterns and try to introduce them into new code. I'd like to make their deprecation more explicit to avoid that from happening. I've already done so for several of the namespace packages like ezmsg-sigproc (not released, but in dev).

[cboulay]

There are still a few usages of @consumer that we might want to deprecate.

• modify_axis
• set_key

[griffinmilsap]

Besides the fact that AI agents will sometimes start writing generators, are there dangers to leaving this in? I have a bunch of older generator-based signal processing modules in some of my notebooks and I'd love if those continue to work. I acknowledge that there are better patterns now, but for simple things is it actually bad to support this pattern too?

If there's not necessarily a need to fully deprecate the generator pattern we can leave lots of comments in encouraging use of the BaseProcessor patterns which hopefully AI agents can catch. Perhaps all that's needed is a short blurb about the limitations of the generator approach and or rationale why its discouraged for many use cases.

If memory serves, it has to do with pickling, right? When we use multiprocessing we can't use these generators? Preston has suggested it may be worthwhile to consider whether we support parallelization via free-threading as opposed to multiprocessing which would come with a multitude of performance wins for many real-world systems. If the generator approach enables simple operations and parallelizes in a threaded ezmsg implementation, perhaps its premature to deprecate the pattern. Or if we do, I .. can't really find any issue with "un-deprecating" it later if we decide it's alright to continue supporting?

[cboulay]

The first problem is related to pickling. I agree that we should push for threading > multiprocessing from Python 3.13 onward, but we aren't there yet. IIRC, cloudpickle actually solved the pickling problem -- there might be a hybrid solution where we use cloudpickle when using multiprocessing.

Regardless, the second problem is that we can't set/get the generator's internal state. The pattern I've established in ezmsg-baseproc makes it possible to dehydrate a unit to store its state in a database, then rehydrate it either with the previous state or even another session's state. This is how cloud-based messaging works (i.e., things built on Kafka) as they are designed to work with multiple concurrent users without spinning up a unique session for each user.

So far I've mostly been targeting high throughput systems where the dehydrate / rehydrate overhead would make this unviable, but I do have one client who is/was using ezmsg in the cloud with lower rate data and they need this functionality.

I guess I wouldn't object to the generator pattern on a component that is entirely stateless. This is true of set_key, but not of modify_axis.