[Flow Control] Implement Garbage Collection for Priority Bands.

[LukeAVanDrie]

What would you like to be added:
A mechanism to clean up and remove PriorityBand structures (queues, atomic counters, mutexes) from the FlowRegistry and registryShard when they are no longer needed.

Requirements:

1. Safe Draining: A band cannot be deleted if it contains active or buffered requests.
2. Triggering: This should ideally be triggered by the deletion of the corresponding InferenceObjective, or via a background GC loop that detects "Empty & Idle" bands if we stick with JIT for a while (see [Flow Control] Refactor: Drive Priority Band lifecycle from InferenceObjective controller. #2011).

Why is this needed:
With the introduction of dynamic provisioning, the system creates resources for every new priority level seen. Currently, these structures are never deleted, even if the associated InferenceObjective is deleted or the priority level is abandoned.

While the memory footprint of an empty band is small (mostly pointers and counters), unbounded growth is a risk over long-running processes with high CRD churn.

[rahulgurnani]

/milestone Flow Control

[evacchi]

I'd like to make an attempt at it, unless someone is already working on it :)

[LukeAVanDrie]

Thanks @evacchi!

I haven't started looking into this yet, so it should still be open. I'm not sure if it's worth doing #2011 first though to move away from the JIT path for flow registration. The Flow Registry code is unfortunately quite complex and brittle. It could benefit from some refactoring to break up responsibilities by perf requirements, but I haven't found time to do that yet.

Some valuable context can be found in these PR discussion threads for why we are handling it the current way:

• flowcontrol: Support dynamic priority provisioning #2006 (comment)
• flowcontrol: Support dynamic priority provisioning #2006 (comment)

Also, generally, looking over #2001 and #2006 may be helpful.

Honestly, this is one of the trickier open Flow Control issues, but if you have ideas here they are more than welcome. This would be a really good way to deeply learn state management for flow control.

[LukeAVanDrie]

If you're still interested, please reach out if you have any questions. I am happy to provide some more code pointers and background context or even hop on a call for a code walkthrough if you would find that helpful.

[evacchi]

hah, doing #2011 first might make sense... I still tried to sketch it within the GC loop. I am opening a draft PR to discuss if it makes sense at all, I feel like I am oversimplifying

[evacchi]

see #2097

[LukeAVanDrie]

@evacchi, this draft is very valuable because it has exposed the hidden complexity in our current locking model. I left some feedback on your PR regarding this, but I have been thinking on this more and I think that instead of trying to prevent these races with complex locks, we should consider switching to an optimistic model that detects and recovers from them.

How it works:

1. No Hot-Path Locks: WithConnection never acquires the global fr.mu.
2. Optimistic Pinning: The JIT path grabs a pointer to the State (Band or Flow) and atomically increments a leaseCount (Pin).
3. Verification: After pinning, it checks: "Is this object still authoritative in the sync.Map?"
   • If Yes: We are safe. The GC guarantees it will never delete an object with lease > 0.
   • If No: We pinned a "Zombie" that the GC is currently deleting. We simply unpin and retry the loop.

Why this may be better:

1. Deadlock Immunity: We remove the lock hierarchy entirely. The Data Path never fights the Control Plane (GC/Admin) for mutexes.
2. Safety: The "Zombie Band" race is impossible because the JIT path "votes" to keep the object alive (via the atomic lease) before it tries to use it.
3. Simplicity: We replace brittle distributed locking logic with a self-healing for loop.

I believe refactoring registry.go to use this Optimistic model is the prerequisite that makes both Dynamic Priority Bands (your PR) and Flow GC safe and stable.

I can try sending out a sketch of this that builds upon your band GC logic. Then we can synthesize the best elements into your existing PR.

[LukeAVanDrie]

^ This is as an alternative to your approach and the other recommendations I made on your PR.

[LukeAVanDrie]

Let me try applying this pattern to replace the existing flow GC only (leaving priority bands without). If it looks promising, you can extend it to your band GC approach; else, we can find another solution. Will try to send out a draft PR tomorrow.

[evacchi]

awesome! thanks for the quick turnaround!