Work stealing breaks local submit of per carrier pollers

[franz1981]

#108 can cause a JDK per carrier read poller to be temporarily stolen, so when a it unpark VTs the will eventually migrate to their "home" scheduler.
This home scheduler is not the one which has run the VT last time, making locality to break, despite be "sticky".
One option would be to resume the VTs back to the last scheduler who run them, making them to return to their home only if the yield again.
Probably stickiness is not very good...

Another option instead is to act like the FJ pool, but we need to keep track the last running scheduler for the poller and submit the VT to unpark to it, saving cross thread submissions.

A third option is not allow JDK pollers to be stolen, but this means that if a carrier is busy or unresponsive, none will make its read poller to make progress, which is still not great.
Having a CHM of read pollers/running scheduler is not a bad option, although it create a shared code path, which can be highly contended with aggressive workstealing of pollers. This is not great as updating the CHM can just happen right before running the JDK pollers, in the carrier which is currently draining, which is not great.

[He-Pin]

Using a local scheduler indefinitely does present problems; this is evident in Akka Pekko, where it can lead to actor starvation and timeouts. It would be ideal if the scheduler could intelligently detect when to call externalSubmit. Designing a good scheduler is indeed challenging, especially one that adapts to the given workload.

[franz1981]

Hi @He-Pin yes it is not a breeze to write a general purpose one, but writing one which operate with a simple enough cost model allow to implement specific use cases better than a generic one.
The existing workstealing mechanism I have implemented aim to unblock tasks on unresponsive carriers assuming the other carriers are not unresponsive as well, so it should help with starvation.
I can publish some charts on how it better behave under saturation compared to FJ in case you are interested to give it a shot.
The good thing of this scheduler is that you can create your own vthread factory which implements the behavior you need, including posting directly to specific carrier your VT - or to FJ too.
In short, is simple (relatively), and have enough flexibility that you can write something which perform well with your use case, WDYT?

The existing readme cover quite deep how it works.
If it is not enough let me know!

[franz1981]

Fyi

These are the current numbers.
And I am working to provide spinning carrier options for who doesn't care about CPU usage and aim to low latency. With Uring it helps a lot, clearly with a CPU hit.

[He-Pin]

Thanks for sharing. We encountered a problem with Pekko where nightly builds frequently failed on Java 25. Our LLM team investigated and found it was related to changes in the ForkJoinPool scheduling. Also, if I remember correctly, iouring might have significant latency issues with small packages.