Java gRPC client timeout failures occurred a handful of times within tens of days when using virtual threads.

[ado611]

Environment
gRPC: 1.71.0
JDK: 21
Spring Boot: 3.3.2
gRPC channel: ManagedChannelBuilder.forAddress(host, port).usePlaintext().build();

In my project, Tomcat's virtual threads are enabled. When invoking gRPC services via BlockingStub with a 300ms timeout configured for Tomcat requests, I observed a handful of timeout failures in monitoring. Notably, some gRPC client requests exhibited abnormal latencies, with durations extending to 10s, 20s, or even 160s instead of being terminated within the preset 300ms threshold.

Is this issue caused by incompatibility between gRPC and virtual threads?

I consulted an AI tool, which provided the following explanation:

When using virtual threads with gRPC, it is recommended to prioritize async stubs (e.g., FutureStub or AsyncStub) over BlockingStub. The cooperative scheduling mechanism of virtual threads may conflict with the blocking wait logic in BlockingStub—which relies on LockSupport.park()—and this conflict can lead to potential request hangs or lost wake-up signals.

[kannanjgithub]

It is plausible that the scheduling mechanism of virtual threads with the blocking wait logic in BlockingStub, that uses LockSupport.park for executing the calls and the callbacks using the call executor. There is also the deadline cancellation scheduled task that runs on the transport event loop group, which can also be impacted by scheduling algorithms.

[ado611]

It is plausible that the scheduling mechanism of virtual threads with the blocking wait logic in BlockingStub, that uses LockSupport.park for executing the calls and the callbacks using the call executor. There is also the deadline cancellation scheduled task that runs on the transport event loop group, which can also be impacted by scheduling algorithms.

So either Tomcat's virtual threads are disabled or using async stubs?

[kannanjgithub]

Yes, using async can help with any contention for virtual threads between the call executor and the transport event pool group executor. You should also check for other IO libraries used by your application such as JDBC drivers that will also contend for the virtual threads.