Summary
TraditionalWorkQueue.isIdle() can report the work queue as idle while a just-submitted operation is still being handed off to a worker thread, so waitUntilIdle() may return while an operation is about to run.
Root cause
isIdle() checks that the operation queue is empty and that no worker thread has an active operation (operation != null). A worker thread is invisible to both checks in the window between taking an operation from the queue (opQueue.poll() inside retryNextOperation) and assigning it to its operation field in TraditionalWorkerThread.run(). waitUntilIdle() polls isIdle() every millisecond, so it can hit exactly this window and return immediately.
How it shows up
TraditionalWorkQueueTestCase.testWaitUntilIdleSlowOpInProgress (TestNG group slow, excluded from CI) is flaky: the test submits an operation delayed by 5 seconds and expects waitUntilIdle(10000) to block for at least 4 seconds, but it occasionally returns right away.
Any production caller of waitUntilIdle() is exposed to the same race: the queue can be declared idle while an accepted operation has not started processing yet.
Expected behaviour
isIdle() must account for accepted-but-not-fully-processed operations, including the queue-to-worker hand-off window, so waitUntilIdle() only returns once all accepted operations have completed.
Summary
TraditionalWorkQueue.isIdle()can report the work queue as idle while a just-submitted operation is still being handed off to a worker thread, sowaitUntilIdle()may return while an operation is about to run.Root cause
isIdle()checks that the operation queue is empty and that no worker thread has an active operation (operation != null). A worker thread is invisible to both checks in the window between taking an operation from the queue (opQueue.poll()insideretryNextOperation) and assigning it to itsoperationfield inTraditionalWorkerThread.run().waitUntilIdle()pollsisIdle()every millisecond, so it can hit exactly this window and return immediately.How it shows up
TraditionalWorkQueueTestCase.testWaitUntilIdleSlowOpInProgress(TestNG groupslow, excluded from CI) is flaky: the test submits an operation delayed by 5 seconds and expectswaitUntilIdle(10000)to block for at least 4 seconds, but it occasionally returns right away.Any production caller of
waitUntilIdle()is exposed to the same race: the queue can be declared idle while an accepted operation has not started processing yet.Expected behaviour
isIdle()must account for accepted-but-not-fully-processed operations, including the queue-to-worker hand-off window, sowaitUntilIdle()only returns once all accepted operations have completed.