fix: flaky Consul embedded test startup#19564
Conversation
Shekharrajak
changed the title
| lastException = e; | ||
| } | ||
|
|
||
| try { |
There was a problem hiding this comment.
means:
Consul is not ready yet.
Wait 500 ms.
Try again.
| } | ||
| } | ||
|
|
||
| throw new RuntimeException( |
There was a problem hiding this comment.
out side the while loop .
we kept retrying until READINESS_TIMEOUT expired
and Consul never became ready
| } | ||
| } | ||
|
|
||
| private void waitForConsulApi() |
There was a problem hiding this comment.
try /v1/status/leader
if ready -> return
if not ready -> sleep 500 ms
if sleep is interrupted -> restore interrupt flag and fail
otherwise -> retry until 30 seconds deadline
|
@jtuglu1 @FrankChen021 - please have a look. |
FrankChen021
left a comment
FrankChen021
left a comment
There was a problem hiding this comment.
| Severity | Findings |
|---|---|
| P0 | 0 |
| P1 | 0 |
| P2 | 1 |
| P3 | 0 |
| Total | 1 |
Reviewed 3 of 3 changed files.
This is an automated review by Codex GPT-5.5
|
Thanks , looks good to merge. This will unblock : #19510 |
abhishekrb19
left a comment
abhishekrb19
left a comment
There was a problem hiding this comment.
Lgtm, thanks for the fix @Shekharrajak. Saw this test flake in https://github.com/apache/druid/actions/runs/27331948177/job/80830684837 as well.
3 participants
Fixes #19563.
Description
This PR hardens the Consul-backed embedded tests against startup races where the Consul container has started but the host-mapped Consul API is not yet reliably accepting requests.
Readiness hardening:
reduce chance of starting Druid before Consul is usable
Announce retry:
tolerate a remaining transient HTTP race if it still happens
Release note
Druid's Consul service discovery announcer now retries transient HTTP registration failures during startup.
Key changed/added classes in this PR
embedded-tests/src/test/java/org/apache/druid/testing/embedded/consul/ConsulClusterResource.javaextensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/ConsulDruidNodeAnnouncer.javaextensions-contrib/consul-extensions/src/test/java/org/apache/druid/consul/discovery/ConsulDruidNodeAnnouncerTest.javaThis PR has: