HDDS-15533. DNS refresh on heartbeat failure for DN to SCM#10844
Open
kerneltime wants to merge 5 commits into
Open
HDDS-15533. DNS refresh on heartbeat failure for DN to SCM#10844kerneltime wants to merge 5 commits into
kerneltime wants to merge 5 commits into
Conversation
Resolves the HDDS-15682 TODO in HostAndPort: makes the cached address threadsafe (volatile) and adds refresh() to re-resolve host:port on demand. On a connection-class heartbeat failure past a threshold (gated by ozone.client.failover.resolve-needed), HeartbeatEndpointTask asks SCMConnectionManager.refreshSCMServer to re-resolve the SCM peer and, on an IP change, rebuild the endpoint under the same host:port key. The stable key means no re-keying or StateContext queue migration is needed. New knob: hdds.heartbeat.address.refresh.threshold (default 3). Generated-by: Claude Code (Opus 4.8)
…eshed address Review follow-up: HostAndPort.refresh() committed the new IP before SCMConnectionManager rebuilt the endpoint, so a buildScmEndpoint failure or a lost race left the cached address ahead of the live proxy -- and the next "IP unchanged" check then blocked recovery until DN restart. Split refresh() into resolveLatest() (no mutation) + setAddress(); refreshSCMServer now builds the replacement with the candidate address and commits it only on success. Adds TestSCMConnectionManager cases for the successful swap and for build-failure-leaves-state-unchanged. Generated-by: Claude Code (Opus 4.8)
…ak-safe Re-review fixes: - refreshSCMServer: log the old -> new address at INFO (the swap was invisible to operators), and treat a RuntimeException from closing the stale proxy as cleanup-only -- the swap is already committed, and the exception would otherwise escape the heartbeat task. - EndpointStateMachine.close(): shut the executor down in a finally block so a proxy close failure cannot leak the endpoint thread; refreshSCMServer makes close() a recurring runtime path. - HostAndPort.setAddress: reject null. Generated-by: Claude Code (Fable 5)
…refresh test - TestHostAndPort: resolveLatest() no-op on unchanged IP, setAddress null rejection, and map-key identity unchanged by address mutation. - TestSCMConnectionManager: removing the endpoint during the unlocked DNS-resolve window abandons the refresh without committing the re-resolved address. Generated-by: Claude Code (Fable 5)
szetszwo
reviewed
Jul 24, 2026
szetszwo
left a comment
szetszwo
left a comment
Contributor
There was a problem hiding this comment.
@kerneltime , thanks for the update! Just have a comment about the conf key.
| "2s"; | ||
| /** Missed heartbeats against one SCM before the DN re-resolves its hostname (HDDS-15533). */ | ||
| public static final String HDDS_HEARTBEAT_ADDRESS_REFRESH_THRESHOLD = | ||
| "hdds.heartbeat.address.refresh.threshold"; |
Contributor
There was a problem hiding this comment.
Let's add "missed-count", i.e.
-
hdds.heartbeat.address.refresh.missed-count-threshold
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Adds DNS refresh on the DN→SCM heartbeat path, completing the
// TODO: HDDS-15533left inHostAndPortby HDDS-15682 (#10612).#10612 keys SCM-node identity on the stable configured
host:port, but it resolves the address once and freezes it — so a datanode keeps dialing a dead IP after an SCM pod is rescheduled to a new IP (e.g. Kubernetes). This change:HostAndPort— makes the cached addressvolatileand splits refresh intoresolveLatest()(pure re-resolve; returns the new address only if the IP changed, never mutates) andsetAddress()(commit). Identity (equals/hashCode) stays keyed on host:port, so instances remain stable map keys across refreshes.SCMConnectionManager.refreshSCMServer()— builds the replacement endpoint dialing the candidate address first, then, only on success and under the write lock, commitssetAddressand swaps the map value under the same host:port key; the stale proxy is closed outside the lock and the swap is logged at INFO. A build failure or a lost race with a concurrentremoveSCMServercommits nothing, so the next heartbeat failure retries cleanly. Because the key is stable there is no re-keying, collision checking, orStateContextqueue migration.HeartbeatEndpointTask— triggers the refresh on a connection-class heartbeat failure (ConnectionFailureUtils) once the consecutive-missed-heartbeat threshold is reached, gated by the existingozone.client.failover.resolve-neededflag. Recon (passive) endpoints are excluded.EndpointStateMachine.close()— shuts the executor down in afinallyblock so a proxy close failure cannot leak the endpoint thread (refresh makesclose()a recurring runtime path).Behavior note: the rebuilt endpoint starts at GETVERSION, so the DN re-runs version/register against the refreshed SCM before heartbeating. This is intentional — a rescheduled SCM pod is effectively a fresh process — and matches what a DN restart would do.
New config:
hdds.heartbeat.address.refresh.threshold(default 3). Withozone.client.failover.resolve-needed=false(the default), behaviour is unchanged.Enabling in Kubernetes
Replaces the earlier stacked attempt (szetszwo#11) and supersedes the closed #10488, rebuilt directly on
masterafter #10612 merged.What is the link to the Apache JIRA?
https://issues.apache.org/jira/browse/HDDS-15533
How was this patch tested?
TestHeartbeatEndpointTaskDnsRefresh(4): a connection-class failure at the threshold triggers the refresh; flag-off, application-level errors, and below-threshold do not.TestSCMConnectionManager(3 new): successful rebuild on IP change; build failure leaves the endpoint and cached address unchanged (guards the stuck-until-restart regression); a concurrentremoveSCMServerduring the unlocked DNS-resolve window abandons the refresh without committing.TestHostAndPort(3):resolveLatest()no-op on unchanged IP;setAddressrejects null; address mutation does not change map-key identity.mvn checkstyle:checkpasses onhadoop-hdds/commonandhadoop-hdds/container-service; full build and test suites via CI.Generated-by: Claude Code (Fable 5)