Skip to content

[KYUUBI #7556] Rearm the ZooKeeper engine deregistration watcher after reconnect#7558

Open
Yangxuhao123 wants to merge 1 commit into
apache:masterfrom
Yangxuhao123:fix/zk-deregister-watcher-rearm
Open

[KYUUBI #7556] Rearm the ZooKeeper engine deregistration watcher after reconnect#7558
Yangxuhao123 wants to merge 1 commit into
apache:masterfrom
Yangxuhao123:fix/zk-deregister-watcher-rearm

Conversation

@Yangxuhao123

@Yangxuhao123 Yangxuhao123 commented Jul 11, 2026

Copy link
Copy Markdown

Why are the changes needed?

Closes #7556.

ZooKeeper watches are one-shot. After a Kyuubi engine's ZooKeeper session expires and reconnects, its service node is restored, but the DeRegisterWatcher is not automatically rearmed. A subsequent node deletion may therefore be missed, preventing graceful shutdown and allowing an unhealthy engine to remain discoverable.

This patch rearms the watcher asynchronously after RECONNECTED, retries when the service node is temporarily unavailable, handles node recreation, and invalidates stale or pending rewatch attempts after a newer reconnect or service deregistration.

How was this patch tested?

Added embedded ZooKeeper test coverage for:

  • Rearming the service watcher after reconnect.
  • Triggering graceful shutdown after a real RECONNECTED event followed by node deletion.
  • Rearming after an absent node is recreated.
  • Cancelling pending rewatch attempts during deregistration.
  • Rearming binary and HTTP discovery clients independently.

The following test suite was executed:

./build/mvn test -pl kyuubi-ha -am \
  -Dtest=none \
  -DwildcardSuites=org.apache.kyuubi.ha.client.zookeeper.EmbeddedZookeeperDiscoveryClientSuite

All 16 tests passed.

Was this patch authored or co-authored using generative AI tooling?

Assisted-by: OpenAI Codex (GPT-5)

@Yangxuhao123 Yangxuhao123 force-pushed the fix/zk-deregister-watcher-rearm branch 2 times, most recently from 0a3eb05 to 33c7223 Compare July 13, 2026 06:58
@pan3793 pan3793 requested a review from cxzl25 July 14, 2026 03:24
@pan3793

pan3793 commented Jul 14, 2026

Copy link
Copy Markdown
Member

@Yangxuhao123 rebase on latest master can solve the CI timeout issue.

cc @cxzl25 could you please take a look?

@Yangxuhao123 Yangxuhao123 force-pushed the fix/zk-deregister-watcher-rearm branch from 33c7223 to c9260b5 Compare July 14, 2026 09:47
" ZooKeeper. The server will be shut down after the last client session completes.")
ZookeeperDiscoveryClient.this.deregisterService()
serviceDiscovery.stopGracefully()
} else if (event.getType == Watcher.Event.EventType.NodeCreated) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What version of zookeeper are you using?

@Yangxuhao123 Yangxuhao123 Jul 15, 2026

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cxzl25 zk client is 3.4.14, zk server is 3.8.4, Is this related to zookeeper version change?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Kyuubi Fails to Rearm the Engine Deregistration Watcher After ZooKeeper Reconnect

3 participants