Skip to content

docs(ecosystem): add Kafka SASL_SSL PTR-lookup troubleshooting guide#810

Open
chideat wants to merge 1 commit into
mainfrom
docs/kafka-sasl-ssl-ptr-lookup
Open

docs(ecosystem): add Kafka SASL_SSL PTR-lookup troubleshooting guide#810
chideat wants to merge 1 commit into
mainfrom
docs/kafka-sasl-ssl-ptr-lookup

Conversation

@chideat

@chideat chideat commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds a solution guide for Kafka SASL_SSL connections that are slow or time out because the broker performs a synchronous reverse DNS (PTR) lookup on the peer IP while registering each new connection (KAFKA-8562). When unknown Pod/CNI/SNAT addresses have no PTR record and CoreDNS forwards the query to an unreachable upstream, the broker's network processor threads block, the accept queue saturates, and clients see slow connects, metadata timeouts, and repeated SSL handshake failed messages.

The guide covers:

  • Issue & background — how the SASL_SSL channel-builder path triggers the lookup, and why the peer IP is the post-SNAT source address.
  • Diagnosis — listener/advertised-endpoint checks, in-broker TCP test, kernel accept-queue inspection (ss and /proc/net/tcp), FD/JVM exclusion, JVM thread dumps, reverse-DNS latency measurement, and CoreDNS resolution-flow inspection.
  • Root cause — the combination of Kafka's synchronous PTR lookup and CoreDNS forwarding unknown Pod PTRs to an unreachable upstream.
  • Resolution — make CoreDNS authoritative for the Pod reverse zone so unknown addresses get a fast NXDOMAIN, with backup, validation, and rollback steps.
  • Long-term remediation — the supported upgrade path to a Kafka version containing the KAFKA-8562 fix (2.7.2 / 2.8.1 / 3.0.0).

Verification

All external claims were cross-checked against authoritative sources before merge:

  • KAFKA-8562 title and fix versions (2.7.2 / 2.8.1 / 3.0.0) — Apache JIRA + the 2.7.2/2.8.1/3.0.0 release notes.
  • Reverse-DNS code path — Kafka 2.7.0 source (SaslChannelBuilder.buildTransportLayer calling getHostName() with no client/server guard) and fix PR KAFKA-8562: SaslChannelBuilder - avoid (reverse) DNS lookup while building underlying SslTransportLayer apache/kafka#10059.
  • Strimzi 0.25.0 ships exactly Kafka 2.7.0 / 2.7.1 / 2.8.0 (none containing the fix) — kafka-versions.yaml at the 0.25.0 tag.
  • CoreDNS PTR generation, fallthrough, zone authority, and forward/hosts behavior — CoreDNS plugin docs and source.

🤖 Generated with Claude Code

Add a solution guide for Kafka SASL_SSL connections that are slow or time out due to reverse DNS (PTR) lookups on the broker connection path (KAFKA-8562), covering diagnosis (listener/accept-queue/thread-dump/DNS-latency checks) and a CoreDNS reverse-zone fix that returns fast NXDOMAIN for unknown Pod addresses.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant