fix(clustering): surface silent cache-transport failures and make rewire retryable (#36803) - #36864
Open
wezell wants to merge 1 commit into
Open
fix(clustering): surface silent cache-transport failures and make rewire retryable (#36803)#36864wezell wants to merge 1 commit into
wezell wants to merge 1 commit into
Conversation
Cache invalidations were dropped with no log, no metric and no health signal when the pub/sub cache transport was not initialized, and a failing cluster rewire was logged and forgotten while KNOWN_SERVERS was updated as if it had succeeded (issue #36544 incident). - PubSubCacheTransport.send(): count dropped invalidations and WARN, rate-limited (CACHE_TRANSPORT_DROP_WARN_INTERVAL_MILLIS, default 30s) - PubSubCacheTransport.init(): idempotent - a rewire on a healthy transport no longer tears down/rebuilds the pub/sub listener - ClusterFactory.addMeToCacheIfNeeded(): report success/failure; only update KNOWN_SERVERS on success so failed rewires are retried; track consecutive failures in a counter exposed via getRewireFailures() - New cache-transport health check (MONITOR_MODE by default so it never fails readiness unless an operator opts in via health.check.cache-transport.mode=PRODUCTION - avoids cold-start deadlock) - Micrometer gauges: dotcms.cache.transport.invalidations.dropped, dotcms.cache.transport.initialized, dotcms.cache.transport.rewire.failures Fixes #36803 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XFuXwYdNsb7irpMEeSGTyj
7 tasks
Contributor
|
Claude Code is working… I'll analyze this and get back to you. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed Changes
Fixes #36803 — cluster cache invalidations were dropped silently when the pub/sub transport failed to initialize, and a persistently failing rewire was logged and forgotten (root cause analysis in #36544).
PubSubCacheTransport.send()no longer discards invalidations silently: increments a dropped-message counter and logs at WARN, rate-limited (CACHE_TRANSPORT_DROP_WARN_INTERVAL_MILLIS, default 30s) so a sustained outage cannot flood the log.PubSubCacheTransport.init()is idempotent: a rewire on an already-healthy transport is a no-op instead of a tear-down/rebuild of the pub/sub listener — this removes the rebuild-per-rewire-pass churn measured in Spike: Cluster re-wire re-inits Postgres pub/sub transport and leaks DB connections → pool exhaustion + startup crash-loop #36544 (LISTEN cluster_actions×3,687).ClusterFactory.addMeToCacheIfNeeded()now reports success/failure instead of swallowing exceptions.KNOWN_SERVERSis only updated on success, so a failed transport init is retried on the next heartbeat instead of being forgotten until the server set changes. Consecutive failures are tracked and exposed viaClusterFactory.getRewireFailures()(reset on success).cache-transporthealth check (CacheTransportHealthCheck, registered inCoreHealthCheckProvider): reports unhealthy when the transport is uninitialized or rewires are persistently failing; exposesinitialized/droppedInvalidations/rewireFailuresas structured data. Single-node/community (NullTransport) always reports healthy.CacheMetricsbinder:dotcms.cache.transport.invalidations.droppeddotcms.cache.transport.initializeddotcms.cache.transport.rewire.failuresCacheTransportgains adefault long getDroppedMessages()(0 for all other transports).Readiness decision (AC: "decide and document")
The health check defaults to MONITOR_MODE: it reports degradation but never fails readiness probes. A node that cannot send invalidations can still serve traffic, and gating readiness on the transport risks a cold-start deadlock (transport init happens during cluster wiring). Operators who prefer to drain such nodes can opt in with
health.check.cache-transport.mode=PRODUCTION. It is never a liveness check — restarting pods on transport failure is what amplified the #36544 incident.Testing
PubSubCacheTransportTest(unit): drops are counted and not published before init; publish works after init;init()is idempotent (singlestart()across repeated calls, re-inits aftershutdown()).docker/docker-compose-examples/pubsub-connection-churnon the spike branch) still pending — see checklist.🤖 Generated with Claude Code
https://claude.ai/code/session_01XFuXwYdNsb7irpMEeSGTyj