Skip to content

[fix][broker] Recover bucket delayed delivery tracker asynchronously to avoid blocking dispatcher threads#26105

Open
void-ptr974 wants to merge 1 commit into
apache:masterfrom
void-ptr974:fix/bucket-delayed-tracker-recovery
Open

[fix][broker] Recover bucket delayed delivery tracker asynchronously to avoid blocking dispatcher threads#26105
void-ptr974 wants to merge 1 commit into
apache:masterfrom
void-ptr974:fix/bucket-delayed-tracker-recovery

Conversation

@void-ptr974

Copy link
Copy Markdown
Contributor

Fixes #26103

Motivation

BucketDelayedDeliveryTracker is lazily created from the broker dispatch path
when the first delayed message needs a delayed delivery tracker.

For the bucket implementation, tracker creation also recovers delayed bucket
snapshots. This recovery can depend on metadata or storage operations, but it was
previously waited for synchronously during tracker creation.

As a result, a dispatcher thread could be blocked while holding the dispatcher
lock. If bucket snapshot recovery is slow, dispatch for the current subscription
is stalled, and other tasks queued on the same broker-topic worker can also be
delayed.

Modifications

  • Add an asynchronous tracker creation path to DelayedDeliveryTrackerFactory.
  • Recover bucket delayed delivery snapshots asynchronously before installing the
    tracker into the dispatcher.
  • Keep delayed messages pending while tracker creation is in progress, and pause
    dispatch until the tracker is ready.
  • Replay pending delayed messages when tracker creation fails or when their
    delivery time has already elapsed.
  • Fall back to the in-memory delayed delivery tracker when bucket recovery fails
    or times out.
  • Clean up pending delayed messages on dispatcher close/unload, and close
    trackers that complete after they are no longer needed.
  • Add lifecycle logs for tracker creation start, pending creation, completion,
    failure, fallback, and stale tracker cleanup.
  • Add tests for async bucket recovery, fallback behavior, dispatcher
    close/clear/replay behavior, and a broker-path delayed message scenario using
    the real bucket tracker.

Verifying this change

  • Make sure that the change passes the CI checks.

This change added tests and can be verified as follows:

  • ./gradlew :pulsar-broker:test --tests org.apache.pulsar.broker.delayed.DelayedDeliveryTrackerFactoryTest
  • ./gradlew :pulsar-broker:checkstyleMain :pulsar-broker:checkstyleTest
  • git diff --check

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

This changes delayed delivery tracker creation from synchronous recovery on the
dispatcher path to asynchronous recovery with dispatcher-side pending state and
replay handling.

@void-ptr974 void-ptr974 force-pushed the fix/bucket-delayed-tracker-recovery branch from a8e8932 to b497567 Compare June 28, 2026 15:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Bucket delayed delivery recovery can block dispatcher threads

1 participant