Skip to content

Bound persistence event replay batches#3325

Open
He-Pin wants to merge 1 commit into
apache:mainfrom
He-Pin:fix/async-replay-mailbox-oom
Open

Bound persistence event replay batches#3325
He-Pin wants to merge 1 commit into
apache:mainfrom
He-Pin:fix/async-replay-mailbox-oom

Conversation

@He-Pin

@He-Pin He-Pin commented Jul 11, 2026

Copy link
Copy Markdown
Member

Motivation

AsyncWriteJournal can emit an entire recovery stream without waiting for the recovering actor to process it. A large journal can therefore enqueue an unbounded number of replay messages in the persistent actor mailbox and exhaust the heap.

Modification

  • Add configurable, acknowledged replay batching with a default batch size of 1000.
  • Let classic and typed persistent actors opt in while retaining the existing ReplayMessages request for compatibility with journal plugins that do not understand batching.
  • Preserve replay limits, ordering, sender semantics, replay-filter state, cancellation, and recovery isolation across actor or journal restarts.
  • Forward the batching protocol through the persistence plugin proxy.
  • Add focused classic, typed, replay-filter, protocol, and remote-proxy regression coverage.
  • Document the replay batching behavior and configuration.

Result

Recovery bounds the number of replayed events in flight without truncating recovered history. Existing journal plugins continue to use the legacy replay path unless they support the opt-in batching marker.

Tests

  • Focused classic protocol, replay filter, and replay batching tests: 24 passed
  • Focused typed replay batching tests: 2 passed
  • sbt "persistence / Test / test": 63 passed; LevelDB JNI emitted an Apple ARM64/x86 architecture warning
  • sbt "persistence-typed-tests / Test / test": 204 passed
  • sbt "persistence-shared / Test / testOnly org.apache.pekko.persistence.journal.leveldb.PersistencePluginProxySpec": 1 passed
  • sbt headerCreateAll, sbt +headerCheckAll, sbt checkCodeStyle, sbt docs/paradox, and sbt +mimaReportBinaryIssues: passed; Paradox reported duplicate-anchor warnings
  • scalafmt --list --mode diff-ref=origin/main and git diff --check: passed
  • sbt sortImports: environment/task failure; the project-scoped key was unavailable and root scalafix raised NoSuchMethodError
  • sbt validatePullRequest: ran for 28m26s and failed on this macOS ARM64 host because LevelDB JNI 1.8 contains x86 binaries; unrelated docs tests also failed from a missing pekko.dispatch.UnboundedMailbox class and UDP multicast timeouts on all local interfaces

References

Fixes #3241

Motivation:
AsyncWriteJournal can enqueue an unbounded number of replayed events in a recovering actor mailbox and exhaust memory.

Modification:
Add acknowledged, configurable replay batching for classic and typed persistence while preserving replay-filter state, sender ordering, restart isolation, and plugin compatibility.

Result:
Recovery limits in-flight replay events without truncating recovered history.

Tests:
- Focused classic protocol, replay filter, and replay batching tests: 24 passed
- Focused typed replay batching tests: 2 passed
- sbt "persistence / Test / test": 63 passed; LevelDB JNI emitted an Apple ARM64/x86 architecture warning
- sbt "persistence-typed-tests / Test / test": 204 passed
- sbt "persistence-shared / Test / testOnly org.apache.pekko.persistence.journal.leveldb.PersistencePluginProxySpec": 1 passed
- sbt headerCreateAll, +headerCheckAll, checkCodeStyle, docs/paradox, and +mimaReportBinaryIssues: passed; Paradox reported duplicate-anchor warnings
- scalafmt --list --mode diff-ref=origin/main and git diff --check: passed
- sbt sortImports: environment/task failure; the project-scoped key was unavailable and root scalafix raised NoSuchMethodError
- sbt validatePullRequest: failed after 28m26s because LevelDB JNI 1.8 is x86-only on macOS ARM64 and unrelated docs tests failed (UnboundedMailbox classpath and UDP multicast interface timeouts)

References:
Fixes apache#3241
@He-Pin He-Pin added this to the 2.0.0-M4 milestone Jul 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

asyncReplayMessages can cause mailbox OOM for large event logs

1 participant