Bound persistence event replay batches#3325
Open
He-Pin wants to merge 1 commit into
Open
Conversation
Motivation: AsyncWriteJournal can enqueue an unbounded number of replayed events in a recovering actor mailbox and exhaust memory. Modification: Add acknowledged, configurable replay batching for classic and typed persistence while preserving replay-filter state, sender ordering, restart isolation, and plugin compatibility. Result: Recovery limits in-flight replay events without truncating recovered history. Tests: - Focused classic protocol, replay filter, and replay batching tests: 24 passed - Focused typed replay batching tests: 2 passed - sbt "persistence / Test / test": 63 passed; LevelDB JNI emitted an Apple ARM64/x86 architecture warning - sbt "persistence-typed-tests / Test / test": 204 passed - sbt "persistence-shared / Test / testOnly org.apache.pekko.persistence.journal.leveldb.PersistencePluginProxySpec": 1 passed - sbt headerCreateAll, +headerCheckAll, checkCodeStyle, docs/paradox, and +mimaReportBinaryIssues: passed; Paradox reported duplicate-anchor warnings - scalafmt --list --mode diff-ref=origin/main and git diff --check: passed - sbt sortImports: environment/task failure; the project-scoped key was unavailable and root scalafix raised NoSuchMethodError - sbt validatePullRequest: failed after 28m26s because LevelDB JNI 1.8 is x86-only on macOS ARM64 and unrelated docs tests failed (UnboundedMailbox classpath and UDP multicast interface timeouts) References: Fixes apache#3241
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
AsyncWriteJournalcan emit an entire recovery stream without waiting for the recovering actor to process it. A large journal can therefore enqueue an unbounded number of replay messages in the persistent actor mailbox and exhaust the heap.Modification
ReplayMessagesrequest for compatibility with journal plugins that do not understand batching.Result
Recovery bounds the number of replayed events in flight without truncating recovered history. Existing journal plugins continue to use the legacy replay path unless they support the opt-in batching marker.
Tests
sbt "persistence / Test / test": 63 passed; LevelDB JNI emitted an Apple ARM64/x86 architecture warningsbt "persistence-typed-tests / Test / test": 204 passedsbt "persistence-shared / Test / testOnly org.apache.pekko.persistence.journal.leveldb.PersistencePluginProxySpec": 1 passedsbt headerCreateAll,sbt +headerCheckAll,sbt checkCodeStyle,sbt docs/paradox, andsbt +mimaReportBinaryIssues: passed; Paradox reported duplicate-anchor warningsscalafmt --list --mode diff-ref=origin/mainandgit diff --check: passedsbt sortImports: environment/task failure; the project-scoped key was unavailable and root scalafix raisedNoSuchMethodErrorsbt validatePullRequest: ran for 28m26s and failed on this macOS ARM64 host because LevelDB JNI 1.8 contains x86 binaries; unrelated docs tests also failed from a missingpekko.dispatch.UnboundedMailboxclass and UDP multicast timeouts on all local interfacesReferences
Fixes #3241