Fix store checkpoint ordering#40
Open
mattfaltyn wants to merge 2 commits into
Open
Conversation
mattfaltyn
marked this pull request as ready for review
July 19, 2026 20:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stores::take_all_pending_ingestsbatching APIRoot cause
Store::take_pending_ingestpreviously wrote its completed-height metadatabefore the returned deferred ingest closure ran. A crash or ingest failure in
that window left a checkpoint claiming a height was complete even though its
transaction, address, or UTXO mutations were missing. Restart then trusted the
checkpoint and skipped the missing work.
This change treats the height as the final commit marker. Forward commits now
ingest all pending mutations, persist Fjall with
SyncData, and only thenpublish checkpoints. The background indexer persists vector data before
running that ordered store commit. Rollback uses the conservative inverse:
persist the lowered checkpoints first, then persist destructive removals.
If a crash leaves vectors ahead of store checkpoints, startup now selects the
lower shared durable tip. Store rollback consumes the still-ahead vector
history before vector truncation, then normal replay resumes without a full
rebuild. The former public batching method remains available as a compatibility
wrapper around the single ordered task.
Impact
Interrupted indexing can no longer leave store metadata ahead of durable Fjall
data. Data or vectors may lead a checkpoint after a failure, which is safe
because BRK recovers from the shared checkpoint and replays the affected
height.
Validation
cargo test -p brk_store -p brk_indexer --libcargo test --workspacecargo check -p brk_indexer --all-targetscargo clippy -p brk_store -p brk_indexer --lib --tests --no-deps -- -D warningscargo package --workspace --allow-dirty --no-verifyrustfmt --checkon the changed store files andcrates/brk_indexer/src/stores.rsgit diff --checkAll commands above pass. A full-file rustfmt check of
crates/brk_indexer/src/lib.rsstill reports an unchanged pre-existingformatting difference outside this PR's hunks.
The repository's unchanged JavaScript/Python release harnesses were also
attempted. They are currently broken on
main: the generated-client tests usethe removed
series.pricespath, and the Python mempool-compat suite importsuntracked
_lib.pyand_endpoints.pyhelpers. Neither suite exercises thechanged Rust persistence flow.
Fixes #39