fix(seidb): refuse a digest replay that would repair the live changelog - #3983
fix(seidb): refuse a digest replay that would repair the live changelog#3983blindchaser wants to merge 7 commits into
Conversation
Use an immutable WAL view for read-only replay so digest tooling never repairs or copies the live changelog. Return actionable retry errors when a point-in-time view cannot safely reach the requested version. Co-authored-by: Cursor <cursoragent@cursor.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #3983 +/- ##
==========================================
- Coverage 58.95% 57.88% -1.08%
==========================================
Files 2306 2207 -99
Lines 197441 185925 -11516
==========================================
- Hits 116410 107618 -8792
+ Misses 70277 68484 -1793
+ Partials 10754 9823 -931
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
|
The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 33c6cde16d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Treat a segment removed between directory listing and open as WAL churn so read-only callers return the actionable retry path. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Solid change: the immutable read-only changelog reader correctly avoids tidwall's writable open (no file creation, no tail truncation, no .START/.END recovery completion), retains fds for a stable point-in-time view, and the new OpenDB cleanup defer fixes real mtree/WAL/file-lock leaks on failure. No blockers, but the fail-closed policy now also applies to production read-only callers (CommitStore.Exporter, LoadVersion(v, true)) that have no retry, and a few guards/classifications could be tightened.
Findings: 0 blocking | 6 non-blocking | 4 posted inline
Blockers
- None at the file/PR level.
Non-blocking
- [suggestion]
openReadOnlyWALeagerlyio.ReadAlls every segment in the changelog directory and keeps one fd open per segment for the lifetime of the view. With the default 20 MB tidwall segment size and a multi-GB changelog that is hundreds of fds plus a full sequential read at open, even thoughMultiTree.Catchuptypically only replays a suffix starting at the selected snapshot version, and each entry is thenpreadagain onReadAt(so the bytes are read twice). Consider indexing segments lazily, or at least skipping segments entirely below the first needed offset. - [suggestion]
Options.ReadOnlyis not only the seidb replay tool:memiavl.CommitStore.Exporter(state-sync snapshot export, whenOnlyAllowExportOnSnapshotVersionis false) andCommitStore.LoadVersion(v, true)/CompositeCommitStore.LoadVersionReadOnlyalso flow through it, and they now inherit both the fail-closed WAL open and the new strictmtree.Version() != targetVersioncheck. The test plan covers only the WAL package,db_test.go, and the CLI. A memiavl-level test that opens read-only while a writer is committing in the same process (the analogue ofTestOpenReadOnlyChangelogWALConcurrentWriter) would pin the behavior those callers actually see. - 4 suggestion(s)/nit(s) flagged inline on specific lines.
Use the complete tail prefix during concurrent writes, treat a missing changelog as empty, and classify deferred read failures through the retryable WAL error path. Co-authored-by: Cursor <cursoragent@cursor.com>
PR SummaryMedium Risk Overview Adds read-only Reviewed by Cursor Bugbot for commit 9075ecb. Bugbot is set up for automated code reviews on this repo. Configure here. |
Keep existing memiavl read-only callers on their prior WAL path and require digest replay to opt into immutable, fail-loud access. Co-authored-by: Cursor <cursoragent@cursor.com>
The digest replay mode refused a torn changelog through a purpose-built read-only WAL reader and an opt-in memIAVL mode. A check before the open reaches the same outcome without either: the opener repairs only the tail segment, so reading that one segment answers whether the open would repair anything. This leaves memIAVL and the changelog opener untouched. Co-authored-by: Cursor <cursoragent@cursor.com>
| _ = db.Close() | ||
| return nil, fmt.Errorf("memiavl replay reached version %d, not the requested height %d; "+ | ||
| "the changelog does not cover that height", reached, height) | ||
| } |
There was a problem hiding this comment.
Pruned WAL gap check removed
High Severity
Removing FailOnWALRepair also dropped the immutable-view check that the changelog still covers every version after the selected snapshot. Replay now only compares the final Version() to --height, so a pruned gap can replay a contiguous suffix, reach the requested height, and return a digest that silently omitted intermediate versions.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit dcdfa41. Configure here.
VerifyIntact sat in its own file, away from truncateCorruptedTail and loadNextBinaryEntry, which are the repair it guards against and the framing it reuses. Co-authored-by: Cursor <cursoragent@cursor.com>
Catchup starts at the changelog's first offset whenever the snapshot ends before it, so a changelog pruned past the snapshot replays a contiguous suffix, reaches the requested height, and omits the versions in between. Comparing the final version to --height does not catch that, and a digest missing intermediate versions reads as a state mismatch between nodes. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 9075ecb. Configure here.
| "%d, so versions %d-%d would be missing from the replay; digest a height at or below %d, "+ | ||
| "or use --memiavl-open-mode snapshot", | ||
| firstVersion, snapshotVersion, snapshotVersion+1, firstVersion-1, snapshotVersion) | ||
| } |
There was a problem hiding this comment.
Gap check rejects InitialVersion jumps
Medium Severity
verifyReplayCoverage flags a gap whenever firstVersion > snapshotVersion+1. That matches a pruned changelog, but it also matches a normal InitialVersion > 1 chain whose snapshot is still at version 0: the first WAL entry is at the initial height, versions below it never existed, and Catchup correctly starts there. Until the first snapshot rewrite, replay digests on such chains are refused even though coverage is complete. The prune case is when the expected next version maps to a positive WAL index below FirstOffset, which is what Catchup already clamps on.
Reviewed by Cursor Bugbot for commit 9075ecb. Configure here.


Summary
seidb evm-logical-digest --memiavl-open-mode replayopens the changelog of a live node. The changelog opener repairs what it finds: it truncates a tail that ends mid-record, and completes an interrupted truncation. On a live node that torn tail is usually a write in progress rather than lasting damage, so the repair can discard a committed record.Options.ReadOnlydoes not prevent it.Check the changelog before the digest opens it, and refuse the run instead. The operator reruns.
sei-db/wal/utils.go: addVerifyIntact, which reports whether the log in a directory can be opened without repair. It reads the tail segment and returnsErrCorruptfor a record that ends mid-frame or for a truncation marker left behind. It never creates, truncates, renames, or removes a file. It sits besidetruncateCorruptedTail, the repair it exists to avoid.sei-db/tools/cmd/seidb/operations/evm_logical_digest.go: call it from replay mode before the open, and report that the live WAL was not modified. Then check that the replay actually covered the height it reports.MultiTree.Catchupstarts at the changelog's first offset whenever the snapshot ends before it, so a changelog pruned past the snapshot replays a contiguous suffix, reaches the requested height, and silently omits the versions in between. The final version alone does not reveal that, and a digest missing intermediate versions reads as a state mismatch between nodes.memIAVL and the changelog opener are unchanged, so every other reader keeps its current behavior. Both checks live in
openMemiAVLReplayReadOnly, the one function every digest replay open passes through. The silent-gap replay is a memIAVL property rather than a digest one, so other readers still have it; guarding it there is a separate change.The check and the open are separate steps, so a writer can still tear the tail in the millisecond between them. That window is far smaller than the run it guards, and the outcome is the same rerun message.
Test plan
sei-db/wal/wal_test.go: accept a complete log, a missing one, and an empty one; reject a torn tail and an interrupted truncation; assert the directory bytes are identical after every rejection; pin thatopentruncates the same tail the check rejects.sei-db/tools/cmd/seidb/operations/memiavl_open_test.go: replay mode refuses a torn changelog, tells the operator to rerun, and leaves the segment byte-identical; refuses a changelog pruned past the snapshot even though the replay reaches the requested height; accepts a height the snapshot and changelog fully cover.go test -race ./sei-db/wal ./sei-db/state_db/sc/memiavl ./sei-db/tools/cmd/seidb/operationsmake dblint