Skip to content

fix(seidb): refuse a digest replay that would repair the live changelog - #3983

Open
blindchaser wants to merge 7 commits into
mainfrom
fix/seidb-digest-readonly-wal
Open

fix(seidb): refuse a digest replay that would repair the live changelog#3983
blindchaser wants to merge 7 commits into
mainfrom
fix/seidb-digest-readonly-wal

Conversation

@blindchaser

@blindchaser blindchaser commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Summary

seidb evm-logical-digest --memiavl-open-mode replay opens the changelog of a live node. The changelog opener repairs what it finds: it truncates a tail that ends mid-record, and completes an interrupted truncation. On a live node that torn tail is usually a write in progress rather than lasting damage, so the repair can discard a committed record. Options.ReadOnly does not prevent it.

Check the changelog before the digest opens it, and refuse the run instead. The operator reruns.

  • sei-db/wal/utils.go: add VerifyIntact, which reports whether the log in a directory can be opened without repair. It reads the tail segment and returns ErrCorrupt for a record that ends mid-frame or for a truncation marker left behind. It never creates, truncates, renames, or removes a file. It sits beside truncateCorruptedTail, the repair it exists to avoid.
  • sei-db/tools/cmd/seidb/operations/evm_logical_digest.go: call it from replay mode before the open, and report that the live WAL was not modified. Then check that the replay actually covered the height it reports. MultiTree.Catchup starts at the changelog's first offset whenever the snapshot ends before it, so a changelog pruned past the snapshot replays a contiguous suffix, reaches the requested height, and silently omits the versions in between. The final version alone does not reveal that, and a digest missing intermediate versions reads as a state mismatch between nodes.

memIAVL and the changelog opener are unchanged, so every other reader keeps its current behavior. Both checks live in openMemiAVLReplayReadOnly, the one function every digest replay open passes through. The silent-gap replay is a memIAVL property rather than a digest one, so other readers still have it; guarding it there is a separate change.

The check and the open are separate steps, so a writer can still tear the tail in the millisecond between them. That window is far smaller than the run it guards, and the outcome is the same rerun message.

Test plan

  • sei-db/wal/wal_test.go: accept a complete log, a missing one, and an empty one; reject a torn tail and an interrupted truncation; assert the directory bytes are identical after every rejection; pin that open truncates the same tail the check rejects.
  • sei-db/tools/cmd/seidb/operations/memiavl_open_test.go: replay mode refuses a torn changelog, tells the operator to rerun, and leaves the segment byte-identical; refuses a changelog pruned past the snapshot even though the replay reaches the requested height; accepts a height the snapshot and changelog fully cover.
  • go test -race ./sei-db/wal ./sei-db/state_db/sc/memiavl ./sei-db/tools/cmd/seidb/operations
  • make dblint

Use an immutable WAL view for read-only replay so digest tooling never repairs or copies the live changelog. Return actionable retry errors when a point-in-time view cannot safely reach the requested version.

Co-authored-by: Cursor <cursoragent@cursor.com>
@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.22222% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.88%. Comparing base (25ac13a) to head (9075ecb).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...b/tools/cmd/seidb/operations/evm_logical_digest.go 64.28% 6 Missing and 4 partials ⚠️
sei-db/wal/utils.go 80.76% 3 Missing and 2 partials ⚠️
Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3983      +/-   ##
==========================================
- Coverage   58.95%   57.88%   -1.08%     
==========================================
  Files        2306     2207      -99     
  Lines      197441   185925   -11516     
==========================================
- Hits       116410   107618    -8792     
+ Misses      70277    68484    -1793     
+ Partials    10754     9823     -931     
Flag Coverage Δ
sei-chain-pr 35.57% <72.22%> (?)
sei-db 69.80% <ø> (ø)
sei-db-state-db ?

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sei-db/wal/utils.go 72.04% <80.76%> (+15.32%) ⬆️
...b/tools/cmd/seidb/operations/evm_logical_digest.go 24.91% <64.28%> (+2.00%) ⬆️

... and 99 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Aug 21, 2026

Copy link
Copy Markdown

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedAug 22, 2026, 3:04 AM

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 33c6cde16d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sei-db/wal/readonly.go Outdated
Comment thread sei-db/wal/readonly.go Outdated
Treat a segment removed between directory listing and open as WAL churn so read-only callers return the actionable retry path.

Co-authored-by: Cursor <cursoragent@cursor.com>

@seidroid seidroid Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid change: the immutable read-only changelog reader correctly avoids tidwall's writable open (no file creation, no tail truncation, no .START/.END recovery completion), retains fds for a stable point-in-time view, and the new OpenDB cleanup defer fixes real mtree/WAL/file-lock leaks on failure. No blockers, but the fail-closed policy now also applies to production read-only callers (CommitStore.Exporter, LoadVersion(v, true)) that have no retry, and a few guards/classifications could be tightened.

Findings: 0 blocking | 6 non-blocking | 4 posted inline

Blockers

  • None at the file/PR level.

Non-blocking

  • [suggestion] openReadOnlyWAL eagerly io.ReadAlls every segment in the changelog directory and keeps one fd open per segment for the lifetime of the view. With the default 20 MB tidwall segment size and a multi-GB changelog that is hundreds of fds plus a full sequential read at open, even though MultiTree.Catchup typically only replays a suffix starting at the selected snapshot version, and each entry is then pread again on ReadAt (so the bytes are read twice). Consider indexing segments lazily, or at least skipping segments entirely below the first needed offset.
  • [suggestion] Options.ReadOnly is not only the seidb replay tool: memiavl.CommitStore.Exporter (state-sync snapshot export, when OnlyAllowExportOnSnapshotVersion is false) and CommitStore.LoadVersion(v, true) / CompositeCommitStore.LoadVersionReadOnly also flow through it, and they now inherit both the fail-closed WAL open and the new strict mtree.Version() != targetVersion check. The test plan covers only the WAL package, db_test.go, and the CLI. A memiavl-level test that opens read-only while a writer is committing in the same process (the analogue of TestOpenReadOnlyChangelogWALConcurrentWriter) would pin the behavior those callers actually see.
  • 4 suggestion(s)/nit(s) flagged inline on specific lines.

Comment thread sei-db/wal/readonly.go Outdated
Comment thread sei-db/state_db/sc/memiavl/db.go Outdated
Comment thread sei-db/tools/cmd/seidb/operations/evm_logical_digest.go Outdated
Comment thread sei-db/state_db/sc/memiavl/db.go Outdated
Use the complete tail prefix during concurrent writes, treat a missing changelog as empty, and classify deferred read failures through the retryable WAL error path.

Co-authored-by: Cursor <cursoragent@cursor.com>
@cursor

cursor Bot commented Aug 21, 2026

Copy link
Copy Markdown

PR Summary

Medium Risk
Touches changelog integrity checks used against live node data; a false positive/negative in VerifyIntact or coverage logic could block valid digest runs or still allow a truncated/gapped replay. The memIAVL opener itself is unchanged.

Overview
Stops seidb evm-logical-digest --memiavl-open-mode replay from opening a live memIAVL changelog that OpenDB would auto-repair (even in ReadOnly), which can truncate a write-in-progress tail.

Adds read-only wal.VerifyIntact and calls it before replay open, asking the operator to rerun instead of mutating the WAL. After open, verifyReplayCoverage rejects a changelog pruned past the snapshot so a suffix replay cannot silently skip versions while still reaching the requested height.

Reviewed by Cursor Bugbot for commit 9075ecb. Bugbot is set up for automated code reviews on this repo. Configure here.

blindchaser and others added 2 commits August 21, 2026 17:04
Keep existing memiavl read-only callers on their prior WAL path and require digest replay to opt into immutable, fail-loud access.

Co-authored-by: Cursor <cursoragent@cursor.com>
The digest replay mode refused a torn changelog through a purpose-built
read-only WAL reader and an opt-in memIAVL mode. A check before the open
reaches the same outcome without either: the opener repairs only the tail
segment, so reading that one segment answers whether the open would repair
anything.

This leaves memIAVL and the changelog opener untouched.

Co-authored-by: Cursor <cursoragent@cursor.com>
@blindchaser blindchaser changed the title fix(seidb): fail closed on live memiavl WAL reads fix(seidb): refuse a digest replay that would repair the live changelog Aug 21, 2026
_ = db.Close()
return nil, fmt.Errorf("memiavl replay reached version %d, not the requested height %d; "+
"the changelog does not cover that height", reached, height)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pruned WAL gap check removed

High Severity

Removing FailOnWALRepair also dropped the immutable-view check that the changelog still covers every version after the selected snapshot. Replay now only compares the final Version() to --height, so a pruned gap can replay a contiguous suffix, reach the requested height, and return a digest that silently omitted intermediate versions.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit dcdfa41. Configure here.

blindchaser and others added 2 commits August 21, 2026 17:48
VerifyIntact sat in its own file, away from truncateCorruptedTail and
loadNextBinaryEntry, which are the repair it guards against and the framing
it reuses.

Co-authored-by: Cursor <cursoragent@cursor.com>
Catchup starts at the changelog's first offset whenever the snapshot ends
before it, so a changelog pruned past the snapshot replays a contiguous
suffix, reaches the requested height, and omits the versions in between.
Comparing the final version to --height does not catch that, and a digest
missing intermediate versions reads as a state mismatch between nodes.

Co-authored-by: Cursor <cursoragent@cursor.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 9075ecb. Configure here.

"%d, so versions %d-%d would be missing from the replay; digest a height at or below %d, "+
"or use --memiavl-open-mode snapshot",
firstVersion, snapshotVersion, snapshotVersion+1, firstVersion-1, snapshotVersion)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gap check rejects InitialVersion jumps

Medium Severity

verifyReplayCoverage flags a gap whenever firstVersion > snapshotVersion+1. That matches a pruned changelog, but it also matches a normal InitialVersion > 1 chain whose snapshot is still at version 0: the first WAL entry is at the initial height, versions below it never existed, and Catchup correctly starts there. Until the first snapshot rewrite, replay digests on such chains are refused even though coverage is complete. The prune case is when the expected next version maps to a positive WAL index below FirstOffset, which is what Catchup already clamps on.

Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 9075ecb. Configure here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant