feat: add --resume to continue an interrupted export

[gacevicljubisa]

Adds export --resume <file> so an interrupted export continues from where it stopped instead of restarting from the contract's start block.

./dist/batch-export export --resume dist/export.ndjson
./dist/batch-export export --resume dist/export.ndjson.gzip

Design

• Format detection by magic bytes, not extension — .ndjson, .gz, and .gzip all work, which matters since this repo's own archives use .gzip.
• Inclusive resume. An interrupted run may have written only part of a block, so the last block is re-queried and entries already present are skipped via Cursor.Skip. No gaps, no duplicates.
• Append in place. Plain files use O_APPEND; gzip files get a new gzip member. Concatenated members are a valid gzip stream (RFC 1952), so gzcat, gunzip, and Go's compress/gzip read the result as one continuous file and the existing ~90 MB is never rewritten. Verified against a copy of the real archive with gzcat, gunzip -t, Go, and Python.
• --resume overrides --start and --output, warning when either was set explicitly. --compress becomes a no-op when resuming an already-compressed file.

Recovering from an unclean shutdown

A hard kill (SIGKILL, OOM, short write) can leave a partial line or a gzip member with no trailer. Appending onto any of those corrupts the file, so the reader identifies the last offset at which the file is known complete — the end of the last newline-terminated line that parses as a log entry, or of the last whole gzip member — and the export truncates to it before appending, logging what it dropped:

"level"="warning" "msg"="resume file ends with a partial write, discarding it" "offset"=89649991 "discardedBytes"=317

This is lossless for export content: everything discarded sits at or after the resume point, so the resumed query re-fetches it. If no clean boundary can be identified, resume refuses (ErrNoCleanBoundary) rather than guessing — the safety property does not depend on recovery succeeding.

Structure

Package	Change
pkg/resume	New. Reads the tail of a previous export, returns the cursor plus the clean-boundary offset.
pkg/gzipstore	AppendWriter — appends a gzip member. CompressFile unchanged.
pkg/filestore	AppendLogsAsync with a skip filter; shared writeLogs core. SaveLogsAsync signature unchanged.
cmd/export.go	Flag wiring; opens the destination up front so a failure can't hang the producer.

Dependency direction is one-way: cmd depends on all three, none of the three depend on each other — filestore takes a bare func(types.Log) bool that Cursor.Skip satisfies structurally.

Also fixes a pre-existing bug: the ctx.Done() path logged "waiting for logs to be saved..." and then returned without waiting, which could run CompressFile over a still-writing file.

Testing

The repo previously had no tests; this adds the first, covering the tail reader's edge cases (truncated tails, multi-member gzip, a valid line straddling a 64 KiB window boundary), an append-resume round trip across all three packages, and a regression test for each of the three corruption scenarios above. go test ./..., go vet ./..., and golangci-lint run are clean.

cmd/export.go has no unit test — RunE needs a live RPC endpoint and there's no HTTP fixture harness here — so the testable logic lives in pkg/. It was verified manually against copies of the real archives with the endpoint reachable.

Known limitations

• Resume assumes the file came from this tool against the same chain and contract. Guarding that would need a header, which changes the format batch-archive consumes.
• A --resume target that exists but holds no parseable entries fails rather than starting fresh.
• --compress with a plain resume can overwrite an existing .gzip of the same name.
• Multi-member gzip is legal and transparently readable, but if anything in batch-archive parses gzip by hand rather than through a standard library, that is where it would surface — worth a glance before this is relied on.

🤖 Generated with Claude Code