Skip to content

fix: stream SQL database dumps#206

Open
Treasure520520 wants to merge 1 commit into
outerbase:mainfrom
Treasure520520:bounty/59-streaming-dump
Open

fix: stream SQL database dumps#206
Treasure520520 wants to merge 1 commit into
outerbase:mainfrom
Treasure520520:bounty/59-streaming-dump

Conversation

@Treasure520520
Copy link
Copy Markdown

/claim #59

Summary

This is a focused streaming slice for the large database dump issue.

Instead of building the entire SQL dump in a single in-memory string/blob, /export/dump now returns a ReadableStream and writes the dump incrementally:

  • emits the existing SQLite header first
  • loads table schemas one table at a time
  • reads table rows in bounded LIMIT ? OFFSET ? pages
  • streams each batch of INSERT statements as it is produced
  • quotes table identifiers and escapes SQL values, including NULL, booleans, bigint, and binary values
  • preserves the existing download headers and response shape for callers

This does not attempt to solve the full async/R2 continuation path in one PR. It removes the immediate O(database dump size) string allocation while keeping the existing endpoint behavior intact, so it can compose with later R2/alarm work.

Validation

  • ./node_modules/.bin/prettier --check src/export/dump.ts src/export/dump.test.ts
  • ./node_modules/.bin/vitest run src/export/dump.test.ts (6 tests)
  • git diff --check

The repo-wide ./node_modules/.bin/tsc --noEmit is still blocked by pre-existing type errors outside this PR (src/do.ts, src/operation.ts, existing tests). The focused dump tests and formatting checks pass.

@Treasure520520
Copy link
Copy Markdown
Author

Adding a short demo GIF for the Algora bounty review guideline:

StarbaseDB #59 streaming dump demo

It summarizes the focused change in this PR: stream the SQL dump response, read rows in bounded pages, and avoid a single full-database dump string allocation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant