Skip to content

fix: stream and async R2 database dumps#202

Open
akamabu wants to merge 1 commit into
outerbase:mainfrom
akamabu:fix/issue-59-r2-async-exports
Open

fix: stream and async R2 database dumps#202
akamabu wants to merge 1 commit into
outerbase:mainfrom
akamabu:fix/issue-59-r2-async-exports

Conversation

@akamabu
Copy link
Copy Markdown

@akamabu akamabu commented May 13, 2026

/claim #59

Summary

Fixes #59 by moving database dumps away from building the entire dump in memory and adding an R2-backed asynchronous export path for large databases.

What changed

  • /export/dump now returns a ReadableStream instead of assembling a single giant string/blob in memory.
  • Table data is exported in bounded LIMIT/OFFSET pages (EXPORT_CHUNK_SIZE configurable, default 500).
  • SQL identifiers and values are safely escaped for generated INSERT statements.
  • Added optional async mode: GET /export/dump?async=true
    • Requires EXPORT_BUCKET R2 binding.
    • Writes dump_YYYYMMDD-HHMMSSZ.sql to R2 by default.
    • Supports filename= query parameter.
    • Supports callback notifications via callbackUrl= or EXPORT_CALLBACK_URL.
    • Uses ctx.waitUntil when available so the request can return 202 Accepted while R2 upload continues.
  • Added Env/config wiring for EXPORT_BUCKET, EXPORT_CALLBACK_URL, and EXPORT_CHUNK_SIZE.
  • Expanded dump tests for streaming, pagination, escaping, async R2 upload, custom filename, and callback behavior.

Testing

  • npx vitest run src/export/dump.test.ts
  • npx vitest run was also run; the new dump tests pass, while 4 pre-existing unrelated RLS tests fail in src/rls/index.test.ts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Database dumps do not work on large databases

1 participant