From the acuantia migration dogfood: acc_mariadb.sales_order — a core business table — hit the extract guard rails (1M rows / 512MB, JSON-measured) and was truncated at 72,973 rows. The run logs a warning and continues, so every downstream model computes on a fraction of the table. Fine as a branch-CI guard rail; wrong as a hard product limit.
Proposal
extract: { maxRows, maxBytes } config on the declaration (overrides) and on the connection (default for its declarations), with 0/"none" meaning uncapped.
- Keep current defaults when unset (branch CI keeps its guard rails; hosted workflow runs on production data opt in to bigger/uncapped).
- Truncation should be more visible: besides the log warning, consider failing the extract when a cap is hit unless an explicit
allowTruncation: true is set — silent partial data is worse than a red action.
Also worth noting: the byte measure is JSON.stringify per row (generous); a real 512MB cap admits much less actual data.
Found in SQLAnvil Cloud run of 206-extract project; caps live in cli/api/dbadapters/bigquery_extract.ts (DEFAULT_ROW_CAP/DEFAULT_BYTE_CAP), plumbing via ExtractSpec proto + declare() in core/session.ts.
From the acuantia migration dogfood:
acc_mariadb.sales_order— a core business table — hit the extract guard rails (1M rows / 512MB, JSON-measured) and was truncated at 72,973 rows. The run logs a warning and continues, so every downstream model computes on a fraction of the table. Fine as a branch-CI guard rail; wrong as a hard product limit.Proposal
extract: { maxRows, maxBytes }config on the declaration (overrides) and on the connection (default for its declarations), with0/"none"meaning uncapped.allowTruncation: trueis set — silent partial data is worse than a red action.Also worth noting: the byte measure is
JSON.stringifyper row (generous); a real 512MB cap admits much less actual data.Found in SQLAnvil Cloud run of 206-extract project; caps live in
cli/api/dbadapters/bigquery_extract.ts(DEFAULT_ROW_CAP/DEFAULT_BYTE_CAP), plumbing viaExtractSpecproto +declare()incore/session.ts.