Skip to content

runner-extract: per-declaration/connection row and byte caps (512MB truncation is silent data loss at scale) #50

Description

@ihistand

From the acuantia migration dogfood: acc_mariadb.sales_order — a core business table — hit the extract guard rails (1M rows / 512MB, JSON-measured) and was truncated at 72,973 rows. The run logs a warning and continues, so every downstream model computes on a fraction of the table. Fine as a branch-CI guard rail; wrong as a hard product limit.

Proposal

  • extract: { maxRows, maxBytes } config on the declaration (overrides) and on the connection (default for its declarations), with 0/"none" meaning uncapped.
  • Keep current defaults when unset (branch CI keeps its guard rails; hosted workflow runs on production data opt in to bigger/uncapped).
  • Truncation should be more visible: besides the log warning, consider failing the extract when a cap is hit unless an explicit allowTruncation: true is set — silent partial data is worse than a red action.

Also worth noting: the byte measure is JSON.stringify per row (generous); a real 512MB cap admits much less actual data.

Found in SQLAnvil Cloud run of 206-extract project; caps live in cli/api/dbadapters/bigquery_extract.ts (DEFAULT_ROW_CAP/DEFAULT_BYTE_CAP), plumbing via ExtractSpec proto + declare() in core/session.ts.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions