From the acuantia migration dogfood: a full hosted run of a converted BigQuery project spends ~35 of 40 minutes in the extract layer — 206 sources re-materialized every run, even though most change rarely. This is the structural cost of cross-warehouse sources (Dataform never moves data; our extract bridge does), so the win is to move as few bytes as possible per run.
Proposal
Skip an extract when the source is provably unchanged since the last materialization:
- BigQuery:
last_modified_time from table metadata (free lookup) vs. the recorded extract time — skip if older. Record extract watermarks in a small state table in the warehouse (e.g. <conn>_ext._sqlanvil_extracts(source, extracted_at, source_modified_at, rows)), so state lives with the data and works for CLI + hosted alike.
- MySQL:
information_schema.tables.update_time where reliable; fall back to always-extract.
- Per-declaration override:
extract: { refresh: "always" | "on_change" } (default on_change once proven, always initially for safe rollout).
- A skipped extract is reported as SKIPPED-fresh (visible in the run log), not silently absent.
Combined with per-declaration caps (#50) this turns the typical scheduled run from "re-copy the world" into "copy the handful that changed".
Also worth exploring later (separate work): Postgres COPY protocol instead of batched INSERTs for several-fold load throughput, and Parquet via the existing export/import bridge for very large sources.
From the acuantia migration dogfood: a full hosted run of a converted BigQuery project spends ~35 of 40 minutes in the extract layer — 206 sources re-materialized every run, even though most change rarely. This is the structural cost of cross-warehouse sources (Dataform never moves data; our extract bridge does), so the win is to move as few bytes as possible per run.
Proposal
Skip an extract when the source is provably unchanged since the last materialization:
last_modified_timefrom table metadata (free lookup) vs. the recorded extract time — skip if older. Record extract watermarks in a small state table in the warehouse (e.g.<conn>_ext._sqlanvil_extracts(source, extracted_at, source_modified_at, rows)), so state lives with the data and works for CLI + hosted alike.information_schema.tables.update_timewhere reliable; fall back to always-extract.extract: { refresh: "always" | "on_change" }(defaulton_changeonce proven,alwaysinitially for safe rollout).Combined with per-declaration caps (#50) this turns the typical scheduled run from "re-copy the world" into "copy the handful that changed".
Also worth exploring later (separate work): Postgres
COPYprotocol instead of batched INSERTs for several-fold load throughput, and Parquet via the existing export/import bridge for very large sources.