-
Notifications
You must be signed in to change notification settings - Fork 354
Description
Bug: Destructive schema changes on forward-only models not surfaced until prod deploy
Describe the bug
When a forward-only model has columns removed from its definition, the plan/apply cycle succeeds
in dev and PR environments but fails with a DestructiveChangeError when deploying to production.
This leaves the production environment in an unfinalized state with downstream models falling behind.
The failure is deterministic — the change cannot be applied to prod — but the user receives no
warning until the final prod deploy step.
To Reproduce
- Have any forward-only model in production with columns
A,B,C— this includes:INCREMENTAL_BY_PARTITION(always forward-only by design)- Any model with
forward_only = truein its config - Models categorized as forward-only at plan time
- Remove column
Cfrom the model definition - Run
sqlmesh plan devand apply — succeeds ✓ - Open a PR; CI checks pass — succeeds ✓
- Attempt to deploy to production — fails with:
Plan requires destructive change to forward-only model ''s schema that drops columns 'C'.
To allow the destructive change, set the model's on_destructive_change setting to warn, allow,
or ignore or include the model in the plan's --allow-destructive-model option.
Expected behavior
The user should be warned — at minimum during the PR environment plan, ideally during plan dev —
that this change will fail at prod deploy. The outcome is fully predictable from:
- the model's
forward_onlystatus on_destructive_changedefaulting toERROR- the column diff between the previous snapshot schema and the new model definition
There is no reason to let the user reach prod before surfacing this.
Root cause (identified)
The destructive schema change check (_check_destructive_schema_change in
snapshot/evaluator.py) only runs as part of MigrateSchemasStage, which is explicitly skipped
for dev plans:
if plan.is_dev:
snapshots_with_schema_migration = [] # always empty in devAdditionally, dev tables are created fresh with a versioned suffix, so no ALTER TABLE is
generated — the check would have nothing to compare even if it ran.
All the information needed to predict the failure is available at plan-build time: the previous
snapshot schema, the new model schema, whether the model is forward-only, and the
on_destructive_change setting. The check simply isn't performed proactively.
Impact
- CI/CD bot deployments fail at the last step, leaving the prod environment in an unfinalized state
- Downstream models fall behind until an operator manually intervenes
- No actionable signal is given to the user before the failure occurs
Suggested fix
During plan construction, for any forward-only model where on_destructive_change = ERROR, diff
the previous snapshot schema against the new model schema. If a column drop is detected, raise (or
warn) at plan time — regardless of target environment. This would surface the issue during
plan dev, as a failed PR check, or at any earlier stage in the pipeline before the user reaches
prod.
Environment
- SQLMesh version: 0.230.1