Skip to content

Destructive schema changes on forward-only models not surfaced until prod deploy #5719

@gjesse

Description

@gjesse

Bug: Destructive schema changes on forward-only models not surfaced until prod deploy

Describe the bug

When a forward-only model has columns removed from its definition, the plan/apply cycle succeeds
in dev and PR environments but fails with a DestructiveChangeError when deploying to production.
This leaves the production environment in an unfinalized state with downstream models falling behind.

The failure is deterministic — the change cannot be applied to prod — but the user receives no
warning until the final prod deploy step.

To Reproduce

  1. Have any forward-only model in production with columns A, B, C — this includes:
    • INCREMENTAL_BY_PARTITION (always forward-only by design)
    • Any model with forward_only = true in its config
    • Models categorized as forward-only at plan time
  2. Remove column C from the model definition
  3. Run sqlmesh plan dev and apply — succeeds ✓
  4. Open a PR; CI checks pass — succeeds ✓
  5. Attempt to deploy to production — fails with:

Plan requires destructive change to forward-only model ''s schema that drops columns 'C'.
To allow the destructive change, set the model's on_destructive_change setting to warn, allow,
or ignore or include the model in the plan's --allow-destructive-model option.

Expected behavior

The user should be warned — at minimum during the PR environment plan, ideally during plan dev
that this change will fail at prod deploy. The outcome is fully predictable from:

  • the model's forward_only status
  • on_destructive_change defaulting to ERROR
  • the column diff between the previous snapshot schema and the new model definition

There is no reason to let the user reach prod before surfacing this.

Root cause (identified)

The destructive schema change check (_check_destructive_schema_change in
snapshot/evaluator.py) only runs as part of MigrateSchemasStage, which is explicitly skipped
for dev plans:

  if plan.is_dev:
      snapshots_with_schema_migration = []  # always empty in dev

Additionally, dev tables are created fresh with a versioned suffix, so no ALTER TABLE is
generated — the check would have nothing to compare even if it ran.

All the information needed to predict the failure is available at plan-build time: the previous
snapshot schema, the new model schema, whether the model is forward-only, and the
on_destructive_change setting. The check simply isn't performed proactively.

Impact

  • CI/CD bot deployments fail at the last step, leaving the prod environment in an unfinalized state
  • Downstream models fall behind until an operator manually intervenes
  • No actionable signal is given to the user before the failure occurs

Suggested fix

During plan construction, for any forward-only model where on_destructive_change = ERROR, diff
the previous snapshot schema against the new model schema. If a column drop is detected, raise (or
warn) at plan time — regardless of target environment. This would surface the issue during
plan dev, as a failed PR check, or at any earlier stage in the pipeline before the user reaches
prod.

Environment

  • SQLMesh version: 0.230.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions