Skip to content

Add SKIP_MIGRATIONS to run migrations out-of-band#2169

Merged
EItanya merged 2 commits into
iplay88keys/migrations-improvements-pt-4from
iplay88keys/migrations-improvements-pt-5
Jul 6, 2026
Merged

Add SKIP_MIGRATIONS to run migrations out-of-band#2169
EItanya merged 2 commits into
iplay88keys/migrations-improvements-pt-4from
iplay88keys/migrations-improvements-pt-5

Conversation

@iplay88keys

@iplay88keys iplay88keys commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Important

Branched off of #2168, so will be rebased after that merges.

Description

Adds a SKIP_MIGRATIONS opt-out so database migrations can run out-of-band (e.g. from a CI/CD pipeline) instead of at controller startup.

  • New --skip-migrations controller flag (settable via the SKIP_MIGRATIONS env var; default off — startup migrations remain the default behavior).
  • When set, the controller applies nothing and instead calls the new migrations.VerifyMigrated, refusing to start against a database that is not fully migrated. Per source: a missing tracking table, a version behind the binary's embedded max, or a dirty tracking table is a startup error with an actionable message; a database ahead of the binary boots in compatibility mode, matching RunUp.
  • VerifyMigrated issues only SELECTs — it deliberately does not open golang-migrate, which creates the tracking table on every open — so the verification works on a connection whose role has no DDL privileges.
  • Helm: new database.postgres.skipMigrations value (default false) rendered into the controller configmap as SKIP_MIGRATIONS, plus a NOTES warning when enabled reminding the operator that migrations must be applied out-of-band before install/upgrade.

Testing

Verified on a kind cluster: upgrade with skipMigrations=true boots and logs database schema verified; renaming a tracking table makes the controller exit with tracking table "vector_schema_migrations" does not exist — the database has not been migrated; apply migrations out-of-band or unset SKIP_MIGRATIONS; restoring the table recovers.

Fresh install without migrations returns:

{"level":"error","ts":"2026-07-06T17:37:17Z","logger":"setup","msg":"database migration verification failed","error":"source core: tracking table \"schema_migrations\" does not exist - the database has not been migrated; apply migrations out-of-band or unset SKIP_MIGRATIONS","stacktrace":"github.com/kagent-dev/kagent/go/core/pkg/app.Start\n\t/workspace/core/pkg/app/app.go:482\nmain.main\n\t/workspace/core/cmd/controller/main.go:34\nruntime.main\n\t/usr/lib/go/src/runtime/proc.go:290"}

Controller starts up correctly when migrations are manually run against the db:

❯ ./go/core/bin/kagent-local db migrate up
applied 9 migration(s); schema is up to date

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
@iplay88keys iplay88keys marked this pull request as ready for review July 6, 2026 17:41
@EItanya EItanya merged commit d104f2f into iplay88keys/migrations-improvements-pt-4 Jul 6, 2026
5 checks passed
@EItanya EItanya deleted the iplay88keys/migrations-improvements-pt-5 branch July 6, 2026 18:25
@iplay88keys iplay88keys restored the iplay88keys/migrations-improvements-pt-5 branch July 6, 2026 19:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants