Skip to content

Add SKIP_MIGRATIONS to run migrations out-of-band#2170

Draft
iplay88keys wants to merge 9 commits into
mainfrom
iplay88keys/migrations-improvements-pt-5
Draft

Add SKIP_MIGRATIONS to run migrations out-of-band#2170
iplay88keys wants to merge 9 commits into
mainfrom
iplay88keys/migrations-improvements-pt-5

Conversation

@iplay88keys

Copy link
Copy Markdown
Contributor

Important

Branched off of #2168, so will be rebased after that merges.

Description

Adds a SKIP_MIGRATIONS opt-out so database migrations can run out-of-band (e.g. from a CI/CD pipeline) instead of at controller startup.

  • New --skip-migrations controller flag (settable via the SKIP_MIGRATIONS env var; default off — startup migrations remain the default behavior).
  • When set, the controller applies nothing and instead calls the new migrations.VerifyMigrated, refusing to start against a database that is not fully migrated. Per source: a missing tracking table, a version behind the binary's embedded max, or a dirty tracking table is a startup error with an actionable message; a database ahead of the binary boots in compatibility mode, matching RunUp.
  • VerifyMigrated issues only SELECTs — it deliberately does not open golang-migrate, which creates the tracking table on every open — so the verification works on a connection whose role has no DDL privileges.
  • Helm: new database.postgres.skipMigrations value (default false) rendered into the controller configmap as SKIP_MIGRATIONS, plus a NOTES warning when enabled reminding the operator that migrations must be applied out-of-band before install/upgrade.

Testing

Verified on a kind cluster: upgrade with skipMigrations=true boots and logs database schema verified; renaming a tracking table makes the controller exit with tracking table "vector_schema_migrations" does not exist — the database has not been migrated; apply migrations out-of-band or unset SKIP_MIGRATIONS; restoring the table recovers.

Fresh install without migrations returns:

{"level":"error","ts":"2026-07-06T17:37:17Z","logger":"setup","msg":"database migration verification failed","error":"source core: tracking table \"schema_migrations\" does not exist - the database has not been migrated; apply migrations out-of-band or unset SKIP_MIGRATIONS","stacktrace":"github.com/kagent-dev/kagent/go/core/pkg/app.Start\n\t/workspace/core/pkg/app/app.go:482\nmain.main\n\t/workspace/core/cmd/controller/main.go:34\nruntime.main\n\t/usr/lib/go/src/runtime/proc.go:290"}

Controller starts up correctly when migrations are manually run against the db:

❯ ./go/core/bin/kagent-local db migrate up
applied 9 migration(s); schema is up to date

iplay88keys and others added 9 commits June 26, 2026 08:22
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
…s-improvements-pt-3

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants