Add kagent db migrate CLI for out-of-band migrations#2168
Draft
iplay88keys wants to merge 7 commits into
Draft
Conversation
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
…s-improvements-pt-3 Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
EItanya
pushed a commit
that referenced
this pull request
Jul 6, 2026
> [!IMPORTANT] > Branched off of #2168, so will be rebased after that merges. ## Description Adds a `SKIP_MIGRATIONS` opt-out so database migrations can run out-of-band (e.g. from a CI/CD pipeline) instead of at controller startup. - New `--skip-migrations` controller flag (settable via the `SKIP_MIGRATIONS` env var; default off — startup migrations remain the default behavior). - When set, the controller applies nothing and instead calls the new `migrations.VerifyMigrated`, refusing to start against a database that is not fully migrated. Per source: a missing tracking table, a version behind the binary's embedded max, or a dirty tracking table is a startup error with an actionable message; a database ahead of the binary boots in compatibility mode, matching `RunUp`. - `VerifyMigrated` issues only `SELECT`s — it deliberately does not open golang-migrate, which creates the tracking table on every open — so the verification works on a connection whose role has no DDL privileges. - Helm: new `database.postgres.skipMigrations` value (default `false`) rendered into the controller configmap as `SKIP_MIGRATIONS`, plus a NOTES warning when enabled reminding the operator that migrations must be applied out-of-band before install/upgrade. ## Testing Verified on a kind cluster: upgrade with `skipMigrations=true` boots and logs `database schema verified`; renaming a tracking table makes the controller exit with `tracking table "vector_schema_migrations" does not exist — the database has not been migrated; apply migrations out-of-band or unset SKIP_MIGRATIONS`; restoring the table recovers. Fresh install without migrations returns: ``` {"level":"error","ts":"2026-07-06T17:37:17Z","logger":"setup","msg":"database migration verification failed","error":"source core: tracking table \"schema_migrations\" does not exist - the database has not been migrated; apply migrations out-of-band or unset SKIP_MIGRATIONS","stacktrace":"github.com/kagent-dev/kagent/go/core/pkg/app.Start\n\t/workspace/core/pkg/app/app.go:482\nmain.main\n\t/workspace/core/cmd/controller/main.go:34\nruntime.main\n\t/usr/lib/go/src/runtime/proc.go:290"} ``` Controller starts up correctly when migrations are manually run against the db: ``` ❯ ./go/core/bin/kagent-local db migrate up applied 9 migration(s); schema is up to date ``` --------- Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
d104f2f to
03bf283
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Important
Branched off of #2094, so will be rebased after that merges.
Description
Adds a
kagent db migratecommand group for running and inspecting database migrations out-of-band from server startup.go/core/pkg/cli/dbandgo/core/pkg/cli/db/migrate;NewCommand(sources ...migrations.Source)takes the source list explicitly so downstream consumers can wire their own sources.--db-url, falling back to the controller'sPOSTGRES_DATABASE_URLenv var. The vector track is gated on the controller'sDATABASE_VECTOR_ENABLEDenv var (default enabled).migrations.WithMigratorexport: the CLI opens migrators through the orchestrator's own schema handling, tracking-table config, and advisory-lock identity, so a CLI invocation racing a booting server serializes.up,down, andgotorefuse a dirty source and report theforceinvocation that clears it; the CLI never auto-recovers (startup remains the automatic tier).status/versionreport dirty state instead of refusing.force 0clears the version record entirely (recovery when a track's first migration fails dirty); any other value must be a shipped migration version.downon an empty track reports "no migrations to roll back" instead of golang-migrate's raw "file does not exist".status --output jsonhas a frozen field shape locked byTestStatusJSONShape.dbsubtree hides the root's server-oriented persistent flags (--kagent-url,-n,-o,--timeout, etc.); db commands talk to Postgres directly.Testing
Version:
Fresh install:
Down (Rollback):
Goto:
Force:
Goto + Up: