Skip to content

Add kagent db migrate CLI for out-of-band migrations#2168

Draft
iplay88keys wants to merge 7 commits into
mainfrom
iplay88keys/migrations-improvements-pt-4
Draft

Add kagent db migrate CLI for out-of-band migrations#2168
iplay88keys wants to merge 7 commits into
mainfrom
iplay88keys/migrations-improvements-pt-4

Conversation

@iplay88keys

@iplay88keys iplay88keys commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Important

Branched off of #2094, so will be rebased after that merges.

Description

Adds a kagent db migrate command group for running and inspecting database migrations out-of-band from server startup.

kagent db migrate up                    # apply all pending migrations, every registered source
kagent db migrate status [--output json] # aggregate + per-source applied/pending breakdown
kagent db migrate version               # per-source current versions (--source filters)
kagent db migrate down N  --source core # roll back the N most-recent migrations on one source
kagent db migrate goto V  --source core # move one source to version V (0 = empty schema)
kagent db migrate force V --source core # assert a version without running SQL (recovery)
  • New exported packages go/core/pkg/cli/db and go/core/pkg/cli/db/migrate; NewCommand(sources ...migrations.Source) takes the source list explicitly so downstream consumers can wire their own sources.
  • The connection comes from --db-url, falling back to the controller's POSTGRES_DATABASE_URL env var. The vector track is gated on the controller's DATABASE_VECTOR_ENABLED env var (default enabled).
  • New migrations.WithMigrator export: the CLI opens migrators through the orchestrator's own schema handling, tracking-table config, and advisory-lock identity, so a CLI invocation racing a booting server serializes.
  • up, down, and goto refuse a dirty source and report the force invocation that clears it; the CLI never auto-recovers (startup remains the automatic tier). status/version report dirty state instead of refusing.
  • force 0 clears the version record entirely (recovery when a track's first migration fails dirty); any other value must be a shipped migration version.
  • down on an empty track reports "no migrations to roll back" instead of golang-migrate's raw "file does not exist".
  • status --output json has a frozen field shape locked by TestStatusJSONShape.
  • Help output for the db subtree hides the root's server-oriented persistent flags (--kagent-url, -n, -o, --timeout, etc.); db commands talk to Postgres directly.

Testing

Version:

❯ ./go/core/bin/kagent-local db migrate version
core: 6
vector: 3

Fresh install:

❯ ./go/core/bin/kagent-local db migrate status
9 migration(s) applied, 0 pending
  core: 6 applied (at v6), 0 pending
  vector: 3 applied (at v3), 0 pending

kagent=# \dt
                   List of tables
 Schema |           Name           | Type  | Owner
--------+--------------------------+-------+--------
⋮
(17 rows)

Down (Rollback):

kagent=# \dt
                   List of tables
 Schema |           Name           | Type  | Owner
--------+--------------------------+-------+--------
⋮
(17 rows)

❯ ./go/core/bin/kagent-local db migrate down 1 --source core 
rolled back 1 migration(s)

❯ ./go/core/bin/kagent-local db migrate status              
8 migration(s) applied, 1 pending
  core: 5 applied (at v5), 1 pending
  vector: 3 applied (at v3), 0 pending

kagent=# \dt
                   List of tables
 Schema |           Name           | Type  | Owner
--------+--------------------------+-------+--------
⋮
(15 rows)

Goto:

❯ ./go/core/bin/kagent-local db migrate goto 1 --source core
schema is at version 1

kagent=# \dt
                   List of tables
 Schema |           Name           | Type  | Owner
--------+--------------------------+-------+--------
 public | memory                   | table | kagent
 public | schema_migrations        | table | kagent
 public | vector_schema_migrations | table | kagent
(3 rows)

Force:

kagent=# select * from schema_migrations;
 version | dirty
---------+-------
       6 | f
(1 row)

❯ ./go/core/bin/kagent-local db migrate force 1 --source core 
version 1 marked as applied

kagent=# select * from schema_migrations;
 version | dirty
---------+-------
       1 | f
(1 row)

Goto + Up:

❯ ./go/core/bin/kagent-local db migrate goto 1 --source core
schema is at version 1

❯ ./go/core/bin/kagent-local db migrate goto 1 --source vector
schema is at version 1

❯ ./go/core/bin/kagent-local db migrate status                
2 migration(s) applied, 7 pending
  core: 1 applied (at v1), 5 pending
  vector: 1 applied (at v1), 2 pending

❯ ./go/core/bin/kagent-local db migrate up                    
applied 7 migration(s); schema is up to date

iplay88keys and others added 7 commits June 26, 2026 08:22
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
…s-improvements-pt-3

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
@iplay88keys iplay88keys marked this pull request as ready for review July 6, 2026 15:29
@iplay88keys iplay88keys changed the base branch from iplay88keys/migrations-improvements-pt-3 to main July 6, 2026 15:29
@iplay88keys iplay88keys requested a review from EItanya July 6, 2026 15:42
@iplay88keys iplay88keys changed the title add kagent db migrate CLI for out-of-band migrations Add kagent db migrate CLI for out-of-band migrations Jul 6, 2026
@iplay88keys iplay88keys changed the base branch from main to iplay88keys/migrations-improvements-pt-3 July 6, 2026 16:18
EItanya pushed a commit that referenced this pull request Jul 6, 2026
> [!IMPORTANT]
> Branched off of #2168, so
will be rebased after that merges.


## Description

Adds a `SKIP_MIGRATIONS` opt-out so database migrations can run
out-of-band (e.g. from a CI/CD pipeline) instead of at controller
startup.

- New `--skip-migrations` controller flag (settable via the
`SKIP_MIGRATIONS` env var; default off — startup migrations remain the
default behavior).
- When set, the controller applies nothing and instead calls the new
`migrations.VerifyMigrated`, refusing to start against a database that
is not fully migrated. Per source: a missing tracking table, a version
behind the binary's embedded max, or a dirty tracking table is a startup
error with an actionable message; a database ahead of the binary boots
in compatibility mode, matching `RunUp`.
- `VerifyMigrated` issues only `SELECT`s — it deliberately does not open
golang-migrate, which creates the tracking table on every open — so the
verification works on a connection whose role has no DDL privileges.
- Helm: new `database.postgres.skipMigrations` value (default `false`)
rendered into the controller configmap as `SKIP_MIGRATIONS`, plus a
NOTES warning when enabled reminding the operator that migrations must
be applied out-of-band before install/upgrade.

## Testing
Verified on a kind cluster: upgrade with `skipMigrations=true` boots and
logs `database schema verified`; renaming a tracking table makes the
controller exit with `tracking table "vector_schema_migrations" does not
exist — the database has not been migrated; apply migrations out-of-band
or unset SKIP_MIGRATIONS`; restoring the table recovers.

Fresh install without migrations returns:
```
{"level":"error","ts":"2026-07-06T17:37:17Z","logger":"setup","msg":"database migration verification failed","error":"source core: tracking table \"schema_migrations\" does not exist - the database has not been migrated; apply migrations out-of-band or unset SKIP_MIGRATIONS","stacktrace":"github.com/kagent-dev/kagent/go/core/pkg/app.Start\n\t/workspace/core/pkg/app/app.go:482\nmain.main\n\t/workspace/core/cmd/controller/main.go:34\nruntime.main\n\t/usr/lib/go/src/runtime/proc.go:290"}
```

Controller starts up correctly when migrations are manually run against
the db:
```
❯ ./go/core/bin/kagent-local db migrate up
applied 9 migration(s); schema is up to date
```

---------

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
@EItanya EItanya requested review from ilackarms and yuval-k as code owners July 6, 2026 18:25
@iplay88keys iplay88keys marked this pull request as draft July 6, 2026 19:57
@iplay88keys iplay88keys changed the base branch from iplay88keys/migrations-improvements-pt-3 to main July 6, 2026 19:57
@iplay88keys iplay88keys force-pushed the iplay88keys/migrations-improvements-pt-4 branch from d104f2f to 03bf283 Compare July 6, 2026 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants