-
Notifications
You must be signed in to change notification settings - Fork 29
feat(migrations): Unify schema and data journals #896
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
dcramer
wants to merge
4
commits into
main
Choose a base branch
from
codex/unified-migration-journals
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
2dd177d
feat(migrations): Unify schema and data journals
dcramer efd628c
fix(migrations): Embed durable migration implementations
dcramer b26c218
ref(migrations): Inject the database adapter
dcramer fd37ff2
ref(migrations): Add a versioned helper boundary
dcramer File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| # `@sentry/junior-migrations` | ||
|
|
||
| Junior's migration format extends a Drizzle Kit journal with TypeScript data | ||
| migrations. Every journal entry resolves to exactly one `<tag>.sql` or | ||
| `<tag>.ts` file. Drizzle Kit continues to own numbering and schema snapshots; | ||
| the Junior migration runner owns execution and exact per-entry tracking. | ||
|
|
||
| TypeScript migrations target a versioned capability API and must not import | ||
| Junior runtime internals. Every parser, transform, and migration-specific | ||
| helper belongs permanently in the migration file so application refactors or | ||
| deletions cannot break pending upgrades. | ||
|
|
||
| ## Authoring | ||
|
|
||
| Generate schema migrations with the owning package's normal Drizzle command. | ||
| Generate data migrations through this package's wrapper: | ||
|
|
||
| ```bash | ||
| junior-migrations generate \ | ||
| --config drizzle.config.ts \ | ||
| --out migrations \ | ||
| --name backfill_actor | ||
| ``` | ||
|
|
||
| The wrapper asks Drizzle Kit to create a custom journal entry and snapshot, | ||
| then replaces the empty SQL file with a `MigrationV1` TypeScript scaffold. | ||
|
|
||
| ## Compatibility | ||
|
|
||
| Drizzle Kit remains the supported authoring tool. Drizzle ORM's stock | ||
| `migrate()` function is not a supported executor for mixed journals because it | ||
| requires every entry to have a SQL file. Call `runMigrationJournal` instead and | ||
| provide the host database adapter, state adapter, and TypeScript loader. The | ||
| same database adapter drives the journal ledger and is exposed as | ||
| `context.database`, so migration files never own connection or driver setup. | ||
|
|
||
| The runner rejects runtime imports of application source, relative modules, | ||
| and unversioned `@sentry/junior` modules. Migrations may import the append-only | ||
| `@sentry/junior/migration-helpers/v1` surface for parsing primitives, adapters, | ||
| stores, and key resolution. One-off migration decisions and data transforms | ||
| must still remain in the journal entry. Add a new helper or capability version | ||
| rather than changing an existing contract. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| { | ||
| "name": "@sentry/junior-migrations", | ||
| "version": "0.101.0", | ||
| "private": false, | ||
| "publishConfig": { | ||
| "access": "public" | ||
| }, | ||
| "type": "module", | ||
| "repository": { | ||
| "type": "git", | ||
| "url": "git+https://github.com/getsentry/junior.git", | ||
| "directory": "packages/junior-migrations" | ||
| }, | ||
| "bin": { | ||
| "junior-migrations": "dist/cli.js" | ||
| }, | ||
| "exports": { | ||
| ".": { | ||
| "types": "./src/index.ts", | ||
| "default": "./dist/index.js" | ||
| } | ||
| }, | ||
| "files": [ | ||
| "dist", | ||
| "src" | ||
| ], | ||
| "scripts": { | ||
| "build": "tsup && tsc -p tsconfig.build.json --emitDeclarationOnly", | ||
| "prepare": "pnpm run build", | ||
| "prepack": "pnpm run build", | ||
| "lint": "oxlint --config ../junior/.oxlintrc.json --deny-warnings src tests tsup.config.ts", | ||
| "test": "vitest run", | ||
| "typecheck": "tsc --noEmit" | ||
| }, | ||
| "peerDependencies": { | ||
| "drizzle-kit": "catalog:" | ||
| }, | ||
| "devDependencies": { | ||
| "@types/node": "^25.9.1", | ||
| "drizzle-kit": "catalog:", | ||
| "drizzle-orm": "catalog:", | ||
| "oxlint": "^1.66.0", | ||
| "tsup": "^8.5.1", | ||
| "typescript": "^6.0.3", | ||
| "vitest": "^4.1.7" | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,24 @@ | ||
| import { parseArgs } from "node:util"; | ||
| import { generateTypeScriptMigration } from "./generate"; | ||
|
|
||
| const { positionals, values } = parseArgs({ | ||
| allowPositionals: true, | ||
| options: { | ||
| config: { type: "string" }, | ||
| name: { type: "string" }, | ||
| out: { type: "string", default: "./migrations" }, | ||
| }, | ||
| }); | ||
|
|
||
| if (positionals[0] !== "generate" || !values.config || !values.name) { | ||
| throw new Error( | ||
| "Usage: junior-migrations generate --config <path> --name <name> [--out <dir>]", | ||
| ); | ||
| } | ||
|
|
||
| const path = await generateTypeScriptMigration({ | ||
| configPath: values.config, | ||
| migrationsFolder: values.out, | ||
| name: values.name, | ||
| }); | ||
| console.log(`Created TypeScript migration ${path}`); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| import { spawn } from "node:child_process"; | ||
| import { readFile, rename, writeFile } from "node:fs/promises"; | ||
| import { resolve } from "node:path"; | ||
| import { readMigrationJournal } from "./journal"; | ||
|
|
||
| /** Drizzle Kit inputs used to create one TypeScript migration entry. */ | ||
| export interface GenerateTypeScriptMigrationOptions { | ||
| configPath: string; | ||
| cwd?: string; | ||
| migrationsFolder: string; | ||
| name: string; | ||
| } | ||
|
|
||
| function run(command: string, args: string[], cwd: string): Promise<void> { | ||
| return new Promise((resolvePromise, reject) => { | ||
| const child = spawn(command, args, { cwd, stdio: "inherit" }); | ||
| child.on("error", reject); | ||
| child.on("exit", (code) => { | ||
| if (code === 0) { | ||
| resolvePromise(); | ||
| } else { | ||
| reject(new Error(`${command} exited with code ${code ?? "unknown"}`)); | ||
| } | ||
| }); | ||
| }); | ||
| } | ||
|
|
||
| function scaffold(): string { | ||
| return `import type { MigrationV1 } from "@sentry/junior-migrations";\n\nconst migration = {\n apiVersion: 1,\n async up(context) {\n void context;\n },\n} satisfies MigrationV1;\n\nexport default migration;\n`; | ||
| } | ||
|
|
||
| /** Create a journaled TypeScript migration through Drizzle Kit's custom generator. */ | ||
| export async function generateTypeScriptMigration( | ||
| options: GenerateTypeScriptMigrationOptions, | ||
| ): Promise<string> { | ||
| const cwd = resolve(options.cwd ?? process.cwd()); | ||
| const folder = resolve(cwd, options.migrationsFolder); | ||
| const before = await readMigrationJournal(folder).catch(() => []); | ||
| await run( | ||
| "drizzle-kit", | ||
| [ | ||
| "generate", | ||
| "--custom", | ||
| "--config", | ||
| options.configPath, | ||
| "--name", | ||
| options.name, | ||
| ], | ||
| cwd, | ||
| ); | ||
| const after = await readMigrationJournal(folder); | ||
| if (after.length !== before.length + 1) { | ||
| throw new Error("Drizzle Kit did not create exactly one migration entry"); | ||
| } | ||
| const entry = after.at(-1); | ||
| if (!entry) { | ||
| throw new Error("Drizzle Kit did not create a migration entry"); | ||
| } | ||
| const sqlPath = resolve(folder, `${entry.tag}.sql`); | ||
| const source = await readFile(sqlPath, "utf8"); | ||
| if (!source.includes("Custom SQL migration file")) { | ||
| throw new Error(`Refusing to replace non-custom migration ${entry.tag}`); | ||
| } | ||
| const typescriptPath = resolve(folder, `${entry.tag}.ts`); | ||
| await rename(sqlPath, typescriptPath); | ||
| await writeFile(typescriptPath, scaffold(), "utf8"); | ||
| return typescriptPath; | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
| export { generateTypeScriptMigration } from "./generate"; | ||
| export { readMigrationJournal, resolveMigrations } from "./journal"; | ||
| export { runMigrationJournal } from "./runner"; | ||
| export type { GenerateTypeScriptMigrationOptions } from "./generate"; | ||
| export type { RunMigrationJournalOptions } from "./runner"; | ||
| export type { | ||
| MigrationContextV1, | ||
| MigrationDatabaseAdapter, | ||
| MigrationJsonValue, | ||
| MigrationJournalEntry, | ||
| MigrationLockV1, | ||
| MigrationProgressV1, | ||
| MigrationRedisV1, | ||
| MigrationRunResult, | ||
| MigrationStateV1, | ||
| MigrationV1, | ||
| ResolvedMigration, | ||
| TypeScriptMigrationLoader, | ||
| } from "./types"; |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.