Skip to content

Commit 2099677

Browse files
anth-volkanth-volk
authored
Merge pull request #97 from PolicyEngine/app-v2-migration
Enable app v2 to run on API v2
2 parents 633576b + 2dc1bee commit 2099677

123 files changed

Lines changed: 19077 additions & 1046 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/skills/database-migrations.md‎

Lines changed: 301 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,301 @@
1+
# Database Migration Guidelines
2+
3+
## Overview
4+
5+
This project uses **Alembic** for database migrations with **SQLModel** models. Alembic is the industry-standard migration tool for SQLAlchemy/SQLModel projects.
6+
7+
**CRITICAL**: SQL migrations are the single source of truth for database schema. All table creation and schema changes MUST go through Alembic migrations.
8+
9+
## Architecture
10+
11+
```
12+
┌─────────────────────────────────────────────────────────────┐
13+
│ SQLModel Models (src/policyengine_api/models/) │
14+
│ - Define Python classes │
15+
│ - Used for ORM queries │
16+
│ - NOT the source of truth for schema │
17+
└─────────────────────────────────────────────────────────────┘
18+
│
19+
│ alembic revision --autogenerate
20+
▼
21+
┌─────────────────────────────────────────────────────────────┐
22+
│ Alembic Migrations (alembic/versions/) │
23+
│ - Create/alter tables │
24+
│ - Add indexes, constraints │
25+
│ - SOURCE OF TRUTH for schema │
26+
└─────────────────────────────────────────────────────────────┘
27+
│
28+
│ alembic upgrade head
29+
▼
30+
┌─────────────────────────────────────────────────────────────┐
31+
│ PostgreSQL Database (Supabase) │
32+
│ - Actual schema │
33+
│ - Tracked by alembic_version table │
34+
└─────────────────────────────────────────────────────────────┘
35+
```
36+
37+
## Essential Rules
38+
39+
### 1. NEVER use SQLModel.metadata.create_all() for schema creation
40+
41+
The old pattern of using `SQLModel.metadata.create_all()` is deprecated. All tables are created via Alembic migrations.
42+
43+
### 2. Every schema change requires a migration
44+
45+
When you modify a SQLModel model (add column, change type, add index), you MUST:
46+
1. Update the model in `src/policyengine_api/models/`
47+
2. Generate a migration: `uv run alembic revision --autogenerate -m "Description"`
48+
3. **Read and verify the generated migration** (see below)
49+
4. Apply it: `uv run alembic upgrade head`
50+
51+
### 3. ALWAYS verify auto-generated migrations before applying
52+
53+
**This is critical for AI agents.** After running `alembic revision --autogenerate`, you MUST:
54+
55+
1. **Read the generated migration file** in `alembic/versions/`
56+
2. **Verify the `upgrade()` function** contains the expected changes:
57+
- Correct table/column names
58+
- Correct column types (e.g., `sa.String()`, `sa.Uuid()`, `sa.Integer()`)
59+
- Proper foreign key references
60+
- Appropriate nullable settings
61+
3. **Verify the `downgrade()` function** properly reverses the changes
62+
4. **Check for Alembic autogenerate limitations:**
63+
- It may miss renamed columns (shows as drop + add instead)
64+
- It may not detect some index changes
65+
- It doesn't handle data migrations
66+
5. **Edit the migration if needed** before applying
67+
68+
Example verification:
69+
```python
70+
# Generated migration - verify this looks correct:
71+
def upgrade() -> None:
72+
op.add_column('users', sa.Column('phone', sa.String(), nullable=True))
73+
74+
def downgrade() -> None:
75+
op.drop_column('users', 'phone')
76+
```
77+
78+
**Never blindly apply a migration without reading it first.**
79+
80+
### 4. Migrations must be self-contained
81+
82+
Each migration should:
83+
- Create tables it needs (never assume they exist from Python)
84+
- Include both `upgrade()` and `downgrade()` functions
85+
- Be idempotent where possible (use `IF NOT EXISTS` patterns)
86+
87+
### 5. Never use conditional logic based on table existence
88+
89+
Migrations should NOT check if tables exist. Instead:
90+
- Ensure migrations run in the correct order (use `down_revision`)
91+
- The initial migration creates all base tables
92+
- Subsequent migrations build on that foundation
93+
94+
## Common Commands
95+
96+
```bash
97+
# Apply all pending migrations
98+
uv run alembic upgrade head
99+
100+
# Generate migration from model changes
101+
uv run alembic revision --autogenerate -m "Add users email index"
102+
103+
# Create empty migration (for manual SQL)
104+
uv run alembic revision -m "Add custom index"
105+
106+
# Check current migration state
107+
uv run alembic current
108+
109+
# Show migration history
110+
uv run alembic history
111+
112+
# Downgrade one revision
113+
uv run alembic downgrade -1
114+
115+
# Downgrade to specific revision
116+
uv run alembic downgrade <revision_id>
117+
```
118+
119+
## Local Development Workflow
120+
121+
```bash
122+
# 1. Start Supabase
123+
supabase start
124+
125+
# 2. Initialize database (runs migrations + applies RLS policies)
126+
uv run python scripts/init.py
127+
128+
# 3. Seed data
129+
uv run python scripts/seed.py
130+
```
131+
132+
### Reset database (DESTRUCTIVE)
133+
134+
```bash
135+
uv run python scripts/init.py --reset
136+
```
137+
138+
## Adding a New Model
139+
140+
1. Create the model in `src/policyengine_api/models/`
141+
142+
```python
143+
# src/policyengine_api/models/my_model.py
144+
from sqlmodel import SQLModel, Field
145+
from uuid import UUID, uuid4
146+
147+
class MyModel(SQLModel, table=True):
148+
__tablename__ = "my_models"
149+
150+
id: UUID = Field(default_factory=uuid4, primary_key=True)
151+
name: str
152+
```
153+
154+
2. Export in `__init__.py`:
155+
156+
```python
157+
# src/policyengine_api/models/__init__.py
158+
from .my_model import MyModel
159+
```
160+
161+
3. Generate migration:
162+
163+
```bash
164+
uv run alembic revision --autogenerate -m "Add my_models table"
165+
```
166+
167+
4. Review the generated migration in `alembic/versions/`
168+
169+
5. Apply the migration:
170+
171+
```bash
172+
uv run alembic upgrade head
173+
```
174+
175+
6. Update `scripts/init.py` to include the table in RLS policies if needed.
176+
177+
## Adding an Index
178+
179+
1. Generate a migration:
180+
181+
```bash
182+
uv run alembic revision -m "Add index on users.email"
183+
```
184+
185+
2. Edit the migration:
186+
187+
```python
188+
def upgrade() -> None:
189+
op.create_index("idx_users_email", "users", ["email"])
190+
191+
def downgrade() -> None:
192+
op.drop_index("idx_users_email", "users")
193+
```
194+
195+
3. Apply:
196+
197+
```bash
198+
uv run alembic upgrade head
199+
```
200+
201+
## Production Considerations
202+
203+
### Applying migrations to production
204+
205+
1. Migrations are automatically applied when deploying
206+
2. Always test migrations locally first
207+
3. For data migrations, consider running during low-traffic periods
208+
209+
### Transitioning production from old system to Alembic
210+
211+
Production databases that were created before Alembic (using the old `SQLModel.metadata.create_all()` approach or raw Supabase migrations) need special handling. Running `alembic upgrade head` would fail because the tables already exist.
212+
213+
**The solution: `alembic stamp`**
214+
215+
The `alembic stamp` command marks a migration as "already applied" without actually running it. This tells Alembic "the database is already at this state, start tracking from here."
216+
217+
**How it works:**
218+
219+
1. `alembic stamp <revision_id>` inserts a row into the `alembic_version` table with the specified revision ID
220+
2. Alembic now thinks that migration (and all migrations before it) have been applied
221+
3. Future migrations will run normally starting from that point
222+
223+
**Step-by-step production transition:**
224+
225+
```bash
226+
# 1. Connect to production database
227+
# (set SUPABASE_DB_URL or other connection env vars)
228+
229+
# 2. Check if alembic_version table exists
230+
# If not, Alembic will create it automatically
231+
232+
# 3. Verify production schema matches the initial migration
233+
# Compare tables/columns in production against alembic/versions/20260204_d6e30d3b834d_initial_schema.py
234+
235+
# 4. Stamp the initial migration as applied
236+
uv run alembic stamp d6e30d3b834d
237+
238+
# 5. If production also has the indexes from the second migration, stamp that too
239+
uv run alembic stamp a17ac554f4aa
240+
241+
# 6. Verify the stamp worked
242+
uv run alembic current
243+
# Should show: a17ac554f4aa (head)
244+
245+
# 7. From now on, new migrations will apply normally
246+
uv run alembic upgrade head
247+
```
248+
249+
**Handling partially applied migrations:**
250+
251+
If production has some but not all changes from a migration:
252+
253+
1. Manually apply the missing changes via SQL
254+
2. Then stamp that migration as complete
255+
3. Or: create a new migration that only adds the missing pieces
256+
257+
**After stamping:**
258+
259+
- All future schema changes go through Alembic migrations
260+
- Developers generate migrations with `alembic revision --autogenerate`
261+
- Deployments run `alembic upgrade head` to apply pending migrations
262+
- The `alembic_version` table tracks what's been applied
263+
264+
## File Structure
265+
266+
```
267+
alembic/
268+
├── env.py # Alembic configuration (imports models, sets DB URL)
269+
├── script.py.mako # Template for new migrations
270+
├── versions/ # Migration files
271+
│ ├── 20260204_d6e30d3b834d_initial_schema.py
272+
│ └── 20260204_a17ac554f4aa_add_parameter_values_indexes.py
273+
alembic.ini # Alembic settings
274+
275+
supabase/
276+
├── migrations/ # Supabase-specific migrations (storage only)
277+
│ ├── 20241119000000_storage_bucket.sql
278+
│ └── 20241121000000_storage_policies.sql
279+
└── migrations_archived/ # Old table migrations (now in Alembic)
280+
```
281+
282+
## Troubleshooting
283+
284+
### "Target database is not up to date"
285+
286+
Run `alembic upgrade head` to apply pending migrations.
287+
288+
### "Can't locate revision"
289+
290+
The alembic_version table has a revision that doesn't exist in your migrations folder. This can happen if someone deleted a migration file. Fix by stamping to a known revision:
291+
292+
```bash
293+
alembic stamp head # If tables are current
294+
alembic stamp d6e30d3b834d # If at initial schema
295+
```
296+
297+
### "Table already exists"
298+
299+
The migration is trying to create a table that already exists. Options:
300+
1. If this is a fresh setup, drop and recreate: `uv run python scripts/init.py --reset`
301+
2. If in production, stamp the migration as applied: `alembic stamp <revision>`

‎.env.example‎

Lines changed: 16 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -61,11 +61,22 @@ AGENT_USE_MODAL=false
6161
POLICYENGINE_API_URL=http://localhost:8000
6262

6363
# =============================================================================
64-
# MODAL SECRETS (for production)
64+
# MODAL SERVERLESS COMPUTE
6565
# =============================================================================
66-
# Modal secrets are NOT set via .env - they're managed via Modal CLI:
67-
#
68-
# 1. modal secret create policyengine-db \
66+
# Modal environment to use (main, staging, testing).
67+
# Only relevant when AGENT_USE_MODAL=true.
68+
# The Modal SDK authenticates via ~/.modal.toml (from `modal setup`).
69+
# For production (Cloud Run), set MODAL_TOKEN_ID and MODAL_TOKEN_SECRET instead.
70+
MODAL_ENVIRONMENT=main
71+
72+
# For production (Cloud Run) only:
73+
# MODAL_TOKEN_ID=ak-...
74+
# MODAL_TOKEN_SECRET=as-...
75+
76+
# =============================================================================
77+
# MODAL SECRETS (managed via Modal CLI, not .env)
78+
# =============================================================================
79+
# 1. modal secret create policyengine-db [--env testing] \
6980
# DATABASE_URL='postgresql://...' \
7081
# SUPABASE_URL='https://...' \
7182
# SUPABASE_KEY='...' \
@@ -75,5 +86,5 @@ POLICYENGINE_API_URL=http://localhost:8000
7586
# 2. modal secret create anthropic-api-key \
7687
# ANTHROPIC_API_KEY='sk-ant-...'
7788
#
78-
# 3. modal secret create logfire-token \
89+
# 3. modal secret create policyengine-logfire \
7990
# LOGFIRE_TOKEN='...'

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,3 +52,6 @@ docs/.env.local
5252
data/
5353
*.h5
5454
*.db
55+
56+
# macOS
57+
.DS_Store

‎CLAUDE.md‎

Lines changed: 15 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,21 @@ Use `gh` CLI for GitHub operations to ensure Actions run correctly.
7575

7676
## Database
7777

78-
`make init` resets tables and storage. `make seed` populates UK/US models with variables, parameters, and datasets.
78+
This project uses **Alembic** for database migrations. See `.claude/skills/database-migrations.md` for detailed guidelines.
79+
80+
**Key rules:**
81+
- All schema changes go through Alembic migrations (never use `SQLModel.metadata.create_all()`)
82+
- After modifying a model: `uv run alembic revision --autogenerate -m "Description"`
83+
- Apply migrations: `uv run alembic upgrade head`
84+
85+
**Local development:**
86+
```bash
87+
supabase start # Start local Supabase
88+
uv run python scripts/init.py # Run migrations + apply RLS policies
89+
uv run python scripts/seed.py # Seed data
90+
```
91+
92+
`scripts/init.py --reset` drops and recreates everything (destructive).
7993

8094
## Modal sandbox + Claude Code CLI gotchas
8195

0 commit comments

Comments
 (0)