Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,16 @@ For a single model, every emitter must agree on:
and `db_check_constraint_name` (Rust).
9. **Default values** — server-side defaults must serialize identically.
10. **Nullability** — must agree.
11. **Enum label additions** — the label-addition decision (which model-declared
labels a live enum type is missing, and which live labels are extra) and
the rendered `ALTER TYPE ... ADD VALUE IF NOT EXISTS` statement are decided
by ONE pair of functions: `ferro_ddl_lowering::missing_enum_labels` /
`extra_enum_labels` + `render_pg_enum_add_value`. The auto-migrate
reconciliation pass consumes them directly; the Alembic autogenerate
comparator consumes them over FFI (`_core._plan_enum_label_addition`) and
executes the byte-identical statements. Pinned by
`tests/test_cross_emitter_parity.py` and the ferro-ddl-lowering unit pins.
See ADR-0011 (append-only; update-gated; warn-never-act for extras).

### Why this invariant exists

Expand Down
12 changes: 10 additions & 2 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,13 +129,21 @@ The auto-migrate step that brings missing tables into existence — the table, i
_Avoid_: Bootstrap, ensure-tables, table sync

**Reconciliation pass**:
The `migrate_updates` step that alters existing tables to match the registered modelsthe only authority for DDL against a table that already exists. Within one table, column changes land before the indexes and constraints that reference them.
The `migrate_updates` step that alters existing schema objects — tables and ferro-owned enum types — to match the registered models; the only authority for DDL against an object that already exists. Within one table, column changes land before the indexes and constraints that reference them; label additions land before any table's changes.
_Avoid_: Update pass, schema sync, drift repair

**Ferro-owned artifact**:
An index or constraint whose name follows ferro's canonical naming (`idx_`, `uq_`, `fk_`, `ck_`), marking it as reconcilable: auto-migrate may create, rebuild, or drop it to match the declared model. Artifacts named any other way belong to the user and are never altered or dropped.
A schema object ferro may reconcile to match the declared model. Indexes and constraints are ferro-owned by naming (`idx_`, `uq_`, `fk_`, `ck_`); native enum types are ferro-owned by derivation — the type's name matches the name ferro derives from the model. Artifacts owned neither way belong to the user and are never altered or dropped.
_Avoid_: Managed index, system constraint, internal index

**Enum label**:
One storable value of a native Postgres enum type, mirrored from a Python `StrEnum` member's value. Members are the Python-side declaration; labels are what the database accepts and stores.
_Avoid_: Enum value, variant, choice

**Label addition**:
The reconciliation-pass operation appending model-declared labels missing from a live ferro-owned enum type. Append-only and metadata-only: rows are never touched, and labels the database has but the model lacks are warned about loudly and never removed — removal and rename are reviewed-migration territory.
_Avoid_: Enum sync, label reconciliation, enum evolution

**Constraint rebuild**:
Drop-and-recreate of a ferro-owned constraint whose live definition no longer matches the declared model — a foreign key's `on_delete`, its target, its columns. Metadata-only: rows are never touched. On a backend that cannot alter constraints, ferro warns loudly and skips; it never diverges silently.
_Avoid_: Constraint alter, FK patch, in-place constraint update
106 changes: 106 additions & 0 deletions crates/ferro-ddl-lowering/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -498,6 +498,63 @@ pub fn render_pg_enum_create_type(type_name: &str, labels: &[String]) -> String
)
}

/// The label-addition decision (ADR-0011): which model-declared labels a live
/// ferro-owned enum type is missing, in declared order. This is the single
/// decision table for enum label drift — the auto-migrate planner and the
/// Alembic autogenerate comparator both consume it mechanically (AGENTS.md
/// § I-1); neither side re-derives it. Append-only by construction: labels the
/// live type has but the model lacks are not this function's concern.
pub fn missing_enum_labels(declared: &[String], live: &[String]) -> Vec<String> {
declared
.iter()
.filter(|label| !live.contains(label))
.cloned()
.collect()
}

/// The warn-never-act half of the label-addition decision (ADR-0011): labels
/// the live type carries that the model no longer declares, in live (enum
/// sort) order. Rows may still hold these labels and older code may still be
/// running against the schema, so callers warn loudly and never remove —
/// removal and rename are reviewed-migration territory.
pub fn extra_enum_labels(declared: &[String], live: &[String]) -> Vec<String> {
live.iter()
.filter(|label| !declared.contains(label))
.cloned()
.collect()
}

/// The warn-never-act message for one drifted enum type, or `None` when
/// nothing is extra. Single-sourced like [`refused_conversion_warning`]:
/// callers emit it verbatim, never re-derive the wording.
pub fn extra_enum_labels_warning(type_name: &str, extra: &[String]) -> Option<String> {
if extra.is_empty() {
return None;
}
let listed: Vec<String> = extra.iter().map(|l| format!("'{l}'")).collect();
Some(format!(
"Enum type '{}' has label(s) {} that the model no longer declares. \
Label addition is append-only: ferro never removes enum labels \
(existing rows may still hold them). Remove or rename labels with \
a reviewed Alembic migration.",
type_name,
listed.join(", ")
))
}

/// One `ALTER TYPE ... ADD VALUE IF NOT EXISTS` for a label addition.
/// `IF NOT EXISTS` makes concurrent boots and shared-type replans harmless;
/// executed outside transactions (autocommit) so the statement is legal on
/// every supported Postgres version and the label is committed before any
/// table plan that references it.
pub fn render_pg_enum_add_value(type_name: &str, label: &str) -> String {
format!(
"ALTER TYPE {} ADD VALUE IF NOT EXISTS '{}'",
quote_ident(type_name),
label.replace('\'', "''"),
)
}

/// Detect a refused conversion from a live column to a resolved storage
/// target. Extends [`refused_scalar_conversion`] with the native-enum case:
/// a live non-enum (varchar/text) column targeted at a native Postgres enum
Expand Down Expand Up @@ -1464,6 +1521,55 @@ mod tests {
);
}

#[test]
fn missing_enum_labels_returns_additions_in_declared_order() {
let declared = vec!["plaid".to_string(), "mx".to_string(), "teller".to_string()];
let live = vec!["plaid".to_string()];
assert_eq!(missing_enum_labels(&declared, &live), vec!["mx", "teller"]);
}

#[test]
fn missing_enum_labels_empty_when_live_covers_declared() {
let declared = vec!["plaid".to_string()];
let live = vec!["plaid".to_string(), "legacy".to_string()];
assert!(missing_enum_labels(&declared, &live).is_empty());
}

#[test]
fn extra_enum_labels_returns_undeclared_live_labels_in_live_order() {
let declared = vec!["plaid".to_string()];
let live = vec!["legacy".to_string(), "plaid".to_string(), "old".to_string()];
assert_eq!(extra_enum_labels(&declared, &live), vec!["legacy", "old"]);
assert!(extra_enum_labels(&live, &live).is_empty());
}

#[test]
fn extra_enum_labels_warning_is_pinned_and_names_the_exit() {
assert_eq!(
extra_enum_labels_warning("provider", &["legacy".to_string()]),
Some(
"Enum type 'provider' has label(s) 'legacy' that the model no longer \
declares. Label addition is append-only: ferro never removes enum \
labels (existing rows may still hold them). Remove or rename labels \
with a reviewed Alembic migration."
.to_string()
)
);
assert_eq!(extra_enum_labels_warning("provider", &[]), None);
}

#[test]
fn render_pg_enum_add_value_is_pinned_and_escapes() {
assert_eq!(
render_pg_enum_add_value("provider", "mx"),
"ALTER TYPE \"provider\" ADD VALUE IF NOT EXISTS 'mx'"
);
assert_eq!(
render_pg_enum_add_value("od'd", "it's"),
"ALTER TYPE \"od'd\" ADD VALUE IF NOT EXISTS 'it''s'"
);
}

#[test]
fn render_pg_enum_create_type_escapes_quotes() {
let sql = render_pg_enum_create_type("od'd", &["it's".to_string()]);
Expand Down
65 changes: 65 additions & 0 deletions docs/adr/0011-label-addition-update-gated-append-only.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Label addition is update-gated and append-only

A native Postgres enum type that already exists is an existing schema object,
so evolving its label set belongs to the **reconciliation pass**
(`migrate_updates`), not the create pass — the same line ADR-0010 drew for
indexes. The reconciliation pass performs **label addition** (`CONTEXT.md`):
it appends model-declared labels missing from a live ferro-owned enum type,
and does nothing else. Labels the database has but the model lacks are warned
about loudly and never removed — rows may still hold them, and old code may
still be running against the new schema during a rolling deploy. A type is
ferro-owned by **derivation**: its name matches the name ferro derives from
the model (enum types carry no `idx_`/`uq_`-style prefix, so prefix doctrine
cannot apply). The additive-only scope is what makes derivation-based
ownership safe: the worst misattribution appends a label; it never drops or
rewrites anything.

Both migration doors consume one decision: the label diff (model labels +
live `pg_enum` labels → additions + warnings) lives in the Rust core and is
consumed mechanically by the auto-migrate planner and by an Alembic
autogenerate comparator in the bridge — the same one-decision-table seam as
`resolve_column_storage` (AGENTS.md I-1). Alembic autogenerate has no
`migrate_updates` gate: running autogenerate is itself the request for a
diff, so the comparator always reports; parity is in the decision, not the
gate. `ALTER TYPE ... ADD VALUE` executes outside transactions on both paths
(an autocommit pre-pass before the per-table transactions in auto-migrate;
`autocommit_block()` in generated revisions) — Prisma shipped the
in-transaction version and it is a graveyard (prisma#7251, #5290, #8424).

Decision by owner (2026-08-05), grilling #328.

Rejected alternatives:

- **Ensure semantics under plain `auto_migrate`** ("the type guard already
runs in the create pass; label addition is just a stronger ensure"):
re-opens the two-owners divergence ADR-0010 closed — an existing object
whose shape plain `auto_migrate` sometimes changes. The create pass stays
introspection-free and silent about drift of every kind, enums included.
- **Automatic removal/rename** (full sync, as `alembic-postgresql-enum`
does): removal requires a type-replacement dance and human judgment about
rows holding the removed label; every implementation that automated it
grew a bug tracker around it. Reviewed-migration territory, permanently.
- **Recommending `alembic-postgresql-enum`** instead of our own comparator:
it autogenerates removals, contradicting append-only — the two doors would
disagree about what the same model means.
- **Ownership markers** (`COMMENT ON TYPE` stamped at create time): sound
provenance, but every existing deployment's types are unmarked, and the
backfill machinery buys nothing additive-only doesn't already guarantee.

## Consequences

- Under plain `auto_migrate=True`, an evolved `StrEnum` still fails at first
use with the database's `invalid input value for enum` error — silently at
boot, by design. The documented answer to drift, enum drift included, is
`migrate_updates=True`. Auto-migrate docs must carry this trap loudly:
fresh schemas always get the complete label set, so no app-side test
against a throwaway schema can catch it (#328).
- A new table created under plain `auto_migrate` that references an existing
stale type gets the stale label set — the create-pass type guard only
fires for missing types. Same flag fixes it.
- Appended labels land at the end of the Postgres enum ordering regardless of
their position in the Python declaration; `ORDER BY` on an enum column
follows database order, not declaration order.
- Live introspection must read `pg_enum` labels (today it records only
"is an enum"), and the cross-emitter parity tests extend to pin the label
diff across both consumers.
71 changes: 71 additions & 0 deletions docs/pages/guide/migrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,8 @@ What it covers is capability-relative per backend:
| Change column type | ⚠️ `UserWarning`, no DDL (SQLite type affinity makes drift mostly cosmetic) | ✅ `ALTER COLUMN ... TYPE ... USING` cast |
| Change nullability | ⚠️ `UserWarning`, no DDL | ✅ `SET NOT NULL` / `DROP NOT NULL` |
| Drop orphaned Ferro-named index (`idx_*` / `uq_*`) | ✅ with `migrate_destructive=True` | ✅ with `migrate_destructive=True` |
| Add a missing enum label (a `StrEnum` grew a member) | ✅ nothing to do — enums store as text | ✅ `ALTER TYPE ... ADD VALUE` *0.18.0+* |
| Remove or rename an enum label | ✅ nothing to do | ⚠️ `UserWarning`, no DDL — Alembic territory |
| Inline single-column `UNIQUE` on existing column, index option changes | ❌ never — Alembic territory | ❌ never |
| Rename column/table, change primary key, drop table | ❌ never — Alembic territory | ❌ never |

Expand All @@ -59,6 +61,74 @@ Rules worth knowing:
- **Postgres type changes take an exclusive lock** and fail the connect if existing data does not cast cleanly — fine for a development flag, but worth knowing.
- **The pool refreshes after any schema change**, so no cached statement or stale identity-mapped instance can observe the pre-migration schema.

### Evolving enums: label addition

*Added in 0.18.0.* On PostgreSQL, `StrEnum` fields create a **native enum type**, and a type that already exists in the database does not learn new members on its own. When a `StrEnum` grows, `migrate_updates=True` performs **label addition**: it compares the model's members against the live type and appends what's missing with `ALTER TYPE ... ADD VALUE IF NOT EXISTS`.

!!! danger "This gap is invisible to your tests"
Under plain `auto_migrate=True` (without `migrate_updates`), an existing enum type is **never** updated — like every existing object, it belongs to the update pass. The failure mode is nasty: every test suite that creates its schema fresh gets the complete enum and stays green, while every *existing* database rejects the new member at runtime with `invalid input value for enum`. No app-side test against a throwaway schema can catch this. If your models' enums evolve, run with `migrate_updates=True` (or generate the migration with Alembic — the [autogenerate bridge](#alembic-for-production) sees the same drift).

=== "Assignment"

```python
from enum import StrEnum

import ferro
from ferro import Model


class Provider(StrEnum):
PLAID = "plaid"
MX = "mx" # new member — the live type only has 'plaid'


class Feed(Model):
id: int | None = ferro.Field(primary_key=True, default=None)
provider: Provider


await ferro.connect("postgres://...", migrate_updates=True)
# → ALTER TYPE "provider" ADD VALUE IF NOT EXISTS 'mx'

feed = await Feed.create(provider=Provider.MX)
recent = await Feed.where(lambda feed: feed.provider == Provider.MX).all()
```

=== "Annotated"

```python
from enum import StrEnum
from typing import Annotated

import ferro
from ferro import FerroField, Model


class Provider(StrEnum):
PLAID = "plaid"
MX = "mx" # new member — the live type only has 'plaid'


class Feed(Model):
id: Annotated[int | None, FerroField(primary_key=True)] = None
provider: Provider


await ferro.connect("postgres://...", migrate_updates=True)
# → ALTER TYPE "provider" ADD VALUE IF NOT EXISTS 'mx'

feed = await Feed.create(provider=Provider.MX)
recent = await Feed.where(lambda feed: feed.provider == Provider.MX).all()
```

The contract, precisely:

- **Append-only, metadata-only.** Label addition adds labels and does nothing else; rows are never touched. A shared `StrEnum` used by several models is one type and reconciles once.
- **Removals and renames are never automatic.** A live label the model no longer declares raises a `UserWarning` naming the type and labels — rows may still hold that label, and older code may still be running against the schema mid-deploy — and the label stays. Remove or rename labels in a reviewed Alembic migration.
- **Labels commit before table changes.** Additions run as their own autocommit statements ahead of the per-table plans, so a new column whose literal default is a brand-new member works in a single deploy, on every supported PostgreSQL version.
- **Appended labels sort last.** `ADD VALUE` appends: a member inserted mid-enum in Python lands at the end of the database ordering, and `ORDER BY` on an enum column follows *database* order, not declaration order.
- **SQLite is unaffected.** Enums store as text there; a new member needs no DDL.

### Destructive drops with `migrate_destructive`

*Added in 0.11.0.* Also **drop** live columns that no longer exist on the model (never whole tables):
Expand Down Expand Up @@ -134,6 +204,7 @@ target_metadata = get_metadata()
- **Composite constraints** (`__ferro_composite_uniques__`, `__ferro_composite_indexes__`) emit matching `UniqueConstraint` / `Index` objects, including the automatic constraints on many-to-many join tables.
- **One-to-one** relations (`ForeignKey(unique=True)`) emit the same `UNIQUE` on the shadow column that `auto_migrate` creates at runtime.
- **Enums** map to named `sqlalchemy.Enum` types (class name lowercased, e.g. `UserRole` → `userrole`) so revisions compile on PostgreSQL, which rejects anonymous enum types.
- **Enum label drift is diffed.** *0.18.0+.* Alembic core is blind to enum value changes; ferro's bridge registers an autogenerate comparator that diffs each named enum type against the live PostgreSQL catalog — the same decision (and the same rendered SQL) the auto-migrate pass uses. A grown `StrEnum` generates `ALTER TYPE ... ADD VALUE IF NOT EXISTS` inside an `autocommit_block()` (placed before table operations, runnable on every supported PostgreSQL version); a live label the model no longer declares generates a comment in the revision telling you removal needs a hand-written step. Models in sync generate nothing.

### Autogenerate

Expand Down
12 changes: 12 additions & 0 deletions src/ferro/_core.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -221,3 +221,15 @@ def _resolve_storage_type(column_ir_json: str, dialect: str) -> str:
def _render_check_body(column: str, values: list[str]) -> str:
"""The shared db_check CHECK body, byte-identical to the Rust emitters."""
...

def _plan_enum_label_addition(
type_name: str, declared: list[str], live: list[str]
) -> str:
"""The label-addition decision (ADR-0011) for one enum type.

Returns JSON: ``{"statements": [...], "extra_labels": [...]}`` — the
Rust-rendered ``ADD VALUE IF NOT EXISTS`` statements for model-declared
labels the live type is missing, and the live labels the model no longer
declares (warn-never-act).
"""
...
Loading
Loading