Skip to content

Add plain postgres ready and build connectivity waits into schema commands#80

Open
davegaeddert wants to merge 2 commits into
masterfrom
postgres-ready-gate
Open

Add plain postgres ready and build connectivity waits into schema commands#80
davegaeddert wants to merge 2 commits into
masterfrom
postgres-ready-gate

Conversation

@davegaeddert

Copy link
Copy Markdown
Member

What this does

plain postgres ready — a serving gate for deployment entrypoints. A readiness engine (plain/postgres/readiness.py) classifies whether the database is ready for this code to serve, and the CLI maps it to an operator contract:

Exit Meaning
0 Ready to serve
1 Not ready, retryable — migrations pending, schema missing objects, DB unreachable
2 Configuration error a human must fix — bad credentials, bad URL, missing database

--wait [--timeout N] standardizes the initContainer/entrypoint poll loop (config errors still exit 2 immediately); --json includes the exit code alongside the truthful classification.

Built-in connectivity waits for schema commandssync, migrations apply, and converge now wait for the database themselves (up to POSTGRES_WAIT_TIMEOUT, default 60s), retrying connection failures while failing immediately on configuration errors via the same classifier. plain postgres wait is removed; plain-dev drops its wait step and relies on sync's built-in wait.

The resulting grammar: writers wait for connectivity, servers wait for readiness, one classifier decides retry-vs-refuse.

Design decisions

  • The verb belongs to the operator. The engine returns a truthful classification; exit-code policy (including DEBUG downgrading schema gaps to warnings) lives in the CLI. JSON status and the Python API (check_database_ready()) never lie.
  • Data-only migrations warn, not gate — a pending all-RunPython backfill shouldn't hold autoscaled capacity hostage while existing pods serve fine.
  • Existence-only schema check — one read-only catalog query for model tables/columns (works against replicas). Drift is convergence's job; extra/unknown objects are invisible, which is what makes code rollbacks against a newer database pass by design (directional satisfaction — a contraction is the rollback horizon).
  • Unknown connection failures default to retryable. psycopg3 exposes no sqlstate on connection-time failures, so classification uses a small marker set for clearly-permanent errors; a DB restarting mid-deploy must never read as a config problem.
  • Direct probe connections, never the pool — pool timeouts are classification-opaque. The engine installs its connection as the active connection so migration-recorder ORM reads ride it too.

Also: float-annotated settings now accept int values (PEP 484 numeric tower) — PLAIN_POSTGRES_WAIT_TIMEOUT=30 and X = 30 in settings.py work naturally instead of failing type validation.

Testing

  • 766 plain-postgres + 433 plain + 26 plain-dev tests pass; type checks clean across all three packages.
  • New public tests pin the classification contract and CLI exit codes; internal tests pin the connection-failure taxonomy and the wait loop.
  • Live-verified against real postgres: wait-through-outage recovery, bad password/scheme/option → exit 2, DEBUG warn-through, JSON reconciliation, --timeout bounds.

plain postgres sync, plain migrations apply, plain postgres converge, and
plain postgres drop-unknown-tables now take a single session-level advisory
lock (key 1047265496 = crc32(b"plain/schema")) held on a dedicated
connection, so concurrent deploy processes (a retried migrate job, two
operators, overlapping release phases) can't interleave schema changes.

- Non-blocking acquire with retry via POSTGRES_SCHEMA_LOCK_RETRY_INTERVAL /
  POSTGRES_SCHEMA_LOCK_MAX_RETRIES (defaults wait up to an hour for long
  index builds); warns about once a minute with the holder's pid while
  waiting; timeouts surface as a clean CLI error, not a traceback.
- Commands re-plan under the lock, so the loser of a race sees the winner's
  applied work instead of executing a stale plan; interactive converge
  aborts if the plan grew past what the operator confirmed; drop-unknown-
  tables re-checks and skips names that are no longer unknown.
- The lock connection uses TCP keepalives, sync re-verifies the lock between
  its migrate and converge phases (SchemaLockLost), nested acquisition fails
  fast, and a dead holder self-clears when its session closes.
…mands

A readiness engine (plain/postgres/readiness.py) classifies whether the
database is ready for this code to serve: reachable, migrations applied,
and every model's table and columns present (existence only — drift is
convergence's job). `plain postgres ready [--wait --timeout] [--json]`
maps the classification to an operator contract: exit 0 = serve,
1 = not ready but retryable, 2 = configuration error a human must fix.
DEBUG downgrades schema gaps to warnings at the CLI; data-only
(all-RunPython) migrations warn instead of gate.

Schema commands (sync, migrations apply, converge) now wait for the
database themselves — connection failures retry for up to
POSTGRES_WAIT_TIMEOUT seconds while configuration errors fail
immediately via the same classifier — so release phases and dev
environments no longer need a separate wait step. The `plain postgres
wait` command is removed and plain-dev relies on sync's built-in wait.

Float-annotated settings now accept int values (PEP 484 numeric tower),
so PLAIN_POSTGRES_WAIT_TIMEOUT=30 and friends work naturally.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant