Add plain postgres ready and build connectivity waits into schema commands#80
Open
davegaeddert wants to merge 2 commits into
Open
Add plain postgres ready and build connectivity waits into schema commands#80davegaeddert wants to merge 2 commits into
davegaeddert wants to merge 2 commits into
Conversation
plain postgres sync, plain migrations apply, plain postgres converge, and plain postgres drop-unknown-tables now take a single session-level advisory lock (key 1047265496 = crc32(b"plain/schema")) held on a dedicated connection, so concurrent deploy processes (a retried migrate job, two operators, overlapping release phases) can't interleave schema changes. - Non-blocking acquire with retry via POSTGRES_SCHEMA_LOCK_RETRY_INTERVAL / POSTGRES_SCHEMA_LOCK_MAX_RETRIES (defaults wait up to an hour for long index builds); warns about once a minute with the holder's pid while waiting; timeouts surface as a clean CLI error, not a traceback. - Commands re-plan under the lock, so the loser of a race sees the winner's applied work instead of executing a stale plan; interactive converge aborts if the plan grew past what the operator confirmed; drop-unknown- tables re-checks and skips names that are no longer unknown. - The lock connection uses TCP keepalives, sync re-verifies the lock between its migrate and converge phases (SchemaLockLost), nested acquisition fails fast, and a dead holder self-clears when its session closes.
…mands A readiness engine (plain/postgres/readiness.py) classifies whether the database is ready for this code to serve: reachable, migrations applied, and every model's table and columns present (existence only — drift is convergence's job). `plain postgres ready [--wait --timeout] [--json]` maps the classification to an operator contract: exit 0 = serve, 1 = not ready but retryable, 2 = configuration error a human must fix. DEBUG downgrades schema gaps to warnings at the CLI; data-only (all-RunPython) migrations warn instead of gate. Schema commands (sync, migrations apply, converge) now wait for the database themselves — connection failures retry for up to POSTGRES_WAIT_TIMEOUT seconds while configuration errors fail immediately via the same classifier — so release phases and dev environments no longer need a separate wait step. The `plain postgres wait` command is removed and plain-dev relies on sync's built-in wait. Float-annotated settings now accept int values (PEP 484 numeric tower), so PLAIN_POSTGRES_WAIT_TIMEOUT=30 and friends work naturally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
plain postgres ready— a serving gate for deployment entrypoints. A readiness engine (plain/postgres/readiness.py) classifies whether the database is ready for this code to serve, and the CLI maps it to an operator contract:012--wait [--timeout N]standardizes the initContainer/entrypoint poll loop (config errors still exit 2 immediately);--jsonincludes the exit code alongside the truthful classification.Built-in connectivity waits for schema commands —
sync,migrations apply, andconvergenow wait for the database themselves (up toPOSTGRES_WAIT_TIMEOUT, default 60s), retrying connection failures while failing immediately on configuration errors via the same classifier.plain postgres waitis removed; plain-dev drops its wait step and relies on sync's built-in wait.The resulting grammar: writers wait for connectivity, servers wait for readiness, one classifier decides retry-vs-refuse.
Design decisions
statusand the Python API (check_database_ready()) never lie.RunPythonbackfill shouldn't hold autoscaled capacity hostage while existing pods serve fine.Also: float-annotated settings now accept int values (PEP 484 numeric tower) —
PLAIN_POSTGRES_WAIT_TIMEOUT=30andX = 30in settings.py work naturally instead of failing type validation.Testing
--timeoutbounds.