From f26860f68530d224905809288da8231bf83fad88 Mon Sep 17 00:00:00 2001 From: andi Date: Mon, 1 Jun 2026 15:28:56 +0200 Subject: [PATCH 1/6] A stagable a/b concept --- SPEC.md | 277 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 277 insertions(+) create mode 100644 SPEC.md diff --git a/SPEC.md b/SPEC.md new file mode 100644 index 0000000..e094760 --- /dev/null +++ b/SPEC.md @@ -0,0 +1,277 @@ +# pg-dev — a/b backends behind pgbouncer + +A specification for the next iteration of this repo's Postgres dev wrapper. + +## Purpose + +This setup exists for **two** things, both squarely in the local-development +loop: + +1. **Make 90-minute `pg_restore` imports non-blocking.** While a fresh dump is + loading, the previously imported database stays available for queries. +2. **Preserve the existing snapshottability** of an already-imported state, so + schema/data migrations can be tested by snapshot-then-restore as today. + +Switching between "the database I was using" and "the freshly imported one" +should be *mostly* seamless — clients keep their endpoint, the change is a +single command, the worst case is a reconnect. + +## Non-purpose + +This is **not** a production or HA design. Specifically: + +- No failover, no replication, no zero-downtime SLA. +- No protection against a malicious actor on the host. The colima VM is a + trusted boundary. +- No multi-user concerns. One developer, one laptop. + +Anything that adds friction in the name of robustness — extra auth steps, +credential rotation rituals, certificate management — is out of scope. + +## Architecture + +Three containers inside the colima VM. The bouncer container runs **two** +pgbouncer processes side-by-side, one per client-facing port: + +``` + ┌──────────────────────────────────────┐ +client (app) ──►│ pg-bouncer (stable IP 10.x.x.10) │ + │ │ +pg_restore ──►│ :5432 ── active ini ─┐ │ + │ :5433 ── staging ini ─┼─ cross │ + │ │ connect │ + │ admin :6432 / :6433 │ │ + └──────────────┬───────────┴───────────┘ + │ + ┌────────────┴────────────┐ + │ :5432 → active backend │ + │ :5433 → staging backend │ + ▼ ▼ + pg-dev-a pg-dev-b + (Postgres 17, (Postgres 17, + own snapshots, own snapshots, + own data) own data) +``` + +- `pg-bouncer` — single container, fixed identity, lives for the life of the + colima VM. Hosts two pgbouncer instances: + - **active** — `listen_port=5432`, admin `6432`, `[databases] $PG_DB` → + whichever backend is currently active. + - **staging** — `listen_port=5433`, admin `6433`, `[databases] $PG_DB` → + the *other* backend. + Both `.ini` files are rendered from the single state file `var/active-slot`, + so they are always cross-connected and cannot drift. +- `pg-dev-a`, `pg-dev-b` — symmetric backends, both long-lived, both running. + At any moment one is **active** (served on :5432) and the other is + **staging** (served on :5433). + +Client-facing semantics are stable forever; the physical slot underneath +flips, the port semantics don't: + +- **:5432** = current data — psql, application, anything that wants the + active dataset. +- **:5433** = where new dumps land / verify staging — `pg_restore` aims here, + sanity checks aim here. + +There is no third "transient" container during import. The staging slot is +permanently there, ready to receive the next dump. + +## Network identity + +The user-facing endpoint is **the pgbouncer container's IP, on ports 5432 and +5433**. That IP must be stable across the life of the colima VM and not +change on promote, restart, or backend snapshot/restore. This is achieved by +pinning the pgbouncer container's address at the device level, e.g.: + +``` +incus config device override pg-bouncer eth0 ipv4.address=10.x.x.10 +``` + +The two backend containers' IPs are an internal detail. Tooling reaches them +by container name (`pg-dev-a`, `pg-dev-b`) via incus's bridge DNS; they are +never written into a client `.pgpass` or connection string. + +## Authentication + +Single guiding principle: **set up `.pgpass` once, never touch it again.** + +- One Postgres role (`$PG_USER` from `.env`) with one password, used for both + application access through the bouncer and direct backend operations. +- That same role/password is registered in a single + `/etc/pgbouncer/userlist.txt`, shared by both pgbouncer instances and + generated once during `cmd_up`. No `auth_query`, no SCRAM verifier + regeneration after every reconfigure. +- Both pgbouncer pools run in **session** mode — preserves prepared + statements, `SET`, advisory locks; behaves indistinguishably from a direct + connection from the client's point of view. +- Each pgbouncer's admin interface is exposed only on `127.0.0.1` inside the + bouncer container (`:6432` active, `:6433` staging). Reachable via + `incus exec pg-bouncer -- psql …` for promote/observability. Not exposed + to the host. +- Backend Postgres `pg_hba.conf` accepts the role from the bouncer's IP and + from the local socket. No host-wide open auth. + +The `.pgpass` line a developer adds once is port-wildcarded so it covers both +listeners: + +``` +10.x.x.10:*:*:$PG_USER:$PG_PASSWORD +``` + +After that, `psql`, application clients, migration tools and `pg_restore` all +just work, forever, regardless of which slot is active. + +## Snapshot model + +Snapshots are **per backend slot**. `pg-dev-a` and `pg-dev-b` have +independent snapshot timelines that never merge or get copied between slots. + +- `initial` on each slot = a clean, role-and-database-bootstrapped Postgres + with no user data. Taken once during `cmd_up`, used to reset that slot + before each import. +- Subsequent named snapshots on a slot record points-in-time after data has + been loaded or modified on that slot. + +The active slot keeps producing snapshots as you work. The staging slot +typically holds `initial` (waiting for the next import) plus, briefly during +an import, intermediate marks like `initial-loaded`. + +A promote does *not* touch snapshots. The previously active slot keeps its +full timeline as the rollback path. + +## Workflow + +### Steady state (no import in flight) + +The active slot serves queries through the bouncer on :5432. The staging slot +is running but idle, reachable through the bouncer on :5433, holding its +`initial` snapshot. Day-to-day snapshot/restore on the active slot works +exactly as today's `make pg.snapshot` / `make pg.restore`. + +### Importing a new dump + +```shell +# 1. Reset staging to its clean initial state. +make pg.staging.reset + +# 2. Run pg_restore through the bouncer's staging port. The active port +# (:5432) keeps serving live queries the whole time. +pg_restore … --host=10.x.x.10 --port=5433 --dbname=$PG_DB … # ~90 min + +# 3. Sanity check the new data while still on staging. +psql --host=10.x.x.10 --port=5433 --dbname=$PG_DB -c '\dt' + +# 4. Take a checkpoint of the loaded state. +make pg.staging.snapshot name=initial-loaded + +# 5. Promote. Atomic from the client's point of view (PAUSE → reload → RESUME +# on both admin consoles). Both port mappings flip together. +make pg.promote +``` + +After step 5, the slot that was staging is now active (served on :5432). The +slot that was active becomes the new staging (served on :5433) — still +holding its data and snapshots, ready to be either rolled back to (re-promote) +or reset for the next import. + +### Rollback + +A bad import is undone by a second `make pg.promote`. The previously active +slot is untouched; the pointer just flips back. No data is regenerated. + +## Command surface + +Two parallel families plus the flip, mirroring today's surface. The staging +family operates **directly on the backend container** via `incus exec`, +because it's used for ops/snapshot work — not through the bouncer: + +| acts on active | acts on staging | meaning | +| -------------------------- | ------------------------------ | ------------------------ | +| `make pg.psql` | `make pg.staging.psql` | psql into that slot (via incus exec) | +| `make pg.shell` | `make pg.staging.shell` | bash in that slot | +| `make pg.logs` | `make pg.staging.logs` | tail postgres logs | +| `make pg.snapshot name=…` | `make pg.staging.snapshot …` | snapshot that slot | +| `make pg.restore name=…` | `make pg.staging.restore …` | restore on that slot | +| `make pg.restore-last` | `make pg.staging.restore-last` | restore most recent | +| `make pg.snapshots` | `make pg.staging.snapshots` | list snapshots | +| | `make pg.staging.reset` | shortcut: restore `initial` | + +Plus the bouncer-aware operations: + +| command | meaning | +| -------------------- | ------------------------------------------------------ | +| `make pg.endpoint` | print both bouncer ports and their roles (see below) | +| `make pg.promote` | flip active/staging on both bouncer instances at once | +| `make pg.status` | print pointer + state of all three containers | + +`make pg.endpoint` prints both port mappings so a client always knows where +to point what: + +``` +active host=10.x.x.10 port=5432 dbname=$PG_DB (current data) +staging host=10.x.x.10 port=5433 dbname=$PG_DB (import target / opposite of active) +``` + +Direct-to-backend tooling (`pg.psql`, `pg.staging.psql`, `pg.shell`, +snapshot/restore ops) goes via `incus exec` against the container, because +snapshots are an incus-level operation on the container, not a Postgres-level +one. The bouncer is in the path of client applications and the import +workflow. + +`pg.staging.host` is no longer needed for the import workflow — the import +endpoint is just `10.x.x.10:5433`. It may be retained as an ops convenience +(printing the staging container's container-level IP for direct-to-backend +debugging), but it is not part of the documented import path. + +## State + +A single file `var/active-slot` contains the literal text `a` or `b` and is +the source of truth for which slot is active. It is written atomically +(tmpfile + rename) and is the *only* thing `make pg.promote` mutates besides +the two pgbouncer `[databases]` lines (one per `.ini`). + +Loss of `var/active-slot` is recoverable: either `.ini` is authoritative +(whatever the :5432 ini names is `a` or `b`; the file just caches that +decision for shell tooling). + +## Implementation outline + +`cmd_up` (one-time provisioning per colima VM): + +1. Launch `pg-dev-a`, install Postgres 17, write config, create role and DB, + `incus snapshot create pg-dev-a initial`. +2. Launch `pg-dev-b` the same way (independent install — keeps the two truly + symmetric and decoupled). Snapshot `initial`. +3. Launch `pg-bouncer`, install pgbouncer, pin its eth0 to the chosen stable + IP. +4. Render `/etc/pgbouncer/userlist.txt` once from `$PG_USER`/`$PG_PASSWORD`, + shared by both instances. +5. Render two `.ini` files from a single template + initial state + (`active-slot=a`): + - `pgbouncer-active.ini` → `listen_port=5432`, admin `6432`, + `$PG_DB` → `pg-dev-a`. + - `pgbouncer-staging.ini` → `listen_port=5433`, admin `6433`, + `$PG_DB` → `pg-dev-b`. +6. Set up systemd template instances `pgbouncer@active` and + `pgbouncer@staging` reading the matching `.ini`, sharing `userlist.txt`. + Enable and start both. +7. Write `var/active-slot=a`. +8. Print `pg.endpoint` and a ready-to-paste `.pgpass` line. + +`cmd_promote`: + +1. Read `var/active-slot`, derive the new `(active, staging)` pair (a/b + swapped). +2. `_bouncer_admin 6432 "PAUSE $PG_DB;"` and + `_bouncer_admin 6433 "PAUSE $PG_DB;"`. +3. Re-render both `.ini` files from the new state, from the same shared + template. They remain cross-connected by construction. +4. `_bouncer_admin 6432 "RELOAD; RESUME $PG_DB;"` and + `_bouncer_admin 6433 "RELOAD; RESUME $PG_DB;"`. +5. Atomically write the new value of `var/active-slot`. +6. Print new status. + +Total promote wall-clock: sub-second. Both port mappings flip together, +dominated by the time PAUSE waits for in-flight transactions to finish (none +on :5433 during the import workflow, because `pg_restore` is the only thing +that talks to it and it has already finished by step 4 of the workflow). From c12abaa24df6978a94367cc1afb7db38f28203ac Mon Sep 17 00:00:00 2001 From: andi Date: Mon, 1 Jun 2026 15:48:23 +0200 Subject: [PATCH 2/6] impl --- .env.example | 12 + Makefile | 100 ++++++-- SPEC.md | 39 ++- scripts/pg-dev-local | 600 +++++++++++++++++++++++++++++++++++-------- 4 files changed, 603 insertions(+), 148 deletions(-) diff --git a/.env.example b/.env.example index 144ac38..f013f4d 100644 --- a/.env.example +++ b/.env.example @@ -4,3 +4,15 @@ PG_PASSWORD=oos4ookahc7si4eejooFee9teePeiqu3quuihuo9Ahhoo7eegh4ohj4fi7gai3vu COLIMA_MEMORY=12 COLIMA_CPU=4 + +# Optional: pin the pg-bouncer container's static IP. +# +# Default (when this is unset): the script reads +# incus network get incusbr0 ipv4.address +# and uses .10 inside whatever subnet incusbr0 happens to be on. So with +# a 192.168.100.0/24 bridge you get 192.168.100.10; with a 10.224.165.0/24 +# bridge you get 10.224.165.10; etc. +# +# Override here only if you need a specific address (e.g. you already have +# something on .10 in that subnet). +# PG_BOUNCER_IP= diff --git a/Makefile b/Makefile index 5230359..29e5459 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,9 @@ +packages := incus colima jq + .PHONY: deps deps: - HOMEBREW_NO_AUTO_UPDATE=1 brew install incus colima - HOMEBREW_NO_AUTO_UPDATE=1 brew upgrade incus colima + HOMEBREW_NO_AUTO_UPDATE=1 brew install $(packages) + HOMEBREW_NO_AUTO_UPDATE=1 brew upgrade $(packages) -include .env export @@ -23,20 +25,50 @@ status: .PHONY: stop stop: - colima stop \ - --verbose + colima stop --verbose PG_DEV := scripts/pg-dev-local +# ----- lifecycle ---------------------------------------------------------- + .PHONY: pg.up pg.up: $(PG_DEV) up - $(MAKE) pg.logs .PHONY: pg.down pg.down: $(PG_DEV) down +.PHONY: pg.status +pg.status: + $(PG_DEV) status + +.PHONY: pg.endpoint +pg.endpoint: + @$(PG_DEV) endpoint + +.PHONY: pg.promote +pg.promote: + $(PG_DEV) promote + +# ----- active backend ----------------------------------------------------- + +.PHONY: pg.psql +pg.psql: + $(PG_DEV) psql + +.PHONY: pg.shell +pg.shell: + $(PG_DEV) shell + +.PHONY: pg.ip +pg.ip: + @$(PG_DEV) ip + +.PHONY: pg.logs +pg.logs: + $(PG_DEV) logs + .PHONY: pg.snapshot pg.snapshot: $(PG_DEV) snapshot $(name) $(if $(force),--force,) @@ -53,21 +85,51 @@ pg.restore-last: pg.snapshots: $(PG_DEV) snapshots -.PHONY: pg.ip -pg.ip: - $(PG_DEV) ip +# ----- staging backend ---------------------------------------------------- -.PHONY: pg.psql -pg.psql: - $(PG_DEV) psql +.PHONY: pg.staging.psql +pg.staging.psql: + $(PG_DEV) staging.psql -.PHONY: pg.shell -pg.shell: - $(PG_DEV) shell +.PHONY: pg.staging.shell +pg.staging.shell: + $(PG_DEV) staging.shell -.PHONY: pg.status -pg.status: - $(PG_DEV) status +.PHONY: pg.staging.ip +pg.staging.ip: + @$(PG_DEV) staging.ip + +.PHONY: pg.staging.logs +pg.staging.logs: + $(PG_DEV) staging.logs + +.PHONY: pg.staging.snapshot +pg.staging.snapshot: + $(PG_DEV) staging.snapshot $(name) $(if $(force),--force,) + +.PHONY: pg.staging.restore +pg.staging.restore: + $(PG_DEV) staging.restore $(name) + +.PHONY: pg.staging.restore-last +pg.staging.restore-last: + $(PG_DEV) staging.restore-last + +.PHONY: pg.staging.snapshots +pg.staging.snapshots: + $(PG_DEV) staging.snapshots + +.PHONY: pg.staging.reset +pg.staging.reset: + $(PG_DEV) staging.reset + +# ----- bouncer ------------------------------------------------------------ + +.PHONY: pg.bouncer.logs +pg.bouncer.logs: + $(PG_DEV) bouncer.logs + +# ----- export / import (active backend) ----------------------------------- .PHONY: pg.export pg.export: @@ -77,9 +139,7 @@ pg.export: pg.import-last: $(PG_DEV) import-last -.PHONY: pg.logs -pg.logs: - incus exec pg-dev -- tail -f /var/log/postgresql/postgresql-17-main.log +# ----- colima ------------------------------------------------------------ .PHONY: delete delete: diff --git a/SPEC.md b/SPEC.md index e094760..0a46021 100644 --- a/SPEC.md +++ b/SPEC.md @@ -39,8 +39,9 @@ client (app) ──►│ pg-bouncer (stable IP 10.x.x.10) │ │ │ pg_restore ──►│ :5432 ── active ini ─┐ │ │ :5433 ── staging ini ─┼─ cross │ - │ │ connect │ - │ admin :6432 / :6433 │ │ + │ (admin is the special │ connect │ + │ `pgbouncer` db on same │ │ + │ port, pgb_admin user) │ │ └──────────────┬───────────┴───────────┘ │ ┌────────────┴────────────┐ @@ -55,10 +56,14 @@ pg_restore ──►│ :5432 ── active ini ─┐ │ - `pg-bouncer` — single container, fixed identity, lives for the life of the colima VM. Hosts two pgbouncer instances: - - **active** — `listen_port=5432`, admin `6432`, `[databases] $PG_DB` → - whichever backend is currently active. - - **staging** — `listen_port=5433`, admin `6433`, `[databases] $PG_DB` → - the *other* backend. + - **active** — `listen_port=5432`, `[databases] $PG_DB` → whichever + backend is currently active. + - **staging** — `listen_port=5433`, `[databases] $PG_DB` → the *other* + backend. + + pgbouncer has no separate admin port: admin commands are issued by + connecting to the special `pgbouncer` virtual database on the same + `listen_port` as the `pgb_admin` user. Both `.ini` files are rendered from the single state file `var/active-slot`, so they are always cross-connected and cannot drift. - `pg-dev-a`, `pg-dev-b` — symmetric backends, both long-lived, both running. @@ -104,10 +109,15 @@ Single guiding principle: **set up `.pgpass` once, never touch it again.** - Both pgbouncer pools run in **session** mode — preserves prepared statements, `SET`, advisory locks; behaves indistinguishably from a direct connection from the client's point of view. -- Each pgbouncer's admin interface is exposed only on `127.0.0.1` inside the - bouncer container (`:6432` active, `:6433` staging). Reachable via - `incus exec pg-bouncer -- psql …` for promote/observability. Not exposed - to the host. +- Each pgbouncer's admin interface is the special `pgbouncer` virtual + database on its own `listen_port` (`:5432` for active, `:5433` for + staging), accessed by connecting as `pgb_admin` (declared in + `admin_users` and listed in `userlist.txt`). Promote and observability + go through `incus exec pg-bouncer -- psql -p {5432,5433} -U pgb_admin + -d pgbouncer …`. +- The `pgb_admin` user shares the application user's password. One secret, + one `.pgpass` line — dev convenience, explicit non-goal w.r.t. privilege + separation. - Backend Postgres `pg_hba.conf` accepts the role from the bouncer's IP and from the local socket. No host-wide open auth. @@ -262,12 +272,13 @@ decision for shell tooling). 1. Read `var/active-slot`, derive the new `(active, staging)` pair (a/b swapped). -2. `_bouncer_admin 6432 "PAUSE $PG_DB;"` and - `_bouncer_admin 6433 "PAUSE $PG_DB;"`. +2. `_bouncer_admin active "PAUSE $PG_DB;"` and + `_bouncer_admin staging "PAUSE $PG_DB;"`. 3. Re-render both `.ini` files from the new state, from the same shared template. They remain cross-connected by construction. -4. `_bouncer_admin 6432 "RELOAD; RESUME $PG_DB;"` and - `_bouncer_admin 6433 "RELOAD; RESUME $PG_DB;"`. +4. `_bouncer_admin active "RELOAD;"` / `_bouncer_admin staging "RELOAD;"` and + `_bouncer_admin active "RESUME $PG_DB;"` / + `_bouncer_admin staging "RESUME $PG_DB;"`. 5. Atomically write the new value of `var/active-slot`. 6. Print new status. diff --git a/scripts/pg-dev-local b/scripts/pg-dev-local index 7c2cb92..05cb7a7 100755 --- a/scripts/pg-dev-local +++ b/scripts/pg-dev-local @@ -1,41 +1,145 @@ #!/usr/bin/env bash set -euo pipefail -CONTAINER=pg-dev -IMAGE=images:ubuntu/24.04/cloud +# pg-dev-local — two PostgreSQL backends behind a dual-listener pgbouncer. +# See SPEC.md for the design. +# +# pg-dev-a, pg-dev-b long-lived backends with their own snapshot timelines +# pg-bouncer one container, two pgbouncer instances: +# :5432 active (current data) +# :5433 staging (import target / opposite of active) +# cross-connect enforced by rendering both inis from +# one state file (var/active-slot). REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" # shellcheck source=../.env [[ -f "$REPO_ROOT/.env" ]] && source "$REPO_ROOT/.env" -PG_USER="${PG_USER}" -PG_DB="${PG_DB}" -PG_PASSWORD="${PG_PASSWORD}" +PG_USER="${PG_USER:?PG_USER must be set in .env}" +PG_DB="${PG_DB:?PG_DB must be set in .env}" +PG_PASSWORD="${PG_PASSWORD:?PG_PASSWORD must be set in .env}" -cmd_up() { - if incus info "$CONTAINER" > /dev/null 2>&1; then - echo "Container '$CONTAINER' already exists. Use: pg-dev-local status" +# Same secret for the bouncer admin user. Dev convenience: one .pgpass line +# covers everything. SPEC.md says this loudly. +PG_BOUNCER_ADMIN_PASSWORD="$PG_PASSWORD" + +IMAGE=images:ubuntu/24.04/cloud +BACKEND_PREFIX=pg-dev +BACKEND_A="${BACKEND_PREFIX}-a" +BACKEND_B="${BACKEND_PREFIX}-b" +BOUNCER=pg-bouncer +STATE_DIR="$REPO_ROOT/var" +STATE_FILE="$STATE_DIR/active-slot" + +# --------------------------------------------------------------------------- +# State helpers +# --------------------------------------------------------------------------- + +_active() { cat "$STATE_FILE" 2>/dev/null || echo a; } +_staging() { [[ "$(_active)" == a ]] && echo b || echo a; } +_c_active() { echo "${BACKEND_PREFIX}-$(_active)"; } +_c_staging() { echo "${BACKEND_PREFIX}-$(_staging)"; } + +_set_active() { + local new="$1" + mkdir -p "$STATE_DIR" + local tmp + tmp="$(mktemp "$STATE_FILE.XXXXXX")" + printf '%s\n' "$new" > "$tmp" + mv "$tmp" "$STATE_FILE" +} + +# --------------------------------------------------------------------------- +# Bouncer helpers +# --------------------------------------------------------------------------- + +_bouncer_ip() { + incus list "$BOUNCER" --format csv -c 4 \ + | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' \ + | head -1 +} + +# Derive a stable IP for the bouncer in the incusbr0 subnet (e.g. 10.x.x.10), +# unless PG_BOUNCER_IP is set in .env. +_pick_bouncer_ip() { + if [[ -n "${PG_BOUNCER_IP:-}" ]]; then + echo "$PG_BOUNCER_IP" + return + fi + local addr + addr=$(incus network get incusbr0 ipv4.address 2>/dev/null || true) + if [[ -z "$addr" || "$addr" == "none" ]]; then + echo "ERROR: cannot read incusbr0 ipv4.address; set PG_BOUNCER_IP in .env" >&2 exit 1 fi + # addr like "10.224.165.1/24" → prefix "10.224.165" → pick .10 + local prefix=${addr%.*} + prefix=${prefix%%/*} + echo "${prefix}.10" +} + +# _bouncer_admin +# Admin access is the special `pgbouncer` virtual database on the same +# listen_port as data — pgbouncer doesn't expose a separate admin port. +_bouncer_admin() { + local role="$1"; shift + local port + case "$role" in + active) port=5432 ;; + staging) port=5433 ;; + *) echo "bad role: $role" >&2; exit 1 ;; + esac + incus exec "$BOUNCER" -- env PGPASSWORD="$PG_BOUNCER_ADMIN_PASSWORD" \ + psql -h 127.0.0.1 -p "$port" -U pgb_admin -d pgbouncer -c "$*" +} + +# _render_ini +_render_ini() { + local role="$1" port="$2" backend="$3" + incus exec "$BOUNCER" -- tee "/etc/pgbouncer/${role}.ini" > /dev/null < Launching $CONTAINER (may take a minute if image is not cached)..." - incus launch --verbose "$IMAGE" "$CONTAINER" +# --------------------------------------------------------------------------- +# Provisioning +# --------------------------------------------------------------------------- - echo "==> Waiting for IPv4 address..." - until incus list "$CONTAINER" --format csv -c 4 | grep -qE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+'; do +_provision_backend() { + local container="$1" + + echo "==> Launching $container..." + incus launch --verbose "$IMAGE" "$container" + + echo "==> Waiting for IPv4 on $container..." + until incus list "$container" --format csv -c 4 | grep -qE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+'; do sleep 2 done - echo "==> Installing PostgreSQL 17..." - incus exec --verbose "$CONTAINER" -- bash << 'SETUP' + echo "==> Installing PostgreSQL 17 on $container..." + incus exec --verbose "$container" -- bash <<'SETUP' set -euo pipefail export DEBIAN_FRONTEND=noninteractive -echo "--- Installing prerequisites ---" apt-get update -q apt-get install -y curl gnupg lsb-release -echo "--- Adding PGDG repository ---" curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc \ | gpg --dearmor -o /etc/apt/trusted.gpg.d/pgdg.gpg echo "deb https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" \ @@ -43,10 +147,8 @@ echo "deb https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" apt-get update -q apt-get install -y --no-install-recommends postgresql-17 -echo "--- Writing configuration ---" mkdir -p /etc/postgresql/17/main/conf.d - -cat > /etc/postgresql/17/main/conf.d/99-dev.conf << 'PGCONF' +cat > /etc/postgresql/17/main/conf.d/99-dev.conf <<'PGCONF' listen_addresses = '*' dynamic_shared_memory_type = mmap @@ -75,170 +177,440 @@ deadlock_timeout = 100 debug_pretty_print = on PGCONF -cat > /etc/postgresql/17/main/pg_hba.conf << 'PGHBA' +cat > /etc/postgresql/17/main/pg_hba.conf <<'PGHBA' local all postgres peer local all all md5 host all all 0.0.0.0/0 scram-sha-256 host all all ::/0 scram-sha-256 PGHBA -echo "--- Starting PostgreSQL ---" systemctl restart postgresql@17-main SETUP - echo "==> Creating role '$PG_USER' and database '$PG_DB'..." - incus exec --verbose "$CONTAINER" -- su -c psql postgres << SQL + echo "==> Creating role '$PG_USER' and database '$PG_DB' on $container..." + incus exec --verbose "$container" -- su -c psql postgres < PostgreSQL ready" - echo " host (container): $ip" - echo " user: $PG_USER" - echo " database: $PG_DB" - echo " incus exec $CONTAINER -- psql -U $PG_USER $PG_DB" + echo "==> Snapshotting $container @ initial..." + incus pause "$container" + incus snapshot create "$container" initial + incus resume "$container" +} + +_provision_bouncer() { + local bouncer_ip="$1" + + echo "==> Launching $BOUNCER..." + incus launch --verbose "$IMAGE" "$BOUNCER" + + echo "==> Waiting for IPv4 on $BOUNCER..." + until incus list "$BOUNCER" --format csv -c 4 | grep -qE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+'; do + sleep 2 + done + + echo "==> Pinning $BOUNCER eth0 → $bouncer_ip..." + incus config device override "$BOUNCER" eth0 ipv4.address="$bouncer_ip" + incus restart "$BOUNCER" + until incus list "$BOUNCER" --format csv -c 4 | grep -q "$bouncer_ip"; do + sleep 1 + done + + echo "==> Installing pgbouncer + psql client on $BOUNCER..." + incus exec --verbose "$BOUNCER" -- bash <<'SETUP' +set -euo pipefail +export DEBIAN_FRONTEND=noninteractive + +apt-get update -q +apt-get install -y --no-install-recommends pgbouncer postgresql-client + +# The Debian package ships a single-instance unit. We replace it with a +# template so we can run active and staging side by side. +systemctl disable --now pgbouncer.service 2>/dev/null || true + +mkdir -p /var/run/pgbouncer-active /var/run/pgbouncer-staging \ + /var/run/pgbouncer /var/log/pgbouncer +chown postgres:postgres /var/run/pgbouncer-active /var/run/pgbouncer-staging \ + /var/run/pgbouncer /var/log/pgbouncer + +cat > /etc/systemd/system/pgbouncer@.service <<'UNIT' +[Unit] +Description=pgbouncer (%i instance) +After=network.target + +[Service] +Type=simple +User=postgres +Group=postgres +ExecStart=/usr/sbin/pgbouncer /etc/pgbouncer/%i.ini +Restart=on-failure +RestartSec=1 + +[Install] +WantedBy=multi-user.target +UNIT + +systemctl daemon-reload +SETUP + + echo "==> Writing userlist.txt..." + incus exec "$BOUNCER" -- tee /etc/pgbouncer/userlist.txt > /dev/null < Rendering pgbouncer inis (active→$BACKEND_A, staging→$BACKEND_B)..." + _render_ini active 5432 "$BACKEND_A" + _render_ini staging 5433 "$BACKEND_B" + + echo "==> Enabling and starting pgbouncer@{active,staging}..." + incus exec "$BOUNCER" -- systemctl enable --now \ + pgbouncer@active pgbouncer@staging + + # Tiny readiness probe. + local tries=0 + until _bouncer_admin active "SHOW VERSION;" >/dev/null 2>&1; do + tries=$((tries + 1)) + if (( tries > 20 )); then + echo "ERROR: pgbouncer@active not responding on :5432" >&2 + incus exec "$BOUNCER" -- journalctl -u pgbouncer@active --no-pager | tail -n 40 >&2 + exit 1 + fi + sleep 1 + done + tries=0 + until _bouncer_admin staging "SHOW VERSION;" >/dev/null 2>&1; do + tries=$((tries + 1)) + if (( tries > 20 )); then + echo "ERROR: pgbouncer@staging not responding on :5433" >&2 + incus exec "$BOUNCER" -- journalctl -u pgbouncer@staging --no-pager | tail -n 40 >&2 + exit 1 + fi + sleep 1 + done +} + +# --------------------------------------------------------------------------- +# Lifecycle commands +# --------------------------------------------------------------------------- + +cmd_up() { + if incus info pg-dev > /dev/null 2>&1; then + cat < /dev/null 2>&1; then + echo "Container '$c' already exists — run '$0 down' first." + exit 1 + fi + done + + local bouncer_ip + bouncer_ip=$(_pick_bouncer_ip) + echo "==> pg-bouncer will be pinned to $bouncer_ip" + + _provision_backend "$BACKEND_A" + _provision_backend "$BACKEND_B" + _provision_bouncer "$bouncer_ip" + + _set_active a + + echo + echo "==> pg-dev ready." + echo + cmd_endpoint } cmd_down() { - incus delete "$CONTAINER" --force + for c in "$BOUNCER" "$BACKEND_A" "$BACKEND_B"; do + incus delete "$c" --force 2>/dev/null || true + done + rm -f "$STATE_FILE" } -cmd_snapshot() { - local name="${1:?Usage: pg-dev-local snapshot [--force]}" - shift - if [[ "${1:-}" == "--force" ]]; then - incus snapshot delete "$CONTAINER" "$name" 2>/dev/null || true +cmd_status() { + local a s + a=$(_active); s=$(_staging) + echo "active slot: $a (active = ${BACKEND_PREFIX}-${a}, staging = ${BACKEND_PREFIX}-${s})" + echo + incus list '^(pg-dev-(a|b)|pg-bouncer)$' +} + +cmd_endpoint() { + local ip + ip=$(_bouncer_ip) + if [[ -z "$ip" ]]; then + echo "ERROR: pg-bouncer has no IP — is it running?" >&2 + exit 1 fi - incus pause "$CONTAINER" - incus snapshot create "$CONTAINER" "$name" - incus resume "$CONTAINER" - echo "Snapshot '$name' created." + cat </dev/null || true +cmd_promote() { + local from to + from=$(_active) + to=$(_staging) + + echo "==> Promoting: active $from → $to" + + # PAUSE drains in-flight transactions; admin commands themselves are + # unaffected, so we can still RELOAD / RESUME while paused. + _bouncer_admin active "PAUSE $PG_DB;" >/dev/null + _bouncer_admin staging "PAUSE $PG_DB;" >/dev/null + + _render_ini active 5432 "${BACKEND_PREFIX}-${to}" + _render_ini staging 5433 "${BACKEND_PREFIX}-${from}" + + _bouncer_admin active "RELOAD;" >/dev/null + _bouncer_admin staging "RELOAD;" >/dev/null + _bouncer_admin active "RESUME $PG_DB;" >/dev/null + _bouncer_admin staging "RESUME $PG_DB;" >/dev/null + + _set_active "$to" + + echo "==> Promoted. Active: ${BACKEND_PREFIX}-${to} Staging: ${BACKEND_PREFIX}-${from}" + echo + cmd_status +} + +# --------------------------------------------------------------------------- +# Per-slot ops +# --------------------------------------------------------------------------- + +_snapshot() { + local container="$1" name="$2" force="${3:-}" + if [[ "$force" == "--force" ]]; then + incus snapshot delete "$container" "$name" 2>/dev/null || true + fi + incus pause "$container" + incus snapshot create "$container" "$name" + incus resume "$container" + echo "Snapshot '$name' created on $container." +} + +_restore() { + local container="$1" name="$2" + incus stop "$container" --force 2>/dev/null || true # ZFS requires removing snapshots taken after the target before restoring. - local snapshots_after - snapshots_after=$(incus snapshot list "$CONTAINER" -f json \ + local after + after=$(incus snapshot list "$container" -f json \ | jq -r --arg name "$name" ' . as $all | ($all | map(.name) | index($name)) as $idx | if $idx == null then [] else $all[$idx+1:] end | .[].name ') - if [[ -n "$snapshots_after" ]]; then - echo "==> The following snapshots will be deleted to allow restore:" - echo "$snapshots_after" | sed 's/^/ /' + if [[ -n "$after" ]]; then + echo "==> Snapshots on $container that will be deleted:" + echo "$after" | sed 's/^/ /' local reply read -r -p "Continue? [Y/n] " reply [[ "${reply:-Y}" =~ ^[Yy]$ ]] || { echo "Aborted."; exit 1; } while IFS= read -r snap; do - incus snapshot delete "$CONTAINER" "$snap" - done <<< "$snapshots_after" + incus snapshot delete "$container" "$snap" + done <<< "$after" fi - incus snapshot restore "$CONTAINER" "$name" - incus start "$CONTAINER" - echo "Restored to snapshot '$name'." + incus snapshot restore "$container" "$name" + incus start "$container" + echo "Restored $container to snapshot '$name'." } -cmd_restore_last() { +_restore_last() { + local container="$1" local name - name=$(incus snapshot list "$CONTAINER" -f json \ + name=$(incus snapshot list "$container" -f json \ | jq -r 'if length == 0 then empty else sort_by(.created_at) | last | .name end') if [[ -z "$name" ]]; then - echo "No snapshots found for '$CONTAINER'." + echo "No snapshots on $container." exit 1 fi - echo "==> Restoring to most recent snapshot: $name" - cmd_restore "$name" + echo "==> Restoring $container to most recent snapshot: $name" + _restore "$container" "$name" } -cmd_snapshots() { - if [[ "${1:-}" == "--json" ]]; then - incus snapshot list "$CONTAINER" -f json - else - incus snapshot list "$CONTAINER" - fi +# active slot +cmd_snapshot() { + local name="${1:?Usage: snapshot [--force]}"; shift || true + _snapshot "$(_c_active)" "$name" "${1:-}" +} +cmd_restore() { + local name="${1:?Usage: restore }" + _restore "$(_c_active)" "$name" } +cmd_restore_last() { _restore_last "$(_c_active)"; } +cmd_snapshots() { incus snapshot list "$(_c_active)"; } -cmd_ip() { - incus list "$CONTAINER" --format csv -c 4 \ +# staging slot +cmd_staging_snapshot() { + local name="${1:?Usage: staging.snapshot [--force]}"; shift || true + _snapshot "$(_c_staging)" "$name" "${1:-}" +} +cmd_staging_restore() { + local name="${1:?Usage: staging.restore }" + _restore "$(_c_staging)" "$name" +} +cmd_staging_restore_last() { _restore_last "$(_c_staging)"; } +cmd_staging_snapshots() { incus snapshot list "$(_c_staging)"; } +cmd_staging_reset() { _restore "$(_c_staging)" initial; } + +# Shell / psql / ip / logs, per slot. +_psql() { + local container="$1"; shift + incus exec "$container" -- env PGPASSWORD="$PG_PASSWORD" \ + psql -U "$PG_USER" "$PG_DB" "$@" +} +cmd_psql() { _psql "$(_c_active)" "$@"; } +cmd_staging_psql() { _psql "$(_c_staging)" "$@"; } + +cmd_shell() { incus exec "$(_c_active)" -- bash; } +cmd_staging_shell() { incus exec "$(_c_staging)" -- bash; } + +_ip() { + incus list "$1" --format csv -c 4 \ | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' \ | head -1 } +cmd_ip() { _ip "$(_c_active)"; } +cmd_staging_ip() { _ip "$(_c_staging)"; } -cmd_psql() { - # Container IPs are on the incus bridge inside the Lima VM and are not - # directly routable from macOS, so we exec psql inside the container. - incus exec "$CONTAINER" -- env PGPASSWORD="$PG_PASSWORD" psql -U "$PG_USER" "$PG_DB" "$@" -} +cmd_logs() { incus exec "$(_c_active)" -- tail -f /var/log/postgresql/postgresql-17-main.log; } +cmd_staging_logs() { incus exec "$(_c_staging)" -- tail -f /var/log/postgresql/postgresql-17-main.log; } -cmd_shell() { - incus exec "$CONTAINER" -- bash +cmd_bouncer_logs() { + incus exec "$BOUNCER" -- \ + tail -f /var/log/pgbouncer/active.log /var/log/pgbouncer/staging.log } -cmd_status() { - incus list "$CONTAINER" -} +# --------------------------------------------------------------------------- +# Export / import (operate on the active backend) +# --------------------------------------------------------------------------- cmd_export() { - local file="$REPO_ROOT/var/pg-dev-$(date +%Y-%m-%dT%H:%M:%S).tar.gz" - echo "==> Freezing $CONTAINER for consistent export..." - incus pause "$CONTAINER" - incus export "$CONTAINER" "$file" - incus resume "$CONTAINER" + mkdir -p "$REPO_ROOT/var" + local container + container=$(_c_active) + local file + file="$REPO_ROOT/var/${container}-$(date +%Y-%m-%dT%H-%M-%S).tar.gz" + echo "==> Pausing $container for consistent export..." + incus pause "$container" + incus export "$container" "$file" + incus resume "$container" echo "Exported to $file" } cmd_import_last() { + local container + container=$(_c_active) local file - file=$(ls -t "$REPO_ROOT/var/pg-dev-"*.tar.gz 2>/dev/null | head -1) + file=$(ls -t "$REPO_ROOT/var/${container}-"*.tar.gz 2>/dev/null | head -1) if [[ -z "$file" ]]; then - echo "No export found in var/ — run 'make pg.export' first" + echo "No export found for $container in var/." exit 1 fi echo "==> Importing from $file..." + incus delete "$container" --force 2>/dev/null || true incus import "$file" - incus start "$CONTAINER" - echo "Imported and started $CONTAINER" + incus start "$container" + echo "Imported and started $container" } +# --------------------------------------------------------------------------- +# Dispatcher +# --------------------------------------------------------------------------- + usage() { - echo "Usage: pg-dev-local [args]" - echo "" - echo "Commands:" - echo " up launch container and install PostgreSQL" - echo " down delete container" - echo " snapshot [--force] create a named snapshot (--force overwrites existing)" - echo " restore restore container to a snapshot" - echo " restore-last restore to the most recent snapshot" - echo " snapshots list snapshots" - echo " ip print container IP" - echo " psql [args] open psql session inside container" - echo " shell open a bash shell in the container" - echo " status show container state" - echo " export export container to var/pg-dev-.tar.gz" - echo " import-last import most recent export from var/" + cat < [args] + +Lifecycle: + up provision pg-dev-a, pg-dev-b, pg-bouncer + down delete all three containers + status show pointer + container states + endpoint print bouncer host/port roles + .pgpass line + promote flip active↔staging on both bouncer instances + +Active slot (whatever is currently active): + psql [args] psql via incus exec into active backend + shell bash into active backend + ip active backend's container IP + logs tail postgres logs on active backend + snapshot [--force] create a snapshot on active backend + restore restore active backend to a snapshot + restore-last restore active backend to its most recent snapshot + snapshots list snapshots on active backend + +Staging slot (the other one): + staging.psql [args] + staging.shell + staging.ip + staging.logs + staging.snapshot [--force] + staging.restore + staging.restore-last + staging.snapshots + staging.reset shortcut: restore staging to 'initial' + +Bouncer: + bouncer.logs tail both pgbouncer logs + +Export / import (act on the active backend): + export export active backend to var/-.tar.gz + import-last re-import most recent export of active backend +EOF } case "${1:-}" in - up) shift; cmd_up "$@" ;; - down) shift; cmd_down "$@" ;; - snapshot) shift; cmd_snapshot "$@" ;; - restore) shift; cmd_restore "$@" ;; - restore-last) shift; cmd_restore_last "$@" ;; - snapshots) shift; cmd_snapshots "$@" ;; - ip) shift; cmd_ip "$@" ;; - psql) shift; cmd_psql "$@" ;; - shell) shift; cmd_shell "$@" ;; - status) shift; cmd_status "$@" ;; - export) shift; cmd_export "$@" ;; - import-last) shift; cmd_import_last "$@" ;; - *) usage; exit 1 ;; + up) shift; cmd_up "$@" ;; + down) shift; cmd_down "$@" ;; + status) shift; cmd_status "$@" ;; + endpoint) shift; cmd_endpoint "$@" ;; + promote) shift; cmd_promote "$@" ;; + + psql) shift; cmd_psql "$@" ;; + shell) shift; cmd_shell "$@" ;; + ip) shift; cmd_ip "$@" ;; + logs) shift; cmd_logs "$@" ;; + snapshot) shift; cmd_snapshot "$@" ;; + restore) shift; cmd_restore "$@" ;; + restore-last) shift; cmd_restore_last "$@" ;; + snapshots) shift; cmd_snapshots "$@" ;; + + staging.psql) shift; cmd_staging_psql "$@" ;; + staging.shell) shift; cmd_staging_shell "$@" ;; + staging.ip) shift; cmd_staging_ip "$@" ;; + staging.logs) shift; cmd_staging_logs "$@" ;; + staging.snapshot) shift; cmd_staging_snapshot "$@" ;; + staging.restore) shift; cmd_staging_restore "$@" ;; + staging.restore-last) shift; cmd_staging_restore_last "$@" ;; + staging.snapshots) shift; cmd_staging_snapshots "$@" ;; + staging.reset) shift; cmd_staging_reset "$@" ;; + + bouncer.logs) shift; cmd_bouncer_logs "$@" ;; + + export) shift; cmd_export "$@" ;; + import-last) shift; cmd_import_last "$@" ;; + + ""|-h|--help) usage ;; + *) usage; exit 1 ;; esac From f891593a2138c0673ac5e8e61568065d27f3ba22 Mon Sep 17 00:00:00 2001 From: andi Date: Mon, 1 Jun 2026 16:42:53 +0200 Subject: [PATCH 3/6] reload --- .gitignore | 1 + Makefile | 4 ++++ scripts/pg-dev-local | 16 +++++++++++++++- 3 files changed, 20 insertions(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 975db50..1c6fdee 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,3 @@ /.zed .env +/var/active-slot diff --git a/Makefile b/Makefile index 29e5459..0e30118 100644 --- a/Makefile +++ b/Makefile @@ -129,6 +129,10 @@ pg.staging.reset: pg.bouncer.logs: $(PG_DEV) bouncer.logs +.PHONY: pg.bouncer.reload +pg.bouncer.reload: + $(PG_DEV) bouncer.reload + # ----- export / import (active backend) ----------------------------------- .PHONY: pg.export diff --git a/scripts/pg-dev-local b/scripts/pg-dev-local index 05cb7a7..a9eb7bb 100755 --- a/scripts/pg-dev-local +++ b/scripts/pg-dev-local @@ -111,7 +111,7 @@ admin_users = pgb_admin stats_users = pgb_admin pool_mode = session server_reset_query = DISCARD ALL -ignore_startup_parameters = extra_float_digits +ignore_startup_parameters = extra_float_digits,statement_timeout,lock_timeout,idle_in_transaction_session_timeout,search_path,default_transaction_isolation,default_transaction_read_only,default_transaction_deferrable logfile = /var/log/pgbouncer/${role}.log pidfile = /var/run/pgbouncer/${role}.pid EOF @@ -502,6 +502,18 @@ cmd_bouncer_logs() { tail -f /var/log/pgbouncer/active.log /var/log/pgbouncer/staging.log } +cmd_bouncer_reload() { + # Re-render both inis from the current pointer state, then RELOAD both + # pgbouncer instances. Useful after a config change to _render_ini. + local a s + a=$(_active); s=$(_staging) + _render_ini active 5432 "${BACKEND_PREFIX}-${a}" + _render_ini staging 5433 "${BACKEND_PREFIX}-${s}" + _bouncer_admin active "RELOAD;" >/dev/null + _bouncer_admin staging "RELOAD;" >/dev/null + echo "Reloaded pgbouncer@active and pgbouncer@staging." +} + # --------------------------------------------------------------------------- # Export / import (operate on the active backend) # --------------------------------------------------------------------------- @@ -573,6 +585,7 @@ Staging slot (the other one): Bouncer: bouncer.logs tail both pgbouncer logs + bouncer.reload re-render both inis from state and RELOAD Export / import (act on the active backend): export export active backend to var/-.tar.gz @@ -607,6 +620,7 @@ case "${1:-}" in staging.reset) shift; cmd_staging_reset "$@" ;; bouncer.logs) shift; cmd_bouncer_logs "$@" ;; + bouncer.reload) shift; cmd_bouncer_reload "$@" ;; export) shift; cmd_export "$@" ;; import-last) shift; cmd_import_last "$@" ;; From 8faaba94a132cf3f926d721018df205bd5ffbb67 Mon Sep 17 00:00:00 2001 From: andi Date: Mon, 1 Jun 2026 16:42:58 +0200 Subject: [PATCH 4/6] readme --- README.md | 142 +++++++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 118 insertions(+), 24 deletions(-) diff --git a/README.md b/README.md index b0056b2..d8288bd 100644 --- a/README.md +++ b/README.md @@ -1,53 +1,147 @@ # Summary -Working with Postgres database dumps can be painful, if you want to test database data- or schema migrations. Since `pg_restore` can be slow if there are complex indexes like `GIN` indexes. +Working with PostgreSQL dumps is painful when `pg_restore` takes ~90 minutes +and your dev database is unreachable for the whole window. This repo wraps +a colima VM running incus + ZFS to give you: -On Linux there is tooling to solve this problem: lightweight system containers via `incus` (backed by LXC, using kernel namespaces and cgroups) and file system snapshots via ZFS. Both are available on macOS through `colima`, which runs a Linux VM transparently in the background. +- **two long-lived Postgres backends** (`pg-dev-a`, `pg-dev-b`) with + independent ZFS snapshot timelines, so already-loaded states stay cheap to + checkpoint and restore; +- **one pgbouncer container** with two listeners on a stable IP, so + `pg_restore` can load a new dump on the staging backend through one port + while your app keeps querying the active backend on the other; +- **a single command** (`make pg.promote`) to atomically swap which backend + is "active" — clients keep their TCP connection, the dataset underneath + changes. -This repository provides some scripting to serve a daily loop of creating and restoring snapshots of a Postgres instance. +Design rationale lives in [SPEC.md](SPEC.md). The scenarios below cover the +day-to-day surface. -# Usage +# Scenarios -## Once +## Once, after cloning ```shell -make deps +make deps # incus, colima, jq on macOS +cp .env.example .env # edit PG_* if you don't like the defaults +make start # boot colima with the incus runtime +make pg.up # provision pg-dev-a, pg-dev-b, pg-bouncer (~15 min) +make pg.endpoint # prints connection info + a ready-to-paste .pgpass line ``` -## Daily +`make pg.endpoint` prints something like (with `` being .10 in +whatever subnet `incus network get incusbr0 ipv4.address` reports — e.g. +`192.168.100.10` if your bridge is on `192.168.100.0/24`): + +``` +active host= port=5432 dbname= (current data) +staging host= port=5433 dbname= (import target / opposite of active) + +.pgpass line (one line, covers both ports): + :*:*:: +``` + +Paste that single line into `~/.pgpass` and you're done with auth forever. +The bouncer IP is pinned at the device level — it survives reboots, +promotes, snapshot restores, everything short of `make pg.down`. Override +the auto-pick by setting `PG_BOUNCER_IP` in `.env`. + +## Day to day: querying the database + +Port **5432** always means *current data*. Pick whatever client you like: ```shell -# Start colima -make pg.up +psql -h -p 5432 -d $PG_DB # any psql, any app +make pg.psql # quick shell via incus exec +make pg.logs # tail postgres logs +``` + +Port **5433** is the staging backend — the opposite slot. Useful for +ad-hoc exploration of "the other dataset" or for verifying a dump +mid-import. + +## Importing a fresh dump (the headline workflow) + +This is what this repo exists for. The import runs on staging through the +bouncer; the active backend keeps serving live queries the entire time. + +```shell +# 1. Wipe staging back to its clean `initial` snapshot. +make pg.staging.reset + +# 2. Restore the dump through the bouncer's staging port (:5433). +pg_restore --host= --port=5433 --dbname=$PG_DB \ + --jobs=4 your-dump.pgdump # ~90 min, no blocking + +# 3. Sanity-check the loaded data while still on staging. +psql -h -p 5433 -d $PG_DB -c '\dt' -# Create a snapshot -make pg.snapshot name=$(date +%Y-%m-%dT%H-%M-%S)_dump_import +# 4. Checkpoint the loaded state on the staging slot. +make pg.staging.snapshot name=initial-loaded -# List snapshots -make pg.snapshots +# 5. Promote. Sub-second; clients keep their TCP connections through the +# bouncer; the dataset underneath flips. +make pg.promote +``` + +After `pg.promote`, apps on :5432 see the freshly imported data. The +previously active backend is now reachable on :5433 with its full snapshot +timeline intact, ready to be rolled back to or wiped for the next import. -# Restore a named snapshot, drops potential following after confirmation -make pg.restore name=initial +## Rolling back a bad import -# Restore the most recent snapshot without confirmation -make pg.restore-last +You promoted, ran the app, and the new data is broken. The previous backend +is untouched on :5433. One command undoes the promote: -# Tail postgres logs -make pg.logs +```shell +make pg.promote # flips back — :5432 now points at the old data again ``` +No data is regenerated. The pointer just inverts. + +## Snapshotting / restoring during normal dev + +Each backend has its own snapshot timeline. The unprefixed commands act on +whichever slot is currently active: + +```shell +make pg.snapshot name=$(date +%Y-%m-%dT%H-%M-%S)_before-migration +# ... run a destructive migration ... +make pg.restore name=$(date +%Y-%m-%dT%H-%M-%S)_before-migration +make pg.restore-last # most recent snapshot, no confirmation +make pg.snapshots # list +``` -## Special +The `pg.staging.*` family mirrors these for the staging slot, if you want +to stage multiple checkpoints before a promote. -Snapshots are bound to one colima instance (`make pg.up`). Destroying the instance will kill all snapshots. You may export and import snapshots, but while faster than `pg_restore` in my case, it still is _not fast_. +## Inspecting state ```shell -make pg.export -time make recreate pg.import-last +make pg.status # active slot + container states +make pg.endpoint # both ports with their roles + .pgpass line +make pg.bouncer.logs # tail both pgbouncer instances ``` +## Tearing down + +```shell +make pg.down # delete pg-dev-a, pg-dev-b, pg-bouncer (irreversible) +make stop # stop colima +make delete # nuke colima entirely; rebuild fresh with `make recreate` +``` + +Snapshots live inside the colima VM. `make delete` loses them all. +`make pg.export` / `make pg.import-last` serialise the *active* backend +(data + snapshots) to a tarball under `var/`, which survives a colima +rebuild — slower than ZFS snapshots, but the only way out of the VM. + # Questions ## Why a `Makefile` if you have a script -Because I like to have the shell autocompletion of `make`. +Because I like the shell autocompletion of `make`. + +## Where's the design rationale + +[SPEC.md](SPEC.md). From 83f5494ab41b1299de6f3f665474d8f25899bbb8 Mon Sep 17 00:00:00 2001 From: andi Date: Mon, 1 Jun 2026 16:58:01 +0200 Subject: [PATCH 5/6] Cleanup --- Makefile | 8 -------- scripts/pg-dev-local | 33 +++++++++++++++++++++++---------- 2 files changed, 23 insertions(+), 18 deletions(-) diff --git a/Makefile b/Makefile index 0e30118..5fe2b75 100644 --- a/Makefile +++ b/Makefile @@ -95,10 +95,6 @@ pg.staging.psql: pg.staging.shell: $(PG_DEV) staging.shell -.PHONY: pg.staging.ip -pg.staging.ip: - @$(PG_DEV) staging.ip - .PHONY: pg.staging.logs pg.staging.logs: $(PG_DEV) staging.logs @@ -115,10 +111,6 @@ pg.staging.restore: pg.staging.restore-last: $(PG_DEV) staging.restore-last -.PHONY: pg.staging.snapshots -pg.staging.snapshots: - $(PG_DEV) staging.snapshots - .PHONY: pg.staging.reset pg.staging.reset: $(PG_DEV) staging.reset diff --git a/scripts/pg-dev-local b/scripts/pg-dev-local index a9eb7bb..da20138 100755 --- a/scripts/pg-dev-local +++ b/scripts/pg-dev-local @@ -459,7 +459,17 @@ cmd_restore() { _restore "$(_c_active)" "$name" } cmd_restore_last() { _restore_last "$(_c_active)"; } -cmd_snapshots() { incus snapshot list "$(_c_active)"; } + +# Unified listing for both slots. +cmd_snapshots() { + local a s + a=$(_active); s=$(_staging) + echo "─── active (${BACKEND_PREFIX}-${a}) ───" + incus snapshot list "${BACKEND_PREFIX}-${a}" + echo + echo "─── staging (${BACKEND_PREFIX}-${s}) ───" + incus snapshot list "${BACKEND_PREFIX}-${s}" +} # staging slot cmd_staging_snapshot() { @@ -471,7 +481,6 @@ cmd_staging_restore() { _restore "$(_c_staging)" "$name" } cmd_staging_restore_last() { _restore_last "$(_c_staging)"; } -cmd_staging_snapshots() { incus snapshot list "$(_c_staging)"; } cmd_staging_reset() { _restore "$(_c_staging)" initial; } # Shell / psql / ip / logs, per slot. @@ -491,8 +500,14 @@ _ip() { | grep -oE '[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+' \ | head -1 } -cmd_ip() { _ip "$(_c_active)"; } -cmd_staging_ip() { _ip "$(_c_staging)"; } + +# Unified IPs for both slots. +cmd_ip() { + local a s + a=$(_active); s=$(_staging) + printf '%-8s %-10s %s\n' active "${BACKEND_PREFIX}-${a}" "$(_ip "${BACKEND_PREFIX}-${a}")" + printf '%-8s %-10s %s\n' staging "${BACKEND_PREFIX}-${s}" "$(_ip "${BACKEND_PREFIX}-${s}")" +} cmd_logs() { incus exec "$(_c_active)" -- tail -f /var/log/postgresql/postgresql-17-main.log; } cmd_staging_logs() { incus exec "$(_c_staging)" -- tail -f /var/log/postgresql/postgresql-17-main.log; } @@ -562,25 +577,25 @@ Lifecycle: endpoint print bouncer host/port roles + .pgpass line promote flip active↔staging on both bouncer instances +Both slots (rendered side by side): + ip container IPs for active and staging + snapshots snapshot lists for active and staging + Active slot (whatever is currently active): psql [args] psql via incus exec into active backend shell bash into active backend - ip active backend's container IP logs tail postgres logs on active backend snapshot [--force] create a snapshot on active backend restore restore active backend to a snapshot restore-last restore active backend to its most recent snapshot - snapshots list snapshots on active backend Staging slot (the other one): staging.psql [args] staging.shell - staging.ip staging.logs staging.snapshot [--force] staging.restore staging.restore-last - staging.snapshots staging.reset shortcut: restore staging to 'initial' Bouncer: @@ -611,12 +626,10 @@ case "${1:-}" in staging.psql) shift; cmd_staging_psql "$@" ;; staging.shell) shift; cmd_staging_shell "$@" ;; - staging.ip) shift; cmd_staging_ip "$@" ;; staging.logs) shift; cmd_staging_logs "$@" ;; staging.snapshot) shift; cmd_staging_snapshot "$@" ;; staging.restore) shift; cmd_staging_restore "$@" ;; staging.restore-last) shift; cmd_staging_restore_last "$@" ;; - staging.snapshots) shift; cmd_staging_snapshots "$@" ;; staging.reset) shift; cmd_staging_reset "$@" ;; bouncer.logs) shift; cmd_bouncer_logs "$@" ;; From 3899c3def72bd386c0116f161eca87ee6c3a9103 Mon Sep 17 00:00:00 2001 From: andi Date: Mon, 1 Jun 2026 17:24:41 +0200 Subject: [PATCH 6/6] Render `host:port` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ``` q:qb pansen/colima-incus feat/andi/a_b_import 244ms$ make pg.status pg.ip pg.snapshots scripts/pg-dev-local status active slot: b (active = pg-dev-b, staging = pg-dev-a) +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | pg-bouncer | RUNNING | 192.168.100.2 (eth0) | fd42:5a94:fea:bc22:1266:6aff:fe3f:2049 (eth0) | CONTAINER | 0 | +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | pg-dev-a | RUNNING | 192.168.100.53 (eth0) | fd42:5a94:fea:bc22:1266:6aff:fede:a68c (eth0) | CONTAINER | 2 | +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | pg-dev-b | RUNNING | 192.168.100.27 (eth0) | fd42:5a94:fea:bc22:1266:6aff:fe5f:4b9b (eth0) | CONTAINER | 1 | +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ active 192.168.100.2:5432 → pg-dev-b (192.168.100.27) staging 192.168.100.2:5433 → pg-dev-a (192.168.100.53) scripts/pg-dev-local snapshots ─── active (pg-dev-b) ─── +---------+-----------------------+------------+----------+ | NAME | TAKEN AT | EXPIRES AT | STATEFUL | +---------+-----------------------+------------+----------+ | initial | 2026/06/01 16:02 CEST | | NO | +---------+-----------------------+------------+----------+ ─── staging (pg-dev-a) ─── +---------------------------------+-----------------------+------------+----------+ | NAME | TAKEN AT | EXPIRES AT | STATEFUL | +---------------------------------+-----------------------+------------+----------+ | initial | 2026/06/01 16:01 CEST | | NO | +---------------------------------+-----------------------+------------+----------+ | 2026-06-01T17-20-21_dump_import | 2026/06/01 17:20 CEST | | NO | +---------------------------------+-----------------------+------------+----------+ q:qb pansen/colima-incus feat/andi/a_b_import 774ms$ make pg.promote scripts/pg-dev-local promote ==> Promoting: active b → a ==> Promoted. Active: pg-dev-a Staging: pg-dev-b active slot: a (active = pg-dev-a, staging = pg-dev-b) +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | pg-bouncer | RUNNING | 192.168.100.2 (eth0) | fd42:5a94:fea:bc22:1266:6aff:fe3f:2049 (eth0) | CONTAINER | 0 | +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | pg-dev-a | RUNNING | 192.168.100.53 (eth0) | fd42:5a94:fea:bc22:1266:6aff:fede:a68c (eth0) | CONTAINER | 2 | +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | pg-dev-b | RUNNING | 192.168.100.27 (eth0) | fd42:5a94:fea:bc22:1266:6aff:fe5f:4b9b (eth0) | CONTAINER | 1 | +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ q:qb pansen/colima-incus feat/andi/a_b_import 612ms$ make pg.status pg.ip pg.snapshots scripts/pg-dev-local status active slot: a (active = pg-dev-a, staging = pg-dev-b) +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | pg-bouncer | RUNNING | 192.168.100.2 (eth0) | fd42:5a94:fea:bc22:1266:6aff:fe3f:2049 (eth0) | CONTAINER | 0 | +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | pg-dev-a | RUNNING | 192.168.100.53 (eth0) | fd42:5a94:fea:bc22:1266:6aff:fede:a68c (eth0) | CONTAINER | 2 | +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ | pg-dev-b | RUNNING | 192.168.100.27 (eth0) | fd42:5a94:fea:bc22:1266:6aff:fe5f:4b9b (eth0) | CONTAINER | 1 | +------------+---------+-----------------------+-----------------------------------------------+-----------+-----------+ active 192.168.100.2:5432 → pg-dev-a (192.168.100.53) staging 192.168.100.2:5433 → pg-dev-b (192.168.100.27) scripts/pg-dev-local snapshots ─── active (pg-dev-a) ─── +---------------------------------+-----------------------+------------+----------+ | NAME | TAKEN AT | EXPIRES AT | STATEFUL | +---------------------------------+-----------------------+------------+----------+ | initial | 2026/06/01 16:01 CEST | | NO | +---------------------------------+-----------------------+------------+----------+ | 2026-06-01T17-20-21_dump_import | 2026/06/01 17:20 CEST | | NO | +---------------------------------+-----------------------+------------+----------+ ─── staging (pg-dev-b) ─── +---------+-----------------------+------------+----------+ | NAME | TAKEN AT | EXPIRES AT | STATEFUL | +---------+-----------------------+------------+----------+ | initial | 2026/06/01 16:02 CEST | | NO | +---------+-----------------------+------------+----------+ ``` --- scripts/pg-dev-local | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/scripts/pg-dev-local b/scripts/pg-dev-local index da20138..35039c1 100755 --- a/scripts/pg-dev-local +++ b/scripts/pg-dev-local @@ -501,12 +501,19 @@ _ip() { | head -1 } -# Unified IPs for both slots. +# Unified port + backend view for both slots. The bouncer endpoint +# (host:port) is the client-facing fact; the backend container name and +# its container IP are shown for ops/debugging context. cmd_ip() { - local a s + local a s ip_a ip_s bouncer_ip a=$(_active); s=$(_staging) - printf '%-8s %-10s %s\n' active "${BACKEND_PREFIX}-${a}" "$(_ip "${BACKEND_PREFIX}-${a}")" - printf '%-8s %-10s %s\n' staging "${BACKEND_PREFIX}-${s}" "$(_ip "${BACKEND_PREFIX}-${s}")" + ip_a=$(_ip "${BACKEND_PREFIX}-${a}") + ip_s=$(_ip "${BACKEND_PREFIX}-${s}") + bouncer_ip=$(_bouncer_ip) + printf '%-8s %-22s → %-10s (%s)\n' \ + active "${bouncer_ip}:5432" "${BACKEND_PREFIX}-${a}" "$ip_a" + printf '%-8s %-22s → %-10s (%s)\n' \ + staging "${bouncer_ip}:5433" "${BACKEND_PREFIX}-${s}" "$ip_s" } cmd_logs() { incus exec "$(_c_active)" -- tail -f /var/log/postgresql/postgresql-17-main.log; } @@ -578,7 +585,7 @@ Lifecycle: promote flip active↔staging on both bouncer instances Both slots (rendered side by side): - ip container IPs for active and staging + ip bouncer endpoint → backend, for both slots snapshots snapshot lists for active and staging Active slot (whatever is currently active):