Archiving & Disaster Recovery: schema through base-backup generation (M1-M5) - #1186
Draft
dimitri wants to merge 53 commits into
Draft
Archiving & Disaster Recovery: schema through base-backup generation (M1-M5)#1186dimitri wants to merge 53 commits into
dimitri wants to merge 53 commits into
Conversation
…or API) Adds the SQL-only foundation for the Archiver process identity / ARCHIVING node membership / base-backup policy / PITR schema described in the archiving-disaster-recovery design doc, milestone 1 (schema + monitor API only -- no service_archiver process involved yet, everything is exercised via direct SQL calls against a plain cluster). - pgautofailover.replication_state gains a new 'archiving' terminal state. - pgautofailover.node gains haspgdata bool, distinguishing ordinary Postgres instances from lightweight ARCHIVING membership rows (a pg_receivewal client, no PGDATA). The old unconditional UNIQUE (nodehost, nodeport) constraint is replaced with a partial unique index scoped to haspgdata rows, since one archiver's (hostname, 0) pair is deliberately shared across every group it serves. - New types, tables and ~26 plpgsql/SQL functions covering: archiver registration and storage targets (local + rclone), formation/group archiver policy (quorum, base-backup policy, replication-quorum eligibility), WAL capture confirmation (wal_archived()/ report_wal_received()), base-backup lifecycle and pruning, warm-standby archiver_node rows with a maxresidentreplay cap, and PITR node lifecycle + command queue. - pgautofailover--2.2--2.3.sql mirrors the same DDL incrementally, since 2.3 hasn't shipped yet; verified end-to-end against a real 1.0 -> ... -> 2.2 -> 2.3 upgrade (including the pre-existing node_nodehost_nodeport_key1 constraint name quirk from two earlier migrations each recreating the table). - New archiving_schema regress test exercising the full schema end-to-end via direct SQL, added at the end of regress_schedule (after cluster_init_failover_rule_attribution, before the dummy_update/ drop_extension/upgrade trio that must stay last) since its expected output pins literal id values tied to its exact position in the shared contrib_regression database, same as every other test in this schedule. Full local regress (20/20) + isolation (6/6) schedules pass, plus a verified real extension upgrade from 2.2 to 2.3.
Adds monitor-side (SQL FSM + C) support for the ARCHIVING replication state, so an ARCHIVING node row (haspgdata = false, created by M1's archiver_add_formation()) is driven through the same node_active() protocol as an ordinary node instead of being stuck at wait_standby forever. No keeper-side/service_archiver work yet -- this is groundwork, verified via node_active() calls made directly against the monitor. - ReplicationState gains REPLICATION_STATE_ARCHIVING (C) / 'archiving' already existed on the SQL enum from M1. - AutoFailoverNode gains hasPgData, populated via TupleToAutoFailoverNode. Looked up by name (SPI_fnumber), not the file's usual hardcoded Anum_ constant: this function is also called against a "RETURNING node.*" tuple descriptor whose physical column order diverges from the explicit SELECT list's logical order once pg_versionnum/pg_version/ pg_versionstring/citus_version are in the mix, so a hardcoded ordinal would silently read the wrong (and wrongly-typed) column for that caller. - MonitorFSM[]: pos 307/309/315/317/319 (report_lsn/wait_standby, primary converged -> secondary/catchingup) gain an explicit hasPgData = TRUE restriction, paired with 5 new hasPgData = FALSE mirror rows (pos 394-398) assigning ARCHIVING instead -- appended after the existing MS-failover cluster since the ordinary rows are numbered with no room between them for 5 more, and the hasPgData split makes their relative order irrelevant to first-match-wins. Pos 367's MS-failover fan-out row (and BuildCandidateList's own C-side secondaryStates list) now also admits ARCHIVING, pulling it into report_lsn during elections exactly like SECONDARY/CATCHINGUP. - system_identifier_is_null_at_init_only loosened to also allow a NULL sysidentifier while reportedstate is 'archiving' or 'report_lsn': an ARCHIVING row never gets a real one. The 2.2--2.3 migration mirror casts the column to text instead of the literals to the enum, since this script's own earlier ADD VALUE 'archiving' and this constraint run in the same ALTER EXTENSION UPDATE transaction and Postgres refuses to create new instances of a not-yet-committed enum value. - keeper_fsm_edges.sql's own "expect zero rows" comment updated: 8 rows are now expected there, a real and currently correct gap -- the monitor side landed first, with no service_archiver/KeeperFSM[] support yet to report ARCHIVING or drive pg_receivewal (next milestone). Verified against a hand-run node_active() scenario (register primary + secondary, converge to primary/secondary, attach an archiver, confirm wait_standby -> archiving instead of catchingup, steady-state archiving stays archiving, replication_quorum = true fans out apply_settings to the primary exactly like an ordinary quorum standby, and rule_pos attribution points at the new rows) in addition to the full regress (20/20) + isolation (6/6) suites and a real 2.2 -> 2.3 extension upgrade.
Adds the keeper-side counterpart to the monitor-side ARCHIVING FSM support (previous commit): KeeperFSM[] rows for WAIT_STANDBY->ARCHIVING/ARCHIVING->REPORT_LSN/REPORT_LSN->ARCHIVING, each dispatching to a new, archiver-specific transition function (fsm_init_archiver/fsm_archiver_report_lsn/fsm_archiver_follow_new_primary, fsm_transition.c) -- mirroring the existing NODE_KIND_CITUS_* pattern of adding separate functions per node kind rather than branching inside the shared ones (fsm_init_standby, keeper_update_pg_state, keeper_ensure_current_state, keeper_node_active_loop stay untouched). New service_archiver.c launches and tracks the one pg_receivewal child an ARCHIVING node keeps running against its group's current primary -- milestone 2's own "colocated fast path" scope (see the Build order in ~/dev/temp/archiving-disaster-recovery.md): pg_receivewal is a real, unmodified Postgres client talking straight to the real primary's own walsender, so no new wire protocol is needed here at all. Not yet wired into supervisor.c's Service/RestartPolicy machinery or a replication slot -- both noted as follow-ups in that file's own header comment. Also: ARCHIVING_STATE added to NodeState (state.h/state.c) and to nodestate_utils.c's nodestateConnectionType() switch (grouped with the other "Postgres known to be stopped" states, since an ARCHIVING row never has a postmaster of its own -- this switch has no default case by design, so a missing case here would have failed the build). Verified: full regress (20/20) + isolation (6/6) suites still pass, and the monitor/keeper reachability cross-check in keeper_fsm_edges.sql -- which the previous commit deliberately left showing 8 unresolved rows, documented as this milestone's own known gap -- now shows zero rows again, both directions, confirming the two sides agree. Live-checked via `pg_autoctl inspect fsm list --json`, which is also how keeper_fsm_edges.json was regenerated (pretty-printed to match the existing file's own review-friendly formatting, not the CLI's compact default). citus_indent and ci/banned.h.sh both pass (the latter caught a raw strerror()/fprintf(stderr) call in service_archiver.c's own exec- failure path, fixed to the project's own log_fatal(..., "%m") convention already used at the other execv() call sites in this codebase).
… (M2 continued)
Adds NODE_KIND_ARCHIVER as a real PgInstanceKind (pgsetup.h/pgsetup.c,
name<->enum both directions), two monitor RPC wrapper functions
(monitor_register_archiver/monitor_archiver_add_formation, monitor.c,
calling M1's own register_archiver()/archiver_add_formation() plpgsql
functions -- not the ordinary C register_node() RPC, since an Archiver is
a process identity, not a (formation, group) membership by itself), and
`pg_autoctl create archiver` (cli_create_node.c): a deliberately minimal,
hand-rolled getopts (not the shared cli_create_node_getopts every ordinary
node kind uses, since that parser's defaults assume a real PostgresSetup
an archiver never has) that registers with the monitor, writes a
KeeperConfig + initial state file (WAIT_STANDBY_STATE, mirroring
archiver_add_formation()'s own starting point), and with --run hands off
to service_archiver_loop() (previous commit).
Verified live against the real monitor RPC layer (not just static
review): registration, formation attachment, and config/state file
writing all confirmed end-to-end against a real running monitor extension
instance, including two real bugs the empirical run caught that manual
review missed --
- config_find_pg_ctl() unconditionally clears pgSetup.pg_ctl before
searching, silently discarding a caller-supplied --pgctl value; fixed
by only calling it when pg_ctl is still empty (also added the missing
--pgctl flag itself -- this dev machine has two pg_ctl on PATH and
needs it to disambiguate, a real scenario, not a test-only one).
- keeper_config_write_file() requires pg_autoctl.role set (validated
against KEEPER_ROLE, not defaulted on write) -- config.role was never
populated, since this path deliberately skips keeper_config_init()'s
ordinary defaults (Postgres-specific probing that doesn't apply here).
`--run`'s actual pg_receivewal launch is still unverified against a real
streaming primary -- needs a real replication-configured Postgres pair,
which is exactly what the next step (pgaftest wiring) provides.
citus_indent and ci/banned.h.sh both pass.
autoctl_node was only ever granted EXECUTE on the function, never SELECT on pgautofailover.basebackup itself, matching every other autoctl_node- callable helper that reads a table it has no direct grant on (e.g. archiver_add_formation) -- get_latest_basebackup was the odd one out. Found via a real end-to-end test of `pg_autoctl archiver serve` against a live monitor: calling it as autoctl_node failed with "permission denied for table basebackup".
The archiver's serving half: a standalone binary (no pg_autoctl/*.c
dependency, only src/bin/common/ and src/bin/lib/log/) that speaks enough
of the real Postgres replication protocol to serve IDENTIFY_SYSTEM, SHOW,
BASE_BACKUP, and a non-standard FETCH_FILE side-channel, backed by an
archiver's local WAL cache and base backups instead of a live postmaster.
No frontend-linkable server-side protocol library exists anywhere in
Postgres (confirmed against pqcomm.c/backend_startup.c/repl_gram.y/
walsender.c, all backend-only) -- this is a genuine reimplementation
guided by that source, not a linking exercise. Two pieces are vendored
near-verbatim since they're already frontend-safe: vendor/tar.c + pgtar.h
(PostgreSQL's own ustar header/checksum logic, src/port/tar.c).
Also ships fetch_client.c / `pg_walsender fetch-file`, the client side of
the FETCH_FILE side-channel, for use as pg_autoctl's own restore_command.
Verified against real, unmodified PostgreSQL client tools:
- psql (replication=1): IDENTIFY_SYSTEM, SHOW wal_segment_size
- pg_basebackup --format=plain -X none --no-manifest: fetched a real
base backup byte-identical to the source, then booted a live Postgres
instance from the result
- pg_walsender fetch-file: fetched a full 16MB WAL segment byte-
identical, plus clean error handling (missing file, path traversal,
unknown route)
See ~/dev/temp/archiving-disaster-recovery.md for the design this
implements milestone 2 of.
`pg_autoctl archiver serve` (cli_archiver.c) is the supervisor verb that
execs pg_walsender as a persistent child (service_archiver_serve.c),
mirroring exactly how service_archiver.c already execs real pg_receivewal
for the outbound WAL-capture direction -- same pattern, new direction.
Keeps pg_walsender's routes file ("<formation>/<group>" -> { walcache,
basebackup }) current, refreshed periodically and on SIGHUP.
The routes file is built from *local* config (formation/groupId/pgSetup.
pgdata), not a monitor round-trip: archiver_add_formation()'s own SQL
inserts the new archiver_node row's pgdata as an empty string, since the
monitor has no way to know an archiver's local WAL cache path -- that's
inherently archiver-host-local information. The one genuinely monitor-
tracked piece is the latest base backup's storage location
(monitor_get_latest_basebackup_location, new in monitor.c).
KeeperConfig gains archiverId/archiverIdStr (keeper_config.h/.c) so a
later, separate `archiver serve` invocation can identify itself to the
monitor -- ini_file.c's INI_INT_T is a plain int, too narrow for a
bigserial id, so this follows citusRoleStr/citusRole's existing string-
plus-parsed-value pattern in the same struct.
Verified against a real, freshly-created cluster (create monitor ->
create postgres -> create archiver -> archiver serve): archiverId
persists and round-trips correctly, the routes file is generated
correctly from live monitor state, pg_walsender starts and serves real
clients through it, and SIGTERM shuts the whole thing down cleanly.
…(M2c)
Completes milestone 2's command surface:
- TIMELINE_HISTORY <tli>: serves a "<tli>.history" file straight out of
the WAL cache directory (RowDescription/DataRow, no COPY involved --
traced from walsender.c's own SendTimeLineHistory()).
- CREATE_REPLICATION_SLOT / READ_REPLICATION_SLOT (physical only): a
slot is a bookkeeping marker file under the WAL cache directory, not
a real Postgres slot -- there's no live server to hold one. Not yet
wired into WAL-retention enforcement (prune_archiver_wal()'s job).
- START_REPLICATION [SLOT <name>] <lsn> TIMELINE <tli>: streams raw WAL
bytes straight from the WAL cache directory. Deliberately does NOT
vendor xlogreader.c: real walsender's own WalSndSegmentOpen just opens
a path computed from TLI+segno and streams bytes -- no WAL *record*
decoding is needed to serve a byte range, only the offset/segment
bookkeeping this file does directly. Handles the actively-growing
(".partial") segment case by polling, matching pg_receivewal's own
producer on the other end of this same protocol.
- wal_dir_scan.c: shared helper -- finds the newest fully-captured WAL
segment and derives its boundary LSN from the filename (XLogFileName
format, fixed 16MB segments). Used by START_REPLICATION's default
position, CREATE_REPLICATION_SLOT's consistent_point, and improves
IDENTIFY_SYSTEM's xlogpos (previously a "0/0" placeholder).
One correctness fix alongside: IDENTIFY_SYSTEM's dbname column must be
NULL for a plain replication=1/true connection (pg_receivewal's style) --
only replication=database (pg_basebackup's style) gets a real dbname back.
Always returning a value broke real pg_receivewal outright ("replication
connection using slot ... is unexpectedly database specific"), caught by
this milestone's own end-to-end testing, not by any narrower unit check.
Verified against real, unmodified PostgreSQL client tools:
- psql: TIMELINE_HISTORY round-trips real file content
- psql: CREATE_REPLICATION_SLOT / READ_REPLICATION_SLOT round-trip
consistent_point/restart_lsn correctly, including the "slot doesn't
exist" all-NULL-row case
- pg_receivewal -S <slot> --endpos=...: streamed a full 16MB WAL segment
byte-identical to the source via START_REPLICATION
(and `pg_autoctl stop`) now supervises an archiver's two
halves together -- WAL capture (service_archiver.c's service_archiver_loop,
outbound pg_receivewal against the primary) and serving (service_archiver_
serve.c's service_archiver_serve_loop, inbound pg_walsender) -- as two real
supervisor.c Service[] entries under one restart-on-crash process tree, the
same way start_keeper() already supervises postgres + node-active together
for an ordinary node (service_archiver_run.c). Dispatched from cli_service.
c's cli_keeper_run(), which already reaches an archiver's config file via
the existing role=keeper path; branches on nodeKind before the Postgres-
instance-specific local_postgres_init()/start_keeper() calls, which don't
apply to an archiver.
Two real bugs surfaced by actually running the full archiver process tree
end-to-end for the first time this session (service_archiver_loop's own
monitor-reporting loop was never previously exercised against a live
monitor for more than a few ticks):
- keeper->postgres.currentLSN was never initialized for an archiver (it
has no real Postgres instance to query it from, so keeper_update_pg_
state() -- the only place that ever set it -- is never called on this
path). node_active()'s own pg_lsn parameter rejected the resulting
empty string outright. Fixed by seeding it to "0/0" once, matching
keeper_update_pg_state()'s own placeholder before a real reading
exists; an archiver's actual capture progress is tracked separately
via archiver_wal, not through this per-node report.
- An ordinary node's own get_other_nodes()/current_state listings now
legitimately include ARCHIVING rows with nodeport = 0 (a deliberate
sentinel, see archiver_add_formation()'s own SQL comment: no
postmaster to be reachable on) -- but monitor.c's node-parsing helpers
treated a parsed port of exactly 0 as an unconditional error, so any
ordinary primary/secondary in a formation with an archiver attached
would fail its own node-active loop entirely. Relaxed the two multi-
node-listing parsers to only reject a genuine parse failure, not the
value 0 itself; left the single-node lookup (which can never
legitimately return an archiver, candidate_priority = 0 excludes it)
unchanged.
Verified against a real, freshly-created cluster (monitor + primary +
archiver): `pg_autoctl run --pgdata archiver1` starts both services
cleanly, the FSM transitions wait_standby -> archiving and real
pg_receivewal starts against the primary, pg_walsender serves real
clients through it, and `pg_autoctl stop` cascades a graceful shutdown
through both services and their own child processes (pg_walsender,
pg_receivewal) with no orphans left behind. Also re-verified the primary
node's own node-active loop, previously broken by the port=0 regression,
now runs cleanly with an archiver attached to its formation.
service_archiver.c gains service_archiver_report_captured_wal(), called once per node_active tick from service_archiver_loop(): it scans the archiver's local WAL cache directory for segments pg_receivewal has completed (i.e. no longer ".partial") since the last one reported, and calls the new monitor_report_wal_received() (monitor.c/.h) -- a thin wrapper around the already-existing pgautofailover.report_wal_received() SQL function -- for each one. This is what actually populates archiver_wal and makes wal_archived() return true; until now nothing in the codebase ever called that SQL function. Also fixes a liveness gap this uncovered: pg_receivewal was only ever (re)started from the FSM transition functions that move a node *into* ARCHIVING_STATE (fsm_init_archiver, fsm_archiver_follow_new_primary). An archiver process restarted while already ARCHIVING (or one whose pg_receivewal child died on its own) had nothing to bring it back up, despite this file's own header comment already describing that as the design. service_archiver_loop() now checks service_archiver_pgreceivewal_is_running() every tick and restarts it when needed, exactly matching that comment. Verified end-to-end against a real monitor + primary + archiver: forced WAL switches on the primary, confirmed archiver_wal gets populated with the correct end-of-segment LSNs and wal_archived() correctly reflects archiver_quorum, confirmed the liveness restart itself by killing and restarting the archiver process while already ARCHIVING. Full SQL regression schedule (src/monitor, 20/20) still passes. Dockerfile: copy pg_walsender into the "run" stage image alongside pg_autoctl -- needed for any archiver node in a pgaftest Docker environment, and a prerequisite for M4's own pgaftest specs.
Adds the "archiver" node kind to pgaftest's own DSL, needed to write any
.pgaf spec that includes an ARCHIVING node:
- test_spec_scan.l/.y: new "archiver" keyword (T_ARCHIVER), usable the
same way "coordinator"/"worker" already are: `archiver1 archiver`
inside a formation{} block.
- compose_gen.c: writes kind = archiver into the node's .ini, and links
service_archiver.c into pgaftest's own SHARED_SRCS (Makefile) so the
binary can drive an archiver node the same way it already drives
postgres/coordinator/worker ones.
- nodespec.c (pg_autoctl, not pgaftest): teaches `pg_autoctl node run
<node.ini>` -- the one command every pgaftest container actually
execs -- to recognize kind = archiver and build the right `pg_autoctl
create archiver` argv. An archiver's own getopts is deliberately
minimal (no --pgport/--ssl-*/--auth/...), so it gets its own argv
branch rather than falling through into the generic postgres-flags
path every other kind shares.
Also fixes a real bug in cli_indent.c's print_node() found while
writing the first archiver spec and round-tripping it through `pgaftest
indent`: the node-kind-to-keyword switch only had cases for coordinator
and worker, so indenting a spec containing an archiver node silently
dropped the "archiver" keyword on write-back, turning it into a plain
postgres node. Added the missing NODE_KIND_ARCHIVER case.
test_spec_parse.c/.h and test_spec_scan.c are bison/flex output,
regenerated from the .y/.l changes above.
First pgaftest spec exercising an ARCHIVING node, covering the two things Milestone 4 adds: - test_001/test_002: forcing WAL segment switches on the primary gets each completed segment reported to the monitor (service_archiver_report_captured_wal(), service_archiver.c) and reflected by pgautofailover.wal_archived() -- the archive_command confirmation check nothing populated before this milestone. - test_002 also exercises the liveness fix in service_archiver_loop(): killing and restarting the archiver process while it's already ARCHIVING must bring pg_receivewal back up on its own, not just on the FSM transition that first enters that state. - test_003: fails node1 over to node2 and confirms the archiver passes through REPORT_LSN_STATE and back to ARCHIVING_STATE (following the new primary), and that segments recorded before the failover are still there afterwards. Segment filenames are asserted directly (a fresh cluster deterministically starts WAL at 000000010000000000000001, and each pg_switch_wal() on an idle test database advances exactly one segment) since autoctl_node has no direct SELECT on archiver_wal -- wal_archived() is the only accessor it can call. Registered in tests/tap/schedule and tests/tap/schedules/node.sch, alongside the other node-lifecycle/FSM specs.
First half of Milestone 5 per the design doc's own Build order ("live
first, then replay/volatile"). New file service_archiver_basebackup.c
adds service_archiver_maybe_generate_basebackup(), called once per tick
from service_archiver_loop() alongside the M4 WAL-report/liveness calls.
Trigger scope for this pass: bootstrap only -- a group with zero
existing base backups (monitor_get_latest_basebackup_location() reports
not-found) gets one immediately. Scheduled/timeline-change/retention
triggers need basebackup_policy wired through the CLI first, a later
milestone.
Target selection follows the design doc's `live` precedence, minus its
warm-standby tier (nothing to select from yet, also later): the first
healthy secondary in the group (monitor_get_nodes(), skipping port == 0
ARCHIVING rows), falling back to the primary when none exists.
Generation itself is a one-shot forked child (basebackupPid, tracked the
same way service_archiver.c tracks pgReceivewalPid) rather than a
persistent service, so a potentially long-running pg_basebackup can't
stall the main loop's own node_active()/WAL-report tick. The child execs
the real, unmodified pg_basebackup client with --wal-method=none -- this
backup is deliberately not self-consistent on its own, since the
archiver's already-running WAL capture is what supplies the WAL needed
to reach consistency on replay -- then reads the resulting backup_label
for the authoritative start LSN/timeline and reports both start and
completion to the monitor via two new wrappers,
monitor_report_basebackup_started()/_completed() (monitor.c/.h), calling
the SQL functions M1's schema already shipped but nothing had called
yet. endlsn is best-effort: a live read of the source's current WAL (or
last-replayed, if the source is a standby) position right after the
backup finishes: not Postgres's own internal stop-backup LSN (not
observable from a plain CLI wrapper around pg_basebackup), but a
reasonable upper bound, and never fatal to the backup itself if that one
query fails.
Verified end-to-end against a real monitor + primary + archiver: the
bootstrap backup fires automatically, archiver_wal / basebackup rows
land correctly (source = 'live', status = 'complete', a real endlsn
distinct from startlsn), and the resulting directory passes
pg_verifybackup. Full SQL regression schedule (src/monitor, 20/20)
still passes.
Second half of Milestone 5 ("live first, then replay/volatile" per the
design doc's Build order). service_archiver_maybe_generate_basebackup()
now takes a bootstrap `live` backup as before, then -- on the very next
tick -- exercises the `replay`/`volatile` pipeline exactly once: extract
the last retained backup into a throwaway staging directory, point its
recovery at this archiver's own already-captured WAL (restore_command +
recovery.signal, entirely local, no network round trip), let it replay
forward and promote once it runs out of locally-captured segments, then
pg_basebackup it over loopback and discard the staging instance --
'volatile' means nothing survives between cycles.
Real frequency-driven scheduling (basebackup_policy's own frequency/
onpromotion/retention, resolved through get_archiver_policy()/
get_basebackup_policy()) is a deliberate follow-up, not built here: the
milestone-defining new capability is the replay mechanism itself, not a
general scheduler (matching the design doc's own build order, which
lists warm standby's scheduling machinery as a later milestone).
monitor_report_basebackup_started() (added in the `live`-only commit)
now takes real source/replaymode parameters instead of a hardcoded
'live', and monitor_get_latest_basebackup_location() is renamed to
monitor_get_latest_basebackup_info() and returns the latest backup's
source alongside its storage location -- what the trigger above uses to
tell "only the bootstrap has run" from "the replay exercise is already
done".
Getting a working staging instance up took two real, load-bearing fixes
along the way:
- pg_ctl start, invoked here through both a hand-rolled fork()/execl()
and this project's own run_program() helper, reproducibly misparsed
its own arguments in this exact process tree (deep in a supervised
archiver's own fork chain) even though byte-identical argv worked
fine in every standalone reproduction attempted. Root cause not
fully isolated; worked around by execing the real "postgres" binary
directly instead of going through pg_ctl at all -- the same
fork()/execv() pattern already used for pg_receivewal and
pg_basebackup in this codebase, with readiness confirmed by polling
a real SQL connection rather than relying on pg_ctl's own "-w".
- recovery_target_lsn set to "the end of the latest complete segment"
is not actually a reachable record boundary on a mostly-idle source
(a renamed, "complete" segment file is always its full fixed size
regardless of how much of it is real WAL) -- recovery correctly
refused to pause there ("recovery ended before configured recovery
target was reached"). Replaying to "everything locally available"
and letting Postgres promote on its own sidesteps needing a precise
target at all, which a `volatile`, discard-after-use snapshot never
actually needed in the first place.
Verified end-to-end against a real monitor + primary + archiver, from a
cold start through both the live bootstrap and the replay follow-up:
basebackup rows land correctly (source/replaymode/status all correct,
real distinct startlsn/endlsn across the sequence), and both resulting
directories pass pg_verifybackup. Full SQL regression schedule
(src/monitor) passed 20/20 twice earlier against this same unchanged
schema in this session; a later re-run hit an apparent local
pg_regress/DROP DATABASE environment hang (ProcSignalBarrier) unrelated
to any change in this commit -- no .sql files are touched here.
pgaftest coverage for Milestone 5's own base backup generation (both live and replay/volatile): brings up a monitor + primary + archiver, then waits for both the bootstrap live backup and the one-time replay/volatile follow-up to land, checking the group's final pgautofailover.get_latest_basebackup() row (source = 'replay', replaymode = 'volatile', status = 'complete'). No explicit trigger step is needed here, unlike archiver_wal_capture.pgaf's pg_switch_wal() calls -- both backups fire on their own within a couple of service_archiver_loop() ticks of the archiver starting. That also makes the intermediate 'live'-only state unsafe to assert on directly (this pass's own trigger logic produces at most one live and one replay backup before going quiet for the group, a couple of ticks apart, with nothing in this spec's control over exactly when to look) -- only the final state, once both have landed, is deterministic. Registered in tests/tap/schedule and tests/tap/schedules/node.sch, alongside archiver_wal_capture.pgaf.
New "Archiving & Disaster Recovery Architecture" section in intro.rst, between "Single Standby Architecture" and "Multiple Standby Architecture" -- an archiver is orthogonal to standby count, so it reads best as the thing you add on top of the simplest case before the doc branches into standby-count variations. New docs/tikz/arch-archiver.tex, rendered to .svg the same way every other architecture diagram in this directory is (latexmk -lualatex + pdftocairo, verified locally): primary + secondary + archiver, with the archiver's own WAL cache / base backups called out, a distinct WAL streaming (pg_receivewal) edge separate from real streaming replication, and the monitor's health-check/WAL-report edges to all three. common.tex gains one new color pair (abox/atxt, MS amber) and one new edge style (wal, dashed) for the archiver box and its WAL-streaming edge -- deliberately not reusing the primary/standby colors, since an archiver is a different kind of entity, not a replica. Terminology: uses "archiver" for the physical entity and "archiving node" for its per-group FSM membership, per-project decision -- avoids colliding with pgautofailover.archiver_node, the broader schema table that also covers warm-standby/pitr instances which don't participate in elections at all.
keeper->postgres.currentLSN was set to "0/0" once at service_archiver_ loop() startup and never updated again -- an archiving node's own reportedlsn in pgautofailover.node stayed at that placeholder forever, no matter how much WAL it had actually captured. This mattered more than it looked: pgautofailover.get_most_advanced_ standby() -- the query fast-forward uses to pick a WAL source during a failover election -- has no kind-based exclusion at all, and an archiving node already passes through REPORT_LSN_STATE during an election exactly like any other node (ARCHIVING_STATE -> REPORT_LSN_STATE, fsm.c). A "0/0" reportedlsn was the only thing keeping an archiver from ever being ranked as a candidate WAL source. service_archiver_update_current_lsn() now scans the local WAL cache for the newest complete segment each tick and updates currentLSN to its end LSN before keeper_node_active() reports it -- verified against a real cluster: after two pg_switch_wal() calls, the archiver's own pgautofailover.node.reportedlsn row tracks the primary's position almost exactly (0/A000000 vs. the primary's own 0/A000060).
…m an archiver
Confirms (and builds out) the reframing from the previous commit: an
archiving node already passes through REPORT_LSN_STATE during elections
and get_most_advanced_standby() has no kind-based exclusion, so once its
currentLSN is real, fast-forward's existing streaming-replication code
path can already select and target one -- no new restore_command
plumbing needed. Four real gaps stood between that and actually working,
found and fixed by testing a genuine, unmodified Postgres standby against
a real archiver end to end (not just pg_receivewal, which never exercises
any of these):
- pg_walsender routing is dbname-based (formation/group as dbname), but a
real standby's own walreceiver never forwards the operator's dbname for
a physical replication connection -- it always sends the literal
"replication", confirmed against a real standby. accept_loop.c now
falls back to the single configured route when it sees that sentinel,
matching this milestone's own one-membership-per-archiver scope; a
multi-route archiver (later milestone) needs a different mechanism
(e.g. application_name, which real walreceiver does forward).
- IDENTIFY_SYSTEM's systemid always fell back to the "unknown" placeholder
"0" because nothing ever populated route->systemId: service_archiver_
serve.c's own routes-file writer never wrote a systemid key, even
though routes.c already knew how to parse one. A real standby rejects
a mismatched system identifier outright ("database system identifier
differs between the primary and standby"). Fixed with a new monitor
RPC, monitor_get_group_system_identifier() (pgautofailover.
get_group_system_identifier(), new SQL function in both
pgautofailover.sql and the 2.2--2.3 migration -- an archiving node has
no sysidentifier of its own, but every other node in its group shares
the same one), wired into the routes-file refresh.
- cmd_start_replication.c read raw fread() bytes from a ".partial"
segment without knowing where pg_receivewal's actually-written data
ends -- pg_receivewal pre-allocates the full segment size up front
(matching real Postgres's own WAL file pre-allocation), so reading past
the real tail returns zeros indistinguishable from real content at the
byte level. Sending that tail as WAL data is exactly what a real
standby's recovery logic flags as "invalid record length ... got 0",
and on seeing it, terminates its own walreceiver outright rather than
treating it as "nothing new yet, retry" -- with no automatic
reconnection afterward. Fixed by trimming any trailing zero run before
ever sending a ".partial" chunk (self-correcting: an in-progress
boundary just gets re-read next tick instead of shipped early).
- get_most_advanced_standby() returns an ARCHIVING row's real nodeport,
which is the port == 0 sentinel (no postmaster of its own), not the
archiver's actual pg_walsender serve port -- the monitor has no column
for that (archiver-host-local information, same reasoning service_
archiver_serve.c's own routes file exists for). keeper_get_most_
advanced_standby() now resolves a port == 0 candidate to
PG_AUTOCTL_ARCHIVER_SERVE_PORT, matching this milestone's single-
well-known-port scope.
Verified end-to-end: a real pg_basebackup-seeded standby, given nothing
but an ordinary primary_conninfo pointing at the archiver's serve port,
completed backup recovery, reached consistent recovery state, streamed
live via START_REPLICATION, stayed connected indefinitely (pg_stat_wal_
receiver: status = streaming), and correctly applied newly-written data
(a table created and populated on the real primary afterward) -- with
zero restore_command, zero new replication-source machinery, and zero
changes to fsm_fast_forward's own selection logic beyond the port fix
above.
Bootstraps a brand new node from a registered archiver's base backup
plus captured WAL instead of the group's live primary -- the disaster-
recovery case: rebuild after every live standby (or even the primary)
is gone, with only the archiver left standing. Verified end to end
against a real cluster: `create postgres --from-archiver` completed
pg_basebackup from the archiver, replayed WAL, and settled into a
genuinely healthy "secondary" (pg_stat_wal_receiver: status =
streaming), matching reportedlsn against the real primary once it
re-parented there.
New plumbing:
- KeeperConfig.fromArchiver (keeper_config.h) plus the `--from-archiver`
CLI flag on `create postgres` (cli_create_node.c, cli_common.c) --
runtime-only, same as createAndRun, since reach_initial_state() runs
in the same `pg_autoctl create` invocation that parses it.
- pgautofailover.get_archiver_node() (pgautofailover.sql, the 2.2--2.3
migration) plus its monitor_get_archiver_node()/keeper_get_archiver_
node() C wrappers (monitor.c, keeper.c): finds the ARCHIVING row for
(formation, group) directly. Deliberately not get_most_advanced_
standby() -- that function filters on reportedstate = 'report_lsn', a
transient state an archiving node only visits during a FAST_FORWARD
election, never during its normal steady-state 'archiving' operation,
so it can never find an idle archiver outside of an election.
- fsm_init_standby() (fsm_transition.c) branches on config->fromArchiver
to resolve the archiver via the above instead of keeper_get_primary(),
and passes an empty replication slot name -- pg_walsender has no
slot-based retention in this milestone (cmd_start_replication.c's own
header comment), so standby_init_database's pre-flight replication-
slot check must be skipped rather than asked to verify a slot that
will never exist, matching that function's own existing "initialising
from another standby, no primary yet" precedent.
Four further real, narrow gaps stood between that and actually working,
each found by running the real `pg_basebackup`/`pg_autoctl` code paths
end to end rather than by inspection:
- pg_walsender's BASE_BACKUP had no manifest support (documented scope
cut, cmd_base_backup.c), but PG13+ pg_basebackup requests one by
default -- ReplicationSource.noManifest (pgsql.h) plus pg_basebackup()
passing --no-manifest when set (pgctl.c) works around it for an
archiver-sourced clone specifically, without touching a real primary's
own backup path.
- pgctl_identify_system() (pgctl.c) built its replication connection
string with no dbname at all, relying on real pg_basebackup's and
real walreceiver's own respective "default unset dbname to the literal
'replication'" behaviors -- neither of which this is: it's pg_auto_
failover's own raw libpq connection, which has no such default and
instead falls back to plain libpq's own "dbname = username" rule
(fe-connect.c), a route pg_walsender's routes file was never going to
have an entry for. Passing "replication" explicitly matches what every
other replication client already sends on the wire, and is a no-op
against a real primary (which ignores dbname for replication=true
connections regardless).
- A "replay" base backup (basebackup_replay_mode, milestone 5) promotes
a throwaway extracted copy to make it self-consistent, which genuinely
puts it on a *later* timeline than whatever the archiver's own walcache
has actually captured (which only ever advances on the real primary's
timeline) -- serving that pairing breaks a real pg_basebackup's own
timeline consistency check once it reaches its background WAL-streaming
step ("starting timeline N is not present in the server", comparing
the backup's own timeline against IDENTIFY_SYSTEM's). Fixed at the
source: pgautofailover.get_latest_basebackup() grew an optional
preferred_source filter (both SQL files), and service_archiver_serve.c's
routes refresh now asks for 'live' specifically -- a live-sourced
backup always shares the walcache's timeline by construction. A second,
independent, defense-in-depth check (walcache_current_timeline(),
comparing the walcache's own newest captured segment's embedded
timeline against whatever's about to be advertised) keeps the routes
file from ever serving a mismatched pairing even if that invariant is
ever violated by a future backup mode. monitor_get_latest_basebackup_
info() also grew a timeline out-param, threaded into the routes file's
own (previously unpopulated) "timeline" key -- already parsed by
routes.c, never written by anyone until now.
- cmd_start_replication.c ended a stream with bare CopyDone and nothing
else. A real, long-lived streaming client (real walreceiver, via
primary_conninfo) never triggers the gap because it never decides to
stop on its own -- which is exactly why this went unnoticed through
all of the earlier fast-forward-from-archiver verification. But
pg_basebackup's --wal-method=stream background WAL receiver does
decide to stop, once it reaches its own target LSN, and real receive-
log.c's ReceiveXlogStream only accepts that as a *successful* stop
when it can read a matching CommandComplete afterward (matching real
walsender.c's own WalSndDone, which sends exactly that on controlled
shutdown) -- without it, the client falls through to "unexpected
termination of replication stream" and exits non-zero even though
nothing was actually wrong on the wire. Fixed by sending a CommandComplete
tagged "COPY" right after CopyDone.
…iver Adds archiver_bootstrap_and_fast_forward.pgaf, the disaster-recovery scenario this whole investigation was driven by: a primary, an archiver, and a secondary that's created via `pg_autoctl create postgres --from-archiver` (not from the live primary) after the archiver's first live base backup is ready, then a FAST_FORWARD election where the archiver is the only node with the WAL the winning candidate is missing. node2 is declared `create and launch deferred`: the normal ini-driven node bring-up (`pg_autoctl node start`) has no hook for a custom flag like --from-archiver (NodeSpec/nodespec.c carries no such field -- fromArchiver lives only in KeeperConfig, populated exclusively by cli_create_node.c's own direct CLI parsing), so test_001 `exec`s into node2's own container and runs `pg_autoctl create postgres --from-archiver` by hand, then backgrounds `pg_autoctl run` the same way debug_citus_worker_switchover.pgaf backgrounds a long-lived process (`bash -c "nohup ... &"` -- a foreground `pg_autoctl run` would hang `docker compose exec -T` forever otherwise). test_002-004 engineer a real WAL gap rather than relying on race timing: stop node2 so it can't stream from node1 anymore, generate more WAL on the primary and give the archiver (still capturing independently via pg_receivewal) time to land it, kill the primary, then bring node2 back -- at that point the archiver is strictly ahead of node2 and is the only viable FAST_FORWARD WAL source. The final row-count check on node2 post-promotion confirms real WAL bytes were fetched and applied, not just that the FSM passed through the right state label. Verified: `pgaftest show spec`/`show compose` parse this spec cleanly (exit 0) and `pgaftest indent` round-trips it losslessly, confirming the DSL usage (deferred node declaration, exec/nohup backgrounding, multi-state `passing through` clause) is syntactically valid against the real grammar. Could not run it against a live docker compose cluster in this session: `make -f Makefile.docker build-pg17` fails fetching ghcr.io/hapostgres/pg_auto_failover/pgaf-base (401 Unauthorized, no registry credentials available here), and no local base image is cached to build from instead. Every C-level behavior this spec exercises (--from-archiver's own bootstrap, and fast-forward sourcing WAL from an archiver) was independently verified working end-to-end by hand against a real cluster in the two preceding commits on this branch.
wal_archived() is a plain LANGUAGE sql function (not SECURITY DEFINER), so it runs under the caller's own privileges. autoctl_node never got a direct SELECT grant on pgautofailover.archiver_wal: the blanket `GRANT SELECT ON ALL TABLES IN SCHEMA pgautofailover TO autoctl_node` only covers tables that already existed when that statement ran, and archiver_wal (like every other table in the M1 archiving schema) was created after it. Confirmed live: calling wal_archived() as autoctl_node (the role node_active() actually uses) failed with "permission denied for table archiver_wal". get_latest_basebackup() had this exact same bug, already fixed the same way (SECURITY DEFINER) in a prior commit -- apply the same fix here.
BuildForPrimaryNodeNodeActiveContext() counted every other node in the group toward replicationQuorumCount/secondaryNodesCount/ secondaryQuorumNodesCount, including ARCHIVING rows -- which are never real Postgres secondaries and can never report SECONDARY. In a formation with only a primary and an archiver, that miscount let the archiver's own bootstrap WAIT_STANDBY reading trip anyOtherNodeWaitingStandby (pos 401) and bump the primary off SINGLE, while secondaryQuorumNodesCount could then never legitimately reach zero -- so the primary got stuck between SINGLE and PRIMARY forever. Skip ARCHIVING (hasPgData=false) rows in that loop, matching the hasPgData-based exclusion this file's own REPORTING_NODE section already applies for a different purpose. Also adds the archiver-mirror FSM rows (pos 394/396/399) their own SINGLE|WAIT_PRIMARY|JOIN_PRIMARY match set, since a primary attached only to an archiver legitimately stays SINGLE the whole time instead of ever reaching WAIT_PRIMARY.
…(M5)
Several pieces of the Archiving & Disaster Recovery milestone, landing
together since they build on each other:
WAL-capture reliability
- service_archiver_start_pgreceivewal() now creates a replication slot
for pg_receivewal (pgautofailover_standby_<nodeId>), the same one
keeper_create_and_drop_replication_slots() already creates eagerly
on any primary for every other node regardless of kind. Without a
slot, a pg_receivewal that loses the startup HBA-propagation race
restarts from the server's then-current position, silently and
permanently skipping whatever WAL existed in between.
- pg_walsender's START_REPLICATION now fails loudly ("58P01") instead
of waiting forever when asked for a segment that predates this
archiver's own captured history and will never arrive.
- The archiver's real captured-WAL position is now tracked out of
band (a position file, service_archiver_position_path() and
friends) so it can cross the fork() boundary between the capture
and serve processes -- consumed by cmd_base_backup.c's own
end-of-backup position (previously could re-send a stale start
position and hang a real pg_basebackup's background WAL streamer
forever) and by cmd_identify_system.c indirectly via the routes
file's new "position" key.
- service_archiver_loop() now sets pgIsRunning = true for the
archiver's own keeper state, which the monitor's NodeIsHealthy()
unconditionally requires before ever selecting a node as a
FAST_FORWARD WAL source.
Telemetry
- service_archiver_report_storage() reports disk usage/free space to
the monitor periodically; monitor_get_archivers() surfaces it (and
each archiver's FSM state) to `pg_autoctl watch`'s new archivers
section.
Base-backup production/retention policy
- New SQL: get_basebackup_policy_for_group(), list_basebackups();
get_basebackup_policy() gains SECURITY DEFINER (needed now that
`pg_autoctl show basebackup-policy` calls it directly).
- service_archiver_basebackup.c's scheduling is now policy-driven
instead of the previous hardcoded "bootstrap live, then exactly one
replay, then quiet" scope: frequency/source/replaymode/onpromotion
read from whichever policy resolves for the group, plus
maxcount/maxage retention pruning after each successful backup. The
very first backup for a group is always sourced live regardless of
policy (nothing to replay from yet).
- The replay/volatile staging instance now starts with ssl = off:
the copied postgresql.conf/postgresql.auto.conf still carries the
source node's own ssl_cert_file/ssl_key_file paths, meaningless
here since the archiver has no Postgres SSL certs of its own --
left enabled, the staging instance failed outright at startup.
- New CLI: `pg_autoctl create/show/set basebackup-policy`, and
`pg_autoctl create archiver --basebackup-policy <name>` to attach
one at creation time via set_archiver_policy().
Verified via a full --no-cache Docker rebuild plus the archiver_wal_
capture, archiver_basebackup_generation, archiver_basebackup_policy,
and archiver_bootstrap_and_fast_forward pgaftest specs, all passing.
archiver_wal_capture.pgaf: fixed a wrong segment-1 assumption (the archiver's replication slot only protects WAL from its own creation time onward -- by the time it's created, node1+node2's own bootstrap has typically already consumed segments up to the empirically observed floor, segment 3) and switched two `wait until ... state is primary` assertions to the real terminal state after a permanent primary loss (`wait_primary`: WAIT_PRIMARY -> PRIMARY requires another node to reach reported SECONDARY, which an archiver never will). archiver_basebackup_generation.pgaf: the schema's own 'default' policy (frequency 24h) no longer produces a second, replay-sourced backup within any sane test window now that scheduling is policy-driven instead of hardcoded "bootstrap live, then exactly one replay". Attach a short-frequency, source=replay policy during setup so the spec's own remaining job -- proving the replay/volatile generation pipeline itself still works -- is still genuinely exercised. archiver_bootstrap_and_fast_forward.pgaf: same wait_primary fix as above, applied where this spec also stops the original primary for good partway through. New: archiver_basebackup_policy.pgaf, covering the base-backup policy feature end to end -- a fast-cycling, maxcount=3 policy created via the real CLI, attached via set_archiver_policy(), reaching and holding a stable retained count after several times its frequency has elapsed. Registered both archiver_basebackup_policy and (previously missing) archiver_bootstrap_and_fast_forward in tests/tap/schedules/node.sch. All four specs verified passing against a from-scratch --no-cache Docker rebuild.
Intro - Rewrote the opening paragraph: pg_auto_failover is a complete system (pg_autoctl runs as its own pid 1 supervising postmaster), not just an extension -- dynamic topology, automated or operator- driven, two modes of operation (command-driven CLI and node.ini + `pg_autoctl node run`). - New "High Availability, Disaster Recovery, and Backups: One System" section with a new two-panel diagram (arch-ha-dr-unified.tex/.svg) contrasting the typical separate-HA-tool/separate-backup-tool split against pg_auto_failover's single control plane for both. Failover State Machine - New "Archiving" subsection in the State reference, covering the ARCHIVING state's real transitions (verified against live `pg_autoctl inspect fsm list --json` output), its exclusion from candidacy/quorum, and its role as a Fast_forward-eligible WAL source. - Added the 3 real archiving edges to the "Node init / join" and "Failover / promotion" mermaid diagrams, with a new archiverState color class and cross-reference notes. Updated the "20 states and 77 transitions" summary line to 21/80. Fault Tolerance - New "Archiving Nodes and Disaster Recovery" section: WAL capture independent of any standby, base backups on a policy, rebuilding a node (or a whole formation) from an archiver's cache, and how archiving nodes participate in (and are excluded from) failover. Operations - New docs/archiving.rst page: registering an archiver, creating and attaching base-backup policies, watching an archiver, and rebuilding a node with `pg_autoctl create postgres --from-archiver` -- including the disaster-recovery case of rebuilding a whole formation from a single surviving archiver. Reference - New CLI reference pages for `pg_autoctl create/show/set basebackup-policy`, registered in their respective toctrees. Verified with a clean `sphinx-build -W --keep-going` (no warnings, no broken references).
…anels Replaces the single stacked arch-ha-dr-unified diagram with two separate figures, each a "production architecture" style pair of dashed service-boundary boxes with a header + inner service pills: - arch-ha-dr-typical: High Availability (Patroni, repmgr) next to Disaster Recovery + Backups (pgBackRest, pgBarman) -- two entirely separate boundaries, naming the actual products a typical setup reaches for. - arch-ha-dr-pgautofailover: High Availability + Disaster Recovery collapse into a single pg_auto_failover box; Backups (pgBackRest, pgBarman) remains its own separate boundary. Colors are a muted, readable palette local to these two diagrams (dark-tinted text, pale tints for fills) rather than raw saturated brand colors used directly as text -- the previous version's bright green header/body text (mbox, #9BF00B) was a real readability problem. Node heights are compact (1.35cm pills) instead of the previous 2.3cm/6.4cm boxes, since most of these boxes hold a single line of text. intro.rst's "High Availability, Disaster Recovery, and Backups: One System" section is retitled "High Availability and Disaster Recovery: One System" and its body adjusted to match: Backups, in the narrower sense of retention/cataloguing/cloud tiers, is now described as its own remaining concern rather than folded into "one system," matching what the new diagrams actually show.
…maid The five keeper-FSM mermaid diagrams had drifted from real KeeperFSM[] output -- verified by re-running `pg_autoctl inspect fsm mermaid <phase>` for all five phases and diffing byte-for-byte against what was checked into the docs. Real gaps found and fixed: - Failover / promotion was missing the entire "wherever you were, you're being demoted now" fan-out (init/single/catchingup/secondary/ prepare_promotion/stop_replication/maintenance/prepare_maintenance/ wait_maintenance/report_lsn/fast_forward, each with both a -> demoted and -> demote_timeout edge), plus several report_lsn fan-in edges (fast_forward/prepare_promotion/stop_replication/ demote_timeout/join_secondary -> report_lsn) -- 28 missing edges in this diagram alone. - Node removal / drop was missing wait_maintenance -> single and fast_forward -> single. - Maintenance was missing wait_maintenance -> report_lsn. - The archiving state's edges (added in an earlier, hand-written pass) are now the tool's own generated labels/coloring (electionState amber, not a separate hand-added archiverState class) instead of hand-embellished text not backed by any real KeeperFSM[] comment. Node init / join and Steady-state / config changes already matched exactly. Updated the summary line and the "replaces the old Graphviz diagram" note from the stale 80/68 transition counts to the real total: 21 states, 102 transitions (111 raw KeeperFSM[] edges minus the 9 excluded join_primary ones). Fixed the Failover / promotion intro paragraph's "still less than half the size of the full graph" claim -- at 57 of 102 edges it's now over half, which the added fan-out edges explain (most of that diagram's size is exactly that "interrupted from anywhere" fan-out). Added an explicit `archiving_state` label on the State reference's Archiving entry so other pages can :ref: it directly instead of relying on an implicit, same-document-only section-title link.
New docs/archiving-internals.rst, in the Architecture toctree: the technical reference for how archiving is actually built, meant to be the main place to extend for later milestones (warm standby, PITR, cloud push). Covers, grounded directly in the current source (function names, exact invocations, exact file paths): - The two forked processes per archiver (capture, serve) and the two files (archiver-position, archiver-routes.ini) that are their only channel to each other -- new arch-archiver-internals diagram. - WAL capture: how an archiver's replication slot reuses the exact same mechanism a real standby's slot uses, with zero primary-side special-casing; the exact pg_receivewal invocation; how the real captured position is computed (including .partial-segment trailing- zero trimming) and shared across the fork boundary; what happens to pg_receivewal across a failover. - Base backup generation: the basebackup_policy table and its 3-tier resolution chain; exactly when a backup is due (bootstrap, onpromotion, frequency); the live pg_basebackup invocation; the full replay/volatile pipeline (staging instance, recovery config, promote, basebackup over loopback, discard); retention pruning. - pg_walsender: why it's a from-scratch reimplementation (no frontend-linkable server-side replication library exists), its process model, the routes-file-based auth/routing mechanism, and a full table of every wire command it implements. - How pg_autoctl create postgres --from-archiver and FAST_FORWARD reuse the port==0 archiver-serve-port resolution trick to talk to pg_walsender with no archiver-specific code past that one lookup. - Build/process wiring, and an explicit "extension points" section listing what M6/M7/M8 build on top of, and what's schema-complete but not yet enforced (concurrency, allowed_hosts). Verified clean with sphinx-build -W --keep-going (no warnings, no broken references).
Mermaid diagrams already get pan/scroll-to-zoom via mermaid_d3_zoom (conf.py), but that's specific to Mermaid's own inline-SVG rendering and never applied to the tikz-rendered figures the rest of the docs embed via `.. figure::` -- those render as plain <img src="....svg">, which d3-zoom can't attach to. New docs/_static/js/zoom.js + css/zoom.css: a small, dependency-free overlay wired to every `figure img` at page load. Click (or Enter/Space when focused) opens the image full-screen on a dark backdrop; scroll to zoom, drag to pan, double-click to reset, Esc/backdrop-click/close-button to dismiss. Wired site-wide via conf.py's existing add_css_file/ add_js_file setup() hook, the same mechanism already used for the project's custom CSS. Verified interactively: click opens the overlay, wheel/drag/dblclick/Esc all behave as expected, and a clean sphinx-build -W --keep-going.
New top-level section right after the page's own intro, before "The pg_auto_failover Monitor": frames High Availability as two distinct guarantees -- Service Availability (the Postgres service stays reachable, what the rest of this page/failover-state-machine.rst/ fault-tolerance.rst describe) and Disaster Recovery (the data survives even total loss of every node that ever held it, what archiving-internals.rst and the archiver covers) -- cross-referencing into both rather than duplicating either. Adds a page-level `fault_tolerance` label to fault-tolerance.rst (it had no explicit label of its own) so this new section can :ref: it directly.
Three boxes -- High Availability, Disaster Recovery, Backups -- on a single horizontal line in both diagrams, with whichever pair shares a provider wrapped in one outer box: - arch-ha-dr-typical: High Availability stands alone; Disaster Recovery and Backups are wrapped together (the same two products, pgBackRest/pgBarman, cover both roles in a typical setup). - arch-ha-dr-pgautofailover: the same three boxes, same colors, same layout, just regrouped -- High Availability and Disaster Recovery are now the wrapped pair, inside a box labeled pg_auto_failover; Backups stands alone in the slot Disaster Recovery and Backups shared on the other diagram. Replaces the previous, more complicated pass at this (2-column, vertically-stacked nested sub-boxes) with the simpler request: 3 boxes, one line, 2 of them wrapped.
arch-ha-dr-pgautofailover.tex had High Availability and Disaster Recovery wrapped in the pg_auto_failover box on the right and Backups standalone on the left -- reading right-to-left relative to arch-ha-dr-typical.tex's High Availability, Disaster Recovery, Backups order. Swapped positions (same widths/gaps, mirrored placement) so both diagrams read in the same order, only the wrapping differs.
…e for operators Full rewrite from an operator's viewpoint instead of a contributor's: no function names, no file:line citations, no build/Makefile details, no "extension points for future milestones" section aimed at future code contributors. Keeps the process-tree diagram. New structure: - Data flow: what actually moves (WAL streaming, base backup generation live vs. replay, serving it back out), and that none of it routes through the monitor. - Storage: a concrete directory-listing example, what each file/ subdirectory actually is, and how to reason about disk sizing (base backups ~ maxcount x one backup's size, WAL ~ however much has accumulated since the oldest still-retained backup). - Network exposure: what's listening, how it's authenticated, how to think about firewalling it. - Process model: an ASCII process tree for the simple case, then three scenarios by request -- more/fewer standby nodes in the group (doesn't change the archiver's own process tree at all), several independent formations (one archiver process tree per formation, fully independent), and a Citus formation (honest about current scope: only the coordinator's own group is covered today, worker groups are not yet). - What you can point at an archiver: the wire-protocol commands its serving side understands, as a definition list (matching this project's own CLI-option documentation style) instead of a table. Keeps the `archiving_architecture` label the file already carried, so existing :ref: links to it from architecture.rst/fault-tolerance.rst (currently being edited separately) keep resolving without changes there. Verified with sphinx-build -W --keep-going: clean, no broken references.
A fair question from review: does /var/lib/pgaf/archiver1/ need a <formation>/<group>/<nodeid> path inside it? No -- but the doc didn't say why where it mattered. One archiver directory belongs to exactly one source (pgReceivewalPid/basebackupPid are file-scope globals, one of each per running archiver process), so nothing else ever writes there to collide with. Covering more than one source is several separate archivers, each its own directory, not one archiver partitioning a shared one -- already explained in "Several formations" below, now stated up front where the directory example itself appears.
pgautofailover.archiver_add_formation() is now safe to call again for a formation some of whose groups are already attached (ON CONFLICT DO NOTHING on the per-group node insert) -- an archiver's own reconciler calls this periodically to pick up newly-added groups, not just once at creation time. New pgautofailover.list_archiver_memberships(archiverid): every (formation, group) an archiver currently holds, across every formation it's attached to -- what an archiver process itself discovers at runtime, distinct from get_archivers()'s own "every archiver attached to one formation" scope. pgautofailover.archiver.region is now wired up end to end (was schema-only, always NULL, dead code): register_archiver() gained a region parameter (default 'default', matching pgautofailover.node's own convention), get_archivers() now returns it. Multiple archivers can already attach to the very same formation (archiver_add_formation() names each ARCHIVING row after its own archiverid, so two different archivers never collide) -- distinct regions across them is the intended shape for geographically-redundant DR coverage of one formation, and archiver_policy's own archiverquorum column already anticipates requiring more than one archiver's confirmation. archiving_schema.sql/.out extended to cover all of the above.
An archiver process can now hold ARCHIVING memberships in more than one (formation, group) at once -- every group of a Citus formation, or several unrelated formations altogether -- with dynamic runtime reconciliation, not just static coverage at creation time. New intermediate supervised process, the reconciler (service_archiver_reconciler.c/h), forked from start_archiver() alongside "serve": periodically calls pgautofailover.list_archiver_memberships() and diffs it against its own dynamic Service array, starting one WAL-capture child per membership (one pg_receivewal each, since a single process can only ever follow one primary) and stopping any no longer attached. Each membership gets its own <root>/<formation>/<group>/ storage subdirectory and Keeper (build_membership_keeper()), independent of every other membership's own state. Runs as its own process rather than folding into start_archiver()'s own top-level supervisor for blast-radius reasons: a bug in this genuinely new dynamic-reconciliation logic can only crash and restart the reconciler itself, "serve" is unaffected. On its own crash recovery, it SIGTERMs any leftover children (tracked in a small persisted file) and starts every currently-discovered membership fresh rather than trying to adopt still-running orphans -- a replication slot retains WAL regardless of how many times its consumer reconnects, so this costs nothing. supervisor.c gained supervisor_add_service()/supervisor_remove_service() and a periodic-callback hook (supervisor_start_with_callback(), with supervisor_start() becoming a thin wrapper -- zero behavior change for every existing caller) so the reconciler can grow/shrink its own supervised Service array at runtime; only usable by a caller whose services array is heap-allocated to begin with, which every ordinary supervisor_start() caller's plain stack array is not. service_archiver_serve.c now writes one "[formation/group]" section per membership to the shared routes file (routes discovered fresh from monitor_list_archiver_memberships() every refresh) instead of exactly one, since a single pg_walsender already multiplexes any number of client connections and needs no per-membership fan-out the way capture does. cli_create_node.c: `create archiver --formation` may now be repeated, to attach the same archiver to several formations right from creation. `create archiver --run` now goes through start_archiver() (the same path `pg_autoctl node run` already used) instead of calling service_archiver_loop() directly, which only ever ran the WAL-capture half and skipped "serve" entirely -- a real, pre-existing inconsistency between the two invocation styles. New `--region` flag on `create archiver` (register_archiver()'s own new region parameter), and nodespec.c now forwards a node.ini's own [settings] region to the archiver create command it execs into -- previously read into the ini parser but silently dropped for kind = archiver specifically, even though ordinary nodes' own --region already worked. pg_autoctl watch's archivers panel gained a Region column.
ci/banned.h.sh (part of `make lint`) had never actually been run against this binary before -- only citus_indent had. Fixes both severity tiers: - snprintf -> sformat (this project's own bounds-checked wrapper), matching the convention already used throughout pg_autoctl. - atoi -> stringToInt/stringToInt64, which report parse failure instead of silently returning 0. - Every remaining memcpy/strcpy call verified bounds-checked (a Min(..., destSize - 1) guard before the copy, or a fixed-size protocol/filename field into a matching fixed-size buffer) and annotated IGNORE-BANNED. No behavior change; `sh ci/banned.h.sh` now exits 0 for this binary.
An archiver can now be declared directly inside cluster { }, as its own
"archiver <name> { }" block -- a sibling of monitor/formation, not nested
inside a formation_block the way it had to be before. This matches the
real data model (pgautofailover.archiver has no formation column at all;
it attaches to one or more formations by name, it isn't a member of any
one of them):
archiver archiver1 {
formation default # required; exactly one at create time
region eu-west # optional; default "default"
create and launch deferred # optional
}
A real shift/reduce ambiguity turned up while designing this: a brace-less
"formation <name>" archiver option is indistinguishable, at one token of
lookahead, from a brand new top-level formation_block starting right
after (formation_block's own opening also accepts a bare name). Braces
around the archiver's own option list resolve it.
Internally, fold_archivers_into_formations() (called once, right after
yyparse() returns) turns each declared archiver into an ordinary TestNode
appended to its own named formation's node list -- so every existing
per-node code path (ini writing, "pg_autoctl node run <ini>" as the
container command, healthcheck/depends_on ordering, create/launch
deferred handling) already used for an archiver nested inside a
formation_block just works for these too, completely unmodified. Only
one formation is accepted per archiver, since pg_autoctl create
archiver's own ini-driven bootstrap has no notion of attaching to more
than one at create time (unlike the CLI's own repeatable --formation) --
a clear error directs to attaching the rest dynamically once running
instead of silently dropping them.
TestNode's own existing "region" support (already wired through to
ordinary nodes' --region) now also reaches an archiver, via
build_membership_keeper()'s sibling in nodespec.c gaining the matching
--region push for kind = archiver.
archiver_multi_formation.pgaf: one archiver, brought up ARCHIVING for a
"default" formation, then attached to a second, independent formation
purely through a monitor-side RPC while already running -- proving the
reconciler's own dynamic-attach discovery, not just static coverage at
creation time.
citus_basic_operation.pgaf: test_011/test_012 cover a single archiver
attached to a whole Citus formation, ending up with one membership per
group (coordinator's group 0, plus each worker group) -- proving Citus
coverage isn't limited to the coordinator. Uses the new top-level
"archiver { }" syntax with "create and launch deferred" +
`pg_autoctl node start`, triggered only once every worker group is
already registered.
archiver_budget_architecture_regions.pgaf: the "budget architecture" (see
docs/architecture.rst) -- node1/dc1, node2/dc2, archiver1/dc3 -- proving
--region round-trips correctly for both node kinds and that the archiver
still does real work regardless of its own label.
archiver_two_regions.pgaf: two archivers (eu-west/us-east) attached to
the same formation, proving both independently capture WAL (via an
archiverQuorum raise after the fact, since wal_archived() itself
aggregates across every attached archiver and doesn't expose a
per-archiver breakdown).
…tive intro.rst, architecture.rst, fault-tolerance.rst: fold Archiving & Disaster Recovery into the existing Service Availability / Business Continuity framing rather than treating it as a bolt-on -- the "budget architecture" (two Postgres nodes plus an archiver, in place of a third live standby) as a named trade-off, WARM standby via cascading replication-protocol serving, and PITR via "transient" nodes that can be reified into new groups.
…rence fixes
archiving-details.rst: rewrite storage layout for per-<formation>/<group>
subdirectories, process model for the reconciler + N capture children
(diagram regenerated to match), and the "Several formations"/"A Citus
formation" sections that previously and incorrectly described a
one-archiver-per-formation limitation.
New docs/ref/pg_autoctl_create_archiver.rst (this command had no
reference page or man page at all before), plus create/show/set
basebackup-policy man page entries that were missing from conf.py's own
man_pages list despite already having rst pages.
archiving.rst: document repeatable --formation and --region on `create
archiver`, including the geographically-redundant-DR use case for
multiple archivers on one formation.
pg_autoctl_node.rst / pg_autoctl_node_run.rst / pg_autoctl_node_start.rst
/ operations.rst: fix a real, pre-existing inaccuracy found while
verifying this session's own reliance on the ini-driven archiver
bring-up path -- these pages documented a single "[launch] mode =
deferred" key, but the actual implementation (nodespec.c) has always
used two independent keys, "create" and "run", each gating a different
step (node creation vs. actually starting Postgres/the supervisor).
Also: "[node] kind" was missing "archiver" as a valid value entirely.
pgaftest.rst: document the new top-level "archiver { }" syntax and its
single-formation-at-create-time constraint, and fix its own stale
"launch deferred = sleep infinity" claim (containers deferred this way
still run the ordinary `pg_autoctl node run <ini>` command; the ini's own
[launch] section is what makes it wait).
PostgreSQL 19 (beta) changed pg_lsn's own text output to always zero-pad the lower 32 bits to 8 hex digits (0/500000 -> 0/00500000); 14-18 all still use the shorter, non-padded form. This is an upstream Postgres change, unrelated to this branch's own work -- the affected lines are archiving_schema.sql's pre-existing basebackup/pitr_node_status content. Matches the project's own established per-version override convention (src/monitor/expected/pg19/expected/, already used by several other tests) -- archiving_schema just never had one yet, since nobody had run it against PG19 before. Fixes the "Build run image (PG19)" and "Build test image (PG19)" CI job failures from run 84226284129.
tests/upgrade builds the current Dockerfile against old-release source trees that predate pg_walsender, so the build stage never produces that binary and the literal COPY fails. The [r] bracket-expression is treated as a glob by BuildKit; an empty match is not an error for COPY (unlike a literal missing path), so this makes the copy optional without touching the old-release source or the upgrade-test tooling. Temporary workaround -- revisit after the release.
…collision Two real bugs found by Docker verification of the archiver/region redesign: - nodespec_write() had no case for NODE_KIND_ARCHIVER, falling through to "postgres". cli_node_start() (pg_autoctl node start) reads the spec, clears the deferred flags, and rewrites the ini via this same function -- silently downgrading a deferred archiver's own ini to a plain postgres node kind, losing --formation/--region in the process. Root cause of citus_basic_operation.pgaf's test_011 failure. While in there, also stopped nodespec_write() from dropping [settings] region entirely on every round-trip (it was never emitted at all), which would silently erase a deferred node's region on 'node start'. - build_membership_keeper() shallow-copies the archiver-level template Keeper, inheriting its already-computed config.pathnames. The subsequent keeper_config_set_pathnames_from_pgdata() call is then a no-op, since SetConfigFilePath/SetStateFilePath/SetNodesFilePath all skip already-nonempty fields -- so every membership beyond the first silently pointed at the template's (or an earlier membership's) state file instead of its own. Root cause of archiver_multi_formation.pgaf's test_003 failure. Fixed by memset-ing pathnames to zero before recomputing them. Both confirmed via a fresh build + citus_indent + banned-API check; Docker/pgaftest re-verification of the two previously-failing specs to follow.
pgautofailover.archiver_add_formation() always synthesizes an ARCHIVING membership row's nodename as 'archiver-<archiverid>-<groupid>', never the plain --name given at create-archiver time. citus_basic_operation. pgaf and archiver_multi_formation.pgaf both asserted WHERE nodename = 'archiver1' in their monitor SQL checks (test_011/ test_012 and test_003/final teardown check respectively), which never matched -- these checks always returned zero rows. Both specs already constrain formationid (and groupid where relevant) and have exactly one archiver each, so WHERE nodename LIKE 'archiver-%' is a safe, unambiguous fix. Also corrects both header comments' stale assumption that these rows share the plain --name. Confirmed via live Docker/pgaftest run: citus_basic_operation.pgaf now passes all 16 steps (previously failed at test_011). archiver_multi_ formation.pgaf's test_003 (the dynamic-attach step this nodename fix covers) now passes too; a separate, unrelated bug in that spec's test_004 (goalstate incorrectly reset to wait_standby for a dynamically-attached second membership) is still under investigation.
WS_SERVER_VERSION was a fixed "16.4" MVP placeholder (already flagged as a known follow-up in its own comment). Real libpq clients (pg_basebackup, pg_receivewal) reject a server reporting a version newer than themselves, so every pg_walsender build for a PG version other than 16 made "create postgres --from-archiver" fail with "pg_basebackup: error: incompatible server version 16.4" -- caught by CI run 84233594160's node schedule on PG14/PG15/PG19. pg_walsender is built once per PGVERSION, against that version's own server headers (Makefile.common's pg_config --includedir-server), so PG_VERSION/PG_VERSION_NUM (from pg_config.h, via postgres_fe.h) are already this build's real target version -- no new plumbing needed, just stop shadowing them with a fixed string.
Adding archiver_wal_capture/archiver_basebackup_generation/archiver_ basebackup_policy/archiver_bootstrap_and_fast_forward to node.sch pushed every PG version over the pgaftest job's 20-minute step timeout: CI run 84233594160 shows PG16/PG18 timing out outright, and PG14/PG15/PG19 hitting real failures before they'd have gotten there either. node.sch's own header already documents this exact pattern from when the FSM edge-gap specs were split out to node-fsm-gaps.sch. New tests/tap/schedules/archiver.sch runs on every PG version, not PG17 only like node-fsm-gaps.sch: pg_walsender speaks the real Postgres wire protocol, so its correctness is genuinely version- sensitive (see the previous commit's server_version fix, caught by exactly this multi-version coverage).
…nodeport)
Every node's periodic node_active() report writes through this
function. For ordinary Postgres nodes, (nodehost, nodeport) is a
unique key. For an ARCHIVING membership row it isn't: archiver_add_
formation() gives every membership nodeport = 0 (a permanent sentinel
-- an archiver has no postmaster to be reachable on) and nodehost =
the owning archiver's own hostname, both identical across every
(formation, group) membership belonging to the same archiver
identity. Once an archiver has 2+ memberships, any one membership's
routine report silently overwrote reportedstate on every other
membership sharing the same archiver, without touching their
goalstate -- leaving reportedstate/goalstate inconsistent and putting
the affected membership's local FSM into an unrecoverable crash loop
("does not know how to reach state wait_standby from archiving").
Root-caused live (reproduced twice) via SQL statement logging while
debugging archiver_multi_formation.pgaf's test_004 failure.
Fixed by scoping the UPDATE on nodeid, the column that's actually
unique per row, using the nodeId the caller (node_active_protocol.c)
already has on hand from its own node lookup -- no new plumbing
needed.
Verified: full src/monitor SQL regression suite passes (20/20,
including node_active_protocol, archiving_schema, and all 6
concurrent-report tests -- no regression for ordinary-node reporting).
Live Docker/pgaftest: the target bug is confirmed fixed end-to-end
(both memberships now hold independent, consistent reportedstate/
goalstate, no crash loop); citus_basic_operation.pgaf still 16/16,
archiver_wal_capture.pgaf and archiver_budget_architecture_regions.pgaf
unaffected.
…ulti-region specs into CI test_004_capture_formation2_wal assumed formation2's archiver slot restart_lsn floor would start at an early, low-numbered segment since it's a fresh formation/group. Live testing (during the ReportAutoFailoverNodeState fix's own verification) showed this is wrong: the floor is segment 3, the same bootstrap-consumption behavior archiver_wal_capture.pgaf's own header comment already documents for the "default" formation -- a freshly-created replication slot's restart_lsn reflects whatever WAL bootstrap/registration itself generated before the slot existed, independent of which formation it belongs to. Segments 1/2 never actually get archived; only 3 onward do. Fixed the two hardcoded segment numbers and corrected the header comment's wrong assumption. Also adds a new tests/tap/schedules/archiver-multi.sch (PG17-only, matching node-fsm-gaps.sch's own rationale -- this is reconciler/SQL logic coverage, not pg_walsender wire-protocol coverage) wiring archiver_multi_formation.pgaf, archiver_budget_architecture_regions. pgaf, and archiver_two_regions.pgaf into CI for the first time -- all three existed as pgaftest specs but were never reachable by any CI schedule until now. Verified: all 3 specs pass in full (5/5, 2/2, 2/2); the combined schedule runs in 3m25s wall-clock, comfortably under the 20-minute CI step timeout that a related schedule in this same PR already blew once.
…lay-backup assertion test_001_replay_backup_lands slept a fixed 60s then asserted the replay/volatile backup had landed. CI run 84233594160 (PG14 node schedule) failed this with "expected replay, got live": the bootstrap live backup had landed but the subsequent replay backup hadn't yet, under CI resource contention. pgaftest's DSL has no generic SQL-condition polling primitive to switch to (test_spec_parse.y's "wait until" forms are all node-state- specific -- state/assigned-state/stopped/replays-lsn), so this follows archiver_basebackup_policy.pgaf's own established precedent for backup-timing checks: a generous fixed sleep, not a tight one. Bumped 60s to 120s -- scheduling itself is checked every 1s (PG_AUTOCTL_KEEPER_SLEEP_TIME) so the 10s policy interval is noticed promptly, but generating a replay/volatile backup is several real Postgres-instance lifecycles (extract, replay, promote, backup, discard), not a single fast pg_basebackup call, so its own wall-clock cost is the real variable here. Verified: 3/3 consecutive local runs pass, ~120.4s each (no C files touched; docker-check/banned.h.sh clean regardless).
Several archiver specs used a "sleep N seconds, then run one SQL
query, then assert" pattern to wait for an async condition (a WAL
segment archived, an archiver reaching a state, a base backup
landing) instead of actually polling. This caused real CI flakiness --
a fixed sleep either wastes time past a condition that was already
true, or isn't long enough under load and produces a flaky failure.
Adds a new CMD_WAIT_SQL command:
wait until sql <service> { SQL } is { value } [timeout Ns]
which polls exec_sql_on_service() every second until its (substring-
matched, same semantics as `expect { }`) output contains <value>, or
times out. This is the building block; three sugar forms cover the
repeated shapes found across the archiver specs, all lowering to
CMD_WAIT_SQL at parse time with no new runtime machinery:
wait until wal segment "<segment>" archived in <formation>/<group>
wait until archiver state is <state> in <formation>[/<group>]
wait until basebackup <source|status|replaymode> is <value> in <formation>/<group>
The archiver-state form exists because an ARCHIVING membership row's
nodename is always synthesized by archiver_add_formation() as
'archiver-<archiverid>-<groupid>', never the plain --name given at
create-archiver time -- the ordinary "wait until <node> state is
<state>" form (which matches on nodename = $1) can't see these rows
at all, let alone disambiguate more than one membership sharing the
same archiver.
Grammar changes regenerated via `make generate` (src/bin/pgaftest),
zero bison conflicts. docs/ref/pgaftest.rst documents all four forms.
Verified: full grammar round-trip via `pgaftest indent` on every new
form, a hand-written timeout-path spec confirms clean 5s failure (no
hang, no false pass), and end-to-end Docker/pgaftest runs across all
8 specs that use or sit next to this feature (see next commit for the
migration itself).
Replaces every "sleep N + sql + expect" call site that was polling for an async condition with the new wait-until-SQL forms (previous commit), across: archiver_wal_capture.pgaf archiver_multi_formation.pgaf citus_basic_operation.pgaf archiver_budget_architecture_regions.pgaf archiver_two_regions.pgaf archiver_basebackup_generation.pgaf archiver_bootstrap_and_fast_forward.pgaf archiver_basebackup_generation.pgaf's own fixed 120s sleep (added in an earlier commit as a stopgap for CI flakiness) is replaced outright by the new "wait until basebackup ... is ..." polling form, which is the real fix that stopgap was standing in for. archiver_basebackup_policy.pgaf is deliberately NOT migrated: its `count(*) = 3` check needs a stable, settled value after enough retention cycles have elapsed, not a first-reach-true poll -- a naive poll-until-true would risk a false pass on a transient count. Its existing fixed sleep is correct by design, not a flakiness bug. Also fixes a second, real bug this migration surfaced in archiver_bootstrap_and_fast_forward.pgaf's test_001: the monitor's own "basebackup complete" status and pg_walsender's actual ability to serve that backup are two different things. cmd_base_backup.c checks route->basebackupDir, which service_archiver_serve.c only refreshes every ARCHIVER_SERVE_ROUTES_REFRESH_TICKS (30) ticks -- so there's a real window where the monitor says "complete" before the archiver's own route is servable. The instant wait-until-SQL poll exposed this race (the old spec's blind sleep 30s happened to also absorb it by accident). There's no SQL-observable signal for "the archiver's route is ready", so this bridges the known 30s refresh window with a documented sleep rather than guessing at, or inventing new machinery for, something that isn't visible from the monitor side. Verified end-to-end via Docker/pgaftest: all 7 migrated specs pass in full, plus archiver_basebackup_policy.pgaf as an unmigrated regression check (2/2, unaffected). citus_basic_operation.pgaf runs its full 16-step Citus HA suite clean (~3.5 min). No C files touched by this commit; make docker-check / banned.h.sh are clean regardless.
Real PG19 libpq performs a "GREASE" self-test on every new connection
(borrowed from TLS): it deliberately requests a bogus minor protocol
version (major=3, minor=9999) plus a "_pq_.test_protocol_negotiation"
startup option, to verify the server negotiates down properly rather
than silently accepting whatever was asked. A server that accepts it
without negotiating is treated as broken and the connection is
refused: "server incorrectly accepted \"grease\" protocol version
3.9999 without negotiation" -- this broke every PG19 archiver
connection in CI (pgaftest / archiver (PG19), the exact "create
postgres --from-archiver" bootstrap path).
ws_startup_negotiate() only ever checked the major version
((code >> 16) != 3) and ignored the minor version entirely, so it
just proceeded with whatever was requested, including the grease
probe's own nonsense value.
Fixed with a real NegotiateProtocolVersion ('v') response, matching
Postgres's own backend behaviour:
- new ws_send_negotiate_protocol_version() (framing.c/.h) sends the
full encoded version (major<<16 | newest supported minor) followed
by a count and list of unrecognized "_pq_.*" startup options --
real libpq's own pqGetNegotiateProtocolVersion3() rejects a
response that isn't properly encoded as "downgrade to pre-3.0",
and separately requires any _pq_.* option the client sent to be
echoed back as unsupported (we don't parse any, so every one seen
is unsupported by definition).
- ws_startup_negotiate() now parses the startup packet's key/value
pairs before responding (needed to collect the _pq_.* option
names), and sends the negotiate message whenever the requested
minor version isn't 0 (all pg_walsender actually implements),
continuing the connection at that version rather than closing it.
Verified against real PG19 beta2 psql (which performs the same
GREASE probe as libpq) connecting directly to a standalone
pg_walsender: IDENTIFY_SYSTEM succeeds, no negotiation error. End to
end: archiver_bootstrap_and_fast_forward.pgaf passes all 4 steps on
PG19, including the exact bootstrap step that failed in CI. Full
regression pass (archiver_wal_capture, archiver_basebackup_generation,
archiver_basebackup_policy) on PG19 unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Archiving & Disaster Recovery, Milestones 1–5 of the design (schema through base-backup generation + policy), plus the docs coverage for all of it.
An archiver is a new node kind that captures a group's WAL continuously (
pg_receivewal, via the same replication-slot mechanism a standby already uses) and produces periodic base backups (live or replayed locally from its own WAL cache), independent of whether any standby is healthy or even present. A new standalone binary,pg_walsender, serves that captured data back out over a real (subset of the) PostgreSQL replication protocol, so a realpg_basebackup, a real streaming standby, or this project's ownrestore_commandcan all talk to an archiver with no archiver-aware code beyond finding it.This is not ready to merge — opening now for early review of direction and approach while the remaining work continues. See "Known gaps" below.
What's in this PR
pgautofailoverSQL schema for archivers —archiver,basebackup_policy,archiver_policy,archiver_wal,basebackup,wal_archived(), retention/pruning functions.service_archiver,archiver serve,pg_walsender(M2): the keeper-sideARCHIVINGFSM state;service_archiver.c's WAL-capture loop; a brand-new standalone binary (src/bin/pg_walsender/) implementing enough of the replication wire protocol from scratch (no frontend-linkable server-side implementation exists anywhere in PostgreSQL to link against) —IDENTIFY_SYSTEM,SHOW,BASE_BACKUP,TIMELINE_HISTORY,CREATE/READ_REPLICATION_SLOT,START_REPLICATION, and aFETCH_FILEside channel forrestore_command.pg_autoctl node runsupport forkind = archiver(M3).archiver_wal/wal_archived()tracking, re-pointingpg_receivewalat a new primary after a failover.live(realpg_basebackupagainst a healthy node) andreplay/volatile(extract the last backup, replay locally captured WAL forward against a throwaway staging instance, snapshot over loopback, discard) sources; policy-driven scheduling (frequency,onpromotion) and retention (maxcount,maxage); new CLI (pg_autoctl create/show/set basebackup-policy,--basebackup-policyoncreate archiver).pg_autoctl create postgres --from-archiver(bootstrap a new node straight from an archiver's cache) andFAST_FORWARDreusing an archiver as a WAL source during a multi-standby election, both via the same well-known-port resolution trick, no archiver-specific code in the ordinary standby-init/fast-forward paths themselves.SINGLEandPRIMARYforever (group_state_machine.c), and the replay staging instance failed to start under SSL (missing certs it has no reason to have).archiving-internals.rst) written as the technical reference/extension point for the milestones after this one; a new Operations page;ARCHIVINGstate coverage in the FSM docs (plus a full regeneration of the five FSM mermaid diagrams frompg_autoctl inspect fsm mermaid— real drift was found and fixed there, unrelated to archiving); fault-tolerance coverage; a rewritten intro with new architecture diagrams; and a small site-wide docs feature (click-to-zoom on figures, generalizing the zoom Mermaid diagrams already had).Known gaps — why this isn't ready yet
pg_walsender(all-new wire-protocol code) and the FSM fix ingroup_state_machine.c.conf.pychange (adds the click-to-zoom JS/CSS): Sphinx's incremental build doesn't reliably re-emit the<script>/<link>tags on every already-built page just becauseconf.pychanged — only pages whose own.rstsource changed get regenerated. If a localdocs/_buildpredates this PR,make -C docs htmlalone won't retrofit the zoom feature onto older pages; runmake -C docs clean html(or deletedocs/_build) once to pick it up everywhere.archiving-internals.rst's "Extension points" section is written to be where that work plugs in.Testing
New/updated
pgaftestspecs:archiver_wal_capture,archiver_bootstrap_and_fast_forward,archiver_basebackup_generation,archiver_basebackup_policy— all passing against a from-scratch--no-cacheDocker rebuild.citus_indent --checkclean.sphinx-build -W --keep-goingclean (no warnings, no broken references).