diff --git a/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md b/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md new file mode 100644 index 000000000..cbd1cb248 --- /dev/null +++ b/docs/en/solutions/How_to_Deploy_CloudBeaver_for_PostgreSQL_Management.md @@ -0,0 +1,161 @@ +--- +kind: + - How To +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515008 +--- + +# How to Deploy CloudBeaver for PostgreSQL Management + +## Issue + +You need a web-based SQL client for browsing and querying PostgreSQL instances managed by Alauda Application Services, without installing a desktop tool on every operator's workstation. CloudBeaver is the open-source web edition of DBeaver and runs as a single-pod Deployment on Kubernetes. This how-to deploys CloudBeaver and connects it to a PostgreSQL cluster managed by the PostgreSQL Operator. + +## Environment + +- Any Kubernetes cluster reachable to operators (NodePort exposure is used below; Ingress / Route works equally well) +- A `StorageClass` capable of provisioning ReadWriteOnce PVCs — used to persist CloudBeaver workspace state across pod restarts +- Network reachability from the CloudBeaver pod to the target PostgreSQL Service (`` on port 5432) + +## Resolution + +### 1. Prepare the manifest + +Save the following to `cloudbeaver.yaml`. Adjust `storageClassName`, image registry, and resource requests to match your environment: + +```yaml +--- +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: cloudbeaver +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + storageClassName: sc-topolvm # replace with any RWO StorageClass available in the cluster + volumeMode: Filesystem +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: cloudbeaver +spec: + replicas: 1 + selector: + matchLabels: + app: cloudbeaver + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 1 + maxUnavailable: 1 + template: + metadata: + labels: + app: cloudbeaver + spec: + containers: + - name: cloudbeaver + image: docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest + imagePullPolicy: Always + ports: + - name: web + containerPort: 8978 + protocol: TCP + resources: + limits: + cpu: "1" + memory: 1Gi + requests: + cpu: 100m + memory: 256Mi + volumeMounts: + - name: cloudbeaver-data + mountPath: /opt/cloudbeaver/workspace + volumes: + - name: cloudbeaver-data + persistentVolumeClaim: + claimName: cloudbeaver +--- +apiVersion: v1 +kind: Service +metadata: + name: cloudbeaver +spec: + type: NodePort + selector: + app: cloudbeaver + ports: + - name: web + port: 8978 + targetPort: 8978 + protocol: TCP +``` + +> CloudBeaver stores its administrator password, saved connections, and query history under `/opt/cloudbeaver/workspace`. Without a persistent volume, everything is lost on every pod restart. Confirm the chosen `storageClassName` exists in the target cluster before applying. + +> **Air-gapped / IPv6-only clusters:** the public `docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest` image may be unreachable from the cluster. Mirror it into the cluster's own registry first and reference that path in the Deployment, for example `skopeo copy docker://docker-mirrors.alauda.cn/dbeaver/cloudbeaver:latest docker:///dbeaver/cloudbeaver:latest`. + +### 2. Deploy + +```bash +kubectl -n apply -f cloudbeaver.yaml +kubectl -n rollout status deploy/cloudbeaver +``` + +### 3. Discover the access URL + +The Service uses NodePort, so any node IP plus the allocated NodePort exposes the UI: + +```bash +namespace= +HOST=$(kubectl -n "$namespace" get pod -l app=cloudbeaver \ + -o jsonpath='{.items[0].status.hostIP}') +PORT=$(kubectl -n "$namespace" get svc cloudbeaver \ + -o jsonpath='{.spec.ports[0].nodePort}') +echo "http://$HOST:$PORT" +``` + +Open the URL in a browser. + +### 4. Initial setup + +1. On first launch CloudBeaver prompts you to set the administrator password. Choose a strong password and store it safely — this account governs all subsequent server-side configuration. +2. Log in with the administrator user. +3. (Optional) Switch the UI language under the user menu in the top-right. + +### 5. Connect to a PostgreSQL instance + +1. Click **New Connection** and choose **PostgreSQL**. +2. Fill **Host** with the cluster Service name and **Port** `5432`. From inside the cluster the host is `.` (the read-write Service); the `-repl` Service points at replicas. From outside the cluster, retrieve the NodePort or LoadBalancer address: + + ```bash + kubectl -n get svc + ``` + +3. Enter the database user and password. The Operator stores each role's credentials in a Secret named `..credentials.postgresql.acid.zalan.do`. Retrieve the `postgres` superuser password from its Secret: + + ```bash + kubectl get secret postgres..credentials.postgresql.acid.zalan.do \ + -n -o jsonpath='{.data.password}' | base64 -d; echo + ``` + + The same Secret also carries `username`, `host` and `port` keys. + +4. (Optional) Set **Database** to the target database; otherwise CloudBeaver connects to the default `postgres` database. +5. Under **Access Management**, grant the current CloudBeaver user permission to use the new connection. +6. Save and test the connection. SQL editors can now be opened from the browser and queries executed against the PostgreSQL cluster. + +### 6. Uninstall + +```bash +kubectl -n delete -f cloudbeaver.yaml +``` + +The PVC is removed alongside the rest of the manifest. To preserve CloudBeaver state for a future redeploy, delete only the Deployment and Service and re-attach the existing PVC during the next install. diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md new file mode 100644 index 000000000..db2fb7e8b --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md @@ -0,0 +1,364 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - 4.3 +id: KB260720001 +--- + +# PostgreSQL Instance Cross-Cluster Migration Guide (Operator v4.3.3) + +## Background + +### The Challenge + +Sometimes a running PostgreSQL instance must move to a different Kubernetes cluster — cluster decommissioning, hardware refresh, moving a workload between ACP platforms, or consolidating environments. Dump-and-restore migrations require downtime proportional to database size and lose writes made after the dump. + +### The Solution + +This guide migrates a PostgreSQL instance across clusters using the operator's hot standby (cross-cluster replication) feature: the target instance is created as a *standby cluster* that bootstraps directly from the source via streaming replication, stays continuously in sync, and is then promoted in a controlled two-phase switchover. Downtime is limited to the switchover itself (seconds to a couple of minutes), and data integrity is verified with checksums before and after cutover. + +This procedure was validated end-to-end with PostgreSQL Operator v4.3.3 on two separate ACP platforms (source and target in different platforms, connected via NodePort). It builds on the [PostgreSQL Hot Standby Cluster Configuration Guide](./How_to_Use_PostgreSQL_Hot_Standby_Cluster.md) (KB251000009); read that first for concept background. + +## Environment Information + +- PostgreSQL Operator: v4.3.3 on **both** source and target clusters (see [Important Limitations](#important-limitations)) +- ACP: any 4.x cluster able to run the v4.3.3 operator; source and target may be in different ACP platforms +- PostgreSQL: same major version on both sides (this guide uses 16) + +## Important Limitations + +- **The operator version must match on both sides.** Cross-version pairing (e.g. a v4.2+/v4.3 standby against a primary managed by v4.1.x) fails with `pq: column "external_ip" does not exist` and — dangerously — leaves the "standby" running as an empty independent primary (tracked as ECO-703). See [Troubleshooting](#troubleshooting). +- Source and target must run the same PostgreSQL major version. +- The standby cluster must initially be created with `numberOfInstances: 1`; scale it up after promotion. +- `replSvcType` must be identical on both clusters. +- The target cluster must be able to reach the source cluster's node IP + NodePort (standby pulls from primary). Verify this before starting — see Step 2. If there is **no network path between the clusters**, the streaming approach cannot work — use the [workstation-relayed logical migration](#alternative-migration-without-inter-cluster-connectivity) instead. +- On mixed-architecture target clusters, pin the instance (and ideally the operator) to nodes matching the source architecture. Streaming replication copies the data directory bit-for-bit; PostgreSQL does not support mixed-architecture replication. + +## Migration Overview + +``` +Step 1 Prepare source: enable clusterReplication, record NodePort, baseline checksums +Step 2 Preflight: network reachability, version/arch checks +Step 3 Create standby on target (bootstraps from source, stays streaming) +Step 4 Verify sync: identity, lag, checksums +Step 5 Cutover: two-phase switchover (demote source, promote target) +Step 6 Post-migration: repoint clients; keep reverse standby or dismantle +``` + +## Step 1: Prepare the Source Instance + +If the source instance does not yet have cluster replication enabled, patch it (this is an online change; the operator creates the replication metadata and exposes the master service): + +```bash +SRC_NS="pg-migrate" +SRC_CLUSTER="acid-mig" + +kubectl -n $SRC_NS patch postgresql $SRC_CLUSTER --type merge -p '{ + "spec": { + "clusterReplication": {"enabled": true, "replSvcType": "NodePort"}, + "postgresql": {"parameters": {"max_slot_wal_keep_size": "10GB"}} + } +}' +``` + +> `max_slot_wal_keep_size` bounds WAL retained for the replication slot if the standby disconnects. Size it to your volume: it must fit in the instance's free disk space, and it defines how long a standby outage you can tolerate before a re-bootstrap is needed. + +Record the connection coordinates the standby will use: + +```bash +# NodePort of the source master service +kubectl -n $SRC_NS get svc $SRC_CLUSTER -o jsonpath='{.spec.ports[0].nodePort}{"\n"}' + +# A node IP hosting the instance (any node IP of the cluster works for NodePort) +kubectl -n $SRC_NS get pod -l cluster-name=$SRC_CLUSTER -o jsonpath='{range .items[*]}{.status.hostIP}{"\n"}{end}' +``` + +Confirm the source registered itself in the replication metadata (role `primary`, correct `node_port`): + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -x \ + -c "SELECT * FROM sys_operator.multi_cluster_info;" +``` + +Take a data integrity baseline. Adapt the checksum query to your schema — the point is a number you can compare after migration: + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -d -tA -c " +SELECT relname, n_live_tup FROM pg_stat_user_tables ORDER BY relname;" +# Example per-table checksum: +# SELECT count(*), sum(hashtext(id::text || payload)) FROM your_table; +``` + +Finally, force a checkpoint so the standby's basebackup starts from a consistent point: + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -c "CHECKPOINT;" +``` + +## Step 2: Preflight Checks on the Target + +**Network reachability** — the standby (operator pod *and* PostgreSQL pods) must reach the source NodePort. Test from a pod on the overlay network of the target cluster, on the node(s) where the instance will run: + +```bash +# From any pod with bash on the target cluster: +kubectl exec -- bash -c 'timeout 4 bash -c "echo > /dev/tcp//" && echo OPEN || echo CLOSED' +``` + +If this prints `CLOSED`, stop and fix connectivity first. Note that reachability can differ per node (broken egress on individual nodes has been observed); test from the nodes you will schedule onto. + +**Version check** — both operators must be v4.3.3 (or at least the same version): + +```bash +kubectl -n operators get csv | grep postgres-operator +``` + +**Architecture** — on mixed-architecture targets, decide the node set now and pin with `nodeAffinity` (shown in Step 3). + +## Step 3: Create the Standby on the Target Cluster + +Create the namespace and a bootstrap secret holding the **source** cluster's admin credentials: + +```bash +TGT_NS="pg-migrate" + +# Read the admin password from the source cluster: +kubectl --context -n $SRC_NS get secret \ + postgres.${SRC_CLUSTER}.credentials.postgresql.acid.zalan.do \ + -o jsonpath='{.data.password}' | base64 -d +``` + +```yaml +kind: Secret +apiVersion: v1 +metadata: + name: standby-bootstrap-secret + namespace: pg-migrate +type: kubernetes.io/basic-auth +stringData: + username: postgres + password: "" +``` + +Create the standby instance. Keep PostgreSQL version, parameters, and volume size aligned with the source; start with a single replica: + +```yaml +apiVersion: acid.zalan.do/v1 +kind: postgresql +metadata: + name: acid-mig # name may differ from the source + namespace: pg-migrate +spec: + teamId: acid + numberOfInstances: 1 # required initially for standby clusters + postgresql: + version: "16" # must match the source major version + parameters: + max_slot_wal_keep_size: '10GB' + volume: + size: 5Gi # same capacity as source + storageClass: + # Only needed on mixed-architecture clusters — match the source arch: + nodeAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + nodeSelectorTerms: + - matchExpressions: + - key: kubernetes.io/arch + operator: In + values: ["arm64"] + clusterReplication: + enabled: true + isReplica: true + peerHost: "" + peerPort: + replSvcType: NodePort + bootstrapSecret: standby-bootstrap-secret +``` + +What happens on creation: the operator connects to the source, copies the database user credential secrets into the target namespace, creates a local `-xcr` service whose endpoints point at the source nodes, and bootstraps the pod with `pg_basebackup` from the source. `patronictl list` shows `creating replica` during the basebackup (duration depends on database size and link bandwidth), then `Standby Leader | streaming`. + +## Step 4: Verify Synchronization + +All three checks below must pass before cutover. + +**1. Streaming state and shared identity** — the cluster identifier printed by `patronictl list` must be *identical* on source and target (it is the PostgreSQL system identifier). If the target shows a different identifier, it bootstrapped as an independent cluster and contains none of your data — see [Troubleshooting](#troubleshooting). + +```bash +# Target — expect: Standby Leader | streaming +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list + +# Source — expect the standby connected, plus the slot active: +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -c \ + "SELECT application_name, client_addr, state FROM pg_stat_replication; + SELECT slot_name, active FROM pg_replication_slots WHERE slot_name='xdc_hotstandby';" +``` + +**2. Replication lag** — write on the source, confirm it appears on the target within seconds, and compare LSNs: + +```bash +kubectl -n $SRC_NS exec ${SRC_CLUSTER}-0 -c postgres -- psql -U postgres -tA -c "SELECT pg_current_wal_lsn();" +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -tA -c "SELECT pg_last_wal_replay_lsn();" +``` + +**3. Data checksums** — rerun the Step 1 baseline queries on the target; every value must match. + +## Step 5: Cutover (Two-Phase Switchover) + +Perform the switchover in the documented order — demote first, then promote — so there is never a moment with two writable primaries. + +1. **Stop application writes** to the source (scale writers down, or hold traffic at the application layer). + +2. **Confirm zero lag** (Step 4, check 2 — the two LSNs must be equal once writes stop). + +3. **Phase 1 — demote the source** to a standby: + +```bash +kubectl --context -n $SRC_NS patch postgresql $SRC_CLUSTER --type merge \ + -p '{"spec":{"clusterReplication":{"isReplica":true},"numberOfInstances":1}}' +``` + +Wait until the source shows `Standby Leader` (it may pass through `stopped` briefly). The demoted source finds the target automatically through the replication metadata — no `peerHost` needs to be added to its spec. + +4. **Phase 2 — promote the target** and scale it to full size: + +```bash +kubectl --context -n $TGT_NS patch postgresql acid-mig --type merge \ + -p '{"spec":{"clusterReplication":{"isReplica":false},"numberOfInstances":2}}' +``` + +5. **Gate on the real promotion signal.** Do not rely on `pg_is_in_recovery()` alone — during promotion and scale-up the cluster status can read `Running` while Patroni is still converting roles. Wait until **all** of the following hold: + +```bash +# a) Patroni shows a Leader in state running (a timeline increase here is normal): +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list +# b) the CR reports Running: +kubectl -n $TGT_NS get postgresql acid-mig -o jsonpath='{.status.PostgresClusterStatus}{"\n"}' +# c) the new replica is streaming (after scale-up): +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -c \ + "SELECT application_name, state FROM pg_stat_replication;" +``` + +6. **Verify writes and integrity** on the new primary: insert a marker row, rerun the checksum queries, compare with the baseline. + +> A leadership change between the target's pods during promote+scale-up (with an extra timeline bump) is normal operator rolling behavior and does not indicate a problem. + +## Step 6: Post-Migration + +- **Repoint clients** to the target cluster's service (and update any external access such as NodePort/LoadBalancer/ingress used by applications). +- The demoted source is now a live **reverse DR standby** of the target: writes on the target replicate back to it. Choose one: + - **Keep it** as disaster-recovery / rollback insurance (recommended for at least a soak period). Rolling back is the same two-phase switchover in the opposite direction. + - **Dismantle it** and remove the replication setup entirely — see below. +- Scale/tune the target (replica count, resources, backup schedule, monitoring) to match what the source had. + +### Dismantling the Source and Removing the Replication Configuration + +Perform these steps **in order** — the standby must be gone before the primary's replication configuration is removed, otherwise the standby loses its upstream while still streaming. + +**1. Delete the demoted source instance** (on the source cluster): + +```bash +kubectl --context -n $SRC_NS delete postgresql $SRC_CLUSTER +# PVC retention on CR deletion depends on operator configuration — check and +# remove any leftovers to reclaim storage: +kubectl --context -n $SRC_NS get pvc -l cluster-name=$SRC_CLUSTER +kubectl --context -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER --ignore-not-found +``` + +The operator deletes the credential secrets it owns along with the CR. + +**2. Convert the target to a normal (non-replicated) instance** by removing the `clusterReplication` block: + +```bash +kubectl --context -n $TGT_NS patch postgresql acid-mig --type json \ + -p '[{"op":"remove","path":"/spec/clusterReplication"}]' +``` + +The operator handles this transition: it re-applies normal-primary Patroni configuration, drops the `sys_operator` metadata schema from the database, and removes the `xdc_hotstandby` slot from the replication configuration. (The reverse — converting an existing normal instance *into* a standby — is not a supported transition; standbys must be created as standbys.) + +**3. Clean up leftover objects** that the operator does not remove: + +```bash +# Bootstrap secret (user-created, never operator-owned): +kubectl --context -n $TGT_NS delete secret standby-bootstrap-secret + +# The -xcr service lingers after the role change (it is only removed +# with the CR itself) — delete it: +kubectl --context -n $TGT_NS delete svc acid-mig-xcr --ignore-not-found + +# The physical replication slot can also linger even though it was removed +# from the Patroni configuration — drop it if the Step 4 check finds it: +kubectl --context -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -c \ + "SELECT pg_drop_replication_slot('xdc_hotstandby') + WHERE EXISTS (SELECT 1 FROM pg_replication_slots WHERE slot_name='xdc_hotstandby' AND NOT active);" +``` + +**4. Verify the target is a clean standalone instance:** + +```bash +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- psql -U postgres -tA -c \ + "SELECT count(*) FROM pg_replication_slots WHERE slot_name='xdc_hotstandby'; + SELECT count(*) FROM pg_namespace WHERE nspname='sys_operator';" +# Both counts must be 0 +kubectl -n $TGT_NS exec acid-mig-0 -c postgres -- patronictl list +# Expect a plain Leader/Replica cluster, no Standby Leader +``` + +## Alternative: Migration Without Inter-Cluster Connectivity + +When the clusters have **no network path between them** but your workstation can reach both Kubernetes API servers, the migration can be relayed through the workstation as a logical dump/restore piped between two `kubectl exec` sessions — the clusters never talk to each other. Downtime equals the full copy duration (versus seconds for the streaming switchover), but there are no operator-version or CPU-architecture constraints, and the PostgreSQL major only needs to be the same or newer on the target. + +The full validated procedure is a separate solution: [How to Migrate a PostgreSQL Instance Between Network-Isolated Clusters](./How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md) (KB260721001). + +## Troubleshooting + +### Standby fails with `pq: column "external_ip" does not exist` — and runs as an empty independent primary + +**Cause:** operator version mismatch between source and target (e.g. source primary managed by v4.1.x, target operator v4.2+/v4.3). The replication metadata table schema differs between these lines and is never migrated (ECO-703). The standby create hook aborts midway and the pod falls through to a normal bootstrap: it comes up as a *fresh, empty, writable* primary while its CR still says `isReplica: true`. + +**Detection:** compare the cluster identifier in `patronictl list` on both sides — different identifier means independent cluster, not a standby. + +**Fix:** upgrade both operators to the same version (preferred). If the source operator cannot be upgraded immediately, add the missing column on the source primary (safe for both versions — all statements reference columns by name): + +```sql +ALTER TABLE sys_operator.multi_cluster_info ADD COLUMN IF NOT EXISTS external_ip CHAR(64) DEFAULT ''; +``` + +Then recreate the standby cleanly — see the next item. + +### Recreating a failed standby: create hook fails with secret "already exists" + +The operator copies the source's credential secrets into the target namespace *before* bootstrapping, and the copy step fails hard if they already exist from a previous attempt. To retry a standby creation from scratch, delete all of: + +```bash +kubectl -n $TGT_NS delete postgresql acid-mig +kubectl -n $TGT_NS delete pvc -l cluster-name=acid-mig # data from the failed attempt must go +kubectl -n $TGT_NS delete secret \ + postgres.acid-mig.credentials.postgresql.acid.zalan.do \ + standby.acid-mig.credentials.postgresql.acid.zalan.do +``` + +then re-apply the standby manifest. + +### Standby stuck in `creating replica` + +The basebackup is either still copying (large database — check network throughput) or cannot connect. Verify Step 2 reachability *from the node the standby pod landed on*; per-node egress differences are a real failure mode. Also confirm the bootstrap secret contains the current source admin password. + +### Replication slot errors (`TypeError ... 'int' and 'NoneType'`) in standby logs + +Known Patroni issue; replication generally continues. See the [Hot Standby guide's troubleshooting section](./How_to_Use_PostgreSQL_Hot_Standby_Cluster.md#troubleshooting) — drop the `xdc_hotstandby` slot if needed. + +## Verification Checklist + +| Check | When | Pass condition | +|---|---|---| +| Network preflight | before Step 3 | target overlay pod reaches `SRC_NODE_IP:NODEPORT` | +| Operator versions | before Step 3 | identical on both clusters | +| Shared system identifier | after Step 3 | `patronictl list` identifier equal on both sides | +| Streaming | after Step 3 | target `Standby Leader / streaming`; source slot `xdc_hotstandby` active | +| Checksums (pre-cutover) | Step 4 | all values match baseline | +| LSN parity | Step 5, writes stopped | `pg_current_wal_lsn()` == `pg_last_wal_replay_lsn()` | +| Promotion | Step 5 | Patroni Leader running + CR Running + replica streaming | +| Checksums (post-cutover) | Step 5 | all values match baseline; new write on target succeeds | +| Reverse replication (if source kept) | Step 6 | target write appears on demoted source | diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md new file mode 100644 index 000000000..725f20d0f --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_Between_Network_Isolated_Clusters.md @@ -0,0 +1,382 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - 4.x +id: KB260721001 +--- + +# How to Migrate a PostgreSQL Instance Between Network-Isolated Clusters + +## Background + +### The Challenge + +The streaming-replication migration described in the [PostgreSQL Instance Cross-Cluster Migration Guide](./How_to_Migrate_a_PostgreSQL_Instance_Across_Clusters.md) requires the target cluster to reach the source cluster's network. In many real deployments the clusters are network-isolated — separate platforms, firewalled sites, disconnected security zones — and no such path exists or can be opened. + +### The Solution + +When an administrator workstation can reach **both** Kubernetes API servers, the migration can be relayed through the workstation: `pg_dump` runs in the source pod, `pg_restore` runs in the target pod, and the data flows between them through two `kubectl exec` channels piped together on the workstation. The clusters never exchange a packet. + +The procedure is four human operations: create the target instance (Step 1), stop application writes (Step 2), run the migration script (Step 3), and cut over (Steps 4–5). The script migrates **every** application database of the instance and verifies each one. + +Because the transfer is logical (SQL-level), this method has no same-version requirement: it works across different operator versions, across CPU architectures, and from an older PostgreSQL major to the same or a newer major on the target. Migrating to an **older** PostgreSQL major (a downgrade) is not supported. + +**Trade-off:** unlike streaming replication, this is a point-in-time copy. Writes made on the source after the dump starts are not transferred — application writes must be stopped for the whole dump+restore window, so downtime equals the full copy duration. + +## Environment Information + +- PostgreSQL Operator: any 4.x version on each side (versions do **not** need to match) +- PostgreSQL: target major version equal to or newer than the source +- Workstation: `bash` and `kubectl` access (kubeconfig/context) to both clusters + +## Prerequisites + +- Two kubeconfig contexts on the workstation, one per cluster; verify both work: + +```bash +SRC_CTX=""; SRC_NS=""; SRC_CLUSTER="" +TGT_CTX=""; TGT_NS=""; TGT_CLUSTER="" + +kubectl --context $SRC_CTX -n $SRC_NS get postgresql $SRC_CLUSTER +kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER 2>/dev/null || echo "target instance not created yet" +``` + +- Enough workstation bandwidth to both API servers for the database size (all data flows through the workstation). +- A maintenance window sized to the dump+restore duration. +- Sufficient Kubernetes permissions in both namespaces: `get` on `postgresqls.acid.zalan.do`, `pods`, and `secrets`; `create` on `pods/exec`; plus `create` on `postgresqls` (target side) and, for the final cleanup, `delete` on `postgresqls` and `persistentvolumeclaims` (source side). +- The commands assume the operator's standard Spilo image: pod labels `spilo-role`/`cluster-name`, database container named `postgres`, and client binaries for every supported major under `/usr/lib/postgresql//bin/`. Custom or non-Spilo images require adapting these labels and paths. + +## Step 1: Create the Target Instance + +List the application databases and owners the source actually holds: + +```bash +SRC_POD=$(kubectl --context $SRC_CTX -n $SRC_NS get pod \ + -l spilo-role=master,cluster-name=$SRC_CLUSTER -o jsonpath='{.items[0].metadata.name}') + +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -tA -c \ + "SELECT datname, pg_get_userbyid(datdba) AS owner FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' ORDER BY 1;" +``` + +Every CR-managed database in this list must be declared in the target CR (`spec.databases`, with its owner in `spec.users`) — the operator creates the roles and databases and manages their credentials; do not attempt to dump global objects (roles) from the source. The `postgres` maintenance database is managed by the operator/image on each side and is not migrated. Databases owned by `postgres` (created outside the CR spec) do not go into the CR — the migration script creates them on the target automatically. + +The `users:`/`databases:` sections of the target CR can be generated directly from the source instead of hand-written: + +```bash +{ + echo " users:" + kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -tA -c \ + "SELECT DISTINCT ' ' || pg_get_userbyid(datdba) || ': []' FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' AND pg_get_userbyid(datdba) <> 'postgres';" + echo " databases:" + kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -tA -c \ + "SELECT ' ' || datname || ': ' || pg_get_userbyid(datdba) FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' AND pg_get_userbyid(datdba) <> 'postgres' ORDER BY 1;" +} +``` + +### Preserve application credentials + +By default the target operator generates **new** passwords for the CR users, which would force every application to be re-configured at cutover. To keep the source passwords, copy each application user's credential secret into the target namespace **before creating the CR** — when the operator finds an existing credential secret, it adopts it and sets the role's password from it instead of generating a new one: + +```bash +SUFFIX="credentials.postgresql.acid.zalan.do" +for SEC in $(kubectl --context $SRC_CTX -n $SRC_NS get secret \ + -l application=spilo,cluster-name=$SRC_CLUSTER -o name); do + NAME=${SEC#secret/} + U=${NAME%.$SRC_CLUSTER.$SUFFIX} + [ "$U" = "$NAME" ] && continue # not a credential secret + case "$U" in postgres|standby|pooler) continue ;; esac # operator-managed system users + kubectl --context $TGT_CTX -n $TGT_NS apply -f - < +``` + +Wait for status `Running`: + +```bash +kubectl --context $TGT_CTX -n $TGT_NS get postgresql $TGT_CLUSTER \ + -o jsonpath='{.status.PostgresClusterStatus}{"\n"}' +``` + +## Step 2: Stop Writes + +Stop application writes on the source — the transfer is a point-in-time copy, and anything written after the dump starts is lost. + +The migration script (Step 3) records and compares exact per-table row counts automatically. For stronger guarantees on critical tables, additionally record per-table checksums now and re-check them in Step 4: + +```bash +kubectl --context $SRC_CTX -n $SRC_NS exec $SRC_POD -c postgres -- \ + psql -U postgres -d -tA -c \ + "SELECT count(*), sum(hashtext(id::text || payload)) FROM ;" +``` + +## Step 3: Run the Migration Script + +The script performs the whole migration in one run: it enumerates the source's application databases, creates any that are missing on the target (e.g. `postgres`-owned databases created outside the CR spec — their owner role must exist on the target), pre-creates each database's extensions, transfers each database as its owner, and verifies exact per-table row counts — reporting `PASS`/`FAIL` per database and exiting non-zero if anything failed. + +Fill in the six variables at the top, then run it with `bash`. Without arguments it migrates every database; pass database names (`bash migrate.sh appdb`) to migrate only those — used to redo a single database after a failure. Every transfer goes through the same script in one of two modes: + +- **Pipe mode** (default): source streams straight into the target — no intermediate storage. +- **File mode** (`DUMP_DIR=/path bash migrate.sh`): each database is dumped to `$DUMP_DIR/.dump` first, then restored from the file — use this for large databases or unreliable links. By default every run takes a **fresh** dump; add `REUSE_DUMPS=1` to reuse completed dumps (resumable). Reuse is only safe **within the same stopped-write maintenance window** — a dump taken before source writes resumed would silently migrate stale data. + +```bash +#!/usr/bin/env bash +# Whole-instance PostgreSQL migration relayed through the workstation. +# Migrates every application database of $SRC_CLUSTER into $TGT_CLUSTER. +set -u -o pipefail + +SRC_CTX=""; SRC_NS=""; SRC_CLUSTER="" +TGT_CTX=""; TGT_NS=""; TGT_CLUSTER="" + +SRC_POD=$(kubectl --context "$SRC_CTX" -n "$SRC_NS" get pod \ + -l spilo-role=master,cluster-name="$SRC_CLUSTER" -o jsonpath='{.items[0].metadata.name}') \ + && [ -n "$SRC_POD" ] || { echo "ERROR: cannot find source master pod for $SRC_CLUSTER" >&2; exit 1; } +TGT_POD=$(kubectl --context "$TGT_CTX" -n "$TGT_NS" get pod \ + -l spilo-role=master,cluster-name="$TGT_CLUSTER" -o jsonpath='{.items[0].metadata.name}') \ + && [ -n "$TGT_POD" ] || { echo "ERROR: cannot find target master pod for $TGT_CLUSTER" >&2; exit 1; } + +srcsql() { local db=$1; shift; kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- psql -U postgres -d "$db" -tA -v ON_ERROR_STOP=1 "$@"; } +tgtsql() { local db=$1; shift; kubectl --context "$TGT_CTX" -n "$TGT_NS" exec "$TGT_POD" -c postgres -- psql -U postgres -d "$db" -tA -v ON_ERROR_STOP=1 "$@"; } + +# Client binaries matching the SOURCE server major (present in both Spilo images). +SRC_MAJOR=$(srcsql postgres -c "SHOW server_version;" | cut -d. -f1) +case "$SRC_MAJOR" in ''|*[!0-9]*) echo "ERROR: cannot determine source PostgreSQL major (psql via $SRC_POD failed)" >&2; exit 1 ;; esac +PG_BIN=/usr/lib/postgresql/$SRC_MAJOR/bin +echo "Source PostgreSQL major: $SRC_MAJOR (client binaries: $PG_BIN)" + +# Count every user table outside the schemas the dump excludes. +COUNT_QUERY="SELECT schemaname||'.'||relname, (xpath('/row/c/text()', query_to_xml(format( + 'SELECT count(*) AS c FROM %I.%I', schemaname, relname), false, true, '')))[1]::text::bigint + FROM pg_stat_user_tables + WHERE schemaname NOT IN ('metric_helpers','user_management') ORDER BY 1;" + +FAILED=""; ATTEMPTED=0 +while read -r DB OWNER; do + [ -z "$DB" ] && continue + # With arguments, migrate only the named databases (e.g. to redo one FAIL). + if [ "$#" -gt 0 ]; then + case " $* " in *" $DB "*) ;; *) continue ;; esac + fi + ATTEMPTED=$((ATTEMPTED+1)) + echo "=== migrating database: $DB (owner: $OWNER) ===" + + # Ensure the database exists on the target (covers databases created + # outside the CR spec; the owner role must already exist on the target). + if [ "$(tgtsql postgres -c "SELECT 1 FROM pg_database WHERE datname = '$DB';")" != "1" ]; then + echo " creating database $DB on target" + tgtsql postgres -c "CREATE DATABASE \"$DB\" OWNER \"$OWNER\";" >/dev/null || { FAILED="$FAILED $DB(create)"; continue; } + fi + + # Pre-create the source's extensions (extension creation needs superuser). + for EXT in $(srcsql "$DB" -c "SELECT extname FROM pg_extension;"); do + tgtsql "$DB" -c "CREATE EXTENSION IF NOT EXISTS \"$EXT\";" >/dev/null 2>&1 \ + || echo " WARNING: could not create extension $EXT on target — restore may fail" + done + + # Baseline on the source (exact counts). + SRC_COUNTS=$(srcsql "$DB" -c "$COUNT_QUERY") + + # Restore as the database's owner, with the target-managed password. + OWNER_PW=$(kubectl --context "$TGT_CTX" -n "$TGT_NS" get secret \ + "$OWNER.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do" \ + -o jsonpath='{.data.password}' | base64 -d) || { FAILED="$FAILED $DB(secret)"; continue; } + + # Transfer: straight pipe by default; with DUMP_DIR set, dump to a file + # first and restore from it. REUSE_DUMPS=1 reuses completed dumps — only + # safe while source writes have stayed stopped since the dump was taken. + if [ -n "${DUMP_DIR:-}" ]; then + mkdir -p "$DUMP_DIR" + DUMP_FILE="$DUMP_DIR/$DB.dump" + if [ -n "${REUSE_DUMPS:-}" ] && [ -s "$DUMP_FILE" ]; then + echo " reusing existing dump $DUMP_FILE" + else + kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- \ + "$PG_BIN/pg_dump" -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management "$DB" > "$DUMP_FILE.partial" \ + && mv "$DUMP_FILE.partial" "$DUMP_FILE" \ + || { rm -f "$DUMP_FILE.partial"; FAILED="$FAILED $DB(dump)"; continue; } + fi + kubectl --context "$TGT_CTX" -n "$TGT_NS" exec -i "$TGT_POD" -c postgres -- \ + env PGPASSWORD="$OWNER_PW" \ + "$PG_BIN/pg_restore" -U "$OWNER" -h localhost -d "$DB" --no-owner -x < "$DUMP_FILE" + RC=$? + else + kubectl --context "$SRC_CTX" -n "$SRC_NS" exec "$SRC_POD" -c postgres -- \ + "$PG_BIN/pg_dump" -U postgres -Fc --no-comments \ + -N metric_helpers -N user_management "$DB" \ + | kubectl --context "$TGT_CTX" -n "$TGT_NS" exec -i "$TGT_POD" -c postgres -- \ + env PGPASSWORD="$OWNER_PW" \ + "$PG_BIN/pg_restore" -U "$OWNER" -h localhost -d "$DB" --no-owner -x + RC=$? + fi + + # Verify: exact counts must match the baseline. + TGT_COUNTS=$(tgtsql "$DB" -c "$COUNT_QUERY") + if [ "$RC" -eq 0 ] && [ "$SRC_COUNTS" = "$TGT_COUNTS" ]; then + echo " PASS: $DB (tables/rows identical)" + else + echo " FAIL: $DB (transfer exit $RC; counts $( [ "$SRC_COUNTS" = "$TGT_COUNTS" ] && echo match || echo DIFFER ))" + FAILED="$FAILED $DB" + fi +done < <(srcsql postgres -F' ' -c \ + "SELECT datname, pg_get_userbyid(datdba) FROM pg_database + WHERE NOT datistemplate AND datname <> 'postgres' ORDER BY 1;") + +echo +if [ "$ATTEMPTED" -eq 0 ]; then + echo "ERROR: no databases migrated — source enumeration failed, or no database matched the arguments" >&2 + exit 1 +fi +if [ -n "$FAILED" ]; then echo "MIGRATION INCOMPLETE — failed:$FAILED"; exit 1; fi +echo "MIGRATION COMPLETE — $ATTEMPTED database(s) verified." +``` + +If a database reports `FAIL`, see [Troubleshooting](#troubleshooting); after fixing the cause, redo just that database by passing its name to the script (in file mode, add `REUSE_DUMPS=1` to skip re-dumping if writes have stayed stopped). + +### Security notes + +- While a restore runs, the owner password is expanded into the workstation's `kubectl` argument list and is visible in local process listings (`ps`). Run the migration from a trusted, single-user workstation, and do not enable shell tracing (`set -x`) around these commands. +- In file mode, `$DUMP_DIR` holds full plaintext logical copies of the databases. Point it at a directory with restrictive permissions (`umask 077` before running, or `chmod 700` on the directory), encrypt at rest if your policy requires, and delete the dumps once the migration is verified. +- The data itself only traverses the two TLS-protected `kubectl exec` channels; nothing is exposed on the network beyond the two Kubernetes API connections. + +## Step 4: Verify Application Access + +The script has already verified per-table row counts. Confirm in addition that each application user can actually query its data with the **target**-managed credentials (this catches ownership problems immediately), and re-check any Step 2 checksums. If you pre-staged the credential secrets in Step 1, this password is identical to the source one — applications keep their existing credentials and only the connection endpoint changes at cutover: + +```bash +TGT_POD=$(kubectl --context $TGT_CTX -n $TGT_NS get pod \ + -l spilo-role=master,cluster-name=$TGT_CLUSTER -o jsonpath='{.items[0].metadata.name}') + +APP_PW=$(kubectl --context $TGT_CTX -n $TGT_NS get secret \ + .$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ + -o jsonpath='{.data.password}' | base64 -d) + +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- \ + env PGPASSWORD="$APP_PW" psql -U -h localhost -d -tA -c \ + "SELECT current_user, count(*) FROM ;" +``` + +The dump deliberately skips ACL statements (`-x`) because they reference roles managed by the source operator — if additional (non-owner) users need privileges on the migrated data, re-grant them on the target now. + +## Step 5: Cut Over and Clean Up + +- Repoint applications to the target instance's service and re-enable writes. +- Delete the source instance when satisfied. There is no replication configuration to tear down: + +```bash +kubectl --context $SRC_CTX -n $SRC_NS delete postgresql $SRC_CLUSTER +# PVC retention depends on operator configuration — remove leftovers: +kubectl --context $SRC_CTX -n $SRC_NS delete pvc -l cluster-name=$SRC_CLUSTER --ignore-not-found +``` + +## Troubleshooting + +### Application user gets `permission denied for table ...` after a successful restore + +The restore was run as `postgres` instead of the application user, so all objects are owned by `postgres` (the script always restores as the owner; this typically comes from a hand-run restore). Either redo the database — drop and recreate it, then rerun the script with the database name — or transfer ownership in place: + +```sql +DO $$ +DECLARE r record; +BEGIN + FOR r IN SELECT tablename FROM pg_tables WHERE schemaname = 'public' LOOP + EXECUTE format('ALTER TABLE public.%I OWNER TO appuser', r.tablename); + END LOOP; + FOR r IN SELECT sequencename FROM pg_sequences WHERE schemaname = 'public' LOOP + EXECUTE format('ALTER SEQUENCE public.%I OWNER TO appuser', r.sequencename); + END LOOP; +END $$; +``` + +This block covers tables and sequences in the `public` schema only. Views, functions, types, and objects in other schemas need analogous `ALTER ... OWNER TO` statements (enumerate them via `pg_views`, `pg_proc`, `pg_type`, or `\dn`/`\df` in psql). + +Related scope note: the migration's `--no-owner` model assumes the operator's standard layout — one database wholly owned by one CR-defined user. If a database has multiple owning roles, `SECURITY DEFINER` functions, or objects in non-`public` schemas with distinct owners, ownership collapses onto the restoring user; plan an explicit post-restore ownership and grant pass for those objects. + +### Restore reports `permission denied to create extension ...` + +The source database uses an extension that is not pre-installed on the target, and the application user cannot create it. The script warns about this (`WARNING: could not create extension`). Create the extension as `postgres` on the target database, then rerun the restore. If `CREATE EXTENSION` itself fails with a missing-file error, the extension's packages are not present in the target image at all — it must be added to the image/instance before this migration can carry that database. + +### Restore reports `must be owner of extension pg_stat_statements` (and similar) + +`--no-comments` was omitted from `pg_dump`; the extension comments require superuser. The data restores fine — the errors only break the clean exit code. Rerun with `--no-comments` for a verifiable result. + +### Restore reports `unrecognized configuration parameter "transaction_timeout"` + +The dump was taken with a `pg_dump` newer than the target server (typically the image's default binary — a hand-run dump without the explicit `PG_BIN` path; the script auto-detects it from `SHOW server_version`). Rerun with the version-matched binary path. When migrating to a *newer* target major, use the **target** major's binaries instead — they are present in the source pod's image too. + +### Restore reports `schema "metric_helpers" already exists` (and similar) + +The `-N metric_helpers -N user_management` exclusions were omitted. These errors are harmless for the excluded schemas' objects, but rerun with the exclusions for a clean, verifiable exit code. + +### Rerunning after a failed restore + +Drop the target database first — a partial restore leaves objects that collide (`relation already exists`, duplicate-key COPY failures). Note `DROP DATABASE` must be executed as its own single statement: + +```bash +kubectl --context $TGT_CTX -n $TGT_NS exec $TGT_POD -c postgres -- psql -U postgres -c "DROP DATABASE appdb" +``` + +Then rerun the script with **only the failed database** as an argument (`bash migrate.sh `) — it recreates the database with the correct owner; in file mode, add `REUSE_DUMPS=1` to restore from the completed dump instead of re-dumping (only if source writes have stayed stopped). Do not rerun it without arguments after a partial success: restoring into the databases that already transferred produces `already exists` collisions that mark them as failed. + +### Application passwords changed after the migration + +The target CR was created **before** the credential secrets were pre-staged (Step 1), so the operator generated new passwords. To restore the source credentials after the fact, update both the secret and the role together — the secret alone is not enough, and an `ALTER ROLE` alone would be reverted by the operator's next sync from the secret. The commands below stay safe for passwords containing quotes, backslashes, or JSON metacharacters: the secret is patched with the base64 value (JSON-safe by construction), and the SQL uses psql's `:'pw'` literal quoting instead of string interpolation: + +```bash +PW_B64=$(kubectl --context $SRC_CTX -n $SRC_NS get secret \ + appuser.$SRC_CLUSTER.credentials.postgresql.acid.zalan.do -o jsonpath='{.data.password}') + +kubectl --context $TGT_CTX -n $TGT_NS patch secret \ + appuser.$TGT_CLUSTER.credentials.postgresql.acid.zalan.do \ + --type merge -p "{\"data\":{\"password\":\"$PW_B64\"}}" + +kubectl --context $TGT_CTX -n $TGT_NS exec -i $TGT_POD -c postgres -- \ + psql -U postgres -v pw="$(printf %s "$PW_B64" | base64 -d)" <<'SQL' +ALTER ROLE appuser PASSWORD :'pw'; +SQL +``` + +Verify with a login test as in Step 4. + +### Pipe is slow + +Throughput is bounded by the workstation's link to both API servers (every byte traverses it twice: exec stream in, exec stream out). Run the migration from a machine with good connectivity to both platforms (e.g. a jump host), and use file mode (`DUMP_DIR`) so progress is resumable. If a very large restore needs parallelism, note `pg_restore -j N` cannot read from stdin — copy the dump file into the target pod (`kubectl cp`) and restore from the local path there. diff --git a/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md new file mode 100644 index 000000000..efb7e789f --- /dev/null +++ b/docs/en/solutions/How_to_Migrate_a_PostgreSQL_Instance_to_Another_Node.md @@ -0,0 +1,134 @@ +--- +kind: + - Solution +products: + - Alauda Application Services +ProductsVersion: + - 4.0,4.1,4.2,4.3 +id: KB260515007 +--- + +# How to Migrate a PostgreSQL Instance to Another Node + +## Issue + +A PostgreSQL cluster managed by the PostgreSQL Operator uses node-local storage +(for example TopoLVM). One or more instances must be moved off their current +node — because the node is being decommissioned, or an instance must run on a +specific compute node. Because the data lives in a node-local PersistentVolume, +the pod cannot simply be rescheduled; the volume is pinned to its node. + +## Environment + +- Alauda Application Services PostgreSQL Operator (Zalando-based, + `acid.zalan.do/v1` `postgresql` resource). +- Node-local storage such as TopoLVM (each PVC is bound to one node). +- At least one target node with enough free capacity. Because migration + re-clones data, the target node should have capacity for roughly **twice** the + instance's PVC size during the transition. + +## Resolution + +The migration relies on a property of the Operator: when an instance's PVC and +pod are deleted, the StatefulSet recreates the pod, a fresh PVC is provisioned +wherever the pod is scheduled, and Patroni re-clones the data from the current +leader. Data is preserved through streaming replication, not by moving the +volume. + +> Validated on ACP 4.2 and 4.3: after deleting a replica's PVC and pod, the +> member was recreated on a node, re-synced from the leader, and previously +> written rows were present on the resynced member. + +In the examples below set `$NAMESPACE` and `$CLUSTER_NAME` for the target +cluster. Replace placeholder node names with your own. + +### 1. Confirm the current placement + +```bash +kubectl get pod -n $NAMESPACE -o wide | grep $CLUSTER_NAME +kubectl get pvc -n $NAMESPACE -o wide | grep $CLUSTER_NAME +``` + +Note which member is the leader (do not delete the leader's volume first): + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- patronictl list +``` + +### 2. Restrict scheduling to the target node(s) + +So the recreated pod lands only on a desired node, cordon the other eligible +nodes (or use a `nodeSelector`/label that only the target nodes carry). Keep at +least the target node schedulable. + +```bash +# Label ONLY the target node(s) so the recreated pod can land there and nowhere else +kubectl label node target=true --overwrite +``` + +Do **not** label the source node — labeling both source and target would let the +pod reschedule back onto the source. If you prefer cordoning over labels, cordon +all non-target nodes instead and skip the `nodeSelector` step below. + +If you use a label-based selector, set it on the instance: + +```bash +kubectl patch postgresql -n $NAMESPACE $CLUSTER_NAME --type merge \ + -p '{"spec":{"nodeSelector":{"target":"true"}}}' +``` + +### 3. Migrate one member at a time + +Always migrate a **non-leader** member first. Delete its PVC and pod together — +the PVC deletion blocks until the pod that mounts it is gone, so delete the pod +in parallel: + +```bash +# Delete the data PVC (it will stay in Terminating until the pod is gone) +kubectl delete pvc pgdata-$CLUSTER_NAME-1 -n $NAMESPACE --wait=false + +# Delete the pod to release the PVC +kubectl delete pod $CLUSTER_NAME-1 -n $NAMESPACE +``` + +The StatefulSet recreates `$CLUSTER_NAME-1`; a new PVC is provisioned on the +scheduled node and Patroni re-clones from the leader. + +### 4. Verify the member rejoined with its data + +```bash +kubectl get pod -n $NAMESPACE -o wide | grep $CLUSTER_NAME-1 +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- patronictl list +``` + +The migrated member should return to role `Replica`, state `running`/`streaming` +with `Lag in MB` `0`. Spot-check data on the member (it is read-only / in +recovery): + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-1 -c postgres -- \ + psql -U postgres -tAc "SELECT pg_is_in_recovery();" # expect t +``` + +### 5. Migrate the (former) leader if required + +To move the leader, first perform a switchover so another member becomes leader, +then repeat steps 3–4 for the old leader: + +```bash +kubectl exec -n $NAMESPACE $CLUSTER_NAME-0 -c postgres -- \ + patronictl switchover $CLUSTER_NAME --force +``` + +### 6. Restore scheduling + +Uncordon any nodes you cordoned and remove temporary labels/`nodeSelector` once +all members are on their intended nodes. + +## Notes + +- Migrate members one at a time and wait for each to fully re-sync (`Lag = 0`) + before moving the next, so the cluster always retains a healthy quorum. +- For a single-instance cluster, temporarily scale to two instances, let the new + member sync on the target node, switch over, then scale back to one — this + avoids downtime that a delete-and-reclone of the sole instance would cause.