diff --git a/CHANGELOG.md b/CHANGELOG.md index e08dd1a49..5665ab606 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,11 +4,13 @@ ### Security - **Bumped CloudNative-PG dependency from chart 0.27.0 (app 1.28.0) to 0.28.1 (app 1.29.1)** to pick up the fix for [CVE-2026-44477 / GHSA-423p-g724-fr39](https://github.com/cloudnative-pg/cloudnative-pg/security/advisories/GHSA-423p-g724-fr39): a privilege-escalation vulnerability in the CNPG metrics exporter that could allow a low-privilege PostgreSQL user to escalate to superuser and execute arbitrary commands in the database pod. Operators upgrading via `helm upgrade` will get the patched CNPG operator automatically. +- **Hardened PostgreSQL `pg_hba` rules**: The operator no longer emits the permissive `host all all 0.0.0.0/0 trust` / `host all all ::0/0 trust` rules that allowed passwordless PostgreSQL connections from any address. Non-replication access is now limited to `host all all localhost trust` (the gateway sidecar, which connects over the shared pod loopback), and cross-instance/cross-region replication uses `hostssl replication streaming_replica all cert`. Clients that previously relied on unauthenticated direct PostgreSQL access over the pod network will be rejected after upgrade; use the DocumentDB gateway (SCRAM-SHA-256) for application traffic. In multi-region deployments where `spec.tls.postgres` is not set, replication falls back to `host replication streaming_replica all trust`, which relies entirely on network-layer security (e.g. an Istio/service-mesh mTLS boundary) — provide `spec.tls.postgres` certificates to require TLS client certificates for replication instead. ### Major Features - **io_uring opt-in feature gate**: A new `IOUring` feature gate (`spec.featureGates.IOUring: true`) enables PostgreSQL 18 asynchronous I/O (`io_method=io_uring`). Because the container runtime's default seccomp profile strips the `io_uring_setup/enter/register` syscalls, the operator also relaxes the postgres container seccomp profile — pointing the pods at a hardened Localhost profile that re-allows only those three syscalls — when the gate is on, so no external Kyverno policy is needed. It is **opt-in** (default off) since io_uring relaxes the sandbox. The Localhost profile path is operator-level config via the Helm value `operator.ioUring.seccompProfile` (default `profiles/documentdb-iouring.json`). See [io_uring documentation](docs/operator-public-documentation/io-uring.md) and the [feature playground](documentdb-playground/io-uring-feature/). - **Gateway OTLP metrics in the per-pod sidecar**: when `spec.monitoring.enabled=true`, the OTel Collector sidecar now exposes an OTLP/gRPC receiver on `127.0.0.1:4317` and the documentdb-gateway is configured (via `OTEL_EXPORTER_OTLP_ENDPOINT` and `OTEL_METRICS_ENABLED`) to push its `db_client_*` metrics there. The sidecar's existing prometheus exporter re-exports them alongside the existing `documentdb.postgres.up` sqlquery output, with per-pod attribution added by the collector's resource processor. No new CRD fields; this turns on automatically wherever monitoring was already enabled. - **Two-Phase Extension Upgrade**: New `spec.schemaVersion` field separates binary upgrades (`spec.documentDBVersion`) from irreversible schema migrations (`ALTER EXTENSION UPDATE`). The default behavior gives you a rollback-safe window — update the binary first, validate, then finalize the schema. Set `schemaVersion: "auto"` for single-step upgrades in development environments. See the [upgrade guide](docs/operator-public-documentation/preview/operations/upgrades.md) for details. +- **PostgreSQL/replication TLS via `spec.tls.postgres`**: A new `spec.tls.postgres` field (backed by CloudNative-PG's `CertificatesConfiguration`) lets you supply your own CA and certificate Secrets for PostgreSQL server and replication connections instead of the CloudNative-PG self-signed defaults. Supported keys are `serverTLSSecret` + `serverCASecret` (server certificate) and `replicationTLSSecret` + `clientCASecret` (the `streaming_replica` client certificate). A CEL validation rule enforces the pairing invariants: server and replication secrets must each be provided together, and `serverTLSSecret` requires `replicationTLSSecret`. When client certificates are provided, cross-region `externalClusters` connect with `sslmode=require` (or `verify-full` when a server CA is present) and present the replication client certificate. See the [TLS configuration guide](docs/operator-public-documentation/preview/configuration/tls.md). ### Breaking Changes - **CRD restructure into domain-grouped stanzas**: image, postgres and plugin fields have moved into dedicated groups. Migrate as follows: `spec.documentDBImage` → `spec.image.documentDB`, `spec.gatewayImage` → `spec.image.gateway`, `spec.postgresImage` → `spec.image.postgres`, `spec.sidecarInjectorPluginName` → `spec.plugins.sidecarInjectorName`. A new `spec.postgres` group exposes `uid`, `gid` and `postInitSQL` (the operator's mandatory bootstrap statements always run first; user statements are appended after). A new root-level `spec.imagePullSecrets` is propagated to the underlying CNPG cluster. diff --git a/docs/crd-ref-docs-config.yaml b/docs/crd-ref-docs-config.yaml index 6e5f11392..414bb8929 100644 --- a/docs/crd-ref-docs-config.yaml +++ b/docs/crd-ref-docs-config.yaml @@ -22,3 +22,6 @@ render: - name: AffinityConfiguration package: github.com/cloudnative-pg/cloudnative-pg/api/v1 link: https://pkg.go.dev/github.com/cloudnative-pg/cloudnative-pg/api/v1#AffinityConfiguration + - name: CertificatesConfiguration + package: github.com/cloudnative-pg/cloudnative-pg/api/v1 + link: https://pkg.go.dev/github.com/cloudnative-pg/cloudnative-pg/api/v1#CertificatesConfiguration diff --git a/docs/operator-public-documentation/preview/api-reference.md b/docs/operator-public-documentation/preview/api-reference.md index f3c900a06..ef63617df 100644 --- a/docs/operator-public-documentation/preview/api-reference.md +++ b/docs/operator-public-documentation/preview/api-reference.md @@ -376,19 +376,7 @@ _Appears in:_ | `uid` _integer_ | UID is the numeric user ID under which the PostgreSQL server process runs.
When set, GID must also be set. | | Optional: \{\}
| | `gid` _integer_ | GID is the numeric group ID under which the PostgreSQL server process runs.
When set, UID must also be set. | | Optional: \{\}
| | `postInitSQL` _string array_ | PostInitSQL is an ordered list of SQL statements executed after the
cluster is initialized. These statements run AFTER the operator's
mandatory bootstrap (CREATE EXTENSION documentdb, CREATE ROLE
documentdb, ALTER ROLE documentdb), so they can safely reference the
documentdb extension and role. | | Optional: \{\}
| - - -#### PostgresTLS - - - -PostgresTLS acts as a placeholder for future Postgres TLS settings. - - - -_Appears in:_ -- [TLSConfiguration](#tlsconfiguration) - +| `parameters` _object (keys:string, values:string)_ | Parameters allows users to override PostgreSQL configuration parameters
(postgresql.conf settings) passed through to the underlying CNPG Cluster.
The operator applies memory-aware defaults (shared_buffers, effective_cache_size,
work_mem, maintenance_work_mem) computed from the pod memory limit, plus static
best-practice defaults for autovacuum, IO, WAL, and connection settings.
Values specified here override computed and static defaults.
Protected parameters (cron.database_name, max_replication_slots, max_wal_senders,
max_prepared_transactions) cannot be overridden. | | Optional: \{\}
| #### PrometheusExporterSpec @@ -454,6 +442,8 @@ _Appears in:_ | Field | Description | Default | Validation | | --- | --- | --- | --- | | `storage` _[StorageConfiguration](#storageconfiguration)_ | Storage configuration for DocumentDB persistent volumes. | | | +| `memory` _string_ | Memory specifies the memory limit for each DocumentDB instance pod.
This value is passed to the CNPG Cluster's spec.resources.limits.memory
and spec.resources.requests.memory (Guaranteed QoS).
Memory-aware PostgreSQL parameters (shared_buffers, effective_cache_size, etc.)
are auto-computed from this value.
If not specified or set to "0", no memory limit is applied and static
defaults are used for memory-aware parameters.
Examples: "2Gi", "4Gi", "8Gi" | | Optional: \{\}
| +| `cpu` _string_ | CPU specifies the CPU limit for each DocumentDB instance pod.
This value is passed to the CNPG Cluster's spec.resources.limits.cpu
and spec.resources.requests.cpu (Guaranteed QoS).
If not specified or set to "0", no CPU limit is applied.
Examples: "2", "4", "500m" | | Optional: \{\}
| #### ScheduledBackup @@ -524,7 +514,7 @@ _Appears in:_ | Field | Description | Default | Validation | | --- | --- | --- | --- | | `gateway` _[GatewayTLS](#gatewaytls)_ | Gateway configures TLS for the gateway sidecar (Phase 1: certificate provisioning only). | | | -| `postgres` _[PostgresTLS](#postgrestls)_ | Postgres configures TLS for the Postgres server (placeholder for future phases). | | | +| `postgres` _[CertificatesConfiguration](https://pkg.go.dev/github.com/cloudnative-pg/cloudnative-pg/api/v1#CertificatesConfiguration)_ | Postgres configures TLS for the Postgres server. | | | | `globalEndpoints` _[GlobalEndpointsTLS](#globalendpointstls)_ | GlobalEndpoints configures TLS for global endpoints (placeholder for future phases). | | | diff --git a/docs/operator-public-documentation/preview/architecture/overview.md b/docs/operator-public-documentation/preview/architecture/overview.md index 2c0e16a23..d8f037741 100644 --- a/docs/operator-public-documentation/preview/architecture/overview.md +++ b/docs/operator-public-documentation/preview/architecture/overview.md @@ -309,7 +309,7 @@ flowchart LR NEW_PVC --> NEW_CLUSTER ``` -For backup configuration, see [Backup and Restore](../backup-and-restore.md). +For backup configuration, see [Backup and Restore](../operations/backup-and-restore.md). ## Dependencies diff --git a/docs/operator-public-documentation/preview/configuration/tls.md b/docs/operator-public-documentation/preview/configuration/tls.md index 0cd970478..9a1e2d32b 100644 --- a/docs/operator-public-documentation/preview/configuration/tls.md +++ b/docs/operator-public-documentation/preview/configuration/tls.md @@ -1,13 +1,13 @@ --- -title: TLS Configuration -description: Configure TLS certificate management for DocumentDB gateway connections — SelfSigned, CertManager, and Provided modes, private CA guidance, certificate rotation, and troubleshooting. +title: TLS configuration +description: Configure TLS certificate management for DocumentDB gateway and PostgreSQL connections, including SelfSigned, CertManager, and Provided modes, private CA guidance, certificate rotation, and troubleshooting. tags: - configuration - tls - security --- -# TLS Configuration +# TLS configuration !!! warning "Breaking change in this release" The `Disabled` TLS mode has been removed. If you previously set `spec.tls.gateway.mode: Disabled`, update it to `SelfSigned` (or remove the field — `SelfSigned` is now the default). See the [CHANGELOG](https://github.com/documentdb/documentdb-kubernetes-operator/blob/main/CHANGELOG.md) for details. @@ -259,6 +259,94 @@ Example TLS status output: } ``` +## PostgreSQL certificates + +The `spec.tls.gateway` settings above secure client connections to the DocumentDB gateway. A separate field, `spec.tls.postgres`, configures the certificates that CloudNative-PG uses for PostgreSQL server and replication connections. + +In a single-region deployment, CloudNative-PG provisions self-signed certificates automatically. You don't need to set `spec.tls.postgres` unless you want PostgreSQL inter-pod, intra-Kubernetes-cluster connections to use certificate material from your own CA instead of the CloudNative-PG generated self-signed certificates. + +When you provide PostgreSQL certificate Secrets, the operator passes them to the underlying CloudNative-PG `Cluster`. This changes the certificates used between DocumentDB instance pods inside the same Kubernetes cluster; it doesn't change how clients connect to the DocumentDB gateway. + +```yaml title="documentdb-postgres-provided-certs.yaml" +apiVersion: documentdb.io/preview +kind: DocumentDB +metadata: + name: my-documentdb + namespace: default +spec: + nodeCount: 1 + instancesPerNode: 3 + resource: + storage: + pvcSize: 100Gi + tls: + postgres: + replicationTLSSecret: postgres-replication-cert + clientCASecret: postgres-replication-cert + serverTLSSecret: postgres-server-cert + serverCASecret: postgres-server-cert +``` + +!!! note + `replicationTLSSecret` and `clientCASecret` must be provided together. `serverTLSSecret` and `serverCASecret` must be provided together, and `serverTLSSecret` requires `replicationTLSSecret`. + +### Replication client certificate name + +The PostgreSQL replication client certificate referenced by `replicationTLSSecret` must authenticate as the `streaming_replica` PostgreSQL role. The Kubernetes Secret name can be any name that you reference from `spec.tls.postgres.replicationTLSSecret`, but the certificate identity must use `streaming_replica` as the common name. + +If you create the client certificate with cert-manager, set `spec.commonName` to `streaming_replica` and include the `client auth` usage: + +```yaml title="postgres-replication-certificate.yaml" +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: postgres-replication-cert + namespace: default +spec: + secretName: postgres-replication-cert + usages: + - client auth + commonName: streaming_replica + issuerRef: + name: my-ca-issuer + kind: Issuer + group: cert-manager.io +``` + +### Server certificate SANs + +The PostgreSQL server certificate referenced by `serverTLSSecret` must include the CloudNative-PG read-write Service names for the DocumentDB object as Subject Alternative Names (SANs). In a single-region deployment, the operator uses the DocumentDB object name as the CloudNative-PG cluster name. + +For a DocumentDB object named `my-documentdb` in the `default` namespace, include all three service-name forms: + +- `my-documentdb-rw` +- `my-documentdb-rw.default` +- `my-documentdb-rw.default.svc` + +If you create the server certificate with cert-manager, put these names in `spec.dnsNames`: + +```yaml title="postgres-server-certificate.yaml" +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: postgres-server-cert + namespace: default +spec: + secretName: postgres-server-cert + usages: + - server auth + dnsNames: + - my-documentdb-rw + - my-documentdb-rw.default + - my-documentdb-rw.default.svc + issuerRef: + name: my-ca-issuer + kind: Issuer + group: cert-manager.io +``` + +For cross-Kubernetes-cluster replication, see [Replication TLS (PostgreSQL)](../multi-region-deployment/setup.md#replication-tls-postgresql). + ## Additional resources The [`documentdb-playground/tls/`](https://github.com/documentdb/documentdb-kubernetes-operator/tree/main/documentdb-playground/tls) directory provides automated scripts and end-to-end guides for TLS setup on AKS: diff --git a/docs/operator-public-documentation/preview/getting-started/deploy-on-aks.md b/docs/operator-public-documentation/preview/getting-started/deploy-on-aks.md index 8a872280d..d843e361f 100644 --- a/docs/operator-public-documentation/preview/getting-started/deploy-on-aks.md +++ b/docs/operator-public-documentation/preview/getting-started/deploy-on-aks.md @@ -72,7 +72,7 @@ For available classes, see - [AKS storage classes](https://learn.microsoft.com/azure/aks/concepts-storage#storage-classes) - [Azure Disk CSI driver on AKS](https://learn.microsoft.com/azure/aks/azure-disk-csi) -- [DocumentDB storage configuration](../advanced-configuration/README.md#storage-configuration) +- [DocumentDB storage configuration](../configuration/storage.md) ## Monitoring and troubleshooting @@ -111,7 +111,7 @@ kubectl get pods -n kube-system | grep csi-azuredisk enabled - [Encryption at rest](https://learn.microsoft.com/azure/aks/enable-host-encryption) on managed disks -- [TLS configuration](../advanced-configuration/README.md#tls-configuration) +- [TLS configuration](../configuration/tls.md) for database traffic - [Azure RBAC integration](https://learn.microsoft.com/azure/aks/manage-azure-rbac) diff --git a/docs/operator-public-documentation/preview/multi-region-deployment/overview.md b/docs/operator-public-documentation/preview/multi-region-deployment/overview.md index 341b3e7df..ebbdfe8b6 100644 --- a/docs/operator-public-documentation/preview/multi-region-deployment/overview.md +++ b/docs/operator-public-documentation/preview/multi-region-deployment/overview.md @@ -156,7 +156,14 @@ an equal or greater volume of available storage compared to the primary. Enable TLS for all connections: - **Client-to-gateway:** Encrypt application connections (see [TLS configuration](../configuration/tls.md)) -- **Replication traffic:** PostgreSQL SSL for inter-cluster replication +- **Replication traffic:** Cross-Kubernetes-cluster streaming replication supports + two TLS paths. Path 1 uses `spec.tls.postgres.replicationTLSSecret` and + `clientCASecret` with `sslmode=require`. + Path 2 adds `serverTLSSecret` and `serverCASecret` and uses + `sslmode=verify-full` for server identity verification and mutual TLS (mTLS). + Configure shared certificate Secrets across all member Kubernetes clusters. + See [Replication TLS (PostgreSQL)](setup.md#replication-tls-postgresql) for + the complete setup. - **Service mesh:** mTLS for cross-cluster service communication ### Authentication and authorization diff --git a/docs/operator-public-documentation/preview/multi-region-deployment/setup.md b/docs/operator-public-documentation/preview/multi-region-deployment/setup.md index 5754ac235..f81391840 100644 --- a/docs/operator-public-documentation/preview/multi-region-deployment/setup.md +++ b/docs/operator-public-documentation/preview/multi-region-deployment/setup.md @@ -153,6 +153,81 @@ spec: - name: member-westus3-cluster ``` +#### Replication TLS (PostgreSQL) + +Cross-Kubernetes-cluster streaming replication flows over the network between +member Kubernetes clusters. Set `spec.tls.postgres` to encrypt this traffic with +TLS. To enable mutual TLS (mTLS), include the full server verification fields so +replicas both present a client certificate and verify the server certificate +(`sslmode=verify-full`). Distribute the same certificate material to every member +Kubernetes cluster so that any member Kubernetes cluster can become primary after +failover. + +When you set `spec.tls.postgres.replicationTLSSecret`, the operator configures +each PostgreSQL replication connection to use the `streaming_replica` client +certificate. + +The supported configuration paths are: + +| Configuration path | Fields | Operator behavior | +| --- | --- | --- | +| Client certificate only | `spec.tls.postgres.replicationTLSSecret`, `spec.tls.postgres.clientCASecret` | References the `streaming_replica` client certificate and key and sets `sslmode=require`. | +| Full server verification | `spec.tls.postgres.replicationTLSSecret`, `spec.tls.postgres.clientCASecret`, `spec.tls.postgres.serverTLSSecret`, `spec.tls.postgres.serverCASecret` | References the `streaming_replica` client certificate and key, references `serverCASecret` as `sslRootCert`, and sets `sslmode=verify-full`. | + +!!! important "Use an explicit PostgreSQL TLS configuration for encrypted replication" + If `spec.tls.postgres` is omitted, the operator assumes another layer, such + as Istio mTLS, secures the cross-Kubernetes-cluster path. In that mode, the + generated PostgreSQL replication configuration doesn't set `sslmode`, + `sslCert`, `sslKey`, or `sslRootCert`. The `pg_hba` will also be set to + trust all incoming replication connections. + +For `verify-full`, the server certificate SANs must cover every host name that a +replica might use for the primary. Include the generated service names for each +member Kubernetes cluster's PostgreSQL backend and, when using Azure Fleet +Networking, the fleet service DNS name pattern: + +In multi-region mode, the CloudNative-PG cluster name isn't the raw DocumentDB +object name. The operator generates one CloudNative-PG cluster name per member +Kubernetes cluster by taking the DocumentDB object name, truncating it, and +appending an FNV-1a hash of the member Kubernetes cluster name. The AKS Fleet +playground mirrors this logic in +[`generate_cnpg_cluster_name`](https://github.com/documentdb/documentdb-kubernetes-operator/blob/main/documentdb-playground/aks-fleet-deployment/deploy-multi-region.sh) +so `deploy-multi-region.sh` can precompute the `*-rw` service names that must be +included in the PostgreSQL server certificate SAN list. + + +```yaml title="postgres-server-certificate.yaml" +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: cross-region-server-cert + namespace: documentdb-preview-ns +spec: + secretName: cross-region-server-cert + usages: + - server auth + dnsNames: + - --rw.documentdb-preview-ns.svc + - --rw.documentdb-preview-ns + - --rw + - --rw.documentdb-preview-ns.svc + - --rw.documentdb-preview-ns + - --rw + - "*.fleet-system.svc" + issuerRef: + name: selfsigned-cross-region-issuer + kind: Issuer + group: cert-manager.io +``` + +For a working KubeFleet example that propagates the certificate Secrets to every +member Kubernetes cluster with `ResourcePlacement`, see +[documentdb-resource-crp.yaml](https://github.com/documentdb/documentdb-kubernetes-operator/blob/main/documentdb-playground/aks-fleet-deployment/documentdb-resource-crp.yaml) +in the playground. + +See [TLSConfiguration](../api-reference.md#tlsconfiguration) in the API Reference +for where PostgreSQL TLS is configured on the DocumentDB resource. + ## Deployment options Choose a deployment approach based on your infrastructure and operational preferences. @@ -402,6 +477,6 @@ If PVCs aren't provisioning: ## Next steps - [Failover procedures](failover-procedures.md) - Learn how to perform planned and unplanned failovers -- [Backup and restore](../backup-and-restore.md) - Configure multi-region backup strategies +- [Backup and restore](../operations/backup-and-restore.md) - Configure multi-region backup strategies - [TLS configuration](../configuration/tls.md) - Secure connections with proper TLS certificates - [AKS Fleet deployment example](https://github.com/documentdb/documentdb-kubernetes-operator/blob/main/documentdb-playground/aks-fleet-deployment/README.md) - Automated Azure multi-region setup diff --git a/documentdb-playground/aks-fleet-deployment/deploy-fleet-bicep.sh b/documentdb-playground/aks-fleet-deployment/deploy-fleet-bicep.sh index 9fbc87c90..073528db1 100755 --- a/documentdb-playground/aks-fleet-deployment/deploy-fleet-bicep.sh +++ b/documentdb-playground/aks-fleet-deployment/deploy-fleet-bicep.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -set -eu +set -euo pipefail # Define variables (allow env overrides) RESOURCE_GROUP="${RESOURCE_GROUP:-documentdb-aks-fleet-rg}" @@ -112,10 +112,10 @@ done <<< "$MEMBER_CLUSTER_NAMES" ######### KUBEFLEET SETUP ######### kubeDir=$(mktemp -d) -git clone https://github.com/kubefleet-dev/kubefleet.git $kubeDir -pushd $kubeDir +git clone https://github.com/kubefleet-dev/kubefleet.git "$kubeDir" +pushd "$kubeDir" # Set up HUB_CLUSTER as the hub -kubectl config use-context $HUB_CLUSTER +kubectl config use-context "$HUB_CLUSTER" # Install cert manager on hub cluster helm repo add jetstack https://charts.jetstack.io @@ -132,6 +132,10 @@ kubectl get pods -n cert-manager -o wide || true export REGISTRY="ghcr.io/kubefleet-dev/kubefleet" export TAG="v0.3.0" +# Keep chart templates aligned with image tags to avoid unsupported flags. +git fetch --tags --quiet +git checkout --detach "$TAG" + # Install the helm chart for running Fleet agents on the hub cluster. helm upgrade --install hub-agent ./charts/hub-agent/ \ --set image.pullPolicy=Always \ @@ -156,29 +160,49 @@ chmod +x ./hack/membership/joinMC.sh HUB_CA=$(kubectl config view --raw -o jsonpath="{.clusters[?(@.name==\"$HUB_CLUSTER\")].cluster.certificate-authority-data}") sed -i "s/--set namespace=fleet-system/--namespace=fleet-system --create-namespace --set config.hubCA=$HUB_CA/" hack/membership/joinMC.sh -./hack/membership/joinMC.sh $TAG $HUB_CLUSTER $MEMBER_CLUSTER_NAMES +./hack/membership/joinMC.sh "$TAG" "$HUB_CLUSTER" $MEMBER_CLUSTER_NAMES + +# Ensure each generated member hub-access service account can watch hub resources. +for sa in $(kubectl --context "$HUB_CLUSTER" get sa -n connect-to-fleet -o json | jq -r '.items[].metadata.name | select(test("-hub-cluster-access$"))'); do + kubectl --context "$HUB_CLUSTER" create clusterrolebinding "${sa}-hub-agent-role" \ + --clusterrole=hub-agent-role \ + --serviceaccount=connect-to-fleet:"$sa" \ + --dry-run=client -o yaml | kubectl --context "$HUB_CLUSTER" apply -f - + + kubectl --context "$HUB_CLUSTER" create clusterrolebinding "${sa}-hub-net-role" \ + --clusterrole=hub-net-controller-manager-role \ + --serviceaccount=connect-to-fleet:"$sa" \ + --dry-run=client -o yaml | kubectl --context "$HUB_CLUSTER" apply -f - +done popd fleetNetworkingDir=$(mktemp -d) -git clone https://github.com/Azure/fleet-networking.git $fleetNetworkingDir -pushd $fleetNetworkingDir +git clone https://github.com/Azure/fleet-networking.git "$fleetNetworkingDir" +pushd "$fleetNetworkingDir" # Set up HUB_CLUSTER as the hub NETWORKING_TAG="v0.3.30" +git fetch --tags --quiet +git checkout --detach "$NETWORKING_TAG" # Install the helm chart for running Fleet agents on the hub cluster. -kubectl config use-context $HUB_CLUSTER +kubectl config use-context "$HUB_CLUSTER" helm install hub-net-controller-manager ./charts/hub-net-controller-manager/ \ --set fleetSystemNamespace=fleet-system-hub \ --set leaderElectionNamespace=fleet-system-hub \ --set image.tag=$NETWORKING_TAG HUB_CLUSTER_ADDRESS=$(kubectl config view -o jsonpath="{.clusters[?(@.name==\"$HUB_CLUSTER\")].cluster.server}") +HUB_CA=$(kubectl config view --raw -o jsonpath="{.clusters[?(@.name==\"$HUB_CLUSTER\")].cluster.certificate-authority-data}") while read -r MEMBER_CLUSTER; do - kubectl config use-context $MEMBER_CLUSTER + kubectl config use-context "$MEMBER_CLUSTER" kubectl apply -f config/crd/* + # ADD HUB CA to member cluster (temp fix while joinMC.sh is out of date) + kubectl -n fleet-system set env deploy/member-agent \ + HUB_CERTIFICATE_AUTHORITY="$HUB_CA" -c member-agent + echo "Installing mcs-controller-manager..." helm install mcs-controller-manager ./charts/mcs-controller-manager/ \ --set refreshtoken.repository=$REGISTRY/refresh-token \ @@ -187,7 +211,7 @@ while read -r MEMBER_CLUSTER; do --set image.pullPolicy=Always \ --set refreshtoken.pullPolicy=Always \ --set config.hubURL=$HUB_CLUSTER_ADDRESS \ - --set config.memberClusterName=$MEMBER_CLUSTER \ + --set config.memberClusterName="$MEMBER_CLUSTER" \ --set enableV1Beta1APIs=true \ --namespace fleet-system \ --create-namespace \ @@ -201,7 +225,7 @@ while read -r MEMBER_CLUSTER; do --set image.pullPolicy=Always \ --set refreshtoken.pullPolicy=Always \ --set config.hubURL=$HUB_CLUSTER_ADDRESS \ - --set config.memberClusterName=$MEMBER_CLUSTER \ + --set config.memberClusterName="$MEMBER_CLUSTER" \ --set enableV1Beta1APIs=true \ --namespace fleet-system \ --create-namespace \ diff --git a/documentdb-playground/aks-fleet-deployment/deploy-multi-region.sh b/documentdb-playground/aks-fleet-deployment/deploy-multi-region.sh index 81a2f9949..908777951 100755 --- a/documentdb-playground/aks-fleet-deployment/deploy-multi-region.sh +++ b/documentdb-playground/aks-fleet-deployment/deploy-multi-region.sh @@ -17,6 +17,124 @@ set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +WEBHOOK_WAIT_TIMEOUT_SECONDS="${WEBHOOK_WAIT_TIMEOUT_SECONDS:-300}" +APPLY_RETRY_COUNT="${APPLY_RETRY_COUNT:-4}" +DOCDB_NAME="documentdb-preview" +DOCDB_NAMESPACE="documentdb-preview-ns" + +generate_cnpg_cluster_name() { + local docdb_name="$1" + local cluster="$2" + local h=0xcbf29ce484222325 + local prime=0x100000001b3 + local max_docdb_len=$((50 - 9)) + local i byte hash + local LC_ALL=C + + for ((i = 0; i < ${#cluster}; i++)); do + printf -v byte '%d' "'${cluster:i:1}" + h=$(((h ^ byte) * prime)) + done + + printf -v hash '%x' "$h" + printf '%s\n' "${docdb_name:0:max_docdb_len}-${hash}" | cut -c1-50 +} + +wait_for_context_connectivity() { + local context="$1" + local timeout_seconds="${2:-60}" + local end_time=$((SECONDS + timeout_seconds)) + + echo "Validating API connectivity for context: $context" + while [ $SECONDS -lt $end_time ]; do + if kubectl --context "$context" version --request-timeout=10s >/dev/null 2>&1; then + echo "✓ API reachable for context: $context" + return 0 + fi + sleep 5 + done + + echo "✗ Unable to reach Kubernetes API for context: $context" + echo " This is often a kubeconfig endpoint or local DNS/network issue." + return 1 +} + +wait_for_service_endpoints() { + local context="$1" + local namespace="$2" + local service="$3" + local timeout_seconds="${4:-300}" + local end_time=$((SECONDS + timeout_seconds)) + + echo "Waiting for webhook service endpoints: ${namespace}/${service} (context: ${context})" + while [ $SECONDS -lt $end_time ]; do + if ! kubectl --context "$context" get service "$service" -n "$namespace" >/dev/null 2>&1; then + sleep 5 + continue + fi + + local endpoint_count + endpoint_count=$(kubectl --context "$context" get endpoints "$service" -n "$namespace" -o jsonpath='{range .subsets[*].addresses[*]}{.ip}{"\n"}{end}' 2>/dev/null | wc -l ) + if [ "$endpoint_count" -gt 0 ]; then + echo "✓ Ready endpoints found for ${namespace}/${service}: ${endpoint_count}" + return 0 + fi + + sleep 5 + done + + echo "✗ Timed out waiting for endpoints on ${namespace}/${service}" + echo "Service summary:" + kubectl --context "$context" get service "$service" -n "$namespace" -o wide || true + echo "Endpoints summary:" + kubectl --context "$context" get endpoints "$service" -n "$namespace" -o wide || true + + local selector + selector=$(kubectl --context "$context" get service "$service" -n "$namespace" -o jsonpath='{range $k,$v := .spec.selector}{$k}={$v}{","}{end}' 2>/dev/null | sed 's/,$//') + if [ -n "$selector" ]; then + echo "Selected pods (${selector}):" + kubectl --context "$context" get pods -n "$namespace" -l "$selector" -o wide || true + fi + + echo "Recent namespace events:" + kubectl --context "$context" get events -n "$namespace" --sort-by=.metadata.creationTimestamp | tail -20 || true + return 1 +} + +apply_with_webhook_retries() { + local context="$1" + local file_path="$2" + local max_attempts="${3:-4}" + local attempt=1 + + while [ "$attempt" -le "$max_attempts" ]; do + echo "Apply attempt ${attempt}/${max_attempts}..." + + local output + if output=$(kubectl --context "$context" apply -f "$file_path" 2>&1); then + echo "$output" + return 0 + fi + + echo "$output" + if echo "$output" | grep -Eq 'no endpoints available for service|failed calling webhook|context deadline exceeded'; then + if [ "$attempt" -lt "$max_attempts" ]; then + echo "Transient webhook failure detected; re-checking webhook readiness before retrying..." + wait_for_service_endpoints "$context" "documentdb-operator" "documentdb-webhook-service" "$WEBHOOK_WAIT_TIMEOUT_SECONDS" || true + wait_for_service_endpoints "$context" "fleet-system-hub" "fleetwebhook" "$WEBHOOK_WAIT_TIMEOUT_SECONDS" || true + sleep $((attempt * 5)) + fi + else + echo "Non-webhook apply failure detected. Aborting retries." + return 1 + fi + + attempt=$((attempt + 1)) + done + + return 1 +} + RESOURCE_GROUP="${RESOURCE_GROUP:-documentdb-aks-fleet-rg}" HUB_REGION="${HUB_REGION:-westus3}" @@ -82,6 +200,23 @@ for cluster in "${CLUSTER_ARRAY[@]}"; do fi done +# Build the certificate SAN list for CNPG cross-region replication services. +CROSS_REGION_DNS_NAMES="" +for cluster in "${CLUSTER_ARRAY[@]}"; do + CNPG_CLUSTER_NAME=$(generate_cnpg_cluster_name "$DOCDB_NAME" "$cluster") + for dns_name in \ + "${CNPG_CLUSTER_NAME}-rw.${DOCDB_NAMESPACE}.svc" \ + "${CNPG_CLUSTER_NAME}-rw.${DOCDB_NAMESPACE}" \ + "${CNPG_CLUSTER_NAME}-rw"; do + if [ -z "$CROSS_REGION_DNS_NAMES" ]; then + CROSS_REGION_DNS_NAMES=" - ${dns_name}" + else + CROSS_REGION_DNS_NAMES="${CROSS_REGION_DNS_NAMES}"$'\n'" - ${dns_name}" + fi + done +done +CROSS_REGION_DNS_NAMES="${CROSS_REGION_DNS_NAMES}"$'\n'" - \"*.fleet-system.svc\"" + # Step 1: Create cluster identification ConfigMaps on each member cluster echo "" echo "=======================================" @@ -127,6 +262,21 @@ fi echo "Using hub context: $HUB_CLUSTER" +if ! wait_for_context_connectivity "$HUB_CLUSTER" 60; then + echo "Fix kubeconfig connectivity for context '$HUB_CLUSTER' and retry." + exit 1 +fi + +if ! wait_for_service_endpoints "$HUB_CLUSTER" "documentdb-operator" "documentdb-webhook-service" "$WEBHOOK_WAIT_TIMEOUT_SECONDS"; then + echo "DocumentDB validating webhook is not ready. Ensure documentdb-operator pod is Running/Ready." + exit 1 +fi + +if ! wait_for_service_endpoints "$HUB_CLUSTER" "fleet-system-hub" "fleetwebhook" "$WEBHOOK_WAIT_TIMEOUT_SECONDS"; then + echo "Fleet validating webhook is not ready. Ensure hub-agent/webhook pods in fleet-system-hub are Running/Ready." + exit 1 +fi + # Check if resources already exist EXISTING_RESOURCES="" if kubectl --context "$HUB_CLUSTER" get namespace documentdb-preview-ns; then @@ -187,6 +337,8 @@ sed -e "s/{{DOCUMENTDB_PASSWORD}}/$DOCUMENTDB_PASSWORD/g" \ while IFS= read -r line; do if [[ "$line" == '{{CLUSTER_LIST}}' ]]; then echo "$CLUSTER_LIST" + elif [[ "$line" == '{{CROSS_REGION_DNS_NAMES}}' ]]; then + echo "$CROSS_REGION_DNS_NAMES" else echo "$line" fi @@ -199,6 +351,8 @@ echo "--------------------------------" echo "Primary cluster: $PRIMARY_CLUSTER" echo "Cluster list:" echo "$CLUSTER_LIST" +echo "Cross-region certificate SANs:" +echo "$CROSS_REGION_DNS_NAMES" echo "--------------------------------" # cat "$TEMP_YAML" @@ -206,7 +360,11 @@ echo "--------------------------------" # Apply the configuration echo "" echo "Applying DocumentDB multi-region configuration..." -kubectl --context "$HUB_CLUSTER" apply -f "$TEMP_YAML" +if ! apply_with_webhook_retries "$HUB_CLUSTER" "$TEMP_YAML" "$APPLY_RETRY_COUNT"; then + echo "Failed to apply configuration after ${APPLY_RETRY_COUNT} attempts." + rm -f "$TEMP_YAML" + exit 1 +fi # Clean up temp file rm -f "$TEMP_YAML" diff --git a/documentdb-playground/aks-fleet-deployment/documentdb-operator-crp.yaml b/documentdb-playground/aks-fleet-deployment/documentdb-operator-crp.yaml index 7520babbc..121b586be 100644 --- a/documentdb-playground/aks-fleet-deployment/documentdb-operator-crp.yaml +++ b/documentdb-playground/aks-fleet-deployment/documentdb-operator-crp.yaml @@ -105,5 +105,19 @@ spec: name: wal-replica-manager-binding policy: placementType: PickAll + affinity: + clusterAffinity: + requiredDuringSchedulingIgnoredDuringExecution: + clusterSelectorTerms: + - labelSelector: + matchExpressions: + - key: documentdb.io/fleet-hub + operator: DoesNotExist + - labelSelector: + matchExpressions: + - key: documentdb.io/fleet-hub + operator: NotIn + values: + - "true" strategy: - type: RollingUpdate \ No newline at end of file + type: RollingUpdate diff --git a/documentdb-playground/aks-fleet-deployment/documentdb-resource-crp.yaml b/documentdb-playground/aks-fleet-deployment/documentdb-resource-crp.yaml index 93a6a4029..49f7dc6ba 100644 --- a/documentdb-playground/aks-fleet-deployment/documentdb-resource-crp.yaml +++ b/documentdb-playground/aks-fleet-deployment/documentdb-resource-crp.yaml @@ -16,6 +16,62 @@ stringData: username: default_user password: {{DOCUMENTDB_PASSWORD}} +## Cert-manager resources for cross-region replication TLS +--- +apiVersion: cert-manager.io/v1 +kind: Issuer +metadata: + name: selfsigned-cross-region-issuer + namespace: documentdb-preview-ns +spec: + selfSigned: {} +--- +apiVersion: v1 +kind: Secret +metadata: + name: cross-region-client-cert + namespace: documentdb-preview-ns + labels: + cnpg.io/reload: "" +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: cross-region-client-cert + namespace: documentdb-preview-ns +spec: + secretName: cross-region-client-cert + usages: + - client auth + commonName: streaming_replica + issuerRef: + name: selfsigned-cross-region-issuer + kind: Issuer + group: cert-manager.io +--- +apiVersion: v1 +kind: Secret +metadata: + name: cross-region-server-cert + namespace: documentdb-preview-ns + labels: + cnpg.io/reload: "" +--- +apiVersion: cert-manager.io/v1 +kind: Certificate +metadata: + name: cross-region-server-cert + namespace: documentdb-preview-ns +spec: + secretName: cross-region-server-cert + usages: + - server auth + dnsNames: +{{CROSS_REGION_DNS_NAMES}} + issuerRef: + name: selfsigned-cross-region-issuer + kind: Issuer + group: cert-manager.io --- apiVersion: documentdb.io/preview @@ -39,6 +95,12 @@ spec: exposeViaService: serviceType: LoadBalancer logLevel: info + tls: + postgres: + replicationTLSSecret: cross-region-client-cert + clientCASecret: cross-region-client-cert + serverTLSSecret: cross-region-server-cert + serverCASecret: cross-region-server-cert --- @@ -75,6 +137,14 @@ spec: version: v1 kind: Secret name: documentdb-credentials + - group: "" + version: v1 + kind: Secret + name: cross-region-client-cert + - group: "" + version: v1 + kind: Secret + name: cross-region-server-cert policy: placementType: PickAll affinity: diff --git a/operator/documentdb-helm-chart/crds/documentdb.io_dbs.yaml b/operator/documentdb-helm-chart/crds/documentdb.io_dbs.yaml index 672830030..924e5c1a4 100644 --- a/operator/documentdb-helm-chart/crds/documentdb.io_dbs.yaml +++ b/operator/documentdb-helm-chart/crds/documentdb.io_dbs.yaml @@ -1125,6 +1125,12 @@ spec: - Istio - None type: string + disableTLS: + default: false + description: |- + Disables TLS for replication traffic between clusters. + Only for use when an existing mesh is already providing TLS. + type: boolean highAvailability: description: Whether or not to have replicas on the primary cluster. type: boolean @@ -1518,15 +1524,75 @@ spec: (placeholder for future phases). type: object postgres: - description: Postgres configures TLS for the Postgres server (placeholder - for future phases). + description: Postgres configures TLS for the Postgres server. + properties: + clientCASecret: + description: |- + The secret containing the Client CA certificate. If not defined, a new secret will be created + with a self-signed CA and will be used to generate all the client certificates.
+
+ Contains:
+
+ - `ca.crt`: CA that should be used to validate the client certificates, + used as `ssl_ca_file` of all the instances.
+ - `ca.key`: key used to generate client certificates, if ReplicationTLSSecret is provided, + this can be omitted.
+ type: string + replicationTLSSecret: + description: |- + The secret of type kubernetes.io/tls containing the client certificate to authenticate as + the `streaming_replica` user. + If not defined, ClientCASecret must provide also `ca.key`, and a new secret will be + created using the provided CA. + type: string + serverAltDNSNames: + description: The list of the server alternative DNS names + to be added to the generated server TLS certificates, when + required. + items: + type: string + type: array + serverCASecret: + description: |- + The secret containing the Server CA certificate. If not defined, a new secret will be created + with a self-signed CA and will be used to generate the TLS certificate ServerTLSSecret.
+
+ Contains:
+
+ - `ca.crt`: CA that should be used to validate the server certificate, + used as `sslrootcert` in client connection strings.
+ - `ca.key`: key used to generate Server SSL certs, if ServerTLSSecret is provided, + this can be omitted.
+ type: string + serverTLSSecret: + description: |- + The secret of type kubernetes.io/tls containing the server TLS certificate and key that will be set as + `ssl_cert_file` and `ssl_key_file` so that clients can connect to postgres securely. + If not defined, ServerCASecret must provide also `ca.key` and a new secret will be + created using the provided CA. + type: string type: object type: object + x-kubernetes-validations: + - message: spec.tls.postgres replicationTLSSecret and clientCASecret + must be provided together; serverTLSSecret and serverCASecret + must be provided together; serverTLSSecret requires replicationTLSSecret + rule: '!has(self.postgres) || (has(self.postgres.replicationTLSSecret) + == has(self.postgres.clientCASecret) && has(self.postgres.serverTLSSecret) + == has(self.postgres.serverCASecret) && (!has(self.postgres.serverTLSSecret) + || has(self.postgres.replicationTLSSecret)))' required: - instancesPerNode - nodeCount - resource type: object + x-kubernetes-validations: + - message: when spec.clusterReplication is set, either spec.clusterReplication.disableTLS + must be true or spec.tls.postgres.replicationTLSSecret and spec.tls.postgres.clientCASecret + must be provided + rule: '!has(self.clusterReplication) || ((has(self.clusterReplication.disableTLS) + && self.clusterReplication.disableTLS) || (has(self.tls) && has(self.tls.postgres) + && has(self.tls.postgres.replicationTLSSecret) && has(self.tls.postgres.clientCASecret)))' status: description: DocumentDBStatus defines the observed state of DocumentDB. properties: diff --git a/operator/src/api/preview/documentdb_types.go b/operator/src/api/preview/documentdb_types.go index b0cae72f1..e2dfc4287 100644 --- a/operator/src/api/preview/documentdb_types.go +++ b/operator/src/api/preview/documentdb_types.go @@ -23,6 +23,7 @@ const ( ) // DocumentDBSpec defines the desired state of DocumentDB. +// +kubebuilder:validation:XValidation:rule="!has(self.clusterReplication) || ((has(self.clusterReplication.disableTLS) && self.clusterReplication.disableTLS) || (has(self.tls) && has(self.tls.postgres) && has(self.tls.postgres.replicationTLSSecret) && has(self.tls.postgres.clientCASecret)))",message="when spec.clusterReplication is set, either spec.clusterReplication.disableTLS must be true or spec.tls.postgres.replicationTLSSecret and spec.tls.postgres.clientCASecret must be provided" type DocumentDBSpec struct { // NodeCount is the number of nodes in the DocumentDB cluster. Must be 1. // +kubebuilder:validation:Minimum=1 @@ -327,6 +328,10 @@ type ClusterReplication struct { ClusterList []MemberCluster `json:"clusterList"` // Whether or not to have replicas on the primary cluster. HighAvailability bool `json:"highAvailability,omitempty"` + // Disables TLS for replication traffic between clusters. + // Only for use when an existing mesh is already providing TLS. + // +kubebuilder:default=false + DisableTLS bool `json:"disableTLS,omitempty"` } type MemberCluster struct { @@ -353,12 +358,13 @@ type Timeouts struct { } // TLSConfiguration aggregates TLS settings across DocumentDB components. +// +kubebuilder:validation:XValidation:rule="!has(self.postgres) || (has(self.postgres.replicationTLSSecret) == has(self.postgres.clientCASecret) && has(self.postgres.serverTLSSecret) == has(self.postgres.serverCASecret) && (!has(self.postgres.serverTLSSecret) || has(self.postgres.replicationTLSSecret)))",message="spec.tls.postgres replicationTLSSecret and clientCASecret must be provided together; serverTLSSecret and serverCASecret must be provided together; serverTLSSecret requires replicationTLSSecret" type TLSConfiguration struct { // Gateway configures TLS for the gateway sidecar (Phase 1: certificate provisioning only). Gateway *GatewayTLS `json:"gateway,omitempty"` - // Postgres configures TLS for the Postgres server (placeholder for future phases). - Postgres *PostgresTLS `json:"postgres,omitempty"` + // Postgres configures TLS for the Postgres server. + Postgres *cnpgv1.CertificatesConfiguration `json:"postgres,omitempty"` // GlobalEndpoints configures TLS for global endpoints (placeholder for future phases). GlobalEndpoints *GlobalEndpointsTLS `json:"globalEndpoints,omitempty"` @@ -379,9 +385,6 @@ type GatewayTLS struct { Provided *ProvidedTLS `json:"provided,omitempty"` } -// PostgresTLS acts as a placeholder for future Postgres TLS settings. -type PostgresTLS struct{} - // GlobalEndpointsTLS acts as a placeholder for future global endpoint TLS settings. type GlobalEndpointsTLS struct{} diff --git a/operator/src/api/preview/zz_generated.deepcopy.go b/operator/src/api/preview/zz_generated.deepcopy.go index 804104c90..7908588bd 100644 --- a/operator/src/api/preview/zz_generated.deepcopy.go +++ b/operator/src/api/preview/zz_generated.deepcopy.go @@ -8,6 +8,7 @@ package preview import ( + apiv1 "github.com/cloudnative-pg/cloudnative-pg/api/v1" "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/runtime" ) @@ -572,21 +573,6 @@ func (in *PostgresSpec) DeepCopy() *PostgresSpec { return out } -// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. -func (in *PostgresTLS) DeepCopyInto(out *PostgresTLS) { - *out = *in -} - -// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new PostgresTLS. -func (in *PostgresTLS) DeepCopy() *PostgresTLS { - if in == nil { - return nil - } - out := new(PostgresTLS) - in.DeepCopyInto(out) - return out -} - // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *PrometheusExporterSpec) DeepCopyInto(out *PrometheusExporterSpec) { *out = *in @@ -782,8 +768,8 @@ func (in *TLSConfiguration) DeepCopyInto(out *TLSConfiguration) { } if in.Postgres != nil { in, out := &in.Postgres, &out.Postgres - *out = new(PostgresTLS) - **out = **in + *out = new(apiv1.CertificatesConfiguration) + (*in).DeepCopyInto(*out) } if in.GlobalEndpoints != nil { in, out := &in.GlobalEndpoints, &out.GlobalEndpoints diff --git a/operator/src/config/crd/bases/documentdb.io_dbs.yaml b/operator/src/config/crd/bases/documentdb.io_dbs.yaml index 672830030..924e5c1a4 100644 --- a/operator/src/config/crd/bases/documentdb.io_dbs.yaml +++ b/operator/src/config/crd/bases/documentdb.io_dbs.yaml @@ -1125,6 +1125,12 @@ spec: - Istio - None type: string + disableTLS: + default: false + description: |- + Disables TLS for replication traffic between clusters. + Only for use when an existing mesh is already providing TLS. + type: boolean highAvailability: description: Whether or not to have replicas on the primary cluster. type: boolean @@ -1518,15 +1524,75 @@ spec: (placeholder for future phases). type: object postgres: - description: Postgres configures TLS for the Postgres server (placeholder - for future phases). + description: Postgres configures TLS for the Postgres server. + properties: + clientCASecret: + description: |- + The secret containing the Client CA certificate. If not defined, a new secret will be created + with a self-signed CA and will be used to generate all the client certificates.
+
+ Contains:
+
+ - `ca.crt`: CA that should be used to validate the client certificates, + used as `ssl_ca_file` of all the instances.
+ - `ca.key`: key used to generate client certificates, if ReplicationTLSSecret is provided, + this can be omitted.
+ type: string + replicationTLSSecret: + description: |- + The secret of type kubernetes.io/tls containing the client certificate to authenticate as + the `streaming_replica` user. + If not defined, ClientCASecret must provide also `ca.key`, and a new secret will be + created using the provided CA. + type: string + serverAltDNSNames: + description: The list of the server alternative DNS names + to be added to the generated server TLS certificates, when + required. + items: + type: string + type: array + serverCASecret: + description: |- + The secret containing the Server CA certificate. If not defined, a new secret will be created + with a self-signed CA and will be used to generate the TLS certificate ServerTLSSecret.
+
+ Contains:
+
+ - `ca.crt`: CA that should be used to validate the server certificate, + used as `sslrootcert` in client connection strings.
+ - `ca.key`: key used to generate Server SSL certs, if ServerTLSSecret is provided, + this can be omitted.
+ type: string + serverTLSSecret: + description: |- + The secret of type kubernetes.io/tls containing the server TLS certificate and key that will be set as + `ssl_cert_file` and `ssl_key_file` so that clients can connect to postgres securely. + If not defined, ServerCASecret must provide also `ca.key` and a new secret will be + created using the provided CA. + type: string type: object type: object + x-kubernetes-validations: + - message: spec.tls.postgres replicationTLSSecret and clientCASecret + must be provided together; serverTLSSecret and serverCASecret + must be provided together; serverTLSSecret requires replicationTLSSecret + rule: '!has(self.postgres) || (has(self.postgres.replicationTLSSecret) + == has(self.postgres.clientCASecret) && has(self.postgres.serverTLSSecret) + == has(self.postgres.serverCASecret) && (!has(self.postgres.serverTLSSecret) + || has(self.postgres.replicationTLSSecret)))' required: - instancesPerNode - nodeCount - resource type: object + x-kubernetes-validations: + - message: when spec.clusterReplication is set, either spec.clusterReplication.disableTLS + must be true or spec.tls.postgres.replicationTLSSecret and spec.tls.postgres.clientCASecret + must be provided + rule: '!has(self.clusterReplication) || ((has(self.clusterReplication.disableTLS) + && self.clusterReplication.disableTLS) || (has(self.tls) && has(self.tls.postgres) + && has(self.tls.postgres.replicationTLSSecret) && has(self.tls.postgres.clientCASecret)))' status: description: DocumentDBStatus defines the observed state of DocumentDB. properties: diff --git a/operator/src/internal/cnpg/cnpg_cluster.go b/operator/src/internal/cnpg/cnpg_cluster.go index 591c5410b..8ceef9a00 100644 --- a/operator/src/internal/cnpg/cnpg_cluster.go +++ b/operator/src/internal/cnpg/cnpg_cluster.go @@ -116,6 +116,7 @@ func GetCnpgClusterSpec(req ctrl.Request, documentdb *dbpreview.DocumentDB, docu PostgresConfiguration: buildPostgresConfiguration(documentdb, extensionImageSource), Bootstrap: getBootstrapConfiguration(documentdb, isPrimaryRegion, log), LogLevel: cmp.Or(documentdb.Spec.LogLevel, "info"), + Certificates: postgresCertificates(documentdb), Backup: &cnpgv1.BackupConfiguration{ VolumeSnapshot: &cnpgv1.VolumeSnapshotConfiguration{ SnapshotOwnerReference: "backup", // Set owner reference to 'backup' so that snapshots are deleted when Backup resource is deleted @@ -291,6 +292,13 @@ func pluginsSidecarInjectorName(documentdb *dbpreview.DocumentDB) string { return documentdb.Spec.Plugins.SidecarInjectorName } +func postgresCertificates(documentdb *dbpreview.DocumentDB) *cnpgv1.CertificatesConfiguration { + if documentdb.Spec.TLS == nil { + return nil + } + return documentdb.Spec.TLS.Postgres +} + // toCNPGImagePullSecrets translates a list of corev1.LocalObjectReference // (the Kubernetes-native shape used on spec.imagePullSecrets) into the // CNPG-flavoured cnpgv1.LocalObjectReference shape that @@ -357,9 +365,8 @@ func applyIOUringSeccomp(spec *cnpgv1.ClusterSpec, documentdb *dbpreview.Documen // operator-managed GUCs. func buildPostgresConfiguration(documentdb *dbpreview.DocumentDB, extensionImageSource corev1.ImageVolumeSource) cnpgv1.PostgresConfiguration { pgHBA := []string{ - "host all all 0.0.0.0/0 trust", - "host all all ::0/0 trust", - "host replication all all trust", + "host all all localhost trust", + "hostssl replication streaming_replica all cert", } return cnpgv1.PostgresConfiguration{ diff --git a/operator/src/internal/cnpg/cnpg_cluster_test.go b/operator/src/internal/cnpg/cnpg_cluster_test.go index 8b7c7ad19..96112489d 100644 --- a/operator/src/internal/cnpg/cnpg_cluster_test.go +++ b/operator/src/internal/cnpg/cnpg_cluster_test.go @@ -191,6 +191,60 @@ var _ = Describe("getDefaultBootstrapConfiguration", func() { }) }) +var _ = Describe("Postgres certificate configuration", func() { + It("omits Postgres certificate configuration", func() { + req := ctrl.Request{} + req.Name = "test-cluster" + req.Namespace = "default" + + documentdb := &dbpreview.DocumentDB{ + Spec: dbpreview.DocumentDBSpec{ + InstancesPerNode: 1, + Resource: dbpreview.Resource{ + Storage: dbpreview.StorageConfiguration{PvcSize: "10Gi"}, + }, + }, + } + + result := GetCnpgClusterSpec(req, documentdb, "ext:1.0", "test-sa", "", true, zap.New(zap.WriteTo(GinkgoWriter))) + + Expect(result.Spec.Certificates).To(BeNil()) + }) + + It("includes Postgres certificate configuration when TLS is set", func() { + req := ctrl.Request{} + req.Name = "test-cluster" + req.Namespace = "default" + + // Create a certificates configuration + certificatesConfig := &cnpgv1.CertificatesConfiguration{ + ServerTLSSecret: "server-tls-secret", + ServerCASecret: "server-ca-secret", + ReplicationTLSSecret: "replication-tls-secret", + ClientCASecret: "client-ca-secret", + } + + documentdb := &dbpreview.DocumentDB{ + Spec: dbpreview.DocumentDBSpec{ + InstancesPerNode: 1, + Resource: dbpreview.Resource{ + Storage: dbpreview.StorageConfiguration{PvcSize: "10Gi"}, + }, + TLS: &dbpreview.TLSConfiguration{ + Postgres: certificatesConfig, + }, + }, + } + + result := GetCnpgClusterSpec(req, documentdb, "ext:1.0", "test-sa", "", true, zap.New(zap.WriteTo(GinkgoWriter))) + + Expect(result.Spec.Certificates).ToNot(BeNil()) + Expect(result.Spec.Certificates).To(Equal(certificatesConfig)) + Expect(result.Spec.Certificates.ServerTLSSecret).To(Equal("server-tls-secret")) + Expect(result.Spec.Certificates.ClientCASecret).To(Equal("client-ca-secret")) + }) +}) + var _ = Describe("GetCnpgClusterSpec", func() { var log = zap.New(zap.WriteTo(GinkgoWriter)) @@ -231,7 +285,9 @@ var _ = Describe("GetCnpgClusterSpec", func() { Expect(result.Spec.PostgresConfiguration.Extensions[0].LdLibraryPath).To(Equal([]string{"lib", "system"})) Expect(result.Spec.PostgresConfiguration.AdditionalLibraries).To(ConsistOf("pg_cron", "pg_documentdb_core", "pg_documentdb")) Expect(result.Spec.PostgresConfiguration.Parameters).To(HaveKeyWithValue("cron.database_name", "postgres")) - Expect(result.Spec.PostgresConfiguration.PgHBA).To(HaveLen(3)) + Expect(result.Spec.PostgresConfiguration.PgHBA).To(HaveLen(2)) + Expect(result.Spec.PostgresConfiguration.PgHBA[0]).To(Equal("host all all localhost trust")) + Expect(result.Spec.PostgresConfiguration.PgHBA[1]).To(Equal("hostssl replication streaming_replica all cert")) Expect(result.Spec.PostgresUID).To(Equal(int64(0))) Expect(result.Spec.PostgresGID).To(Equal(int64(0))) }) @@ -375,6 +431,88 @@ var _ = Describe("GetCnpgClusterSpec", func() { Expect(result.Spec.Plugins[0].Parameters["gatewayTLSSecret"]).To(Equal("my-tls-secret")) }) + It("uses custom SidecarInjectorName when specified", func() { + req := ctrl.Request{} + req.Name = "test-cluster" + req.Namespace = "default" + + documentdb := &dbpreview.DocumentDB{ + Spec: dbpreview.DocumentDBSpec{ + InstancesPerNode: 3, + Resource: dbpreview.Resource{ + Storage: dbpreview.StorageConfiguration{ + PvcSize: "10Gi", + }, + }, + Plugins: &dbpreview.PluginsSpec{ + SidecarInjectorName: "custom-injector", + }, + }, + } + + result := GetCnpgClusterSpec(req, documentdb, "postgres:16", "test-sa", "", true, log) + Expect(result).ToNot(BeNil()) + Expect(result.Spec.Plugins).To(HaveLen(1)) + Expect(result.Spec.Plugins[0].Name).To(Equal("custom-injector")) + }) + + It("applies TLS and certificate configuration together", func() { + req := ctrl.Request{} + req.Name = "test-cluster" + req.Namespace = "default" + + certificatesConfig := &cnpgv1.CertificatesConfiguration{ + ServerTLSSecret: "server-tls-secret", + ServerCASecret: "server-ca-secret", + ReplicationTLSSecret: "replication-tls-secret", + ClientCASecret: "client-ca-secret", + } + + documentdb := &dbpreview.DocumentDB{ + Spec: dbpreview.DocumentDBSpec{ + InstancesPerNode: 3, + Resource: dbpreview.Resource{ + Storage: dbpreview.StorageConfiguration{ + PvcSize: "10Gi", + }, + }, + TLS: &dbpreview.TLSConfiguration{ + Postgres: certificatesConfig, + }, + }, + } + + result := GetCnpgClusterSpec(req, documentdb, "postgres:16", "test-sa", "", true, log) + Expect(result).ToNot(BeNil()) + Expect(result.Spec.Certificates).ToNot(BeNil()) + Expect(result.Spec.Certificates).To(Equal(certificatesConfig)) + }) + + It("handles nil plugins and nil TLS gracefully", func() { + req := ctrl.Request{} + req.Name = "test-cluster" + req.Namespace = "default" + + documentdb := &dbpreview.DocumentDB{ + Spec: dbpreview.DocumentDBSpec{ + InstancesPerNode: 1, + Resource: dbpreview.Resource{ + Storage: dbpreview.StorageConfiguration{ + PvcSize: "10Gi", + }, + }, + Plugins: nil, + TLS: nil, + }, + } + + result := GetCnpgClusterSpec(req, documentdb, "postgres:16", "test-sa", "", true, log) + Expect(result).ToNot(BeNil()) + Expect(result.Spec.Plugins).To(HaveLen(1)) + Expect(result.Spec.Plugins[0].Name).To(Equal(util.DEFAULT_SIDECAR_INJECTOR_PLUGIN)) + Expect(result.Spec.Certificates).To(BeNil()) + }) + It("passes gatewayImagePullPolicy to plugin params when env var is set", func() { req := ctrl.Request{} req.Name = "test-cluster" diff --git a/operator/src/internal/cnpg/cnpg_patch.go b/operator/src/internal/cnpg/cnpg_patch.go index f42e414e3..c0ab477f8 100644 --- a/operator/src/internal/cnpg/cnpg_patch.go +++ b/operator/src/internal/cnpg/cnpg_patch.go @@ -24,6 +24,8 @@ const ( PatchPathPlugins = "/spec/plugins" PatchPathReplicationSlots = "/spec/replicationSlots" PatchPathExternalClusters = "/spec/externalClusters" + PatchPathCertificates = "/spec/certificates" + PatchPathPostgresPgHBA = "/spec/postgresql/pg_hba" PatchPathManagedServices = "/spec/managed/services/additional" PatchPathSynchronous = "/spec/postgresql/synchronous" PatchPathBootstrap = "/spec/bootstrap" diff --git a/operator/src/internal/cnpg/cnpg_sync.go b/operator/src/internal/cnpg/cnpg_sync.go index 4d32f66e3..903fcb745 100644 --- a/operator/src/internal/cnpg/cnpg_sync.go +++ b/operator/src/internal/cnpg/cnpg_sync.go @@ -215,6 +215,21 @@ func SyncCnpgCluster( }) } + if !reflect.DeepEqual(current.Spec.Certificates, desired.Spec.Certificates) { + certificatesPatch := JSONPatch{ + Op: PatchOpReplace, + Path: PatchPathCertificates, + Value: desired.Spec.Certificates, + } + if current.Spec.Certificates == nil { + certificatesPatch.Op = PatchOpAdd + } else if desired.Spec.Certificates == nil { + certificatesPatch.Op = PatchOpRemove + certificatesPatch.Value = nil + } + patchOps = append(patchOps, certificatesPatch) + } + // Extra operations (e.g., replication changes) patchOps = append(patchOps, extraOps...) diff --git a/operator/src/internal/cnpg/cnpg_sync_test.go b/operator/src/internal/cnpg/cnpg_sync_test.go index 098511382..c7d88e8df 100644 --- a/operator/src/internal/cnpg/cnpg_sync_test.go +++ b/operator/src/internal/cnpg/cnpg_sync_test.go @@ -477,6 +477,95 @@ var _ = Describe("SyncCnpgCluster - mutable spec fields", func() { // No restart annotation — CNPG handles all mutable spec fields natively Expect(updated.Annotations).ToNot(HaveKey("kubectl.kubernetes.io/restartedAt")) }) + + It("adds certificates when desired has certificates and current does not", func() { + current := baseCluster("test-cluster", namespace) + current.Spec.Certificates = nil + desired := current.DeepCopy() + desired.Spec.Certificates = &cnpgv1.CertificatesConfiguration{ + ServerTLSSecret: "server-tls-secret", + ServerCASecret: "server-ca-secret", + ReplicationTLSSecret: "replication-tls-secret", + ClientCASecret: "client-ca-secret", + } + + c := buildFakeClient(current).Build() + err := SyncCnpgCluster(context.Background(), c, current, desired, nil) + Expect(err).ToNot(HaveOccurred()) + + updated := &cnpgv1.Cluster{} + Expect(c.Get(context.Background(), types.NamespacedName{Name: "test-cluster", Namespace: namespace}, updated)).To(Succeed()) + Expect(updated.Spec.Certificates).ToNot(BeNil()) + Expect(updated.Spec.Certificates.ServerTLSSecret).To(Equal("server-tls-secret")) + Expect(updated.Spec.Certificates.ClientCASecret).To(Equal("client-ca-secret")) + }) + + It("removes certificates when desired has no certificates but current does", func() { + current := baseCluster("test-cluster", namespace) + current.Spec.Certificates = &cnpgv1.CertificatesConfiguration{ + ServerTLSSecret: "server-tls-secret", + ServerCASecret: "server-ca-secret", + ReplicationTLSSecret: "replication-tls-secret", + ClientCASecret: "client-ca-secret", + } + desired := current.DeepCopy() + desired.Spec.Certificates = nil + + c := buildFakeClient(current).Build() + err := SyncCnpgCluster(context.Background(), c, current, desired, nil) + Expect(err).ToNot(HaveOccurred()) + + updated := &cnpgv1.Cluster{} + Expect(c.Get(context.Background(), types.NamespacedName{Name: "test-cluster", Namespace: namespace}, updated)).To(Succeed()) + Expect(updated.Spec.Certificates).To(BeNil()) + }) + + It("updates certificates when both current and desired have certificates but they differ", func() { + current := baseCluster("test-cluster", namespace) + current.Spec.Certificates = &cnpgv1.CertificatesConfiguration{ + ServerTLSSecret: "old-server-tls-secret", + ServerCASecret: "old-server-ca-secret", + ReplicationTLSSecret: "old-replication-tls-secret", + ClientCASecret: "old-client-ca-secret", + } + desired := current.DeepCopy() + desired.Spec.Certificates = &cnpgv1.CertificatesConfiguration{ + ServerTLSSecret: "new-server-tls-secret", + ServerCASecret: "new-server-ca-secret", + ReplicationTLSSecret: "new-replication-tls-secret", + ClientCASecret: "new-client-ca-secret", + } + + c := buildFakeClient(current).Build() + err := SyncCnpgCluster(context.Background(), c, current, desired, nil) + Expect(err).ToNot(HaveOccurred()) + + updated := &cnpgv1.Cluster{} + Expect(c.Get(context.Background(), types.NamespacedName{Name: "test-cluster", Namespace: namespace}, updated)).To(Succeed()) + Expect(updated.Spec.Certificates).ToNot(BeNil()) + Expect(updated.Spec.Certificates.ServerTLSSecret).To(Equal("new-server-tls-secret")) + Expect(updated.Spec.Certificates.ClientCASecret).To(Equal("new-client-ca-secret")) + }) + + It("applies multiple certificate and cluster configuration changes", func() { + current := baseCluster("test-cluster", namespace) + current.Spec.Certificates = &cnpgv1.CertificatesConfiguration{ + ServerTLSSecret: "old-tls", + } + + desired := current.DeepCopy() + desired.Spec.Certificates = &cnpgv1.CertificatesConfiguration{ + ServerTLSSecret: "new-tls", + } + + c := buildFakeClient(current).Build() + err := SyncCnpgCluster(context.Background(), c, current, desired, nil) + Expect(err).ToNot(HaveOccurred()) + + updated := &cnpgv1.Cluster{} + Expect(c.Get(context.Background(), types.NamespacedName{Name: "test-cluster", Namespace: namespace}, updated)).To(Succeed()) + Expect(updated.Spec.Certificates.ServerTLSSecret).To(Equal("new-tls")) + }) }) var _ = Describe("Helper functions", func() { diff --git a/operator/src/internal/controller/backup_controller_test.go b/operator/src/internal/controller/backup_controller_test.go index 0c174b996..881546272 100644 --- a/operator/src/internal/controller/backup_controller_test.go +++ b/operator/src/internal/controller/backup_controller_test.go @@ -313,66 +313,66 @@ var _ = Describe("Backup Controller", func() { }) Describe("Reconcile", func() { - It("creates CNPG Backup using ReplicationContext.CNPGClusterName when no CNPG Backup exists", func() { - backup := &dbpreview.Backup{ - ObjectMeta: metav1.ObjectMeta{ - Name: backupName, - Namespace: backupNamespace, - }, - Spec: dbpreview.BackupSpec{ - Cluster: cnpgv1.LocalObjectReference{Name: clusterName}, - }, - Status: dbpreview.BackupStatus{ - Phase: cnpgv1.BackupPhasePending, - }, - } + It("creates CNPG Backup using ReplicationContext.CNPGClusterName when no CNPG Backup exists", func() { + backup := &dbpreview.Backup{ + ObjectMeta: metav1.ObjectMeta{ + Name: backupName, + Namespace: backupNamespace, + }, + Spec: dbpreview.BackupSpec{ + Cluster: cnpgv1.LocalObjectReference{Name: clusterName}, + }, + Status: dbpreview.BackupStatus{ + Phase: cnpgv1.BackupPhasePending, + }, + } - // DocumentDB cluster without replication (IsPrimary() and EndpointEnabled() are true) - cluster := &dbpreview.DocumentDB{ - ObjectMeta: metav1.ObjectMeta{ - Name: clusterName, - Namespace: backupNamespace, - }, - } - - // Pre-existing default VolumeSnapshotClass so ensureVolumeSnapshotClass succeeds - vsc := &snapshotv1.VolumeSnapshotClass{ - ObjectMeta: metav1.ObjectMeta{ - Name: "default-snapclass", - Annotations: map[string]string{ - "snapshot.storage.kubernetes.io/is-default-class": "true", - }, - }, - Driver: "fake.csi.driver", - DeletionPolicy: snapshotv1.VolumeSnapshotContentDelete, - } - - fakeClient := fake.NewClientBuilder(). - WithScheme(scheme). - WithObjects(backup, cluster, vsc). - WithStatusSubresource(&dbpreview.Backup{}). - Build() - - reconciler := &BackupReconciler{ - Client: fakeClient, - Scheme: scheme, - Recorder: recorder, - } - - res, err := reconciler.Reconcile(ctx, reconcile.Request{ - NamespacedName: types.NamespacedName{ - Name: backupName, - Namespace: backupNamespace, + // DocumentDB cluster without replication (IsPrimary() and EndpointEnabled() are true) + cluster := &dbpreview.DocumentDB{ + ObjectMeta: metav1.ObjectMeta{ + Name: clusterName, + Namespace: backupNamespace, + }, + } + + // Pre-existing default VolumeSnapshotClass so ensureVolumeSnapshotClass succeeds + vsc := &snapshotv1.VolumeSnapshotClass{ + ObjectMeta: metav1.ObjectMeta{ + Name: "default-snapclass", + Annotations: map[string]string{ + "snapshot.storage.kubernetes.io/is-default-class": "true", }, - }) - Expect(err).ToNot(HaveOccurred()) - Expect(res.RequeueAfter).To(Equal(5 * time.Second)) - - // Verify a CNPG Backup was created with cluster name matching the DocumentDB name - // (no replication → CNPGClusterName == DocumentDB name) - cnpgBackup := &cnpgv1.Backup{} - Expect(fakeClient.Get(ctx, client.ObjectKey{Name: backupName, Namespace: backupNamespace}, cnpgBackup)).To(Succeed()) - Expect(cnpgBackup.Spec.Cluster.Name).To(Equal(clusterName)) + }, + Driver: "fake.csi.driver", + DeletionPolicy: snapshotv1.VolumeSnapshotContentDelete, + } + + fakeClient := fake.NewClientBuilder(). + WithScheme(scheme). + WithObjects(backup, cluster, vsc). + WithStatusSubresource(&dbpreview.Backup{}). + Build() + + reconciler := &BackupReconciler{ + Client: fakeClient, + Scheme: scheme, + Recorder: recorder, + } + + res, err := reconciler.Reconcile(ctx, reconcile.Request{ + NamespacedName: types.NamespacedName{ + Name: backupName, + Namespace: backupNamespace, + }, }) + Expect(err).ToNot(HaveOccurred()) + Expect(res.RequeueAfter).To(Equal(5 * time.Second)) + + // Verify a CNPG Backup was created with cluster name matching the DocumentDB name + // (no replication → CNPGClusterName == DocumentDB name) + cnpgBackup := &cnpgv1.Backup{} + Expect(fakeClient.Get(ctx, client.ObjectKey{Name: backupName, Namespace: backupNamespace}, cnpgBackup)).To(Succeed()) + Expect(cnpgBackup.Spec.Cluster.Name).To(Equal(clusterName)) + }) }) }) diff --git a/operator/src/internal/controller/certificate_controller_test.go b/operator/src/internal/controller/certificate_controller_test.go index a9b5f7004..9753fc249 100644 --- a/operator/src/internal/controller/certificate_controller_test.go +++ b/operator/src/internal/controller/certificate_controller_test.go @@ -10,8 +10,10 @@ import ( cmapi "github.com/cert-manager/cert-manager/pkg/apis/certmanager/v1" cmmeta "github.com/cert-manager/cert-manager/pkg/apis/meta/v1" + v1 "github.com/cloudnative-pg/cloudnative-pg/api/v1" "github.com/stretchr/testify/require" corev1 "k8s.io/api/core/v1" + "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/types" @@ -58,6 +60,24 @@ func baseDocumentDB(name, ns string) *dbpreview.DocumentDB { } } +func enableAzureFleetReplication(ddb *dbpreview.DocumentDB) { + ddb.Spec.ClusterReplication = &dbpreview.ClusterReplication{ + CrossCloudNetworkingStrategy: string(util.AzureFleet), + Primary: "member-a", + ClusterList: []dbpreview.MemberCluster{ + {Name: "member-a"}, + {Name: "member-b"}, + }, + } +} + +func fleetMemberNameConfigMap(memberName string) *corev1.ConfigMap { + return &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: "cluster-name", Namespace: "kube-system"}, + Data: map[string]string{"name": memberName}, + } +} + func TestEnsureProvidedSecret(t *testing.T) { ctx := context.Background() ddb := baseDocumentDB("ddb-prov", "default") @@ -145,6 +165,127 @@ func TestEnsureSelfSignedCert(t *testing.T) { require.NotEmpty(t, ddb.Status.TLS.SecretName) } +func TestReconcileCertificatesDoesNotManagePostgresCertificates(t *testing.T) { + ctx := context.Background() + ddb := baseDocumentDB("ddb-pg", "default") + enableAzureFleetReplication(ddb) + r := buildCertificateReconciler(t, ddb, fleetMemberNameConfigMap("member-a")) + + res, err := r.reconcileCertificates(ctx, ddb) + require.NoError(t, err) + require.Equal(t, RequeueAfterShort, res.RequeueAfter) + + issuer := &cmapi.Issuer{} + err = r.Client.Get(ctx, types.NamespacedName{Name: "ddb-pg-postgres-selfsigned", Namespace: "default"}, issuer) + require.True(t, errors.IsNotFound(err)) + + caCert := &cmapi.Certificate{} + err = r.Client.Get(ctx, types.NamespacedName{Name: "ddb-pg-postgres-ca", Namespace: "default"}, caCert) + require.True(t, errors.IsNotFound(err)) + + serverCert := &cmapi.Certificate{} + err = r.Client.Get(ctx, types.NamespacedName{Name: "ddb-pg-postgres-server", Namespace: "default"}, serverCert) + require.True(t, errors.IsNotFound(err)) + + replicationCert := &cmapi.Certificate{} + err = r.Client.Get(ctx, types.NamespacedName{Name: "ddb-pg-postgres-replication", Namespace: "default"}, replicationCert) + require.True(t, errors.IsNotFound(err)) + + gatewayIssuer := &cmapi.Issuer{} + require.NoError(t, r.Client.Get(ctx, types.NamespacedName{Name: "ddb-pg-gateway-selfsigned", Namespace: "default"}, gatewayIssuer)) +} + +func TestEnsurePostgresCertificatesWithPartialPostgresTLSDoesNotFillDefaults(t *testing.T) { + ctx := context.Background() + ddb := baseDocumentDB("ddb-pg-provided", "default") + enableAzureFleetReplication(ddb) + ddb.Spec.TLS = &dbpreview.TLSConfiguration{ + Postgres: &v1.CertificatesConfiguration{ServerCASecret: "provided-server-ca"}, + } + r := buildCertificateReconciler(t, ddb, fleetMemberNameConfigMap("member-a")) + + res, err := r.reconcileCertificates(ctx, ddb) + require.NoError(t, err) + require.Equal(t, RequeueAfterShort, res.RequeueAfter) + + replicationCert := &cmapi.Certificate{} + err = r.Client.Get(ctx, types.NamespacedName{Name: "ddb-pg-provided-postgres-replication", Namespace: "default"}, replicationCert) + require.True(t, errors.IsNotFound(err)) +} + +func TestEnsurePostgresCertificatesAzureFleetWithProvidedCertsSkipsPostgresResources(t *testing.T) { + ctx := context.Background() + ddb := baseDocumentDB("ddb-pg-explicit", "default") + enableAzureFleetReplication(ddb) + ddb.Spec.TLS = &dbpreview.TLSConfiguration{ + Postgres: &v1.CertificatesConfiguration{ + ServerCASecret: "provided-server-ca", + ServerTLSSecret: "provided-server-tls", + ReplicationTLSSecret: "provided-replication-tls", + ClientCASecret: "provided-client-ca", + }, + } + r := buildCertificateReconciler(t, ddb, fleetMemberNameConfigMap("member-a")) + + res, err := r.reconcileCertificates(ctx, ddb) + require.NoError(t, err) + require.Equal(t, RequeueAfterShort, res.RequeueAfter) + + postgresIssuer := &cmapi.Issuer{} + err = r.Client.Get(ctx, types.NamespacedName{Name: "ddb-pg-explicit-postgres-selfsigned", Namespace: "default"}, postgresIssuer) + require.True(t, errors.IsNotFound(err)) + + providedCert := &cmapi.Certificate{} + err = r.Client.Get(ctx, types.NamespacedName{Name: "provided-server-ca", Namespace: "default"}, providedCert) + require.True(t, errors.IsNotFound(err)) +} + +func TestEnsurePostgresCertificatesIstioSkipsPostgresResources(t *testing.T) { + ctx := context.Background() + ddb := baseDocumentDB("ddb-pg-istio", "default") + ddb.Spec.ClusterReplication = &dbpreview.ClusterReplication{ + CrossCloudNetworkingStrategy: string(util.Istio), + Primary: "member-a", + ClusterList: []dbpreview.MemberCluster{ + {Name: "member-a"}, + {Name: "member-b"}, + }, + } + ddb.Spec.TLS = &dbpreview.TLSConfiguration{ + Postgres: &v1.CertificatesConfiguration{}, + } + r := buildCertificateReconciler(t, ddb, fleetMemberNameConfigMap("member-a")) + + res, err := r.reconcileCertificates(ctx, ddb) + require.NoError(t, err) + require.Equal(t, RequeueAfterShort, res.RequeueAfter) + + postgresIssuer := &cmapi.Issuer{} + err = r.Client.Get(ctx, types.NamespacedName{Name: "ddb-pg-istio-postgres-selfsigned", Namespace: "default"}, postgresIssuer) + require.True(t, errors.IsNotFound(err)) +} + +func TestEnsurePostgresCertificatesOmittedSkipsPostgresResources(t *testing.T) { + ctx := context.Background() + ddb := baseDocumentDB("ddb-pg-omitted", "default") + r := buildCertificateReconciler(t, ddb) + + res, err := r.reconcileCertificates(ctx, ddb) + require.NoError(t, err) + require.Equal(t, RequeueAfterShort, res.RequeueAfter) + + postgresIssuer := &cmapi.Issuer{} + err = r.Client.Get(ctx, types.NamespacedName{Name: "ddb-pg-omitted-postgres-selfsigned", Namespace: "default"}, postgresIssuer) + require.True(t, errors.IsNotFound(err)) + + postgresCert := &cmapi.Certificate{} + err = r.Client.Get(ctx, types.NamespacedName{Name: "ddb-pg-omitted-postgres-ca", Namespace: "default"}, postgresCert) + require.True(t, errors.IsNotFound(err)) + + gatewayIssuer := &cmapi.Issuer{} + require.NoError(t, r.Client.Get(ctx, types.NamespacedName{Name: "ddb-pg-omitted-gateway-selfsigned", Namespace: "default"}, gatewayIssuer)) +} + // TestEmptyModeDefaultsToSelfSigned verifies that when mode is empty, // the controller treats it as SelfSigned to ensure TLS is always enabled. // This is a security fix - see https://github.com/documentdb/documentdb-kubernetes-operator/issues/356 diff --git a/operator/src/internal/controller/physical_replication.go b/operator/src/internal/controller/physical_replication.go index 668117bd1..827fdbd85 100644 --- a/operator/src/internal/controller/physical_replication.go +++ b/operator/src/internal/controller/physical_replication.go @@ -8,6 +8,7 @@ import ( "fmt" "io" "net/http" + "reflect" "slices" "strings" "time" @@ -107,6 +108,23 @@ func (r *DocumentDBReconciler) AddClusterReplicationToClusterSpec( Self: replicationContext.CNPGClusterName, } + // If we are multi-cluster and no certs are provided, trust that the DocumentDB cluster is running in a secure environment and allow replication without TLS + // (probably istio or some other service mesh) + postgresReplicationTLSSecret := "" + postgresServerCASecret := "" + if cnpgCluster.Spec.Certificates != nil { + postgresReplicationTLSSecret = cnpgCluster.Spec.Certificates.ReplicationTLSSecret + postgresServerCASecret = cnpgCluster.Spec.Certificates.ServerCASecret + } + postgresClientCertificateProvided := postgresReplicationTLSSecret != "" + postgresServerCAProvided := postgresServerCASecret != "" + if documentdb.Spec.ClusterReplication.DisableTLS { + cnpgCluster.Spec.PostgresConfiguration.PgHBA = []string{ + "host all all localhost trust", + "host replication streaming_replica all trust", + } + } + if replicationContext.IsAzureFleetNetworking() { // need to create services for each of the other clusters cnpgCluster.Spec.Managed = &cnpgv1.ManagedConfiguration{ @@ -139,15 +157,46 @@ func (r *DocumentDBReconciler) AddClusterReplicationToClusterSpec( }, } for clusterName, serviceName := range replicationContext.GenerateExternalClusterServices(documentdb.Name, documentdb.Namespace, replicationContext.IsAzureFleetNetworking()) { - cnpgCluster.Spec.ExternalClusters = append(cnpgCluster.Spec.ExternalClusters, cnpgv1.ExternalCluster{ - Name: clusterName, - ConnectionParameters: map[string]string{ - "host": serviceName, - "port": "5432", - "dbname": "postgres", - "user": "postgres", - }, - }) + connectionParameters := map[string]string{ + "host": serviceName, + "port": "5432", + "dbname": "postgres", + "user": "streaming_replica", + } + if postgresClientCertificateProvided { + connectionParameters["sslmode"] = "require" + if postgresServerCAProvided { + connectionParameters["sslmode"] = "verify-full" + } + } + + externalCluster := cnpgv1.ExternalCluster{ + Name: clusterName, + ConnectionParameters: connectionParameters, + } + if postgresClientCertificateProvided { + externalCluster.SSLCert = &corev1.SecretKeySelector{ + LocalObjectReference: corev1.LocalObjectReference{ + Name: postgresReplicationTLSSecret, + }, + Key: "tls.crt", + } + externalCluster.SSLKey = &corev1.SecretKeySelector{ + LocalObjectReference: corev1.LocalObjectReference{ + Name: postgresReplicationTLSSecret, + }, + Key: "tls.key", + } + } + if postgresClientCertificateProvided && postgresServerCAProvided { + externalCluster.SSLRootCert = &corev1.SecretKeySelector{ + LocalObjectReference: corev1.LocalObjectReference{ + Name: postgresServerCASecret, + }, + Key: "ca.crt", + } + } + cnpgCluster.Spec.ExternalClusters = append(cnpgCluster.Spec.ExternalClusters, externalCluster) } return nil @@ -363,11 +412,8 @@ func (r *DocumentDBReconciler) syncReplicationChanges(ctx context.Context, curre log.Log.Info("Clearing stale promotionToken", "cluster", current.Name) } - // Update if the cluster list has changed - replicasChanged := externalClusterNamesChanged(current.Spec.ExternalClusters, desired.Spec.ExternalClusters) - if replicasChanged { - getReplicasChangePatchOps(&patchOps, desired, replicationContext) - } + // Update if replication connection entries or their PgHBA rules have changed. + getReplicasChangePatchOps(&patchOps, current, desired, replicationContext) return patchOps, nil, -1 } @@ -483,50 +529,49 @@ func (r *DocumentDBReconciler) getPrimaryChangePatchOps(ctx context.Context, pat return nil, -1 } -func externalClusterNamesChanged(currentClusters, desiredClusters []cnpgv1.ExternalCluster) bool { - if len(currentClusters) != len(desiredClusters) { - return true - } - - if len(currentClusters) == 0 { - return false - } - - nameSet := make(map[string]bool, len(currentClusters)) - for _, cluster := range currentClusters { - nameSet[cluster.Name] = true - } - - for _, cluster := range desiredClusters { - if found := nameSet[cluster.Name]; !found { - return true - } - delete(nameSet, cluster.Name) +func getReplicasChangePatchOps(patchOps *[]cnpg.JSONPatch, current, desired *cnpgv1.Cluster, replicationContext *util.ReplicationContext) { + externalClusterSpecChanged := !reflect.DeepEqual(current.Spec.ExternalClusters, desired.Spec.ExternalClusters) + if externalClusterSpecChanged { + *patchOps = append(*patchOps, cnpg.JSONPatch{ + Op: cnpg.PatchOpReplace, + Path: cnpg.PatchPathExternalClusters, + Value: desired.Spec.ExternalClusters, + }) } - - return len(nameSet) != 0 -} - -func getReplicasChangePatchOps(patchOps *[]cnpg.JSONPatch, desired *cnpgv1.Cluster, replicationContext *util.ReplicationContext) { - *patchOps = append(*patchOps, cnpg.JSONPatch{ - Op: cnpg.PatchOpReplace, - Path: cnpg.PatchPathExternalClusters, - Value: desired.Spec.ExternalClusters, - }) - if replicationContext.IsAzureFleetNetworking() { + if !reflect.DeepEqual(current.Spec.PostgresConfiguration.PgHBA, desired.Spec.PostgresConfiguration.PgHBA) { *patchOps = append(*patchOps, cnpg.JSONPatch{ Op: cnpg.PatchOpReplace, - Path: cnpg.PatchPathManagedServices, - Value: desired.Spec.Managed.Services.Additional, + Path: cnpg.PatchPathPostgresPgHBA, + Value: desired.Spec.PostgresConfiguration.PgHBA, }) } - if replicationContext.IsPrimary() { + if externalClusterSpecChanged && replicationContext.IsAzureFleetNetworking() { *patchOps = append(*patchOps, cnpg.JSONPatch{ Op: cnpg.PatchOpReplace, - Path: cnpg.PatchPathSynchronous, - Value: desired.Spec.PostgresConfiguration.Synchronous, + Path: cnpg.PatchPathManagedServices, + Value: desired.Spec.Managed.Services.Additional, }) } + if externalClusterSpecChanged && replicationContext.IsPrimary() { + currentSynchronous := current.Spec.PostgresConfiguration.Synchronous + desiredSynchronous := desired.Spec.PostgresConfiguration.Synchronous + if !reflect.DeepEqual(currentSynchronous, desiredSynchronous) { + synchronousPatch := cnpg.JSONPatch{ + Path: cnpg.PatchPathSynchronous, + Value: desiredSynchronous, + } + switch { + case currentSynchronous == nil: + synchronousPatch.Op = cnpg.PatchOpAdd + case desiredSynchronous == nil: + synchronousPatch.Op = cnpg.PatchOpRemove + synchronousPatch.Value = nil + default: + synchronousPatch.Op = cnpg.PatchOpReplace + } + *patchOps = append(*patchOps, synchronousPatch) + } + } } func (r *DocumentDBReconciler) ReadToken(ctx context.Context, documentdb *dbpreview.DocumentDB, replicationContext *util.ReplicationContext) (string, error, time.Duration) { diff --git a/operator/src/internal/controller/physical_replication_test.go b/operator/src/internal/controller/physical_replication_test.go index 50cd65978..2db056e17 100644 --- a/operator/src/internal/controller/physical_replication_test.go +++ b/operator/src/internal/controller/physical_replication_test.go @@ -653,4 +653,591 @@ var _ = Describe("Physical Replication", func() { Expect(updated.Spec.ExternalClusters).To(HaveLen(3)) Expect(updated.Spec.PostgresConfiguration.Synchronous.Number).To(Equal(2)) }) + + It("applies external cluster detail changes for non-HA primary without synchronous config", func() { + ctx := context.Background() + namespace := "default" + + documentdb := baseDocumentDB("docdb-repl-nonha", namespace) + documentdb.Spec.ClusterReplication = &dbpreview.ClusterReplication{ + CrossCloudNetworkingStrategy: string(util.None), + Primary: documentdb.Name, + ClusterList: []dbpreview.MemberCluster{ + {Name: documentdb.Name}, + {Name: "member-2"}, + }, + } + + current := &cnpgv1.Cluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "docdb-repl-nonha", + Namespace: namespace, + }, + Spec: cnpgv1.ClusterSpec{ + ReplicaCluster: &cnpgv1.ReplicaClusterConfiguration{ + Self: documentdb.Name, + Primary: documentdb.Name, + Source: documentdb.Name, + }, + ExternalClusters: []cnpgv1.ExternalCluster{ + { + Name: documentdb.Name, + ConnectionParameters: map[string]string{ + "host": documentdb.Name + "-rw." + namespace + ".svc", + "port": "5432", + "dbname": "postgres", + "user": "postgres", + }, + }, + { + Name: "member-2", + ConnectionParameters: map[string]string{ + "host": "member-2-rw." + namespace + ".svc", + "port": "5432", + "dbname": "postgres", + "user": "postgres", + }, + }, + }, + }, + } + + desired := current.DeepCopy() + desired.Spec.ExternalClusters[1].ConnectionParameters["user"] = "streaming_replica" + desired.Spec.ExternalClusters[1].ConnectionParameters["sslmode"] = "verify-full" + desired.Spec.ExternalClusters[1].SSLCert = &corev1.SecretKeySelector{ + LocalObjectReference: corev1.LocalObjectReference{Name: "replication-tls"}, + Key: "tls.crt", + } + desired.Spec.ExternalClusters[1].SSLKey = &corev1.SecretKeySelector{ + LocalObjectReference: corev1.LocalObjectReference{Name: "replication-tls"}, + Key: "tls.key", + } + desired.Spec.ExternalClusters[1].SSLRootCert = &corev1.SecretKeySelector{ + LocalObjectReference: corev1.LocalObjectReference{Name: "server-ca"}, + Key: "ca.crt", + } + + reconciler := buildDocumentDBReconciler(current) + replicationContext, err := util.GetReplicationContext(ctx, reconciler.Client, *documentdb) + Expect(err).ToNot(HaveOccurred()) + + patchOps, err, requeue := reconciler.syncReplicationChanges(ctx, current, desired, documentdb, replicationContext) + Expect(err).ToNot(HaveOccurred()) + Expect(requeue).To(Equal(time.Duration(-1))) + Expect(current.Spec.PostgresConfiguration.Synchronous).To(BeNil()) + Expect(desired.Spec.PostgresConfiguration.Synchronous).To(BeNil()) + for _, op := range patchOps { + Expect(op.Path).ToNot(Equal(cnpg.PatchPathSynchronous)) + } + + syncErr := cnpg.SyncCnpgCluster(ctx, reconciler.Client, current, desired, patchOps) + Expect(syncErr).ToNot(HaveOccurred()) + + updated := &cnpgv1.Cluster{} + Expect(reconciler.Client.Get(ctx, types.NamespacedName{Name: current.Name, Namespace: namespace}, updated)).To(Succeed()) + Expect(updated.Spec.ExternalClusters).To(Equal(desired.Spec.ExternalClusters)) + Expect(updated.Spec.PostgresConfiguration.Synchronous).To(BeNil()) + }) + + It("applies synchronous config when HA primary gains synchronous replica configuration", func() { + ctx := context.Background() + namespace := "default" + + // Current: HA primary with no synchronous config + current := &cnpgv1.Cluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "docdb-ha-primary", + Namespace: namespace, + }, + Spec: cnpgv1.ClusterSpec{ + Instances: 1, + ReplicaCluster: &cnpgv1.ReplicaClusterConfiguration{ + Self: "primary-cluster", + }, + ExternalClusters: []cnpgv1.ExternalCluster{ + {Name: "standby-1"}, + {Name: "standby-2"}, + }, + }, + } + + // Desired: HA primary WITH synchronous config and additional external cluster + desired := current.DeepCopy() + desired.Spec.ExternalClusters = append(desired.Spec.ExternalClusters, + cnpgv1.ExternalCluster{Name: "standby-3"}) + desired.Spec.PostgresConfiguration = cnpgv1.PostgresConfiguration{ + Synchronous: &cnpgv1.SynchronousReplicaConfiguration{ + Method: cnpgv1.SynchronousReplicaConfigurationMethodAny, + Number: 2, + }, + } + + documentdb := &dbpreview.DocumentDB{} + reconciler := buildDocumentDBReconciler(current) + replicationContext := &util.ReplicationContext{ + OtherCNPGClusterNames: []string{"standby-1", "standby-2", "standby-3"}, + } + + patchOps, err, requeue := reconciler.syncReplicationChanges(ctx, current, desired, documentdb, replicationContext) + Expect(err).ToNot(HaveOccurred()) + Expect(requeue).To(Equal(time.Duration(-1))) + // Should include patches for external clusters and synchronous config + Expect(len(patchOps)).To(BeNumerically(">", 0)) + }) + + It("removes synchronous config when HA primary replication is downgraded", func() { + ctx := context.Background() + namespace := "default" + + // Current: HA primary WITH synchronous config + current := &cnpgv1.Cluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "docdb-ha-primary-sync", + Namespace: namespace, + }, + Spec: cnpgv1.ClusterSpec{ + Instances: 1, + ReplicaCluster: &cnpgv1.ReplicaClusterConfiguration{ + Self: "primary-cluster", + }, + ExternalClusters: []cnpgv1.ExternalCluster{ + {Name: "standby-1"}, + }, + PostgresConfiguration: cnpgv1.PostgresConfiguration{ + Synchronous: &cnpgv1.SynchronousReplicaConfiguration{ + Method: cnpgv1.SynchronousReplicaConfigurationMethodAny, + Number: 1, + }, + }, + }, + } + + // Desired: HA primary WITHOUT synchronous config, with different external cluster + desired := current.DeepCopy() + desired.Spec.ExternalClusters = []cnpgv1.ExternalCluster{ + {Name: "standby-2"}, + } + desired.Spec.PostgresConfiguration.Synchronous = nil + + documentdb := &dbpreview.DocumentDB{} + reconciler := buildDocumentDBReconciler(current) + replicationContext := &util.ReplicationContext{ + OtherCNPGClusterNames: []string{"standby-2"}, + } + + patchOps, err, requeue := reconciler.syncReplicationChanges(ctx, current, desired, documentdb, replicationContext) + Expect(err).ToNot(HaveOccurred()) + Expect(requeue).To(Equal(time.Duration(-1))) + // Should include patches for external clusters and synchronous config removal + Expect(len(patchOps)).To(BeNumerically(">", 0)) + }) + + It("handles synchronous replica config change on HA primary", func() { + ctx := context.Background() + namespace := "default" + + // Current: HA primary with synchronous config + current := &cnpgv1.Cluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "docdb-ha-primary-sync-update", + Namespace: namespace, + }, + Spec: cnpgv1.ClusterSpec{ + Instances: 1, + ReplicaCluster: &cnpgv1.ReplicaClusterConfiguration{ + Self: "primary-cluster", + }, + ExternalClusters: []cnpgv1.ExternalCluster{ + {Name: "standby-1"}, + {Name: "standby-2"}, + }, + PostgresConfiguration: cnpgv1.PostgresConfiguration{ + Synchronous: &cnpgv1.SynchronousReplicaConfiguration{ + Method: cnpgv1.SynchronousReplicaConfigurationMethodAny, + Number: 1, + }, + }, + }, + } + + // Desired: HA primary with DIFFERENT synchronous config and different external cluster + desired := current.DeepCopy() + desired.Spec.ExternalClusters = []cnpgv1.ExternalCluster{ + {Name: "standby-3"}, + {Name: "standby-4"}, + } + desired.Spec.PostgresConfiguration.Synchronous = &cnpgv1.SynchronousReplicaConfiguration{ + Method: cnpgv1.SynchronousReplicaConfigurationMethodAny, + Number: 2, + } + + documentdb := &dbpreview.DocumentDB{} + reconciler := buildDocumentDBReconciler(current) + replicationContext := &util.ReplicationContext{ + OtherCNPGClusterNames: []string{"standby-3", "standby-4"}, + } + + patchOps, err, requeue := reconciler.syncReplicationChanges(ctx, current, desired, documentdb, replicationContext) + Expect(err).ToNot(HaveOccurred()) + Expect(requeue).To(Equal(time.Duration(-1))) + // Synchronous config change + external cluster changes should generate patches + Expect(len(patchOps)).To(BeNumerically(">", 0)) + }) + + It("propagates external cluster changes with PgHBA updates", func() { + ctx := context.Background() + namespace := "default" + + documentdb := baseDocumentDB("docdb-repl-pghba", namespace) + documentdb.Spec.ClusterReplication = &dbpreview.ClusterReplication{ + CrossCloudNetworkingStrategy: string(util.None), + Primary: documentdb.Name, + ClusterList: []dbpreview.MemberCluster{ + {Name: documentdb.Name}, + {Name: "member-2"}, + }, + } + + current := &cnpgv1.Cluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "docdb-repl-pghba", + Namespace: namespace, + }, + Spec: cnpgv1.ClusterSpec{ + ReplicaCluster: &cnpgv1.ReplicaClusterConfiguration{ + Self: documentdb.Name, + Primary: documentdb.Name, + Source: documentdb.Name, + }, + ExternalClusters: []cnpgv1.ExternalCluster{ + { + Name: documentdb.Name, + ConnectionParameters: map[string]string{ + "host": documentdb.Name + "-rw." + namespace + ".svc", + }, + }, + }, + PostgresConfiguration: cnpgv1.PostgresConfiguration{ + PgHBA: []string{ + "host all all localhost trust", + }, + }, + }, + } + + desired := current.DeepCopy() + desired.Spec.ExternalClusters = append(desired.Spec.ExternalClusters, cnpgv1.ExternalCluster{ + Name: "member-2", + ConnectionParameters: map[string]string{ + "host": "member-2-rw." + namespace + ".svc", + }, + }) + + reconciler := buildDocumentDBReconciler(current) + replicationContext, err := util.GetReplicationContext(ctx, reconciler.Client, *documentdb) + Expect(err).ToNot(HaveOccurred()) + + patchOps, err, requeue := reconciler.syncReplicationChanges(ctx, current, desired, documentdb, replicationContext) + Expect(err).ToNot(HaveOccurred()) + Expect(requeue).To(Equal(time.Duration(-1))) + + // Should have patch operations when external clusters change + Expect(patchOps).ToNot(BeEmpty()) + + syncErr := cnpg.SyncCnpgCluster(ctx, reconciler.Client, current, desired, patchOps) + Expect(syncErr).ToNot(HaveOccurred()) + + updated := &cnpgv1.Cluster{} + Expect(reconciler.Client.Get(ctx, types.NamespacedName{Name: current.Name, Namespace: namespace}, updated)).To(Succeed()) + // Verify clusters were added + Expect(updated.Spec.ExternalClusters).To(HaveLen(2)) + Expect(updated.Spec.ExternalClusters[1].Name).To(Equal("member-2")) + }) + + It("applies synchronous config changes for HA primary on external cluster changes", func() { + ctx := context.Background() + namespace := "default" + + documentdb := baseDocumentDB("docdb-repl-ha-sync", namespace) + documentdb.Spec.ClusterReplication = &dbpreview.ClusterReplication{ + CrossCloudNetworkingStrategy: string(util.None), + Primary: documentdb.Name, + ClusterList: []dbpreview.MemberCluster{ + {Name: documentdb.Name}, + {Name: "member-2"}, + }, + } + + current := &cnpgv1.Cluster{ + ObjectMeta: metav1.ObjectMeta{ + Name: "docdb-repl-ha-sync", + Namespace: namespace, + }, + Spec: cnpgv1.ClusterSpec{ + ReplicaCluster: &cnpgv1.ReplicaClusterConfiguration{ + Self: documentdb.Name, + Primary: documentdb.Name, + Source: documentdb.Name, + }, + ExternalClusters: []cnpgv1.ExternalCluster{ + { + Name: documentdb.Name, + ConnectionParameters: map[string]string{ + "host": documentdb.Name + "-rw." + namespace + ".svc", + }, + }, + }, + }, + } + + desired := current.DeepCopy() + desired.Spec.ExternalClusters = append(desired.Spec.ExternalClusters, cnpgv1.ExternalCluster{ + Name: "member-2", + ConnectionParameters: map[string]string{ + "host": "member-2-rw." + namespace + ".svc", + }, + }) + + reconciler := buildDocumentDBReconciler(current) + replicationContext, err := util.GetReplicationContext(ctx, reconciler.Client, *documentdb) + Expect(err).ToNot(HaveOccurred()) + + patchOps, err, requeue := reconciler.syncReplicationChanges(ctx, current, desired, documentdb, replicationContext) + Expect(err).ToNot(HaveOccurred()) + Expect(requeue).To(Equal(time.Duration(-1))) + + syncErr := cnpg.SyncCnpgCluster(ctx, reconciler.Client, current, desired, patchOps) + Expect(syncErr).ToNot(HaveOccurred()) + + updated := &cnpgv1.Cluster{} + Expect(reconciler.Client.Get(ctx, types.NamespacedName{Name: current.Name, Namespace: namespace}, updated)).To(Succeed()) + // Verify clusters were added + Expect(updated.Spec.ExternalClusters).To(HaveLen(2)) + }) +}) + +var _ = Describe("AddClusterReplicationToClusterSpec - cert management fields", func() { + // Helper to build a minimal cnpgCluster suitable for AddClusterReplicationToClusterSpec. + buildCnpgCluster := func(name, namespace string) *cnpgv1.Cluster { + return &cnpgv1.Cluster{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace}, + Spec: cnpgv1.ClusterSpec{ + InheritedMetadata: &cnpgv1.EmbeddedObjectMetadata{ + Labels: map[string]string{}, + }, + }, + } + } + + // Helper to build a ReplicationContext in primary state (zero value state == NoReplication which + // satisfies IsPrimary()) with two remote cluster members, using the None networking strategy so + // no service import/export objects are required. + buildPrimaryReplicationContext := func(name string, tlsSecret, caSecret string) *util.ReplicationContext { + return &util.ReplicationContext{ + CNPGClusterName: name + "-local", + OtherCNPGClusterNames: []string{name + "-remote-a", name + "-remote-b"}, + PrimaryCNPGClusterName: name + "-local", + CrossCloudNetworkingStrategy: util.None, + } + } + + It("uses provided certificate names when all Postgres cert secrets are provided", func() { + ctx := context.Background() + namespace := "default" + + documentdb := baseDocumentDB("docdb-cert-provided", namespace) + documentdb.Spec.ClusterReplication = &dbpreview.ClusterReplication{ + CrossCloudNetworkingStrategy: string(util.None), + Primary: "cluster-a", + ClusterList: []dbpreview.MemberCluster{ + {Name: "cluster-a"}, + {Name: "cluster-b"}, + }, + } + documentdb.Spec.TLS = &dbpreview.TLSConfiguration{ + Postgres: &cnpgv1.CertificatesConfiguration{ + ServerCASecret: "provided-server-ca", + ClientCASecret: "provided-client-ca", + ServerTLSSecret: "provided-server-tls", + ReplicationTLSSecret: "provided-replication-tls", + }, + } + + cnpgCluster := buildCnpgCluster("docdb-cert-provided", namespace) + cnpgCluster.Spec.Certificates = documentdb.Spec.TLS.Postgres + replicationContext := buildPrimaryReplicationContext("docdb-cert-provided", "", "") + + reconciler := buildDocumentDBReconciler() + Expect(reconciler.AddClusterReplicationToClusterSpec(ctx, documentdb, replicationContext, cnpgCluster)).To(Succeed()) + + Expect(cnpgCluster.Spec.Certificates).ToNot(BeNil()) + Expect(cnpgCluster.Spec.Certificates.ServerCASecret).To(Equal("provided-server-ca")) + Expect(cnpgCluster.Spec.Certificates.ClientCASecret).To(Equal("provided-client-ca")) + Expect(cnpgCluster.Spec.Certificates.ServerTLSSecret).To(Equal("provided-server-tls")) + Expect(cnpgCluster.Spec.Certificates.ReplicationTLSSecret).To(Equal("provided-replication-tls")) + Expect(cnpgCluster.Spec.Certificates.ServerAltDNSNames).To(BeEmpty()) + // Self + two remote external clusters + Expect(cnpgCluster.Spec.ExternalClusters).To(HaveLen(3)) + for _, ec := range cnpgCluster.Spec.ExternalClusters { + if ec.Name == replicationContext.CNPGClusterName { + // Self cluster still uses the superuser for self-loopback. + Expect(ec.ConnectionParameters["user"]).To(Equal("postgres")) + continue + } + // External (remote) clusters use the dedicated replication user with generated TLS material. + Expect(ec.ConnectionParameters["user"]).To(Equal("streaming_replica")) + Expect(ec.ConnectionParameters).To(HaveKeyWithValue("sslmode", "verify-full")) + Expect(ec.SSLCert.Name).To(Equal("provided-replication-tls")) + Expect(ec.SSLKey.Name).To(Equal("provided-replication-tls")) + Expect(ec.SSLRootCert.Name).To(Equal("provided-server-ca")) + } + }) + + It("uses provided replication client certificate with sslmode require when server CA is omitted", func() { + ctx := context.Background() + namespace := "default" + + documentdb := baseDocumentDB("docdb-cert-partial", namespace) + documentdb.Spec.ClusterReplication = &dbpreview.ClusterReplication{ + CrossCloudNetworkingStrategy: string(util.None), + Primary: "cluster-a", + ClusterList: []dbpreview.MemberCluster{ + {Name: "cluster-a"}, + {Name: "cluster-b"}, + }, + } + documentdb.Spec.TLS = &dbpreview.TLSConfiguration{ + Postgres: &cnpgv1.CertificatesConfiguration{ + ReplicationTLSSecret: "replication-tls", + ClientCASecret: "replication-tls", + }, + } + + cnpgCluster := buildCnpgCluster("docdb-cert-partial", namespace) + cnpgCluster.Spec.Certificates = documentdb.Spec.TLS.Postgres + replicationContext := buildPrimaryReplicationContext("docdb-cert-partial", "", "") + + reconciler := buildDocumentDBReconciler() + Expect(reconciler.AddClusterReplicationToClusterSpec(ctx, documentdb, replicationContext, cnpgCluster)).To(Succeed()) + + Expect(cnpgCluster.Spec.Certificates).ToNot(BeNil()) + for _, ec := range cnpgCluster.Spec.ExternalClusters { + if ec.Name == replicationContext.CNPGClusterName { + continue + } + Expect(ec.ConnectionParameters).To(HaveKeyWithValue("sslmode", "require")) + Expect(ec.SSLCert.Name).To(Equal("replication-tls")) + Expect(ec.SSLKey.Name).To(Equal("replication-tls")) + Expect(ec.SSLRootCert).To(BeNil()) + } + }) + + It("uses the server CA as the external cluster root certificate", func() { + ctx := context.Background() + namespace := "default" + + documentdb := baseDocumentDB("docdb-distinct-ca", namespace) + documentdb.Spec.ClusterReplication = &dbpreview.ClusterReplication{ + CrossCloudNetworkingStrategy: string(util.None), + Primary: "cluster-a", + ClusterList: []dbpreview.MemberCluster{ + {Name: "cluster-a"}, + {Name: "cluster-b"}, + }, + } + documentdb.Spec.TLS = &dbpreview.TLSConfiguration{ + Postgres: &cnpgv1.CertificatesConfiguration{ + ReplicationTLSSecret: "cross-region-client-cert", + ClientCASecret: "cross-region-client-cert", + ServerTLSSecret: "cross-region-server-cert", + ServerCASecret: "cross-region-server-cert", + }, + } + + cnpgCluster := buildCnpgCluster("docdb-distinct-ca", namespace) + cnpgCluster.Spec.Certificates = documentdb.Spec.TLS.Postgres + replicationContext := buildPrimaryReplicationContext("docdb-distinct-ca", "", "") + + reconciler := buildDocumentDBReconciler() + Expect(reconciler.AddClusterReplicationToClusterSpec(ctx, documentdb, replicationContext, cnpgCluster)).To(Succeed()) + + for _, ec := range cnpgCluster.Spec.ExternalClusters { + if ec.Name == replicationContext.CNPGClusterName { + continue + } + Expect(ec.SSLCert.Name).To(Equal("cross-region-client-cert")) + Expect(ec.SSLKey.Name).To(Equal("cross-region-client-cert")) + Expect(ec.SSLRootCert.Name).To(Equal("cross-region-server-cert")) + } + }) + + It("omits certificate configuration and TLS external cluster refs when Postgres TLS is omitted", func() { + ctx := context.Background() + namespace := "default" + + documentdb := baseDocumentDB("docdb-cert-omitted", namespace) + documentdb.Spec.ClusterReplication = &dbpreview.ClusterReplication{ + CrossCloudNetworkingStrategy: string(util.Istio), + Primary: "cluster-a", + DisableTLS: true, + ClusterList: []dbpreview.MemberCluster{ + {Name: "cluster-a"}, + {Name: "cluster-b"}, + }, + } + + cnpgCluster := buildCnpgCluster("docdb-cert-omitted", namespace) + replicationContext := buildPrimaryReplicationContext("docdb-cert-omitted", "", "") + + reconciler := buildDocumentDBReconciler() + Expect(reconciler.AddClusterReplicationToClusterSpec(ctx, documentdb, replicationContext, cnpgCluster)).To(Succeed()) + + Expect(cnpgCluster.Spec.Certificates).To(BeNil()) + Expect(cnpgCluster.Spec.PostgresConfiguration.PgHBA).To(Equal([]string{ + "host all all localhost trust", + "host replication streaming_replica all trust", + })) + Expect(cnpgCluster.Spec.ExternalClusters).To(HaveLen(3)) + for _, ec := range cnpgCluster.Spec.ExternalClusters { + if ec.Name == replicationContext.CNPGClusterName { + continue + } + Expect(ec.ConnectionParameters).NotTo(HaveKey("sslmode")) + Expect(ec.SSLCert).To(BeNil()) + Expect(ec.SSLKey).To(BeNil()) + Expect(ec.SSLRootCert).To(BeNil()) + } + }) + + It("does not downgrade PgHBA to trust when disableTLS is false and Postgres TLS is omitted", func() { + ctx := context.Background() + namespace := "default" + + documentdb := baseDocumentDB("docdb-cert-omitted-default", namespace) + documentdb.Spec.ClusterReplication = &dbpreview.ClusterReplication{ + CrossCloudNetworkingStrategy: string(util.Istio), + Primary: "cluster-a", + ClusterList: []dbpreview.MemberCluster{ + {Name: "cluster-a"}, + {Name: "cluster-b"}, + }, + } + + cnpgCluster := buildCnpgCluster("docdb-cert-omitted-default", namespace) + cnpgCluster.Spec.PostgresConfiguration.PgHBA = []string{ + "host all all localhost trust", + "hostssl replication streaming_replica all cert", + } + replicationContext := buildPrimaryReplicationContext("docdb-cert-omitted-default", "", "") + + reconciler := buildDocumentDBReconciler() + Expect(reconciler.AddClusterReplicationToClusterSpec(ctx, documentdb, replicationContext, cnpgCluster)).To(Succeed()) + + Expect(cnpgCluster.Spec.Certificates).To(BeNil()) + Expect(cnpgCluster.Spec.PostgresConfiguration.PgHBA).To(Equal([]string{ + "host all all localhost trust", + "hostssl replication streaming_replica all cert", + })) + }) })