From a1d6eafdd313372b86d9211e8b8bee031df8d09f Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Wed, 22 Jul 2026 16:52:35 +0100 Subject: [PATCH 1/9] docs: add architecture diagrams, 2/2/2 layout, and failure guidance Chooser page: - Add a topology comparison diagram (multi-AZ, Stretch Cluster, Shadowing) above the comparison table - Add a Stretch Cluster vs Shadowing comparison table (replication, RPO, RTO, client failover, topology, produce latency, best for) - Add Tiered Storage and relative cost rows to the topology table, concrete produce-latency figures, and a Whole Cluster Restore pointer Stretch page: - Add an architecture diagram separating the operator control plane from the broker data plane, with split-brain protection and two-level leader election explained - Recommend a 2/2/2 six-broker layout (survives location loss and restores full RF=5 replication after a single broker failure); validated end-to-end on kind - Add a Limitations section: no Tiered Storage (and no Iceberg topics or Cloud Topics), same-continent placement with example region trios - Add an Understand failure scenarios section (one location down, two or more down, beyond quorum with Shadowing as defense in depth) - Document spec-drift behavior (SpecSynced condition, DriftDetected) with a troubleshooting entry, the fixed peer list and how to change it, managed EKS/GKE/AKS support, plugin auto-download, kubeconfig caching, ordered_racks leader preference with follower fetching, and the operator's automatic stale-disk wipe recovery --- .../kubernetes/k-choose-deployment.adoc | 90 +++++++++++++++- .../kubernetes/k-stretch-clusters.adoc | 100 +++++++++++++++--- 2 files changed, 169 insertions(+), 21 deletions(-) diff --git a/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc b/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc index ba087731f9..58b99ea60c 100644 --- a/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc +++ b/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc @@ -33,6 +33,43 @@ Weigh the options against these dimensions: == Compare topologies +.The three topologies at a glance +[mermaid] +---- +graph TB + subgraph single["Single cluster, multi-AZ: zonal resilience"] + direction LR + sa["Broker
AZ 1"] <--> sb["Broker
AZ 2"] <--> sc["Broker
AZ 3"] + end + subgraph stretch["Stretch Cluster: one cluster across Kubernetes clusters"] + direction LR + subgraph k1["Kubernetes cluster 1"] + t1["2 brokers"] + end + subgraph k2["Kubernetes cluster 2"] + t2["2 brokers"] + end + subgraph k3["Kubernetes cluster 3"] + t3["2 brokers"] + end + t1 <-->|"synchronous Raft"| t2 + t2 <--> t3 + end + subgraph shadow["Independent clusters with Shadowing: disaster recovery"] + direction LR + subgraph src["Source cluster"] + p1["Brokers"] + end + subgraph dst["Shadow cluster"] + p2["Brokers"] + end + p1 -->|"asynchronous shadow link"| p2 + end + single ~~~ stretch ~~~ shadow +---- + +A single cluster replicates synchronously between brokers, whether those brokers share one Kubernetes cluster (multi-AZ) or span several (Stretch Cluster). Shadowing instead copies data asynchronously between two independent clusters. + [cols="1h,2,2,2"] |=== | | Single cluster, multi-AZ | Stretch Cluster (one cluster across Kubernetes clusters) | Independent clusters with asynchronous replication (Shadowing) @@ -53,9 +90,19 @@ Weigh the options against these dimensions: | Runbook-driven: fail over shadow topics, then redirect clients | Latency requirement -| Low inter-AZ latency -| Low inter-cluster latency (under 50 ms recommended); requires WAN tuning -| None; any distance +| Low inter-AZ latency; produce round trips stay in the single-digit milliseconds +| Low inter-cluster latency (under 50 ms recommended, which in practice means same-continent locations); produce pays a cross-location round trip of tens of milliseconds; requires WAN tuning +| None; any distance; produce stays local because replication is out of band + +| Relative cost +| $$ +| $$$: cross-location replication traffic on every write +| $$: two clusters, but the shadow can be provisioned for recovery rather than full production load + +| Tiered Storage +| Supported +| Not supported (including Iceberg topics and Cloud Topics) +| Supported | License | Community @@ -68,7 +115,7 @@ Weigh the options against these dimensions: | xref:manage:disaster-recovery/shadowing/overview.adoc[] |=== -For read-only offloading rather than disaster recovery, see xref:manage:kubernetes/k-remote-read-replicas.adoc[Remote Read Replicas]. +For read-only offloading rather than disaster recovery, see xref:manage:kubernetes/k-remote-read-replicas.adoc[Remote Read Replicas]. For the lowest-cost recovery option, xref:manage:kubernetes/tiered-storage/k-whole-cluster-restore.adoc[Whole Cluster Restore] rebuilds a cluster from Tiered Storage in the same region, trading minutes of RPO and a high RTO for the price of object storage. == Quick picks @@ -81,6 +128,41 @@ For read-only offloading rather than disaster recovery, see xref:manage:kubernet Stretch Clusters and Shadowing solve different problems and can complement each other. A Stretch Cluster is one logical cluster with synchronous replication and a single set of offsets; Shadowing asynchronously replicates data and metadata between two independent clusters for disaster recovery, protecting against events that affect a whole cluster, including some it cannot survive synchronously. +If you are deciding between the two, compare them dimension by dimension: + +[cols="1h,2,2"] +|=== +| |Stretch Cluster |Shadowing + +|Replication +|Synchronous: the Raft quorum spans locations. +|Asynchronous: data is copied after the write is acknowledged. + +|RPO +|Zero: acknowledged writes survive the loss of a location. +|Greater than zero: a replication-lag window always exists. + +|RTO +|Near zero: leaders re-elect automatically, with no cutover. +|Minutes: you promote the shadow cluster and repoint clients. + +|Client failover +|No re-bootstrap: clients keep one broker list and reconnect on their own. +|Re-bootstrap required: you repoint clients to the promoted cluster. Offsets are preserved, so consumers resume where they left off. + +|Topology +|One logical cluster with one set of offsets. +|Two independent clusters: source and shadow. + +|Produce latency +|Cross-location round trip on every acknowledged write. +|Local: replication happens out of band. + +|Best for +|Workloads that cannot lose an acknowledged write. +|Disaster recovery where seconds of data loss is acceptable, or any deployment that spans distances too large for synchronous replication. +|=== + A Stretch Cluster can act as the shadow (target) cluster of a shadow link. Using a Stretch Cluster as the source cluster of a shadow link is not supported. == Next steps diff --git a/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc b/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc index 0db34edf3a..8826861191 100644 --- a/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc +++ b/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc @@ -31,6 +31,46 @@ A Stretch Cluster deployment includes: * *Multicluster operators*: One Redpanda Operator in each Kubernetes cluster. The operators coordinate through Raft consensus to manage the Stretch Cluster as a single unit. The rpk commands for this operator plane live under xref:reference:rpk/rpk-k8s/rpk-k8s-multicluster.adoc[`rpk k8s multicluster`]. * *Cross-cluster networking*: Direct broker-to-broker communication across Kubernetes clusters for Raft replication. +.Stretch Cluster architecture +[mermaid] +---- +graph TB + subgraph kc1["Kubernetes cluster 1"] + direction TB + o1["Redpanda Operator"] + b1["RedpandaBrokerPool
2 brokers"] + end + subgraph kc2["Kubernetes cluster 2"] + direction TB + o2["Redpanda Operator"] + b2["RedpandaBrokerPool
2 brokers"] + end + subgraph kc3["Kubernetes cluster 3"] + direction TB + o3["Redpanda Operator"] + b3["RedpandaBrokerPool
2 brokers"] + end + o1 <-. "operator Raft group
TLS, port 9443" .-> o2 + o2 <-. " " .-> o3 + o1 <-. " " .-> o3 + b1 <--> |"data replication
Raft, port 33145"| b2 + b2 <--> b3 + b1 <--> b3 +---- + +The operators form the control plane: they coordinate through their own Raft group over TLS and manage the StretchCluster as a single unit. The brokers form the data plane: they replicate partition data directly with each other across clusters, without going through the operators. + +This design protects the control plane against split brain. Every operator is installed with the same fixed peer list (`multicluster.peers`), and the operators elect a single leader through Raft, which drives reconciliation for the whole Stretch Cluster. During a network partition, only the side that holds a majority of the peer list can elect a leader and reconcile changes; operators on the minority side stop reconciling until the partition heals. Because the operators are not on the data path, brokers continue to serve traffic while reconciliation is paused. + +Leader election happens at two levels. Within each Kubernetes cluster, operator replicas elect a local leader through a standard Kubernetes Lease, so you can run multiple operator replicas per cluster for high availability. Across clusters, only each cluster's local leader participates in the Raft group, representing its cluster with a single identity. + +NOTE: The peer list is fixed at installation time. To add or remove a participating cluster, rerun the bootstrap command with the new set of contexts, then upgrade every operator installation with the updated `multicluster.peers` list and restart the operator Deployments. + +== Limitations + +* *Tiered Storage is not supported* on Stretch Clusters in this release. This also means that features that depend on Tiered Storage are unavailable, including xref:manage:iceberg/about-iceberg-topics.adoc[Iceberg topics] and xref:manage:kubernetes/k-cloud-topics.adoc[Cloud Topics]. +* *Keep all locations on the same continent*. Transoceanic deployments are technically possible but produce latency degrades with distance, and pairwise latencies over the documented budgets cause broker liveness failures. See <>. + == Prerequisites Before deploying a Stretch Cluster, ensure you have: @@ -45,14 +85,16 @@ Before deploying a Stretch Cluster, ensure you have: ---- rpk k8s install ---- ++ +The `k8s` plugin is a managed rpk plugin: it also downloads automatically, with checksum verification, the first time you run any `rpk k8s` command. -* *At least three Kubernetes clusters, running a xref:upgrade:k-compatibility.adoc#compatibility-matrix[supported Kubernetes version]*: The operators form a Raft group that needs a quorum of `⌊n/2⌋ + 1` members to elect a leader. With three clusters, the deployment tolerates the loss of one cluster. With only two clusters, the loss of either cluster stops the operator control plane, so two-cluster deployments are not recommended. +* *At least three Kubernetes clusters, running a xref:upgrade:k-compatibility.adoc#compatibility-matrix[supported Kubernetes version]*: The operators form a Raft group that needs a quorum of `⌊n/2⌋ + 1` members to elect a leader. With three clusters, the deployment tolerates the loss of one cluster. With only two clusters, the loss of either cluster stops the operator control plane, so two-cluster deployments are not recommended. Stretch Clusters work on managed Kubernetes services, including Amazon EKS, Google GKE, and Azure AKS, as well as self-managed clusters. * *Cross-cluster networking*: One of the networking modes described in <>. * *Network requirements*: ** Pod-to-pod connectivity between all clusters. -** Inter-cluster network latency under 50 ms for optimal performance. +** Inter-cluster network latency under 50 ms for optimal performance. In practice, this means placing all Kubernetes clusters on the same continent, for example a same-continent region trio such as AWS us-east-1, us-east-2, and us-west-2, or GCP us-east1, us-central1, and us-west1. Measure the pairwise round-trip times between your candidate regions before deploying. Transoceanic links typically exceed this budget; for replication across greater distances, use xref:manage:disaster-recovery/shadowing/overview.adoc[Shadowing] instead. ** Sufficient bandwidth for cross-cluster replication traffic. * *Kubernetes API access*: Each operator must be able to reach the Kubernetes API servers of all peer clusters, and you need a kubeconfig with contexts for all participating clusters to run the bootstrap command. @@ -90,7 +132,7 @@ You can stretch a cluster across Kubernetes clusters in different cloud provider * *Site-to-site VPN connectivity between all clouds*: A full mesh of IPsec VPN tunnels between the provider networks, for example AWS VPN Gateway, Google Cloud HA VPN, and Azure VPN Gateway, with BGP or static routes advertising each cloud's CIDR ranges. Node IPs in every cloud must be routable from the others. * *Non-overlapping CIDR ranges* for the VPC or VNet networks and for the pod networks in every cloud. -* *Pairwise latency under 100 ms* between all regions. Broker liveness checks fail at higher latencies, even with the WAN tuning shown in this page's StretchCluster example. +* *Pairwise latency under 100 ms* between all regions, which means choosing regions on the same continent. Broker liveness checks fail at higher latencies, even with the WAN tuning shown in this page's StretchCluster example. * *A CNI that supports cross-cluster networking*, such as Cilium with Cluster Mesh (`mesh` mode), installed in place of each cloud's default CNI. Avoid double encryption: if the VPN layer already encrypts all cross-cloud traffic, don't also enable CNI-level node encryption, such as WireGuard, on the same path. @@ -150,7 +192,7 @@ This command: + * Generates an ECDSA P-256 CA, valid for 10 years, and an individual certificate for each operator, valid for one year. * Provisions a LoadBalancer Service named `-multicluster-peer` in each cluster for operator peer communication on port 9443, and includes the resolved LoadBalancer addresses in the certificate Subject Alternative Names (SANs). -* Creates the TLS and kubeconfig Secrets that the operators read at startup, in the given namespace on each cluster. +* Creates the TLS and kubeconfig Secrets that the operators read at startup, in the given namespace on each cluster. Every operator caches kubeconfigs for all peer clusters, so a newly elected leader can manage resources in the other clusters immediately. + When the command completes, it prints a `multicluster.peers` block with each cluster's name and peer address. Use these values in the Helm installation. + @@ -284,7 +326,7 @@ The `users.txt` key defines the cluster's superusers in `username:password:mecha . Create a StretchCluster resource. + -Save the following to `stretchcluster.yaml`. The spec must be identical on every cluster, so that the operators' consistency check passes. +Save the following to `stretchcluster.yaml`. You apply this same manifest to every participating cluster, and the spec must be identical everywhere: the operators compare the spec across clusters before reconciling, and if any copy differs, they block changes until the specs are aligned. See <>. + [,yaml] ---- @@ -317,8 +359,8 @@ spec: key: license.key config: cluster: - # Five replicas spread over a 2/2/1 broker layout keep a Raft - # majority through the loss of any one location. + # Five replicas spread over three locations keep a Raft majority + # through the loss of any one location. default_topic_replications: 5 # Wait 10 min before relocating replicas off an unavailable broker # to avoid cross-region rebalancing on brief network blips. @@ -349,7 +391,7 @@ done + Each RedpandaBrokerPool defines a group of brokers in one Kubernetes cluster and binds to the StretchCluster through `spec.clusterRef`. -Redpanda Data recommends five brokers distributed 2/2/1 across the three Kubernetes clusters, with `default_topic_replications: 5`. With this layout, every partition and the cluster's Raft groups keep a majority through the loss of any one location, including either two-broker location, and the single-broker location acts as a tiebreaker. Save the following to `brokerpool-cluster1.yaml`: +Redpanda Data recommends six brokers distributed 2/2/2 across the three Kubernetes clusters, with `default_topic_replications: 5`. With five replicas over three locations, at most two replicas of any partition share a location, so every partition and the cluster's Raft groups keep a majority through the loss of any one location. The sixth broker also lets the cluster restore full replication after a single broker failure: five brokers remain, which is enough to host all five replicas of every partition. Save the following to `brokerpool-cluster1.yaml`: + [,yaml,subs="attributes+"] ---- @@ -363,7 +405,7 @@ spec: group: cluster.redpanda.com kind: StretchCluster name: cluster - replicas: 2 # 2/2/1 across the three clusters + replicas: 2 # 2/2/2 across the three clusters image: # <2> repository: docker.redpanda.com/redpandadata/redpanda tag: {latest-redpanda-tag} @@ -388,7 +430,7 @@ spec: For all RedpandaBrokerPool fields, see the xref:reference:k-crd.adoc#k8s-api-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-redpandabrokerpool[RedpandaBrokerPool reference]. + -Apply each pool to its own cluster, changing `metadata.name` for each one and setting `replicas` to 2, 2, and 1: +Apply each pool to its own cluster, changing only `metadata.name` for each one: + [,bash] ---- @@ -452,6 +494,16 @@ kubectl exec --namespace redpanda --context -it - + The output lists the brokers from all pools across all Kubernetes clusters. If you deployed without SASL, omit the `-X` flags. +== Understand failure scenarios + +With the recommended 2/2/2 layout and `default_topic_replications: 5`, at most two of any partition's five replicas share a location, so the deployment behaves as follows: + +* *One location down*: Every partition keeps a three-of-five majority, leaders re-elect automatically (along your leader preference, if set), and clients keep working with zero data loss. After the configured timeouts, continuous partition balancing decommissions the unreachable brokers and restores full replication on the survivors (see <>). When the location returns, its brokers rejoin as fresh brokers and pinned leaders migrate back. + +* *Two or more locations down*: Partitions lose their Raft majority and the cluster stops serving writes, and the operator control plane loses quorum and stops reconciling. This is an availability outage, not data loss: acknowledged writes persist on the surviving disks, and the cluster resumes when a location is restored. If a lost location cannot be restored, contact Redpanda Support before attempting recovery. + +* *Beyond quorum loss*: A Stretch Cluster cannot survive the simultaneous loss of a majority of its locations. For protection against failures of that scale, or for an isolated recovery copy, use xref:manage:disaster-recovery/shadowing/overview.adoc[Shadowing] between independent clusters. Failing over to a shadow cluster costs a replication-lag window (an RPO greater than zero) and requires repointing clients. For how the two features combine, see xref:deploy:redpanda/kubernetes/k-choose-deployment.adoc#stretch-clusters-and-shadowing[Stretch Clusters and Shadowing]. + == Optimize for region failures To keep a Stretch Cluster performing predictably through a region outage, enable rack awareness on every broker pool, pin partition leaders near your clients, and let Redpanda eject brokers that are permanently gone. @@ -470,7 +522,7 @@ spec: nodeAnnotation: topology.kubernetes.io/zone # <2> ---- <1> Rack awareness requires per-pool RBAC, and the operator Helm chart must be installed with `rbac.createAdditionalControllerCRs=true`. -<2> The Kubernetes Node label whose value becomes each broker's rack ID. The default is `topology.kubernetes.io/zone`, which gives each availability zone its own rack. To treat each Kubernetes cluster or region as one rack, use a label whose value is shared by all nodes in that cluster. +<2> The Kubernetes Node label whose value becomes each broker's rack ID. The default is `topology.kubernetes.io/zone`, which gives each availability zone its own rack. To treat each Kubernetes cluster or region as one rack, use a label whose value is shared by all nodes in that cluster, such as `topology.kubernetes.io/region`. === Pin partition leaders near your clients @@ -481,11 +533,11 @@ Without a leader preference, partition leaders can move to any region after a fa spec: config: cluster: - default_leaders_preference: "racks:," # <1> + default_leaders_preference: "ordered_racks:," # <1> ---- -<1> One or more rack IDs, which are the values of the node label configured in `rackAwareness.nodeAnnotation`. With multiple racks, Redpanda balances leaders across them. +<1> Rack IDs in priority order, which are the values of the node label configured in `rackAwareness.nodeAnnotation`. Redpanda places leaders in the first listed rack when it's available and fails over down the list in order. `ordered_racks` requires Redpanda 26.1 or later. To balance leaders across racks without a priority order, use `racks:,` instead. -Leaders fail over to other regions when the preferred racks are unavailable and move back automatically when they recover. For details, see xref:develop:produce-data/leader-pinning.adoc[]. +Leaders fail over along the preference list when the preferred racks are unavailable and move back automatically when they recover. To let consumers read from replicas in their own region instead of fetching from a leader in another region, pair leader pinning with xref:develop:consume-data/follower-fetching.adoc[follower fetching]. For details, see xref:develop:produce-data/leader-pinning.adoc[]. === Automatically eject ghost brokers @@ -608,6 +660,20 @@ kubectl get redpandabrokerpool --namespace redpanda \ kubectl logs --namespace redpanda -l app.kubernetes.io/name=operator | grep -i "forbidden\|unauthorized" ---- +=== StretchCluster specs differ across clusters + +If you change the StretchCluster on one cluster but not the others, the operators detect the drift and block all changes until the specs are aligned. While blocked, the operators still remediate local failures, such as restarting downed pods, but do not apply any spec changes. + +The drift is reported in the StretchCluster's `SpecSynced` condition and in the operator logs: + +[,bash] +---- +kubectl get stretchcluster --namespace redpanda --context \ + -o jsonpath='{.status.conditions[?(@.type=="SpecSynced")].message}' +---- + +The message names the clusters and the top-level fields that differ, for example `StretchCluster .spec differs on clusters: (fields: config)`. To recover, apply the same manifest to every cluster. + === Bootstrap user password mismatch If the StretchCluster doesn't deploy brokers and the operator logs show `bootstrap user password mismatch`, the operators created the internal `-bootstrap-user` Secret independently with different passwords. This can happen when the StretchCluster manifest is applied to all clusters at nearly the same time. @@ -638,7 +704,9 @@ done If a region comes back online after its broker was automatically decommissioned (see <>), the returning broker cannot rejoin with its old identity: the pod starts but never becomes ready, and its logs show that the broker was removed from the cluster. -Delete the broker's PersistentVolumeClaim and Pod so it rejoins as a new broker with a fresh ID: +The operator recovers from this automatically. When the pod stays not-ready past a threshold (five minutes by default) and its on-disk identity conflicts with the cluster's authoritative broker list, the operator wipes the stale disk by deleting the pod's PersistentVolumeClaim and Pod. The broker reschedules onto a fresh disk, joins with a new broker ID, and continuous balancing moves partition replicas back onto it. To change the threshold, set the operator's `--wipe-stale-disk-after` flag; a negative duration disables the wipe. + +To recover immediately, or if you disabled the automatic wipe, delete the broker's PersistentVolumeClaim and Pod yourself: [,bash] ---- @@ -646,8 +714,6 @@ kubectl delete pvc --namespace redpanda --context --namespace redpanda --context ---- -The broker joins with a new broker ID, and continuous balancing moves partition replicas back onto it. - === Brokers cannot communicate across clusters If brokers in different Kubernetes clusters cannot reach each other: From 7fb10527cec3de1ff3b682779ca53e60d97bb484 Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Wed, 22 Jul 2026 17:40:18 +0100 Subject: [PATCH 2/9] docs: scope RedpandaBrokerPool to Stretch Clusters Per engineering (K8S-907 discussion): in this release the RedpandaBrokerPool CRD is installed only in multicluster mode and its controller reconciles pools only for StretchCluster resources; NodePool (beta) remains the pool mechanism for standalone clusters. - Replace the release-notes section that claimed NodePool documentation was removed (the NodePool docs are still published) with an accurate 'Broker pool resources by deployment type' section - Add a Limitations bullet on the stretch page pointing standalone deployments at NodePool Verified against operator source at operator/v26.2.1-beta.3: multiclusterCRDs gating in cmd/crd/crd.go and the StretchCluster-only clusterRef resolution in brokerpool_controller.go --- .../pages/redpanda/kubernetes/k-stretch-clusters.adoc | 1 + modules/get-started/pages/release-notes/operator.adoc | 6 ++++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc b/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc index 8826861191..712b8159a5 100644 --- a/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc +++ b/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc @@ -68,6 +68,7 @@ NOTE: The peer list is fixed at installation time. To add or remove a participat == Limitations +* *RedpandaBrokerPool works only with StretchCluster resources*. The CRD is installed only in multicluster mode, and standalone Redpanda resources cannot use it. To manage broker pools in a standalone cluster, use the beta xref:manage:kubernetes/k-node-pools.adoc[NodePool resource] instead. * *Tiered Storage is not supported* on Stretch Clusters in this release. This also means that features that depend on Tiered Storage are unavailable, including xref:manage:iceberg/about-iceberg-topics.adoc[Iceberg topics] and xref:manage:kubernetes/k-cloud-topics.adoc[Cloud Topics]. * *Keep all locations on the same continent*. Transoceanic deployments are technically possible but produce latency degrades with distance, and pairwise latencies over the documented budgets cause broker liveness failures. See <>. diff --git a/modules/get-started/pages/release-notes/operator.adoc b/modules/get-started/pages/release-notes/operator.adoc index 3fd2de832c..c22d39ede0 100644 --- a/modules/get-started/pages/release-notes/operator.adoc +++ b/modules/get-started/pages/release-notes/operator.adoc @@ -22,9 +22,11 @@ The release also adds the xref:reference:rpk/rpk-k8s/rpk-k8s.adoc[`rpk k8s` plug For deployment steps, see xref:deploy:redpanda/kubernetes/k-stretch-clusters.adoc[]. -=== NodePool CRD documentation removed +=== Broker pool resources by deployment type -The NodePool custom resource (beta in v26.1.x) is intended for internal use in Bring Your Own Cloud (BYOC) deployments and is no longer documented for self-managed deployments. The CRD remains available and actively maintained, and existing configurations continue to work. +The new RedpandaBrokerPool custom resource manages broker pools for Stretch Clusters only. The RedpandaBrokerPool CRD is installed only in multicluster mode, and its controller does not reconcile pools for standalone Redpanda resources. + +For standalone clusters, the xref:manage:kubernetes/k-node-pools.adoc[NodePool custom resource] (beta) remains the way to manage broker pools. Existing NodePool configurations continue to work. RedpandaBrokerPool support for standalone clusters is planned for a later release. == Redpanda Operator v26.1.x From f32a83d9431f098f2356fff413fb95f8100fb9ee Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Wed, 22 Jul 2026 17:54:11 +0100 Subject: [PATCH 3/9] docs: apply standards review fixes to stretch follow-up - Fix the Cloud Topics xref, which moved under cloud-topics/ on beta - Spell out RPO on first use on the stretch page - Rework the Limitations bullets to the page's bold-term-colon pattern - Tie the 50 ms recommendation to the 100 ms liveness ceiling in one sentence and drop a doubled example marker - Replace 'as well as' with 'and'; use active voice for the peer-list sentence - Chooser: parallel phrasing in the Relative cost row, clearer Whole Cluster Restore trade-off sentence, table spacing consistency --- .../redpanda/kubernetes/k-choose-deployment.adoc | 6 +++--- .../redpanda/kubernetes/k-stretch-clusters.adoc | 14 +++++++------- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc b/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc index 58b99ea60c..58d7c7dc83 100644 --- a/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc +++ b/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc @@ -95,7 +95,7 @@ A single cluster replicates synchronously between brokers, whether those brokers | None; any distance; produce stays local because replication is out of band | Relative cost -| $$ +| $$: one cluster, intra-region traffic only | $$$: cross-location replication traffic on every write | $$: two clusters, but the shadow can be provisioned for recovery rather than full production load @@ -115,7 +115,7 @@ A single cluster replicates synchronously between brokers, whether those brokers | xref:manage:disaster-recovery/shadowing/overview.adoc[] |=== -For read-only offloading rather than disaster recovery, see xref:manage:kubernetes/k-remote-read-replicas.adoc[Remote Read Replicas]. For the lowest-cost recovery option, xref:manage:kubernetes/tiered-storage/k-whole-cluster-restore.adoc[Whole Cluster Restore] rebuilds a cluster from Tiered Storage in the same region, trading minutes of RPO and a high RTO for the price of object storage. +For read-only offloading rather than disaster recovery, see xref:manage:kubernetes/k-remote-read-replicas.adoc[Remote Read Replicas]. For the lowest-cost recovery option, xref:manage:kubernetes/tiered-storage/k-whole-cluster-restore.adoc[Whole Cluster Restore] rebuilds a cluster from Tiered Storage in the same region, accepting minutes of data loss (RPO) and a high recovery time (RTO) in exchange for the lower cost of object storage. == Quick picks @@ -132,7 +132,7 @@ If you are deciding between the two, compare them dimension by dimension: [cols="1h,2,2"] |=== -| |Stretch Cluster |Shadowing +| | Stretch Cluster | Shadowing |Replication |Synchronous: the Raft quorum spans locations. diff --git a/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc b/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc index 712b8159a5..7c1cffa4e4 100644 --- a/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc +++ b/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc @@ -60,7 +60,7 @@ graph TB The operators form the control plane: they coordinate through their own Raft group over TLS and manage the StretchCluster as a single unit. The brokers form the data plane: they replicate partition data directly with each other across clusters, without going through the operators. -This design protects the control plane against split brain. Every operator is installed with the same fixed peer list (`multicluster.peers`), and the operators elect a single leader through Raft, which drives reconciliation for the whole Stretch Cluster. During a network partition, only the side that holds a majority of the peer list can elect a leader and reconcile changes; operators on the minority side stop reconciling until the partition heals. Because the operators are not on the data path, brokers continue to serve traffic while reconciliation is paused. +This design protects the control plane against split brain. You install every operator with the same fixed peer list (`multicluster.peers`), and the operators elect a single leader through Raft, which drives reconciliation for the whole Stretch Cluster. During a network partition, only the side that holds a majority of the peer list can elect a leader and reconcile changes; operators on the minority side stop reconciling until the partition heals. Because the operators are not on the data path, brokers continue to serve traffic while reconciliation is paused. Leader election happens at two levels. Within each Kubernetes cluster, operator replicas elect a local leader through a standard Kubernetes Lease, so you can run multiple operator replicas per cluster for high availability. Across clusters, only each cluster's local leader participates in the Raft group, representing its cluster with a single identity. @@ -68,9 +68,9 @@ NOTE: The peer list is fixed at installation time. To add or remove a participat == Limitations -* *RedpandaBrokerPool works only with StretchCluster resources*. The CRD is installed only in multicluster mode, and standalone Redpanda resources cannot use it. To manage broker pools in a standalone cluster, use the beta xref:manage:kubernetes/k-node-pools.adoc[NodePool resource] instead. -* *Tiered Storage is not supported* on Stretch Clusters in this release. This also means that features that depend on Tiered Storage are unavailable, including xref:manage:iceberg/about-iceberg-topics.adoc[Iceberg topics] and xref:manage:kubernetes/k-cloud-topics.adoc[Cloud Topics]. -* *Keep all locations on the same continent*. Transoceanic deployments are technically possible but produce latency degrades with distance, and pairwise latencies over the documented budgets cause broker liveness failures. See <>. +* *RedpandaBrokerPool scope*: RedpandaBrokerPool works only with StretchCluster resources. The CRD is installed only in multicluster mode, and standalone Redpanda resources cannot use it. To manage broker pools in a standalone cluster, use the beta xref:manage:kubernetes/k-node-pools.adoc[NodePool resource] instead. +* *Tiered Storage*: Not supported on Stretch Clusters in this release. This also means that features that depend on Tiered Storage are unavailable, including xref:manage:iceberg/about-iceberg-topics.adoc[Iceberg topics] and xref:manage:kubernetes/cloud-topics/k-cloud-topics.adoc[Cloud Topics]. +* *Distance*: Keep all locations on the same continent. Transoceanic deployments are technically possible, but produce latency degrades with distance, and pairwise latencies over the documented budgets cause broker liveness failures. See <>. == Prerequisites @@ -89,13 +89,13 @@ rpk k8s install + The `k8s` plugin is a managed rpk plugin: it also downloads automatically, with checksum verification, the first time you run any `rpk k8s` command. -* *At least three Kubernetes clusters, running a xref:upgrade:k-compatibility.adoc#compatibility-matrix[supported Kubernetes version]*: The operators form a Raft group that needs a quorum of `⌊n/2⌋ + 1` members to elect a leader. With three clusters, the deployment tolerates the loss of one cluster. With only two clusters, the loss of either cluster stops the operator control plane, so two-cluster deployments are not recommended. Stretch Clusters work on managed Kubernetes services, including Amazon EKS, Google GKE, and Azure AKS, as well as self-managed clusters. +* *At least three Kubernetes clusters, running a xref:upgrade:k-compatibility.adoc#compatibility-matrix[supported Kubernetes version]*: The operators form a Raft group that needs a quorum of `⌊n/2⌋ + 1` members to elect a leader. With three clusters, the deployment tolerates the loss of one cluster. With only two clusters, the loss of either cluster stops the operator control plane, so two-cluster deployments are not recommended. Stretch Clusters work on managed Kubernetes services, including Amazon EKS, Google GKE, and Azure AKS, and on self-managed clusters. * *Cross-cluster networking*: One of the networking modes described in <>. * *Network requirements*: ** Pod-to-pod connectivity between all clusters. -** Inter-cluster network latency under 50 ms for optimal performance. In practice, this means placing all Kubernetes clusters on the same continent, for example a same-continent region trio such as AWS us-east-1, us-east-2, and us-west-2, or GCP us-east1, us-central1, and us-west1. Measure the pairwise round-trip times between your candidate regions before deploying. Transoceanic links typically exceed this budget; for replication across greater distances, use xref:manage:disaster-recovery/shadowing/overview.adoc[Shadowing] instead. +** Inter-cluster network latency under 50 ms for optimal performance; broker liveness checks begin to fail above roughly 100 ms, even with WAN tuning. In practice, this means choosing regions on the same continent, for example AWS us-east-1, us-east-2, and us-west-2, or GCP us-east1, us-central1, and us-west1. Measure the pairwise round-trip times between your candidate regions before deploying. Transoceanic links typically exceed this budget; for replication across greater distances, use xref:manage:disaster-recovery/shadowing/overview.adoc[Shadowing] instead. ** Sufficient bandwidth for cross-cluster replication traffic. * *Kubernetes API access*: Each operator must be able to reach the Kubernetes API servers of all peer clusters, and you need a kubeconfig with contexts for all participating clusters to run the bootstrap command. @@ -503,7 +503,7 @@ With the recommended 2/2/2 layout and `default_topic_replications: 5`, at most t * *Two or more locations down*: Partitions lose their Raft majority and the cluster stops serving writes, and the operator control plane loses quorum and stops reconciling. This is an availability outage, not data loss: acknowledged writes persist on the surviving disks, and the cluster resumes when a location is restored. If a lost location cannot be restored, contact Redpanda Support before attempting recovery. -* *Beyond quorum loss*: A Stretch Cluster cannot survive the simultaneous loss of a majority of its locations. For protection against failures of that scale, or for an isolated recovery copy, use xref:manage:disaster-recovery/shadowing/overview.adoc[Shadowing] between independent clusters. Failing over to a shadow cluster costs a replication-lag window (an RPO greater than zero) and requires repointing clients. For how the two features combine, see xref:deploy:redpanda/kubernetes/k-choose-deployment.adoc#stretch-clusters-and-shadowing[Stretch Clusters and Shadowing]. +* *Beyond quorum loss*: A Stretch Cluster cannot survive the simultaneous loss of a majority of its locations. For protection against failures of that scale, or for an isolated recovery copy, use xref:manage:disaster-recovery/shadowing/overview.adoc[Shadowing] between independent clusters. Failing over to a shadow cluster costs a replication-lag window (a recovery point objective, or RPO, greater than zero) and requires repointing clients. For how the two features combine, see xref:deploy:redpanda/kubernetes/k-choose-deployment.adoc#stretch-clusters-and-shadowing[Stretch Clusters and Shadowing]. == Optimize for region failures From a36996ba7b564a81a614dfa4be77f8b5e3d1934b Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Wed, 22 Jul 2026 18:02:15 +0100 Subject: [PATCH 4/9] docs: prefer separate sentences over semicolons --- .../pages/redpanda/kubernetes/k-choose-deployment.adoc | 8 ++++---- .../pages/redpanda/kubernetes/k-stretch-clusters.adoc | 6 +++--- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc b/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc index 58d7c7dc83..4007a43145 100644 --- a/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc +++ b/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc @@ -82,7 +82,7 @@ A single cluster replicates synchronously between brokers, whether those brokers | Replication and RPO | Synchronous (Raft), zero data loss | Synchronous (Raft), zero data loss, one set of offsets -| Asynchronous, near real time, offset-preserving; RPO greater than zero +| Asynchronous, near real time, and offset-preserving, with an RPO greater than zero | Failover | Automatic @@ -90,9 +90,9 @@ A single cluster replicates synchronously between brokers, whether those brokers | Runbook-driven: fail over shadow topics, then redirect clients | Latency requirement -| Low inter-AZ latency; produce round trips stay in the single-digit milliseconds -| Low inter-cluster latency (under 50 ms recommended, which in practice means same-continent locations); produce pays a cross-location round trip of tens of milliseconds; requires WAN tuning -| None; any distance; produce stays local because replication is out of band +| Low inter-AZ latency. Produce round trips stay in the single-digit milliseconds. +| Low inter-cluster latency (under 50 ms recommended, which in practice means same-continent locations). Produce pays a cross-location round trip of tens of milliseconds. Requires WAN tuning. +| None. Produce stays local at any distance because replication is out of band. | Relative cost | $$: one cluster, intra-region traffic only diff --git a/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc b/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc index 7c1cffa4e4..7dfd5b20fe 100644 --- a/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc +++ b/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc @@ -60,7 +60,7 @@ graph TB The operators form the control plane: they coordinate through their own Raft group over TLS and manage the StretchCluster as a single unit. The brokers form the data plane: they replicate partition data directly with each other across clusters, without going through the operators. -This design protects the control plane against split brain. You install every operator with the same fixed peer list (`multicluster.peers`), and the operators elect a single leader through Raft, which drives reconciliation for the whole Stretch Cluster. During a network partition, only the side that holds a majority of the peer list can elect a leader and reconcile changes; operators on the minority side stop reconciling until the partition heals. Because the operators are not on the data path, brokers continue to serve traffic while reconciliation is paused. +This design protects the control plane against split brain. You install every operator with the same fixed peer list (`multicluster.peers`), and the operators elect a single leader through Raft, which drives reconciliation for the whole Stretch Cluster. During a network partition, only the side that holds a majority of the peer list can elect a leader and reconcile changes. Operators on the minority side stop reconciling until the partition heals. Because the operators are not on the data path, brokers continue to serve traffic while reconciliation is paused. Leader election happens at two levels. Within each Kubernetes cluster, operator replicas elect a local leader through a standard Kubernetes Lease, so you can run multiple operator replicas per cluster for high availability. Across clusters, only each cluster's local leader participates in the Raft group, representing its cluster with a single identity. @@ -95,7 +95,7 @@ The `k8s` plugin is a managed rpk plugin: it also downloads automatically, with * *Network requirements*: ** Pod-to-pod connectivity between all clusters. -** Inter-cluster network latency under 50 ms for optimal performance; broker liveness checks begin to fail above roughly 100 ms, even with WAN tuning. In practice, this means choosing regions on the same continent, for example AWS us-east-1, us-east-2, and us-west-2, or GCP us-east1, us-central1, and us-west1. Measure the pairwise round-trip times between your candidate regions before deploying. Transoceanic links typically exceed this budget; for replication across greater distances, use xref:manage:disaster-recovery/shadowing/overview.adoc[Shadowing] instead. +** Inter-cluster network latency under 50 ms for optimal performance. Broker liveness checks begin to fail above roughly 100 ms, even with WAN tuning. In practice, this means choosing regions on the same continent, for example AWS us-east-1, us-east-2, and us-west-2, or GCP us-east1, us-central1, and us-west1. Measure the pairwise round-trip times between your candidate regions before deploying. Transoceanic links typically exceed this budget. For replication across greater distances, use xref:manage:disaster-recovery/shadowing/overview.adoc[Shadowing] instead. ** Sufficient bandwidth for cross-cluster replication traffic. * *Kubernetes API access*: Each operator must be able to reach the Kubernetes API servers of all peer clusters, and you need a kubeconfig with contexts for all participating clusters to run the bootstrap command. @@ -705,7 +705,7 @@ done If a region comes back online after its broker was automatically decommissioned (see <>), the returning broker cannot rejoin with its old identity: the pod starts but never becomes ready, and its logs show that the broker was removed from the cluster. -The operator recovers from this automatically. When the pod stays not-ready past a threshold (five minutes by default) and its on-disk identity conflicts with the cluster's authoritative broker list, the operator wipes the stale disk by deleting the pod's PersistentVolumeClaim and Pod. The broker reschedules onto a fresh disk, joins with a new broker ID, and continuous balancing moves partition replicas back onto it. To change the threshold, set the operator's `--wipe-stale-disk-after` flag; a negative duration disables the wipe. +The operator recovers from this automatically. When the pod stays not-ready past a threshold (five minutes by default) and its on-disk identity conflicts with the cluster's authoritative broker list, the operator wipes the stale disk by deleting the pod's PersistentVolumeClaim and Pod. The broker reschedules onto a fresh disk, joins with a new broker ID, and continuous balancing moves partition replicas back onto it. To change the threshold, set the operator's `--wipe-stale-disk-after` flag. A negative duration disables the wipe. To recover immediately, or if you disabled the automatic wipe, delete the broker's PersistentVolumeClaim and Pod yourself: From ccb582e045747303eee10e0d8ccc624be246c8bd Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Thu, 23 Jul 2026 10:31:56 +0100 Subject: [PATCH 5/9] docs: add breathing room under Mermaid subgraph titles Set flowchart.subGraphTitleMargin (bottom: 16) in the Mermaid initialize options so subgraph labels no longer sit directly on the nested boxes below them. The top margin must stay 0 because Mermaid 10 shifts the label up by the top margin without growing the subgraph box, which clips the title. --- local-antora-playbook.yml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/local-antora-playbook.yml b/local-antora-playbook.yml index 1636d5f304..5967474c32 100644 --- a/local-antora-playbook.yml +++ b/local-antora-playbook.yml @@ -88,6 +88,12 @@ antora: theme_variables: line_color: '#e2401b' font_family: Inter, sans-serif + flowchart: + # top must stay 0: mermaid 10 shifts the label up by the top + # margin without growing the subgraph box, clipping the title + sub_graph_title_margin: + top: 0 + bottom: 16 - require: '@redpanda-data/docs-extensions-and-macros/extensions/collect-bloblang-samples' - require: '@redpanda-data/docs-extensions-and-macros/extensions/generate-rp-connect-categories' - require: '@redpanda-data/docs-extensions-and-macros/extensions/modify-redirects' From d3599d8319746193f473dece4028de813f747a35 Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Thu, 23 Jul 2026 11:22:28 +0100 Subject: [PATCH 6/9] docs: add spacer line under nested-subgraph titles in topology diagram Mermaid 10 and 11 never reserve vertical space for a cluster label above nested subgraphs: the label overlaps the inner box borders, and neither flowchart.subGraphTitleMargin nor flowchart.padding changes the label-to-content distance (measured at -3px for any value). The only working lever is a trailing blank line in the title itself. --- .../deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc b/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc index 4007a43145..892e4e142c 100644 --- a/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc +++ b/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc @@ -41,7 +41,7 @@ graph TB direction LR sa["Broker
AZ 1"] <--> sb["Broker
AZ 2"] <--> sc["Broker
AZ 3"] end - subgraph stretch["Stretch Cluster: one cluster across Kubernetes clusters"] + subgraph stretch["Stretch Cluster: one cluster across Kubernetes clusters
 "] direction LR subgraph k1["Kubernetes cluster 1"] t1["2 brokers"] @@ -55,7 +55,7 @@ graph TB t1 <-->|"synchronous Raft"| t2 t2 <--> t3 end - subgraph shadow["Independent clusters with Shadowing: disaster recovery"] + subgraph shadow["Independent clusters with Shadowing: disaster recovery
 "] direction LR subgraph src["Source cluster"] p1["Brokers"] From 81f41d06b94b53970cc085951fb06ec03d7aaca9 Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Thu, 23 Jul 2026 12:59:06 +0100 Subject: [PATCH 7/9] Revert "docs: add spacer line under nested-subgraph titles in topology diagram" This reverts commit d3599d8319746193f473dece4028de813f747a35. --- .../deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc b/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc index 892e4e142c..4007a43145 100644 --- a/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc +++ b/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc @@ -41,7 +41,7 @@ graph TB direction LR sa["Broker
AZ 1"] <--> sb["Broker
AZ 2"] <--> sc["Broker
AZ 3"] end - subgraph stretch["Stretch Cluster: one cluster across Kubernetes clusters
 "] + subgraph stretch["Stretch Cluster: one cluster across Kubernetes clusters"] direction LR subgraph k1["Kubernetes cluster 1"] t1["2 brokers"] @@ -55,7 +55,7 @@ graph TB t1 <-->|"synchronous Raft"| t2 t2 <--> t3 end - subgraph shadow["Independent clusters with Shadowing: disaster recovery
 "] + subgraph shadow["Independent clusters with Shadowing: disaster recovery"] direction LR subgraph src["Source cluster"] p1["Brokers"] From 6e4e5eff31a8e4c877d4d7be46a9d8dde40c81ec Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Thu, 23 Jul 2026 12:59:08 +0100 Subject: [PATCH 8/9] Revert "docs: add breathing room under Mermaid subgraph titles" This reverts commit ccb582e045747303eee10e0d8ccc624be246c8bd. --- local-antora-playbook.yml | 6 ------ 1 file changed, 6 deletions(-) diff --git a/local-antora-playbook.yml b/local-antora-playbook.yml index 5967474c32..1636d5f304 100644 --- a/local-antora-playbook.yml +++ b/local-antora-playbook.yml @@ -88,12 +88,6 @@ antora: theme_variables: line_color: '#e2401b' font_family: Inter, sans-serif - flowchart: - # top must stay 0: mermaid 10 shifts the label up by the top - # margin without growing the subgraph box, clipping the title - sub_graph_title_margin: - top: 0 - bottom: 16 - require: '@redpanda-data/docs-extensions-and-macros/extensions/collect-bloblang-samples' - require: '@redpanda-data/docs-extensions-and-macros/extensions/generate-rp-connect-categories' - require: '@redpanda-data/docs-extensions-and-macros/extensions/modify-redirects' From 510e702c448b851b4f7c6972818a263069f64925 Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Fri, 24 Jul 2026 18:18:44 +0100 Subject: [PATCH 9/9] docs: revert to the 2/2/1 broker layout and arrange clusters side by side Per review: 2/2/1 with default_topic_replications: 5 keeps an odd broker count, so the cluster always holds a clean three-of-five majority through the loss of any one location, and with five replicas on five brokers every broker holds a copy of every partition, so any surviving location retains a complete copy of the data even if quorum is lost. Also lay the architecture diagram out left-to-right so the three Kubernetes clusters appear side by side with the full cross-cluster mesh visible, instead of stacked vertically. --- .../redpanda/kubernetes/k-choose-deployment.adoc | 2 +- .../redpanda/kubernetes/k-stretch-clusters.adoc | 14 +++++++------- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc b/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc index 4007a43145..1e29ead3d1 100644 --- a/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc +++ b/modules/deploy/pages/redpanda/kubernetes/k-choose-deployment.adoc @@ -50,7 +50,7 @@ graph TB t2["2 brokers"] end subgraph k3["Kubernetes cluster 3"] - t3["2 brokers"] + t3["1 broker"] end t1 <-->|"synchronous Raft"| t2 t2 <--> t3 diff --git a/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc b/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc index 7dfd5b20fe..8da52b5d10 100644 --- a/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc +++ b/modules/deploy/pages/redpanda/kubernetes/k-stretch-clusters.adoc @@ -34,7 +34,7 @@ A Stretch Cluster deployment includes: .Stretch Cluster architecture [mermaid] ---- -graph TB +graph LR subgraph kc1["Kubernetes cluster 1"] direction TB o1["Redpanda Operator"] @@ -48,7 +48,7 @@ graph TB subgraph kc3["Kubernetes cluster 3"] direction TB o3["Redpanda Operator"] - b3["RedpandaBrokerPool
2 brokers"] + b3["RedpandaBrokerPool
1 broker"] end o1 <-. "operator Raft group
TLS, port 9443" .-> o2 o2 <-. " " .-> o3 @@ -392,7 +392,7 @@ done + Each RedpandaBrokerPool defines a group of brokers in one Kubernetes cluster and binds to the StretchCluster through `spec.clusterRef`. -Redpanda Data recommends six brokers distributed 2/2/2 across the three Kubernetes clusters, with `default_topic_replications: 5`. With five replicas over three locations, at most two replicas of any partition share a location, so every partition and the cluster's Raft groups keep a majority through the loss of any one location. The sixth broker also lets the cluster restore full replication after a single broker failure: five brokers remain, which is enough to host all five replicas of every partition. Save the following to `brokerpool-cluster1.yaml`: +Redpanda Data recommends five brokers distributed 2/2/1 across the three Kubernetes clusters, with `default_topic_replications: 5`. The odd broker count means the cluster always keeps a clean three-of-five majority through the loss of any one location, with the single-broker location acting as a tiebreaker. With five replicas on five brokers, every broker also holds a replica of every partition, so any surviving location holds a complete copy of the data even if the cluster loses quorum. Save the following to `brokerpool-cluster1.yaml`: + [,yaml,subs="attributes+"] ---- @@ -406,7 +406,7 @@ spec: group: cluster.redpanda.com kind: StretchCluster name: cluster - replicas: 2 # 2/2/2 across the three clusters + replicas: 2 # 2/2/1 across the three clusters image: # <2> repository: docker.redpanda.com/redpandadata/redpanda tag: {latest-redpanda-tag} @@ -431,7 +431,7 @@ spec: For all RedpandaBrokerPool fields, see the xref:reference:k-crd.adoc#k8s-api-github-com-redpanda-data-redpanda-operator-operator-api-redpanda-v1alpha2-redpandabrokerpool[RedpandaBrokerPool reference]. + -Apply each pool to its own cluster, changing only `metadata.name` for each one: +Apply each pool to its own cluster, changing `metadata.name` for each one and setting `replicas` to 2, 2, and 1: + [,bash] ---- @@ -497,9 +497,9 @@ The output lists the brokers from all pools across all Kubernetes clusters. If y == Understand failure scenarios -With the recommended 2/2/2 layout and `default_topic_replications: 5`, at most two of any partition's five replicas share a location, so the deployment behaves as follows: +With the recommended 2/2/1 layout and `default_topic_replications: 5`, at most two of any partition's five replicas share a location, so the deployment behaves as follows: -* *One location down*: Every partition keeps a three-of-five majority, leaders re-elect automatically (along your leader preference, if set), and clients keep working with zero data loss. After the configured timeouts, continuous partition balancing decommissions the unreachable brokers and restores full replication on the survivors (see <>). When the location returns, its brokers rejoin as fresh brokers and pinned leaders migrate back. +* *One location down*: Every partition keeps a three-of-five majority, leaders re-elect automatically (along your leader preference, if set), and clients keep working with zero data loss. After the configured timeouts, continuous partition balancing decommissions the unreachable brokers (see <>). When the location returns, its brokers rejoin as fresh brokers, replication is fully restored, and pinned leaders migrate back. * *Two or more locations down*: Partitions lose their Raft majority and the cluster stops serving writes, and the operator control plane loses quorum and stops reconciling. This is an availability outage, not data loss: acknowledged writes persist on the surviving disks, and the cluster resumes when a location is restored. If a lost location cannot be restored, contact Redpanda Support before attempting recovery.