diff --git a/modules/upgrade/pages/k-rolling-upgrade.adoc b/modules/upgrade/pages/k-rolling-upgrade.adoc index f625947993..30a523677a 100644 --- a/modules/upgrade/pages/k-rolling-upgrade.adoc +++ b/modules/upgrade/pages/k-rolling-upgrade.adoc @@ -295,9 +295,84 @@ kubectl get console --namespace -o jsonpath='{.spec.w + If warnings are present, review them and manually update your Console configuration as needed. For details about migrating to Console v3, see xref:migrate:console-v3.adoc[]. +== Defer upgrade finalization + +By default, Redpanda finalizes an upgrade automatically as soon as every broker runs the new version: version-gated features activate and downgrade is no longer possible. Deferred finalization keeps the cluster in an unfinalized state after the roll completes, so you can soak test the new version and still <>, even after all brokers are upgraded. This feature requires an xref:get-started:licensing/overview.adoc[Enterprise Edition license]. + +Deferred finalization is available when upgrading to 26.2 or later, and the cluster must be running at least version 26.1.9 or 25.3.15 before the upgrade so that the configuration property is available. + +. Before you upgrade, on the version you are upgrading from, disable automatic finalization: ++ +[,bash] +---- +kubectl exec --namespace -c redpanda -- \ + rpk cluster config set features_auto_finalization false +---- ++ +This is a runtime change to the xref:reference:properties/cluster-properties.adoc#features_auto_finalization[`features_auto_finalization`] cluster property; no restart is required. You can also set it declaratively through your usual xref:manage:kubernetes/k-cluster-property-configuration.adoc[cluster property configuration]. + +. <> as usual. Nothing about the roll changes, and the Redpanda Operator and Helm chart require no special configuration. + +. After all brokers run the new version, verify that the cluster is holding the upgrade open: ++ +[,bash] +---- +kubectl exec --namespace -c redpanda -- \ + rpk cluster upgrade status +---- ++ +The output reports `ready to finalize` while the rollback window is open, along with the active logical version (the downgrade floor) and each broker's reported version. ++ +.Example output +[%collapsible] +==== +[.no-copy] +---- +STATE ready to finalize +ACTIVE VERSION 18 +VERSION AFTER FINALIZATION 19 +AUTO-FINALIZATION ENABLED false + +NODE RELEASE-VERSION LOGICAL-VERSION VERSION-KNOWN ALIVE +0 v26.2.1 19 true true +1 v26.2.1 19 true true +2 v26.2.1 19 true true +---- +==== + +. Soak test the new version. During this window: ++ +* Version-gated features remain inactive, and configuration that depends on them reports that it cannot be configured until the upgrade is finalized. +* To abandon the upgrade, follow <>. Rolling back, re-upgrading, and rolling back again is supported. + +. When you are confident in the new version, finalize the upgrade: ++ +[,bash] +---- +kubectl exec --namespace -c redpanda -- \ + rpk cluster upgrade finalize +---- ++ +Finalization is irreversible: the previous version is no longer available for downgrade. Run `rpk cluster upgrade status` again to confirm the state reports `finalized`. + +. Optionally, restore automatic finalization for future upgrades: ++ +[,bash] +---- +kubectl exec --namespace -c redpanda -- \ + rpk cluster config set features_auto_finalization true +---- + +[NOTE] +==== +* Setting `features_auto_finalization` back to `true` while an upgrade is pending does not reliably finalize it. Use `rpk cluster upgrade finalize`, then restore the property afterward. +* A finalize request is not persisted across controller leadership changes. If leadership moves before the version advances, re-run the command. A successful `finalize` means the request was accepted; poll `rpk cluster upgrade status` to confirm the version advanced. +* Deferred finalization holds back one upgrade at a time. You cannot defer across multiple feature releases. +==== + == Roll back -If something does not go as planned during a rolling upgrade, you can roll back to the original version as long as you have not upgraded all brokers. +If something does not go as planned during a rolling upgrade, you can roll back to the original version as long as you have not upgraded all brokers. With <>, you can roll back even after all brokers are upgraded, at any time before you finalize the upgrade: revert the version change as described in this section, and the cluster returns to normal operation on the previous release with the same data. The StatefulSet uses the `RollingUpdate` strategy by default in xref:reference:k-redpanda-helm-spec.adoc#statefulsetupdatestrategytype[`statefulset.updateStrategy.type`], which means all Pods in the StatefulSet are restarted in reverse-ordinal order. For details, see the https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/#update-strategies[Kubernetes documentation^]. @@ -360,6 +435,19 @@ Leaderless partitions: [] == Troubleshooting +=== Broker crash-loops with Incompatible downgrade detected + +If a broker is downgraded to a previous feature release after the upgrade was finalized, it refuses to start and its logs show: + +[.no-copy] +---- +Incompatible downgrade detected! My version 18, feature table 19 indicates that all nodes in cluster were previously >= that version +---- + +This is a guard rail, not a bug: after finalization, the previous version can no longer read the cluster's state. The StatefulSet rolls brokers one at a time, so the roll halts at the first refused broker and the rest of the cluster stays available. + +To recover, set the image back to the finalized version. The rolling update must pass through the already-updated brokers before it reaches the crash-looping one, so the recovery can take several minutes; deleting the crash-looping Pod does not speed this up, because the StatefulSet recreates it at the old revision until the roll reaches its ordinal. + include::troubleshoot:partial$errors-and-solutions.adoc[tags=deployment] include::shared:partial$suggested-reading.adoc[] diff --git a/modules/upgrade/partials/rolling-upgrades/upgrade-limitations.adoc b/modules/upgrade/partials/rolling-upgrades/upgrade-limitations.adoc index c81a51ba60..ee7fcccda5 100644 --- a/modules/upgrade/partials/rolling-upgrades/upgrade-limitations.adoc +++ b/modules/upgrade/partials/rolling-upgrades/upgrade-limitations.adoc @@ -8,10 +8,10 @@ The following limitations ensure a smooth transition between versions and help m ** You can upgrade only one feature release at a time, for example from 22.2 to 22.3. Skipping feature releases is not supported. ** If you have xref:get-started:licensing/overview.adoc[enterprise features] enabled without a valid license, Redpanda fails to upgrade to new feature releases, starting from version 24.3. -* *Rollbacks*: You can roll back to the original version only if at least one broker is still running the original version (not yet upgraded) and the cluster hasn't yet restarted. +* *Rollbacks*: You can roll back to the original version only if at least one broker is still running the original version (not yet upgraded) and the cluster hasn't yet restarted. Exception: if you defer upgrade finalization (an Enterprise feature available when upgrading to 26.2 or later), you can roll back until you finalize the upgrade, even after all brokers are upgraded. * *Downgrades*: -Downgrades are possible only between patch releases of the same feature release. For example, you can downgrade from 22.2.2 to 22.2.1. Downgrading to previous feature releases, such as 22.1.x, is not supported. +Downgrades are possible only between patch releases of the same feature release. For example, you can downgrade from 22.2.2 to 22.2.1. Downgrading to previous feature releases, such as 22.1.x, is not supported after the upgrade is finalized. If you defer upgrade finalization, you can downgrade to the version you upgraded from until you finalize. * *Tiered Storage*: If you have xref:manage:tiered-storage.adoc[Tiered Storage] enabled and you're upgrading to 23.2, object storage uploads are paused until all brokers are upgraded. If the cluster cannot upgrade, roll it back to the original version.