From d592b6a2be0d96a9df20825bbe8e9363f8232420 Mon Sep 17 00:00:00 2001 From: Amber Alston Date: Mon, 6 Jul 2026 14:11:54 -0600 Subject: [PATCH 1/3] docs: document EC v3 deployment pipeline, Helm hook support, and upgrade failure handling Adds three documentation improvements for EC v3: 1. New "Deployment pipeline" section in the overview documenting the fixed sequential install/upgrade pipeline (k0s -> preflights -> addons -> extensions -> app preflights -> app charts), including built-in addon ordering. 2. Clarified extension chart docs: where extensions sit in the pipeline, weight sorting details (default 0, negative supported, stable alphabetical tiebreak), and that Helm hooks are supported via --wait-for-jobs. 3. New "Upgrade failure handling" section covering retry behavior, Helm chart rollback semantics, and vendor responsibility for DB migrations. Co-Authored-By: Claude Opus 4.6 --- embedded-cluster/embedded-config.mdx | 6 ++++-- embedded-cluster/embedded-overview.mdx | 18 ++++++++++++++++++ embedded-cluster/updating-embedded.mdx | 18 ++++++++++++++++++ 3 files changed, 40 insertions(+), 2 deletions(-) diff --git a/embedded-cluster/embedded-config.mdx b/embedded-cluster/embedded-config.mdx index b30e3bffe9..507692302d 100644 --- a/embedded-cluster/embedded-config.mdx +++ b/embedded-cluster/embedded-config.mdx @@ -180,7 +180,7 @@ spec: ## extensions -Embedded Cluster installs Helm charts before your application as part of Embedded Cluster itself. Specify these charts in `spec.extensions.helmCharts`. Each item describes one chart release. +Embedded Cluster installs Helm chart extensions after built-in addons (OpenEBS, registry) and before application charts. This positions extension charts in the infrastructure layer of the [deployment pipeline](embedded-overview#deployment-pipeline). Specify these charts in `spec.extensions.helmCharts`. Each item describes one chart release. ### helmCharts @@ -191,9 +191,11 @@ Each object in `helmCharts` can include the following fields: * `chartVersion`: Chart version string to install. * `releaseName`: (Required) Helm release name. * `namespace`: (Required) Namespace for the release. -* `weight`: Controls installation order relative to other Helm chart extensions. Lower weights install before higher weights. +* `weight`: Controls installation order relative to other Helm chart extensions. Lower weights install before higher weights. When weights are equal, charts install in alphabetical order by namespace and release name. The default weight is `0`. Negative weights are supported. * `values`: Multi-line string of Helm values. This field supports [Replicated template functions](/reference/template-functions-about). Embedded Cluster updates Helm extensions when users deploy new versions of your application. For example, you can change `values` from one release to another, and those changes apply when the user deploys the new version. +Each extension chart is installed with `helm upgrade --install --wait --wait-for-jobs`, so Helm hooks (such as `pre-install` and `pre-upgrade` Jobs) run and must succeed before the next chart begins. + #### Requirements * You must provide `chart.chartVersion`. Omitting it can cause upgrade issues. diff --git a/embedded-cluster/embedded-overview.mdx b/embedded-cluster/embedded-overview.mdx index 042e4943b8..b3e4906c3d 100644 --- a/embedded-cluster/embedded-overview.mdx +++ b/embedded-cluster/embedded-overview.mdx @@ -40,6 +40,24 @@ For more information about how to install with Embedded Cluster, see: * [Air Gap Installation with Embedded Cluster](installing-embedded-air-gap) * [Install with a customer-managed registry](installing-embedded-byo-registry) +## Deployment pipeline {#deployment-pipeline} + +During both installation and upgrade, Embedded Cluster deploys components in a fixed, sequential pipeline. Each stage must complete before the next begins: + +1. **Kubernetes bootstrap:** Embedded Cluster installs and starts the k0s Kubernetes distribution. +1. **Host preflight checks:** Verifies that the host meets requirements (disk space, memory, CPU, ports, hostname length). +1. **Built-in addons:** Installs infrastructure addons in a fixed order: + 1. **OpenEBS** (local PersistentVolume storage) + 1. **SeaweedFS** (S3-compatible object store, HA airgap clusters only) + 1. **Image registry** (internal image registry, airgap only unless using a customer-managed registry) +1. **Helm chart extensions:** Installs vendor-provided extension charts, sorted by `weight` (ascending). Charts with equal weights install in alphabetical order. See [extensions](embedded-config#extensions). +1. **Application preflight checks:** Runs vendor-defined preflight checks against the cluster. +1. **Application charts:** Installs the application's HelmChart custom resources, sorted by `weight` (ascending). + +Each Helm chart (addons, extensions, and application charts) is installed with `--wait` and `--wait-for-jobs`, which means Helm hooks (`pre-install`, `post-install`, `pre-upgrade`, `post-upgrade`) run and must succeed before the next chart begins. This enables patterns like running database migrations in a Helm pre-upgrade hook Job. + +Extensions install after built-in addons and before application charts. This provides a platform-then-infrastructure-then-application deployment order when combined with `weight`-based sorting within each tier. + ## About upgrading with Embedded Cluster Upgrades can be performed from the CLI on a controller. Alternatively, an opt-in [persistent admin console](embedded-persistent-console) can be installed on a controller to give end users a browser-driven upgrade experience without SSH access. The persistent admin console is not enabled by default. diff --git a/embedded-cluster/updating-embedded.mdx b/embedded-cluster/updating-embedded.mdx index b1b27a796a..82108d0d61 100644 --- a/embedded-cluster/updating-embedded.mdx +++ b/embedded-cluster/updating-embedded.mdx @@ -196,3 +196,21 @@ sudo ./APP_SLUG upgrade --license LICENSE_FILE --headless ``` Optionally pass `--config-values` with the path to your [ConfigValues](/reference/custom-resource-configvalues) file to change configuration during the upgrade. Embedded Cluster automatically detects the air gap installation. + +## Upgrade failure handling {#upgrade-failures} + +Embedded Cluster upgrades follow the same sequential [deployment pipeline](embedded-overview#deployment-pipeline) as installations. If any stage fails, the upgrade stops and reports the error. The stages that already completed are not rolled back. + +### Retrying a failed upgrade + +To retry a failed upgrade, resolve the underlying issue and re-run the `upgrade` command with the same binary. Each stage of the upgrade is safe to re-run. + +### Helm chart rollback + +If a Helm chart fails during the upgrade (addon, extension, or application chart), Embedded Cluster does not automatically roll back that chart's Helm release to its previous revision. The failed release remains in a `failed` state. Re-running `upgrade` cleans up the failed release and retries the installation. + +### Database migrations and application-level rollback + +Embedded Cluster manages infrastructure and chart deployment. Application-level concerns like database migration rollback are the vendor's responsibility. If your application uses Helm hooks (such as a `pre-upgrade` Job) to run database migrations, design those migrations to be idempotent so they are safe to re-run if the upgrade is retried. + +Embedded Cluster does not attempt to reverse database migrations or other application state changes when an upgrade fails. Plan your migration strategy accordingly. From 495c7dbeae0118a02fe949b88293f41a619aac0e Mon Sep 17 00:00:00 2001 From: Amber Alston Date: Mon, 6 Jul 2026 15:53:05 -0600 Subject: [PATCH 2/3] Soften upgrade failure handling tone and add DR recovery guidance Reframes the upgrade failure section to lead with the common case (retry resolves most failures) and adds disaster recovery as the recommended path for rare unrecoverable upgrades. Recommends bundling Velero as a Helm chart extension for vendor-controlled backup and restore. Co-Authored-By: Claude Opus 4.6 --- embedded-cluster/updating-embedded.mdx | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/embedded-cluster/updating-embedded.mdx b/embedded-cluster/updating-embedded.mdx index 82108d0d61..a3675a2b6d 100644 --- a/embedded-cluster/updating-embedded.mdx +++ b/embedded-cluster/updating-embedded.mdx @@ -199,18 +199,24 @@ Optionally pass `--config-values` with the path to your [ConfigValues](/referenc ## Upgrade failure handling {#upgrade-failures} -Embedded Cluster upgrades follow the same sequential [deployment pipeline](embedded-overview#deployment-pipeline) as installations. If any stage fails, the upgrade stops and reports the error. The stages that already completed are not rolled back. +Embedded Cluster upgrades follow the same sequential [deployment pipeline](embedded-overview#deployment-pipeline) as installations. If any stage fails, the upgrade stops and reports the error. Stages that already completed successfully are not rolled back automatically. + +Most upgrade failures can be resolved by fixing the underlying issue and re-running the upgrade. In rare cases where an upgrade cannot be recovered, disaster recovery (restore from backup) is the recommended path. ### Retrying a failed upgrade -To retry a failed upgrade, resolve the underlying issue and re-run the `upgrade` command with the same binary. Each stage of the upgrade is safe to re-run. +Re-run the `upgrade` command with the same binary after resolving the issue. Each stage is idempotent and safe to re-run. ### Helm chart rollback -If a Helm chart fails during the upgrade (addon, extension, or application chart), Embedded Cluster does not automatically roll back that chart's Helm release to its previous revision. The failed release remains in a `failed` state. Re-running `upgrade` cleans up the failed release and retries the installation. +When a Helm chart fails during the upgrade (addon, extension, or application chart), the failed release remains in a `failed` state rather than rolling back to its previous revision. Re-running `upgrade` cleans up the failed release and retries the installation. ### Database migrations and application-level rollback -Embedded Cluster manages infrastructure and chart deployment. Application-level concerns like database migration rollback are the vendor's responsibility. If your application uses Helm hooks (such as a `pre-upgrade` Job) to run database migrations, design those migrations to be idempotent so they are safe to re-run if the upgrade is retried. +Embedded Cluster handles infrastructure and chart deployment. Application-level concerns like database migration rollback are managed by your application. If you use Helm hooks (such as a `pre-upgrade` Job) to run database migrations, design those migrations to be idempotent so they are safe to re-run when an upgrade is retried. + +### Disaster recovery {#upgrade-dr} + +For cases where an upgrade fails and cannot be recovered by retrying, treat the situation as a disaster recovery scenario. Replicated recommends bundling a backup solution like [Velero](https://velero.io/) as a Helm chart extension in your application package. This gives your customers a backup and restore path that you control, including the Velero version, backup scope, and upgrade cadence. -Embedded Cluster does not attempt to reverse database migrations or other application state changes when an upgrade fails. Plan your migration strategy accordingly. +In Embedded Cluster v3, application backups are scoped to your application data only. Embedded Cluster infrastructure state is rebuilt from the install artifacts during a restore, so a restore onto a fresh machine recovers the full environment. From 5904863d7b390728c517ee6c7e5d264ca07f4610 Mon Sep 17 00:00:00 2001 From: Amber Alston Date: Mon, 6 Jul 2026 16:19:17 -0600 Subject: [PATCH 3/3] Address review feedback: preflight order, console retry, DR wording - Swap host preflights before k0s bootstrap to match actual code order - Add persistent admin console as a retry path for failed upgrades - Reword DR backup sentence to clarify that the vendor bundles the backup solution, not that EC provides it out of the box Co-Authored-By: Claude Opus 4.6 --- embedded-cluster/embedded-overview.mdx | 2 +- embedded-cluster/updating-embedded.mdx | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/embedded-cluster/embedded-overview.mdx b/embedded-cluster/embedded-overview.mdx index b3e4906c3d..c68c17a894 100644 --- a/embedded-cluster/embedded-overview.mdx +++ b/embedded-cluster/embedded-overview.mdx @@ -44,8 +44,8 @@ For more information about how to install with Embedded Cluster, see: During both installation and upgrade, Embedded Cluster deploys components in a fixed, sequential pipeline. Each stage must complete before the next begins: -1. **Kubernetes bootstrap:** Embedded Cluster installs and starts the k0s Kubernetes distribution. 1. **Host preflight checks:** Verifies that the host meets requirements (disk space, memory, CPU, ports, hostname length). +1. **Kubernetes bootstrap:** Embedded Cluster installs and starts the k0s Kubernetes distribution. 1. **Built-in addons:** Installs infrastructure addons in a fixed order: 1. **OpenEBS** (local PersistentVolume storage) 1. **SeaweedFS** (S3-compatible object store, HA airgap clusters only) diff --git a/embedded-cluster/updating-embedded.mdx b/embedded-cluster/updating-embedded.mdx index a3675a2b6d..9f9708e791 100644 --- a/embedded-cluster/updating-embedded.mdx +++ b/embedded-cluster/updating-embedded.mdx @@ -205,7 +205,7 @@ Most upgrade failures can be resolved by fixing the underlying issue and re-runn ### Retrying a failed upgrade -Re-run the `upgrade` command with the same binary after resolving the issue. Each stage is idempotent and safe to re-run. +Re-run the `upgrade` command with the same binary after resolving the issue. Each stage is idempotent and safe to re-run. For upgrades initiated from the persistent admin console, use the console's upgrade flow to retry. ### Helm chart rollback @@ -219,4 +219,4 @@ Embedded Cluster handles infrastructure and chart deployment. Application-level For cases where an upgrade fails and cannot be recovered by retrying, treat the situation as a disaster recovery scenario. Replicated recommends bundling a backup solution like [Velero](https://velero.io/) as a Helm chart extension in your application package. This gives your customers a backup and restore path that you control, including the Velero version, backup scope, and upgrade cadence. -In Embedded Cluster v3, application backups are scoped to your application data only. Embedded Cluster infrastructure state is rebuilt from the install artifacts during a restore, so a restore onto a fresh machine recovers the full environment. +When you bundle a backup solution this way, backups cover your application data only. Embedded Cluster infrastructure state is not included in application backups because it is rebuilt from the install artifacts during a restore, so a restore onto a fresh machine recovers the full environment.