Document EC v3 deployment pipeline and upgrade failure handling

embedded-cluster/embedded-config.mdx

@@ -180,7 +180,7 @@ spec:
 
 ## extensions
 
-Embedded Cluster installs Helm charts before your application as part of Embedded Cluster itself. Specify these charts in `spec.extensions.helmCharts`. Each item describes one chart release.
+Embedded Cluster installs Helm chart extensions after built-in addons (OpenEBS, registry) and before application charts. This positions extension charts in the infrastructure layer of the [deployment pipeline](embedded-overview#deployment-pipeline). Specify these charts in `spec.extensions.helmCharts`. Each item describes one chart release.
 
 ### helmCharts
 
@@ -191,9 +191,11 @@ Each object in `helmCharts` can include the following fields:
   * `chartVersion`: Chart version string to install.
 * `releaseName`: (Required) Helm release name.
 * `namespace`: (Required) Namespace for the release.
-* `weight`: Controls installation order relative to other Helm chart extensions. Lower weights install before higher weights.
+* `weight`: Controls installation order relative to other Helm chart extensions. Lower weights install before higher weights. When weights are equal, charts install in alphabetical order by namespace and release name. The default weight is `0`. Negative weights are supported.
 * `values`: Multi-line string of Helm values. This field supports [Replicated template functions](/reference/template-functions-about). Embedded Cluster updates Helm extensions when users deploy new versions of your application. For example, you can change `values` from one release to another, and those changes apply when the user deploys the new version.
 
+Each extension chart is installed with `helm upgrade --install --wait --wait-for-jobs`, so Helm hooks (such as `pre-install` and `pre-upgrade` Jobs) run and must succeed before the next chart begins.
+
 #### Requirements
 
 * You must provide `chart.chartVersion`. Omitting it can cause upgrade issues.

embedded-cluster/embedded-overview.mdx

@@ -40,6 +40,24 @@ For more information about how to install with Embedded Cluster, see:
 * [Air Gap Installation with Embedded Cluster](installing-embedded-air-gap)
 * [Install with a customer-managed registry](installing-embedded-byo-registry)
 
+## Deployment pipeline {#deployment-pipeline}
+
+During both installation and upgrade, Embedded Cluster deploys components in a fixed, sequential pipeline. Each stage must complete before the next begins:
+
+1. **Host preflight checks:** Verifies that the host meets requirements (disk space, memory, CPU, ports, hostname length).
+1. **Kubernetes bootstrap:** Embedded Cluster installs and starts the k0s Kubernetes distribution.
+1. **Built-in addons:** Installs infrastructure addons in a fixed order:
+   1. **OpenEBS** (local PersistentVolume storage)
+   1. **SeaweedFS** (S3-compatible object store, HA airgap clusters only)
+   1. **Image registry** (internal image registry, airgap only unless using a customer-managed registry)
+1. **Helm chart extensions:** Installs vendor-provided extension charts, sorted by `weight` (ascending). Charts with equal weights install in alphabetical order. See [extensions](embedded-config#extensions).
+1. **Application preflight checks:** Runs vendor-defined preflight checks against the cluster.
+1. **Application charts:** Installs the application's HelmChart custom resources, sorted by `weight` (ascending).
+
+Each Helm chart (addons, extensions, and application charts) is installed with `--wait` and `--wait-for-jobs`, which means Helm hooks (`pre-install`, `post-install`, `pre-upgrade`, `post-upgrade`) run and must succeed before the next chart begins. This enables patterns like running database migrations in a Helm pre-upgrade hook Job.
+
+Extensions install after built-in addons and before application charts. This provides a platform-then-infrastructure-then-application deployment order when combined with `weight`-based sorting within each tier.
+
 ## About upgrading with Embedded Cluster
 
 Upgrades can be performed from the CLI on a controller. Alternatively, an opt-in [persistent admin console](embedded-persistent-console) can be installed on a controller to give end users a browser-driven upgrade experience without SSH access. The persistent admin console is not enabled by default.

embedded-cluster/updating-embedded.mdx

@@ -196,3 +196,27 @@ sudo ./APP_SLUG upgrade --license LICENSE_FILE --headless
 ```
 
 Optionally pass `--config-values` with the path to your [ConfigValues](/reference/custom-resource-configvalues) file to change configuration during the upgrade. Embedded Cluster automatically detects the air gap installation.
+
+## Upgrade failure handling {#upgrade-failures}
+
+Embedded Cluster upgrades follow the same sequential [deployment pipeline](embedded-overview#deployment-pipeline) as installations. If any stage fails, the upgrade stops and reports the error. Stages that already completed successfully are not rolled back automatically.
+
+Most upgrade failures can be resolved by fixing the underlying issue and re-running the upgrade. In rare cases where an upgrade cannot be recovered, disaster recovery (restore from backup) is the recommended path.
+
+### Retrying a failed upgrade
+
+Re-run the `upgrade` command with the same binary after resolving the issue. Each stage is idempotent and safe to re-run. For upgrades initiated from the persistent admin console, use the console's upgrade flow to retry.
+
+### Helm chart rollback
+
+When a Helm chart fails during the upgrade (addon, extension, or application chart), the failed release remains in a `failed` state rather than rolling back to its previous revision. Re-running `upgrade` cleans up the failed release and retries the installation.
+
+### Database migrations and application-level rollback
+
+Embedded Cluster handles infrastructure and chart deployment. Application-level concerns like database migration rollback are managed by your application. If you use Helm hooks (such as a `pre-upgrade` Job) to run database migrations, design those migrations to be idempotent so they are safe to re-run when an upgrade is retried.
+
+### Disaster recovery {#upgrade-dr}
+
+For cases where an upgrade fails and cannot be recovered by retrying, treat the situation as a disaster recovery scenario. Replicated recommends bundling a backup solution like [Velero](https://velero.io/) as a Helm chart extension in your application package. This gives your customers a backup and restore path that you control, including the Velero version, backup scope, and upgrade cadence.
+
+When you bundle a backup solution this way, backups cover your application data only. Embedded Cluster infrastructure state is not included in application backups because it is rebuilt from the install artifacts during a restore, so a restore onto a fresh machine recovers the full environment.