Document EC v3 deployment pipeline and upgrade failure handling#4232
Merged
Conversation
…ade failure handling Adds three documentation improvements for EC v3: 1. New "Deployment pipeline" section in the overview documenting the fixed sequential install/upgrade pipeline (k0s -> preflights -> addons -> extensions -> app preflights -> app charts), including built-in addon ordering. 2. Clarified extension chart docs: where extensions sit in the pipeline, weight sorting details (default 0, negative supported, stable alphabetical tiebreak), and that Helm hooks are supported via --wait-for-jobs. 3. New "Upgrade failure handling" section covering retry behavior, Helm chart rollback semantics, and vendor responsibility for DB migrations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
✅ Deploy Preview for replicated-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
✅ Deploy Preview for replicated-docs-upgrade ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
replicated-ci
added
type::docs
Reframes the upgrade failure section to lead with the common case (retry resolves most failures) and adds disaster recovery as the recommended path for rare unrecoverable upgrades. Recommends bundling Velero as a Helm chart extension for vendor-controlled backup and restore. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
AmberAlston
commented
Jul 6, 2026
|
|
||
| For cases where an upgrade fails and cannot be recovered by retrying, treat the situation as a disaster recovery scenario. Replicated recommends bundling a backup solution like [Velero](https://velero.io/) as a Helm chart extension in your application package. This gives your customers a backup and restore path that you control, including the Velero version, backup scope, and upgrade cadence. | ||
|
|
||
| In Embedded Cluster v3, application backups are scoped to your application data only. Embedded Cluster infrastructure state is rebuilt from the install artifacts during a restore, so a restore onto a fresh machine recovers the full environment. |
Member
Author
There was a problem hiding this comment.
We'll want to add more to DR section when we have stuff like reference implementation available, but this will work for now until we have more to link to
sgalsaleh
reviewed
Jul 6, 2026
| During both installation and upgrade, Embedded Cluster deploys components in a fixed, sequential pipeline. Each stage must complete before the next begins: | ||
|
|
||
| 1. **Kubernetes bootstrap:** Embedded Cluster installs and starts the k0s Kubernetes distribution. | ||
| 1. **Host preflight checks:** Verifies that the host meets requirements (disk space, memory, CPU, ports, hostname length). |
Member
There was a problem hiding this comment.
This actually comes before bootstrapping k0s
sgalsaleh
reviewed
Jul 6, 2026
|
|
||
| ### Retrying a failed upgrade | ||
|
|
||
| Re-run the `upgrade` command with the same binary after resolving the issue. Each stage is idempotent and safe to re-run. |
Member
There was a problem hiding this comment.
What about upgrades initiated from the persistent admin console?
sgalsaleh
reviewed
Jul 6, 2026
|
|
||
| For cases where an upgrade fails and cannot be recovered by retrying, treat the situation as a disaster recovery scenario. Replicated recommends bundling a backup solution like [Velero](https://velero.io/) as a Helm chart extension in your application package. This gives your customers a backup and restore path that you control, including the Velero version, backup scope, and upgrade cadence. | ||
|
|
||
| In Embedded Cluster v3, application backups are scoped to your application data only. Embedded Cluster infrastructure state is rebuilt from the install artifacts during a restore, so a restore onto a fresh machine recovers the full environment. |
Member
There was a problem hiding this comment.
The way this sentence starts gives the impression that EC v3 provides application level backups out of the box.
- Swap host preflights before k0s bootstrap to match actual code order - Add persistent admin console as a retry path for failed upgrades - Reword DR backup sentence to clarify that the vendor bundles the backup solution, not that EC provides it out of the box Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
sgalsaleh
approved these changes
Jul 6, 2026
xavpaice
approved these changes
Jul 6, 2026
xavpaice
left a comment
xavpaice
left a comment
Member
There was a problem hiding this comment.
Appreciate the note on backups too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
--waitand--wait-for-jobs), enabling patterns like DB migration Jobs in pre-upgrade hooksweightdocs with default value, negative weight support, and stable alphabetical tiebreakingContext
These gaps surfaced during a BYOC RFI where internal team members had to read EC v3 source code to answer deployment orchestration questions (section 4 of the RFI). All claims in this PR were verified against the EC v3 codebase before writing.
Key code references verified:
pkg/addons/install.go:88-144- addon ordering (OpenEBS -> SeaweedFS -> registry -> extensions)pkg/appcharts/sorting.go:20-29- weight-based sorting with stable alphabetical tiebreakpkg/helm/helm.go:35-53-helm upgrade -i --wait --wait-for-jobscmd/cli/commands/headless.go:51-126- full pipeline sequence (preflights -> k0s -> addons -> app preflights -> charts)pkg/helm/reconciler.go:22-72- sequential chart installation with pending release cleanupTest plan
embedded-overview#deployment-pipeline,embedded-config#extensions)🤖 Generated with Claude Code