feat: implement HIP-0025 resource creation sequencing#32038
Conversation
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
Implements HIP-0025 ordered resource creation sequencing in Helm v4 by introducing DAG-based ordering for subcharts and resource groups, along with optional custom readiness evaluation and CLI support.
Changes:
- Added generic DAG utilities plus subchart/resource-group DAG builders and parsers.
- Implemented
--wait=orderedexecution across install/upgrade/rollback/uninstall and added--readiness-timeout. - Added ordered delimiter output for
helm template, new lint rules, and extensive unit/integration test coverage.
Reviewed changes
Copilot reviewed 47 out of 47 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/release/v1/util/resource_group.go | Parses resource-group annotations and builds a DAG for ordered batching. |
| pkg/release/v1/util/resource_group_test.go | Unit tests for resource-group parsing, warnings, and DAG behavior. |
| pkg/release/v1/release.go | Adds SequencingInfo metadata to releases for rollback/uninstall behavior. |
| pkg/release/v1/release_test.go | Tests JSON backward compatibility and round-tripping for SequencingInfo. |
| pkg/kube/readiness.go | Adds custom readiness expression parsing and evaluation via JSONPath. |
| pkg/kube/readiness_test.go | Unit tests for custom readiness evaluation and expression parsing. |
| pkg/kube/custom_readiness_status_reader.go | Integrates custom readiness into kstatus waiting via a StatusReader. |
| pkg/kube/custom_readiness_status_reader_test.go | Verifies waiter integration and status reader behavior. |
| pkg/kube/options.go | Adds waiter option to enable the custom readiness status reader. |
| pkg/kube/client.go | Adds OrderedWaitStrategy and wires in optional custom readiness readers. |
| pkg/kube/client_wait_strategy_test.go | Tests waiter selection and error messaging for ordered strategy. |
| pkg/cmd/flags.go | Adds --wait=ordered parsing support (where applicable) and --readiness-timeout. |
| pkg/cmd/flags_test.go | Tests CLI flag acceptance/rejection and readiness-timeout validation. |
| pkg/cmd/install.go | Wires --readiness-timeout and ordered wait handling into install command. |
| pkg/cmd/upgrade.go | Wires --wait=ordered and --readiness-timeout into upgrade command. |
| pkg/cmd/template.go | Adds ordered template rendering with resource-group delimiters and subchart ordering. |
| pkg/cmd/template_test.go | Adds golden tests for ordered delimiter output and absence without ordered wait. |
| pkg/cmd/testdata/testcharts/sequenced-chart/Chart.yaml | Test fixture chart for ordered template output. |
| pkg/cmd/testdata/testcharts/sequenced-chart/templates/aa-databases-configmap.yaml | Test fixture manifest with resource-group annotation. |
| pkg/cmd/testdata/testcharts/sequenced-chart/templates/bb-app-configmap.yaml | Test fixture manifest with depends-on resource-group annotation. |
| pkg/cmd/testdata/testcharts/sequenced-chart/templates/cc-unsequenced-configmap.yaml | Test fixture manifest without sequencing annotations. |
| pkg/cmd/testdata/testcharts/sequenced-chart/charts/worker/Chart.yaml | Test subchart fixture for ordered template output. |
| pkg/cmd/testdata/testcharts/sequenced-chart/charts/worker/templates/aa-worker-configmap.yaml | Test subchart fixture manifest with resource-group annotation. |
| pkg/cmd/testdata/output/template-ordered-delimiters.txt | Golden output verifying START/END delimiters and ordering. |
| pkg/chart/v2/util/dag.go | Introduces generic DAG with deterministic batching and cycle detection. |
| pkg/chart/v2/util/dag_test.go | Unit tests for DAG batching, cycles, and helper methods. |
| pkg/chart/v2/util/subchart_dag.go | Builds subchart dependency DAG from Chart.yaml fields and annotation. |
| pkg/chart/v2/util/subchart_dag_test.go | Unit tests for subchart DAG ordering, aliases, disabled deps, and errors. |
| pkg/chart/v2/dependency.go | Adds depends-on field to chart dependencies for sequencing declarations. |
| pkg/chart/v2/dependency_json_test.go | Tests JSON/YAML tags and backward compatibility for the new dependency field. |
| pkg/chart/v2/lint/lint.go | Registers new sequencing lint rule. |
| pkg/chart/v2/lint/lint_test.go | Validates sequencing rules are registered and produce expected errors. |
| pkg/chart/v2/lint/rules/sequencing.go | Lint rules for subchart/resource-group cycles and readiness annotation completeness. |
| pkg/chart/v2/lint/rules/sequencing_test.go | Tests sequencing lint behavior across error/warning scenarios. |
| pkg/chart/common/util/jsonschema.go | Minor change to schema validation error formatting implementation. |
| pkg/action/action.go | Refactors render pipeline to optionally return sorted manifests for sequencing-aware actions. |
| pkg/action/sequencing.go | Adds sequenced deployment engine (two-level DAG, per-batch waits, annotation stripping). |
| pkg/action/sequencing_test.go | Comprehensive tests for ordered installs, batching, timeouts, and hooks behavior. |
| pkg/action/install.go | Adds ordered install execution path with per-batch readiness timeout. |
| pkg/action/upgrade.go | Adds ordered upgrade execution path and sequencing metadata recording. |
| pkg/action/upgrade_sequenced_test.go | Tests ordered upgrade behaviors including rollback-on-failure/cleanup and transitions. |
| pkg/action/rollback.go | Adds ordered rollback execution path based on stored sequencing metadata. |
| pkg/action/rollback_sequenced_test.go | Tests ordered rollback behavior, cleanup, and sequencing info preservation. |
| pkg/action/uninstall.go | Adds ordered uninstall path using reverse DAG order when sequencing metadata is present. |
| pkg/action/uninstall_sequenced_test.go | Tests sequenced uninstall reverse ordering, hooks, keep policy/history, and dry-run. |
| pkg/action/warning_system_test.go | Tests slog warnings for partial readiness, isolated groups, and orphan dependencies. |
| pkg/action/backward_compat_test.go | Verifies backward-compat behavior for ordered wait without annotations and legacy releases. |
Comments suppressed due to low confidence (1)
pkg/kube/client.go:1
- Because
newCustomReadinessStatusReader()is prepended and itsSupports()returns true for all kinds, enabling it changes which StatusReader is used for all resources in that wait (including those without custom readiness annotations). If the intent is to only affect resources that actually define both readiness annotations, consider tightening the enablement condition (e.g., only enable the reader when at least one resource in the batch has both annotations) to reduce behavior drift and minimize surprises in mixed batches.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
41f231a to
e5409e8
Compare
Document the new resource sequencing feature in Helm v4, including: - Resource group annotations and dependencies - Subchart sequencing via Chart.yaml - Custom readiness conditions with JSONPath - CLI flags (--wait=ordered, --readiness-timeout) - Lint rules for sequencing issues - Execution order for install, upgrade, rollback, and uninstall Refs: HIP-0025, helm/helm#32038 Signed-off-by: promptless[bot] <promptless[bot]@users.noreply.github.com>
|
Promptless prepared a documentation update related to this change. Triggered by helm/helm#32038 Created comprehensive documentation for HIP-0025 resource sequencing, covering the new Review at helm/helm-www#2068 |
e5409e8 to
da559ec
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 47 out of 47 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
pkg/kube/readiness.go:1
evaluateExpressionparses JSONPath (jsonpath.New().Parse(...)) on every evaluation. DuringstatusWaiterpolling this can become hot (expressions × resources × poll iterations). To reduce overhead, consider compiling/parsing expressions once (e.g., parse the annotation JSON into a slice of pre-parsed{template, op, expected}structs and reuse compiled JSONPath objects) and reusing across polling iterations for the same resource/batch.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 47 out of 47 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
pkg/kube/readiness.go:1
- All JSONPath execution errors are currently swallowed and treated as 'condition not met'. This can hide real authoring errors (e.g., incorrect JSONPath that parses but fails at execution time due to type/indexing issues) and lead to resources waiting indefinitely with no actionable signal. Prefer returning an error for execution failures, or at least only suppress a narrow 'field not found / empty result' case while surfacing other failures with context (expression + JSONPath template).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 47 out of 47 changed files in this pull request and generated 6 comments.
Comments suppressed due to low confidence (1)
pkg/kube/readiness.go:1
- readinessJSONPath()
always prefixes the provided path with.status, which means an expression like{.status.phase} == "Ready"turns into{.status.status.phase}and will never match. This is easy to hit because several fixtures/tests in this PR use{.status.*}style. Consider either (a) explicitly documenting/enforcing that paths are relative to.status(so{.phase}), and updating fixtures accordingly, or (b) supporting both by stripping a leading.status/.status.prefix before prefixing, so{.status.phase}and{.phase}` behave the same.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 47 out of 47 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
pkg/release/v1/util/resource_group.go:1
- resourceGroupResourceID does not include namespace. If a chart renders same Kind/Name into multiple namespaces (possible via templating), ParseResourceGroups can falsely report “assigned to multiple resource groups” even though the resources are distinct. Include namespace in the identity when available (e.g., apiVersion/kind/namespace/name) to match Kubernetes object identity semantics.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 48 out of 48 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 48 out of 48 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 48 out of 48 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
e3c0e2a to
0c95ee2
Compare
Dependency.Validate() sanitized Name/Version/Repository/Condition/Tags but not the HIP-0025 DependsOn field, whose entries are used verbatim as subchart lookup keys when building the sequencing DAG. Stray control characters or embedded whitespace could therefore cause spurious "unknown or disabled subchart" failures. Sanitize DependsOn entries like the sibling fields. Addresses a Copilot review comment on PR helm#32038. Signed-off-by: Rohit Gudi <50377477+caretak3r@users.noreply.github.com>
The depends-on field is documented as accepting subchart names or aliases, but BuildSubchartDAG only registered each subchart under its effective name (alias if set), so referencing an aliased subchart by its original name was rejected as "unknown or disabled subchart". Resolve references by either the effective name or the original name, and reject a reference that maps to more than one subchart as ambiguous rather than silently picking one. Addresses a Copilot review comment on PR helm#32038. Signed-off-by: Rohit Gudi <50377477+caretak3r@users.noreply.github.com>
|
@joejulian Any suggestions? I can re-submit the PR in small pieces, one for each DAG? |
49ee576 to
1ab51f3
Compare
|
Rebased onto current
What changed since your review
TestingBeyond unit coverage, I ran a live matrix against a kind cluster: 35 capability rows Open questions for maintainers
Re-requesting review. |
|
@joejulian when you have a chance — this is rebased, re-layered, and your review items are all addressed (details above). Would appreciate another look, and happy to split it into the per-layer stack if that's easier to review. |
…ion keys Address Copilot review on PR helm#32038: - dag.go: printChild returned early for subcharts whose chart metadata is unavailable (storage-decoded charts at rollback/uninstall), hiding the structural resource-group batches buildStructuralLevel generates. Print an accurate note and still recurse into the structural child level. The old 'not found in chart dependencies' message was also misleading. - sequencing.go: hoist HelmInternalSequencingAnnotations() out of the per-resource Visit closure so the slice is cloned once per batch.
…ion keys Address Copilot review on PR helm#32038: - dag.go: printChild returned early for subcharts whose chart metadata is unavailable (storage-decoded charts at rollback/uninstall), hiding the structural resource-group batches buildStructuralLevel generates. Print an accurate note and still recurse into the structural child level. The old 'not found in chart dependencies' message was also misleading. - sequencing.go: hoist HelmInternalSequencingAnnotations() out of the per-resource Visit closure so the slice is cloned once per batch.
9cd32c1 to
c0fed30
Compare
| annotations := manifest.Head.Metadata.Annotations | ||
| _, hasSuccess := annotations[releaseutil.AnnotationReadinessSuccess] | ||
| _, hasFailure := annotations[releaseutil.AnnotationReadinessFailure] | ||
| if hasSuccess != hasFailure { | ||
| b.warnf(WarningKindPartialReadiness, chartPath, "resource %q has only one of %s and %s; falling back to kstatus readiness", manifest.Head.Metadata.Name, releaseutil.AnnotationReadinessSuccess, releaseutil.AnnotationReadinessFailure) | ||
| } |
Generic DAG with deterministic topological batching and cycle detection, the subchart dependency DAG builder (reading depends-on field and the helm.sh/depends-on/subcharts annotation against post-processed c.Dependencies() state), and the new depends-on field on Chart.yaml dependency entries. Refs: HIP-0025 Signed-off-by: Rohit Gudi <50377477+caretak3r@users.noreply.github.com>
…adata Resource-group annotation parsing into a DAG, the SequencingInfo field recorded on releases for sequenced uninstall/rollback, and the helm-internal annotation stripping helper. Backward-compatible JSON round-trip for releases stored without SequencingInfo. Refs: HIP-0025 Signed-off-by: Rohit Gudi <50377477+caretak3r@users.noreply.github.com>
OrderedWaitStrategy, custom readiness evaluation via .status-scoped JSONPath expressions (helm.sh/readiness-success / -failure), and the status reader that layers custom readiness over kstatus. Refs: HIP-0025 Signed-off-by: Rohit Gudi <50377477+caretak3r@users.noreply.github.com>
Lint validation for subchart and resource-group dependency cycles, orphan group references, and the both-or-neither readiness annotation rule. Refs: HIP-0025 Signed-off-by: Rohit Gudi <50377477+caretak3r@users.noreply.github.com>
The sequenced deployment engine driving per-batch create-and-wait across the resource-group and subchart DAGs, wired into install, upgrade, rollback (respecting stored SequencingInfo), and reverse-order uninstall. Default (non-ordered) paths are unchanged. Refs: HIP-0025 Signed-off-by: Rohit Gudi <50377477+caretak3r@users.noreply.github.com>
…command The --wait=ordered and --readiness-timeout flags across install/upgrade/uninstall/rollback, ordered helm template output with resource-group delimiters, and the helm dag debugging command. Refs: HIP-0025 Signed-off-by: Rohit Gudi <50377477+caretak3r@users.noreply.github.com>
…ion keys Address Copilot review on PR helm#32038: - dag.go: printChild returned early for subcharts whose chart metadata is unavailable (storage-decoded charts at rollback/uninstall), hiding the structural resource-group batches buildStructuralLevel generates. Print an accurate note and still recurse into the structural child level. The old 'not found in chart dependencies' message was also misleading. - sequencing.go: hoist HelmInternalSequencingAnnotations() out of the per-resource Visit closure so the slice is cloned once per batch. Signed-off-by: Rohit Gudi <50377477+caretak3r@users.noreply.github.com>
c0fed30 to
9470c5a
Compare
| @@ -513,13 +520,13 @@ func (cfg *Configuration) renderResources(ctx context.Context, ch *chart.Chart, | |||
| // used by install or upgrade | |||
| err = writeToFile(newDir, m.Name, m.Content, fileWritten[m.Name]) | |||
| fileWritten[m.Path] = true | ||
| } | ||
|
|
||
| err = writeToFile(newDir, m.Path, m.Manifest, fileWritten[m.Path]) |
What this PR does / why we need it
Implements HIP-0025: Better Support for Resource Creation Sequencing, adding native DAG-based ordering for resource groups and subcharts to Helm v4.
Today Helm applies all rendered manifests simultaneously. Application distributors who need ordered deployment must use hooks (tedious to maintain) or bake sequencing into the application itself (unnecessary complexity). HIP-0025 gives chart authors a first-class mechanism to define deployment order using annotations and
Chart.yamlfields, and gives operators a--wait=orderedflag to enable sequenced execution.Overview
When
--wait=orderedis specified, Helm builds two levels of dependency graphs:depends-onfields inChart.yamldependencies and/or thehelm.sh/depends-on/subchartsannotation.helm.sh/resource-groupandhelm.sh/depends-on/resource-groupsannotations.Resources are deployed in topological order (Kahn's algorithm). Helm waits for each batch to reach readiness before proceeding to the next. Readiness defaults to kstatus evaluation; chart authors can override it with custom JSONPath expressions via
helm.sh/readiness-successandhelm.sh/readiness-failureannotations supporting 6 operators (==,!=,<,<=,>,>=).Execution Flow
Reverse-DAG order is used for
uninstallandrollbackof sequenced releases.Commit Organization
This PR was previously organized as 7 stacked commits to aid review. Following review iterations and a rebase onto current
upstream/main, it has been consolidated into 2 commits to keep the history clean:feat: implement HIP-0025 resource creation sequencingfix(hip-0025): thread context through updateAndWait and chartPath through nested subchartsNew Annotations
helm.sh/resource-grouphelm.sh/depends-on/resource-groupshelm.sh/depends-on/subchartshelm.sh/readiness-successhelm.sh/readiness-failureAll six annotation keys match the HIP-0025 specification byte-for-byte. The
helm.sh/depends-on/resource-groupsandhelm.sh/depends-on/subchartskeys contain multiple/(not valid Kubernetes qualified names) — these are stripped before SSA-applying to the API server, so K8s never sees them; they are consumed at render/sequencing time only.New Chart.yaml Fields
depends-onondependencies[]entries — declares subchart ordering dependencies by name or alias.New CLI Flags
--wait=orderedinstall,upgrade,rollback,uninstall--readiness-timeoutinstall,upgrade,rollback--timeout)Schema Changes
DependsOn []stringadded tochart.Dependency(preserves backward compatibility — optional field).SequencingInfostruct added torelease.Release(Enabled bool,Strategy string). Persisted with the release so rollback/uninstall know to use sequenced flow.OrderedWaitStrategyconstant added topkg/kube.Key Design Decisions
pkg/chart/v2/util/dag.go).parent → parent/charts/sub → parent/charts/sub/charts/nested.--wait=orderedare upgraded/rolled back/uninstalled using the traditional single-batch flow. TheSequencingInfo.Enabledflag gates the sequenced path on rollback and uninstall.helm.sh/hook-weightas before.helm.sh/depends-on/resource-groupsis invalid as a K8s annotation key (multiple/); it's stripped from manifests before SSA.helm template --wait=orderedoutput emits## START/ENDdelimiters and is not intended to be piped directly tokubectl apply.Review Feedback Addressed
Joejulian's CHANGES_REQUESTED feedback (April 2026) has been addressed in commit 1:
subchart_dag.go— use ProcessDependencies-resolved chart state.BuildSubchartDAGnow readsc.Dependencies()(post-processed: aliases applied, disabled deps pruned, conditions/tags resolved) instead of re-deriving enablement fromchart.Metadata.Dependenciesheuristics. Closes the misclassification path joejulian called out for conditions/tags/aliases.sequencing.go— context cancellation throughcreateAndWait. Addedctx.Done()select gates before Build, before Create, and before Wait. New testTestSequencedDeployment_CreateAndWait_RespectsContextCancellationcovers the four cancellation points.Latest Copilot review on the consolidated commit (May 3 2026) flagged two further issues, addressed in commit 2:
updateAndWaitignored its context arg. Sequenced upgrades and rollbacks did not honor cancellation through Build/Update/Wait. Now threaded with the same select gates ascreateAndWait.GroupManifestsByDirectSubchartflattened nested subcharts. Used the bare chart name as path prefix; on recursion the prefix didn't match top-level-rooted manifest names. Now threadschartPaththroughdeployChartLevelanddeleteChartLevelReverseso each recursion level matches the full path. AddedTestSequencing_GroupManifestsByDirectSubchart_Nested; updated the uninstall test that previously documented this as "a known limitation."Deferred (tracked for follow-up issues, not blockers):
groupManifestsByChartPathhelper betweencmd/template.goandaction/sequencing.go(acknowledged scope-deferred).# Source:headers innormalizedManifestContentfor uninstall path recovery.How to Test
Unit tests:
Integration tests (requires kind cluster):
Manual verification checklist:
helm templatewith--wait=orderedshows resource-group delimitershelm linterrors on circular deps, partial readiness, orphan refs (ErrorSev)--readiness-timeoutis honored per resource group; rejected when greater than--timeoutSequencingInfowhen--wait=orderedis used; rollback/uninstall consult itReference Documents
Special notes for reviewers:
rebase -ion the previous 19-commit branch produced N rounds of conflicts against current upstream after the Go-1.26 / k8s-1.36 merge; a one-shot 3-waymerge --squashfrom a fresh branch onupstream/mainresolved cleanly.pkg/action,pkg/cmd,pkg/kube,pkg/chart/...,pkg/release/....make lint/make vetclean.If applicable:
Signed-off-by: Rohit Gudi 50377477+caretak3r@users.noreply.github.com
refs https://github.com/helm/community/blob/main/hips/hip-0025.md