Skip to content

feat(anc): wire check-hotfix into node wrapper behind ENABLE_PROVISIONING_HOTFIX#8715

Draft
Devinwong wants to merge 4 commits into
devinwong/anc-check-hotfix-configmapfrom
devinwong/anc-wire-check-hotfix-wrapper
Draft

feat(anc): wire check-hotfix into node wrapper behind ENABLE_PROVISIONING_HOTFIX#8715
Devinwong wants to merge 4 commits into
devinwong/anc-check-hotfix-configmapfrom
devinwong/anc-wire-check-hotfix-wrapper

Conversation

@Devinwong

@Devinwong Devinwong commented Jun 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

2.1c - Wire check-hotfix into the node wrapper (shell only)

POC / M1 draft. Shell-only wiring for the Provisioning-Hotfix flow. No Go changes.

Why this exists (the scale-up gap it closes)

There are two writers of the hotfix pointer file
(/opt/azure/containers/aks-node-controller-hotfix.json):

  • (A) static, model-baked: a cloud-init write_files entry generated from
    hotfix/anc-hotfix-version.json into nodecustomdata.yml
    (hotfix/anc_hotfix_generate.py). This lands in the VMSS custom data, i.e. the VMSS
    model.
  • (B) dynamic, boot-time: check-hotfix (this wiring), which pulls the pointer live at
    boot and refreshes the same file before download-hotfix gates/installs.

With (A) alone, the pointer is only as fresh as the VMSS model. When an existing nodepool
scales up (autoscale or manual), the new instance boots from the nodepool's frozen model,
so it gets whatever hotfix pointer was baked in at the last model PUT - NOT the current
one. A hotfix published after that point silently misses exactly the newest nodes until a
control-plane reconcile refreshes every affected model. That stale-model-on-scale-up gap
is the core problem this design (and the PoC) targets.

(B) closes it: check-hotfix runs in the wrapper's ExecStart and reads the pointer live
from the LPS endpoint (IMDS-attested, reachable pre-kubelet), decoupled from the VMSS
model. A scale-up node booting from a months-old model still converges to the current
hotfix state. The read channel is deliberately pre-kubelet / IMDS-attested (not
apiserver/ConfigMap) precisely so a brand-new scale-up node can fetch it before it has a
kubeconfig. (A) remains useful as the cold-start/offline default baked into the model;
(B) is the authoritative live override.

Enablement (where this sits in the rollout chain)

This env gate is the on-node terminal of the design's region-staged opt-in:
EnableProvisioningHotfix aks-rp toggle (AKS Toggles-as-code, per region) -> absvc
respects toggle -> ANC respects toggle. This PR implements only the last hop ("ANC
respects toggle"). The env var name mirrors the toggle/contract name to match the
existing contract->env convention (e.g. EnableIMDSRestriction -> ENABLE_IMDS_RESTRICTION),
keeping the chain traceable. Wiring absvc to render this var from a contract field is a
separate follow-up PR; the aks-rp toggle + toggle YAML live in the aks-rp repo. Until
those land, the var renders unset everywhere, so this change is inert (default-off).

Note: 2.1d (#8717) relaxes this env gate, moving the on/off decision into the Go binary
via the enable_provisioning_hotfix contract field (single source of truth). This PR
intentionally ADDS the gate; #8717 relaxes it, so each PR stays reviewable on its own.

What this does

Adds one call to the check-hotfix subcommand (added in 2.1b) inside
aks-node-controller-wrapper.sh, gated behind a new env flag ENABLE_PROVISIONING_HOTFIX
that is OFF by default. check-hotfix reads the hotfix pointer from the LPS endpoint
(IMDS-attested) and refreshes
$HOTFIX_JSON, which the existing download-hotfix block consumes - so it must run
first. The call is fail-open (the command always exits 0) and additionally wrapped
defensively so it can never block provisioning.

Default-off / fail-open guarantee

When ENABLE_PROVISIONING_HOTFIX is unset, empty, or any value other than the literal
string true, the wrapper behaves EXACTLY as it does today. This preserves the
6-month VHD backward-compatibility window: older VHDs running newer CSE, and newer
VHDs running older CSE, are unaffected unless the flag is explicitly turned on.

Known-safe: old VHD + flag on

If ENABLE_PROVISIONING_HOTFIX=true ever reaches a node whose VHD-baked ANC binary predates
2.1b, "$BIN_PATH" check-hotfix is an unknown subcommand and exits non-zero. The
if ... else log "...continuing (fail-open)" fi wrapper swallows that error, so
provisioning still proceeds unchanged. This path is covered by shellspec case 4 below
(check-hotfix exits non-zero -> wrapper still provisions), which models the missing
subcommand. This matters for the 6-month VHD support window.

Before / after flow

Flag off (default - unchanged):

guard config/nbc -> [download-hotfix if $HOTFIX_JSON] -> select binary -> provision

Flag on (ENABLE_PROVISIONING_HOTFIX=true):

guard config/nbc -> check-hotfix (refresh pointer) -> [download-hotfix if $HOTFIX_JSON] -> select binary -> provision

Notes

  • check-hotfix takes no flags/args; it reads the AKSNodeConfig from its default
    on-node path internally for the LPS endpoint (IMDS-attested) it reads, so the wrapper passes nothing.
  • HOTFIX_JSON is parameterized as ${HOTFIX_JSON:-<default>} to match the existing
    BIN_PATH / CONFIG_PATH / NBC_CMD_PATH pattern and to allow shellspec to exercise
    the download-hotfix branch. Production default path is unchanged.
  • Write/read handoff verified: check-hotfix writes the pointer to defaultHotfixVersionPath
    (/opt/azure/containers/aks-node-controller-hotfix.json, hotfix.go) and download-hotfix
    reads the same constant. The wrapper's HOTFIX_JSON default is byte-identical, and the
    Go hotfixVersionPath override exists only for tests (no env/production override and
    check-hotfix takes no path flag), so the two never diverge on a node.
  • POSIX compliant ([ ], =, ${VAR:-}); passes shellcheck generic + POSIX (SC3010/SC3014)
    and the wrapper shellspec suite (8 examples, 0 failures).

Tests

New shellspec cases in aks_node_controller_wrapper_spec.sh:

  1. flag unset -> check-hotfix NOT called (no behavior change)
  2. non-true value (e.g. "1") -> treated as disabled
  3. flag = true -> check-hotfix runs BEFORE download-hotfix, provision last
  4. check-hotfix exits non-zero -> wrapper still proceeds to provision (fail-open)

Stack

main
 \- #8694  2.1a  base->version hotfix map (Go)
     \- #8696  2.1b  check-hotfix LPS endpoint reader (Go)
         \- #8715  2.1c  wire check-hotfix into wrapper (shell)   <- this PR
             \- #8717  2.1d  enable_provisioning_hotfix contract field + Go self-gate

Base is set to the 2.1b branch so the diff shows only the wrapper + shellspec changes.
Will retarget to main as the stack merges down.

This unblocks the on-node e2e PoC tests (fail-open and multi-base) since check-hotfix
is otherwise never invoked at boot.

@Devinwong Devinwong mentioned this pull request Jun 16, 2026
Draft
@Devinwong Devinwong changed the title feat(anc): wire check-hotfix into node wrapper behind ANC_HOTFIX_ENABLED feat(anc): wire check-hotfix into node wrapper behind ENABLE_PROVISIONING_HOTFIX Jun 16, 2026
@Devinwong Devinwong marked this pull request as ready for review June 16, 2026 02:27
@Devinwong Devinwong force-pushed the devinwong/anc-check-hotfix-configmap branch 2 times, most recently from 0c90761 to b33ec66 Compare June 16, 2026 18:08
@Devinwong Devinwong force-pushed the devinwong/anc-wire-check-hotfix-wrapper branch from f842590 to 3ebabf0 Compare June 16, 2026 18:11
@Devinwong Devinwong requested a review from xuexu6666 as a code owner June 19, 2026 20:59
@Devinwong

Devinwong commented Jun 19, 2026

Copy link
Copy Markdown
Collaborator Author

Read-channel pivot: the hotfix-pointer read moves from Option 2 (kube-system anc-hotfix-version ConfigMap via apiserver + bootstrap token) to Option 4 (LPS endpoint, IMDS-attested), validated by e2e showing the node can reach LPS pre-kubelet. The fetch/auth rewrite lives in #8696 (2.1b). This wrapper wiring is channel-agnostic: the check-hotfix -> download-hotfix call sequence, the ENABLE_PROVISIONING_HOTFIX gate (relaxed by 2.1d via the enable_provisioning_hotfix contract field), and the fail-open semantics are all unchanged. Only comments/wording were updated to name the new read channel.

@Devinwong Devinwong force-pushed the devinwong/anc-check-hotfix-configmap branch 2 times, most recently from 07b497b to 0d6f945 Compare June 20, 2026 00:25
@Devinwong Devinwong marked this pull request as draft June 20, 2026 01:20
Devin Wong and others added 2 commits June 22, 2026 17:49
feat(anc): wire check-hotfix into node wrapper behind ANC_HOTFIX_ENABLED
09b8cd3 
Add a default-off ANC_HOTFIX_ENABLED-gated call to the 2.1b check-hotfix
subcommand in aks-node-controller-wrapper.sh, placed before the existing
download-hotfix block since check-hotfix refreshes the hotfix pointer that
block consumes. The call is fail-open and wrapped defensively so it can never
block provisioning. When the flag is unset/non-true the wrapper behaves exactly
as before (6-month VHD backward compat). Parameterize HOTFIX_JSON to match the
existing path-var pattern and enable shellspec coverage of the download-hotfix
branch. Add shellspec tests for flag off, flag on ordering, fail-open, and
non-true value handling.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
test(anc): note old-VHD fail-open coverage in wrapper spec
41b325f 
Clarify that the check-hotfix non-zero (fail-open) case also models a node whose
VHD-baked binary predates 2.1b, where check-hotfix is an unknown subcommand.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Devin Wong and others added 2 commits June 22, 2026 17:49
refactor(anc): rename wrapper hotfix gate to ENABLE_PROVISIONING_HOTFIX
b5258bb 
Match the design's EnableProvisioningHotfix aks-rp region toggle and AgentBaker's
contract->env naming convention (EnableIMDSRestriction -> ENABLE_IMDS_RESTRICTION),
so the toggle -> absvc -> ANC opt-in chain stays traceable. No behavior change;
still default-off and fail-open.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
docs(anc): reword wrapper hotfix comment for LPS endpoint read channel
abdcd9f 
The hotfix pointer read channel moved from the kube-system ConfigMap (apiserver +
bootstrap token) to the LPS endpoint (IMDS-attested); the fetch/auth rewrite lives in
2.1b. The wrapper's check-hotfix -> download-hotfix call contract, the
ENABLE_PROVISIONING_HOTFIX gate, and the fail-open semantics are unchanged - only the
explanatory comment is updated to name the new read channel accurately.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@Devinwong Devinwong force-pushed the devinwong/anc-wire-check-hotfix-wrapper branch from 9b0f1fd to abdcd9f Compare June 23, 2026 01:00
@github-actions

github-actions Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Changes cached containers or packages on windows VHDs

Please get a Windows SIG member to approve.

The following dif file shows any additions or deletions from what will be cached on windows VHDs organised by VHD type.

  • Additions are new things cached.
  • Deletions are things no longer cached.
diff --git a/vhd_files/2022-containerd-gen2.txt b/vhd_files/2022-containerd-gen2.txt
index db10c9e..c51a47f 100644
--- a/vhd_files/2022-containerd-gen2.txt
+++ b/vhd_files/2022-containerd-gen2.txt
@@ -122,0 +123 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.34.6-windows-hp
+mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.35.2-windows-hp
@@ -124 +124,0 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.35.3-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.35.4-windows-hp
@@ -129,0 +130 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.11-windows-hpc-1
@@ -131 +131,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hpc-1
@@ -133 +133,2 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.8-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.3-windows-hpc-1
@@ -135 +135,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc-1
@@ -137 +136,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc-1
diff --git a/vhd_files/2022-containerd.txt b/vhd_files/2022-containerd.txt
index 94de353..7312c49 100644
--- a/vhd_files/2022-containerd.txt
+++ b/vhd_files/2022-containerd.txt
@@ -122,0 +123 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.34.6-windows-hp
+mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.35.2-windows-hp
@@ -124 +124,0 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.35.3-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.35.4-windows-hp
@@ -129,0 +130 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.11-windows-hpc-1
@@ -131 +131,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hpc-1
@@ -133 +133,2 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.8-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.3-windows-hpc-1
@@ -135 +135,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc-1
@@ -137 +136,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc-1
diff --git a/vhd_files/2025-gen2.txt b/vhd_files/2025-gen2.txt
index d0ea692..36e3641 100644
--- a/vhd_files/2025-gen2.txt
+++ b/vhd_files/2025-gen2.txt
@@ -52,0 +53 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.34.6-windows-hp
+mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.35.2-windows-hp
@@ -54 +54,0 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.35.3-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.35.4-windows-hp
@@ -59,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.11-windows-hpc-1
@@ -61 +61,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hpc-1
@@ -63 +63,2 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.8-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.3-windows-hpc-1
@@ -65 +65,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc-1
@@ -67 +66,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc-1
diff --git a/vhd_files/2025.txt b/vhd_files/2025.txt
index ab44d8b..b8873d5 100644
--- a/vhd_files/2025.txt
+++ b/vhd_files/2025.txt
@@ -52,0 +53 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.34.6-windows-hp
+mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.35.2-windows-hp
@@ -54 +54,0 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.35.3-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes-csi/azurefile-csi:v1.35.4-windows-hp
@@ -59,0 +60 @@ mcr.microsoft.com/oss/v2/kubernetes-csi/secrets-store/driver:v1.5.4
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.11-windows-hpc-1
@@ -61 +61,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.13-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.33.14-windows-hpc-1
@@ -63 +63,2 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.10-windows-hp
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.11-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.34.8-windows-hpc-1
+mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.3-windows-hpc-1
@@ -65 +65,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.5-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.35.6-windows-hpc-1
@@ -67 +66,0 @@ mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.1-windows-hpc
-mcr.microsoft.com/oss/v2/kubernetes/azure-cloud-node-manager:v1.36.2-windows-hpc-1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cameronmeissner cameronmeissner Awaiting requested review from cameronmeissner cameronmeissner is a code owner
@ganeshkumarashok ganeshkumarashok Awaiting requested review from ganeshkumarashok ganeshkumarashok is a code owner
@lilypan26 lilypan26 Awaiting requested review from lilypan26 lilypan26 is a code owner
@AbelHu AbelHu Awaiting requested review from AbelHu AbelHu is a code owner
@djsly djsly Awaiting requested review from djsly djsly is a code owner
@phealy phealy Awaiting requested review from phealy phealy is a code owner
@r2k1 r2k1 Awaiting requested review from r2k1 r2k1 is a code owner
@timmy-wright timmy-wright Awaiting requested review from timmy-wright timmy-wright is a code owner
@zachary-bailey zachary-bailey Awaiting requested review from zachary-bailey zachary-bailey is a code owner
@awesomenix awesomenix Awaiting requested review from awesomenix awesomenix is a code owner
@mxj220 mxj220 Awaiting requested review from mxj220 mxj220 is a code owner
@pdamianov-dev pdamianov-dev Awaiting requested review from pdamianov-dev pdamianov-dev is a code owner
@calvin197 calvin197 Awaiting requested review from calvin197 calvin197 is a code owner
@sulixu sulixu Awaiting requested review from sulixu sulixu is a code owner
@surajssd surajssd Awaiting requested review from surajssd surajssd is a code owner
@SriHarsha001 SriHarsha001 Awaiting requested review from SriHarsha001 SriHarsha001 is a code owner
@runzhen runzhen Awaiting requested review from runzhen runzhen is a code owner
@xuexu6666 xuexu6666 Awaiting requested review from xuexu6666 xuexu6666 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@Devinwong