Tenant csi HelmRelease remediates by uninstall, and its 10m install budget is spent pulling latest-tagged images

[lexfrei]

packages/apps/kubernetes/templates/helmreleases/csi.yaml carries remediation.retries: -1 on install and upgrade (:24-30) with no install.strategy, so a failed install is remediated by uninstalling the release. That is the same configuration #3552 changes for cilium. The general form is known and covers every tenant addon HelmRelease the chart renders; this issue is about the one chain that has now been seen ending a suite on its own.

What was observed

Run 31025557528 on 4d16f4fdb, job E2E Tests, suite kubernetes-previous, failing at hack/e2e-chainsaw/_lib/run-kubernetes.sh:677:

error: timed out waiting for the condition on helmreleases/kubernetes-test-previous-version-csi

That run carries #3552, so the tenant cilium release no longer remediates by uninstall. Its events are InstallFailed and then UpgradeSucceeded with nothing torn down between them, both tenant workers reach Ready, and node join passes for the first time on that branch. With cilium no longer ending the suite earlier, csi is what ends it.

The csi events from the same run, ages as printed in a dump taken at 18:52:36:

6m20s  Warning  InstallFailed       helmrelease/kubernetes-test-previous-version-csi  Helm install failed ... timeout waiting for: [DaemonSet/cozy-csi/kubevirt-csi-node status: 'InProgress']
6m20s  Normal   UninstallSucceeded  helmrelease/kubernetes-test-previous-version-csi  Helm uninstall remediation for release ...

Three things compound

Times below are the dump timestamp minus the printed event age, and carry that granularity.

dependsOn starts the clock too early. csi.yaml:60 depends on <release>-cilium, so the install begins when cilium reports Ready, at roughly 18:36:36. The tenant workers reached Ready at roughly 18:44, from node ages of 4m41s and 5m30s read at 18:49. kubevirt-csi-node is a DaemonSet, so for about eight of its ten minutes there were no nodes to schedule on and no way for the rollout to progress.

The strategy turns a slow install into a restart. With no install.strategy.name: RetryOnFailure next to the retries: -1 that never exhausts (csi.yaml:24-27), the install that failed at 18:46:16 is remediated by an uninstall, and the next attempt starts from a torn-down release instead of resuming a rollout that had finally acquired nodes.

The suite budget cannot absorb one retry. csi.yaml:12 sets timeout: 10m, and run-kubernetes.sh:677 waits --timeout=10m on the same release. The two are equal, so the first attempt to exhaust its own timeout is terminal for the suite whatever the retry does next. The uninstall at 18:46:16 falls inside the suite window, which ran from 18:39:29 to 18:49:29.

Relation to #3555

#3555 records that the dependency gate protects the tenant addons declaring dependsOn, since a release waiting on a dependency returns before the release reconciler runs and never increments a failure count. That holds, and this narrows it: the gate protects only while the dependency is not ready. csi fails right after the gate opens, and it fails partly because the gate opened, since that is the moment its own timeout starts counting while the workers still do not exist. A release whose readiness depends on nodes needs a budget that starts from something later than the CNI.

Shape of a fix

#3552 is the precedent for the strategy part, on both actions for the reason given there. The other two are independent of it. The budget equality is a test-side choice, and the early clock is a property of what csi waits for rather than of the remediation strategy. Fixing only the strategy leaves a csi install that still spends most of its timeout before a node exists.

[lexfrei]

Two claims in the body do not survive a recheck. Both are corrected below, and the recheck found the actual cause.

The first is "for about eight of its ten minutes there were no nodes to schedule on". The tenant snapshot has the DaemonSet creating both pods at 18:36:16 and both Scheduled in the same second, onto md0-ncqcf-7zdx6 and md0-ncqcf-txw2m. Nodes were present for the whole window. That claim came from reading a node AGE column against the timestamp of the chainsaw block containing it. Chainsaw buffers a script's stdout and emits it once when the script exits, so every line in the block carries the exit time and a relative age read against it means nothing. run-kubernetes.sh:539 prints that table right after the node-join gate at :462, minutes before the csi wait at :677.

The second is "the suite budget cannot absorb one retry". The csi events are InstallFailed and UninstallSucceeded at 18:46:16, then InstallSucceeded at 18:46:43. Teardown and reinstall cost 27 seconds and finished 2m46s before the suite wait expired at 18:49:29.

What consumed the ten minutes is image pulls. Both pods were scheduled at 18:36:16 and then sat fetching. On one node quay.io/openshift/origin-csi-node-driver-registrar:latest took 2m16.833s, on the other 4m1.224s, and the kubevirt-csi-driver image ahead of it took about five minutes there. The second pod started its last container at 18:46:22, six seconds after the install timeout fired at 18:46:16, and the uninstall killed it at 18:46:23. Both registrar and livenessprobe are latest tags from a public registry, at packages/system/kubevirt-csi-node/templates/deploy.yaml:204 and :228 and again at packages/apps/kubernetes/templates/csi/deploy.yaml:141, so every tenant install fetches them fresh.

The release still never went Ready, and that part is not explained by the evidence. The reinstall created new pods at 18:46:28; both stayed Pending with all three containers in ContainerCreating and PodReadyToStartContainers: False through to 18:49:29, emitting no events in those three minutes, and the DaemonSet read desiredNumberScheduled: 2, numberReady: 0, numberUnavailable: 2. The images were already local by then, so this is not a pull. The tenant crust-gather's 180s window closed at 18:49:29 and the tenant was torn down, so what held the sandbox is not recoverable from this run.

Two things stand. The remediation shape is real, and here it tore down a rollout that was six seconds from complete. And the timers are offset: the install began at 18:36:16 when the dependency gate opened while the suite's wait began at 18:39:29, so with both budgets at 10m the suite only ever sees the install's last 6m47s.