Helm chart: `imageRegistry` strips source registry prefix from CSI sidecar images, making air-gapped/mirror deployments impossible

[broboa]

Describe the bug

When imageRegistry is set in values.yaml, the operator applies it to all images — including CSI sidecars — using ReplaceImageRegistry, which discards the entire source registry and path, keeping only the base image name and tag.

The Trident image (netapp/trident) and CSI sidecar images originate from different upstream registries:

• docker.io/netapp/trident:* — NetApp image
• docker.io/netapp/trident-autosupport:* — NetApp image
• registry.k8s.io/sig-storage/csi-provisioner:* — Kubernetes SIG Storage image
• registry.k8s.io/sig-storage/csi-attacher:*
• registry.k8s.io/sig-storage/csi-resizer:*
• registry.k8s.io/sig-storage/csi-snapshotter:*
• registry.k8s.io/sig-storage/csi-node-driver-registrar:*
• registry.k8s.io/sig-storage/livenessprobe:*

A standard Artifactory (or similar) mirror preserves the upstream path structure under a prefix, e.g.:

artifactory.example.com/docker.io/netapp/trident:26.02.1
artifactory.example.com/registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.16.0

These two sets of images cannot share a single imageRegistry prefix because their upstream org paths differ. Setting imageRegistry: artifactory.example.com/docker.io/netapp resolves the Trident image correctly but produces broken paths for sidecars:

# produced (broken)
artifactory.example.com/docker.io/netapp/csi-node-driver-registrar:v2.16.0

# correct
artifactory.example.com/registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.16.0

Setting imageRegistry: artifactory.example.com/registry.k8s.io/sig-storage fixes the sidecars but breaks Trident:

# produced (broken)
artifactory.example.com/registry.k8s.io/sig-storage/trident:26.02.1

There is no value for imageRegistry that satisfies both sets simultaneously.

Root cause

ReplaceImageRegistry in pkg/network/network.go unconditionally strips everything except the basename:

func ReplaceImageRegistry(image, registry string) string {
    remainder := GetBaseImageName(image) // returns only "csi-node-driver-registrar:v2.16.0"
    if registry == "" {
        return remainder
    }
    return registry + "/" + remainder
}

This is called for every sidecar image in installer.go:

for _, sidecarImage := range sidecarImages {
    if *sidecarImage != "" {
        *sidecarImage = network.ReplaceImageRegistry(*sidecarImage, cr.Spec.ImageRegistry)
    }
}

The function discards the source registry and org path entirely, so different-origin images cannot be mirrored under a single registry prefix that preserves their layout.

Environment

• Trident version: 26.02.1
• Installation method: Helm chart (trident-operator), TridentOrchestrator CR created by operator
• Kubernetes version: 1.33.6
• Container runtime: containerd 1.7.29
• OS: Rocky Linux 9.7 (amd64)
• Registry: Artifactory with virtual repositories proxying docker.io and registry.k8s.io under separate paths

To reproduce

1. Set up an Artifactory mirror with:

   • artifactory.example.com/docker.io/netapp/trident:26.02.1
   • artifactory.example.com/registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.16.0
   • (and the remaining CSI sidecars under registry.k8s.io/sig-storage/)

2. Install via Helm with:

   imageRegistry: "artifactory.example.com/docker.io/netapp"

3. Observe that the Trident operator pod starts but the CSI sidecar containers enter ImagePullBackOff, attempting to pull from artifactory.example.com/docker.io/netapp/csi-node-driver-registrar:v2.16.0 instead of artifactory.example.com/registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.16.0.

Expected behavior

The Helm chart should expose an extraEnv field in values.yaml that appends entries to the operator container's env: block in templates/deployment.yaml. The operator already reads TRIDENT_CSI_SIDECAR_*_IMAGE env vars (operator/config/config.go), so no binary changes are needed — the fix is entirely in the chart.

A values.yaml shape that would work:

extraEnv:
  - name: TRIDENT_CSI_SIDECAR_PROVISIONER_IMAGE
    value: "artifactory.example.com/registry.k8s.io/sig-storage/csi-provisioner:v6.2.0"
  - name: TRIDENT_CSI_SIDECAR_ATTACHER_IMAGE
    value: "artifactory.example.com/registry.k8s.io/sig-storage/csi-attacher:v4.11.0"
  - name: TRIDENT_CSI_SIDECAR_RESIZER_IMAGE
    value: "artifactory.example.com/registry.k8s.io/sig-storage/csi-resizer:v2.1.0"
  - name: TRIDENT_CSI_SIDECAR_SNAPSHOTTER_IMAGE
    value: "artifactory.example.com/registry.k8s.io/sig-storage/csi-snapshotter:v8.5.0"
  - name: TRIDENT_CSI_SIDECAR_NODE_DRIVER_REGISTRAR_IMAGE
    value: "artifactory.example.com/registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.16.0"
  - name: TRIDENT_CSI_SIDECAR_LIVENESS_PROBE_IMAGE
    value: "artifactory.example.com/registry.k8s.io/sig-storage/livenessprobe:v2.18.0"

And the corresponding addition to templates/deployment.yaml:

{{- with .Values.extraEnv }}
{{- toYaml . | nindent 8 }}
{{- end }}

This requires no changes to the operator binary. The entire fix is a two-line addition to the chart.

Additional context

The env vars TRIDENT_CSI_SIDECAR_*_IMAGE are defined in operator/config/config.go and consumed in operator/controllers/orchestrator/installer/installer.go. They accept full image references (registry + path + tag), bypassing the ReplaceImageRegistry logic entirely. They are the correct hook point — they just need to be surfaced through the chart.

There is currently no supported way to set these env vars. The templates/deployment.yaml in the Helm chart has a hardcoded env: block with only POD_NAME, OPERATOR_NAME, and an Azure-specific conditional. There is no extraEnv or equivalent escape hatch in values.yaml. The only workaround is a post-install kubectl set env or a manual patch against the operator Deployment, both of which are overwritten on the next helm upgrade.

[torirevilla]

@broboa for air-gapped environments take a look at the docs for helm offline installation here. Under imageRegistry the docs specifically suggest to not use /netapp/ in the repository path. Additionally, setting the specific tridentImage or operatorImage with the full path will use what is provided in that field rather than using the imageRegistry that is set.