toolkit-validation container fails with "nvidia-smi": executable file not found in $PATH (after clean installation on Ubuntu 24.04.2 + Kubespray 1.32.2)

[botterweck]

• Installed Ubuntu 24.04.2 LTS from scratch
• Installed Kubernetes (Kubespray 1.32.2 via Ansible)
• Installed NVIDIA GPU Operator (via Helm)

Problem

• Pod nvidia-operator-validator-lrjkw is stuck in Status CrashLoopBackOff
• Container toolkit-validation failes with "Error: error validating toolkit installation: exec: "nvidia-smi": executable file not found in $PATH"

Notes

• I did NOT select "install third-party drivers like NVIDIA" during Ubuntu install, since I assume it would be better leave it to GPU Operator to pick the right drivers.

Similar Issues

• "nvidia-smi": executable file not found in $PATH: unknown #707
• toolkit installation container can't find nvidia-smi #265
• toolkit-validation container errors out with "nvidia-smi": executable file not found in $PATH after migrating to containerd #936

System Summary

OS: Ubuntu 24.04.2 LTS
Kernel: 6.11.0-21-generic
Computer: Dell Precision 7680
GPU: NVIDIA RTX 4090 (Laptop)
Container Runtime Type: contained 2.0.3
K8s: Kubeflow v1.32.2

GPU Operator Installation

helm install --wait --generate-name \
    -n gpu-operator --create-namespace \
    nvidia/gpu-operator \
    --version=v25.3.0

GPU Operator Namespace Pods Status

kubectl get pods -n gpu-operator

NAME                                                              READY   STATUS                  RESTARTS       AGE
gpu-feature-discovery-88m4r                                       0/1     Init:0/1                0              75m
gpu-operator-1743284421-node-feature-discovery-gc-848dd4fb28gnd   1/1     Running                 0              76m
gpu-operator-1743284421-node-feature-discovery-master-5579j6m7k   1/1     Running                 0              76m
gpu-operator-1743284421-node-feature-discovery-worker-zwxkm       1/1     Running                 0              76m
gpu-operator-6c54bfb6d5-p7bxb                                     1/1     Running                 0              76m
nvidia-container-toolkit-daemonset-rgh2q                          1/1     Running                 0              75m
nvidia-dcgm-exporter-n6fzb                                        0/1     Init:0/1                0              75m
nvidia-device-plugin-daemonset-fqtxs                              0/1     Init:0/1                0              75m
nvidia-driver-daemonset-4p7qm                                     1/1     Running                 0              76m
nvidia-operator-validator-lrjkw                                   0/1     Init:CrashLoopBackOff   19 (98s ago)   75m

Log of Container "toolkit-validation"

kubectl logs -n gpu-operator nvidia-operator-validator-lrjkw -c toolkit-validation

time="2025-03-29T22:55:39Z" level=info msg="version: b5479aaa-amd64, commit: b5479aa"
toolkit is not ready
time="2025-03-29T22:55:39Z" level=info msg="Error: error validating toolkit installation: exec: \"nvidia-smi\": executable file not found in $PATH"

Description for pod nvidia-operator-validator-lrjkw

kubectl describe pod -n gpu-operator nvidia-operator-validator-lrjkw

Name:                 nvidia-operator-validator-lrjkw
Namespace:            gpu-operator
Priority:             2000001000
Priority Class Name:  system-node-critical
Runtime Class Name:   nvidia
Service Account:      nvidia-operator-validator
Node:                 tensor/192.168.1.111
Start Time:           Sat, 29 Mar 2025 21:41:31 +0000
Labels:               app=nvidia-operator-validator
                      app.kubernetes.io/managed-by=gpu-operator
                      app.kubernetes.io/part-of=gpu-operator
                      controller-revision-hash=7fcd444d54
                      helm.sh/chart=gpu-operator-v25.3.0
                      pod-template-generation=1
Annotations:          cni.projectcalico.org/containerID: 62e5554f1d12b7abc6a447b23682285922cbd6626a442d0912c67ccf4b5ef240
                      cni.projectcalico.org/podIP: 10.233.106.205/32
                      cni.projectcalico.org/podIPs: 10.233.106.205/32
Status:               Pending
IP:                   10.233.106.205
IPs:
  IP:           10.233.106.205
Controlled By:  DaemonSet/nvidia-operator-validator
Init Containers:
  driver-validation:
    Container ID:  containerd://9d9b53912922d5bbe3d768d8d5305d7cb8988819f33763e5db7195eaa82ab2fa
    Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
    Image ID:      nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:07b93914425148f936157ad295649ce100b91b29394669031a585d2458c9f39f
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      nvidia-validator
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 29 Mar 2025 21:43:27 +0000
      Finished:     Sat, 29 Mar 2025 21:43:27 +0000
    Ready:          True
    Restart Count:  0
    Environment:
      WITH_WAIT:           true
      COMPONENT:           driver
      OPERATOR_NAMESPACE:  gpu-operator (v1:metadata.namespace)
    Mounts:
      /host from host-root (ro)
      /host-dev-char from host-dev-char (rw)
      /run/nvidia/driver from driver-install-dir (rw)
      /run/nvidia/validations from run-nvidia-validations (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xbqr5 (ro)
  toolkit-validation:
    Container ID:  containerd://3e2b201f61a786725407f0ca66426fabc50734ea2b2acac8f1164298cdba6669
    Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
    Image ID:      nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:07b93914425148f936157ad295649ce100b91b29394669031a585d2458c9f39f
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      nvidia-validator
    State:          Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sat, 29 Mar 2025 23:00:42 +0000
      Finished:     Sat, 29 Mar 2025 23:00:42 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sat, 29 Mar 2025 22:55:39 +0000
      Finished:     Sat, 29 Mar 2025 22:55:39 +0000
    Ready:          False
    Restart Count:  20
    Environment:
      NVIDIA_VISIBLE_DEVICES:  all
      WITH_WAIT:               false
      COMPONENT:               toolkit
    Mounts:
      /run/nvidia/validations from run-nvidia-validations (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xbqr5 (ro)
  cuda-validation:
    Container ID:
    Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      nvidia-validator
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      WITH_WAIT:                    false
      COMPONENT:                    cuda
      NODE_NAME:                     (v1:spec.nodeName)
      OPERATOR_NAMESPACE:           gpu-operator (v1:metadata.namespace)
      VALIDATOR_IMAGE:              nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
      VALIDATOR_IMAGE_PULL_POLICY:  IfNotPresent
      VALIDATOR_RUNTIME_CLASS:      nvidia
    Mounts:
      /run/nvidia/validations from run-nvidia-validations (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xbqr5 (ro)
  plugin-validation:
    Container ID:
    Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      nvidia-validator
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      COMPONENT:                    plugin
      WITH_WAIT:                    false
      WITH_WORKLOAD:                false
      MIG_STRATEGY:                 single
      NODE_NAME:                     (v1:spec.nodeName)
      OPERATOR_NAMESPACE:           gpu-operator (v1:metadata.namespace)
      VALIDATOR_IMAGE:              nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
      VALIDATOR_IMAGE_PULL_POLICY:  IfNotPresent
      VALIDATOR_RUNTIME_CLASS:      nvidia
    Mounts:
      /run/nvidia/validations from run-nvidia-validations (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xbqr5 (ro)
Containers:
  nvidia-operator-validator:
    Container ID:
    Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      echo all validations are successful; sleep infinity
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /run/nvidia/validations from run-nvidia-validations (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xbqr5 (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 False
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  run-nvidia-validations:
    Type:          HostPath (bare host directory volume)
    Path:          /run/nvidia/validations
    HostPathType:  DirectoryOrCreate
  driver-install-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /run/nvidia/driver
    HostPathType:
  host-root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:
  host-dev-char:
    Type:          HostPath (bare host directory volume)
    Path:          /dev/char
    HostPathType:
  kube-api-access-xbqr5:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              nvidia.com/gpu.deploy.operator-validator=true
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
                             nvidia.com/gpu:NoSchedule op=Exists
Events:
  Type     Reason   Age                   From     Message
  ----     ------   ----                  ----     -------
  Warning  BackOff  4m7s (x342 over 77m)  kubelet  Back-off restarting failed container toolkit-validation in pod nvidia-operator-validator-lrjkw_gpu-operator(e08c4646-dd57-4a5a-8df9-c1a4191e5fb2)
  Normal   Pulled   5s (x21 over 77m)     kubelet  Container image "nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0" already present on machine

[cdesiniotis]

Hi @botterweck, can you provide the following information:

• nvidia-smi output, by running sudo chroot /run/nvidia/driver nvidia-smi on the host
• driver system logs, by running sudo dmesg | grep -i nvrm
• the current containerd configuration, by running containerd config dump

[ameijer]

hi @botterweck & @cdesiniotis! We had the exact same issue, only difference is we are using cri-o. We set cdi.enabled: true and cdi.default: true in our chart values for the operator, and that resolved the issue. We successfully reproduced that sequence multiple times. Exact same installation without CDI would fail to find the SMI and would send the validator pod into a crash loop. Setting those CDI helm values to true fixed the problem.

[botterweck]

@cdesiniotis thanks for your questions.

I have meanwhile done another attempt with a clean reinstall. This time before starting the installation of GPU Operator, I installed an NVIDIA driver from the CUDA repository, that resulted in NVIDIA driver 570.124.06, which is listed in the
GPU Operator Component Matrix as "default, recommended" for GPU Operator v25.3.0 (current version).

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt-get update
sudo apt-get install -y nvidia-open

Output of nvidia-smi

Tue Apr  1 22:07:51 2025
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.124.06             Driver Version: 570.124.06     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 4090 ...    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   41C    P3            593W /   90W |      15MiB /  16376MiB |     14%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A            1953      G   /usr/lib/xorg/Xorg                        4MiB |
+-----------------------------------------------------------------------------------------+

Output of sudo chroot /run/nvidia/driver nvidia-smi

chroot: failed to run command ‘nvidia-smi’: No such file or directory

Output of sudo dmesg | grep -i nvrm

[    3.904100] NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64  570.124.06  Release Build  (dvs-builder@U22-I3-AE18-09-6)  Wed Feb 26 01:52:55 UTC 2025
[    5.000036] NVRM: testIfDsmSubFunctionEnabled: GPS ACPI DSM called before _acpiDsmSupportedFuncCacheInit subfunction = 11.
[    6.288849] NVRM: testIfDsmSubFunctionEnabled: GPS ACPI DSM called before _acpiDsmSupportedFuncCacheInit subfunction = 11

Output of containerd config dump

version = 3
root = '/var/lib/containerd'
state = '/run/containerd'
temp = ''
plugin_dir = ''
disabled_plugins = []
required_plugins = []
oom_score = 0
imports = []

[grpc]
  address = '/run/containerd/containerd.sock'
  tcp_address = ''
  tcp_tls_ca = ''
  tcp_tls_cert = ''
  tcp_tls_key = ''
  uid = 0
  gid = 0
  max_recv_message_size = 16777216
  max_send_message_size = 16777216

[ttrpc]
  address = ''
  uid = 0
  gid = 0

[debug]
  address = ''
  uid = 0
  gid = 0
  level = 'info'
  format = ''

[metrics]
  address = ''
  grpc_histogram = false

[plugins]
  [plugins.'io.containerd.cri.v1.images']
    snapshotter = 'overlayfs'
    disable_snapshot_annotations = true
    discard_unpacked_layers = true
    max_concurrent_downloads = 3
    image_pull_progress_timeout = '5m'
    image_pull_with_sync_fs = false
    stats_collect_period = 10

    [plugins.'io.containerd.cri.v1.images'.pinned_images]
      sandbox = 'registry.k8s.io/pause:3.10'

    [plugins.'io.containerd.cri.v1.images'.registry]
      config_path = '/etc/containerd/certs.d'

    [plugins.'io.containerd.cri.v1.images'.image_decryption]
      key_model = 'node'

  [plugins.'io.containerd.cri.v1.runtime']
    enable_selinux = false
    selinux_category_range = 1024
    max_container_log_line_size = 16384
    disable_apparmor = false
    restrict_oom_score_adj = false
    disable_proc_mount = false
    unset_seccomp_profile = ''
    tolerate_missing_hugetlb_controller = true
    disable_hugetlb_controller = true
    device_ownership_from_security_context = false
    ignore_image_defined_volumes = false
    netns_mounts_under_state_dir = false
    enable_unprivileged_ports = false
    enable_unprivileged_icmp = false
    enable_cdi = true
    cdi_spec_dirs = ['/etc/cdi', '/var/run/cdi']
    drain_exec_sync_io_timeout = '0s'
    ignore_deprecation_warnings = []

    [plugins.'io.containerd.cri.v1.runtime'.containerd]
      default_runtime_name = 'nvidia'
      ignore_blockio_not_enabled_errors = false
      ignore_rdt_not_enabled_errors = false

      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nvidia]
          runtime_type = 'io.containerd.runc.v2'
          runtime_path = ''
          pod_annotations = []
          container_annotations = []
          privileged_without_host_devices = false
          privileged_without_host_devices_all_devices_allowed = false
          base_runtime_spec = '/etc/containerd/cri-base.json'
          cni_conf_dir = ''
          cni_max_conf_num = 0
          snapshotter = ''
          sandboxer = 'podsandbox'
          io_type = ''

          [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nvidia.options]
            BinaryName = '/usr/local/nvidia/toolkit/nvidia-container-runtime'
            CriuImagePath = ''
            CriuWorkPath = ''
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            Root = ''
            ShimCgroup = ''
            binaryName = '/usr/local/bin/runc'
            systemdCgroup = true

        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nvidia-cdi]
          runtime_type = 'io.containerd.runc.v2'
          runtime_path = ''
          pod_annotations = []
          container_annotations = []
          privileged_without_host_devices = false
          privileged_without_host_devices_all_devices_allowed = false
          base_runtime_spec = '/etc/containerd/cri-base.json'
          cni_conf_dir = ''
          cni_max_conf_num = 0
          snapshotter = ''
          sandboxer = 'podsandbox'
          io_type = ''

          [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nvidia-cdi.options]
            BinaryName = '/usr/local/nvidia/toolkit/nvidia-container-runtime.cdi'
            CriuImagePath = ''
            CriuWorkPath = ''
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            Root = ''
            ShimCgroup = ''
            binaryName = '/usr/local/bin/runc'
            systemdCgroup = true

        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nvidia-legacy]
          runtime_type = 'io.containerd.runc.v2'
          runtime_path = ''
          pod_annotations = []
          container_annotations = []
          privileged_without_host_devices = false
          privileged_without_host_devices_all_devices_allowed = false
          base_runtime_spec = '/etc/containerd/cri-base.json'
          cni_conf_dir = ''
          cni_max_conf_num = 0
          snapshotter = ''
          sandboxer = 'podsandbox'
          io_type = ''

          [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.nvidia-legacy.options]
            BinaryName = '/usr/local/nvidia/toolkit/nvidia-container-runtime.legacy'
            CriuImagePath = ''
            CriuWorkPath = ''
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            Root = ''
            ShimCgroup = ''
            binaryName = '/usr/local/bin/runc'
            systemdCgroup = true

        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
          runtime_type = 'io.containerd.runc.v2'
          runtime_path = ''
          pod_annotations = []
          container_annotations = []
          privileged_without_host_devices = false
          privileged_without_host_devices_all_devices_allowed = false
          base_runtime_spec = '/etc/containerd/cri-base.json'
          cni_conf_dir = ''
          cni_max_conf_num = 0
          snapshotter = ''
          sandboxer = 'podsandbox'
          io_type = ''

          [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
            BinaryName = ''
            CriuImagePath = ''
            CriuWorkPath = ''
            IoGid = 0
            IoUid = 0
            NoNewKeyring = false
            Root = ''
            ShimCgroup = ''
            binaryName = '/usr/local/bin/runc'
            systemdCgroup = true

    [plugins.'io.containerd.cri.v1.runtime'.cni]
      bin_dir = '/opt/cni/bin'
      conf_dir = '/etc/cni/net.d'
      max_conf_num = 1
      setup_serially = false
      conf_template = ''
      ip_pref = ''
      use_internal_loopback = false

  [plugins.'io.containerd.gc.v1.scheduler']
    pause_threshold = 0.02
    deletion_threshold = 0
    mutation_threshold = 100
    schedule_delay = '0s'
    startup_delay = '100ms'

  [plugins.'io.containerd.grpc.v1.cri']
    disable_tcp_service = true
    stream_server_address = '127.0.0.1'
    stream_server_port = '0'
    stream_idle_timeout = '4h0m0s'
    enable_tls_streaming = false

    [plugins.'io.containerd.grpc.v1.cri'.x509_key_pair_streaming]
      tls_cert_file = ''
      tls_key_file = ''

  [plugins.'io.containerd.image-verifier.v1.bindir']
    bin_dir = '/opt/containerd/image-verifier/bin'
    max_verifiers = 10
    per_verifier_timeout = '10s'

  [plugins.'io.containerd.internal.v1.opt']
    path = '/opt/containerd'

  [plugins.'io.containerd.internal.v1.tracing']

  [plugins.'io.containerd.metadata.v1.bolt']
    content_sharing_policy = 'shared'

  [plugins.'io.containerd.monitor.container.v1.restart']
    interval = '10s'

  [plugins.'io.containerd.monitor.task.v1.cgroups']
    no_prometheus = false

  [plugins.'io.containerd.nri.v1.nri']
    disable = false
    socket_path = '/var/run/nri/nri.sock'
    plugin_path = '/opt/nri/plugins'
    plugin_config_path = '/etc/nri/conf.d'
    plugin_registration_timeout = '5s'
    plugin_request_timeout = '2s'
    disable_connections = false

  [plugins.'io.containerd.runtime.v2.task']
    platforms = ['linux/amd64']

  [plugins.'io.containerd.service.v1.diff-service']
    default = ['walking']
    sync_fs = false

  [plugins.'io.containerd.service.v1.tasks-service']
    blockio_config_file = ''
    rdt_config_file = ''

  [plugins.'io.containerd.shim.v1.manager']
    env = []

  [plugins.'io.containerd.snapshotter.v1.blockfile']
    root_path = ''
    scratch_file = ''
    fs_type = ''
    mount_options = []
    recreate_scratch = false

  [plugins.'io.containerd.snapshotter.v1.btrfs']
    root_path = ''

  [plugins.'io.containerd.snapshotter.v1.devmapper']
    root_path = ''
    pool_name = ''
    base_image_size = ''
    async_remove = false
    discard_blocks = false
    fs_type = ''
    fs_options = ''

  [plugins.'io.containerd.snapshotter.v1.native']
    root_path = ''

  [plugins.'io.containerd.snapshotter.v1.overlayfs']
    root_path = ''
    upperdir_label = false
    sync_remove = false
    slow_chown = false
    mount_options = []

  [plugins.'io.containerd.snapshotter.v1.zfs']
    root_path = ''

  [plugins.'io.containerd.tracing.processor.v1.otlp']

  [plugins.'io.containerd.transfer.v1.local']
    max_concurrent_downloads = 3
    max_concurrent_uploaded_layers = 3
    config_path = ''

[cgroup]
  path = ''

[timeouts]
  'io.containerd.timeout.bolt.open' = '0s'
  'io.containerd.timeout.metrics.shimstats' = '2s'
  'io.containerd.timeout.shim.cleanup' = '5s'
  'io.containerd.timeout.shim.load' = '5s'
  'io.containerd.timeout.shim.shutdown' = '3s'
  'io.containerd.timeout.task.state' = '2s'

[stream_processors]
  [stream_processors.'io.containerd.ocicrypt.decoder.v1.tar']
    accepts = ['application/vnd.oci.image.layer.v1.tar+encrypted']
    returns = 'application/vnd.oci.image.layer.v1.tar'
    path = 'ctd-decoder'
    args = ['--decryption-keys-path', '/etc/containerd/ocicrypt/keys']
    env = ['OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf']

  [stream_processors.'io.containerd.ocicrypt.decoder.v1.tar.gzip']
    accepts = ['application/vnd.oci.image.layer.v1.tar+gzip+encrypted']
    returns = 'application/vnd.oci.image.layer.v1.tar+gzip'
    path = 'ctd-decoder'
    args = ['--decryption-keys-path', '/etc/containerd/ocicrypt/keys']
    env = ['OCICRYPT_KEYPROVIDER_CONFIG=/etc/containerd/ocicrypt/ocicrypt_keyprovider.conf']

[botterweck]

A bit more information.

Note the "Failed to create pod sandbox: rpc error: code = Unknown desc = unable to get OCI runtime for sandbox "c7c4e...": no runtime for "nvidia" is configured" in the events section of kubectl describe pod nvidia-operator-validator-qpv7s -n gpu-operator.

Output of kubectl get pods -n gpu-operator

NAME                                                              READY   STATUS                  RESTARTS      AGE
gpu-feature-discovery-bmrgf                                       0/1     Init:0/1                0             22m
gpu-operator-1743541061-node-feature-discovery-gc-5c45647cwbxj6   1/1     Running                 0             22m
gpu-operator-1743541061-node-feature-discovery-master-6c6cbbhfk   1/1     Running                 0             22m
gpu-operator-1743541061-node-feature-discovery-worker-55m2l       1/1     Running                 0             22m
gpu-operator-748c9796cf-7ct6m                                     1/1     Running                 0             22m
nvidia-container-toolkit-daemonset-ktrv6                          1/1     Running                 0             22m
nvidia-dcgm-exporter-bxj8q                                        0/1     Init:0/1                0             22m
nvidia-device-plugin-daemonset-4ghzp                              0/1     Init:0/1                0             22m
nvidia-operator-validator-qpv7s                                   0/1     Init:CrashLoopBackOff   9 (65s ago)   22m

Output of kubectl describe pod nvidia-operator-validator-qpv7s -n gpu-operator

(container toolkit-validation is in state CrashLoopBackOff)

Name:                 nvidia-operator-validator-qpv7s
Namespace:            gpu-operator
Priority:             2000001000
Priority Class Name:  system-node-critical
Runtime Class Name:   nvidia
Service Account:      nvidia-operator-validator
Node:                 tensor/192.168.1.111
Start Time:           Tue, 01 Apr 2025 21:57:48 +0100
Labels:               app=nvidia-operator-validator
                      app.kubernetes.io/managed-by=gpu-operator
                      app.kubernetes.io/part-of=gpu-operator
                      controller-revision-hash=7fcd444d54
                      helm.sh/chart=gpu-operator-v25.3.0
                      pod-template-generation=1
Annotations:          cni.projectcalico.org/containerID: f0f04edb2f0ed7a6a426d34505d8421277f2e34c634ed8ec7e4f7823e53c07cb
                      cni.projectcalico.org/podIP: 10.233.106.217/32
                      cni.projectcalico.org/podIPs: 10.233.106.217/32
Status:               Pending
IP:                   10.233.106.217
IPs:
  IP:           10.233.106.217
Controlled By:  DaemonSet/nvidia-operator-validator
Init Containers:
  driver-validation:
    Container ID:  containerd://c74d8a341fddd45dad6f636ccafcd804535e148f5ccbc2825cb1c00917505361
    Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
    Image ID:      nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:07b93914425148f936157ad295649ce100b91b29394669031a585d2458c9f39f
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      nvidia-validator
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Tue, 01 Apr 2025 21:58:12 +0100
      Finished:     Tue, 01 Apr 2025 21:58:14 +0100
    Ready:          True
    Restart Count:  0
    Environment:
      WITH_WAIT:           true
      COMPONENT:           driver
      OPERATOR_NAMESPACE:  gpu-operator (v1:metadata.namespace)
    Mounts:
      /host from host-root (ro)
      /host-dev-char from host-dev-char (rw)
      /run/nvidia/driver from driver-install-dir (rw)
      /run/nvidia/validations from run-nvidia-validations (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qzq8h (ro)
  toolkit-validation:
    Container ID:  containerd://9b05caa9a95b43bd3da30151312083ba582d2f2b9eafbbcb89c77db8f4774bac
    Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
    Image ID:      nvcr.io/nvidia/cloud-native/gpu-operator-validator@sha256:07b93914425148f936157ad295649ce100b91b29394669031a585d2458c9f39f
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      nvidia-validator
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 01 Apr 2025 22:19:07 +0100
      Finished:     Tue, 01 Apr 2025 22:19:07 +0100
    Ready:          False
    Restart Count:  9
    Environment:
      NVIDIA_VISIBLE_DEVICES:  all
      WITH_WAIT:               false
      COMPONENT:               toolkit
    Mounts:
      /run/nvidia/validations from run-nvidia-validations (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qzq8h (ro)
  cuda-validation:
    Container ID:
    Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      nvidia-validator
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      WITH_WAIT:                    false
      COMPONENT:                    cuda
      NODE_NAME:                     (v1:spec.nodeName)
      OPERATOR_NAMESPACE:           gpu-operator (v1:metadata.namespace)
      VALIDATOR_IMAGE:              nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
      VALIDATOR_IMAGE_PULL_POLICY:  IfNotPresent
      VALIDATOR_RUNTIME_CLASS:      nvidia
    Mounts:
      /run/nvidia/validations from run-nvidia-validations (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qzq8h (ro)
  plugin-validation:
    Container ID:
    Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      nvidia-validator
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      COMPONENT:                    plugin
      WITH_WAIT:                    false
      WITH_WORKLOAD:                false
      MIG_STRATEGY:                 single
      NODE_NAME:                     (v1:spec.nodeName)
      OPERATOR_NAMESPACE:           gpu-operator (v1:metadata.namespace)
      VALIDATOR_IMAGE:              nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
      VALIDATOR_IMAGE_PULL_POLICY:  IfNotPresent
      VALIDATOR_RUNTIME_CLASS:      nvidia
    Mounts:
      /run/nvidia/validations from run-nvidia-validations (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qzq8h (ro)
Containers:
  nvidia-operator-validator:
    Container ID:
    Image:         nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      echo all validations are successful; sleep infinity
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /run/nvidia/validations from run-nvidia-validations (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qzq8h (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 False
  Ready                       False
  ContainersReady             False
  PodScheduled                True
Volumes:
  run-nvidia-validations:
    Type:          HostPath (bare host directory volume)
    Path:          /run/nvidia/validations
    HostPathType:  DirectoryOrCreate
  driver-install-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /run/nvidia/driver
    HostPathType:
  host-root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:
  host-dev-char:
    Type:          HostPath (bare host directory volume)
    Path:          /dev/char
    HostPathType:
  kube-api-access-qzq8h:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              nvidia.com/gpu.deploy.operator-validator=true
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
                             nvidia.com/gpu:NoSchedule op=Exists
Events:
  Type     Reason                  Age                   From               Message
  ----     ------                  ----                  ----               -------
  Normal   Scheduled               24m                   default-scheduler  Successfully assigned gpu-operator/nvidia-operator-validator-qpv7s to tensor
  Warning  FailedCreatePodSandBox  24m                   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = unable to get OCI runtime for sandbox "c7c4e0875eae17e9093756168cc4c9fafc995ff3e194267d7c3ea072780ff9cf": no runtime for "nvidia" is configured
  Warning  FailedCreatePodSandBox  24m                   kubelet            Failed to create pod sandbox: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/containerd/containerd.sock: connect: connection refused"
  Normal   Pulled                  24m                   kubelet            Container image "nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0" already present on machine
  Normal   Created                 24m                   kubelet            Created container: driver-validation
  Normal   Started                 24m                   kubelet            Started container driver-validation
  Normal   Created                 13m (x8 over 24m)     kubelet            Created container: toolkit-validation
  Normal   Started                 13m (x8 over 24m)     kubelet            Started container toolkit-validation
  Warning  BackOff                 4m41s (x92 over 24m)  kubelet            Back-off restarting failed container toolkit-validation in pod nvidia-operator-validator-qpv7s_gpu-operator(6548ae72-c2f3-4755-a532-2b841e9bfe81)
  Normal   Pulled                  3m33s (x10 over 24m)  kubelet            Container image "nvcr.io/nvidia/cloud-native/gpu-operator-validator:v25.3.0" already present on machine

Output of kubectl logs nvidia-operator-validator-qpv7s -n gpu-operator -c toolkit-validation

time="2025-04-01T21:29:21Z" level=info msg="version: b5479aaa-amd64, commit: b5479aa"
time="2025-04-01T21:29:21Z" level=info msg="Error: error validating toolkit installation: exec: \"nvidia-smi\": executable file not found in $PATH"
toolkit is not ready

Output of kubectl logs nvidia-container-toolkit-daemonset-ktrv6 -n gpu-operator

(since that is responsible for installing the nvidia runtime, afaik)

Defaulted container "nvidia-container-toolkit-ctr" out of: nvidia-container-toolkit-ctr, driver-validation (init)
IS_HOST_DRIVER=true
NVIDIA_DRIVER_ROOT=/
DRIVER_ROOT_CTR_PATH=/host
NVIDIA_DEV_ROOT=/
DEV_ROOT_CTR_PATH=/host
time="2025-04-01T20:57:55Z" level=info msg="Parsing arguments"
time="2025-04-01T20:57:55Z" level=info msg="Starting nvidia-toolkit"
time="2025-04-01T20:57:55Z" level=info msg="disabling device node creation since --cdi-enabled=false"
time="2025-04-01T20:57:55Z" level=info msg="Verifying Flags"
time="2025-04-01T20:57:55Z" level=info msg=Initializing
time="2025-04-01T20:57:55Z" level=info msg="Using config version 3"
time="2025-04-01T20:57:55Z" level=info msg="Using CRI runtime plugin name \"io.containerd.cri.v1.runtime\""
time="2025-04-01T20:57:55Z" level=info msg="Installing NVIDIA container toolkit to '/usr/local/nvidia/toolkit'"
time="2025-04-01T20:57:55Z" level=info msg="Removing existing NVIDIA container toolkit installation"
time="2025-04-01T20:57:55Z" level=info msg="Creating directory '/usr/local/nvidia/toolkit'"
time="2025-04-01T20:57:55Z" level=info msg="Creating directory '/usr/local/nvidia/toolkit/.config/nvidia-container-runtime'"
time="2025-04-01T20:57:55Z" level=info msg="Installing NVIDIA container library to '/usr/local/nvidia/toolkit'"
time="2025-04-01T20:57:55Z" level=info msg="Finding library libnvidia-container.so.1 (root=)"
time="2025-04-01T20:57:55Z" level=info msg="Checking library candidate '/usr/lib64/libnvidia-container.so.1'"
time="2025-04-01T20:57:55Z" level=info msg="Skipping library candidate '/usr/lib64/libnvidia-container.so.1': error resolving link '/usr/lib64/libnvidia-container.so.1': lstat /usr/lib64/libnvidia-container.so.1: no such file or directory"
time="2025-04-01T20:57:55Z" level=info msg="Checking library candidate '/usr/lib/x86_64-linux-gnu/libnvidia-container.so.1'"
time="2025-04-01T20:57:55Z" level=info msg="Resolved link: '/usr/lib/x86_64-linux-gnu/libnvidia-container.so.1' => '/usr/lib/x86_64-linux-gnu/libnvidia-container.so.1.17.5'"
time="2025-04-01T20:57:55Z" level=info msg="Installing '/usr/lib/x86_64-linux-gnu/libnvidia-container.so.1.17.5' to '/usr/local/nvidia/toolkit/libnvidia-container.so.1.17.5'"
time="2025-04-01T20:57:55Z" level=info msg="Installed '/usr/lib/x86_64-linux-gnu/libnvidia-container.so.1.17.5' to '/usr/local/nvidia/toolkit/libnvidia-container.so.1.17.5'"
time="2025-04-01T20:57:55Z" level=info msg="Creating symlink '/usr/local/nvidia/toolkit/libnvidia-container.so.1' -> 'libnvidia-container.so.1.17.5'"
time="2025-04-01T20:57:55Z" level=info msg="Finding library libnvidia-container-go.so.1 (root=)"
time="2025-04-01T20:57:55Z" level=info msg="Checking library candidate '/usr/lib64/libnvidia-container-go.so.1'"
time="2025-04-01T20:57:55Z" level=info msg="Skipping library candidate '/usr/lib64/libnvidia-container-go.so.1': error resolving link '/usr/lib64/libnvidia-container-go.so.1': lstat /usr/lib64/libnvidia-container-go.so.1: no such file or directory"
time="2025-04-01T20:57:55Z" level=info msg="Checking library candidate '/usr/lib/x86_64-linux-gnu/libnvidia-container-go.so.1'"
time="2025-04-01T20:57:55Z" level=info msg="Resolved link: '/usr/lib/x86_64-linux-gnu/libnvidia-container-go.so.1' => '/usr/lib/x86_64-linux-gnu/libnvidia-container-go.so.1.17.5'"
time="2025-04-01T20:57:55Z" level=info msg="Installing '/usr/lib/x86_64-linux-gnu/libnvidia-container-go.so.1.17.5' to '/usr/local/nvidia/toolkit/libnvidia-container-go.so.1.17.5'"
time="2025-04-01T20:57:55Z" level=info msg="Installed '/usr/lib/x86_64-linux-gnu/libnvidia-container-go.so.1.17.5' to '/usr/local/nvidia/toolkit/libnvidia-container-go.so.1.17.5'"
time="2025-04-01T20:57:55Z" level=info msg="Creating symlink '/usr/local/nvidia/toolkit/libnvidia-container-go.so.1' -> 'libnvidia-container-go.so.1.17.5'"
time="2025-04-01T20:57:55Z" level=info msg="Installing executable '/usr/bin/nvidia-container-runtime' to /usr/local/nvidia/toolkit"
time="2025-04-01T20:57:55Z" level=info msg="Installing '/usr/bin/nvidia-container-runtime' to '/usr/local/nvidia/toolkit/nvidia-container-runtime.real'"
time="2025-04-01T20:57:55Z" level=info msg="Installed '/usr/local/nvidia/toolkit/nvidia-container-runtime.real'"
time="2025-04-01T20:57:55Z" level=info msg="Installed wrapper '/usr/local/nvidia/toolkit/nvidia-container-runtime'"
time="2025-04-01T20:57:55Z" level=info msg="Installing executable '/usr/bin/nvidia-container-runtime.cdi' to /usr/local/nvidia/toolkit"
time="2025-04-01T20:57:55Z" level=info msg="Installing '/usr/bin/nvidia-container-runtime.cdi' to '/usr/local/nvidia/toolkit/nvidia-container-runtime.cdi.real'"
time="2025-04-01T20:57:56Z" level=info msg="Installed '/usr/local/nvidia/toolkit/nvidia-container-runtime.cdi.real'"
time="2025-04-01T20:57:56Z" level=info msg="Installed wrapper '/usr/local/nvidia/toolkit/nvidia-container-runtime.cdi'"
time="2025-04-01T20:57:56Z" level=info msg="Installing executable '/usr/bin/nvidia-container-runtime.legacy' to /usr/local/nvidia/toolkit"
time="2025-04-01T20:57:56Z" level=info msg="Installing '/usr/bin/nvidia-container-runtime.legacy' to '/usr/local/nvidia/toolkit/nvidia-container-runtime.legacy.real'"
time="2025-04-01T20:57:56Z" level=info msg="Installed '/usr/local/nvidia/toolkit/nvidia-container-runtime.legacy.real'"
time="2025-04-01T20:57:56Z" level=info msg="Installed wrapper '/usr/local/nvidia/toolkit/nvidia-container-runtime.legacy'"
time="2025-04-01T20:57:56Z" level=info msg="Installing NVIDIA container CLI from '/usr/bin/nvidia-container-cli'"
time="2025-04-01T20:57:56Z" level=info msg="Installing executable '/usr/bin/nvidia-container-cli' to /usr/local/nvidia/toolkit"
time="2025-04-01T20:57:56Z" level=info msg="Installing '/usr/bin/nvidia-container-cli' to '/usr/local/nvidia/toolkit/nvidia-container-cli.real'"
time="2025-04-01T20:57:56Z" level=info msg="Installed '/usr/local/nvidia/toolkit/nvidia-container-cli.real'"
time="2025-04-01T20:57:56Z" level=info msg="Installed wrapper '/usr/local/nvidia/toolkit/nvidia-container-cli'"
time="2025-04-01T20:57:56Z" level=info msg="Installing NVIDIA container runtime hook from '/usr/bin/nvidia-container-runtime-hook'"
time="2025-04-01T20:57:56Z" level=info msg="Installing executable '/usr/bin/nvidia-container-runtime-hook' to /usr/local/nvidia/toolkit"
time="2025-04-01T20:57:56Z" level=info msg="Installing '/usr/bin/nvidia-container-runtime-hook' to '/usr/local/nvidia/toolkit/nvidia-container-runtime-hook.real'"
time="2025-04-01T20:57:56Z" level=info msg="Installed '/usr/local/nvidia/toolkit/nvidia-container-runtime-hook.real'"
time="2025-04-01T20:57:56Z" level=info msg="Installed wrapper '/usr/local/nvidia/toolkit/nvidia-container-runtime-hook'"
time="2025-04-01T20:57:56Z" level=info msg="Creating symlink '/usr/local/nvidia/toolkit/nvidia-container-toolkit' -> 'nvidia-container-runtime-hook'"
time="2025-04-01T20:57:56Z" level=info msg="Installing executable '/usr/bin/nvidia-ctk' to /usr/local/nvidia/toolkit"
time="2025-04-01T20:57:56Z" level=info msg="Installing '/usr/bin/nvidia-ctk' to '/usr/local/nvidia/toolkit/nvidia-ctk.real'"
time="2025-04-01T20:57:56Z" level=info msg="Installed '/usr/local/nvidia/toolkit/nvidia-ctk.real'"
time="2025-04-01T20:57:56Z" level=info msg="Installed wrapper '/usr/local/nvidia/toolkit/nvidia-ctk'"
time="2025-04-01T20:57:56Z" level=info msg="Installing executable '/usr/bin/nvidia-cdi-hook' to /usr/local/nvidia/toolkit"
time="2025-04-01T20:57:56Z" level=info msg="Installing '/usr/bin/nvidia-cdi-hook' to '/usr/local/nvidia/toolkit/nvidia-cdi-hook.real'"
time="2025-04-01T20:57:56Z" level=info msg="Installed '/usr/local/nvidia/toolkit/nvidia-cdi-hook.real'"
time="2025-04-01T20:57:56Z" level=info msg="Installed wrapper '/usr/local/nvidia/toolkit/nvidia-cdi-hook'"
time="2025-04-01T20:57:56Z" level=info msg="Installing NVIDIA container toolkit config '/usr/local/nvidia/toolkit/.config/nvidia-container-runtime/config.toml'"
time="2025-04-01T20:57:56Z" level=info msg="Skipping unset option: nvidia-container-runtime.debug"
time="2025-04-01T20:57:56Z" level=info msg="Skipping unset option: nvidia-container-runtime.log-level"
time="2025-04-01T20:57:56Z" level=info msg="Skipping unset option: nvidia-container-runtime.mode"
time="2025-04-01T20:57:56Z" level=info msg="Skipping unset option: nvidia-container-runtime.modes.cdi.annotation-prefixes"
time="2025-04-01T20:57:56Z" level=info msg="Skipping unset option: nvidia-container-runtime.runtimes"
time="2025-04-01T20:57:56Z" level=info msg="Skipping unset option: nvidia-container-cli.debug"
time="2025-04-01T20:57:56Z" level=info msg="Starting 'setup' for nvidia-toolkit"
Using config:
accept-nvidia-visible-devices-as-volume-mounts = false
accept-nvidia-visible-devices-envvar-when-unprivileged = true
disable-require = false
supported-driver-capabilities = "compat32,compute,display,graphics,ngx,utility,video"

[nvidia-container-cli]
  environment = []
  ldconfig = "@/sbin/ldconfig.real"
  load-kmods = true
  path = "/usr/local/nvidia/toolkit/nvidia-container-cli"
  root = "/"

[nvidia-container-runtime]
  log-level = "info"
  mode = "auto"
  runtimes = ["docker-runc", "runc", "crun"]

  [nvidia-container-runtime.modes]

    [nvidia-container-runtime.modes.cdi]
      annotation-prefixes = ["cdi.k8s.io/"]
      default-kind = "management.nvidia.com/gpu"
      spec-dirs = ["/etc/cdi", "/var/run/cdi"]

    [nvidia-container-runtime.modes.csv]
      mount-spec-path = "/etc/nvidia-container-runtime/host-files-for-container.d"

[nvidia-container-runtime-hook]
  path = "/usr/local/nvidia/toolkit/nvidia-container-runtime-hook"
  skip-mode-detection = true

[nvidia-ctk]
  path = "/usr/local/nvidia/toolkit/nvidia-ctk"
time="2025-04-01T20:57:56Z" level=info msg="Using config version 3"
time="2025-04-01T20:57:56Z" level=info msg="Using CRI runtime plugin name \"io.containerd.cri.v1.runtime\""
time="2025-04-01T20:57:56Z" level=info msg="Flushing config to /runtime/config-dir/config.toml"
time="2025-04-01T20:57:56Z" level=info msg="Sending SIGHUP signal to containerd"
time="2025-04-01T20:57:56Z" level=info msg="Successfully signaled containerd"
time="2025-04-01T20:57:56Z" level=info msg="Completed 'setup' for nvidia-toolkit"
time="2025-04-01T20:57:56Z" level=info msg="Waiting for signal"

[botterweck]

@ameijer, thanks for your suggestion

We set cdi.enabled: true and cdi.default: true in our chart values for the operator, and that resolved the issue.

I installed with

helm install --wait --generate-name \
	-n gpu-operator --create-namespace \
	nvidia/gpu-operator \
	--version=v25.3.0 \
	--set cdi.default=true

However, the effect seems to be the same, that is

1. the pod nvidia-operator-validator-XYZ goes into status Error/CrashLoopBackOff
2. when looking at "describe pod" for that pod, the Event section shows "Failed to create pod sandbox: rpc error: code = Unknown desc = unable to get OCI runtime for sandbox "...": no runtime for "nvidia" is configured
   (very similar to the outputs posted in my previous messages)

[RayLin9981]

Hi everyone, I was also stuck on this problem for about two days, and here provided me with the following environment:

• ubuntu24.04, also use kubespray to install k8s 1.32.2
• GPU Worker is dgx-1, with 8 V100 GPUs
• GPU Worker Driver Installation and Container Toolkit Manual

This problem seems to have always existed and there is no appropriate solution. I wonder if anyone can discuss a good solution?

p.s I really want to skip the nvidia-operator-validator process, but I tried some parameters of clusterpolicies.nvidia.com and nothing worked...

---

some information：

kubectl get no -o wide
NAME     STATUS   ROLES           AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
dgx-m1   Ready    control-plane   2d4h    v1.32.2   192.168.147.161   <none>        Ubuntu 24.04.2 LTS   6.8.0-57-generic   containerd://2.0.3
dgx-w1   Ready    <none>          6h29m   v1.32.2   192.168.147.171   <none>        Ubuntu 24.04.2 LTS   6.8.0-57-generic   containerd://2.0.3
dgx1     Ready    <none>          2d1h    v1.32.2   192.168.147.152   <none>        Ubuntu 24.04.1 LTS   6.8.0-57-generic   containerd://2.0.3

dgx1 is GPU node

root@dgx1:~# nvidia-smi -L
GPU 0: Tesla V100-SXM2-32GB (UUID: GPU-1f7e3ae1-7a26-8dd4-8ae9-fb6b9ebab4ae)
GPU 1: Tesla V100-SXM2-32GB (UUID: GPU-bec53afd-8230-2c6a-dbcb-a8088a9b0e88)
GPU 2: Tesla V100-SXM2-32GB (UUID: GPU-f85b169d-7443-05ff-a835-ecbe75205c6c)
GPU 3: Tesla V100-SXM2-32GB (UUID: GPU-eb392dda-8b08-9347-f66f-60f5b6f6bb79)
GPU 4: Tesla V100-SXM2-32GB (UUID: GPU-27441946-5f74-aa7b-4568-8e0fdd151ca9)
GPU 5: Tesla V100-SXM2-32GB (UUID: GPU-bd6173f8-2292-642c-fe6f-d0cdd5a66bb1)
GPU 6: Tesla V100-SXM2-32GB (UUID: GPU-a9a189a1-db9a-6d12-da89-60ab050b6462)
GPU 7: Tesla V100-SXM2-32GB (UUID: GPU-33d6db07-8a24-ca51-78ed-db8d7b342baa)
root@dgx1:~# nvidia-ctk --version
NVIDIA Container Toolkit CLI version 1.17.5
commit: f785e908a7f72149f8912617058644fd84e38cde

helm ：

helm upgrade --install --wait gpu-operator -n gpu-operator --create-namespace nvidia/gpu-operator \
     --version=v25.3.0 \
     --values=values.yaml

values.yaml:

platform:
  openshift: false

nfd:
  enabled: true
  nodefeaturerules: false

psa:
  enabled: false

cdi:
  enabled: false
  default: false

sandboxWorkloads:
  enabled: false
  defaultWorkload: "container"

hostPaths:
  rootFS: "/"

  driverInstallDir: "/run/nvidia/driver"

daemonsets:
  labels: {}
  annotations: {}
  priorityClassName: system-node-critical
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
  updateStrategy: "RollingUpdate"
  rollingUpdate:
    maxUnavailable: "1"

validator:
  repository: nvcr.io/nvidia/cloud-native
  image: gpu-operator-validator
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  env: []
  args: []
  resources: {}
  plugin:
    env:
      - name: WITH_WORKLOAD
        value: "false"

operator:
  defaultRuntime: containerd
  repository: nvcr.io/nvidia
  image: gpu-operator
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  priorityClassName: system-node-critical
  runtimeClass: nvidia
  use_ocp_driver_toolkit: false
  cleanupCRD: false
  upgradeCRD: true
  initContainer:
    image: cuda
    repository: nvcr.io/nvidia
    version: 12.8.1-base-ubi9
    imagePullPolicy: IfNotPresent
  tolerations:
  - key: "node-role.kubernetes.io/master"
    operator: "Equal"
    value: ""
    effect: "NoSchedule"
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Equal"
    value: ""
    effect: "NoSchedule"
  annotations:
    openshift.io/scc: restricted-readonly
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 1
          preference:
            matchExpressions:
              - key: "node-role.kubernetes.io/master"
                operator: In
                values: [""]
        - weight: 1
          preference:
            matchExpressions:
              - key: "node-role.kubernetes.io/control-plane"
                operator: In
                values: [""]
  logging:
    timeEncoding: epoch
    level: info
    develMode: false
  resources:
    limits:
      cpu: 500m
      memory: 350Mi
    requests:
      cpu: 200m
      memory: 100Mi

mig:
  strategy: single

driver:
  enabled: false
  nvidiaDriverCRD:
    enabled: false
    deployDefaultCR: true
    driverType: gpu
    nodeSelector: {}
  kernelModuleType: "auto"

  usePrecompiled: false
  repository: nvcr.io/nvidia
  image: driver
  version: "570.124.06"
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  startupProbe:
    initialDelaySeconds: 60
    periodSeconds: 10
    timeoutSeconds: 60
    failureThreshold: 120
  rdma:
    enabled: false
    useHostMofed: false
  upgradePolicy:
    autoUpgrade: true
    maxParallelUpgrades: 1
    maxUnavailable: 25%
    waitForCompletion:
      timeoutSeconds: 0
      podSelector: ""
    gpuPodDeletion:
      force: false
      timeoutSeconds: 300
      deleteEmptyDir: false
    drain:
      enable: false
      force: false
      podSelector: ""
      timeoutSeconds: 300
      deleteEmptyDir: false
  manager:
    image: k8s-driver-manager
    repository: nvcr.io/nvidia/cloud-native
    version: v0.8.0
    imagePullPolicy: IfNotPresent
    env:
      - name: ENABLE_GPU_POD_EVICTION
        value: "true"
      - name: ENABLE_AUTO_DRAIN
        value: "false"
      - name: DRAIN_USE_FORCE
        value: "false"
      - name: DRAIN_POD_SELECTOR_LABEL
        value: ""
      - name: DRAIN_TIMEOUT_SECONDS
        value: "0s"
      - name: DRAIN_DELETE_EMPTYDIR_DATA
        value: "false"
  env: []
  resources: {}
  repoConfig:
    configMapName: ""
  certConfig:
    name: ""
  licensingConfig:
    configMapName: ""
    nlsEnabled: true
  virtualTopology:
    config: ""
  kernelModuleConfig:
    name: ""

toolkit:
  enabled: false
  repository: nvcr.io/nvidia/k8s
  image: container-toolkit
  version: v1.17.5-ubuntu20.04
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  env:
  - name: CONTAINERD_CONFIG
    value: /etc/containerd/config.toml
  - name: CONTAINERD_SOCKET
    value: /run/containerd/containerd.sock
  - name: CONTAINERD_RUNTIME_CLASS
    value: nvidia
  - name: CONTAINERD_SET_AS_DEFAULT
    value: "true"
  resources: {}
  installDir: "/usr/local/nvidia"

devicePlugin:
  enabled: false
  repository: nvcr.io/nvidia
  image: k8s-device-plugin
  version: v0.17.1
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  args: []
  env:
    - name: PASS_DEVICE_SPECS
      value: "true"
    - name: FAIL_ON_INIT_ERROR
      value: "true"
    - name: DEVICE_LIST_STRATEGY
      value: envvar
    - name: DEVICE_ID_STRATEGY
      value: uuid
    - name: NVIDIA_VISIBLE_DEVICES
      value: all
    - name: NVIDIA_DRIVER_CAPABILITIES
      value: all
  resources: {}
  config:
    create: false
    name: ""
    default: ""
    data: {}
  mps:
    root: "/run/nvidia/mps"

dcgm:
  enabled: false
  repository: nvcr.io/nvidia/cloud-native
  image: dcgm
  version: 4.1.1-2-ubuntu22.04
  imagePullPolicy: IfNotPresent
  args: []
  env: []
  resources: {}

dcgmExporter:
  enabled: true
  repository: nvcr.io/nvidia/k8s
  image: dcgm-exporter
  version: 4.1.1-4.0.4-ubuntu22.04
  imagePullPolicy: IfNotPresent
  env:
    - name: DCGM_EXPORTER_LISTEN
      value: ":9400"
    - name: DCGM_EXPORTER_KUBERNETES
      value: "true"
    - name: DCGM_EXPORTER_COLLECTORS
      value: "/etc/dcgm-exporter/dcp-metrics-included.csv"
  resources: {}
  serviceMonitor:
    enabled: false
    interval: 15s
    honorLabels: false
    additionalLabels: {}
    relabelings: []

gfd:
  enabled: true
  repository: nvcr.io/nvidia
  image: k8s-device-plugin
  version: v0.17.1
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  env:
    - name: GFD_SLEEP_INTERVAL
      value: 60s
    - name: GFD_FAIL_ON_INIT_ERROR
      value: "true"
  resources: {}

migManager:
  enabled: true
  repository: nvcr.io/nvidia/cloud-native
  image: k8s-mig-manager
  version: v0.12.1-ubuntu20.04
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  env:
    - name: WITH_REBOOT
      value: "false"
  resources: {}
  config:
    default: "all-disabled"
    create: false
    name: ""
    data: {}
  gpuClientsConfig:
    name: ""

nodeStatusExporter:
  enabled: false
  repository: nvcr.io/nvidia/cloud-native
  image: gpu-operator-validator
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  resources: {}

gds:
  enabled: false
  repository: nvcr.io/nvidia/cloud-native
  image: nvidia-fs
  version: "2.20.5"
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  env: []
  args: []

gdrcopy:
  enabled: false
  repository: nvcr.io/nvidia/cloud-native
  image: gdrdrv
  version: "v2.4.4"
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  env: []
  args: []

vgpuManager:
  enabled: false
  repository: ""
  image: vgpu-manager
  version: ""
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  env: []
  resources: {}
  driverManager:
    image: k8s-driver-manager
    repository: nvcr.io/nvidia/cloud-native
    version: v0.8.0
    imagePullPolicy: IfNotPresent
    env:
      - name: ENABLE_GPU_POD_EVICTION
        value: "false"
      - name: ENABLE_AUTO_DRAIN
        value: "false"

vgpuDeviceManager:
  enabled: true
  repository: nvcr.io/nvidia/cloud-native
  image: vgpu-device-manager
  version: v0.3.0
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  env: []
  config:
    name: ""
    default: "default"

vfioManager:
  enabled: true
  repository: nvcr.io/nvidia
  image: cuda
  version: 12.8.1-base-ubi9
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  env: []
  resources: {}
  driverManager:
    image: k8s-driver-manager
    repository: nvcr.io/nvidia/cloud-native
    version: v0.8.0
    imagePullPolicy: IfNotPresent
    env:
      - name: ENABLE_GPU_POD_EVICTION
        value: "false"
      - name: ENABLE_AUTO_DRAIN
        value: "false"

kataManager:
  enabled: false
  config:
    artifactsDir: "/opt/nvidia-gpu-operator/artifacts/runtimeclasses"
    runtimeClasses:
      - name: kata-nvidia-gpu
        nodeSelector: {}
        artifacts:
          url: nvcr.io/nvidia/cloud-native/kata-gpu-artifacts:ubuntu22.04-535.54.03
          pullSecret: ""
      - name: kata-nvidia-gpu-snp
        nodeSelector:
          "nvidia.com/cc.capable": "true"
        artifacts:
          url: nvcr.io/nvidia/cloud-native/kata-gpu-artifacts:ubuntu22.04-535.86.10-snp
          pullSecret: ""
  repository: nvcr.io/nvidia/cloud-native
  image: k8s-kata-manager
  version: v0.2.3
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  env: []
  resources: {}

sandboxDevicePlugin:
  enabled: true
  repository: nvcr.io/nvidia
  image: kubevirt-gpu-device-plugin
  version: v1.3.1
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  args: []
  env: []
  resources: {}

ccManager:
  enabled: false
  defaultMode: "off"
  repository: nvcr.io/nvidia/cloud-native
  image: k8s-cc-manager
  version: v0.1.1
  imagePullPolicy: IfNotPresent
  imagePullSecrets: []
  env:
    - name: CC_CAPABLE_DEVICE_IDS
      value: "0x2339,0x2331,0x2330,0x2324,0x2322,0x233d"
  resources: {}

node-feature-discovery:
  enableNodeFeatureApi: true
  priorityClassName: system-node-critical
  gc:
    enable: true
    replicaCount: 1
    serviceAccount:
      name: node-feature-discovery
      create: false
  worker:
    serviceAccount:
      name: node-feature-discovery
      create: false
    tolerations:
    - key: "node-role.kubernetes.io/master"
      operator: "Equal"
      value: ""
      effect: "NoSchedule"
    - key: "node-role.kubernetes.io/control-plane"
      operator: "Equal"
      value: ""
      effect: "NoSchedule"
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
    config:
      sources:
        pci:
          deviceClassWhitelist:
          - "02"
          - "0200"
          - "0207"
          - "0300"
          - "0302"
          deviceLabelFields:
          - vendor
  master:
    serviceAccount:
      name: node-feature-discovery
      create: true
    config:
      extraLabelNs: ["nvidia.com"]

[RayLin9981]

Hi, just a quick update:
https://hackmd.io/MSvd7S3iSYGuD523VM7d5A?view

I downgraded my DGX server from containerd 2.0.3 to 1.7.21, and now the GPU operator works without any issues.
Why is that?

[flpanbin]

I have the same problem. gpu-operator v25.3.0 install failed with containerd v2.0.3, It can be install successed with containerd v1.7.13.

[TaeWoo21]

I experienced the same issue. As @RayLin9981 and @flpanbin already mentioned, gpu-operator v25.3.0 fails to install with containerd v2.0.3.

In my case, I initially attempted to set up the kubernetes cluster using the kubespray master branch, which resulted in containerd v2.0.3 being used as the container runtime. That seems to have been a mistake.

I should have used the latest stable kubespray release version (e.g., release-2.27), which uses containerd v1.7.26 as the container runtime.

[lengrongfu]

I use containerd 2.0.4 version to test is working, but need manual upgrade containerd.

[RayLin9981]

I use containerd 2.0.4 version to test is working, but need manual upgrade containerd.

Hi, just curious, you upgrade to which containerd version?

[lengrongfu]

I use containerd 2.0.4 version to test is working, but need manual upgrade containerd.

Hi, just curious, you upgrade to which containerd version?

containerd 2.0.4 version. https://github.com/containerd/containerd/releases/tag/v2.0.4

[RayLin9981]

I use containerd 2.0.4 version to test is working, but need manual upgrade containerd.

Hi, just curious, you upgrade to which containerd version?

containerd 2.0.4 version.

woo, so you mean 2.0.3 not work but 2.0.4 is ok?
or you have do something about confg containerd.toml?

[flpanbin]

For containerd 2.0.3, I found that gpu operator could be installed successfully after removing the binaryName config in config.toml. Maybe you can check whether the configuration exists and try to remove it. @botterweck @RayLin9981 @TaeWoo21

[RayLin9981]

I'm checking @flpanbin provide solution, my env:

[root@rocky ~]# kubectl get no -o wide
NAME    STATUS   ROLES           AGE     VERSION   INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                      KERNEL-VERSION                 CONTAINER-RUNTIME
rocky   Ready    control-plane   3m38s   v1.32.2   192.168.147.125   <none>        Rocky Linux 9.5 (Blue Onyx)   5.14.0-503.38.1.el9_5.x86_64   containerd://2.0.3

and for easy to look, not use values.yaml:

helm upgrade --install --wait gpu-operator -n gpu-operator --create-namespace nvidia/gpu-operator \
     --version=v25.3.0 \
     --set driver.enabled=false

check /etc/config/containerd.toml, and comment binaryName :

and passed!

Thank you very much for your solution!! @flpanbin

p.s: I check containerd 1.7.x config.toml, binaryName is empty originally:

but there have another binaryName beside Nvidia config: