How to ensure that the nvidia-container-toolkit start before anyone else?

[Wennn]

Once a node is restarted, the absence of a guaranteed sequence of pod restarts may result in pods that were started before the nvidia-container-toolkit cannot access the GPU devices, including device plugin.

Could you please provide some guidance on how to handle this situation? Is there any config from gpu-operator side to ensure nvidia-container-toolkit gets started at first?

Thanks.

[github-actions]

This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.

[shivamerla]

From the GPU Operator side, the expectation is eventual consistency. Once the NVIDIA runtime/toolkit is configured on the node, the device plugin and GPU workloads will converge to healthy state. Kubernetes does not provide a guaranteed pod restart sequence after reboot, so ordering can’t be strictly enforced via a GPU Operator config.