CreateContainerError: context deadline exceeded unpacking an OCI tarball

[jarrillaga]

Hey community :)

I’m running Kairos in an air-gapped environment and use a run.sh script in the install stage to copy a OCI image tarball (~310 MB) into /var/lib/rancher/k3s/agent/images/omnibus-init.tar. Although ctr images ls confirms the image is present, containerd repeatedly times out while unpacking and mounting its many overlayfs layers, hitting the default CRI deadline and failing with:

CreateContainerError: context deadline exceeded

Logs & Observations

Kubelet / K3s

CreateContainer in sandbox from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Normal  Pulled   Container image "xxxx/omnibus-init" already present on machine
Warning Failed   Error: context deadline exceeded

msg="Failed to unpack image layer" error="failed to extract layer sha256:xxxx...: no such file or directory"

containerd snapshots: ~100 active overlayfs layers

dmesg shows no overlayfs mount errors

iostat / top reveal high I/O and CPU load during boot

I can confirm that the tarball is well formed because I was able to run it manually from command line.

Problem
The init-job container never starts because containerd can’t finish unpacking and mounting all the OCI image layers before Kubernetes’s default timeout.

Question
Is there a built-in Kairos or K3s feature to pre-unpack or pre-mount large OCI image tarballs during the install phase so that init-job containers can start reliably on first boot? Any best practices or recommended workflow for handling large, air-gapped images would be greatly appreciated!

[jimmykarily]

Googling and ChatGPTing (new word? :D) says this might help:

ctr image import /var/lib/rancher/k3s/agent/images/my-large-image.tar

Worths a try. If it works, I guess you can run it in a stage in the kairos config.