Workload container exit is invisible: Pod stays Running and the instance leaks

[ky-hiverge]

What happened

A workload whose process exits leaves its Pod reported Running forever and its EC2 instance running forever. Nothing observes container state, so Nebula cannot tell a finished workload from a healthy one.

Observed with 30 replicas of config/samples gpu-workload-sample, whose command is:

sh -c "nvidia-smi --query-gpu=... --format=csv || echo 'no nvidia-smi'; sleep 3600"

Timeline from the cluster (all 30 replicas, one hour after start):

Time	Event
13:27:25	~30 SandD daemons authenticate to headscale
13:44–13:46	controller restarts; all 30 re-authorize
14:27:24–14:27:40	all 30 mesh nodes node has disconnected — exactly 3600s after 13:27:25
14:29:13	SandD controller reaps the stale entries → Active daemons: 0
after	zero further auth attempts; no daemon returns

Meanwhile kubectl get pods still showed all 30 as 1/1 Running, and all 30 instances stayed running in EC2 — 2+ hours after their only process had exited.

Why

buildUserData emits docker run --rm ... as the last line of user-data (pkg/provider/aws/translate.go:78, :105-108) and the script then ends. When the container exits, nothing runs. The instance idles as running indefinitely.

toState (pkg/provider/aws/aws.go:812) derives instance state from the EC2 instance state alone, so a dead container reads as InstanceRunning, and the vnode poll loop keeps synthesizing a Running Pod status from it.

Two consequences:

1. Wrong status — a completed or crashed workload is indistinguishable from a live one. A crash-looping or instantly-failing container looks perfectly healthy.
2. Leaked instances (the expensive half) — the instance is never reclaimed. 30 idle GPU instances kept billing here.

Not solvable by readiness probes, and not a SandD problem

• Probes don't run. There is no kubelet on a Nebula virtual node; VK synthesizes the entire Pod status from what the provider reports. A readinessProbe on the Pod is ignored data.
• SandD can't report it. The shim (translate.go:178-192) backgrounds sandd, then exec "$@" makes the workload PID 1. sandd is a child of the workload, so it dies in the same instant the container is torn down — it cannot be the thing that reports the exit. (That parent/child direction is correct: the daemon should not outlive the workload it is attached to.)

The gap is host-side, in user-data.

Suggested fix

Have the instance stop itself once docker run returns. This reuses machinery that already exists:

• toState already maps stopped/stopping/terminated → InstanceTerminated
• the poll loop already drives a terminated instance to a terminal Pod
• the owning Deployment then replaces the Pod → a fresh NodeClaim → a fresh instance

That is the delete-and-recreate model the NodeClaim ledger already documents, so no new signalling channel and no container-state observer is needed.

Open design question

The simple version makes any container exit terminal, including exit 0. A workload that legitimately completes would churn a replacement instance instead of settling. Distinguishing the two requires capturing the container's exit code host-side and giving Nebula somewhere to carry it — a larger change. Worth deciding before implementing.

Related

Separately, the SandD controller under-counted live daemons (25 of 30) because its reaper could evict a daemon whose socket was still open. That is a different bug with a fix in flight in the SandD repo (HeartbeatAck → re-register in place); it does not address this issue.

[googs1025]

I think the simple host-side fix makes sense here.

For AWS, maybe we can make this more concrete like:

1. In buildUserData, do not leave docker run --rm ... as the last command.
2. Capture the workload container exit, then call shutdown -h now from the host after docker run returns.
3. In the EC2 launch template, set instance initiated shutdown behavior to terminate, so the shutdown does not only stop the instance.

Then the existing flow can still be reused:

• EC2 goes to shutting-down / terminated
• toState maps that to InstanceTerminated
• vnode poll updates the Pod to terminal status
• Deployment can recreate a new Pod / NodeClaim if it wants another replica

This matches the suggested fix in the issue as I understand it. The main tradeoff is also the same open question: exit 0 will be terminal too, so a completed workload under a Deployment may be recreated. I think that is OK for a first fix, and carrying the container exit code back to Nebula can be a later, larger change.

[kerthcet]

If we can use something like supervisor to restart the process, it would be great since scheduling is a non-trivial but if we can not, capture the exit code is also good to me.

Would you like to pick this @googs1025

[googs1025]

Thanks, that makes sense to me. I can pick this.

I will first look into a host-side supervisor/systemd way, so for normal Deployment / restartPolicy=Always, the container can restart on the same EC2 instance instead of doing a full reschedule.

My rough plan is:

• let user-data write a small systemd unit or wrapper for docker run
• use restart behavior for Always
• for no-restart cases, still let the instance shutdown/terminate
• keep exit-code reporting as a follow-up if it needs more status plumbing

I will try to keep the first PR AWS-only and small, with tests for user-data / launch template behavior.