[CHORE] k3s snapshots lifecycle

[Flegma]

As someone on Discord asked, we should register a cronjob to clean snapshots in the /var/lib/rancher/k3s/agent/ directory.

Maybe we can do a cronjob that would clean them on monthly basis for start, and if needed at later point, we can add some .env variable that would allow people to configure how many days they want to keep it.

[lukepolo]

k3s crictl rmi --prune. only issue is it will remove game-server images / game-streamer images too since they wont be running probably

[Flegma]

@lukepolo here's how I'm thinking we tackle this.

First, a quick clarification on what's actually filling up: it's not etcd snapshots, it's containerd's overlayfs image snapshots under /var/lib/rancher/k3s/agent/containerd/... - i.e. the layers behind old ghcr.io/5stackgg/* images, the same stuff fix-versions.sh clears by hand today. I checked one of our nodes (k3s v1.34.4+k3s1) and even a nearly-idle one is already sitting on ~1.6 GB of snapshots; on a node that's churned a lot of image versions this just keeps climbing until the disk fills and the kubelet starts evicting.

One thing to be careful about: we should not rm files out of that directory directly - containerd tracks every snapshot in its own metadata DB, so deleting them underneath it corrupts state and breaks image pulls. The safe path is crictl rmi / kubelet GC, which drop the image and its snapshot together.

So rather than a single cronjob, I'd split it into two layers:

1. A systemd timer running k3s crictl rmi --prune (the main cleanup). --prune removes all unused images (confirmed on the node), and a systemd timer keeps it consistent with the pattern we already use on nodes - same shape as the 5stack-tailscale-state-check.timer we write in game-node-server-setup.sh. Start it weekly/monthly. Works on every node since it only needs the local containerd socket.

2. Kubelet image GC as the native safety net. We already set kubelet-arg in /etc/rancher/k3s/config.yaml, so it's a clean place to add GC thresholds and let the kubelet auto-reclaim under disk pressure between timer runs. And for the "keep N days, configurable" idea from the issue, the kubelet's imageMaximumGCAge is exactly that knob (time-based, unused images only) - we're on 1.34 so it's available; since it's a KubeletConfiguration field we'd wire it through a kubelet config drop-in (exact form is a PR detail).

I'd skip a Kubernetes CronJob for this - it runs as a single pod on one node, but this is per-host cleanup (one pod can't reach every node's containerd), so a per-node timer plus the kubelet itself is the right layer.

A few rollout notes:

• Should apply to both the panel host (install.sh) and game nodes (game-node-server-setup.sh) - both run containerd.
• The host-level units are only written during initial setup right now, so we'll want to fold the unit-writing into the update path too, otherwise existing nodes won't pick it up.
• Make the retention/interval env-configurable (e.g. IMAGE_GC_MAX_AGE_DAYS, default 30) the same way the postgres backup retention works.

If you're good with this direction I'll put up a PR. Happy to keep it to just the timer for a first pass if you'd rather start minimal.