orchestrator/nfsproxy: clean up .nfs* orphan files on sandbox teardown - #3533
Open
AdaAibaby wants to merge 1 commit into
Open
orchestrator/nfsproxy: clean up .nfs* orphan files on sandbox teardown#3533AdaAibaby wants to merge 1 commit into
AdaAibaby wants to merge 1 commit into
Conversation
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
August 4, 2026 11:54
AdaAibaby
force-pushed
the
fix/nfs-orphan-cleanup
branch
from
August 4, 2026 12:08
29936c6 to
ea46cd9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3532.
Problem
When a sandbox VM is forcibly killed while holding open file descriptors on files
inside a persistent volume, the NFS silly-rename mechanism leaves
.nfs*fileson the NFS server that are never cleaned up, leaking storage indefinitely.
Observed on production: a 5 GiB workspace image from Jul 29 remained as
.nfs000000019000ee9800000001— confirmed orphan (no process holds it, confirmedvia
/proc/*/fd), yet persisted because the final NFSREMOVERPC was never sent.Root Cause
NFS silly-rename: when an NFS client deletes a file that is still open locally,
the client kernel sends
RENAME ws_<id>.img → .nfsXXXXto the server instead ofREMOVE, deferring the delete until the last fd is closed. If the VM is killedbefore that point, the
REMOVEnever arrives and the.nfsXXXXfile persistspermanently on the server (same size as the original file, typically several GiB).
OnNetworkReleasepreviously only closed the orchestrator-side chroot with nocleanup of leftover
.nfs*files in the volume directory.Fix
Add
removeNFSOrphanscalled fromOnNetworkReleasebefore closing each chroot.It scans
chroot.Root()(the actual host-side volume directory) for.nfs*filesand removes them. By the time
OnNetworkReleasefires the sandbox VM is alreadydead — all its fds are gone — so orphan files are safe to remove.
Files Changed
packages/orchestrator/pkg/nfsproxy/chroot/nfs.go/cc @jakubno @dobrac @ValentaTomas @arkamar @tvi @tomassrnka Looking forward to your code review.