Skip to content

orchestrator/nfsproxy: clean up .nfs* orphan files on sandbox teardown - #3533

Open
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/nfs-orphan-cleanup
Open

orchestrator/nfsproxy: clean up .nfs* orphan files on sandbox teardown#3533
AdaAibaby wants to merge 1 commit into
e2b-dev:mainfrom
AdaAibaby:fix/nfs-orphan-cleanup

Conversation

@AdaAibaby

@AdaAibaby AdaAibaby commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #3532.

Problem

When a sandbox VM is forcibly killed while holding open file descriptors on files
inside a persistent volume, the NFS silly-rename mechanism leaves .nfs* files
on the NFS server that are never cleaned up, leaking storage indefinitely.

Observed on production: a 5 GiB workspace image from Jul 29 remained as
.nfs000000019000ee9800000001 — confirmed orphan (no process holds it, confirmed
via /proc/*/fd), yet persisted because the final NFS REMOVE RPC was never sent.

Root Cause

NFS silly-rename: when an NFS client deletes a file that is still open locally,
the client kernel sends RENAME ws_<id>.img → .nfsXXXX to the server instead of
REMOVE, deferring the delete until the last fd is closed. If the VM is killed
before that point, the REMOVE never arrives and the .nfsXXXX file persists
permanently on the server (same size as the original file, typically several GiB).

OnNetworkRelease previously only closed the orchestrator-side chroot with no
cleanup of leftover .nfs* files in the volume directory.

Fix

Add removeNFSOrphans called from OnNetworkRelease before closing each chroot.
It scans chroot.Root() (the actual host-side volume directory) for .nfs* files
and removes them. By the time OnNetworkRelease fires the sandbox VM is already
dead — all its fds are gone — so orphan files are safe to remove.

Files Changed

  • packages/orchestrator/pkg/nfsproxy/chroot/nfs.go

/cc @jakubno @dobrac @ValentaTomas @arkamar @tvi @tomassrnka Looking forward to your code review.

@AdaAibaby
AdaAibaby force-pushed the fix/nfs-orphan-cleanup branch from 29936c6 to ea46cd9 Compare August 4, 2026 12:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

persistent volume: .nfs* orphan files leak storage when sandbox VMs are forcibly killed

2 participants