firecracker: serve fork mem-file via per-template uffd page server#219
Draft
sjmiller609 wants to merge 4 commits intohypeship/uffd-prefetch-hotpagesfrom
Draft
firecracker: serve fork mem-file via per-template uffd page server#219sjmiller609 wants to merge 4 commits intohypeship/uffd-prefetch-hotpagesfrom
sjmiller609 wants to merge 4 commits intohypeship/uffd-prefetch-hotpagesfrom
Conversation
Wires firecracker's snapshot restore through a userfaultfd-backed page server when the fork descends from a template. Each template gets one uffd.Server lazily started on the first fork and torn down once the last fork is released, so non-template forks (the symlink-only path) are unaffected. Adds an E2E test that boots a firecracker source, standbys it, promotes it to a template, forks against the template, and asserts: fork reaches Running, mem-file is a symlink to the template's, refcount bumps to 1, the per-template uffd tracker registers the fork, and on fork delete the refcount drops and the tracker detaches. Also adds unit coverage for the uffd tracker lifecycle (lazy start, multi-fork share, last-release teardown, closeAll, empty-acquire rejection).
Unix domain socket paths cap at 108 bytes (sun_path). Putting the per- fork socket under <DataDir>/templates/<25-char-cuid>/uffd/<25-char- cuid>.uffd blew that limit on CI runners where t.TempDir() returns long prefixes, surfacing as "bind: invalid argument" in TestFirecrackerForkFromTemplate. Sockets are ephemeral, so anchoring them at /tmp/h-uffd/<templateID>/ keeps the path well under the limit regardless of how deep DataDir is. The tracker now also rm -rfs the per-template socket dir on the last release so we don't leak stale entries.
The previous fork-mem-file symlink looped through withSnapshotSourceDirAlias during firecracker restore: fork/.../memory -> source/.../memory, but the alias dance temporarily symlinks source dir -> fork dir, so resolution ping-pongs back to fork/.../memory and trips ELOOP. Switching to a hardlink makes the fork's mem-file resolve by inode so the temporary directory alias no longer affects it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Firecracker enables snapshot base reuse, which renames the post-restore snapshot dir from snapshot-latest to snapshot-base. The hardlink survives the rename (same inode), so the test just needs the right path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
uffd.Serverinstead of mmaping the mem-file directly.uffd.Serverper template on the first fork acquire and tears it down once the last fork is released. Non-template forks (the symlink-only path from PR fork: share template mem-file via symlink for fan-out forks #214) are unaffected.UffdSocketPaththroughForkPrepareRequest→ firecrackerrestoreMetadata→snapshotLoadParams.MemBackend({ backend_type: "Uffd", backend_path: <UDS> }), withmem_file_pathcleared when a backend is set.DeleteInstancefor forks (best-effort warn on error).Test plan
go vet ./...cleango build ./...cleango test ./lib/instances/... -run TestUffdTracker -count=1TestFirecrackerForkFromTemplateboots a firecracker source, standbys → promotes → forks → asserts:ForkCountbumps to 1ForkCountand detaches from uffdconfig_test.goupdated for uffd backend assertionsStacked on #218.
🤖 Generated with Claude Code