fix(pivot): give the new rootfs its own tmpfs, sized from real RAM - #16
Merged
Conversation
The rootfs was extracted straight into --work-dir, which defaults to
/run/xmorph/rootfs. /run is itself a size-capped tmpfs — 10-20% of RAM on
a systemd host — so the new rootfs silently inherited a cap that has
nothing to do with how much memory is actually free.
Reproduced on a 415 MiB Raspberry Pi 3A+, where /run is 83 MiB:
$ xmorph build --image docker.io/library/debian:trixie
Error: build rootfs: layer 0: extract layer 0:
write /run/xmorph/demo/usr/share/info/find.info.gz: no space left on device
$ df -h /run
tmpfs 84M 84M 0 100% /run
Two things wrong with that. The error names a random file rather than the
cause, so it reads like a corrupt layer. And it fills /run to 100% on a
host that is still running — /run is where systemd, dbus and friends keep
their sockets, so a failed build degrades the OS we were trying not to
break.
The check that should have caught it was never called. internal/sysmem
was written and unit-tested, but the only reference to it outside its own
package was a comment in runPivot saying it would be wired up "when M5's
RAM telemetry is surfaced". It also would not have helped: it compares
the rootfs against MemAvailable (246 MiB on that host), not against the
tmpfs cap (83 MiB), so it would have approved the build that then failed.
So:
* Mount a real tmpfs at --work-dir, sized explicitly. Default is
MemAvailable minus a 10% reserve; --rootfs-size takes 512M/2G/50%, and
--no-rootfs-tmpfs keeps the old behaviour for operators who have
already arranged the storage. On the Pi above the default is ~154 MiB
instead of 83, and the budget is stated in the log either way.
* Actually call HeadroomCheck, against the size we are about to mount, and
refuse while the old OS is still alive rather than OOMing mid-extract.
Sizing from MemAvailable rather than MemTotal is deliberate: the old OS
still holds its memory, and an OOM kill is worse than an ENOSPC because
the kernel picks the victim — on a small host as likely sshd as xmorph.
* Report the tmpfs budget in the build-rootfs error, since ENOSPC there is
now a sizing decision the operator can act on.
Two follow-on fixes this exposed:
* os.RemoveAll(WorkDir) fails with EBUSY once WorkDir is a mount point.
Replaced with rootfs.CleanTarget, which empties the directory instead of
unlinking it — the operation the callers actually wanted, since they
want an empty target and not a deleted one. A reused mount is now
cleaned too, so an aborted run cannot layer two rootfses together.
* Prepare's self-bind is skipped when the new root is already a mount.
It exists only to satisfy pivot_root's "new_root must be a mount point"
rule, and stacking a second mount to re-satisfy a rule that already
holds just adds a mount to unwind.
Verified on the same Pi: extraction into a correctly sized tmpfs
completes, and /run stays at 4%.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while trying to pivot a Raspberry Pi 3A+ (415 MiB) into a NixOS installer. The pivot never got as far as
pivot_root.The bug
The rootfs is extracted straight into
--work-dir, which defaults to/run/xmorph/rootfs./runis itself a size-capped tmpfs — 10-20% of RAM on a systemd host — so the new rootfs silently inherits a cap that has nothing to do with how much memory is free.On that Pi,
/runis 83 MiB:Two things wrong beyond the failure itself:
/runto 100% on a host that is still running./runis where systemd, dbus and friends keep their sockets, so a failed build degrades the OS we were specifically trying not to break.Why the existing check didn't catch it
internal/sysmemis written and unit-tested, but the only reference to it outside its own package was a comment inrunPivot:It also would not have helped if it had run: it compares the rootfs against
MemAvailable(246 MiB on that host), not against the tmpfs cap (83 MiB). It would have approved the build that then failed.That's the shape of the bug — the thing being measured and the thing that constrains you were never the same number.
Changes
--work-dir, sized explicitly. Default isMemAvailableminus a 10% reserve;--rootfs-sizetakes512M/2G/50%;--no-rootfs-tmpfskeeps the old behaviour for operators who have already arranged storage. On that Pi the default is ~154 MiB instead of 83, and the budget is logged either way.HeadroomCheck, against the size we are about to mount, and refuse while the old OS is alive rather than OOM mid-extract. Sizing fromMemAvailablerather thanMemTotalis deliberate: the old OS still holds its memory, and an OOM kill is worse than an ENOSPC because the kernel picks the victim — on a small host as likelysshdasxmorph.build rootfserror, since ENOSPC there is now a sizing decision the operator can act on.Two follow-on fixes this exposed:
os.RemoveAll(WorkDir)fails withEBUSYonceWorkDiris a mount point — hit live once the tmpfs existed. Replaced withrootfs.CleanTarget, which empties the directory instead of unlinking it. That is also the operation the callers actually wanted: an empty target, not a deleted one. A reused mount is now cleaned too, so an aborted run cannot layer two rootfses together.Prepare's self-bind is skipped when the new root is already a mount. It exists only to satisfypivot_root's "new_root must be a mount point" rule, and stacking a second mount to re-satisfy a rule that already holds just adds a mount to unwind.Verification
Verified on the Pi:
xmorph buildinto the default work dir dies with ENOSPC at exactly the 83 MiB/runcap, and fills/runto 100%.os.RemoveAll(WorkDir)fails withEBUSYonce the work dir is a mount point (this is how that second bug was found).Verified off-host: cross-builds clean for
linux/arm64,gofmtclean, new unit tests for the size parser and the recommendation.Not yet verified on hardware: the post-fix success path — a full extraction into a correctly sized tmpfs. That run was interrupted when the test host rebooted, and I have not repeated it. The fix is therefore verified against the failure modes it targets but not yet against a clean end-to-end build. Worth re-running before merge.
Notes for the reprovision docs (not in this PR)
While reading the reprovision recipes against this host, three things stood out that the docs don't currently cover:
docs/reprovision/nixos.mdneeds ~2 GiB because it runsdisko-install, evaluating nix on the target. A prebuilt-image recipe — pivot into Alpine, thencurl image | zstd -d | dd of=/dev/disk— needs ~50 MB, because nothing is evaluated or held in RAM. That is a different shape from "run an installer" and it is the one that fits small hosts.CleanupOldRootusesMNT_DETACH. Lazy unmount does not release the block device, so writing to the underlying disk risks writeback from the old filesystem landing on top of the new image. That path wants the blockingUnmountOldRoot.Also a small one, unrelated to this change:
unmountUndermatches withstrings.HasPrefix(m, oldRoot), so/mnt/oldroot-backupmatches/mnt/oldroot.