test(pivot): assert the machine survives its own pivot - #18
Merged
Conversation
The bricked-box regression shipped with the NixOS VM suite fully green. That
is the interesting part: nixos-pivot runs Go unit tests inside a VM and proves
pivot_root and the mount ordering are correct -- and they were. The pivot
executed flawlessly. What broke was the lifecycle afterwards: the supervised
entrypoint exited, no userspace was left, and the box sat there answering ICMP
from the kernel while every port was dead. Nothing asserted the machine was
still there, so nothing failed.
These tests assert liveness after the pivot rather than the syscall.
The observation channel is the crux. The NixOS backdoor is a systemd service
on the guest, so it dies with the old root and machine.succeed is unavailable
post-pivot -- which is presumably why this was never covered. The serial
console survives the pivot, and a reboot restores the backdoor, and between
them every outcome we care about is observable:
idle-stays-up pivot with `xmorph idle`, wait for its console line,
then watch for the reboot banner and fail if it
appears. This is the exact incident, inverted.
exit-reboots pivot into /bin/true. Exiting 0 is the point: the
original bug treated a clean exit as success. The
machine must come back on its own, proven by
multi-user.target starting again.
shell-refused-preflight a shell entrypoint with no TTY must be refused
before anything destructive, with the old root
intact and the error naming the way forward.
Offline throughout: --rootfs with a locally built busybox tarball, no registry
pulls, so CI stays hermetic and a dev can run these on a laptop.
resolv.conf is deliberately not retested here. The unit tests in
internal/postpivot cover absent / usable / stub-only / dangling-symlink far
more precisely than a VM can, and a VM test asserting "a file exists" would
cost minutes of CI to catch less.
The test script is Python, and pivotCmd interpolated a shell \ continuation into a string literal. The continued line then ended with a real newline before the closing quote, which Python rejects -- so all three tests failed the driver's type check before a VM ever booted. Emit one line instead, and note why at the definition. Also stop raising inside a try whose except must be broad (the driver's timeout exception type varies across nixpkgs): the raise was being swallowed by its own handler, so a rebooting machine would have passed. Set a flag and assert after. Generated test scripts now compile locally via python3 compile(), which catches this whole class of error without needing a Linux builder.
The lifecycle VM tests hung until the global timeout. Two causes, both in how the test observed the machine rather than in what it asserted: - The guest boots with `console=ttyS0 console=tty0`. The last one wins, so /dev/console is the graphics console and wait_for_console_text watches the serial line. Every byte the pivot wrote went somewhere nobody was reading. Write to /dev/ttyS0 by name. - NixOS test VMs get -no-reboot unless the machine is started with allow_reboot=True. exit-reboots asserts a reboot, so without it the correct behaviour would have killed QEMU and scored as a crash. Also drop the pretence that --contain could stand in for this: it runs the entrypoint directly and never reaches postpivot.Run, so it exercises none of the supervisor lifecycle. The more useful half is internal/postpivot/lifecycle_test.go. What bricked pwu-camera1 was a branch — reboot only on a *failed* exit, when a detached shell's exit is a clean one — and a branch can be tested without a VM. Supervise's reboot is now behind rebootHook so a test can observe the decision instead of taking the machine down with it. Reverting the fix fails TestSuperviseRebootsOnCleanExit, which is the point. idle-stays-up now asserts from a second VM that a TCP connection to the pivoted machine's SSH port still completes. Ping was never the question: the bricked box answered ping for hours, because the kernel does that on its own.
The lifecycle VM test pivoted cleanly, stayed up, and could not be
connected to. On the console:
level=ERROR msg=sshd err="sshd: no auth method configured
(need authorized_keys or password)"
sshd needs a password or authorized keys and gets neither, so it logs
that and never binds — on a console nobody reads, on a machine whose
entire reason for pivoting was to be reachable. docs/rescue.md already
says one of the two is required; only --ssh.password's help disagreed,
promising a random default that nothing ever generated.
So refuse it up front, next to the shell-entrypoint check and for the
same reason: the old root is still mounted and aborting is free.
Found by the test, which is the first useful thing it has done.
idle-stays-up passed every assertion in 79 seconds and then sat for 59
minutes until the global timeout killed it, which the run reported as a
failure.
After the test script returns, the driver does this:
for machine in self.machines:
if machine.is_up():
machine.execute("sync")
execute() calls connect(), and connect() waits on the backdoor shell in a
loop that never gives up. The backdoor is a systemd service on the guest.
It died with the old root — which is the premise of the test.
So end the test by pulling the plug over QMP, which needs nothing from
the guest. exit-reboots is unaffected: it reboots, and the backdoor comes
back with the machine.
The pivot itself is now demonstrably sound end to end. From the run:
prober # Connection to target (2001:db8:1::2) 22 port [tcp/ssh] succeeded!
after a real pivot_root, and again 20 seconds later.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #17. Those were the fixes; this is the net that should have caught them.
Why the suite was green while the box was bricked
nixos-pivotruns Go unit tests inside a VM and provespivot_rootand the mount ordering are correct — and they were. The pivot executed flawlessly. What broke was the lifecycle afterwards: the supervised entrypoint exited, no userspace was left, and the machine sat there answering ICMP from the kernel while every port was dead.Nothing asserted the machine was still there, so nothing failed. ICMP is the trap —
pivot_rootnever touches the kernel, so "it pings" is not evidence the pivot worked.The thing that made this hard to test
The NixOS backdoor is a systemd service on the guest, so it dies with the old root and
machine.succeedis unavailable post-pivot. I suspect that's why this was never covered.Two channels do survive: the serial console, and a reboot (which restores the backdoor). Between them every outcome we care about is observable.
The three tests
idle-stays-upxmorph idle, wait for its console line, then watch for the reboot banner and fail if it appearsexit-reboots/bin/true; machine must come back on its own, proven bymulti-user.targetstarting againshell-refused-preflightexit-rebootsuses/bin/truedeliberately: exiting 0 is the point. The original bug treated a clean exit as success and simply returned, which is what turned a stumble into a box needing physical access.Hermetic
Offline throughout —
--rootfswith a locally built busybox tarball, no registry pulls. CI stays hermetic and a dev can run these on a laptop against a throwaway VM instead of real hardware.Wired into
ci.ymlalongside the existing VM job, so they run on every push.Verification status, honestly
All three evaluate to real derivations (
nix eval .#checks.x86_64-linux.<name>.drvPath), andnix flake checkis clean. ButnixosTestis Linux-only and I'm on darwin, so I could not execute them locally — CI is the first real run. If a test script needs adjusting, that shows up in this PR's checks rather than in a pivot on hardware, which is the point of the change.resolv.confis deliberately not retested here: the unit tests ininternal/postpivotcover absent / usable / stub-only / dangling-symlink far more precisely than a VM can, and a VM test asserting "a file exists" would cost minutes of CI to catch less.