Wait out socket backpressure when streaming fds across a clone

[doanbaotrung]

fork_ipc_send_fds chunks descriptors at 120 per SCM_RIGHTS message, which bounds each control message but not how many sit unread in the socket at once. The parent streams every chunk in a tight loop while the freshly cloned child is still starting, so it outruns the receiver by a full socket buffer.

macOS refuses a control message that does not fit rather than queuing it, so a blocking sendmsg reports EMSGSIZE where a data-only write would block. At the default 8 KiB buffer that lands after about 1900 descriptors, which a guest reaches once its region list grows large enough -- dpkg passed it around the 198th package of an install:

clone: send backing fds failed: Message too long
clone: failed to send process state
dpkg: unrecoverable fatal error, aborting:
fork failed: Cannot allocate memory

Treat EMSGSIZE from a fixed-size chunk as backpressure: wait for writability and retry the same chunk. The child drains concurrently, and POLLOUT stays clear while the buffer holds control mbufs, so this blocks rather than spins. Waiting without a deadline matches fork_ipc_write_all on the same socket; a child that dies surfaces as POLLHUP.

---

Summary by cubic

Handle socket backpressure when streaming file descriptors during clone to prevent EMSGSIZE failures on macOS and avoid host termination from SIGPIPE. Previously, a blocking sendmsg failed with EMSGSIZE and aborted the clone; now we treat it as backpressure, wait for POLLOUT, and retry the same chunk.

• Wait for POLLOUT with poll(-1), handle EINTR, and resend the same fixed-size SCM_RIGHTS chunk.
• Add a small backoff after repeated stalls to avoid spinning if SNDLOWAT is low.
• On POLLHUP/ERR/NVAL, fail with EPIPE instead of looping.
• Set SO_NOSIGPIPE on the clone socketpair so a dying child does not terminate the host.

^{Written for commit 6c3d6f0. Summary will update on new commits.}

[cubic-dev-ai]

1 issue found across 1 file

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/runtime/fork-state.c">

<violation number="1" location="src/runtime/fork-state.c:131">
P2: The new wait-and-retry path can now attempt a `sendmsg` to a freshly cloned child that died while the parent was blocked in `poll`. That `sendmsg` returns `EPIPE`, and the fork IPC socket was created with a bare `socketpair(AF_UNIX, SOCK_STREAM, ...)` in forkipc.c with no `SO_NOSIGPIPE` — unlike every other socket in syscall/net.c, which sets it. Host `SIGPIPE` is unblocked and uses its default terminate disposition (only `SIGUSR2`/`SIGALRM` are masked at bring-up), so an `EPIPE` here raises `SIGPIPE` on the whole process instead of just failing the clone cleanly. Before this change the sender returned `EMSGSIZE` quickly and never waited on the child, so this `EPIPE`/`SIGPIPE` outcome is newly reachable through the retry loop. Consider setting `SO_NOSIGPIPE` on the fork socketpair where it is created (forkipc.c), or suppress the signal around the `sendmsg` retry, so a dead child surfaces as an error return rather than terminating the host.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: The new wait-and-retry path can now attempt a sendmsg to a freshly cloned child that died while the parent was blocked in poll. That sendmsg returns EPIPE, and the fork IPC socket was created with a bare socketpair(AF_UNIX, SOCK_STREAM, ...) in forkipc.c with no SO_NOSIGPIPE — unlike every other socket in syscall/net.c, which sets it. Host SIGPIPE is unblocked and uses its default terminate disposition (only SIGUSR2/SIGALRM are masked at bring-up), so an EPIPE here raises SIGPIPE on the whole process instead of just failing the clone cleanly. Before this change the sender returned EMSGSIZE quickly and never waited on the child, so this EPIPE/SIGPIPE outcome is newly reachable through the retry loop. Consider setting SO_NOSIGPIPE on the fork socketpair where it is created (forkipc.c), or suppress the signal around the sendmsg retry, so a dead child surfaces as an error return rather than terminating the host.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/runtime/fork-state.c, line 131:

<comment>The new wait-and-retry path can now attempt a `sendmsg` to a freshly cloned child that died while the parent was blocked in `poll`. That `sendmsg` returns `EPIPE`, and the fork IPC socket was created with a bare `socketpair(AF_UNIX, SOCK_STREAM, ...)` in forkipc.c with no `SO_NOSIGPIPE` — unlike every other socket in syscall/net.c, which sets it. Host `SIGPIPE` is unblocked and uses its default terminate disposition (only `SIGUSR2`/`SIGALRM` are masked at bring-up), so an `EPIPE` here raises `SIGPIPE` on the whole process instead of just failing the clone cleanly. Before this change the sender returned `EMSGSIZE` quickly and never waited on the child, so this `EPIPE`/`SIGPIPE` outcome is newly reachable through the retry loop. Consider setting `SO_NOSIGPIPE` on the fork socketpair where it is created (forkipc.c), or suppress the signal around the `sendmsg` retry, so a dead child surfaces as an error return rather than terminating the host.</comment>

<file context>
@@ -88,10 +89,48 @@ int fork_ipc_send_fds(int sock, const int *fds, int count)
-        sent += chunk;
+        if (!(pfd.revents & POLLOUT) &&
+            (pfd.revents & (POLLHUP | POLLERR | POLLNVAL))) {
+            errno = EPIPE;
+            return -1;
+        }
</file context>

[doanbaotrung]

Finding (SIGPIPE) — valid, and worse than described. Fixed.

Confirmed by direct test: sendmsg to a dead peer on a bare socketpair(AF_UNIX, SOCK_STREAM) kills the process — the harness exited 141 (128+13), and the line after the send never printed. The setup is exactly as stated: syscall/net.c sets SO_NOSIGPIPE on every guest socket, forkipc.c set it on none, and bring-up masks only SIGUSR2/SIGALRM ([proc.c:2981](third_party/elfuse/src/syscall/proc.c:2981)), so SIGPIPE keeps its default terminate disposition on vCPU threads.

One correction to the framing: this isn't newly reachable. The first sendmsg could always race a dying child, and fork_ipc_write_all's plain write() on the same socket has the same exposure and predates this PR. What the retry loop changes is the width of the window — it now deliberately waits on the child. So the bug is pre-existing, this PR makes it far more likely to fire, and the fix closes both paths at once.

Fixed at the socketpair rather than around the retry: SO_NOSIGPIPE on both ends in [forkipc.c](third_party/elfuse/src/runtime/forkipc.c:1541). It rides on the file description, so the spawned child inherits it. Re-ran the same test with the option set: A: survived, errno=Broken pipe, exit 0. A dead child now fails the clone instead of terminating the host.

[cubic-dev-ai]

No issues found across 1 file

_{Re-trigger cubic}

[maxliu04002]

looks good to me

[maxliu04002]

But I think it is better to check the return value of setsockopt().