Spread egress-lane VMs over several CNI networks - #58
Open
doge-rgb wants to merge 5 commits into
Open
Conversation
The proxy refuses every private and special-purpose target so a guest cannot turn it into an SSRF. That also refuses the corporate services a sandbox legitimately needs: internal names resolve to a ULA v6 (and an RFC1918 v4 the node cannot route anyway), both of which the guard reads as internal. Name the prefixes instead of adding a "permit private" switch. The guest bridges are themselves ULA and RFC1918, so a blanket permit would hand every sandbox a route to its neighbours and to the host's own gateway address — the two things the guard exists to prevent. An explicit list admits the corporate ranges and nothing adjacent to them. The list is validated at load, so a malformed prefix fails startup rather than surfacing as a refused dial hours later. Empty is the default and behaves exactly as before.
A node that lists the whole address space turns the guard off on purpose, loopback and cloud metadata included. That is a deliberate configuration for fleets that put a policy-enforcing forward proxy in front; the test records what it opens so the choice stays visible.
…s out A none-lane guest has no NIC at all — that is what makes the lane fast (1000 VMs in 5s at a flat 200/s, against 86s and a collapsing rate for the same count on the egress lane, where each VM takes a bridge port). Its only way out is silkd's loopback relay on 3128, which splices to the host's egress proxy over vsock. Nothing reached it before because nothing told a client it was there. The variables go on silkd's own unit rather than /etc/environment or a profile script: every exec in the guest is a child of silkd, so this is the one place that covers a bare `curl` as well as a login shell. On the egress lane the NIC carries traffic directly and the relay simply goes unused, so one image serves both.
An exec starts from a clean environment on purpose — silkd's own settings are not the guest's. But that also drops the one thing a guest cannot discover for itself. On the no-network lane there is no NIC at all; the only way out is the loopback relay silkd runs, and an unconfigured client never thinks to look for it. Baking the variables into silkd's unit was not enough for the same reason: they reached silkd and stopped there. Forward exactly the six proxy names, and only when set. Everything else stays uninherited, and a request's own env still layers on top.
A Linux bridge holds at most BR_MAX_PORTS ports — 1024, a compile-time kernel constant with no sysctl — and answers EXFULL for the next one. One bridge per node therefore caps egress density near a thousand VMs however much CPU and memory the node has, and a fleet run reached exactly that: nineteen of twenty nodes wedged between 1004 and 1014 ports. Measured on one node afterwards, the cost is not only the ceiling. Filling a single bridge to 1000 took 86s and the per-second rate collapsed from 105 to single digits as it filled, because br_add_if walks the bridge's port list while holding the global rtnl lock. The same node filled 1000 none-lane VMs, which take no bridge port at all, in 5s at a flat 200/s. Accept a list of conflists and hash the VM name to pick one, so N bridges give N×1024 and each stays in the range where that walk is short. Hashing rather than a counter keeps the choice free of process state: a VM resolves to the same conflist whoever asks and whenever, which matters because its record persists the network it was built on and restore has to agree. The scalar `network` key normalizes into the list, so the binary rolls onto nodes still carrying the old spelling without touching their config, and a one-entry list is byte-for-byte the old argv.
CMGS
force-pushed
the
pr2-none-lane-egress
branch
from
July 29, 2026 16:33
7e6e17d to
296acbc
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A Linux bridge holds 1024 ports:
BR_MAX_PORTSis a compile-time constant, andfind_portno()scans linearly for a free one while holding the global rtnlmutex. So a single bridge caps the egress lane at 1024 VMs per node, and long
before that cap the scan itself becomes the bottleneck — measured on one node,
egress-lane creation peaked at 105/s and collapsed to single digits within five
seconds as the bridge filled, with load average reaching 287 while the CPU sat
idle. The none lane, which attaches nothing, held a flat ~200/s throughout.
A node may now name several CNI networks instead of one. Each VM is assigned by
hashing its own name, so VMs spread evenly across the configured networks and
the assignment is stable for a given VM without any per-node state. Four bridges
move the ceiling to ~4096 per node and cut the average port-scan length by the
same factor.
This does not remove the rtnl serialization, only divides the work that happens
under it. The lane that avoids the lock entirely is
net=none(#57), which iswhy that one is the default.
🤖 Generated with Claude Code