Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 23 additions & 3 deletions docs/egress.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,16 @@ bound to that sandbox's identity by the socket path — no in-band token. The
proxy evaluates the policy, injects the matched rule's secret host-side, dials
the origin, and relays the response.

A sandbox reaches it as a standard HTTP proxy:
The base image sets `http_proxy`/`https_proxy`/`no_proxy` on silkd's unit, and
silkd forwards exactly those variables into every exec — only when the guest
has no NIC beyond `lo`/`sit0`, so a lane with a working NIC is never steered
into a relay whose host side is closed. An unconfigured client just works, and
`-x` still does:

```sh
curl -x http://127.0.0.1:3128 https://api.github.com/user # allowed + credentialed
curl -x http://127.0.0.1:3128 https://evil.example/ # 403 egress denied: evil.example
curl https://api.github.com/user # allowed + credentialed
curl https://evil.example/ # 403 egress denied: evil.example
curl -x http://127.0.0.1:3128 https://api.github.com/user # explicit form, same path
```

## How it works (egress lane)
Expand Down Expand Up @@ -54,6 +59,21 @@ IPv6 forms (NAT64, 6to4, Teredo, IPv4-compatible) — so an allow-listed host
that resolves, or is rebound, to one cannot reach the sandboxd host or a
sibling VM.

`egress_internal_allow` re-admits named prefixes through that guard for nodes
whose sandboxes legitimately need internal services:

```jsonc
{ "egress_internal_allow": ["10.8.0.0/16", "fdc8::/16"] }
```

It is node-wide (every pool and tenant on the node gets the same re-admission)
and checked after NAT64 unwrapping, so an embedded IPv4 matches as the IPv4 it
is. Name service prefixes, never the whole private space: the guest bridges are
themselves ULA/RFC1918, so a blanket permit would open sandbox-to-sandbox and
the host's own gateway. `0.0.0.0/0` + `::/0` turns the guard off entirely — for
fleets with a policy-enforcing proxy in front — and requests still pass the
domain policy first; the allow-list widens the IP gate only.

### Deployment constraints

- **One sandboxd per host.** The restart sweep owns the whole `sandbox_egress_*`
Expand Down
3 changes: 3 additions & 0 deletions docs/silkd.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,9 @@ lane. `SILKD_NET=none|egress` overrides the probe for tests and operators.
On every lane silkd also binds `127.0.0.1:3128` and relays each connection to
the host's [guarded-egress](egress.md) proxy over vsock (`CID2:2049`); when the
host wired no policy the per-connection dial is refused, so the port is inert.
On the no-network lane silkd forwards the image-baked proxy variables
(`http_proxy` and friends) into every exec, so unconfigured clients use the
relay without being told; a lane with its own network never gets them.

## Limits

Expand Down
4 changes: 4 additions & 0 deletions os-image/android/silkd.rc
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ service silkd /system/bin/silkd
user root
group root
setenv SILKD_PORT 2048
# Mirrors the base:24.04 unit; forwarded into execs only on the no-NIC lane.
setenv http_proxy http://127.0.0.1:3128
setenv https_proxy http://127.0.0.1:3128
setenv no_proxy localhost,127.0.0.1,::1

on boot
start silkd
7 changes: 7 additions & 0 deletions os-image/base/24.04/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,10 @@ RUN --mount=type=secret,id=sandbox_install_agent \
# [silkd] product-tier daemon on vsock 2048, started at sysinit alongside
# the agent — this is the sandbox product's readiness signal. Same
# measured-fast ordering as the agent unit (no local-fs wait).
#
# 3128 is silkd's loopback relay over vsock to the host proxy, the none
# lane's only way out. silkd forwards these into execs only on the
# no-network lane, so a lane with its own NIC never dials a closed relay.
printf '%s\n' \
'[Unit]' \
'Description=silkd (sandbox product daemon: exec, files, sessions over vsock)' \
Expand All @@ -65,6 +69,9 @@ RUN --mount=type=secret,id=sandbox_install_agent \
'Type=simple' \
'ExecStart=/usr/local/bin/silkd' \
'Environment=SILKD_PORT=2048' \
'Environment=http_proxy=http://127.0.0.1:3128' \
'Environment=https_proxy=http://127.0.0.1:3128' \
'Environment=no_proxy=localhost,127.0.0.1,::1' \
'Restart=always' \
'RestartSec=2s' \
'LimitNOFILE=65536' \
Expand Down
20 changes: 20 additions & 0 deletions sandboxd/config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ import (
"encoding/json"
"fmt"
"net"
"net/netip"
"os"
"runtime"
"slices"
Expand Down Expand Up @@ -231,6 +232,11 @@ type Config struct {
// revocation bound for a replica a delete broadcast missed). Off by default.
CheckpointPeerHeal bool `json:"checkpoint_peer_heal,omitempty"`

// EgressInternalAllow re-admits CIDRs through the proxy's SSRF guard,
// node-wide (every pool and tenant). Prefixes, not a permit-private
// switch: the guest bridges are themselves ULA/RFC1918.
EgressInternalAllow []string `json:"egress_internal_allow,omitempty"`

// CheckpointTTLHours ages out checkpoints (0 = keep forever); the
// sweep runs hourly and on startup.
CheckpointTTLHours int `json:"checkpoint_ttl_hours,omitempty"`
Expand Down Expand Up @@ -380,6 +386,9 @@ func (c *Config) validate() error {
if c.CheckpointPeerHeal && c.APIToken == "" {
return fmt.Errorf("checkpoint_peer_heal requires api_token: without it resolveScope leaves the raw checkpoint blob GET reachable with no credential")
}
if err := c.validateEgressAllow(); err != nil {
return err
}
if err := c.validateTenants(); err != nil {
return err
}
Expand Down Expand Up @@ -448,6 +457,17 @@ func (c *Config) validateSecrets() (map[string]struct{}, error) {
return names, nil
}

// validateEgressAllow rejects a malformed prefix at load: a typo would
// otherwise surface as a refused dial long after startup.
func (c *Config) validateEgressAllow() error {
for _, cidr := range c.EgressInternalAllow {
if _, err := netip.ParsePrefix(cidr); err != nil {
return fmt.Errorf("egress_internal_allow %q: %w", cidr, err)
}
}
return nil
}

func (c *Config) validateTenants() error {
if len(c.Tenants) > 0 && c.APIToken == "" {
return fmt.Errorf("tenants require api_token: the operator surfaces are unreachable without it")
Expand Down
27 changes: 24 additions & 3 deletions sandboxd/pool/egress.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,13 @@ var (
netip.MustParsePrefix("3fff::/20"), // documentation
netip.MustParsePrefix("5f00::/16"), // SRv6 SIDs
}
)

// egressDialer blocks internal-address targets so the proxy cannot be an SSRF.
egressDialer = &net.Dialer{Control: func(_, address string, _ syscall.RawConn) error {
// newEgressDialer builds the proxy's upstream dialer: internal targets are
// blocked (the proxy must not be an SSRF), then exactly the node-named
// prefixes re-admitted.
func newEgressDialer(allow []netip.Prefix) *net.Dialer {
return &net.Dialer{Control: func(_, address string, _ syscall.RawConn) error {
host, _, err := net.SplitHostPort(address)
if err != nil {
return err
Expand All @@ -62,13 +66,18 @@ var (
b := ip.As16()
ip = netip.AddrFrom4([4]byte{b[12], b[13], b[14], b[15]})
}
// After the NAT64 unwrap (a v4-in-v6 target matches as the v4 it is),
// before the block, so a named prefix wins.
if slices.ContainsFunc(allow, func(p netip.Prefix) bool { return p.Contains(ip) }) {
return nil
}
if !ip.IsGlobalUnicast() || ip.IsPrivate() || ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() ||
slices.ContainsFunc(internalRanges, func(p netip.Prefix) bool { return p.Contains(ip) }) {
return fmt.Errorf("egress: blocked internal address %s", ip)
}
return nil
}}
)
}

// egressListener is one sandbox's egress accept point: an http.Server serving
// egress.Proxy over the per-sandbox UDS the VMM connects when the guest dials
Expand Down Expand Up @@ -223,3 +232,15 @@ func (m *Manager) effectivePolicy(sb *types.Sandbox) (egress.Evaluator, bool) {
return nil, false
}
}

// parsePrefixes turns the allow-list into prefixes; config validation already
// rejected malformed entries, so a leftover is dropped rather than failing dials.
func parsePrefixes(cidrs []string) []netip.Prefix {
out := make([]netip.Prefix, 0, len(cidrs))
for _, c := range cidrs {
if p, err := netip.ParsePrefix(c); err == nil {
out = append(out, p)
}
}
return out
}
59 changes: 57 additions & 2 deletions sandboxd/pool/egress_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -426,13 +426,13 @@ func TestEgressDialerBlocksInternal(t *testing.T) {
"[64:ff9b::a00:1]:80", // NAT64-embedded 10.0.0.1
}
for _, addr := range blocked {
if err := egressDialer.Control("tcp", addr, nil); err == nil {
if err := newEgressDialer(nil).Control("tcp", addr, nil); err == nil {
t.Errorf("dial to internal %s allowed; SSRF not blocked", addr)
}
}
// public v4 direct, public v6, and a public v4 via the NAT64 well-known prefix
for _, addr := range []string{"93.184.216.34:443", "[2606:4700:4700::1111]:443", "[64:ff9b::5db8:d822]:443"} {
if err := egressDialer.Control("tcp", addr, nil); err != nil {
if err := newEgressDialer(nil).Control("tcp", addr, nil); err != nil {
t.Errorf("dial to public %s blocked: %v", addr, err)
}
}
Expand All @@ -455,6 +455,61 @@ func TestEgressLaneCannotForkOrCheckpoint(t *testing.T) {
}
}

// TestEgressDialerAdmitsOnlyNamedInternalPrefixes pins the SSRF boundary:
// named prefixes re-admit corporate services without re-admitting the guest
// bridges, which are themselves ULA/RFC1918.
func TestEgressDialerAdmitsOnlyNamedInternalPrefixes(t *testing.T) {
d := newEgressDialer(parsePrefixes([]string{"fdc8::/16", "10.8.0.0/16"}))
check := func(addr string) error {
return d.Control("tcp", addr, nil)
}
for name, tc := range map[string]struct {
addr string
allowed bool
}{
"public v4": {"93.184.216.34:443", true},
"public v6": {"[2606:2800:220:1:248:1893:25c8:1946]:443", true},
"corporate v6": {"[fdc8:17:9:200f::1]:443", true},
"corporate v4": {"10.8.7.149:443", true},
// Guests and the host gateway share fc00::/7 with the corporate range.
"another sandbox on the bridge": {"[fd00:c0c0:38::5]:443", false},
"the host's bridge gateway": {"[fd00:c0c0:38::1]:443", false},
"other private v4": {"192.168.1.1:443", false},
"loopback": {"127.0.0.1:443", false},
"cloud metadata": {"169.254.169.254:80", false},
"CGNAT metadata": {"100.100.100.200:80", false},
} {
t.Run(name, func(t *testing.T) {
err := check(tc.addr)
if tc.allowed && err != nil {
t.Errorf("%s was blocked: %v", tc.addr, err)
}
if !tc.allowed && err == nil {
t.Errorf("%s was allowed; it must not be reachable from a guest", tc.addr)
}
})
}
}

// TestEgressDialerWildcardAllowsEverything pins the fully-open form as a
// deliberate choice — loopback and cloud metadata included — for fleets with
// a policy-enforcing forward proxy in front.
func TestEgressDialerWildcardAllowsEverything(t *testing.T) {
d := newEgressDialer(parsePrefixes([]string{"0.0.0.0/0", "::/0"}))
for _, addr := range []string{
"93.184.216.34:443", // public
"[fdc8:17:9:200f::1]:443", // corporate v6
"10.8.7.149:443", // corporate v4
"[fd00:c0c0:38::5]:443", // another sandbox
"127.0.0.1:7777", // the node itself
"169.254.169.254:80", // cloud metadata
} {
if err := d.Control("tcp", addr, nil); err != nil {
t.Errorf("%s blocked under a wildcard allow-list: %v", addr, err)
}
}
}

// egressClient builds an HTTP client that reaches the origin through the
// per-sandbox egress proxy UDS, exactly as the guest's proxy client would over
// vsock. Playing the VMM's role lets the host-side path run without a real VM.
Expand Down
2 changes: 1 addition & 1 deletion sandboxd/pool/pool.go
Original file line number Diff line number Diff line change
Expand Up @@ -384,7 +384,7 @@ func NewManager(ctx context.Context, cfg *config.Config, eng Engine, secrets *eg
healPending: map[string]struct{}{},
healAbort: map[string]struct{}{},
egressSecrets: secrets,
dial: egressDialer.DialContext,
dial: newEgressDialer(parsePrefixes(cfg.EgressInternalAllow)).DialContext,
sweep: netfilter.SweepExcept,
refillSem: make(chan struct{}, refill),
probeSem: make(chan struct{}, refill),
Expand Down
2 changes: 2 additions & 0 deletions silkd/src/git.rs
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@ use tokio::io::AsyncWrite;
use tokio::process::Command;

use crate::proto::{self, ErrorKind, GitBranchOp, GitFileStatus, Response};
use crate::sysutil;

/// Clones `url` into `path` (optionally a branch, shallow depth), then Done.
pub async fn clone<W: AsyncWrite + Unpin>(
Expand Down Expand Up @@ -169,6 +170,7 @@ async fn net_verb<W: AsyncWrite + Unpin>(
/// exec could otherwise scrape the token.
fn git_cmd(dir: &str, auth: Option<&str>) -> Command {
let mut cmd = Command::new("git");
sysutil::align_proxy_env(&mut cmd);
cmd.arg("-C").arg(dir);
apply_config(&mut cmd, auth);
cmd.stdin(Stdio::null())
Expand Down
2 changes: 2 additions & 0 deletions silkd/src/lsp.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ use tokio::process::{Child, ChildStdin, ChildStdout, Command};
use tokio::sync::mpsc;

use crate::proto::{self, ErrorKind, Request, Response};
use crate::sysutil;

const MANIFEST_DIR: &str = "/etc/silkd/lsp.d";

Expand Down Expand Up @@ -62,6 +63,7 @@ impl Broker {
}
};
let mut cmd = Command::new(&argv[0]);
sysutil::align_proxy_env(&mut cmd);
cmd.args(&argv[1..])
.stdin(Stdio::piped())
.stdout(Stdio::piped())
Expand Down
40 changes: 32 additions & 8 deletions silkd/src/net.rs
Original file line number Diff line number Diff line change
@@ -1,25 +1,49 @@
//! Guest network-lane detection. The egress lane has a device-backed NIC;
//! the none lane carries only virtual interfaces (lo, plus the tunnels an
//! all-builtin kernel auto-creates). Verbs that need the network (git
//! clone/push/pull) consult this so the none lane fails fast with a typed
//! error instead of hanging on an unreachable remote.
//! all-builtin kernel auto-creates). Network verbs (git clone/push/pull) and
//! proxy-env forwarding consult this.

use std::sync::OnceLock;
use std::sync::atomic::{AtomicI8, Ordering};

static LANE_OVERRIDE: AtomicI8 = AtomicI8::new(-1);

/// Reports whether the guest can reach a network. `SILKD_NET` overrides the
/// probe (`none` / `egress`) for operators and tests; otherwise an interface
/// under /sys/class/net backed by a real device (a `device` symlink) counts.
/// probe (`none` / `egress`) for operators; otherwise an interface under
/// /sys/class/net backed by a real device (a `device` symlink) counts.
/// Name filtering is not enough: the all-builtin sandbox kernel auto-creates
/// virtual tunnels (sit0 and friends) even on the no-NIC lane, and `lo` is
/// virtual too — only a virtio/physical NIC has a device backing. On
/// non-Linux dev hosts (no /sys/class/net) it defaults to true so git verbs
/// are testable.
pub fn has_egress() -> bool {
match std::env::var("SILKD_NET").as_deref() {
Ok("none") => return false,
Ok("egress") => return true,
match LANE_OVERRIDE.load(Ordering::Relaxed) {
0 => return false,
1 => return true,
_ => {}
}
match silkd_net() {
0 => return false,
1 => return true,
_ => {}
}
let Ok(entries) = std::fs::read_dir("/sys/class/net") else {
return true;
};
entries.flatten().any(|e| e.path().join("device").exists())
}

/// Lane override for tests: set_var would race every concurrent getenv.
pub fn override_egress_for_tests(lane: Option<bool>) {
LANE_OVERRIDE.store(lane.map_or(-1, i8::from), Ordering::Relaxed);
}

/// SILKD_NET decoded once: operator config, fixed at service start.
fn silkd_net() -> i8 {
static SILKD_NET: OnceLock<i8> = OnceLock::new();
*SILKD_NET.get_or_init(|| match std::env::var("SILKD_NET").as_deref() {
Ok("none") => 0,
Ok("egress") => 1,
_ => -1,
})
}
Loading
Loading