Skip to content

Commit 7365094

Browse files
kerthcetkerthcet
committed
fix: add socket to avoid network issue
Signed-off-by: kerthcet <kerthcet@gmail.com>
1 parent 3d6cb84 commit 7365094

8 files changed

Lines changed: 437 additions & 6 deletions

File tree

‎.github/workflows/rust-ci.yaml‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,3 +65,29 @@ jobs:
6565

6666
- name: Run E2E tests
6767
run: make test-e2e
68+
69+
test-e2e-tunnel:
70+
name: Tunnel E2E Tests
71+
runs-on: ubuntu-latest
72+
steps:
73+
- uses: actions/checkout@v4
74+
75+
- name: Install Rust toolchain
76+
uses: actions-rust-lang/setup-rust-toolchain@v1
77+
with:
78+
toolchain: stable
79+
80+
- name: Setup Python
81+
uses: actions/setup-python@v5
82+
with:
83+
python-version: '3.12'
84+
85+
- name: Set up Docker Buildx
86+
uses: docker/setup-buildx-action@v3
87+
88+
# Regression coverage for the userspace-networking + SOCKS5 mesh dial path
89+
# (an unprivileged daemon reaching the controller only over the tailnet).
90+
# The controller container needs /dev/net/tun, which GitHub's Linux runners
91+
# provide; the daemon side is deliberately unprivileged.
92+
- name: Run tunnel-mode E2E tests
93+
run: make test-e2e-tunnel

‎Cargo.lock‎

Lines changed: 13 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Makefile‎

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ MATURIN := .venv/bin/maturin
55
# Pinned so lint results don't shift when ruff changes its default rule set.
66
RUFF_VERSION := ruff==0.15.15
77

8-
.PHONY: help build install dev test clean daemon-build daemon-release test-e2e docker-build docker-down
8+
.PHONY: help build install dev test clean daemon-build daemon-release test-e2e test-e2e-tunnel docker-build docker-down
99

1010
help:
1111
@echo "SandD - Sandbox Daemon - Build Commands"
@@ -14,7 +14,8 @@ help:
1414
@echo " make install - Install Python package locally"
1515
@echo " make dev - Install in development mode with hot reload"
1616
@echo " make test - Run unit and integration tests (fast, no Docker)"
17-
@echo " make test-e2e - Run end-to-end tests with Docker (slow)"
17+
@echo " make test-e2e - Run direct-mode end-to-end tests with Docker (slow)"
18+
@echo " make test-e2e-tunnel- Run tunnel-mode (Tailscale mesh) e2e tests (slow)"
1819
@echo " make daemon-build - Build daemon binary (debug)"
1920
@echo " make daemon-release - Build daemon binary (release)"
2021
@echo " make docker-build - Build Docker image for daemon"
@@ -58,12 +59,26 @@ test-e2e: $(PYTEST) dev
5859
@echo "Building Docker images..."
5960
docker compose -f hack/docker/docker-compose.e2e.yml build
6061
@echo ""
61-
@echo "Running E2E tests with Docker..."
62-
$(PYTEST) python/tests/ -m e2e -v -s
62+
@echo "Running direct-mode E2E tests with Docker..."
63+
$(PYTEST) python/tests/ -m "e2e and not tunnel" -v -s
6364
@echo ""
6465
@echo "Cleaning up containers..."
6566
docker compose -f hack/docker/docker-compose.e2e.yml down
6667

68+
# Tunnel-mode e2e uses its OWN compose stack (headscale + mesh) and the test
69+
# fixture mints the auth key mid-bringup, so it runs separately from test-e2e.
70+
# The `tunnel` marker selects only these tests; the fixture handles up/down of
71+
# docker-compose.tunnel-e2e.yml, but we `down` here too as a cleanup backstop.
72+
test-e2e-tunnel: $(PYTEST) dev
73+
@echo "Building tunnel-mode Docker images..."
74+
docker compose -f hack/docker/docker-compose.tunnel-e2e.yml build
75+
@echo ""
76+
@echo "Running tunnel-mode E2E tests (Tailscale/headscale mesh)..."
77+
$(PYTEST) python/tests/ -m tunnel -v -s
78+
@echo ""
79+
@echo "Cleaning up containers..."
80+
docker compose -f hack/docker/docker-compose.tunnel-e2e.yml down -v
81+
6782
docker-build:
6883
docker compose -f hack/docker/docker-compose.e2e.yml build
6984

‎hack/docker/docker-compose.tunnel-e2e.yml‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
# Tunnel-mode E2E — REGRESSION coverage for the userspace-networking dial path.
2+
#
3+
# WHY THIS EXISTS (the bug it guards against):
4+
# The direct-mode e2e (docker-compose.e2e.yml) and examples/tunnel-simple both
5+
# let the daemon reach the controller over the shared Docker BRIDGE (via the
6+
# `controller` hostname / host.docker.internal). That means the tailnet is set
7+
# up but NEVER load-bearing — a plain socket always had a route. So the one path
8+
# Nebula actually uses in prod — an UNPRIVILEGED daemon in
9+
# `--tun=userspace-networking`, where the tailnet has NO kernel route and the
10+
# WebSocket must traverse tailscaled's SOCKS5 proxy — had zero coverage, and a
11+
# regression there (daemons join the mesh yet `Active daemons: 0`) shipped.
12+
#
13+
# HOW THIS TEST FORCES THE MESH PATH:
14+
# The daemon dials `ws://controller.sandd.local:8765/ws` — a MagicDNS name. The
15+
# Docker embedded DNS only knows the container name `controller`, NOT the
16+
# `.sandd.local` FQDN, so the bridge cannot resolve (let alone route) it. The
17+
# name resolves ONLY inside tailscaled (MagicDNS) to the controller's 100.64.x
18+
# mesh IP, and — because the daemon runs userspace-networking with no route to
19+
# 100.64.0.0/10 — the only way the WebSocket connects is THROUGH the SOCKS5 proxy
20+
# with remote DNS (socks5h). Without that wiring in sandd, this test hangs at
21+
# "daemon failed to connect"; with it, the daemon connects and exec works.
22+
#
23+
# ASYMMETRY (mirrors Nebula, deliberately):
24+
# - controller: NET_ADMIN + /dev/net/tun. It is the INFRA side (in-cluster), and
25+
# accepting INBOUND mesh connections on a normal listening socket needs a real
26+
# TUN. This is not the constrained side, so privilege here is fine.
27+
# - daemon: UNPRIVILEGED, userspace-networking. This is the TENANT side (the GPU
28+
# workload container) — exactly what must work without NET_ADMIN/TUN.
29+
#
30+
# Orchestrated by python/tests/test_e2e_tunnel.py (mints the auth key between
31+
# `up headscale` and `up controller daemon`). Not part of the default e2e run.
32+
33+
services:
34+
# Headscale coordination server — assigns mesh IPs, runs MagicDNS for
35+
# *.sandd.local (see base_domain in the example's headscale-config.yaml).
36+
headscale:
37+
image: headscale/headscale:0.23
38+
command: serve
39+
volumes:
40+
# Reuse the example's config verbatim: magic_dns: true, base_domain:
41+
# sandd.local — that base_domain is what makes controller.sandd.local resolve.
42+
- ../../examples/tunnel-simple/headscale-config.yaml:/etc/headscale/config.yaml:ro
43+
- headscale-data:/var/lib/headscale
44+
networks:
45+
- mesh
46+
environment:
47+
- TZ=UTC
48+
49+
# Controller — the SandD server, on the mesh. Loops list_daemons()+exec and
50+
# prints distinctive markers the test greps for. NET_ADMIN/TUN: infra side.
51+
controller:
52+
hostname: controller
53+
build:
54+
context: ../..
55+
dockerfile: hack/docker/Dockerfile.server-tunnel
56+
command:
57+
- /bin/bash
58+
- -c
59+
- |
60+
set -e
61+
echo "[ctrl] starting tailscaled (kernel TUN, infra side)"
62+
tailscaled --state=/var/lib/tailscale/tailscaled.state &
63+
sleep 3
64+
tailscale up \
65+
--authkey="${SANDD_TUNNEL_AUTH_KEY:-}" \
66+
--login-server=http://headscale:8080 \
67+
--hostname=controller \
68+
--accept-routes
69+
echo "[ctrl] mesh IP: $$(tailscale ip -4)"
70+
python3 -u << 'PYEOF'
71+
import os, time
72+
from sandd import Server, TunnelConfig
73+
cfg = TunnelConfig(authkey=os.environ["SANDD_TUNNEL_AUTH_KEY"],
74+
server="http://headscale:8080")
75+
# The server joins the mesh via the running tailscaled/TUN above and
76+
# listens on :8765 across it.
77+
server = Server(host="0.0.0.0", port=8765, connect="tunnel", tunnel_config=cfg)
78+
print("[ctrl] controller ready, waiting for daemons", flush=True)
79+
seen = set()
80+
while True:
81+
for did in server.list_daemons():
82+
if did not in seen:
83+
seen.add(did)
84+
print(f"[ctrl] DAEMON_CONNECTED {did}", flush=True)
85+
r = server.exec(did, "hostname")
86+
if r.success:
87+
print(f"[ctrl] EXEC_OK {did} {r.stdout.strip()}", flush=True)
88+
time.sleep(2)
89+
PYEOF
90+
environment:
91+
- SANDD_TUNNEL_AUTH_KEY=${SANDD_TUNNEL_AUTH_KEY:-}
92+
cap_add:
93+
- NET_ADMIN
94+
devices:
95+
- /dev/net/tun
96+
depends_on:
97+
- headscale
98+
networks:
99+
- mesh
100+
101+
# Daemon — the TENANT side. UNPRIVILEGED, userspace-networking. Dials the
102+
# controller by its MagicDNS name so the bridge cannot carry the connection.
103+
daemon:
104+
build:
105+
context: ../..
106+
dockerfile: hack/docker/Dockerfile.daemon-tunnel
107+
entrypoint: ["/bin/bash", "-c"]
108+
command:
109+
- |
110+
set -e
111+
echo "[daemon] starting sandd --tunnel (userspace-networking, unprivileged)"
112+
# sandd itself brings up tailscaled --tun=userspace-networking with the
113+
# SOCKS5 proxy, joins the mesh, and dials the controller THROUGH the proxy.
114+
# The MagicDNS name is resolved remotely by tailscaled (socks5h), never on
115+
# the Docker bridge — so this only succeeds over the mesh.
116+
exec sandd \
117+
--server-url=ws://controller.sandd.local:8765/ws \
118+
--daemon-id=tunnel-daemon-1 \
119+
--tunnel \
120+
--tunnel-authkey="${SANDD_TUNNEL_AUTH_KEY:-}" \
121+
--tunnel-server=http://headscale:8080
122+
environment:
123+
- SANDD_TUNNEL_AUTH_KEY=${SANDD_TUNNEL_AUTH_KEY:-}
124+
- RUST_LOG=info
125+
# NO cap_add, NO devices — this is the whole point: it must work unprivileged.
126+
depends_on:
127+
- headscale
128+
- controller
129+
networks:
130+
- mesh
131+
132+
volumes:
133+
headscale-data:
134+
135+
networks:
136+
mesh:
137+
driver: bridge

‎pyproject.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,4 +55,5 @@ exclude = [
5555
asyncio_mode = "auto"
5656
markers = [
5757
"e2e: end-to-end tests with Docker (slow, skip by default)",
58+
"tunnel: tunnel-mode (Tailscale/headscale) e2e; needs its own compose stack",
5859
]

0 commit comments

Comments
 (0)