feat(integrations): Harbor agent-phase fidelity — sandbox identity, network policy, terminal, and harness isolation - #527
Closed
jdchawla29 wants to merge 26 commits into
Closed
Conversation
`--clearenv` arrived in bubblewrap 0.5.0, but debian bullseye — and every image built on it — ships 0.4.1. There, bwrap aborted on the unknown option before running anything, so every command in the session exited 1: the agent got a shell where `true` and `echo` failed, spent its budget diagnosing that, and the rollout still graded as a completed run scoring 0.0. Two of the ten terminal-bench 2.0 sample tasks are bullseye-based and lost exactly this way. `usable_bwrap` could not catch it: its probe creates namespaces but passes none of the options a session goes on to use, so an old bwrap looks healthy and then fails every command. Give the payload its environment with `env -i` instead, matching what the non-bwrap dropped-privilege path already does. Isolation is unchanged — the mounts and namespace flags all predate 0.4 — and the option set a session passes is now covered by a test rather than discovered on a user's image.
The adapted layer installed uv and its managed interpreter at uv's defaults, which are relative to the invoking user's home — inside the very filesystem the task grades. The graded party then found a HUD runtime its task never declared, plus interpreter shims pointing into /hud, which the workspace masks and which therefore resolve to nothing. `hud`'s version check compounded it: no user to prompt, a cache that is never warm because the container is fresh each rollout, so it called PyPI on the rollout's critical path and wrote its state into the graded tree. Pin every uv directory under /hud, disable the cache nothing reads after the build, and turn the version check off in the image. /hud is the one path the workspace already masks, so containment is what makes that single mask sufficient instead of something to clean up afterwards.
Two ways an adapted image differed from what Harbor gives the same task, both of which move the score without any model being involved. Harbor uploads /tests when it runs the verifier, so the agent phase never contains the assertions being graded; the adapted image laid that directory down before the agent ran. Mask it like the rest of the grading material and let _grade lay the baked copy down at the point Harbor would. Harbor also installs its agent *into* the container, provisioning python3, pip, git and curl (BaseInstalledAgent.SYSTEM_PACKAGES) before the agent phase. HUD's agent is external and installs nothing, so an adapted image handed the agent a barer machine than Harbor would — a harness difference scored as a model difference. Bake the same set at build time, presence-checked per tool so an image shipping its own interpreter keeps exactly that one.
Every exec channel built its own bwrap sandbox, so each command ran in a
fresh pid namespace that died with it. Files persisted — the root is a real
directory — which made the workspace look stateful right up to the point an
agent started something and used it in the next command. `nohup`, `setsid`,
a listener: all gone, with nothing to say so.
That makes a whole class of task unsolvable while scoring like a weak agent.
Both qemu tasks in the terminal-bench 2.0 sample ("start the image in the
background and leave it running") and configure-git-webserver graded 0.0 on
it; the qemu pair passes now. Harbor has no such problem because its agent
works in one persistent tmux session in the container.
Hold one sandbox and have sessions join it with nsenter. The user namespace
is joined first, which is what confers the privilege to join the rest in a
container given no extra capability; the network namespace is joined only
when the sandbox owns one, since a sandbox sharing the container's netns
cannot rejoin it from inside bwrap's userns. The pid comes from bwrap's
--info-fd, and readiness from the holder's own first line of output: bwrap
names the child before that child finishes building its mount namespace, so
the pid alone can point at a half-assembled root.
A command that completes now keeps its process group, since `some-server &`
is how an agent starts what it means to use next; the sandbox is the
lifetime boundary instead, discarded at the rollout boundary so a reused
container never hands the next agent the last one's daemons. Commands that
time out, get abandoned, or run with no sandbox are still torn down as a
group, having nothing else to bound them.
Output was accumulated in the handler and written to the channel only once the process finished. An agent running a build or a test suite saw nothing until it ended, and a session that does not exit — a shell held open across calls — produced nothing at all, which foreclosed any harness tool built on a persistent session. Worse, a command stopped at the timeout reported only that the deadline had passed: the epilogue returned before writing what had been collected, so the output showing how far it got was discarded precisely when it was the only evidence available. Relay both streams as they arrive. The timeout message then says why the command stopped rather than standing in for its output, and a held-open shell reports each command as it runs — with cwd, exports and aliases persisting, since it is one shell — leaving what tools to build on that to the harness, which is where that choice belongs.
Sessions always ran on pipes, so isatty() was false and every program that branches on it took its non-interactive path: no readline, no curses, no progress rendering, and a terminal task graded on behaviour a terminal would never produce. A pty request from the client was accepted by the protocol and then quietly ignored. Honour it: allocate a terminal pair, hand the child the slave as its three std fds, and relay the master to the channel. stderr merges into stdout, as it does on any tty. The size the client asked for is applied to the terminal rather than left at a default, and TERM goes into the session environment — sessions start from an exact environment, so without that deliberate step curses has no terminal description and fails outright. Verified through nsenter into the sandbox, not just locally: stty reports the requested geometry, stdin is a pts device, and a curses program starts and reads back the right size.
Two gaps left by the pty itself. A session had terminal file descriptors but no controlling terminal, so /dev/tty could not be opened (ENXIO) and job control had no foreground process group: `less`, `vim` reading the terminal directly, and anything expecting Ctrl-C to reach a foreground job behaved unlike they would in a terminal. Claim it with `setsid -c` around the payload. Run there it is not a process-group leader, so it execs in place rather than forking, leaving the pid and exit status the caller waits on intact. And a window resize arrives from asyncssh as an exception raised by the stdin read, not as data — TerminalSizeChanged, which is not an asyncssh.Error and so was caught by nothing. The first resize escaped the relay coroutine and killed the session's keyboard for good. Handle it where it surfaces: resize the terminal and keep reading. Verified in the container: /dev/tty opens, tcgetpgrp reports the session's own process group as the foreground one, and curses still starts clean.
… too The unsandboxed branch built a bare `bash -lc`, so `env` and `tty` were accepted and then dropped: a caller asking for a terminal or passing variables got neither, silently, depending only on whether bwrap happened to be usable. It surfaced as a terminal session with no TERM, which leaves curses with no terminal description at all. Run the same payload the sandboxed forms do. `cwd` stays the one argument the unsandboxed form cannot honour — there is no mount namespace to chdir into — and the docstring now says so rather than leaving the caller to find out. This also removes the TERM patch-up the handler was doing on the side, so one mechanism carries the session environment on every path.
…nd /tmp Two things told an agent what was running it, and one of them handed over a key while doing so. The serving process's environment carried through to sessions whole, so anything HUD was configured with arrived in the agent's `env` — the version check flag this branch added, and on a deployed placement an API key. Filter HUD's own variables out of what a session inherits. What the task or the caller declares is layered on afterwards and still arrives, HUD-shaped name or not: this drops only what the serving process happened to hold. Session key material went to a mkdtemp under /tmp, which is the *task's* /tmp: the agent could read the private key to its own shell, the host key and authorized_keys, in a directory named after the harness. Let a caller place credentials somewhere it masks, and have Harbor put them under /hud. Outside the served root was never the same as out of the session's reach. Verified from inside the container: the agent's environment has no HUD_ anything, /tmp is empty, and a search for key material finds none.
…at all /tests and the verdict directory were masked with a tmpfs, so the agent saw them as empty directories. The base image ships both empty and an earlier rollout left them behind, so they were there for the whole agent phase — two directories the task's own image would not have, announcing that something is holding the assertions and the verdict back. Harbor does not have them during its agent phase either; it uploads the verifier when it runs it. Do the same: remove both when the task starts and again when it ends, and let _grade lay them down in between for as long as the verifier needs. Removing rather than emptying is the point — an empty directory is the signal. Where the serve process cannot remove them (not root, and they sit at the filesystem root) it empties them instead, which is what the mask achieved. /hud keeps its mask: it has to exist for the whole rollout, being what serves. This drops the stricter-than-Harbor property the masks gave, that a process the agent leaves running cannot reach the assertions or the verdict while the verifier works. That exposure is Harbor's own in shared mode, and its answer is environment_mode = "separate", which this integration still refuses rather than grading as if it were shared.
bwrap maps a single id when left to create the user namespace alone, so every file owned by anyone else is nobody inside it: unreadable, unwritable, not even chownable by the sandbox's own root. For an image whose task runs as a non-root user that is most of the filesystem — the agent cannot write its own working directory — and no session can drop to an id the map does not contain, which made setpriv useless inside a sandbox. Install the map from out here instead, over bwrap's --userns-block-fd: the sandbox is held at its own creation until its ids are known. Writing a full range needs CAP_SETUID and CAP_SETGID in this namespace, which a container's root has by default — no added capability, no newuidmap, and both flags are old enough for the bubblewrap on bullseye. Where the kernel refuses, the fallback is the single id bwrap would have mapped: a narrow map is workable, an absent one is not. Also lets a caller keep the workspace root's ownership as the image staged it, rather than handing it to the session's uid.
The layer restored the image's USER for the whole container, which is every process in it — including the one serving. That identity is a statement about the task's *phases*: Harbor runs the agent and the verifier as it, and its own harness is not subject to it. Ours serves from inside the image, so the directive demoted the harness too, and a non-root harness can create neither /tests at the filesystem root nor its own state under /hud. Hence the pre-created, chowned-at-build directories, and hence a rollout that could not start at all once the session keys moved under /hud. Apply it where each phase runs instead: agent sessions drop with shell_uid, the verifier with setpriv at its own invocation, and the serving process stays root. Per phase, because Harbor's are — a task may hand the agent a restricted account and still verify as root, which one USER directive cannot express. Each falls back to the image's own USER, recorded into the layer because the build context is gone by serve time. Both carry a home: dropping ids alone leaves a phase pointed at the harness's, which it cannot write, and Harbor's verifiers commonly install their own tooling. The workspace root keeps the ownership the image gave it. This only works because the sandbox now maps the container's whole id space; with bwrap's single id, setpriv could not reach the declared user at all.
…stem The adaptation layer kept itself at /hud, masked with a tmpfs. That hid what was in it and left the directory: a name at the root of the filesystem that the task's own image never had, plus the same name in the sandbox argv that pid 1 reports. Nothing of ours was readable through it, so this is not about confidentiality — it is the last thing in the agent's namespace identifying what is running it. Keep it under /media instead and rebuild that parent for sessions: a fresh tmpfs, then every sibling bound back as it was, so the entry is absent rather than present-and-empty. Only --tmpfs and --bind, so it holds on the bubblewrap that ships with bullseye, and writes elsewhere — including new directories at the filesystem root, which a task like configure-git-webserver needs — still land in the real filesystem for the verifier. What is given up is entries created directly in that one parent during a rollout, which is why it is a parent nothing writes to: /media is for removable media, where /srv and /opt are where service data and add-on software legitimately go. The path is a placeholder in the templates now rather than a literal, so it is stated once. An agent session sees a root with no /hud, no /tests and no /logs — the image's own filesystem, plus the sandbox it is running in.
A verifier is sandboxed for one reason — a task declaring no-network needs a network namespace, and bwrap can only make one inside a user namespace. That namespace maps a single id, so the verifier could not chown: not untar with ownership, not run apt, which is most of what Harbor's verifiers do before they assert anything. It failed with EINVAL and graded 0.0, indistinguishable from a task the agent got wrong. Hold it at creation and map it, exactly as an agent session's sandbox is now. The handshake moves to the workspace as install_identity_map, since both sides of it were the same fifteen lines. Verified with a verifier that chowns and probes the network: chown=yes, network=severed. Fixing the ids does not hand back the network the task ruled out.
…n it A workspace shared the substrate's network, so a session could address anything else listening on it — the control channel included. That is how a graded party could call tasks.grade, read the verifier's own output, and take the session key out of hello. Reachability was the whole of it: nothing was wrong with the channel, it was simply on the same network as the thing being graded. So give the workspace a network of its own and one way out of it. The proxy listens on a unix socket, which makes reaching it a question of the filesystem rather than the network; a bridge runs in the workspace's network namespace while keeping this one's mount namespace, so it can see a socket the workspace cannot, and offers it as an ordinary proxy port on the workspace's loopback. Nothing of ours is bound inside, and the substrate is not addressable from in there — the agent's namespace ends up with exactly one listener, which is the proxy. Every connection passing one process is also what makes a policy possible, so Harbor's network_mode stops being refused: an allowlist is the hosts the task named, no-network is none of them, and public leaves the network as it was. The verifier's allowlist is still refused, because it runs on the substrate's network with nothing between it and the hosts it dials. Parsing is the standard library's. A hand-rolled proxy got keep-alive and body framing wrong in the way that matters here: apt failed halfway through an index, which reads as a broken task rather than a broken harness.
… own Public was the one policy that still shared the substrate's network, which meant "the agent may reach any host" also meant "the agent may reach the channel grading it, the ssh daemon serving it, and anything else the substrate is running". Every task in the sample is public, so the exposure was the common case rather than the exception. Public now means every host reached the same way as any other policy: through the workspace's own way out. The agent's network ends up with exactly one listener, the proxy. Bounding it moves the verifier too. Harbor grades a service by talking to it — telnet to the VM the agent booted, git clone from the server it configured — and those listen in the workspace now, so the verifier joins its user and network namespaces while keeping this mount namespace: /tests and the verdict stay where the workspace cannot reach them, and the ids map through, so it can still chown what it untars. Behind that boundary it needs the way out too, since Harbor's verifiers commonly install their own tooling first. The bridge is waited for rather than merely spawned. Sessions start as soon as the sandbox does, and a session that opens with a package install against a port not yet bound reads a connection refused as a network that does not work — which looked like a per-host failure until the timing was the thing being measured. Verified on every shape: public, allowlisted, no-network, non-root image, a task whose verifier reaches a service the agent started, and configure-git-webserver end to end.
nsenter opens a directory given to --wd before it joins any namespace, so the path is resolved on the substrate rather than inside the sandbox. The default guest path exists only in the sandbox's mount namespace, so every session on a workspace that took bwrap's defaults died at startup with "nsenter: cannot open /workspace" — the shell never ran, and the failure looked like the command's. Bare --wd is the sandbox's own working directory, which bwrap set to the guest path when it built it, so a session lands where it always meant to. That leaves nothing for enter_argv's cwd to select: it named a directory nsenter could only misread, and no caller passed one. Harbor was unaffected — its guest path is the image's WORKDIR, which the substrate has too — which is why the sample kept scoring.
…ares A workspace with a network of its own cannot address the substrate at all. That is the point where the control channel is concerned, but it applies just as much to services the environment itself runs: a database the task depends on, an API it is meant to call, anything an image starts. Until now those were as unreachable as the channel, so a bounded workspace could only be given to tasks that needed nothing but the internet. A Peer is one of those services, handed back deliberately and at the address the task expects rather than wherever it happens to listen. The proxy's bridge already offered one socket on the workspace's loopback; it now offers one per route, and a peer is a socket that splices to a substrate address instead of applying a host policy. Names come from a hosts file the sandbox mounts over its own, so a task that says "db:5432" finds it there; 127.0.0.1 is used wherever the port is free, since a task that says "localhost:8080" means that one, and only a second service on the same port moves. Peers are reached directly rather than through the proxy — sent there, they would be resolved out on the substrate, where the name means nothing and the address is something else — so no_proxy names them. A workspace that may reach no host is told of no proxy at all, rather than being pointed at one that was never started.
…done Transfer-Encoding is hop-by-hop, and the proxy was passing it through. The response body comes back from http.client already de-chunked, so the client was handed plain bytes under a header telling it to look for chunk lengths: curl reports "illegal or missing hexadecimal sequence in chunked-encoding" and the body is whatever survives that. Chunked is what a server reaches for when it cannot know the length up front — a generated index, a streamed archive — so this was worst for exactly the kind of fetch a task does before it can start work.
…e agent's
The verifier joins the workspace's network namespace so it can reach a
service the agent started — telnet to the VM it booted, git clone from the
server it configured. It was then taking the workspace's way out too, which
applies the *agent's* allowlist. A task that restricted its agent to one host
therefore restricted its grader to the same one, and Harbor's verifiers
almost all begin by installing their own tooling:
Err:1 http://ports.ubuntu.com/ubuntu-ports noble InRelease
403 Forbidden [IP: 127.0.0.1 3128]
/tests/test.sh: line 23: uv: command not found
The verifier then scored the task 0.0 for a reason its author never wrote,
and nothing in the run said so.
A visitor — a process in the workspace's network that is not one of its
sessions — now gets a way out of its own, on its own port, under the policy
its own phase declared. It exists only while the visitor runs: the agent's
sessions share that network, so a second and more permissive route standing
open would be one the agent could simply take instead of the one it was
given. During the agent phase the port refuses connections.
That makes a verifier-declared allowlist enforceable rather than ignored, so
the refusal that existed because it was not is gone, and each phase's hosts
are carried separately in the workspace policy that keys environments.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| length = response.getheader("Content-Length") | ||
| for key, value in response.getheaders(): | ||
| if key.lower() not in _HOP_BY_HOP and key.lower() != "content-length": | ||
| self.send_header(key, value) |
| length = response.getheader("Content-Length") | ||
| for key, value in response.getheaders(): | ||
| if key.lower() not in _HOP_BY_HOP and key.lower() != "content-length": | ||
| self.send_header(key, value) |
| if key.lower() not in _HOP_BY_HOP and key.lower() != "content-length": | ||
| self.send_header(key, value) | ||
| if length is not None: | ||
| self.send_header("Content-Length", length) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Agent-phase fidelity for Harbor environments: the graded party's world must be exactly what the task declared — identity, filesystem, network, terminal — and the harness must not be observable inside it.
What this changes
/media/hud) and keeps its state out of the graded filesystem, session environment, and/tmp; the agent phase can be denied the grading paths entirely.allowed_hostsenforced by an egress proxy, substrate services reachable only as declared peers, and the verifier held to its own declared hosts rather than the agent's.adapt()builds the task's own environment as a base image plus a generated serving layer, with a per-group manifest; the integration consumes the standard environment boundary instead of string-templating Dockerfiles.Verification
Validated end to end against terminal-bench-2 tasks: adapted, deployed, synced, and run to graded verdicts on real container runtimes, locally and hosted. Follow-up fixes found during that validation are in #526.