feat(kubernetes): add sidecar supervisor topology#2076
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
🌿 Preview your docs: https://nvidia-preview-pr-2076.docs.buildwithfern.com/openshell |
|
Label |
4f981c2 to
8c9ae53
Compare
Some comments with help of agent1. Central trust boundary — the workload can use the sandbox's gateway identity . 2. nftables fence only covers TCP/UDP. 3. 4. Doc drift in the debug skill. |
e00a00b to
ed858eb
Compare
Addressed this by moving the supervisor and sandbox to run with different GIDs. Updated the other areas with clarifying comments. |
|
Deployed this branch on an OpenShift 4.22 cluster (K8s 1.35) with Root causeIn
Step 2 runs as root with Note the state dir was already changed to |
|
@mrunalp I reproduced locally and pushed a fix. Please have a look when you get a chance! |
Add the Kubernetes sidecar supervisor topology, its Helm/Skaffold configuration, topology documentation, and sidecar e2e matrix coverage. Skip root-only sandbox identity rewriting when process enforcement is network-only so the low-permission sidecar process container can start successfully. Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
3d80066 to
254cdbe
Compare
| # staging directory before the image build job runs. | ||
|
|
||
| FROM scratch AS supervisor | ||
| FROM alpine:3.22 AS supervisor |
There was a problem hiding this comment.
Making a note that this might break the podman implementation which expects a scratch container. I might be wrong here though. I would expect E2E tests to shake this out.
There was a problem hiding this comment.
e2e tests seem to pass with this change; open to other options if this is a blocker
There was a problem hiding this comment.
I'll have to test. It's going to depend more on the output of the build than the base. Basically the way the podman mount works is it mounts the entire filesystem of the image to the mount point inside the instantiated container runtime and the supervisor binary needs to live at the root of that.
| case "${selected_engine}" in | ||
| docker|podman) | ||
| export CONTAINER_ENGINE="${selected_engine}" | ||
| return | ||
| ;; | ||
| *) | ||
| echo "ERROR: CONTAINER_ENGINE=${CONTAINER_ENGINE} is invalid; expected docker or podman" >&2 | ||
| exit 2 | ||
| ;; | ||
| esac |
There was a problem hiding this comment.
why is there a docker branch here?
Looks like you've rebased now since that's been merged? |
| /// Supervisor pod/runtime topology. Kubernetes sidecar mode sets this to | ||
| /// `"sidecar"`; the default combined supervisor path omits it. | ||
| pub const SUPERVISOR_TOPOLOGY: &str = "OPENSHELL_SUPERVISOR_TOPOLOGY"; | ||
|
|
||
| /// Network enforcement backend selected by the compute driver. | ||
| pub const NETWORK_ENFORCEMENT_MODE: &str = "OPENSHELL_NETWORK_ENFORCEMENT_MODE"; | ||
|
|
||
| /// Process enforcement mode selected by the compute driver. | ||
| /// | ||
| /// The default when unset is `"full"`, where the process supervisor enforces | ||
| /// filesystem/process policy before spawning workloads. Kubernetes sidecar | ||
| /// topology sets this to `"network-only"` so the process wrapper can run as | ||
| /// the sandbox UID without Linux capabilities while preserving SSH/session | ||
| /// behavior. | ||
| pub const PROCESS_ENFORCEMENT_MODE: &str = "OPENSHELL_PROCESS_ENFORCEMENT_MODE"; | ||
|
|
||
| /// Whether network policy evaluation must bind requests to the peer binary. | ||
| /// | ||
| /// The default when unset is `"required"`. Kubernetes sidecar experiments may | ||
| /// set this to `"relaxed"` to enforce endpoint and L7 policy without per-binary | ||
| /// `/proc` identity binding. | ||
| pub const NETWORK_BINARY_IDENTITY: &str = "OPENSHELL_NETWORK_BINARY_IDENTITY"; | ||
|
|
||
| /// File written by the network supervisor when sidecar networking is ready. | ||
| pub const SUPERVISOR_READY_FILE: &str = "OPENSHELL_SUPERVISOR_READY_FILE"; | ||
|
|
||
| /// File written by the process supervisor with the workload entrypoint PID and | ||
| /// read by the network sidecar for process/binary-bound network policy checks. | ||
| pub const ENTRYPOINT_PID_FILE: &str = "OPENSHELL_ENTRYPOINT_PID_FILE"; | ||
|
|
||
| /// Local protobuf policy snapshot written by the network sidecar and read by | ||
| /// the process-only supervisor in Kubernetes sidecar topology. | ||
| pub const SIDECAR_POLICY_SNAPSHOT_FILE: &str = "OPENSHELL_SIDECAR_POLICY_SNAPSHOT_FILE"; | ||
|
|
||
| /// Local provider environment snapshot written by the network sidecar and read | ||
| /// by the process-only supervisor in Kubernetes sidecar topology. | ||
| pub const SIDECAR_PROVIDER_ENV_SNAPSHOT_FILE: &str = "OPENSHELL_SIDECAR_PROVIDER_ENV_SNAPSHOT_FILE"; | ||
|
|
||
| /// Optional TLS server name override used when connecting to the gateway. | ||
| pub const GATEWAY_TLS_SERVER_NAME: &str = "OPENSHELL_GATEWAY_TLS_SERVER_NAME"; | ||
|
|
||
| /// Directory where the network supervisor writes the proxy CA files consumed | ||
| /// by workload child processes. | ||
| pub const PROXY_TLS_DIR: &str = "OPENSHELL_PROXY_TLS_DIR"; | ||
|
|
There was a problem hiding this comment.
why do these constants need to be in core?
There was a problem hiding this comment.
They don't; I'll clean this up. It looks like this sandbox_env is all the configuration that's used by the supervisor, and the config for this isn't located in the driver. Not sure how best to clean this up.
| install_sidecar_iptables_legacy_bypass_rules(proxy_uid).map_err(|iptables_error| { | ||
| miette::miette!( | ||
| "sidecar nft ruleset load failed: {nft_error}; sidecar iptables-legacy fallback failed: {iptables_error}" | ||
| ) | ||
| }) |
There was a problem hiding this comment.
(from agent review)
When nftables loading fails on a dual-stack pod, this fallback reports success after installing only IPv4 iptables-legacy rules. IPv6 TCP/UDP traffic remains unrestricted, allowing workloads to bypass the sidecar proxy and network policy entirely. Install equivalent ip6tables-legacy rules or fail closed when IPv6 cannot be fenced.
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Run nftables setup as individual commands so optional conntrack and log expressions can fail without rolling back required table, chain, and reject rules. Signed-off-by: Seth Jennings <sjenning@redhat.com> Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
|
Discussed with @TaylorMutch: Suggestion: keep Landlock (and seccomp) in sidecar
|
Render sidecar pods with a shared process namespace, keep binary-aware network policy enabled, and move Kubernetes sidecar settings under the nested sidecar config table. Also apply unprivileged Landlock/seccomp setup in NetworkOnly supervisor mode so sidecar topology keeps sandbox child hardening without privileged process setup. Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Coordinate sidecar policy and provider bootstrap over a local Unix socket so the process leaf no longer reads policy/provider snapshot files. Report entrypoint startup through the control channel and keep gateway credentials confined to the network sidecar. Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Summary
Adds an experimental Kubernetes
sidecarsupervisor topology after the combined-topology base from #2074. The defaultcombinedtopology remains unchanged;sidecaris opt-in and may change as we continue validating the permission model, pod layout, and configuration surface across Kubernetes runtimes.The goal of
sidecaris to move pod-level network enforcement, gateway authentication, gateway session handling, and SSH relay ownership out of the agent container and into a dedicated network sidecar. The agent container runs as the resolved sandbox UID/GID withrunAsNonRoot,allowPrivilegeEscalation: false, and all Linux capabilities dropped.Outcome
combinedas the default topology and full OpenShell enforcement path.topology = "sidecar"under[openshell.drivers.kubernetes]and nests sidecar-only settings under[openshell.drivers.kubernetes.sidecar]; Helm renders the same shape fromsupervisor.topologyandsupervisor.sidecar.*.shareProcessNamespace: truefor sidecar pods so the network sidecar can resolve workload process and binary identity through/proc.supervisor.sidecar.processBinaryAwareNetworkPolicy=falsedrops the sidecarSYS_PTRACEcapability and downgrades sidecar network policy to endpoint/L7 enforcement without matchingpolicy.binaries.deploy/helm/openshell/ci/values-sidecar-kata.yamlas the local/CI-dev overlay used for Kata validation.Enforcement Model
combinedremains the complete OpenShell sandbox contract: network policy, filesystem policy, process controls, supervisor privilege drop, supervisor identity mount isolation, gateway relay, SSH, exec, and file sync all run through the existing agent-container supervisor path.sidecaris network-focused but no longer disables all process-side enforcement:/procinspection.Runtime Validation Status
gcloud container clusters create-auto --workload-policies=allow-net-admin; the Google Cloud CLI reference listsallow-net-adminas the supported Autopilot workload policy: https://docs.cloud.google.com/sdk/gcloud/reference/container/clusters/create-auto#--workload-policiesdeploy/helm/openshell/ci/values-sidecar-kata.yaml. This validation passed without using the supervisor sideload init container.Related Issue
References #1827, #981, #899, #1305.
Related PRs: #1973, #2074, #2016.
Changes
sidecartopology and sidecar-specific configuration.supervisor_topologytotopologyand move sidecar settings underopenshell.drivers.kubernetes.sidecar.network-onlybehavior for sidecar mode while preserving workload launch, Landlock/seccomp application, and SSH/session relay behavior.SYS_PTRACEcapability when users accept the downgrade.Testing
git diff --check origin/main..feat/kubernetes-sidecar-topology-v2cargo check -p openshell-driver-kubernetes -p openshell-sandbox -p openshell-supervisor-process -p openshell-supervisor-networkcargo test -p openshell-driver-kubernetes --libcargo test -p openshell-supervisor-process --libmise run helm:testmarkdownlint-cli2 docs/kubernetes/topology.mdx docs/kubernetes/setup.mdxruby -e 'require "yaml"; YAML.load_file(".github/workflows/branch-e2e.yml"); puts "ok"'mise run rust:lintmise run rust:format:check && mise run rust:lintcargo test -p openshell-supervisor-processCARGO_BUILD_JOBS=1 mise run pre-commitOpenShell / Branch Checkspassing on this branch, including Linux amd64 and arm64 Rust jobs.HELM_K3S_LB_HOST_PORT=18080 mise run e2e:kubernetes:sidecar--workload-policies=allow-net-admin.deploy/helm/openshell/ci/values-sidecar-kata.yaml; validation passed without the supervisor sideload init container.Checklist