Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 22 additions & 1 deletion .agents/skills/debug-openshell-cluster/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -299,7 +299,28 @@ all denied, inspect that file and the network sidecar logs.
The shared state directory should preserve `sandbox_gid` inheritance
(`02775`), and the SSH socket should be group-connectable (`0660`) so the
network sidecar can bridge gateway relay requests to the process supervisor.
Inspect all three when sandbox registration or egress enforcement fails:

If `supervisor_topology = "proxy-pod"` is rendered, each sandbox should have a
separate supervisor Deployment with one supervisor pod, a headless supervisor
Service, a proxy CA Secret, and two per-sandbox NetworkPolicies. The agent pod
should have `openshell.ai/sandbox-role=agent`; the supervisor pod should have
`openshell.ai/sandbox-role=supervisor`; both should share the same
`openshell.ai/sandbox-id`. The supervisor Deployment must have a controlling
`Sandbox` ownerReference. The Deployment pod template must carry the
`openshell.io/sandbox-id` annotation so the TokenReview bootstrap path can mint
a sandbox JWT. For supervisor pods, the gateway validates the
`Pod -> ReplicaSet -> Deployment -> Sandbox` owner chain, so missing
`apps/replicasets get` RBAC can also break bootstrap. Helm renders the
Deployment, ReplicaSet, Service, Secret, and NetworkPolicy RBAC only when
`supervisor.topology=proxy-pod`; if those resources fail with forbidden errors,
confirm both the rendered `gateway.toml` and Helm values use proxy-pod topology.
If the agent cannot reach the gateway, check DNS to the headless Service, the
agent egress NetworkPolicy DNS exception for kube-dns/CoreDNS, and the
supervisor ingress NetworkPolicy allowing only that agent pod on ports `3128`
and `18080`.

Inspect the relevant containers when sandbox registration or egress enforcement
fails:

```bash
kubectl -n openshell get configmap openshell-config -o jsonpath='{.data.gateway\.toml}' | grep supervisor_topology
Expand Down
52 changes: 48 additions & 4 deletions .agents/skills/helm-dev-environment/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,13 +70,25 @@ mise run helm:skaffold:run:sidecar
mise run helm:skaffold:run:sidecar-mtls
```

Both commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
**Supervisor proxy-pod topology** (build once and leave running):
```bash
mise run helm:skaffold:run:proxy-pod
```

All Skaffold commands build the `gateway` and `supervisor` images and deploy the OpenShell Helm
chart. The sidecar profile renders an `openshell-network-init` init container for
nftables setup and a non-root `openshell-supervisor-network` runtime sidecar for
proxying. The sidecar-mTLS profile reuses `ci/values-sidecar.yaml` and restores
`server.disableTls=false` inline for Skaffold. The `pkiInitJob` hook (a pre-install
Job that runs `openshell-gateway generate-certs`) generates mTLS secrets on first
install. Envoy Gateway opt-in; see the Optional Add-ons section below.
`server.disableTls=false` inline for Skaffold. The proxy-pod profile renders
network supervision in a separate supervisor Deployment with one pod and relies
on Kubernetes NetworkPolicy enforcement so the agent pod can reach only its
paired supervisor plus DNS. The
default local k3s/k3d cluster keeps k3s's embedded NetworkPolicy controller
enabled; if you replace the CNI, install a policy-enforcing CNI before using
proxy-pod. The
`pkiInitJob` hook (a pre-install Job that runs `openshell-gateway
generate-certs`) generates mTLS secrets on first install. Envoy Gateway opt-in;
see the Optional Add-ons section below.

The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or `kubectl port-forward`.

Expand All @@ -85,6 +97,31 @@ The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or
create the Secret named `openshell-ha-pg` with a `uri` key, then run
`mise run helm:skaffold:run` or `mise run helm:skaffold:dev`.

### Kubernetes e2e profiles

Run the default Kubernetes e2e environment:

```bash
mise run e2e:kubernetes
```

Run the sidecar topology e2e environment:

```bash
mise run e2e:kubernetes:sidecar
```

Run the proxy-pod topology e2e environment:

```bash
mise run e2e:kubernetes:proxy-pod
```

The proxy-pod e2e task applies `ci/values-proxy-pod.yaml` through
`OPENSHELL_E2E_KUBE_EXTRA_VALUES`. Use an existing cluster with NetworkPolicy
enforcement, or let the wrapper create the default local k3d/k3s cluster with
k3s's embedded NetworkPolicy controller enabled.

### TLS behaviour

`ci/values-skaffold.yaml` sets `server.disableTls: true`, so Skaffold-based deploys run
Expand Down Expand Up @@ -147,6 +184,12 @@ For a sidecar-profile deployment:
mise run helm:skaffold:delete:sidecar
```

For a proxy-pod-profile deployment:

```bash
mise run helm:skaffold:delete:proxy-pod
```

### Delete the cluster entirely

```bash
Expand Down Expand Up @@ -272,6 +315,7 @@ for dependencies still declared in `Chart.yaml`.
| `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |
| `deploy/helm/openshell/ci/values-keycloak.yaml` | Keycloak OIDC overlay |
| `deploy/helm/openshell/ci/values-sidecar.yaml` | Supervisor sidecar topology overlay for Kubernetes e2e/dev |
| `deploy/helm/openshell/ci/values-proxy-pod.yaml` | Supervisor proxy-pod topology overlay for Kubernetes e2e/dev; requires NetworkPolicy enforcement |
| `deploy/helm/openshell/ci/values-spire.yaml` | SPIFFE/SPIRE provider token grant overlay |
| `deploy/helm/openshell/ci/values-spire-stack.yaml` | SPIRE hardened chart values for local dev |
| `deploy/helm/openshell/ci/values-tls-disabled.yaml` | Lint-only: TLS + auth disabled (reverse-proxy edge termination) |
Expand Down
4 changes: 4 additions & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ http-body-util = "0.1"
tokio-rustls = { version = "0.26", default-features = false, features = ["logging", "tls12", "ring"] }
rustls = { version = "0.23", default-features = false, features = ["std", "logging", "tls12", "ring"] }
rustls-pemfile = "2"
rcgen = { version = "0.13", features = ["crypto", "pem"] }
rcgen = { version = "0.13", features = ["crypto", "pem", "x509-parser"] }
webpki-roots = "1"

# CLI
Expand Down
8 changes: 5 additions & 3 deletions architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,9 +64,11 @@ Podman, and VM drivers deliver the initial token through supervisor-only
runtime material; Kubernetes supervisors exchange a projected ServiceAccount
token through `IssueSandboxToken`. The gateway validates that projected token
with Kubernetes `TokenReview`, requires the configured sandbox service account,
checks the returned pod binding against the live pod UID, and verifies the pod's
controlling `Sandbox` ownerReference against the live Sandbox CR UID and
sandbox-id label before minting the gateway JWT. The bootstrap path accepts
checks the returned pod binding against the live pod UID, and verifies the
pod's ownership against the live Sandbox CR UID and sandbox-id label before
minting the gateway JWT. Agent pods must be directly controlled by the
`Sandbox` CR. Proxy-pod supervisor pods may be controlled through the Kubernetes
`Pod -> ReplicaSet -> Deployment -> Sandbox` chain. The bootstrap path accepts
both `agents.x-k8s.io/v1beta1` ownerReferences from newer Agent Sandbox
controllers and `agents.x-k8s.io/v1alpha1` ownerReferences from existing
deployments. Supervisors renew gateway JWTs in memory before expiry only while
Expand Down
23 changes: 23 additions & 0 deletions crates/openshell-core/src/sandbox_env.rs
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,17 @@ pub const NETWORK_BINARY_IDENTITY: &str = "OPENSHELL_NETWORK_BINARY_IDENTITY";
/// File written by the network supervisor when sidecar networking is ready.
pub const SUPERVISOR_READY_FILE: &str = "OPENSHELL_SUPERVISOR_READY_FILE";

/// TCP address the process supervisor waits for before starting when the
/// network supervisor runs outside the agent process.
pub const SUPERVISOR_READY_ADDR: &str = "OPENSHELL_SUPERVISOR_READY_ADDR";

/// File written by the process supervisor with the workload entrypoint PID and
/// read by the network sidecar for process/binary-bound network policy checks.
pub const ENTRYPOINT_PID_FILE: &str = "OPENSHELL_ENTRYPOINT_PID_FILE";

/// Address where an external network supervisor forwards gateway gRPC traffic.
pub const GATEWAY_FORWARD_ADDR: &str = "OPENSHELL_GATEWAY_FORWARD_ADDR";

/// Local protobuf policy snapshot written by the network sidecar and read by
/// the process-only supervisor in Kubernetes sidecar topology.
pub const SIDECAR_POLICY_SNAPSHOT_FILE: &str = "OPENSHELL_SIDECAR_POLICY_SNAPSHOT_FILE";
Expand All @@ -61,10 +68,26 @@ pub const SIDECAR_PROVIDER_ENV_SNAPSHOT_FILE: &str = "OPENSHELL_SIDECAR_PROVIDER
/// Optional TLS server name override used when connecting to the gateway.
pub const GATEWAY_TLS_SERVER_NAME: &str = "OPENSHELL_GATEWAY_TLS_SERVER_NAME";

/// Explicit URL injected into sandbox child processes for proxy-mode egress.
///
/// Kubernetes proxy-pod topology uses a headless Service DNS name, which
/// cannot be represented by the policy's `SocketAddr` proxy field.
pub const PROXY_URL: &str = "OPENSHELL_PROXY_URL";

/// Explicit listener address for the network supervisor's HTTP CONNECT proxy.
pub const PROXY_BIND_ADDR: &str = "OPENSHELL_PROXY_BIND_ADDR";

/// Directory where the network supervisor writes the proxy CA files consumed
/// by workload child processes.
pub const PROXY_TLS_DIR: &str = "OPENSHELL_PROXY_TLS_DIR";

/// Optional CA certificate PEM path used by the network supervisor instead of
/// generating an ephemeral CA.
pub const PROXY_CA_CERT_PATH: &str = "OPENSHELL_PROXY_CA_CERT_PATH";

/// Optional CA private key PEM path paired with [`PROXY_CA_CERT_PATH`].
pub const PROXY_CA_KEY_PATH: &str = "OPENSHELL_PROXY_CA_KEY_PATH";

/// Path to the CA certificate for mTLS communication with the gateway.
pub const TLS_CA: &str = "OPENSHELL_TLS_CA";

Expand Down
1 change: 1 addition & 0 deletions crates/openshell-driver-kubernetes/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ tracing = { workspace = true }
tracing-subscriber = { workspace = true }
thiserror = { workspace = true }
miette = { workspace = true }
rcgen = { workspace = true }

[dev-dependencies]
temp-env = "0.3"
Expand Down
8 changes: 8 additions & 0 deletions crates/openshell-driver-kubernetes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,14 @@ process supervisor runs in network-only mode and does not apply Landlock
filesystem policy, process privilege dropping, or process/binary identity
checks. Network endpoint and L7 policy remain enforced by the network sidecar.

The `proxy-pod` supervisor topology runs network enforcement and gateway
forwarding in a separate supervisor Deployment with one pod. The agent pod runs
only the process-mode supervisor and reaches the supervisor through a
per-sandbox headless Service. The driver creates an owner-referenced supervisor
Deployment with one replica plus Service, proxy CA Secret, and NetworkPolicy
resources so agent egress is limited to its paired supervisor pod plus DNS. If
the supervisor pod is deleted, the Deployment recreates it.

Sidecar mode uses the pod `fsGroup` to make the projected service-account token
and sandbox client TLS secret group-readable so the non-root process supervisor
can authenticate to the gateway. Treat the agent container as trusted with
Expand Down
17 changes: 16 additions & 1 deletion crates/openshell-driver-kubernetes/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ pub const DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME: &str = "default";
/// Default storage size for the workspace PVC.
pub const DEFAULT_WORKSPACE_STORAGE_SIZE: &str = "2Gi";

/// Default UID for the long-running Kubernetes network supervisor sidecar.
/// Default UID for the long-running Kubernetes network proxy.
pub const DEFAULT_PROXY_UID: u32 = 1337;

/// How the supervisor binary is delivered into sandbox pods.
Expand Down Expand Up @@ -65,13 +65,17 @@ pub enum SupervisorTopology {
/// Run network supervision in a privileged sidecar and process supervision
/// as a low-capability wrapper in the agent container.
Sidecar,
/// Run network supervision in a separate supervisor pod and process
/// supervision as a low-capability wrapper in the agent pod.
ProxyPod,
}

impl std::fmt::Display for SupervisorTopology {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Combined => f.write_str("combined"),
Self::Sidecar => f.write_str("sidecar"),
Self::ProxyPod => f.write_str("proxy-pod"),
}
}
}
Expand All @@ -83,6 +87,7 @@ impl FromStr for SupervisorTopology {
match s {
"combined" => Ok(Self::Combined),
"sidecar" => Ok(Self::Sidecar),
"proxy-pod" => Ok(Self::ProxyPod),
other => Err(format!("unknown supervisor topology '{other}'")),
}
}
Expand Down Expand Up @@ -516,6 +521,16 @@ mod tests {
assert_eq!(cfg.supervisor_topology, SupervisorTopology::Combined);
}

#[test]
fn serde_override_supervisor_topology_proxy_pod() {
let json = serde_json::json!({
"supervisor_topology": "proxy-pod"
});
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
assert_eq!(cfg.supervisor_topology, SupervisorTopology::ProxyPod);
assert_eq!(cfg.supervisor_topology.to_string(), "proxy-pod");
}

#[test]
fn serde_override_proxy_uid() {
let json = serde_json::json!({
Expand Down
Loading
Loading