diff --git a/docs.json b/docs.json index b5b7db02..812aba5d 100644 --- a/docs.json +++ b/docs.json @@ -221,6 +221,7 @@ "openhands/usage/agent-canvas/backend-setup/local", "openhands/usage/agent-canvas/backend-setup/vm", "openhands/usage/agent-canvas/backend-setup/docker", + "openhands/usage/agent-canvas/backend-setup/kubernetes", "openhands/usage/agent-canvas/backend-setup/cloud", "openhands/usage/agent-canvas/backend-setup/modal" ] diff --git a/openhands/usage/agent-canvas/backend-setup/docker.mdx b/openhands/usage/agent-canvas/backend-setup/docker.mdx index 99454964..85768fe6 100644 --- a/openhands/usage/agent-canvas/backend-setup/docker.mdx +++ b/openhands/usage/agent-canvas/backend-setup/docker.mdx @@ -81,3 +81,4 @@ Then add the Docker backend: - [Connect and Manage Backends](/openhands/usage/agent-canvas/backends) - [Local Backend](/openhands/usage/agent-canvas/backend-setup/local) - [VM / Self-Hosted Installation](/openhands/usage/agent-canvas/backend-setup/vm) +- [Kubernetes (Helm)](/openhands/usage/agent-canvas/backend-setup/kubernetes) diff --git a/openhands/usage/agent-canvas/backend-setup/kubernetes.mdx b/openhands/usage/agent-canvas/backend-setup/kubernetes.mdx new file mode 100644 index 00000000..5f152a39 --- /dev/null +++ b/openhands/usage/agent-canvas/backend-setup/kubernetes.mdx @@ -0,0 +1,515 @@ +--- +title: Kubernetes (Helm) +description: Install Agent Canvas into a Kubernetes cluster with the official Helm chart. +--- + +The official Helm chart deploys the all-in-one Agent Canvas image (frontend + agent-server + automation) on Kubernetes as a `StatefulSet` with a `PersistentVolumeClaim` for durable state, a `Service`, and an optional `Ingress` and RBAC layer. It's the recommended way to run Agent Canvas as a shared, always-on backend that survives pod restarts and image upgrades. + + + **Turn this into an internal vibecoding platform.** Once Agent Canvas runs in your cluster with [RBAC enabled](#rbac), you can give it a skill that teaches the agent how to deploy the small web apps it builds straight into the cluster. From that point on, **anyone with access to the Agent Canvas UI can build and ship code into the cluster — and save it to GitHub — with just a prompt.** No pipelines, no manual `kubectl`, no hand-written manifests: the agent scaffolds the app, applies the manifests, and gives back a live URL. + + The "save to GitHub" half of that loop requires the **GitHub MCP server** to be enabled so the agent can create repos and push commits on the user's behalf. See the [generic app-deployment skill](#skill-deploying-apps-into-the-cluster) below for a ready-to-adapt version, and [RBAC](#rbac) for the permissions it needs. + + + + The agent server can read and write the pod filesystem, execute shell commands, and — when RBAC is enabled — mutate the Kubernetes cluster it runs in. Treat the release namespace as trusted infrastructure, put it behind an authenticated ingress before exposing it to the internet, and only turn on `rbac.clusterAdmin` when you truly need cluster-wide access. + + +## Prerequisites + +- Kubernetes **1.24 or later** (required by the chart's `kubeVersion` constraint). +- [Helm **3.x**](https://helm.sh/docs/intro/install/). +- A working `kubectl` context with permission to create resources in the target namespace. +- A `StorageClass` that supports `ReadWriteOnce` volumes. On GKE this is usually `standard-rwo` on older node pools or `hyperdisk-balanced` on `c4` / `n4` node pools. On EKS it's `gp3`. On DigitalOcean/Linode it's `do-block-storage` / `linode-block-storage`. +- An ingress controller (nginx, Traefik, cloud-provider ingress, etc.) if you want to reach Agent Canvas from outside the cluster. + +## Get the Chart + +The chart lives alongside the source in the `OpenHands/agent-canvas` repo. Clone it and install from the local path: + +```bash +git clone https://github.com/OpenHands/agent-canvas.git +cd agent-canvas +helm install agent-canvas ./helm/agent-canvas \ + --namespace agent-canvas --create-namespace +``` + +That single command deploys everything below. Agent Canvas is now reachable inside the cluster at `http://agent-canvas.agent-canvas.svc.cluster.local:8000`. See [Access It](#access-it) for how to reach it from a browser. + +## What Gets Deployed + +| Resource | Purpose | +|---|---| +| `StatefulSet` | Single-replica pod running the all-in-one image. | +| `PersistentVolumeClaim` (per pod) | Backs `~/.openhands` and `~/workspace` (both mounted from the same PVC via `subPath`): settings, encrypted secrets, conversation history, automation SQLite DB, cloned repos, generated files. | +| `Service` (`ClusterIP`) | Cluster-internal endpoint on port 8000. | +| `Service` (headless) | Required by the `StatefulSet` for stable pod DNS. | +| `ServiceAccount` | Stable identity the pod runs under. | +| `Ingress` (optional) | External HTTP(S) entry point. | +| `RoleBinding` (per namespace) | Created when `rbac.enabled=true`, one per entry in `rbac.namespaces`. | +| `ClusterRoleBinding` (optional) | Created when `rbac.clusterAdmin=true`. | + +## Persistence + +The chart provisions **one** PVC and mounts it at multiple well-known subdirectories of the openhands user's HOME via `subPath`. That preserves the pristine `/home/openhands` the base image ships (dotfiles like `~/.bashrc` and `~/.profile`) while persisting the directories that actually contain state: + +- `~/.openhands` — agent-server settings and encrypted secrets, conversation history and event stores, automation SQLite database (unless you point at external Postgres — see [External Database](#external-database)), the `OH_SECRET_KEY` and session API key auto-generated on first boot +- `~/workspace` — the agent's default working directory: cloned repos, worktrees, anything the agent writes when it treats `~` as the workspace root + +Both paths share the same underlying disk. Add more entries to `persistence.mounts` if you want other subtrees persisted (e.g. `~/.cache`, `~/.config`). + +Defaults: + +```yaml +persistence: + enabled: true + mounts: + - mountPath: /home/openhands/.openhands + subPath: openhands + - mountPath: /home/openhands/workspace + subPath: workspace + size: 20Gi + # storageClassName: "" # empty → cluster default + accessModes: + - ReadWriteOnce +``` + + + The pod runs as `openhands` (UID 10001) from the upstream image. The chart sets `podSecurityContext.fsGroup: 10001` so the kubelet chowns the PVC on mount and the process can write to it. If you override `securityContext` or `podSecurityContext`, make sure UID/GID/fsGroup all point at the same user or `openhands` won't be able to write to the volume. + + + + On GKE clusters using `c4` or `n4` node pools, the default `standard-rwo` StorageClass will fail to attach because those machine types require `hyperdisk-balanced`. Set `persistence.storageClassName: hyperdisk-balanced` explicitly. + + +### Bring Your Own PVC + +If you already manage the volume out of band, point the chart at it and it will skip the `volumeClaimTemplates` path: + +```yaml +persistence: + enabled: true + existingClaim: my-agent-canvas-pvc +``` + +## Ingress + +Ingress is off by default. Enable it and provide the standard knobs — the chart supports `className`, `annotations`, multiple `hosts` with per-path routing, and TLS. + +```yaml +ingress: + enabled: true + className: nginx + annotations: + nginx.ingress.kubernetes.io/proxy-read-timeout: "3600" + nginx.ingress.kubernetes.io/proxy-send-timeout: "3600" + nginx.ingress.kubernetes.io/proxy-body-size: "50m" + cert-manager.io/cluster-issuer: letsencrypt-prod + hosts: + - host: agent-canvas.example.com + paths: + - path: / + pathType: Prefix + tls: + - hosts: + - agent-canvas.example.com + secretName: agent-canvas-tls +``` + + + The read/send timeout annotations matter — Agent Canvas holds long-lived WebSocket connections for streaming agent events. Without generous timeouts, the ingress controller will close idle streams and the UI will drop reconnects mid-turn. Nginx defaults to 60 seconds. + + +## RBAC + +RBAC is **off by default**. The pod runs under its own ServiceAccount but has no in-cluster permissions. Turn it on when the agent needs to inspect or mutate Kubernetes resources (e.g. to deploy things it builds). + +Two independent switches: + +```yaml +rbac: + enabled: true + # Full access to all resources in these namespaces (bound to the + # built-in `admin` ClusterRole via one RoleBinding per namespace). + # Each namespace must already exist in the cluster. + namespaces: + - default + - agent-sandbox + # Optionally grant cluster-admin. OFF by default. Very broad — enable + # only when the agent truly needs to manage the whole cluster. + clusterAdmin: false +``` + + + `rbac.clusterAdmin: true` grants `cluster-admin` — the highest privilege level in Kubernetes. An agent with a compromised prompt (or a bad LLM response) could delete every resource in the cluster. Prefer scoping to specific namespaces with `rbac.namespaces` whenever possible. + + +## Skill: Deploying Apps Into the Cluster + +To unlock the internal vibecoding platform described at the top of this page, give the agent a [skill](/overview/skills/creating) that teaches it the conventions for shipping the apps it builds into a namespace of your cluster. Drop the markdown below into `.openhands/skills/deploy-app/SKILL.md` (or your workspace's skills directory), adjust the placeholders (``, ``, GitHub org), and the agent will scaffold, deploy, and — with the **GitHub MCP server enabled** — push each app to its own repo on request. + +This is a generic version of the skill the OpenHands team runs internally. It assumes the backend was installed with [`rbac.enabled=true`](#rbac) and a `rbac.namespaces` entry for the target namespace, so the pod's ServiceAccount can `kubectl apply` there directly. + +````markdown +# Deploy apps into the cluster + +Use this skill to create and manage the small web apps you build, serving each +one at `https://.` from the `` namespace of the +cluster Agent Canvas runs in. + +## Platform conventions + +Every app follows the same pattern: + +- **Namespace:** ``. The agent runs under a ServiceAccount that has + admin in this namespace (granted via the Helm chart's `rbac.namespaces`), so + `kubectl apply` works directly with no extra credentials. +- **Content:** static files (HTML/JS/CSS) served by an `nginx:*-alpine` pod. + The files live in a **ConfigMap** (`-web`) mounted at + `/usr/share/nginx/html`. Apps that need a backend add their own container. +- **Objects per app:** `Deployment` + `Service` (ClusterIP, port 80) + + `Ingress`. Apps that need scheduled work add a `CronJob`. +- **Host:** `.`. +- **TLS:** if cert-manager is installed, add the + `cert-manager.io/cluster-issuer: ` annotation and a `tls` block with + `secretName: -tls`; the cert is issued automatically. +- **Auth:** put shared apps behind your ingress's authentication (oauth2-proxy, + a forward-auth middleware, Cloudflare Access, etc.) so they aren't exposed + unauthenticated. Reference your cluster's auth middleware/annotation here. +- **Resources:** keep them tiny (requests `10m`/`16Mi`, limits `100m`/`64Mi`). + +### Ingress template + +```yaml +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: + namespace: + labels: + app: + annotations: + cert-manager.io/cluster-issuer: + # Add your cluster's auth middleware/annotation here so the app is + # not exposed unauthenticated. +spec: + ingressClassName: # e.g. nginx or traefik + rules: + - host: . + http: + paths: + - path: / + pathType: Prefix + backend: + service: + name: + port: + number: 80 + tls: + - hosts: + - . + secretName: -tls +``` + +### Deployment + Service template + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: + namespace: + labels: + app: +spec: + replicas: 1 + selector: + matchLabels: + app: + template: + metadata: + labels: + app: + spec: + containers: + - name: web + image: nginx:1.27-alpine + ports: + - containerPort: 80 + resources: + requests: + cpu: 10m + memory: 16Mi + limits: + cpu: 100m + memory: 64Mi + volumeMounts: + - name: web + mountPath: /usr/share/nginx/html + volumes: + - name: web + configMap: + name: -web +--- +apiVersion: v1 +kind: Service +metadata: + name: + namespace: + labels: + app: +spec: + selector: + app: + ports: + - port: 80 + targetPort: 80 +``` + +## Secrets (never commit them) + +If an app needs credentials at runtime, store them in a Kubernetes `Secret` +created out of band — never in a ConfigMap, the git repo, or the manifest, and +never print their values. Create the Secret from environment variables so the +plaintext never appears in a command line or file: + +```bash +kubectl create secret generic - -n \ + --from-literal=="$SOME_ENV_VAR" \ + --dry-run=client -o yaml | kubectl apply -f - +``` + +Consume it in the Deployment via `env` + `secretKeyRef`. Document the one-off +`kubectl create secret ...` command in the app's `README.md` — keep it out of +`deploy.sh` and the committed manifest. + +## Source layout & GitHub + +- Keep each app's source under `~/workspace/`. +- Give each app its own GitHub repo (e.g. `/app-`). + **This requires the GitHub MCP server to be enabled** so the agent can create + the repo and push commits. Create it if it doesn't exist, then push. +- Standard repo layout: + - `README.md` — what the app is, its URL, how to deploy, any one-off secrets. + - `k8s/.yaml` — all Kubernetes objects (Deployment+Service+Ingress). + - `web/` — static assets served via the ConfigMap. + - `deploy.sh` — regenerates the ConfigMap from `web/`, applies `k8s/`, and + rolls the Deployment. + +### Typical `deploy.sh` + +```bash +#!/usr/bin/env bash +set -euo pipefail +NS= +DIR="$(cd "$(dirname "$0")" && pwd)" + +kubectl create configmap -web --namespace "$NS" \ + --from-file="$DIR/web/" \ + --dry-run=client -o yaml | kubectl apply -f - + +kubectl apply -f "$DIR/k8s/.yaml" +kubectl rollout restart deployment/ -n "$NS" +kubectl rollout status deployment/ -n "$NS" +``` + +## Creating a new app + +1. `mkdir -p ~/workspace//{k8s,web}` and add `web/index.html`, + `k8s/.yaml` (from the templates above), `deploy.sh`, and a + `README.md`. +2. If GitHub MCP is enabled, create/verify `/app-`, + commit, and push. +3. Deploy: `./deploy.sh`. +4. If cert-manager is used, wait for the cert: + `kubectl get certificate -tls -n ` should become + `READY=True`. Confirm the Ingress has an address. +5. Report the live URL back to the user. + +## Updating an app + +Edit files under `~/workspace/`, commit + push (via GitHub MCP), then +re-run `./deploy.sh` (which restarts the Deployment so nginx reloads the +ConfigMap). + +## Deleting an app + +1. `kubectl delete -f ~/workspace//k8s/.yaml` and delete the + `-web` ConfigMap. Deleting the Ingress lets cert-manager clean up the + TLS secret; delete any credential secrets explicitly. +2. Optionally archive/delete the GitHub repo and remove + `~/workspace/`. + +## Verifying access + +```bash +kubectl get deploy,svc,ingress,certificate -n -l app= +``` +```` + +## Common Configurations + +### Minimal (defaults + ingress) + +```yaml +# values.yaml +ingress: + enabled: true + className: nginx + hosts: + - host: agent-canvas.example.com + paths: + - path: / + pathType: Prefix + tls: + - hosts: [agent-canvas.example.com] + secretName: agent-canvas-tls +``` + +### With LLM Credentials from a Secret + +Rather than typing your LLM key into the UI on every reinstall, pass it in through the chart. Create the secret separately, then reference it via `config.extraEnv`: + +```bash +kubectl -n agent-canvas create secret generic llm \ + --from-literal=api-key=sk-... +``` + +```yaml +# values.yaml +config: + extraEnv: + - name: LLM_MODEL + value: "openhands/claude-sonnet-4-5-20250929" + - name: LLM_API_KEY + valueFrom: + secretKeyRef: + name: llm + key: api-key +``` + +### Agent That Manages a Sandbox Namespace + +```yaml +# values.yaml +rbac: + enabled: true + namespaces: + - agent-sandbox +``` + +Create the sandbox namespace before installing (`kubectl create namespace agent-sandbox`). Then the pod can `kubectl apply` / `kubectl delete` anything inside `agent-sandbox` but nothing else. + +### External Database + +The automation subsystem uses a SQLite database on the PVC by default. For higher-volume deployments, point it at Postgres: + +```yaml +# values.yaml +config: + automationDbUrl: "postgresql+asyncpg://user:pass@postgres.databases.svc.cluster.local/agent_canvas" +``` + +Store the actual credentials in a Kubernetes Secret and reference them via `config.extraEnv` rather than putting the password in `values.yaml`. + +## Install and Upgrade + +```bash +# First install +helm install agent-canvas ./helm/agent-canvas \ + --namespace agent-canvas --create-namespace \ + -f values.yaml + +# Later upgrades +helm upgrade agent-canvas ./helm/agent-canvas \ + -n agent-canvas -f values.yaml + +# Check rollout +kubectl -n agent-canvas rollout status statefulset/agent-canvas +kubectl -n agent-canvas get pvc,pod,svc,ingress +``` + +To pin a specific image (e.g. a PR preview or a build newer than the chart's `appVersion`): + +```bash +helm upgrade agent-canvas ./helm/agent-canvas \ + -n agent-canvas -f values.yaml \ + --set image.tag=sha- +``` + +## Access It + +The chart's default `Service` is `ClusterIP`. Three common ways to reach the UI: + +1. **Ingress** — configure the `ingress:` block as shown above. This is the production path. +2. **Port-forward** — for quick access from your laptop without touching DNS or ingress: + + ```bash + kubectl -n agent-canvas port-forward svc/agent-canvas 8000:8000 + ``` + + Then open `http://localhost:8000`. +3. **LoadBalancer** — set `service.type: LoadBalancer` if your cloud provisions cloud load balancers for you. Cheaper than ingress for one-off installs, but skips TLS and auth. + + + The agent server accepts any request with the right `LOCAL_BACKEND_API_KEY`, so exposing it via a bare LoadBalancer means anyone on the internet who can guess the key can drive the agent. Prefer the Ingress path with an authenticated proxy (oauth2-proxy, Cloudflare Access, tailscale-serve, ngrok OAuth, etc.) in front of it. + + +## Uninstall + +```bash +helm uninstall agent-canvas -n agent-canvas +``` + +The PVC created by the `StatefulSet` is **retained** on uninstall so a reinstall picks up where you left off. Delete it explicitly if you want a fully clean slate: + +```bash +kubectl -n agent-canvas delete pvc -l app.kubernetes.io/instance=agent-canvas +``` + +## Troubleshooting + +### `FailedAttachVolume: pd-balanced disk type cannot be used by c4-standard-8 machine type` + +The default StorageClass on your cluster is provisioning a disk type your nodes can't attach. On GKE `c4` / `n4` node pools, use `hyperdisk-balanced`: + +```yaml +persistence: + storageClassName: hyperdisk-balanced +``` + +Because `volumeClaimTemplates` on an existing `StatefulSet` are immutable, changing the StorageClass requires deleting the STS and PVC first: + +```bash +kubectl -n agent-canvas delete statefulset agent-canvas +kubectl -n agent-canvas delete pvc -l app.kubernetes.io/instance=agent-canvas +helm upgrade agent-canvas ./helm/agent-canvas -n agent-canvas -f values.yaml +``` + +### `ErrImagePull` on `ghcr.io/openhands/agent-canvas:` + +Verify the tag exists on GHCR — the chart's `appVersion` pins the default. To pull an image built from a specific commit, use `--set image.tag=sha-`. See [GHCR](https://github.com/OpenHands/agent-canvas/pkgs/container/agent-canvas) for the tag list. + +### WebSocket disconnects every minute + +Your ingress is closing idle streams. Bump the timeout annotations on the `Ingress`: + +```yaml +ingress: + annotations: + # nginx + nginx.ingress.kubernetes.io/proxy-read-timeout: "3600" + nginx.ingress.kubernetes.io/proxy-send-timeout: "3600" + # Traefik + traefik.ingress.kubernetes.io/router.middlewares: "" # keep this in mind if you also add auth middlewares +``` + +### Pod stuck in `Pending` — `no persistent volumes available` + +Either no `StorageClass` exists on the cluster, or the one you set doesn't provision on demand. Run `kubectl get storageclass` and set `persistence.storageClassName` to one that shows `VOLUMEBINDINGMODE=WaitForFirstConsumer` (Immediate is fine too). + +## Related Guides + +- [Docker Backend](/openhands/usage/agent-canvas/backend-setup/docker) — single-container equivalent for laptops and single-host VMs. +- [VM / Self-Hosted Installation](/openhands/usage/agent-canvas/backend-setup/vm) — install directly on a Linux VM without Kubernetes. +- [Connect and Manage Backends](/openhands/usage/agent-canvas/backends) — point a local Agent Canvas UI at a remote backend. diff --git a/openhands/usage/agent-canvas/backend-setup/vm.mdx b/openhands/usage/agent-canvas/backend-setup/vm.mdx index ff22fae2..51819907 100644 --- a/openhands/usage/agent-canvas/backend-setup/vm.mdx +++ b/openhands/usage/agent-canvas/backend-setup/vm.mdx @@ -336,4 +336,5 @@ Before exposing Agent Canvas beyond an SSH tunnel: - [Connect and Manage Backends](/openhands/usage/agent-canvas/backends) - [Local Backend](/openhands/usage/agent-canvas/backend-setup/local) - [Docker Backend](/openhands/usage/agent-canvas/backend-setup/docker) +- [Kubernetes (Helm)](/openhands/usage/agent-canvas/backend-setup/kubernetes) - [Cloud Backend](/openhands/usage/agent-canvas/backend-setup/cloud)