From bbc0fd4b2c47ad1805f80c444cfd24a83859d0e7 Mon Sep 17 00:00:00 2001 From: openhands Date: Wed, 8 Jul 2026 07:29:52 +0000 Subject: [PATCH] docs: replace custom sandbox image guide with warm runtime pools approach - Replaces single-image legacy approach with multi-image warm runtime pool documentation - Key benefits: no restarts/redeployments, dynamic updates, multiple images - Complete guide to authentication, configuration, and usage - Includes working shell scripts for managing warm runtime configs - Documents three methods for starting conversations (API, SDK, Web UI) - Adds image lifecycle management strategies for handling updates and resume - Covers troubleshooting for common issues (auth, image pulls, resume failures) Co-authored-by: openhands --- enterprise/custom-sandbox-image.mdx | 1299 +++++++++++++++++++++++++-- 1 file changed, 1228 insertions(+), 71 deletions(-) diff --git a/enterprise/custom-sandbox-image.mdx b/enterprise/custom-sandbox-image.mdx index 85a2da67..c6f88349 100644 --- a/enterprise/custom-sandbox-image.mdx +++ b/enterprise/custom-sandbox-image.mdx @@ -1,115 +1,1272 @@ --- -title: Custom Sandbox Images -description: Preload repos, dependencies, and tooling into a custom sandbox image to make your agents faster and more reliable. +title: Multiple Custom Sandbox Images +description: Configure multiple project-specific sandbox images using warm runtime pools for instant agent startup icon: box --- -Custom sandbox images let you prebake the repository, dependencies, compiled output, and test harness -your agents need. Instead of spending minutes provisioning a workspace on every run, your agents start -on the actual task immediately. +OpenHands agents run inside isolated sandbox containers where they execute code, interact with files, and perform tasks. Each sandbox is based on a container image that defines the runtime environment—the tools, dependencies, and configurations available to the agent. -## Why Use a Custom Image +## Why Use Multiple Custom Images -Custom images eliminate cold-start setup work (clone, install, transpile, and bootstrap) so agents -spend their time on the actual task. They also reduce setup variance and lower sandbox memory requirements -by keeping only what the agent needs. +By default, OpenHands provides a general-purpose agent sandbox image with Python, Node.js, and common development tools. However, enterprise teams often need specialized environments for different projects: -## Build Your Own Custom Image +- **PHP web applications** requiring PHP 8.4, Composer, MySQL client, and Redis +- **Java microservices** needing JDK 21, Maven, and Spring Boot tooling +- **Mobile development** requiring iOS/Android SDKs and simulators +- **Data science workloads** with specialized Python packages, R, or Julia +- **Legacy systems** with older runtime versions and specific system libraries -The [OpenHands agent-server sandbox guide](https://docs.openhands.dev/sdk/guides/agent-server/docker-sandbox) -provides full documentation on building custom sandbox images. The approach is the same for the Enterprise -Replicated VM deployment. +Custom sandbox images eliminate cold-start provisioning time. Instead of spending minutes installing dependencies every time an agent starts, you prebake everything into the image. Agents start working immediately. -### Basic Pattern +**Multiple custom images** let you maintain different environments for different project types, with each image kept ready in a warm pool for instant claim when a conversation starts. -1. Start from the OpenHands agent-server base image. -2. Keep the normal OpenHands entrypoint intact: extend the image, do not replace the entrypoint. -3. Add your repo, docs, tools, and verification wrappers. -4. Pre-run the expensive setup you do not want to repeat at task time. -5. Publish the image to a registry and point the Replicated installer at it. +### Key Advantages Over Legacy Approach - - Do not override the entrypoint or replace the runtime contract of the base image. The installer - expects standard OpenHands agent-server behavior. Only extend, do not replace. - +The warm runtime pool approach offers significant operational benefits: -### Base Image +✅ **No restarts required** - Add, update, or remove images via API without restarting OpenHands +✅ **No redeployments** - Changes take effect within ~1 minute (next reconciliation cycle) +✅ **Dynamic management** - Update image tags, pool sizes, or configurations on the fly +✅ **Multiple images** - Run different images simultaneously for different project types +✅ **Instant updates** - Change pool size from 2 to 10 pods without any downtime -```dockerfile -FROM ghcr.io/openhands/agent-server:1.23.0-python +**Previously**, changing sandbox images required: +1. Editing Helm values or Replicated Admin Console config +2. Redeploying the entire OpenHands application +3. Waiting for all services to restart (5-10 minutes) +4. Only one image could be used at a time + +**Now**, you can: +1. Call a REST API endpoint with new configuration +2. Wait ~1 minute for warm runtime reconciler to create pods +3. Start using the new image immediately +4. Maintain multiple images for different teams/projects + +## How Warm Runtime Pools Work + +OpenHands Enterprise maintains **warm runtime pools**—pre-started sandbox containers with no assigned user. When a conversation begins, instead of spinning up a new container from scratch (20+ seconds), the system claims an already-running warm container and assigns it to the conversation in under 2 seconds. + +You configure warm runtime pools via the Runtime API: + +``` +Admin → Runtime API → Warm Pool Configuration + → Creates warm pods in Kubernetes + +User starts conversation → Claims warm pod → Agent runs immediately ``` -Pin a specific version tag to ensure reproducible builds. Check -[ghcr.io/openhands/agent-server](https://github.com/OpenHands/OpenHands/pkgs/container/agent-server) -for the latest available tags. +**Key capabilities:** +- Configure multiple warm runtime pools (one per custom image) +- Set pool size per image (how many warm containers to keep ready) +- Support for private container registries +- Automatic pool maintenance (reconciliation every minute) - - To get the latest features of OpenHands Enterprise, rebuild your custom image before each upgrade. The agent server base image is updated with every OHE release. - +## Prerequisites + +### 1. Build and Publish Your Custom Image + +Your custom image must extend the OpenHands agent-server base image. Follow the existing image building guide: + + + Complete guide to building agent-server images with custom dependencies + + +**Quick checklist:** +- ✅ Start from `ghcr.io/openhands/agent-server:VERSION-python` +- ✅ Keep the OpenHands entrypoint intact (extend, don't replace) +- ✅ Install your project's dependencies and tools +- ✅ Pre-compile or transpile code if applicable +- ✅ Do NOT bake in secrets or credentials +- ✅ Push to a registry reachable from your OpenHands cluster + + + +### Security Context + +If your OpenHands deployment runs with non-root security contexts (common in enterprise K8s), ensure your Dockerfile sets appropriate ownership: + +```dockerfile +FROM ghcr.io/openhands/agent-server:1.29.0-python + +# Install dependencies as root +RUN apt-get update && apt-get install -y \ + php8.4-cli \ + php8.4-mysql \ + composer + +# Set ownership for non-root execution +RUN chown -R 10001:10001 /workspace +USER 10001 +``` + +### Platform Architecture -### Example: Build and Push +Build for `linux/amd64` if your cluster runs on x86-64 nodes: ```bash docker buildx build \ --platform linux/amd64 \ - -f your-project/Dockerfile \ - -t ghcr.io//openhands-custom-image: \ + -t your-registry.com/project/agent-image:v1.0.0 \ --push \ . ``` -Use `--platform linux/amd64` because the Enterprise Replicated VM runs on `x86-64`. +### Image Size Optimization + +Large images slow down pod startup even in warm pools (initial image pull): +- Use multi-stage builds to exclude build tools from final image +- Combine `RUN` commands to reduce layers +- Clean package manager caches (`apt-get clean`, `npm cache clean`) +- Consider squashing layers for distribution + +### Private Registries + +Configure image pull secrets in your OpenHands Helm values: + +```yaml +# For Replicated VM: configure in Admin Console under "Custom Sandbox Image" +# For K8s Helm: add to runtime-api chart values +runtime-api: + imagePullSecrets: + - name: harbor-registry-credentials +``` + +Then create the secret: + +```bash +kubectl create secret docker-registry harbor-registry-credentials \ + --docker-server=harbor.company.com \ + --docker-username=robot-account \ + --docker-password=token \ + --namespace openhands +``` + + + +### 2. Set Up Admin Authentication + +The Runtime API admin endpoints require authentication. The admin password is not set by default in Replicated VM installations. + +**Set the admin password** (one-time setup): + +```bash +# Generate a secure password +ADMIN_PASSWORD=$(openssl rand -base64 32) + +# Store it in Kubernetes +kubectl patch secret admin-password -n openhands \ + -p="{\"data\":{\"admin-password\":\"$(echo -n $ADMIN_PASSWORD | base64)\"}}" + +# Restart runtime-api to load the new password +kubectl rollout restart deployment -n openhands -l app.kubernetes.io/name=runtime-api + +# Save the password securely - you'll need it for API calls +echo "Admin password (save this in your password manager):" +echo "$ADMIN_PASSWORD" +``` + +Wait for the runtime-api pod to fully restart before proceeding: + +```bash +kubectl rollout status deployment -n openhands -l app.kubernetes.io/name=runtime-api +``` + + + The admin password is stored in the `admin-password` Kubernetes secret. Keep your saved copy secure—it grants full access to Runtime API management endpoints. + + +## Managing Warm Runtime Configurations + +All warm runtime configuration is done via REST API calls to the Runtime API. We'll provide shell scripts that handle authentication and API calls from inside your cluster. + +### Authentication Flow + +Admin endpoints require a JWT token obtained through challenge-response authentication: + +1. **Get challenge**: `GET /api/admin/challenge` returns a challenge string and salt +2. **Hash password**: Compute PBKDF2-HMAC-SHA256 with 10,000 iterations +3. **Login**: `POST /api/admin/login` with challenge and hash → returns JWT token +4. **Use token**: Include `Authorization: Bearer {token}` in subsequent requests + +JWT tokens expire after 24 hours. Re-authenticate if you get 401 errors. + + + +If you prefer to call the API directly: + +```bash +# Set your admin password +ADMIN_PASSWORD="your-password-from-setup" +RUNTIME_API_URL="https://runtime.your-domain.com" + +# Step 1: Get challenge +CHALLENGE_RESP=$(curl -s "$RUNTIME_API_URL/api/admin/challenge") +CHALLENGE=$(echo $CHALLENGE_RESP | jq -r '.challenge') +SALT=$(echo $CHALLENGE_RESP | jq -r '.salt') + +# Step 2: Compute hash (using Python) +HASH=$(python3 -c " +import hashlib, binascii +salt = '$SALT' +challenge = '$CHALLENGE' +password = '$ADMIN_PASSWORD' +combined_salt = (salt + challenge).encode('utf-8') +dk = hashlib.pbkdf2_hmac('sha256', password.encode(), combined_salt, 10000, dklen=32) +print(binascii.hexlify(dk).decode()) +") + +# Step 3: Login and get token +TOKEN=$(curl -s "$RUNTIME_API_URL/api/admin/login" \ + -H "Content-Type: application/json" \ + -d "{\"challenge\":\"$CHALLENGE\",\"hash\":\"$HASH\"}" \ + | jq -r '.token') + +# Step 4: Use token in API calls +curl "$RUNTIME_API_URL/api/warm-runtime-configs" \ + -H "Authorization: Bearer $TOKEN" +``` + + + +### Helper Script: Warm Runtime Config Manager + +Save this script as `manage-warm-runtimes.sh` to simplify warm runtime configuration: + +```bash +#!/bin/bash +# +# manage-warm-runtimes.sh +# +# Manage warm runtime configurations for custom sandbox images. +# Runs commands inside the runtime-api pod via kubectl exec. +# +# Usage: +# ./manage-warm-runtimes.sh list +# ./manage-warm-runtimes.sh add +# ./manage-warm-runtimes.sh update +# ./manage-warm-runtimes.sh delete +# +# Examples: +# ./manage-warm-runtimes.sh list +# ./manage-warm-runtimes.sh add php-symfony php-config.json +# ./manage-warm-runtimes.sh delete php-symfony +# + +set -e + +NAMESPACE="openhands" +RUNTIME_API_URL="http://localhost:5000" + +# Parse command +if [ $# -lt 1 ]; then + echo "Usage: $0 [arguments]" + echo "" + echo "Commands:" + echo " list List all warm runtime configurations" + echo " add Add a new configuration" + echo " update Update an existing configuration" + echo " delete Delete a configuration" + echo "" + echo "Config file format (JSON):" + echo ' {' + echo ' "image": "registry.com/image:tag",' + echo ' "working_dir": "/workspace",' + echo ' "command": ["/usr/local/bin/openhands-agent-server", "--port", "60000"],' + echo ' "environment": {"LOG_JSON": "1"},' + echo ' "count": 2,' + echo ' "run_as_user": 10001,' + echo ' "run_as_group": 10001,' + echo ' "fs_group": 10001,' + echo ' "fuse_s3_mount": false' + echo ' }' + exit 1 +fi + +COMMAND="$1" + +# Find runtime-api pod +echo "Finding runtime-api pod..." +POD=$(kubectl get pods -n "$NAMESPACE" -l app.kubernetes.io/name=runtime-api \ + -o jsonpath='{.items[0].metadata.name}' 2>/dev/null) + +if [ -z "$POD" ]; then + echo "Error: Could not find runtime-api pod in namespace '$NAMESPACE'" + exit 1 +fi +echo " ✓ Using pod: $POD" + +# Get admin password +echo "Retrieving admin credentials..." +ADMIN_PASSWORD=$(kubectl get secret admin-password -n "$NAMESPACE" \ + -o jsonpath='{.data.admin-password}' | base64 -d) + +if [ -z "$ADMIN_PASSWORD" ]; then + echo "Error: Admin password not set. See documentation for setup instructions." + exit 1 +fi +echo " ✓ Credentials retrieved" + +# Execute command +case "$COMMAND" in + list) + echo "Fetching warm runtime configurations..." + kubectl exec -n "$NAMESPACE" "$POD" -- python3 -c " +import json, hashlib, binascii, urllib.request, urllib.error + +ADMIN_PASSWORD = '''$ADMIN_PASSWORD''' +API_URL = '$RUNTIME_API_URL' + +def req(path, method='GET', data=None, token=None): + url = f'{API_URL}{path}' + headers = {'Content-Type': 'application/json'} + if token: + headers['Authorization'] = f'Bearer {token}' + r = urllib.request.Request(url, method=method, headers=headers) + if data: + r.data = json.dumps(data).encode('utf-8') + try: + with urllib.request.urlopen(r) as resp: + return json.loads(resp.read().decode('utf-8')) + except urllib.error.HTTPError as e: + raise Exception(f'HTTP {e.code}: {e.read().decode(\"utf-8\")}') + +# Authenticate +chal = req('/api/admin/challenge') +combined_salt = (chal['salt'] + chal['challenge']).encode('utf-8') +dk = hashlib.pbkdf2_hmac('sha256', ADMIN_PASSWORD.encode(), combined_salt, 10000, dklen=32) +token = req('/api/admin/login', 'POST', { + 'challenge': chal['challenge'], + 'hash': binascii.hexlify(dk).decode() +})['token'] + +# List configurations +configs = req('/api/warm-runtime-configs', token=token)['configs'] +if not configs: + print('No warm runtime configurations found.') +else: + print(f'Found {len(configs)} configuration(s):\\n') + for cfg in configs: + print(f\" Name: {cfg['name']}\") + print(f\" Image: {cfg['image']}\") + print(f\" Count: {cfg.get('count', 'default')}\") + print(f\" Fuse mount: {cfg.get('fuse_s3_mount', False)}\") + print() +" + ;; + + add|update) + if [ $# -ne 3 ]; then + echo "Usage: $0 $COMMAND " + exit 1 + fi + CONFIG_NAME="$2" + CONFIG_FILE="$3" + + if [ ! -f "$CONFIG_FILE" ]; then + echo "Error: Config file not found: $CONFIG_FILE" + exit 1 + fi + + # Validate JSON + if ! jq empty "$CONFIG_FILE" 2>/dev/null; then + echo "Error: Invalid JSON in $CONFIG_FILE" + exit 1 + fi + + CONFIG_JSON=$(cat "$CONFIG_FILE") + ACTION="Adding" + if [ "$COMMAND" = "update" ]; then + ACTION="Updating" + fi + + echo "$ACTION warm runtime configuration '$CONFIG_NAME'..." + kubectl exec -i -n "$NAMESPACE" "$POD" -- python3 -c " +import json, hashlib, binascii, urllib.request, urllib.error, sys + +ADMIN_PASSWORD = '''$ADMIN_PASSWORD''' +API_URL = '$RUNTIME_API_URL' +CONFIG_NAME = '$CONFIG_NAME' +CONFIG_DATA = json.loads('''$CONFIG_JSON''') + +def req(path, method='GET', data=None, token=None): + url = f'{API_URL}{path}' + headers = {'Content-Type': 'application/json'} + if token: + headers['Authorization'] = f'Bearer {token}' + r = urllib.request.Request(url, method=method, headers=headers) + if data: + r.data = json.dumps(data).encode('utf-8') + try: + with urllib.request.urlopen(r) as resp: + return json.loads(resp.read().decode('utf-8')) + except urllib.error.HTTPError as e: + raise Exception(f'HTTP {e.code}: {e.read().decode(\"utf-8\")}') + +# Authenticate +chal = req('/api/admin/challenge') +combined_salt = (chal['salt'] + chal['challenge']).encode('utf-8') +dk = hashlib.pbkdf2_hmac('sha256', ADMIN_PASSWORD.encode(), combined_salt, 10000, dklen=32) +token = req('/api/admin/login', 'POST', { + 'challenge': chal['challenge'], + 'hash': binascii.hexlify(dk).decode() +})['token'] + +# Save configuration +result = req(f'/api/admin/warm-runtime-configs/{CONFIG_NAME}', 'PUT', CONFIG_DATA, token) +print(f\"✓ Configuration '{CONFIG_NAME}' saved successfully\") +print(f\" Image: {result['image']}\") +print(f\" Pool size: {result.get('count', 'default')}\") +" + ;; + + delete) + if [ $# -ne 2 ]; then + echo "Usage: $0 delete " + exit 1 + fi + CONFIG_NAME="$2" + + echo "Deleting warm runtime configuration '$CONFIG_NAME'..." + kubectl exec -n "$NAMESPACE" "$POD" -- python3 -c " +import json, hashlib, binascii, urllib.request, urllib.error + +ADMIN_PASSWORD = '''$ADMIN_PASSWORD''' +API_URL = '$RUNTIME_API_URL' +CONFIG_NAME = '$CONFIG_NAME' + +def req(path, method='GET', data=None, token=None): + url = f'{API_URL}{path}' + headers = {'Content-Type': 'application/json'} + if token: + headers['Authorization'] = f'Bearer {token}' + r = urllib.request.Request(url, method=method, headers=headers) + if data: + r.data = json.dumps(data).encode('utf-8') + try: + with urllib.request.urlopen(r) as resp: + if resp.status == 204: + return None + return json.loads(resp.read().decode('utf-8')) + except urllib.error.HTTPError as e: + raise Exception(f'HTTP {e.code}: {e.read().decode(\"utf-8\")}') + +# Authenticate +chal = req('/api/admin/challenge') +combined_salt = (chal['salt'] + chal['challenge']).encode('utf-8') +dk = hashlib.pbkdf2_hmac('sha256', ADMIN_PASSWORD.encode(), combined_salt, 10000, dklen=32) +token = req('/api/admin/login', 'POST', { + 'challenge': chal['challenge'], + 'hash': binascii.hexlify(dk).decode() +})['token'] + +# Delete configuration +req(f'/api/admin/warm-runtime-configs/{CONFIG_NAME}', 'DELETE', token=token) +print(f\"✓ Configuration '{CONFIG_NAME}' deleted successfully\") +" + ;; + + *) + echo "Error: Unknown command: $COMMAND" + echo "Valid commands: list, add, update, delete" + exit 1 + ;; +esac +``` + +Make the script executable: + +```bash +chmod +x manage-warm-runtimes.sh +``` + +### Configuration Format + +Each warm runtime configuration requires: + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `image` | string | ✅ | Full image reference (e.g., `harbor.company.com/project/agent:v1.0`) | +| `working_dir` | string | ✅ | Working directory inside container (typically `/workspace`) | +| `command` | array | ✅ | Entrypoint command (typically `["/usr/local/bin/openhands-agent-server", "--port", "60000"]`) | +| `environment` | object | ✅ | Environment variables as key-value pairs | +| `count` | integer | ⬜ | Number of warm pods to maintain (default: global pool size) | +| `run_as_user` | integer | ⬜ | User ID for pod security context | +| `run_as_group` | integer | ⬜ | Group ID for pod security context | +| `fs_group` | integer | ⬜ | Filesystem group ID for pod security context | +| `fuse_s3_mount` | boolean | ⬜ | Use S3-backed workspace (requires fusey setup, default: `false`) | + +**Example configuration file** (`php-symfony.json`): + +```json +{ + "image": "harbor.company.com/web-team/php-symfony:8.4", + "working_dir": "/workspace", + "command": ["/usr/local/bin/openhands-agent-server", "--port", "60000"], + "environment": { + "LOG_JSON": "1", + "OH_ENABLE_VNC": "0", + "OH_CONVERSATIONS_PATH": "/workspace/conversations", + "OH_BASH_EVENTS_DIR": "/workspace/bash_events", + "OPENVSCODE_SERVER_ROOT": "/openhands/.openvscode-server" + }, + "count": 3, + "run_as_user": 10001, + "run_as_group": 10001, + "fs_group": 10001, + "fuse_s3_mount": false +} +``` + + + **Environment variables**: Include the standard OpenHands environment variables shown above. The Runtime API may inject additional variables at runtime (e.g., webhook URLs, CORS origins). + + +### Common Operations + +**List all warm runtime configurations:** + +```bash +./manage-warm-runtimes.sh list +``` + +**Add a new configuration:** -### What to Bake In +```bash +# Create config file +cat > php-symfony.json < php-symfony-updated.json -Good candidates for prebaking: +# Update +./manage-warm-runtimes.sh update php-symfony php-symfony-updated.json +``` -- Pinned repository checkouts -- Package manager caches and installed dependencies (`node_modules`, Python virtualenvs, etc.) -- Compiled or transpiled output -- Native system packages (`xvfb`, `libkrb5-dev`, `pkg-config`, etc.) -- Browser or Electron artifacts -- Stable helper scripts such as `prepare-*` and `*-verify` wrappers +**Delete a configuration:** -### What to Keep Out +```bash +./manage-warm-runtimes.sh delete php-symfony +``` - Do not bake the following into your image: + Deleting a configuration removes it from the database but does not immediately stop running warm pods. The warm runtime reconciler will cull excess pods on its next cycle (runs every ~1 minute). + + +## Verifying and Monitoring + +### Check Configuration Status + +After adding or updating a configuration, verify it was saved: + +```bash +./manage-warm-runtimes.sh list +``` + +### Monitor Warm Pool Creation - - Secrets, API keys, or personal credentials - - Machine-specific paths or environment assumptions - - Uncommitted source changes or task-specific fixes - - Rapidly changing dependencies (use a lightweight `prepare-*` helper instead) +The warm runtime reconciler runs every ~1 minute and creates/culls pods to match configured pool sizes. + +**Watch pods being created:** + +```bash +kubectl get pods -n openhands -l openhands.all-hands.dev/warm-runtime=true -w +``` + +**Check pod status:** + +```bash +kubectl get pods -n openhands -l openhands.all-hands.dev/warm-runtime=true -o wide +``` + +You should see pods with names like `warm-runtime--` in `Running` state. + +### Verify Image Pull + +If your custom image is in a private registry, check that pods pulled it successfully: + +```bash +kubectl describe pod -n openhands | grep -A 10 "Events:" +``` + +Look for successful `Pulled` events. If you see `ImagePullBackOff` or `ErrImagePull`, check your image pull secret configuration. + +### Check Warm Runtime Logs + +View logs from a warm runtime pod: + +```bash +# Find a warm pod +POD=$(kubectl get pods -n openhands -l openhands.all-hands.dev/warm-runtime=true \ + -o jsonpath='{.items[0].metadata.name}') + +# View logs +kubectl logs -n openhands $POD +``` + +Dormant warm runtimes should log something like: +``` +Deferred init mode enabled (OH_DEFERRED_INIT=true) +Waiting for POST /api/init to activate... +``` + +## Starting Conversations with Custom Images + +Once you've configured warm runtime pools, you can start using them. This section explains the complete flow from configuration to running conversations. + +### Understanding the Runtime Claim Flow + +When a conversation starts, here's what happens: + +1. **Client sends request** to OpenHands server (web UI, API, SDK) +2. **OpenHands server** forwards `POST /start` to Runtime API with: + - `image`: Which sandbox image to use + - `command`, `working_dir`, `environment`: Runtime parameters + - `session_id`: Unique conversation identifier + +3. **Runtime API** looks for an available warm runtime where ALL of these match: + - ✅ `image` matches exactly + - ✅ `command` matches exactly + - ✅ `working_dir` matches exactly + - ✅ `environment` matches (after key normalization) + - ✅ Pod is unclaimed (`api_key_name IS NULL`) + +4. **If match found**: Runtime is claimed and assigned to the conversation (<2 seconds) +5. **If no match**: New pod is cold-started from the image (20+ seconds) + + + **Exact matching required**: Even small differences (extra environment variable, different command order) prevent warm runtime matching and cause cold starts. -If the repository or dependencies change frequently, include a `prepare-*` script in the image -so the agent can refresh only the parts that need updating without a full rebuild. +### Method 1: Direct Runtime API Calls + +If you're building custom integrations or automation, call the Runtime API directly. + +#### Step 1: Get Your API Key + +The Runtime API requires an API key for authentication. For internal OpenHands server usage, this is typically pre-configured. For external integrations: + +```bash +# Retrieve the default API key from Kubernetes +kubectl get secret default-api-key -n openhands -o jsonpath='{.data.key}' | base64 -d +``` + +Or create a dedicated API key using admin endpoints (see [API Reference](#api-reference)). + +#### Step 2: Start a Runtime + +Call `POST /start` with parameters matching your warm runtime configuration: + +```bash +# Set variables +RUNTIME_API_URL="https://runtime.your-domain.com" +API_KEY="your-api-key" # From step 1 + +# Start runtime with custom PHP image +curl -X POST "$RUNTIME_API_URL/start" \ + -H "X-API-Key: $API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "image": "harbor.company.com/web-team/php-symfony:8.4", + "command": ["/usr/local/bin/openhands-agent-server", "--port", "60000"], + "working_dir": "/workspace", + "environment": { + "LOG_JSON": "1", + "OH_ENABLE_VNC": "0", + "OH_CONVERSATIONS_PATH": "/workspace/conversations", + "OH_BASH_EVENTS_DIR": "/workspace/bash_events", + "OPENVSCODE_SERVER_ROOT": "/openhands/.openvscode-server" + }, + "session_id": "php-project-session-123", + "run_as_user": 10001, + "run_as_group": 10001, + "fs_group": 10001 + }' +``` + +**Response** (warm runtime claimed): +```json +{ + "runtime_id": "abc123", + "session_id": "php-project-session-123", + "status": "RUNNING", + "url": "https://abc123.runtime.your-domain.com", + "image": "harbor.company.com/web-team/php-symfony:8.4", + "created_at": "2026-07-08T10:30:45Z" +} +``` + + + **Warm vs Cold Start**: If the response comes back in under 2-3 seconds, you claimed a warm runtime. If it takes 20+ seconds, the runtime cold-started. Check the `created_at` timestamp. + + +#### Step 3: Verify Runtime is Using Your Image + +```bash +# Get runtime details +RUNTIME_ID="abc123" # From step 2 response + +curl "$RUNTIME_API_URL/runtime/$RUNTIME_ID" \ + -H "X-API-Key: $API_KEY" +``` + +Response shows the image being used: +```json +{ + "runtime_id": "abc123", + "status": "RUNNING", + "image": "harbor.company.com/web-team/php-symfony:8.4", + ... +} +``` + +### Method 2: Via OpenHands SDK + +If you're building applications using the OpenHands SDK: + +```python +from openhands_sdk import OpenHands + +# Initialize client +client = OpenHands( + runtime_api_url="https://runtime.your-domain.com", + api_key="your-api-key" +) + +# Start runtime with custom image +# IMPORTANT: Parameters must match your warm runtime configuration exactly +runtime = client.runtime.start( + image="harbor.company.com/web-team/php-symfony:8.4", + command=["/usr/local/bin/openhands-agent-server", "--port", "60000"], + working_dir="/workspace", + environment={ + "LOG_JSON": "1", + "OH_ENABLE_VNC": "0", + "OH_CONVERSATIONS_PATH": "/workspace/conversations", + "OH_BASH_EVENTS_DIR": "/workspace/bash_events", + "OPENVSCODE_SERVER_ROOT": "/openhands/.openvscode-server" + }, + session_id="php-project-session-456", + run_as_user=10001, + run_as_group=10001, + fs_group=10001 +) + +print(f"Runtime started: {runtime.runtime_id}") +print(f"Using image: {runtime.image}") +print(f"Status: {runtime.status}") + +# Use the runtime for agent operations +conversation = client.conversation.create(runtime_id=runtime.runtime_id) +``` + +### Method 3: Via OpenHands Web UI -## Configure the Replicated VM Installer +The OpenHands web UI currently uses a default sandbox image for all conversations. To use custom images through the UI, you have several options: -Once your image is built and pushed to a registry, point the Replicated Admin Console at it. +#### Option A: Change the Global Default Image -1. Open the **Admin Console** at `https://:30000`. -2. Navigate to **Config** and find the **Sandbox Image** section. -3. Set the following fields: +Configure OpenHands server to use your custom image as the default. This affects **all** web UI conversations. -| Field | Value | -|---|---| -| **Use a Custom Sandbox Image** | Enabled | -| **Sandbox Image Repository** | Your image repository (e.g. `ghcr.io/your-org/openhands-custom-image`) | -| **Sandbox Image Tag** | Your image tag (e.g. `v1.2.0`) | -| **Registry Server** | If your registry requires authentication | -| **Registry Username** | If your registry requires authentication | -| **Registry Password or Credentials** | If your registry requires authentication | +**For Replicated VM installations:** +1. Open Admin Console: `https://:30000` +2. Navigate to **Config** → **Sandbox Image** +3. Enable **Use a Custom Sandbox Image** +4. Set: + - Repository: `harbor.company.com/web-team/php-symfony` + - Tag: `8.4` + - Registry credentials (if needed) +5. Save and deploy -4. Click **Save config** and then **Deploy** to apply the change. +**For Kubernetes Helm installations:** + +Update your `values.yaml`: +```yaml +openhands: + sandbox: + defaultImage: "harbor.company.com/web-team/php-symfony:8.4" +``` + +Then upgrade: +```bash +helm upgrade openhands ./charts/openhands -n openhands -f values.yaml +``` - This setting applies to the **sandbox / agent-server image** only (the image that runs inside each - agent's isolated workspace). It does not replace the other OpenHands service images. + This approach uses a **single** default image for all web UI users. If you need different images per project, see Options B or C below. -## Reference +#### Option B: Per-User or Per-Project Configuration (Requires Code Changes) + +For enterprise deployments where different teams need different images, you can extend the OpenHands server to: + +1. **Map users/teams to images**: Store user→image mappings in database +2. **Detect from repository**: Parse project files (e.g., `.openhands.yaml`) to determine required image +3. **UI selector**: Add a dropdown in the conversation start UI for users to pick an image + +This requires custom development. Example flow: + +```python +# In your OpenHands server code +def determine_sandbox_image(user, repository_url): + """Determine which sandbox image to use.""" + + # Option 1: User/team mapping + if user.team == "web-team": + return "harbor.company.com/web-team/php-symfony:8.4" + elif user.team == "mobile-team": + return "harbor.company.com/mobile-team/ios-android:latest" + + # Option 2: Repository detection + if repository_url and "symfony" in repository_url.lower(): + return "harbor.company.com/web-team/php-symfony:8.4" + + # Fallback to default + return "ghcr.io/openhands/agent-server:1.29.0-python" + +# When starting conversation, pass to Runtime API +runtime = runtime_api_client.start( + image=determine_sandbox_image(current_user, repo_url), + command=[...], + ... +) +``` + +#### Option C: Use the REST API Directly + +Power users can bypass the web UI and call the OpenHands server API directly: + +```bash +# Start conversation with specific sandbox image +curl -X POST "https://your-domain.com/api/v1/app-conversations" \ + -H "Authorization: Bearer $USER_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "github_repo": "https://github.com/company/php-project", + "agent_class": "CodeActAgent", + "language": "en", + "sandbox_config": { + "image": "harbor.company.com/web-team/php-symfony:8.4", + "command": ["/usr/local/bin/openhands-agent-server", "--port", "60000"], + "working_dir": "/workspace", + "environment": { + "LOG_JSON": "1", + "OH_ENABLE_VNC": "0" + }, + "run_as_user": 10001, + "run_as_group": 10001, + "fs_group": 10001 + } + }' +``` + + + The exact API endpoint and parameter names vary by OpenHands version. Check your OpenHands server's OpenAPI documentation at `https://your-domain.com/openapi.json` for current API schema. + + +### Verifying Warm Runtime Usage + +After starting a conversation, verify it used a warm runtime (vs cold start): + +**Method 1: Check response time** +- ✅ Warm runtime claim: 1-3 seconds +- ❌ Cold start: 20-30+ seconds + +**Method 2: Check Runtime API logs** + +```bash +kubectl logs -n openhands -l app.kubernetes.io/name=runtime-api --tail=100 | grep "claim" +``` + +Look for log lines like: +``` +Claimed warm runtime for session +``` + +**Method 3: Query running runtimes** + +```bash +# Get all current runtimes +curl "$RUNTIME_API_URL/list" -H "X-API-Key: $API_KEY" + +# Check which image a specific runtime is using +curl "$RUNTIME_API_URL/runtime/$RUNTIME_ID" -H "X-API-Key: $API_KEY" | jq '.image' +``` + +### Why Warm Runtime Might Not Be Claimed + +If conversations are cold-starting instead of claiming warm runtimes: + +1. **Parameter mismatch**: Request parameters don't exactly match warm config + - Compare request JSON to `./manage-warm-runtimes.sh list` output + - Check for extra/missing environment variables + - Verify command array order and format + +2. **No available warm pods**: All warm pods are already claimed + - Check pool size: `./manage-warm-runtimes.sh list` + - Increase `count` if needed + - Check pod status: `kubectl get pods -n openhands -l openhands.all-hands.dev/warm-runtime=true` + +3. **Pods not ready yet**: Warm pods still starting + - Wait 1-2 minutes after adding configuration + - Check pod logs: `kubectl logs -n openhands ` + +4. **Image pull failed**: Pods stuck pulling image + - Check events: `kubectl describe pod -n openhands ` + - Verify registry credentials + +## Troubleshooting + +### "Admin functionality is disabled" Error + +**Cause**: The `ADMIN_PASSWORD` environment variable is not set in the Runtime API deployment. + +**Solution**: Follow the [Set Up Admin Authentication](#2-set-up-admin-authentication) steps to configure the admin password. + +### Authentication Failures (401 Unauthorized) + +**Cause**: JWT token expired (tokens last 24 hours) or incorrect password. + +**Solution**: +- Verify your saved admin password is correct +- Re-authenticate to get a new token +- Check runtime-api logs for authentication errors + +### Configuration Added But No Pods Created + +**Possible causes:** +1. **Image doesn't exist or can't be pulled**: Check image name and registry credentials +2. **Warm runtime reconciler not running**: Check runtime-api logs +3. **Resource constraints**: Check cluster capacity + +**Debug steps:** + +```bash +# Check runtime-api logs +kubectl logs -n openhands -l app.kubernetes.io/name=runtime-api | grep -i "warm" + +# Check pending pods +kubectl get pods -n openhands -l openhands.all-hands.dev/warm-runtime=true + +# Describe a pending pod to see why it's not starting +kubectl describe pod -n openhands +``` + +### Conversation Uses Wrong Image (Cold Start Instead of Warm Pool) + +**Cause**: Request parameters don't exactly match warm runtime configuration. + +**Solution**: +1. List your warm runtime config: `./manage-warm-runtimes.sh list` +2. Compare request parameters (image, command, working_dir, environment) to config +3. Ensure exact match—even whitespace or key ordering differences prevent matching + +### Image Pull Errors (ImagePullBackOff) + +**Cause**: Private registry credentials not configured or incorrect. + +**Solution**: + +```bash +# Verify image pull secret exists +kubectl get secret -n openhands | grep registry + +# Check secret contents +kubectl get secret -n openhands -o yaml + +# Recreate if needed +kubectl delete secret -n openhands +kubectl create secret docker-registry \ + --docker-server=registry.company.com \ + --docker-username=username \ + --docker-password=password \ + --namespace openhands + +# Verify runtime-api deployment references the secret +kubectl get deployment runtime-api -n openhands -o yaml | grep imagePullSecrets +``` + +### Pods Stuck in Pending State + +**Cause**: Insufficient cluster resources (CPU, memory, storage). + +**Solution**: Check node resources and pod resource requests: + +```bash +# Check node capacity +kubectl describe nodes | grep -A 5 "Allocated resources" + +# Check pod resource requests +kubectl get pods -n openhands -l openhands.all-hands.dev/warm-runtime=true -o yaml | grep -A 3 "resources:" +``` + +## API Reference + +### Warm Runtime Configuration Endpoints + +All endpoints require admin JWT authentication (see [Authentication Flow](#authentication-flow)). + +#### List Configurations + +```http +GET /api/warm-runtime-configs +Authorization: Bearer {admin-jwt} +``` + +**Response** (200 OK): +```json +{ + "configs": [ + { + "name": "php-symfony", + "image": "harbor.company.com/web-team/php-symfony:8.4", + "working_dir": "/workspace", + "command": ["/usr/local/bin/openhands-agent-server", "--port", "60000"], + "environment": {"LOG_JSON": "1"}, + "count": 3, + "run_as_user": 10001, + "run_as_group": 10001, + "fs_group": 10001, + "fuse_s3_mount": false + } + ] +} +``` + +#### Create or Update Configuration + +```http +PUT /api/admin/warm-runtime-configs/{config-name} +Authorization: Bearer {admin-jwt} +Content-Type: application/json + +{ + "image": "string", + "working_dir": "string", + "command": ["string"], + "environment": {"key": "value"}, + "count": 2, + "run_as_user": 10001, + "run_as_group": 10001, + "fs_group": 10001, + "fuse_s3_mount": false +} +``` + +**Response** (200 OK): Returns the saved configuration. + +#### Delete Configuration + +```http +DELETE /api/admin/warm-runtime-configs/{config-name} +Authorization: Bearer {admin-jwt} +``` + +**Response** (204 No Content): Configuration deleted successfully. + +## Image Lifecycle Management + +When managing custom images over time, you need to consider how image updates affect paused conversations that may need to resume later. + +### How Resume Works + +When a conversation is paused and later resumed, the behavior depends on your storage configuration: + +#### S3-Backed Runtimes (Fuse Mount) + +**What happens:** +- Resume claims a new warm pod matching the **original image tag** used when the conversation started +- If no warm pod matches, a fresh pod is created with the original image tag +- Workspace data is restored from S3 + +**Implications:** +- Old image tags must remain available in your registry +- Updating warm pool config to new image tag does NOT affect resumed conversations +- If original image tag is deleted from registry → resume fails with `ImagePullBackOff` + +#### Traditional PVC-Backed Runtimes + +**What happens:** +- Resume scales the existing deployment back to 1 replica +- Before resuming, checks for a warm config named `"current"` and updates deployment to that image +- Workspace data persists in PVC attached to deployment + +**Implications:** +- Resumed conversations automatically get updated to "current" image if configured +- Image updates could introduce compatibility issues with existing workspace state +- Less common in enterprise deployments (S3-backed is preferred) + +### Image Update Strategies + +#### Strategy 1: Keep Old Tags Available (Recommended for S3-backed) + +When you update to a new image version, keep old tags in your registry: + +```bash +# Build and push new version +docker tag harbor.company.com/web-team/php-symfony:8.4 \ + harbor.company.com/web-team/php-symfony:8.4-20260708 + +docker push harbor.company.com/web-team/php-symfony:8.4-20260708 +docker push harbor.company.com/web-team/php-symfony:8.4 + +# Update warm pool config to new tag +jq '.image = "harbor.company.com/web-team/php-symfony:8.4-20260708"' \ + php-config.json > php-config-updated.json +./manage-warm-runtimes.sh update php-symfony php-config-updated.json +``` + +**Advantages:** +- ✅ Old conversations can always resume +- ✅ Clear version history +- ✅ Can rollback if new version has issues + +**Disadvantages:** +- ❌ Registry storage grows over time +- ❌ Need cleanup process for very old tags + +**Cleanup guidance:** +- Check Runtime API for active/paused runtimes using old tags +- Only delete tags with no active or paused runtimes +- Set registry retention policies (e.g., keep last 10 tags) + +```bash +# Check which images are currently in use +curl "$RUNTIME_API_URL/list?show_all=true" -H "X-API-Key: $API_KEY" \ + | jq -r '.[] | .image' | sort | uniq -c + +# Delete old tags only after confirming no usage +``` + +#### Strategy 2: Force Fresh Start (For Breaking Changes) + +If a new image version is incompatible with old workspace state, inform users to start fresh conversations: + +```bash +# Update warm pool to new incompatible version +jq '.image = "harbor.company.com/web-team/php-symfony:9.0"' \ + php-config.json > php-config-v9.json +./manage-warm-runtimes.sh update php-symfony php-config-v9.json + +# Keep old version available temporarily under different config name +jq '.image = "harbor.company.com/web-team/php-symfony:8.4"' \ + php-config.json > php-config-v8-legacy.json +./manage-warm-runtimes.sh add php-symfony-v8-legacy php-config-v8-legacy.json +``` + +**Communication to users:** +- Announce the breaking change in advance +- Ask users to complete or stop ongoing conversations +- After migration window, stop and remove paused runtimes on old version + +```bash +# List paused runtimes (these need attention) +curl "$RUNTIME_API_URL/list?show_all=true" -H "X-API-Key: $API_KEY" \ + | jq '.[] | select(.status == "PAUSED")' + +# After grace period, stop old runtimes +# (Users can't resume anyway if you've deleted the old image) +``` + +#### Strategy 3: Gradual Rollout + +Test new images with a small pool before full replacement: + +```bash +# Add new version alongside existing +./manage-warm-runtimes.sh add php-symfony-v2 php-config-v2.json + +# Gradually shift load by adjusting pool sizes +jq '.count = 1' php-config-v1.json | ./manage-warm-runtimes.sh update php-symfony - +jq '.count = 5' php-config-v2.json | ./manage-warm-runtimes.sh update php-symfony-v2 - + +# Monitor for issues, then complete migration +./manage-warm-runtimes.sh delete php-symfony # Remove v1 +``` + +### Best Practices + +1. **Use semantic versioning** in image tags (e.g., `php-symfony:8.4.1`, not `latest`) +2. **Keep previous 3-5 versions** available for resume +3. **Set registry retention policies** to auto-cleanup very old tags +4. **Monitor paused runtimes** before deleting old images +5. **Document breaking changes** and migration paths for users +6. **Test resume** after image updates to verify compatibility + +### What Happens When Image is Missing + +If a runtime tries to resume but the image tag no longer exists: + +**Symptoms:** +- Resume request times out or returns error after 2+ minutes +- Runtime status stuck in `STARTING` or moves to `ERROR` +- Pod events show `ImagePullBackOff` or `ErrImagePull` + +**Check pod status:** +```bash +# Find the pod +kubectl get pods -n openhands | grep runtime- + +# Check events +kubectl describe pod -n openhands runtime- | grep -A 10 Events +``` + +**Resolution:** +1. Push the missing image tag back to registry, OR +2. Stop the runtime and ask user to start a new conversation + +```bash +# Stop failed runtime +curl -X POST "$RUNTIME_API_URL/stop" \ + -H "X-API-Key: $API_KEY" \ + -d '{"runtime_id": "abc123"}' +``` + +## Migration from Legacy Approach + +If you're currently using the Replicated Admin Console "Custom Sandbox Image" settings (single global image), you can migrate to warm runtime pools: + +1. **Note your current image**: Check Admin Console → Config → Sandbox Image +2. **Create warm runtime config** for your current image using the steps above +3. **(Optional)** Disable the Admin Console setting after verifying warm pools work +4. **Add additional images** for different project types as needed + +The new approach doesn't interfere with the old approach—they can coexist during migration. + +## Related Documentation -- [OpenHands custom image example repo](https://github.com/OpenHands/openhands-custom-image): Dockerfile, benchmark scripts, and analysis tooling for the VS Code custom image example. -- [Agent-server sandbox guide](https://docs.openhands.dev/sdk/guides/agent-server/docker-sandbox): full SDK documentation on building and configuring custom sandbox images. + + + Complete guide to building agent-server images + + + Kubernetes installation overview + + + Configure sandbox resource limits + + + Deep dive into warm pool lifecycle + +