Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions services/cloud-agent-next/.dev.vars.example
Original file line number Diff line number Diff line change
Expand Up @@ -40,8 +40,8 @@ WORKER_URL=http://host.docker.internal:8794
CLI_TIMEOUT_SECONDS=900
REAPER_INTERVAL_MS=300000

# Per-session sandbox org IDs (optional)
# Comma-separated list of org IDs that use per-session sandboxes, or `*` for all orgs
# Per-session Cloudflare sandbox org IDs (optional)
# Comma-separated list of org IDs that use per-session Cloudflare sandboxes, or `*` for all orgs
PER_SESSION_SANDBOX_ORG_IDS=

# Credential containment org IDs (optional)
Expand Down
1 change: 1 addition & 0 deletions services/cloud-agent-next/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ Git tokens (GitHub App installation tokens, managed GitLab tokens) are resolved
- `services/cloud-agent-next/test/e2e/README.md` is the source of truth for setup, fake-LLM routing, lifecycle directives, and troubleshooting.
- Prefer focused scenario debugging first: `pnpm exec tsx services/cloud-agent-next/test/e2e/run.ts <lifecycle> <conversation>`.
- Run the aggregate local regression matrix with `pnpm exec tsx services/cloud-agent-next/test/e2e/smoke.ts` when validating the full real Worker + DO + sandbox + wrapper path.
- Non-zero `portOffset` sessions (the common case) require `WORKER_URL`/`FAKE_LLM_URL` env overrides on every driver invocation — see README "Running". Discover the offset with `pnpm dev:status --json`.
- This harness is local/manual rather than part of normal `pnpm test` or CI; use `dev/logs/cloud-agent-next.log` and `dev/logs/fake-llm.log` when debugging it.
- Read `services/cloud-agent-next/DEBUG.md` when correlating local Wrangler logs with Docker sandbox containers, wrapper log files, Kilo CLI logs, uploaded archives, or stuck session flows.

Expand Down
22 changes: 22 additions & 0 deletions services/cloud-agent-next/DEBUG.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,28 @@ The internal `getWrapperLogs` path also discovers these sandbox-side files direc

- Worker queueing succeeds, but no wrapper logs appear:
- inspect pending flush scheduling, sandbox creation, and wrapper startup logs in `dev/logs/cloud-agent-next.log`.
- Worker log shows `Failed to issue Kilo session capability` / `Worker "git-token-service-dev" not found`:
- the `GIT_TOKEN_SERVICE` service binding did not resolve. This fails before
any wrapper or sandbox work, so no wrapper log or fake-LLM traffic exists.
Check `.wrangler/dev-registry/` for a missing `git-token-service-dev`
entry; if absent, `pnpm dev:restart cloudflare-git-token-service` and
confirm it reappears before retrying the turn.
- Turn stalls in `preparing` (repeats several times, no kilo events, no
fake-LLM traffic):
- the wrapper inside the sandbox is failing to start. Look for
`Reconciling physical wrapper stop … reason: 'startup-failed'`,
`ContainerControlConnection upgrade returned retryable status` (503),
or `Container is not listening to port …` in
`dev/logs/cloud-agent-next.log`, then read the wrapper log inside the
sandbox (`/tmp/kilocode-wrapper-*.log`) for bootstrap/import detail. The
503 from the container control connection means the Docker container's
control plane isn't ready — this is an environmental issue under Docker
Desktop load, not a code regression. If the failure is transient, suspect
Docker contention: leftover stopped containers, a competing dev session
also running Cloud Agent sandboxes, or stale DO alarm timers from
previous sessions. Confirm the fake LLM was never reached with
`curl -s $FAKE_LLM_URL/test/requests` — a flat `chatCompletions` count
proves the stall is upstream of kilo inference.
- Wrapper log reaches bootstrap/import, then repeats:
- inspect import metadata, import exit code, and post-import `getSession()` lookup.
- Wrapper ingest connects, but UI stays stale:
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
import { describe, expect, it, vi } from 'vitest';

vi.mock('@cloudflare/sandbox', () => ({ getSandbox: vi.fn() }));

import type { SessionMetadata } from '../persistence/session-metadata.js';
import type { Env, SandboxInstance } from '../types.js';
import { CloudflareAgentSandbox } from './cloudflare/cloudflare-agent-sandbox.js';
import type { AgentSandbox } from './protocol.js';

/**
* Shared behavioral contract every AgentSandbox adapter must satisfy for a
* session with no running wrapper. Provider-specific flows (ensureWrapper,
* workspace preparation, reconciliation) stay in the per-provider suites.
*/

function sessionMetadata(provider: 'cloudflare'): SessionMetadata {
return {
metadataSchemaVersion: 2,
identity: { sessionId: 'agent_contract', userId: 'user_contract' },
auth: {},
workspace: { sandboxId: 'ses-abcdef', sandboxProvider: provider },
lifecycle: { version: 1, timestamp: 1 },
} satisfies SessionMetadata;
}

function idleCloudflareSandbox(): AgentSandbox {
const sandbox = {
listProcesses: vi.fn().mockResolvedValue([]),
destroy: vi.fn().mockResolvedValue(undefined),
renewActivityTimeout: vi.fn(),
} as unknown as SandboxInstance;
return new CloudflareAgentSandbox({} as Env, sessionMetadata('cloudflare'), {
resolveSandbox: () => sandbox,
sleep: () => Promise.resolve(),
stopObservationDelaysMs: [0],
});
}

describe.each([['cloudflare', idleCloudflareSandbox]] as const)(
'AgentSandbox contract (%s)',
(_provider, createIdleSandbox) => {
it('reports absent wrappers when none are running', async () => {
await expect(createIdleSandbox().discoverSessionWrappers()).resolves.toEqual({
status: 'absent',
});
});

it('treats stopping wrappers as settled when none are running', async () => {
const result = await createIdleSandbox().stopWrappers({
target: { kind: 'session' },
attemptId: 'attempt-contract',
reason: 'session-delete',
});

expect(result.status).toBe('absent');
});

it('scopes instance-targeted stops to the leased instance', async () => {
const result = await createIdleSandbox().stopWrappers({
target: {
kind: 'instance',
instance: { instanceId: 'instance-other', instanceGeneration: 7 },
},
attemptId: 'attempt-contract',
reason: 'startup-failed',
});

expect(result.status).toBe('absent');
});

it('returns no running wrapper when none exists', async () => {
await expect(createIdleSandbox().getRunningWrapper()).resolves.toBeNull();
});

it('resolves recovery deletion without throwing', async () => {
await expect(createIdleSandbox().delete('recovery')).resolves.toBeUndefined();
});

it('keeps the sandbox alive without a running wrapper', async () => {
await expect(createIdleSandbox().keepAlive()).resolves.toBeUndefined();
});

it('reports terminal availability as a declared capability outcome', async () => {
const result = await createIdleSandbox().getRunningTerminalClient();

expect(['ready', 'not-running', 'unhealthy', 'capability-unavailable']).toContain(
result.status
);
});
}
);
14 changes: 14 additions & 0 deletions services/cloud-agent-next/src/agent-sandbox/capabilities.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
import type { AgentSandboxProvider } from '../types.js';

export type ProviderCapabilities = {
terminal: boolean;
devcontainer: boolean;
};

/**
* Static capability matrix per sandbox provider. Metadata validation and
* feature gates read this table instead of hard-coding provider names.
*/
export const PROVIDER_CAPABILITIES: Record<AgentSandboxProvider, ProviderCapabilities> = {
cloudflare: { terminal: true, devcontainer: true },
};
17 changes: 11 additions & 6 deletions services/cloud-agent-next/src/agent-sandbox/factory.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,18 +6,23 @@ import { CloudflareAgentSandbox } from './cloudflare/cloudflare-agent-sandbox.js

vi.mock('@cloudflare/sandbox', () => ({ getSandbox: vi.fn() }));

function metadata(): SessionMetadata {
function metadata(provider?: 'cloudflare'): SessionMetadata {
return {
metadataSchemaVersion: 2,
identity: { sessionId: 'agent_sandbox', userId: 'user_sandbox' },
auth: {},
workspace: { sandboxId: 'ses-abcdef' },
workspace: { sandboxId: 'ses-abcdef', ...(provider ? { sandboxProvider: provider } : {}) },
lifecycle: { version: 1, timestamp: 1 },
};
}

describe('AgentSandbox factory', () => {
it('constructs the Cloudflare runtime adapter', () => {
expect(createAgentSandbox({} as Env, metadata())).toBeInstanceOf(CloudflareAgentSandbox);
});
describe('AgentSandbox provider factory', () => {
it.each([undefined, 'cloudflare'] as const)(
'resolves %s metadata to the Cloudflare runtime adapter',
provider => {
expect(createAgentSandbox({} as Env, metadata(provider))).toBeInstanceOf(
CloudflareAgentSandbox
);
}
);
});
19 changes: 18 additions & 1 deletion services/cloud-agent-next/src/agent-sandbox/factory.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,25 @@
import type { SessionMetadata } from '../persistence/session-metadata.js';
import type { Env } from '../types.js';
import type { AgentSandbox } from './protocol.js';
import type { AgentSandbox, AgentSandboxLifecycle, AgentSandboxLifecycleHost } from './protocol.js';
import { CloudflareAgentSandbox } from './cloudflare/cloudflare-agent-sandbox.js';

export function createAgentSandbox(env: Env, metadata: SessionMetadata): AgentSandbox {
return new CloudflareAgentSandbox(env, metadata);
}

/**
* Provider-side lifecycle reconciliation seam. Cloudflare runtimes are
* locally addressed, so creation never needs reconciliation and deletion is
* driven synchronously by the session; providers whose runtimes are created
* or deleted through remote APIs dispatch to their own lifecycle here.
*/
export function createAgentSandboxLifecycle(
_env: Env,
_host: AgentSandboxLifecycleHost
): AgentSandboxLifecycle {
return {
reconcileCreateIntent: async () => undefined,
planDeletion: async () => ({ kind: 'not-applicable' }),
reconcilePendingDeletion: async () => 'none',
};
}
67 changes: 66 additions & 1 deletion services/cloud-agent-next/src/agent-sandbox/protocol.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,36 @@ import type {
FencedWrapperDispatchRequest,
WorkspaceReady,
} from '../execution/types.js';
import type { SessionMetadata } from '../persistence/session-metadata.js';

export type SandboxDeleteReason = 'explicit' | 'retention-expired' | 'recovery';

export type SessionDeletionIntent = {
reason: Extract<SandboxDeleteReason, 'explicit' | 'retention-expired'>;
startedAt: number;
};

export type AgentSandboxFailure =
| 'provider_not_configured'
| 'provider_auth_failed'
| 'runtime_not_running'
| 'runtime_creation_failed'
| 'runtime_configuration_drift'
| 'runtime_deleted_during_active_work'
| 'runtime_max_duration_reached'
| 'runtime_infrastructure_failed'
| 'capability_unavailable';

export class AgentSandboxUnavailableError extends Error {
constructor(
message: string,
public readonly failure: AgentSandboxFailure = 'capability_unavailable'
) {
super(message);
this.name = 'AgentSandboxUnavailableError';
}
}

export const WRAPPER_DISCOVERY_LIST_PROCESSES_TIMEOUT_REASON =
'wrapper_discovery_list_processes_timeout';
export type WrapperInspectionFailureReason = typeof WRAPPER_DISCOVERY_LIST_PROCESSES_TIMEOUT_REASON;
Expand Down Expand Up @@ -66,7 +93,8 @@ export type StopWrappersResult =
export type TerminalClientResult =
| { status: 'ready'; client: TerminalWrapperClient }
| { status: 'not-running' }
| { status: 'unhealthy' };
| { status: 'unhealthy' }
| { status: 'capability-unavailable'; message: string };

export type WrapperLogs = {
files: Record<string, string>;
Expand Down Expand Up @@ -115,3 +143,40 @@ export type AgentSandbox = {
keepAlive(): Promise<void>;
delete(reason: SandboxDeleteReason): Promise<void>;
};

export type ProviderDeletionPlan =
| { kind: 'not-applicable' }
| { kind: 'complete' }
| { kind: 'deferred'; entries: Record<string, unknown> };

/**
* Session-DO capabilities a provider lifecycle needs: durable storage for its
* intents/tombstones, alarm scheduling for retries, and terminal-state
* transitions owned by the session.
*/
export type AgentSandboxLifecycleHost = {
storage: DurableObjectStorage;
scheduleAlarmAtOrBefore(deadline: number): Promise<void>;
eraseDurableObjectState(): Promise<void>;
purgeDeletedSessionPayload(): Promise<void>;
getSessionIdForLogs(): string | undefined;
};

/**
* Provider-side lifecycle reconciliation for one Cloud Agent session:
* settling interrupted runtime creation and driving deletion to a terminal
* state. Methods self-guard on stored provider state, so the session can
* invoke them on alarm paths without knowing its provider.
*
* `planDeletion` only returns what to persist; the session commits the
* returned entries atomically with its own deletion-intent fence.
*/
export type AgentSandboxLifecycle = {
reconcileCreateIntent(now: number): Promise<void>;
planDeletion(input: {
metadata: SessionMetadata;
intent: SessionDeletionIntent;
now: number;
}): Promise<ProviderDeletionPlan>;
reconcilePendingDeletion(now: number): Promise<'handled' | 'none'>;
};
10 changes: 9 additions & 1 deletion services/cloud-agent-next/src/execution/errors.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,8 @@ export type RetryableErrorCode =
export type PermanentErrorCode =
| 'INVALID_REQUEST' // Bad input (missing fields, invalid format)
| 'SESSION_NOT_FOUND' // Session doesn't exist
| 'WRAPPER_JOB_CONFLICT'; // Wrapper busy (internal error - shouldn't happen)
| 'WRAPPER_JOB_CONFLICT' // Wrapper busy (internal error - shouldn't happen)
| 'SANDBOX_CAPABILITY_UNAVAILABLE'; // This session targets a deliberately disabled runtime capability

/**
* All possible execution error codes.
Expand Down Expand Up @@ -126,6 +127,13 @@ export class ExecutionError extends Error {
static wrapperJobConflict(message: string): ExecutionError {
return new ExecutionError('WRAPPER_JOB_CONFLICT', message, { retryable: false });
}

static sandboxCapabilityUnavailable(message: string, cause?: unknown): ExecutionError {
return new ExecutionError('SANDBOX_CAPABILITY_UNAVAILABLE', message, {
retryable: false,
cause,
});
}
}

/**
Expand Down
25 changes: 17 additions & 8 deletions services/cloud-agent-next/src/execution/orchestrator.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
import { beforeEach, describe, expect, it, vi } from 'vitest';
import type { AgentSandbox, WrapperInstanceLease } from '../agent-sandbox/protocol.js';
import {
AgentSandboxUnavailableError,
type AgentSandbox,
type WrapperInstanceLease,
} from '../agent-sandbox/protocol.js';
import type { Env } from '../types.js';
import { WrapperError } from '../kilo/wrapper-client.js';
import type { ExecutionError } from './errors.js';
Expand Down Expand Up @@ -220,17 +224,10 @@ describe('ExecutionOrchestrator AgentSandbox delivery', () => {
command: 'compact',
arguments: '--aggressive',
},
finalization: {
autoCommit: true,
condenseOnComplete: false,
},
} satisfies FencedWrapperDispatchRequest;
const commandRequest = {
command: 'compact',
args: '--aggressive',
messageId: commandPlan.turn.messageId,
autoCommit: true,
condenseOnComplete: false,
session: prepared.readyRequest.session,
};
buildWrapperSessionReadyAndPromptRequestsMock.mockResolvedValueOnce({
Expand Down Expand Up @@ -264,6 +261,18 @@ describe('ExecutionOrchestrator AgentSandbox delivery', () => {
} satisfies Partial<ExecutionError>);
});

it('classifies intentionally unavailable wrapper delivery as permanent', async () => {
const { orchestrator, ensureWrapper } = createOrchestrator();
ensureWrapper.mockRejectedValueOnce(
new AgentSandboxUnavailableError('Sandbox delivery capability is not enabled')
);

await expect(orchestrator.execute(basePlan)).rejects.toMatchObject({
code: 'SANDBOX_CAPABILITY_UNAVAILABLE',
retryable: false,
} satisfies Partial<ExecutionError>);
});

it('keeps ordinary wrapper bootstrap failure retryable', async () => {
const { orchestrator, ensureWrapper } = createOrchestrator();
ensureWrapper.mockRejectedValueOnce(new Error('wrapper unavailable'));
Expand Down
12 changes: 11 additions & 1 deletion services/cloud-agent-next/src/execution/orchestrator.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,11 @@ import {
getPreparationInfrastructureFailure,
withPreparationInfrastructureRecovery,
} from '../sandbox-recovery.js';
import type { AgentSandbox, WrapperInstanceLease } from '../agent-sandbox/protocol.js';
import {
AgentSandboxUnavailableError,
type AgentSandbox,
type WrapperInstanceLease,
} from '../agent-sandbox/protocol.js';
import { isCodeReviewEphemeralSandboxId } from '../code-review-ephemeral-sandbox.js';

/** Maximum time allowed for complete wrapper readiness, including Kilo startup. */
Expand Down Expand Up @@ -147,6 +151,12 @@ export class ExecutionOrchestrator {
await this.destroyEphemeralSandboxAfterPreAcceptanceFailure(sandbox, plan, error);
if (error instanceof ExecutionError) throw error;
if (error instanceof WrapperError && error.code === 'WRAPPER_FINALIZING') throw error;
if (error instanceof AgentSandboxUnavailableError) {
throw ExecutionError.sandboxCapabilityUnavailable(
'Sandbox runtime delivery is unavailable for this session',
error
);
}
throw ExecutionError.wrapperStartFailed(
`Failed to start wrapper: ${error instanceof Error ? error.message : String(error)}`,
error
Expand Down
Loading
Loading