Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions packages/pi-extension/CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,17 @@

## Unreleased

- Flags name the domain rather than an acronym: `--cua-tools` is now
`--browser-tools`, `--cua-coordinates` is `--browser-coordinates`, and the
commands are `/browser` and `/browser-tools`.
- Browser configuration is one `--browser-options` JSON object forwarded verbatim
to Kernel's browser-create call, replacing `--cua-profile-id`,
`--cua-profile-save-changes`, `--cua-proxy-id`, and `--cua-browser-timeout`. A
flag per create-call field grows every time the SDK does; JSON tracks it for
free. The only default is `timeout_seconds: 600`. `--browser-session` still
attaches an existing browser and cannot be combined with `--browser-options`.
Note that `stealth` is no longer forced on — pass it in the JSON if you want it.

- `@onkernel/cua-cli` and the `cua` binary are removed. Everything the CLI built
because it needed an agent front-end — sessions and resume, skills, the TUI,
print and RPC modes, model selection — pi supplies, so the extension replaces
Expand Down
35 changes: 23 additions & 12 deletions packages/pi-extension/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,17 +25,17 @@ No selector means no Kernel tool is active and no browser is provisioned.

```sh
pi -p --provider openai --model gpt-5.6-sol \
--cua-tools browser,browser-act "Open example.com and report its heading"
--browser-tools browser,browser-act "Open example.com and report its heading"

pi --mode rpc --no-session --provider openai --model gpt-5.6-sol --cua-tools browser
pi --mode rpc --no-session --provider openai --model gpt-5.6-sol --browser-tools browser

pi -p --provider anthropic --model claude-opus-5 --cua-tools anthropic-computer \
pi -p --provider anthropic --model claude-opus-5 --browser-tools anthropic-computer \
"Open example.com and report its heading"
```

### The menu

Eight entries, one per capability. Availability is per model, and `/cua-tools`
Eight entries, one per capability. Availability is per model, and `/browser-tools`
tells you which apply to the one you selected.

| entry | tools | works on |
Expand All @@ -52,16 +52,16 @@ tells you which apply to the one you selected.
`anthropic-browser` and `anthropic-computer` cannot be selected together:
Anthropic rejects the pair because the browser tool addresses a viewport
coordinate frame and the computer tool a display frame. The catalog compiler
refuses it before the request goes out, and `/cua-tools` reports it as a conflict
refuses it before the request goes out, and `/browser-tools` reports it as a conflict
rather than as unavailability.

`--cua-coordinates` selects `pixels` (default) or `normalized-1000` for the
`--browser-coordinates` selects `pixels` (default) or `normalized-1000` for the
`computer` entry's coordinate contract.

### Commands

- `/cua` — current selectors, active tools, and browser status.
- `/cua-tools` — with no argument, list every selector for the current model,
- `/browser` — current selectors, active tools, and browser status.
- `/browser-tools` — with no argument, list every selector for the current model,
marking the selected ones and showing the compiler's own reason for any that
this model cannot take. With an argument, replace the selection. `none` clears
it.
Expand All @@ -80,10 +80,21 @@ no browser is created, and the model answers from memory with exit 0.

| flag | effect |
| --- | --- |
| `--cua-browser-session` | attach an existing session; never deleted on exit |
| `--cua-profile-id`, `--cua-profile-save-changes` | load and optionally persist a profile |
| `--cua-proxy-id` | route through a Kernel proxy |
| `--cua-browser-timeout` | owned-browser timeout in seconds (default 300) |
| `--browser-session` | attach an existing session; never deleted on exit |
| `--browser-options` | JSON forwarded verbatim to Kernel's browser-create call |

`--browser-options` is one JSON object rather than a flag per field, so it tracks
the Kernel SDK without this extension growing an option every time the SDK does:

```sh
pi -p --browser-tools browser \
--browser-options '{"stealth":true,"profile":{"id":"p1","save_changes":true},"proxy_id":"px1"}' \
"open example.com"
```

The only default is `timeout_seconds: 600` — the failure it prevents is a browser
vanishing mid-task. Override it in the same JSON. `--browser-session` attaches an
existing browser, so it cannot be combined with `--browser-options`.

One browser is provisioned lazily per session, on first tool execution.
Compiling declarations, generating headers, and transforming a payload never
Expand Down
25 changes: 16 additions & 9 deletions packages/pi-extension/src/browser-runtime.ts
Original file line number Diff line number Diff line change
@@ -1,13 +1,22 @@
import Kernel from "@onkernel/sdk";
import { CuaExecutionResources } from "@onkernel/cua-agent";

/**
* Browser configuration, passed as one JSON object rather than a flag per field.
*
* `create` is forwarded to Kernel's browser-create call as-is, so it tracks the
* SDK without this extension growing a flag every time the SDK does. The only
* default is a generous timeout: the failure it prevents is a browser vanishing
* mid-task.
*/
export interface BrowserOptions {
/** Attach an existing session instead of creating one. Never deleted on exit. */
sessionId?: string;
profileId?: string;
proxyId?: string;
timeoutSeconds: number;
saveProfileChanges: boolean;
/** Forwarded verbatim to `client.browsers.create`. */
create: Record<string, unknown>;
}

export const DEFAULT_BROWSER_TIMEOUT_SECONDS = 600;
export interface BrowserStatus {
sessionId?: string;
owned?: boolean;
Expand Down Expand Up @@ -62,11 +71,9 @@ export class CuaBrowserRuntime {
const browser = attached
? await client.browsers.retrieve(this.options.sessionId!)
: await client.browsers.create({
stealth: true,
timeout_seconds: this.options.timeoutSeconds,
...(this.options.profileId ? { profile: { id: this.options.profileId, save_changes: this.options.saveProfileChanges } } : {}),
...(this.options.proxyId ? { proxy_id: this.options.proxyId } : {}),
});
timeout_seconds: DEFAULT_BROWSER_TIMEOUT_SECONDS,
...this.options.create,
} as never);
this.client = client;
this.status = {
sessionId: browser.session_id,
Expand Down
91 changes: 48 additions & 43 deletions packages/pi-extension/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,10 @@ import { CONFIG_ENTRY, restoreConfig, type PersistedConfig } from "./state";
import { availabilityText, statusText } from "./render";

export default function cuaPiExtension(pi: ExtensionAPI): void {
pi.registerFlag("cua-tools", { type: "string", description: "Comma-separated explicit CUA tool selectors" });
pi.registerFlag("cua-coordinates", { type: "string", description: "pixels or normalized-1000", default: "pixels" });
pi.registerFlag("cua-browser-session", { type: "string", description: "Attach an existing Kernel browser session" });
pi.registerFlag("cua-profile-id", { type: "string", description: "Kernel browser profile id" });
pi.registerFlag("cua-proxy-id", { type: "string", description: "Kernel proxy id" });
pi.registerFlag("cua-browser-timeout", { type: "string", description: "Owned browser timeout in seconds", default: "300" });
pi.registerFlag("cua-profile-save-changes", { type: "boolean", description: "Save owned browser profile changes", default: false });
pi.registerFlag("browser-tools", { type: "string", description: "Comma-separated tool selectors; see /browser-tools for this model's menu" });
pi.registerFlag("browser-coordinates", { type: "string", description: "pixels or normalized-1000", default: "pixels" });
pi.registerFlag("browser-session", { type: "string", description: "Attach an existing Kernel browser session instead of creating one" });
pi.registerFlag("browser-options", { type: "string", description: "JSON forwarded to Kernel's browser-create call, e.g. {\"stealth\":true}" });
// Parsed flag values are unavailable until after the extension factory returns,
// but session_start errors do not stop print/RPC provider calls.
validateRawCliFlags();
Expand All @@ -54,7 +51,7 @@ export default function cuaPiExtension(pi: ExtensionAPI): void {
for (const [name, spec] of allSpecs) {
const conflict = pi.getAllTools().find((tool) => tool.name === name);
if (conflict && conflict.sourceInfo.path !== extensionPath) {
throw new Error(`cannot register CUA tool "${name}": already owned by ${conflict.sourceInfo.source}`);
throw new Error(`cannot register browser tool "${name}": already owned by ${conflict.sourceInfo.source}`);
}
pi.registerTool({
name: spec.name,
Expand All @@ -63,17 +60,17 @@ export default function cuaPiExtension(pi: ExtensionAPI): void {
parameters: spec.declaration.parameters,
executionMode: "sequential",
async execute(toolCallId, input, signal) {
if (!activeNames.has(name)) throw new Error(`CUA tool "${name}" is not active`);
if (!activeNames.has(name)) throw new Error(`browser tool "${name}" is not active`);
const selected = currentSpecs().find((candidate) => candidate.name === name);
if (!selected || compatibilityError) throw new Error(compatibilityError ?? `CUA tool "${name}" is no longer selected`);
if (!selected || compatibilityError) throw new Error(compatibilityError ?? `browser tool "${name}" is no longer selected`);
const resources = await ensureRuntime().get(signal);
return resources.materialize(selected).execute(toolCallId, input, signal);
},
});
}
}
function ensureRuntime(): CuaBrowserRuntime {
if (!sessionActive) throw new Error("CUA browser runtime is unavailable outside an active pi session");
if (!sessionActive) throw new Error("the browser runtime is unavailable outside an active pi session");
return (runtime ??= new CuaBrowserRuntime(browserOptions));
}
function currentSpecs(): CuaToolSpec[] {
Expand Down Expand Up @@ -121,13 +118,13 @@ export default function cuaPiExtension(pi: ExtensionAPI): void {
}
initialized = true;
if (ctx.mode === "tui") {
ctx.ui.setStatus("cua", statusText(selection.selectors, [...activeNames], runtime?.getStatus() ?? {}, compatibilityError));
ctx.ui.setStatus("browser-tools", statusText(selection.selectors, [...activeNames], runtime?.getStatus() ?? {}, compatibilityError));
} else if (compatibilityError && compatibilityError !== warnedError) {
// Print and RPC have no status line, and silence here is the worst failure
// this extension can produce: the tools vanish, no browser is created, and
// the model answers from memory with exit 0. Say so on stderr, once per
// distinct reason so a multi-turn run does not repeat itself.
process.stderr.write(`cua: no browser tool is active — ${compatibilityError}\n`);
process.stderr.write(`browser tools: none active — ${compatibilityError}\n`);
warnedError = compatibilityError;
}
if (!compatibilityError) warnedError = undefined;
Expand Down Expand Up @@ -185,21 +182,21 @@ export default function cuaPiExtension(pi: ExtensionAPI): void {

registerCuaProviders();

pi.registerCommand("cua", {
description: "Show CUA tool and browser status",
pi.registerCommand("browser", {
description: "Show the selected browser tools and browser status",
handler: async (_args, ctx) => {
reconcile(ctx);
notifyStatus(ctx);
},
});
pi.registerCommand("cua-tools", {
description: "Replace this session's explicit CUA selectors, or list what this model can take",
pi.registerCommand("browser-tools", {
description: "Replace this session's tool selection, or list what this model can take",
handler: async (args, ctx) => {
// No argument lists the menu instead of clearing the selection, because
// clearing is the more destructive reading of an empty command.
if (!args?.trim()) {
if (!ctx.model) {
ctx.ui.notify("cua: no pi model is selected", "error");
ctx.ui.notify("browser tools: no pi model is selected", "error");
return;
}
ctx.ui.notify(availabilityText(selectorAvailability(ctx.model, selection)), "info");
Expand Down Expand Up @@ -227,10 +224,10 @@ export default function cuaPiExtension(pi: ExtensionAPI): void {
const known = saved.selectors.filter((selector) => CUA_SELECTORS.includes(selector));
const dropped = saved.selectors.filter((selector) => !CUA_SELECTORS.includes(selector));
if (dropped.length) {
process.stderr.write(`cua: ignoring retired tool selector(s) from this session: ${dropped.join(", ")}\n`);
process.stderr.write(`browser tools: ignoring retired selector(s) from this session: ${dropped.join(", ")}\n`);
}
// Always apply what was restored, even when nothing survives. A persisted
// selection came from `/cua-tools`, which deliberately overrides the flags,
// selection came from `/browser-tools`, which deliberately overrides the flags,
// so falling back to them would re-enable tools this session had replaced.
// An empty selection with the note above is the honest outcome.
selection = parseSelection(known.join(",") || undefined, saved.coordinates);
Expand Down Expand Up @@ -270,7 +267,7 @@ export default function cuaPiExtension(pi: ExtensionAPI): void {
pi.on("tool_call", (event) => {
if (!allSpecs.has(event.toolName)) return;
if (!activeNames.has(event.toolName) || compatibilityError)
return { block: true, reason: compatibilityError ?? `CUA tool "${event.toolName}" is inactive` };
return { block: true, reason: compatibilityError ?? `browser tool "${event.toolName}" is inactive` };
});
pi.on("session_shutdown", async () => {
sessionActive = false;
Expand All @@ -293,26 +290,41 @@ function validateRawCliFlags(argv = process.argv.slice(2)): void {
const index = argv.indexOf(`--${name}`);
return index >= 0 && !argv[index + 1]?.startsWith("--") ? argv[index + 1] : undefined;
};
parseSelection(read("cua-tools"), read("cua-coordinates") ?? "pixels");
const sessionId = trim(read("cua-browser-session"));
if (sessionId && (trim(read("cua-profile-id")) || trim(read("cua-proxy-id"))))
throw new Error("--cua-browser-session cannot be combined with --cua-profile-id or --cua-proxy-id");
positiveSeconds(read("cua-browser-timeout"));
parseSelection(read("browser-tools"), read("browser-coordinates") ?? "pixels");
parseBrowserOptions(read("browser-session"), read("browser-options"));
}
function readFlags(pi: ExtensionAPI): { selection: CuaSelection; browserOptions: BrowserOptions } {
const browserOptions: BrowserOptions = {
sessionId: trim(asString(pi.getFlag("cua-browser-session"))),
profileId: trim(asString(pi.getFlag("cua-profile-id"))),
proxyId: trim(asString(pi.getFlag("cua-proxy-id"))),
timeoutSeconds: positiveSeconds(asString(pi.getFlag("cua-browser-timeout"))),
saveProfileChanges: pi.getFlag("cua-profile-save-changes") === true,
return {
selection: parseSelection(asString(pi.getFlag("browser-tools")), asString(pi.getFlag("browser-coordinates"))),
browserOptions: parseBrowserOptions(asString(pi.getFlag("browser-session")), asString(pi.getFlag("browser-options"))),
};
if (browserOptions.sessionId && (browserOptions.profileId || browserOptions.proxyId))
throw new Error("--cua-browser-session cannot be combined with --cua-profile-id or --cua-proxy-id");
return { selection: parseSelection(asString(pi.getFlag("cua-tools")), asString(pi.getFlag("cua-coordinates"))), browserOptions };
}
function defaultBrowserOptions(): BrowserOptions {
return { timeoutSeconds: 300, saveProfileChanges: false };
return { create: {} };
}

/**
* One JSON flag instead of a flag per create-call field, so this extension does
* not grow an option every time the Kernel SDK does.
*/
export function parseBrowserOptions(sessionId: string | undefined, optionsJson: string | undefined): BrowserOptions {
const attach = trim(sessionId);
const raw = trim(optionsJson);
let create: Record<string, unknown> = {};
if (raw) {
let parsed: unknown;
try {
parsed = JSON.parse(raw);
} catch (error) {
throw new Error(`--browser-options must be valid JSON: ${error instanceof Error ? error.message : String(error)}`);
}
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) throw new Error("--browser-options must be a JSON object");
create = parsed as Record<string, unknown>;
}
if (attach && Object.keys(create).length > 0) {
throw new Error("--browser-session attaches an existing browser, so --browser-options cannot also configure a new one");
}
return { ...(attach ? { sessionId: attach } : {}), create };
}
function asString(value: boolean | string | undefined): string | undefined {
return typeof value === "string" ? value : undefined;
Expand All @@ -321,13 +333,6 @@ function trim(value: string | undefined): string | undefined {
const result = value?.trim();
return result || undefined;
}
function positiveSeconds(value: string | undefined): number {
const seconds = Number(value ?? "300");
if (!Number.isSafeInteger(seconds) || seconds < 1 || seconds > 259200)
throw new Error("--cua-browser-timeout must be a whole number from 1 to 259200");
return seconds;
}

function withoutCuaToolSchemas(payload: unknown, cuaSpecs: ReadonlyMap<string, CuaToolSpec>): unknown {
if (!isRecord(payload) || !Array.isArray(payload.tools)) return payload;
const tools: unknown[] = [];
Expand Down
4 changes: 2 additions & 2 deletions packages/pi-extension/src/render.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ export function statusText(selectors: readonly string[], active: readonly string
const browserText = browser.sessionId
? `${browser.owned ? "owned" : "attached"} ${browser.sessionId}${browser.liveUrl ? ` ${browser.liveUrl}` : ""}`
: "not provisioned";
return `cua: selected=${selectors.join(",") || "none"}; active=${tools}; browser=${browserText}${error ? `; unavailable=${error}` : ""}`;
return `browser tools: selected=${selectors.join(",") || "none"}; active=${tools}; browser=${browserText}${error ? `; unavailable=${error}` : ""}`;
}

/** One line per selector, so an unavailable one carries the compiler's own reason. */
Expand All @@ -17,5 +17,5 @@ export function availabilityText(entries: readonly SelectorAvailability[]): stri
const conflict = entry.conflictsWith.length ? ` (cannot combine with ${entry.conflictsWith.join(", ")})` : "";
return `${mark} ${entry.selector}${conflict}`;
});
return ["cua selectors for this model (* = selected):", ...lines].join("\n");
return ["browser tool selectors for this model (* = selected):", ...lines].join("\n");
}
10 changes: 5 additions & 5 deletions packages/pi-extension/src/selection.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,16 +51,16 @@ export const CUA_SELECTORS: readonly string[] = Object.freeze(Object.keys(MENU))
export function parseSelection(value: string | undefined, coordinates: string | undefined): CuaSelection {
const coordinateMode = coordinates ?? "pixels";
if (coordinateMode !== "pixels" && coordinateMode !== "normalized-1000") {
throw new Error('--cua-coordinates must be "pixels" or "normalized-1000"');
throw new Error('--browser-coordinates must be "pixels" or "normalized-1000"');
}
const selectors =
value
?.split(",")
.map((item) => item.trim())
.filter(Boolean) ?? [];
if (new Set(selectors).size !== selectors.length) throw new Error("--cua-tools contains duplicate selectors");
if (new Set(selectors).size !== selectors.length) throw new Error("--browser-tools contains duplicate selectors");
for (const selector of selectors) {
if (!CUA_SELECTORS.includes(selector)) throw new Error(`unknown CUA tool selector "${selector}"`);
if (!CUA_SELECTORS.includes(selector)) throw new Error(`unknown browser tool selector "${selector}"`);
}
return Object.freeze({ selectors: Object.freeze(selectors), coordinates: coordinateMode });
}
Expand All @@ -85,12 +85,12 @@ export function expandSelection(selection: CuaSelection): CuaToolSpec[] {
const result: CuaToolSpec[] = [];
for (const selector of selection.selectors) {
const entry = MENU[selector];
if (!entry) throw new Error(`unknown CUA tool selector "${selector}"`);
if (!entry) throw new Error(`unknown browser tool selector "${selector}"`);
result.push(...entry(coordinates));
}
const identities = new Set<string>();
for (const spec of result) {
if (identities.has(spec.identity)) throw new Error(`CUA selection contains duplicate tool identity "${spec.identity}"`);
if (identities.has(spec.identity)) throw new Error(`selection contains duplicate tool identity "${spec.identity}"`);
identities.add(spec.identity);
}
return result;
Expand Down
Loading
Loading