Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 62 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,64 @@ picklab artifacts report # render the latest run
picklab session destroy --all
```

Every screenshot, log, and action lands in `.picklab/runs/<runId>/` with a manifest, so a run is inspectable and reproducible after the fact.
Every screenshot, log, and action lands in a run directory with a manifest, so a run is inspectable and reproducible after the fact. By default that run directory lives outside your project — see [Run storage](#run-storage) below.

### Run storage

By default, run artifacts (screenshots, logs, manifests, evidence journals)
are written under the shared Pickforge company root, **not** inside your
project — a default screenshot or run never shows up in `git status`:

```text
~/.pickforge/picklab/projects/<projectId>/runs/<runId>/
```

`<projectId>` is a stable id derived from the project's canonical (symlink-resolved)
path: the same project always resolves to the same id, and different projects
never collide. Use the platform home-directory equivalent on non-Linux systems.
`PICKLAB_HOME` overrides the PickLab home root (default `~/.pickforge/picklab`);
`picklab doctor` reports the resolved path.

Two other modes are available via `storage` in the **global** config or the
`PICKLAB_STORAGE_MODE` / `PICKLAB_STORAGE_PATH` environment overrides for
automation and tests; `.picklab/config.json` (project-level) can select
`project-local`, but not `custom` — see below:

```json
{
"storage": { "mode": "project-local" }
}
```

- `home` (default) — the layout above.
- `project-local` — restores the previous default: `.picklab/runs/` inside the
project. Generated files then do appear in the project's source-control
view; add `.picklab/runs/` to `.gitignore` if you opt into this mode.
Selectable from project or global config.
- `custom` — an explicit absolute path outside the project directory:
`{ "storage": { "mode": "custom", "path": "/abs/path" } }` writes runs
under `<path>/runs/`. A relative path, a path equal to or nested inside the
project directory, or `custom` mode with no path, is rejected.

**`custom` cannot be selected from project-level `.picklab/config.json`.**
That file is repo-committed and travels with `git clone`; honoring a
`custom` selection from it would let a cloned repository silently redirect
run artifacts (screenshots, which may carry secrets) to any absolute path
with no prompt. Only the user-owned global config or an env override may
select `custom`. A project config that requests `custom` is ignored — the
resolver falls back to global config's mode, then `home` — and `picklab
doctor` surfaces the rejected request as a warning.

`.picklab/config.json` itself always stays project-local regardless of
`storage` mode — only generated runtime artifacts move.

**Upgrading from an earlier version:** existing runs already written under a
project's `.picklab/runs/` remain discoverable by `artifact_list` /
`artifact_report` / MCP resources without any migration step — nothing is
moved or deleted. Likewise, an existing `~/.picklab` global config, agent
state, or session registry (the previous default PickLab home) is still read
as a non-destructive fallback if the new `~/.pickforge/picklab` default has
nothing yet; `picklab doctor` flags a detected legacy home.

### Evidence recording

Expand All @@ -68,7 +125,8 @@ desktop, Android, and session actions share the same append-only timeline as
browser DevTools actions. Destroying a session, or reaping a dead one, finalizes
the run and writes a static `report.html` filmstrip.

A finalized evidence run under `.picklab/runs/<runId>/` contains:
A finalized evidence run directory (see [Run storage](#run-storage) for where
it lives) contains:

- `manifest.json` — run identity, status, and evidence metadata
- `actions.jsonl` — authoritative, append-only sanitized action timeline
Expand Down Expand Up @@ -144,7 +202,8 @@ For any other agent, add the stdio server yourself:

`picklab-browser` is static. Each invocation discovers the one live browser session for the agent's project and derives its loopback CDP URL in memory, so recreating a session never requires an agent config edit. The relay runs the bundled, exact `chrome-devtools-mcp@1.5.0`; it does not use `npx` or connect to a personal browser.

Custom agents can be stored under `~/.picklab/agents`:
Custom agents can be stored under the PickLab home's `agents/` dir (default
`~/.pickforge/picklab/agents`, override via `PICKLAB_HOME`):

```sh
picklab agents add --name my-agent --mcp-command "picklab mcp serve"
Expand Down
10 changes: 6 additions & 4 deletions SECURITY.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,12 @@ fuller model.

## Recorded evidence and screenshots

Computer-use evidence is stored as local project data under
`.picklab/runs/<runId>/`. `actions.jsonl` is the authoritative sanitized
timeline; finalization produces an escaped, no-script `report.html` with a
restrictive content security policy and no external requests.
Computer-use evidence is stored as local run data, by default under
`~/.pickforge/picklab/projects/<projectId>/runs/<runId>/` (outside the
project; see the README's "Run storage" section for storage modes).
`actions.jsonl` is the authoritative sanitized timeline; finalization
produces an escaped, no-script `report.html` with a restrictive content
security policy and no external requests.

The recorder persists only allowlisted metadata. Typed and filled text becomes
length plus input type. Failed network records keep method, URL origin/path
Expand Down
57 changes: 55 additions & 2 deletions docs/releases/UNRELEASED.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,24 @@ release description, then reset it after the release is published.

## User-facing changes

- **Default run storage changed.** Screenshots, logs, manifests, and evidence
journals now default to `~/.pickforge/picklab/projects/<projectId>/runs/`
(outside the project) instead of `<project>/.picklab/runs/`, so a default
run no longer shows up in `git status`. `PICKLAB_HOME` now defaults to
`~/.pickforge/picklab` (was `~/.picklab`); the old default is still read
non-destructively as a fallback for existing global config, agent state,
and sessions — nothing is moved or deleted. `project-local` (restores the
previous layout, selectable from project or global config) and `custom`
(explicit absolute path outside the project directory) storage modes are
available via the user-owned global config's `storage` field or the
`PICKLAB_STORAGE_MODE` / `PICKLAB_STORAGE_PATH` env overrides — a
project-committed `.picklab/config.json` cannot select `custom` (it
travels with `git clone`; a cloned repo requesting it falls back to home
with a `picklab doctor` warning, never a silent redirect of screenshots to
an attacker-chosen path). Existing project-local runs remain discoverable
by `artifact_list` / `artifact_report` / MCP resources without migration,
and retention pruning never touches them — only the resolved primary
storage root is ever pruned. See the README's "Run storage" section.
- Added isolated headed Chrome/Chromium sessions, including CLI and MCP session
lifecycle support and a static `picklab-browser` DevTools MCP entry.
- Added `picklab watch` plus configurable manual/automatic VNC viewer attachment
Expand All @@ -26,6 +44,25 @@ release description, then reset it after the release is published.

## Internal/release changes

- Added a single storage resolver (`resolveRunStorage`) covering home,
project-local, and custom modes with stable per-project id derivation
(sha256 of the canonical project path); routed run creation, the run
catalog, and active-evidence-pointer resolution in core, CLI, and MCP
through it. `openRunCatalog` now layers the resolved primary root with a
read-only legacy project-local fallback root for non-destructive discovery.
The resolver applies a source allowlist so only the global config layer or
an env override may select `custom` storage (project config may still
select `project-local` or `home`), and rejects a custom path that equals
or is nested inside the project directory. `pruneFinalizedEvidenceRuns`
scopes its retention candidate set to the resolved primary root
(`entry.rootDir`) so the read-only legacy fallback root is never a removal
candidate.
Added a legacy read-fallback (`resolveReadablePath`, per-entry) for global
config, agent state, and sessions across the `~/.picklab` →
`~/.pickforge/picklab` default-root change. `destroySessionRecord` now
removes both the new-home and legacy copies of a session record
unconditionally, so a record that exists at both locations cannot
resurrect via the legacy read fallback after being destroyed.
- Added a private browser lifecycle package with confined ephemeral profiles,
loopback-only CDP discovery, verified process-group teardown, concurrent
session safety, and conservative dead-session reaping.
Expand Down Expand Up @@ -60,12 +97,28 @@ release description, then reset it after the release is published.
- `bun install --frozen-lockfile`
- `bun run typecheck`
- Focused run catalog, run, evidence, CLI artifact, and MCP resource/tool suites.
- `bun run test` (79 files, 958 passed / 2 skipped)
- `bun run test:coverage` (79 files, 958 passed / 2 skipped; all thresholds met)
- New `storage.test.ts` (project id derivation, all three storage modes, env
overrides) and `run-catalog.test.ts`'s "openRunCatalog storage modes" suite
(home default, legacy project-local discovery, project isolation, custom
mode), plus a `git status --porcelain` repo-cleanliness smoke test.
- `bun run test` — same pre-existing failure set as unmodified `main`,
verified test-by-test against `main` on this branch's (macOS) dev sandbox.
These are platform-environment failures, not specific to this change; on
CI's Linux runners this class does not apply. At least three distinct
causes, none `/proc`-only as previously stated here:
process-identity/Xvfb/x11vnc verification (Darwin has no `/proc`; most of
the count), a macOS tmpdir realpath mismatch (`/var` vs `/private/var`,
e.g. `run-catalog.test.ts`'s root-precedence test and
`devtools-mcp.test.ts`'s package-root assertion), and inline-screenshot/CDP
association mismatches in `devtools-evidence.test.ts`.
- `bun run build`

### Not tested yet

- `bun run test:coverage` on a machine where the full suite runs clean (this
dev sandbox can't produce a coverage summary because the pre-existing
Darwin-only failures above stop the process before the v8 coverage
provider flushes it — reproduced identically on unmodified `main`).
- Platform smoke checks outside Linux.
- Live remote SSH-tunnel smoke test.
- Tag-triggered npm publish and draft-release creation (runs only after merge and
Expand Down
25 changes: 23 additions & 2 deletions packages/agent-installers/src/state.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
import fs from "node:fs";
import path from "node:path";
import { agentsDir, writeFileAtomic, type EnvLike } from "@pickforge/picklab-core";
import {
agentsDir,
legacyAgentsDir,
resolveReadablePath,
writeFileAtomic,
type EnvLike,
} from "@pickforge/picklab-core";

export interface AgentStateEntry {
registered: boolean;
Expand All @@ -20,12 +26,27 @@ export function agentsStatePath(env: EnvLike = process.env): string {
return path.join(agentsDir(env), "state.json");
}

function legacyAgentsStatePath(env: EnvLike): string | undefined {
const dir = legacyAgentsDir(env);
return dir === undefined ? undefined : path.join(dir, "state.json");
}

/**
* Read agent registration state. Falls back to a pre-#34 `~/.picklab/agents`
* record when the new home has none yet, so "is this agent registered?"
* keeps answering correctly across the default-root change without a
* migration step. Writes (`recordAgentState`) always target the new home.
*/
export async function readAgentsState(
env: EnvLike = process.env,
): Promise<AgentsState> {
const statePath = await resolveReadablePath(
agentsStatePath(env),
legacyAgentsStatePath(env),
);
let raw: string;
try {
raw = await fs.promises.readFile(agentsStatePath(env), "utf8");
raw = await fs.promises.readFile(statePath, "utf8");
} catch (error) {
const code = (error as NodeJS.ErrnoException).code;
if (code === "ENOENT" || code === "ENOTDIR") {
Expand Down
109 changes: 109 additions & 0 deletions packages/agent-installers/test/state.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import { readAgentsState, recordAgentState } from "../src/state.js";

let fakeHome: string;
let homedirSpy: ReturnType<typeof vi.spyOn>;

beforeEach(async () => {
fakeHome = await fs.promises.mkdtemp(
path.join(os.tmpdir(), "picklab-agentstate-fakehome-"),
);
homedirSpy = vi.spyOn(os, "homedir").mockReturnValue(fakeHome);
});

afterEach(async () => {
homedirSpy.mockRestore();
await fs.promises.rm(fakeHome, { recursive: true, force: true });
});

describe("readAgentsState legacy home fallback", () => {
it("reads a legacy ~/.picklab/agents/state.json when PICKLAB_HOME is unset", async () => {
const legacyDir = path.join(fakeHome, ".picklab", "agents");
await fs.promises.mkdir(legacyDir, { recursive: true });
await fs.promises.writeFile(
path.join(legacyDir, "state.json"),
JSON.stringify({
agents: {
"claude-code": {
registered: true,
configPath: "/legacy/.claude.json",
updatedAt: "2026-01-01T00:00:00.000Z",
},
},
}),
);

const state = await readAgentsState({});
expect(state.agents["claude-code"]?.registered).toBe(true);
expect(state.agents["claude-code"]?.configPath).toBe("/legacy/.claude.json");
});

it("prefers the new home once it has its own state", async () => {
await fs.promises.mkdir(path.join(fakeHome, ".picklab", "agents"), {
recursive: true,
});
await fs.promises.writeFile(
path.join(fakeHome, ".picklab", "agents", "state.json"),
JSON.stringify({
agents: { codex: { registered: true, configPath: "/legacy/codex" } },
}),
);
await recordAgentState(
"codex",
{ registered: false, configPath: "/new/codex" },
{},
);

const state = await readAgentsState({});
expect(state.agents.codex?.registered).toBe(false);
expect(state.agents.codex?.configPath).toBe("/new/codex");
});

it("does not fall back once PICKLAB_HOME is set explicitly", async () => {
await fs.promises.mkdir(path.join(fakeHome, ".picklab", "agents"), {
recursive: true,
});
await fs.promises.writeFile(
path.join(fakeHome, ".picklab", "agents", "state.json"),
JSON.stringify({
agents: { codex: { registered: true, configPath: "/legacy/codex" } },
}),
);

const state = await readAgentsState({ PICKLAB_HOME: "/other" });
expect(state.agents).toEqual({});
});

it("writes always target the new home, leaving legacy state untouched", async () => {
const legacyPath = path.join(
fakeHome,
".picklab",
"agents",
"state.json",
);
await fs.promises.mkdir(path.dirname(legacyPath), { recursive: true });
await fs.promises.writeFile(
legacyPath,
JSON.stringify({
agents: { codex: { registered: true, configPath: "/legacy/codex" } },
}),
);

await recordAgentState(
"cursor",
{ registered: true, configPath: "/new/cursor" },
{},
);

const legacyRaw = JSON.parse(await fs.promises.readFile(legacyPath, "utf8"));
expect(legacyRaw.agents.cursor).toBeUndefined();
expect(
fs.existsSync(
path.join(fakeHome, ".pickforge", "picklab", "agents", "state.json"),
),
).toBe(true);
});
});
7 changes: 6 additions & 1 deletion packages/android/test/reaper.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,12 @@ describe("android reaper tracking", () => {
bootTimeoutMs: 5_000,
});

const { run } = await beginEvidenceRun(projectDir, session.id);
const { run } = await beginEvidenceRun(
projectDir,
session.id,
{},
registryEnv,
);

forceStopFailure = true;
try {
Expand Down
9 changes: 6 additions & 3 deletions packages/browser/src/devtools-evidence.ts
Original file line number Diff line number Diff line change
Expand Up @@ -401,9 +401,12 @@ export async function createDevtoolsEvidenceRecorder(
): Promise<DevtoolsEvidenceRecorder | undefined> {
const config = await loadConfig(opts.projectDir, opts.env);
if (!isEvidenceEnabled(config)) return undefined;
const { run } = await beginEvidenceRun(opts.projectDir, opts.sessionId, {
slug: "computer-use",
});
const { run } = await beginEvidenceRun(
opts.projectDir,
opts.sessionId,
{ slug: "computer-use" },
opts.env,
);
const runStat = await fs.promises.lstat(run.dir);
if (!runStat.isDirectory() || runStat.isSymbolicLink()) {
throw new Error("Evidence run directory is not a real directory");
Expand Down
5 changes: 4 additions & 1 deletion packages/browser/test/devtools-evidence.test.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { afterEach, beforeEach, describe, expect, it, vi } from "vitest";
import {
listRuns,
readActions,
Expand All @@ -19,9 +19,12 @@ beforeEach(async () => {
projectDir = await fs.promises.mkdtemp(
path.join(os.tmpdir(), "picklab-devtools-evidence-"),
);
// These tests assert against the literal `.picklab/runs` layout.
vi.stubEnv("PICKLAB_STORAGE_MODE", "project-local");
});

afterEach(async () => {
vi.unstubAllEnvs();
await fs.promises.rm(projectDir, { recursive: true, force: true });
});

Expand Down
Loading
Loading