Add Vercel Blob storage backend for production#15
Open
gnirpaz wants to merge 9 commits into
Open
Conversation
Production (Vercel) has a read-only filesystem, so the JSON-file storage can't persist there. Introduce a durable backend selected at runtime, without duplicating the storage logic. Architecture: - Extract a small KvStore primitive interface (get/put/createIfAbsent/ exists/delete/deletePrefix/listChildren + append-only events). - Move ALL business logic (validation, normalization, activity, agent pointer sync, unique-id reservation, section-isolated summary) into a backend-agnostic DocumentStorage that speaks only KvStore. - FsKvStore: the existing filesystem backend (atomic writes, per-path write serialization, corrupt-file quarantine, ndjson event logs). - BlobKvStore: new @vercel/blob backend. Deterministic pathnames, private access, fresh reads (useCache:false), allowOverwrite:false for atomic unique-id reservation, folded prefix listing for direct children, and one blob per event (no read-modify-write append race). - getStorage() selects the backend: AGENTBOARD_STORAGE=blob|fs forces it, otherwise a present BLOB_READ_WRITE_TOKEN selects Blob (Vercel), with the filesystem as the local default. FsStorage/getStorage/AgentRegistrationError public API is preserved. Tests: - Existing FS suite is unchanged and passes (regression guard for the refactor). - document-storage.test.ts runs the business logic against an in-memory KvStore (backend-agnostic guarantees). - blob-kv.test.ts runs DocumentStorage against a faithful in-memory fake of the @vercel/blob SDK (folded listing, conditional create, not-found), validating BlobKvStore key/prefix handling end-to-end. Note: BlobKvStore cannot be exercised against a live store from CI here; it is validated via the SDK fake and type-checking. Requires the BLOB_READ_WRITE_TOKEN env var (already set by Vercel Blob). https://claude.ai/code/session_014r13EsBf8Kz8YGfD6VxDtS
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Only delete the source blob after copy succeeds, matching FsKvStore behavior where a failed rename leaves the original file intact.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed:
head()missingaccess: 'private'inexists()- Added
{ access: ACCESS }to thehead()call inBlobObjectStore.exists()(formerlyblob-kv.ts) so private blob metadata checks match all other SDK calls.
- Added
Preview (4b00138c57)
diff --git a/bun.lock b/bun.lock
--- a/bun.lock
+++ b/bun.lock
@@ -5,6 +5,7 @@
"": {
"name": "agentboard-init",
"dependencies": {
+ "@vercel/blob": "^2.4.0",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"lucide-react": "^0.576.0",
@@ -495,6 +496,8 @@
"@unrs/resolver-binding-win32-x64-msvc": ["@unrs/resolver-binding-win32-x64-msvc@1.11.1", "", { "os": "win32", "cpu": "x64" }, "sha512-lrW200hZdbfRtztbygyaq/6jP6AKE8qQN2KvPcJ+x7wiD038YtnYtZ82IMNJ69GJibV7bwL3y9FgK+5w/pYt6g=="],
+ "@vercel/blob": ["@vercel/blob@2.4.0", "", { "dependencies": { "async-retry": "^1.3.3", "is-buffer": "^2.0.5", "is-node-process": "^1.2.0", "throttleit": "^2.1.0", "undici": "^6.23.0" } }, "sha512-ncQ8CRb6XoEAYJwjOTRGpACRT6h/AeY+/33gLyeVxG5BIes27OPm1jmqreF+JHjcTmGhClTP+kBpmyLfbV0xew=="],
+
"accepts": ["accepts@2.0.0", "", { "dependencies": { "mime-types": "^3.0.0", "negotiator": "^1.0.0" } }, "sha512-5cvg6CtKwfgdmVqY1WIiXKc3Q1bkRqGLi+2W/6ao+6Y7gu/RCwRuAhGEzh5B4KlszSuTLgZYuqFqo5bImjNKng=="],
"acorn": ["acorn@8.16.0", "", { "bin": { "acorn": "bin/acorn" } }, "sha512-UVJyE9MttOsBQIDKw1skb9nAwQuR5wuGD3+82K6JgJlm/Y+KI92oNsMNGZCYdDsVtRHSak0pcV5Dno5+4jh9sw=="],
@@ -541,6 +544,8 @@
"async-function": ["async-function@1.0.0", "", {}, "sha512-hsU18Ae8CDTR6Kgu9DYf0EbCr/a5iGL0rytQDobUcdpYOKokk8LEjVphnXkDkgpi0wYVsqrXuP0bZxJaTqdgoA=="],
+ "async-retry": ["async-retry@1.3.3", "", { "dependencies": { "retry": "0.13.1" } }, "sha512-wfr/jstw9xNi/0teMHrRW7dsz3Lt5ARhYNZ2ewpadnhaIp5mbALhOAP+EAdsC7t4Z6wqsDVv9+W6gm1Dk9mEyw=="],
+
"available-typed-arrays": ["available-typed-arrays@1.0.7", "", { "dependencies": { "possible-typed-array-names": "^1.0.0" } }, "sha512-wvUjBtSGN7+7SjNpq/9M2Tg350UZD3q62IFZLbRAR1bSMlCo1ZaeW+BJ+D090e4hIIZLBcTDWe4Mh4jvUDajzQ=="],
"axe-core": ["axe-core@4.11.1", "", {}, "sha512-BASOg+YwO2C+346x3LZOeoovTIoTrRqEsqMa6fmfAV0P+U9mFr9NsyOEpiYvFjbc64NMrSswhV50WdXzdb/Z5A=="],
@@ -885,6 +890,8 @@
"is-boolean-object": ["is-boolean-object@1.2.2", "", { "dependencies": { "call-bound": "^1.0.3", "has-tostringtag": "^1.0.2" } }, "sha512-wa56o2/ElJMYqjCjGkXri7it5FbebW5usLw/nPmCMs5DeZ7eziSYZhSmPRn0txqeW4LnAmQQU7FgqLpsEFKM4A=="],
+ "is-buffer": ["is-buffer@2.0.5", "", {}, "sha512-i2R6zNFDwgEHJyQUtJEk0XFi1i0dPFn/oqjK3/vPCcDeJvW5NQ83V8QbicfF1SupOaB0h8ntgBC2YiE7dfyctQ=="],
+
"is-bun-module": ["is-bun-module@2.0.0", "", { "dependencies": { "semver": "^7.7.1" } }, "sha512-gNCGbnnnnFAUGKeZ9PdbyeGYJqewpmc2aKHUEMO5nQPWU9lOmv7jcmQIv+qHD8fXW6W7qfuCwX4rY9LNRjXrkQ=="],
"is-callable": ["is-callable@1.2.7", "", {}, "sha512-1BC0BVFhS/p0qtw6enp8e+8OD0UrK0oFLztSjNzhcKA3WDuJxxAPXzPuPtKkjEY9UUoEWlX/8fgKeu2S8i9JTA=="],
@@ -1213,6 +1220,8 @@
"restore-cursor": ["restore-cursor@5.1.0", "", { "dependencies": { "onetime": "^7.0.0", "signal-exit": "^4.1.0" } }, "sha512-oMA2dcrw6u0YfxJQXm342bFKX/E4sG9rbTzO9ptUcR/e8A33cHuvStiYOwH7fszkZlZ1z/ta9AAoPk2F4qIOHA=="],
+ "retry": ["retry@0.13.1", "", {}, "sha512-XQBQ3I8W1Cge0Seh+6gjj03LbmRFWuoszgK9ooCpwYIrhhoO80pfq4cUkU5DkknwfOfFteRwlZ56PYOGYyFWdg=="],
+
"rettime": ["rettime@0.10.1", "", {}, "sha512-uyDrIlUEH37cinabq0AX4QbgV4HbFZ/gqoiunWQ1UqBtRvTTytwhNYjE++pO/MjPTZL5KQCf2bEoJ/BJNVQ5Kw=="],
"reusify": ["reusify@1.1.0", "", {}, "sha512-g6QUff04oZpHs0eG5p83rFLhHeV00ug/Yf9nZM6fLeUrPguBTkTQOdpAWWspMh55TZfVQDPaN3NQJfbVRAxdIw=="],
@@ -1319,6 +1328,8 @@
"tapable": ["tapable@2.3.0", "", {}, "sha512-g9ljZiwki/LfxmQADO3dEY1CbpmXT5Hm2fJ+QaGKwSXUylMybePR7/67YW7jOrrvjEgL1Fmz5kzyAjWVWLlucg=="],
+ "throttleit": ["throttleit@2.1.0", "", {}, "sha512-nt6AMGKW1p/70DF/hGBdJB57B8Tspmbp5gfJ8ilhLnt7kkr2ye7hzD6NVG8GGErk2HWF34igrL2CXmNIkzKqKw=="],
+
"tiny-invariant": ["tiny-invariant@1.3.3", "", {}, "sha512-+FbBPE1o9QAYvviau/qC5SE3caw21q3xkvWKBtja5vgqOWIHHJ3ioaq1VPfn/Szqctz2bU/oYeKd9/z5BL+PVg=="],
"tinyexec": ["tinyexec@1.0.2", "", {}, "sha512-W/KYk+NFhkmsYpuHq5JykngiOCnxeVL8v8dFnqxSD8qEEdRfXk1SDM6JzNqcERbcGYj9tMrDQBYV9cjgnunFIg=="],
@@ -1365,6 +1376,8 @@
"unbox-primitive": ["unbox-primitive@1.1.0", "", { "dependencies": { "call-bound": "^1.0.3", "has-bigints": "^1.0.2", "has-symbols": "^1.1.0", "which-boxed-primitive": "^1.1.1" } }, "sha512-nWJ91DjeOkej/TA8pXQ3myruKpKEYgqvpw9lz4OPHj/NWFNluYrjbz9j01CJ8yKQd2g4jFoOkINCTW2I5LEEyw=="],
+ "undici": ["undici@6.26.0", "", {}, "sha512-4yqz8a3n5HmGTlsbADNtr/dJlhkh/55Rq798G6ibiULcXbDtaLpTl1pvdqcbFfeoj3iSi52lePFM7h9H21cw/A=="],
+
"undici-types": ["undici-types@6.21.0", "", {}, "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ=="],
"unicorn-magic": ["unicorn-magic@0.3.0", "", {}, "sha512-+QBBXBCvifc56fsbuxZQ6Sic3wqqc3WWaqxs58gvJrcOuN83HGTCwz3oS5phzU9LthRNE9VrJCFCLUgHeeFnfA=="],
diff --git a/package.json b/package.json
--- a/package.json
+++ b/package.json
@@ -20,6 +20,7 @@
"test": "bun test"
},
"dependencies": {
+ "@vercel/blob": "^2.4.0",
"class-variance-authority": "^0.7.1",
"clsx": "^2.1.1",
"lucide-react": "^0.576.0",
diff --git a/src/lib/storage/blob-object-store.test.ts b/src/lib/storage/blob-object-store.test.ts
new file mode 100644
--- /dev/null
+++ b/src/lib/storage/blob-object-store.test.ts
@@ -1,0 +1,180 @@
+import { beforeEach, describe, expect, mock, test } from "bun:test";
+import type { Agent, Initiative } from "@/lib/types";
+
+// In-memory fake of the @vercel/blob SDK surface that BlobObjectStore uses. It
+// faithfully models the behaviors the code depends on: not-found semantics,
+// conditional create (allowOverwrite:false), and folded prefix listing. This
+// lets us validate BlobObjectStore's key/prefix translation end-to-end without a
+// live Blob store.
+const blobs = new Map<string, string>();
+
+class FakeBlobNotFoundError extends Error {
+ constructor() {
+ super("Blob not found");
+ this.name = "BlobNotFoundError";
+ }
+}
+
+function toBlob(pathname: string) {
+ return {
+ pathname,
+ url: `https://fake.blob/${pathname}`,
+ downloadUrl: `https://fake.blob/${pathname}?download=1`,
+ size: (blobs.get(pathname) ?? "").length,
+ uploadedAt: new Date(),
+ etag: "etag",
+ };
+}
+
+mock.module("@vercel/blob", () => ({
+ BlobNotFoundError: FakeBlobNotFoundError,
+ async put(pathname: string, body: string, opts: { allowOverwrite?: boolean }) {
+ if (opts?.allowOverwrite === false && blobs.has(pathname)) {
+ throw new Error(`blob already exists: ${pathname}`);
+ }
+ blobs.set(pathname, typeof body === "string" ? body : String(body));
+ return toBlob(pathname);
+ },
+ async get(pathname: string) {
+ if (!blobs.has(pathname)) return null;
+ return { statusCode: 200, stream: new Response(blobs.get(pathname)).body, headers: new Headers(), blob: toBlob(pathname) };
+ },
+ async head(pathname: string) {
+ if (!blobs.has(pathname)) throw new FakeBlobNotFoundError();
+ return toBlob(pathname);
+ },
+ async del(target: string | string[]) {
+ for (const t of Array.isArray(target) ? target : [target]) blobs.delete(t);
+ },
+ async copy(from: string, to: string) {
+ blobs.set(to, blobs.get(from) ?? "");
+ return toBlob(to);
+ },
+ async list(opts: { prefix?: string; mode?: "expanded" | "folded" } = {}) {
+ const prefix = opts.prefix ?? "";
+ const keys = [...blobs.keys()].filter((k) => k.startsWith(prefix));
+ if (opts.mode === "folded") {
+ const direct: string[] = [];
+ const folders = new Set<string>();
+ for (const k of keys) {
+ const rest = k.slice(prefix.length);
+ if (rest.includes("/")) folders.add(`${prefix}${rest.split("/")[0]}/`);
+ else direct.push(k);
+ }
+ return { blobs: direct.map(toBlob), folders: [...folders], hasMore: false, cursor: undefined };
+ }
+ return { blobs: keys.map(toBlob), hasMore: false, cursor: undefined };
+ },
+}));
+
+const { BlobObjectStore } = await import("./blob-object-store");
+const { DocumentStorage } = await import("./document-storage");
+
+let storage: InstanceType<typeof DocumentStorage>;
+
+beforeEach(() => {
+ blobs.clear();
+ storage = new DocumentStorage(new BlobObjectStore());
+});
+
+function intro(sessionKey: string) {
+ return { intro: { runtime: "claude-code", sessionKey, thread: { id: sessionKey, name: sessionKey }, workingDirectory: "/tmp" } };
+}
+
+async function seed(): Promise<{ boardId: string; agent: Agent; initiative: Initiative }> {
+ const board = await storage.createBoard({ name: "Board", description: "" });
+ const agent = await storage.createAgent(board.id, { name: "Worker", description: "", metadata: intro("s1") });
+ const initiative = await storage.createInitiative(board.id, { name: "Init", description: "" });
+ return { boardId: board.id, agent, initiative };
+}
+
+describe("BlobObjectStore via DocumentStorage", () => {
+ test("create/list/get round-trip through blob keys", async () => {
+ const { boardId, agent, initiative } = await seed();
+ const task = await storage.createTask(
+ boardId,
+ initiative.id,
+ { title: "Do thing", description: "", assigneeAgentId: agent.id, priority: "medium", tags: [] },
+ agent.id,
+ );
+
+ expect(await storage.getTask(boardId, initiative.id, task.id)).not.toBeNull();
+ expect((await storage.listTasks(boardId, initiative.id)).length).toBe(1);
+ expect((await storage.listInitiatives(boardId)).map((i) => i.id)).toContain(initiative.id);
+ expect((await storage.listAgents(boardId)).length).toBe(1);
+ // Deterministic pathname (no random suffix) at the expected key.
+ expect(blobs.has(`boards/${boardId}/initiatives/${initiative.id}/tasks/${task.id}.json`)).toBe(true);
+ });
+
+ test("listChildren folding: initiatives dir yields only the .json entities", async () => {
+ const { boardId, initiative } = await seed();
+ // The initiative file and its subtree (tasks/plans) coexist under initiatives/.
+ const initiatives = await storage.listInitiatives(boardId);
+ expect(initiatives.map((i) => i.id)).toEqual([initiative.id]);
+ });
+
+ test("unique-id reservation handles colliding slugs", async () => {
+ const { boardId, agent, initiative } = await seed();
+ const results = await Promise.all(
+ Array.from({ length: 10 }, () =>
+ storage.createTask(
+ boardId,
+ initiative.id,
+ { title: "Same", description: "", assigneeAgentId: null, priority: "medium", tags: [] },
+ agent.id,
+ ),
+ ),
+ );
+ expect(new Set(results.map((t) => t.id)).size).toBe(10);
+ });
+
+ test("deleteInitiative removes the file and the whole subtree", async () => {
+ const { boardId, agent, initiative } = await seed();
+ await storage.createTask(
+ boardId,
+ initiative.id,
+ { title: "T", description: "", assigneeAgentId: null, priority: "medium", tags: [] },
+ agent.id,
+ );
+ expect(await storage.deleteInitiative(boardId, initiative.id)).toBe(true);
+ const leftover = [...blobs.keys()].filter((k) => k.includes(`/initiatives/${initiative.id}`));
+ expect(leftover).toEqual([]);
+ });
+
+ test("activity events are stored and listed", async () => {
+ const { boardId, agent, initiative } = await seed();
+ await storage.createTask(
+ boardId,
+ initiative.id,
+ { title: "T", description: "", assigneeAgentId: null, priority: "medium", tags: [] },
+ agent.id,
+ );
+ const activity = await storage.listActivity(boardId);
+ expect(activity.map((e) => e.type)).toContain("task.created");
+ });
+
+ test("corrupt blob is quarantined and skipped on list", async () => {
+ const { boardId, agent, initiative } = await seed();
+ const bad = await storage.createTask(
+ boardId,
+ initiative.id,
+ { title: "Bad", description: "", assigneeAgentId: null, priority: "medium", tags: [] },
+ agent.id,
+ );
+ const key = `boards/${boardId}/initiatives/${initiative.id}/tasks/${bad.id}.json`;
+ blobs.set(key, "{ not valid json");
+
+ const tasks = await storage.listTasks(boardId, initiative.id);
+ expect(tasks.map((t) => t.id)).not.toContain(bad.id);
+ // Moved out of the tasks dir into the quarantine prefix.
+ expect(blobs.has(key)).toBe(false);
+ expect([...blobs.keys()].some((k) => k.startsWith(".quarantine/"))).toBe(true);
+ });
+
+ test("getBoardSummary counts via blob backend", async () => {
+ const { boardId } = await seed();
+ const summary = await storage.getBoardSummary(boardId);
+ expect(summary?.agentCount).toBe(1);
+ expect(summary?.initiativeCount).toBe(1);
+ });
+});
diff --git a/src/lib/storage/blob-object-store.ts b/src/lib/storage/blob-object-store.ts
new file mode 100644
--- /dev/null
+++ b/src/lib/storage/blob-object-store.ts
@@ -1,0 +1,187 @@
+import { BlobNotFoundError, copy, del, get, head, list, put } from "@vercel/blob";
+import { randomUUID } from "crypto";
+import { timestamp } from "@/lib/utils";
+import type { ObjectStore } from "./object-store";
+
+const ACCESS = "private" as const;
+const QUARANTINE_PREFIX = ".quarantine";
+
+function isNotFound(err: unknown): boolean {
+ return err instanceof BlobNotFoundError;
+}
+
+/**
+ * Vercel Blob-backed ObjectStore for production / serverless deployments, where the
+ * filesystem is read-only. Keys map directly to blob pathnames.
+ *
+ * Consistency notes:
+ * - Reads use `useCache: false` to avoid stale CDN responses after a write.
+ * - `createIfAbsent` relies on `allowOverwrite: false` for atomic reservation.
+ * - There is no cross-instance lock, so concurrent writers to the *same* key
+ * are last-write-wins (each write is itself atomic). Unique-id reservation
+ * prevents the common create collision; updates to one record assume a single
+ * logical writer, which matches the app's per-entity access pattern.
+ */
+export class BlobObjectStore implements ObjectStore {
+ private async readText(key: string): Promise<string | null> {
+ let res;
+ try {
+ res = await get(key, { access: ACCESS, useCache: false });
+ } catch (err) {
+ if (isNotFound(err)) return null;
+ throw err;
+ }
+ if (!res || res.statusCode !== 200 || !res.stream) return null;
+ return await new Response(res.stream).text();
+ }
+
+ private async quarantine(key: string, reason: string): Promise<void> {
+ try {
+ const dest = `${QUARANTINE_PREFIX}/${timestamp().replace(/[:.]/g, "-")}-${randomUUID().slice(0, 8)}-${key.replace(/\//g, "_")}`;
+ // Only delete the source after the copy succeeds — matching FsObjectStore,
+ // where a failed rename leaves the original intact. If copy throws we fall
+ // to the catch below and the corrupt blob is preserved for inspection.
+ await copy(key, dest, { access: ACCESS });
+ await del(key);
+ console.error(`Quarantined corrupt blob ${key} -> ${dest} (${reason})`);
+ } catch (err) {
+ console.error(`Failed to quarantine blob ${key}:`, err);
+ }
+ }
+
+ async get<T>(key: string, validate?: (value: unknown) => boolean): Promise<T | null> {
+ const content = await this.readText(key);
+ if (content === null) return null;
+
+ let parsed: unknown;
+ try {
+ parsed = JSON.parse(content);
+ } catch (err) {
+ console.error(`Corrupted JSON in blob ${key}:`, (err as Error).message);
+ await this.quarantine(key, "invalid JSON");
+ return null;
+ }
+
+ if (validate && !validate(parsed)) {
+ console.error(`Schema validation failed for blob ${key}; skipping`);
+ return null;
+ }
+
+ return parsed as T;
+ }
+
+ async put(key: string, data: unknown): Promise<void> {
+ await put(key, JSON.stringify(data, null, 2), {
+ access: ACCESS,
+ addRandomSuffix: false,
+ allowOverwrite: true,
+ contentType: "application/json",
+ });
+ }
+
+ async createIfAbsent(key: string): Promise<boolean> {
+ try {
+ await put(key, "{}", {
+ access: ACCESS,
+ addRandomSuffix: false,
+ allowOverwrite: false,
+ contentType: "application/json",
+ });
+ return true;
+ } catch (err) {
+ // allowOverwrite:false rejects when the key exists. Disambiguate a real
+ // conflict from a transient error by confirming the blob is present.
+ if (await this.exists(key)) return false;
+ throw err;
+ }
+ }
+
+ async exists(key: string): Promise<boolean> {
+ try {
+ await head(key, { access: ACCESS });
+ return true;
+ } catch (err) {
+ if (isNotFound(err)) return false;
+ throw err;
+ }
+ }
+
+ async delete(key: string): Promise<void> {
+ try {
+ await del(key);
+ } catch (err) {
+ if (!isNotFound(err)) throw err;
+ }
+ }
+
+ private async listPathnames(prefix: string): Promise<string[]> {
+ const pathnames: string[] = [];
+ let cursor: string | undefined;
+ do {
+ const res = await list({ prefix, cursor, limit: 1000 });
+ for (const blob of res.blobs) pathnames.push(blob.pathname);
+ cursor = res.hasMore ? res.cursor : undefined;
+ } while (cursor);
+ return pathnames;
+ }
+
+ async deletePrefix(prefix: string): Promise<void> {
+ const p = prefix.endsWith("/") ? prefix : `${prefix}/`;
+ const pathnames = await this.listPathnames(p);
+ // del accepts up to 1000 urls/pathnames per call.
+ for (let i = 0; i < pathnames.length; i += 1000) {
+ await del(pathnames.slice(i, i + 1000));
+ }
+ }
+
+ async listChildren(prefix: string): Promise<string[]> {
+ const p = prefix.endsWith("/") ? prefix : `${prefix}/`;
+ const names = new Set<string>();
+ let cursor: string | undefined;
+ do {
+ const res = await list({ prefix: p, mode: "folded", cursor, limit: 1000 });
+ for (const blob of res.blobs) {
+ const rel = blob.pathname.slice(p.length);
+ if (rel && !rel.includes("/")) names.add(rel);
+ }
+ for (const folder of res.folders) {
+ const rel = folder.slice(p.length).replace(/\/$/, "");
+ if (rel) names.add(rel);
+ }
+ cursor = res.hasMore ? res.cursor : undefined;
+ } while (cursor);
+ return [...names];
+ }
+
+ async appendEvent(eventsKey: string, bucket: string, record: unknown): Promise<void> {
+ // One blob per event avoids the read-modify-write race an append-to-file
+ // approach would suffer on object storage.
+ await put(`${eventsKey}/${bucket}/${randomUUID()}.json`, JSON.stringify(record), {
+ access: ACCESS,
+ addRandomSuffix: false,
+ allowOverwrite: true,
+ contentType: "application/json",
+ });
+ }
+
+ async listEventBuckets(eventsKey: string): Promise<string[]> {
+ // Buckets are the immediate subfolders of the events collection.
+ return this.listChildren(eventsKey);
+ }
+
+ async readEventBucket(eventsKey: string, bucket: string): Promise<unknown[]> {
+ const pathnames = await this.listPathnames(`${eventsKey}/${bucket}/`);
+ const records = await Promise.all(
+ pathnames.map(async (pathname) => {
+ const content = await this.readText(pathname);
+ if (content === null) return null;
+ try {
+ return JSON.parse(content);
+ } catch {
+ return null;
+ }
+ }),
+ );
+ return records.filter((r) => r !== null);
+ }
+}
diff --git a/src/lib/storage/document-storage.test.ts b/src/lib/storage/document-storage.test.ts
new file mode 100644
--- /dev/null
+++ b/src/lib/storage/document-storage.test.ts
@@ -1,0 +1,130 @@
+import { beforeEach, describe, expect, test } from "bun:test";
+import { DocumentStorage } from "./document-storage";
+import { InMemoryObjectStore } from "./memory-object-store";
+import type { Agent, Initiative } from "@/lib/types";
+
+// Exercises the backend-agnostic business logic against an in-memory ObjectStore.
+// The same DocumentStorage runs on FsObjectStore (local) and BlobObjectStore (prod), so
+// these guarantees hold for every backend.
+let storage: DocumentStorage;
+
+beforeEach(() => {
+ storage = new DocumentStorage(new InMemoryObjectStore());
+});
+
+function intro(sessionKey: string) {
+ return { intro: { runtime: "claude-code", sessionKey, thread: { id: sessionKey, name: sessionKey }, workingDirectory: "/tmp" } };
+}
+
+async function seed(): Promise<{ boardId: string; agent: Agent; initiative: Initiative }> {
+ const board = await storage.createBoard({ name: "Board", description: "" });
+ const agent = await storage.createAgent(board.id, { name: "Worker", description: "", metadata: intro("s1") });
+ const initiative = await storage.createInitiative(board.id, { name: "Init", description: "" });
+ return { boardId: board.id, agent, initiative };
+}
+
+describe("DocumentStorage core flow", () => {
+ test("create/list/get board, initiative, task round-trip", async () => {
+ const { boardId, agent, initiative } = await seed();
+ const task = await storage.createTask(
+ boardId,
+ initiative.id,
+ { title: "Do thing", description: "", assigneeAgentId: agent.id, priority: "high", tags: ["x"] },
+ agent.id,
+ );
+ expect(task.assigneeAgentIds).toEqual([agent.id]);
+
+ const fetched = await storage.getTask(boardId, initiative.id, task.id);
+ expect(fetched?.title).toBe("Do thing");
+
+ const boards = await storage.listBoards();
+ expect(boards.map((b) => b.id)).toContain(boardId);
+
+ const tasks = await storage.listTasks(boardId, initiative.id);
+ expect(tasks.length).toBe(1);
+ });
+
+ test("concurrent identical-title creates get unique ids", async () => {
+ const { boardId, agent, initiative } = await seed();
+ const results = await Promise.all(
+ Array.from({ length: 20 }, () =>
+ storage.createTask(
+ boardId,
+ initiative.id,
+ { title: "Same", description: "", assigneeAgentId: null, priority: "medium", tags: [] },
+ agent.id,
+ ),
+ ),
+ );
+ expect(new Set(results.map((t) => t.id)).size).toBe(20);
+ expect((await storage.listTasks(boardId, initiative.id)).length).toBe(20);
+ });
+
+ test("status change to in_progress wires up the agent pointer", async () => {
+ const { boardId, agent, initiative } = await seed();
+ const task = await storage.createTask(
+ boardId,
+ initiative.id,
+ { title: "T", description: "", assigneeAgentId: agent.id, priority: "medium", tags: [] },
+ agent.id,
+ );
+ await storage.updateTask(boardId, initiative.id, task.id, { status: "in_progress" }, agent.id);
+ const after = await storage.getAgent(boardId, agent.id);
+ expect(after?.currentTaskId).toBe(task.id);
+ expect(after?.currentInitiativeId).toBe(initiative.id);
+ });
+
+ test("deleting an initiative removes its task subtree", async () => {
+ const { boardId, agent, initiative } = await seed();
+ await storage.createTask(
+ boardId,
+ initiative.id,
+ { title: "T", description: "", assigneeAgentId: null, priority: "medium", tags: [] },
+ agent.id,
+ );
+ expect(await storage.deleteInitiative(boardId, initiative.id)).toBe(true);
+ expect(await storage.getInitiative(boardId, initiative.id)).toBeNull();
+ expect(await storage.listTasks(boardId, initiative.id)).toEqual([]);
+ });
+
+ test("duplicate agent name/session are rejected", async () => {
+ const board = await storage.createBoard({ name: "B", description: "" });
+ await storage.createAgent(board.id, { name: "Dup", description: "", metadata: intro("k1") });
+ await expect(
+ storage.createAgent(board.id, { name: "Dup", description: "", metadata: intro("k2") }),
+ ).rejects.toThrow(/already exists/);
+ });
+
+ test("activity log records and lists events newest-first", async () => {
+ const { boardId, agent, initiative } = await seed();
+ await storage.createTask(
+ boardId,
+ initiative.id,
+ { title: "T", description: "", assigneeAgentId: null, priority: "medium", tags: [] },
+ agent.id,
+ );
+ const activity = await storage.listActivity(boardId);
+ const types = activity.map((e) => e.type);
+ expect(types).toContain("task.created");
+ expect(types).toContain("agent.registered");
+ });
+
+ test("getBoardSummary counts entities and isolates a failing section", async () => {
+ const { boardId } = await seed();
+
+ const summary = await storage.getBoardSummary(boardId);
+ expect(summary?.agentCount).toBe(1);
+ expect(summary?.initiativeCount).toBe(1);
+
+ class Broken extends DocumentStorage {
+ async listAllBoardTasks(): Promise<never> {
+ throw new Error("boom");
+ }
+ }
+ const broken = new Broken(new InMemoryObjectStore());
+ const b = await broken.createBoard({ name: "B2", description: "" });
+ const brokenSummary = await broken.getBoardSummary(b.id);
+ expect(brokenSummary).not.toBeNull();
+ expect(brokenSummary!.taskCount).toBe(0);
+ });
+});
diff --git a/src/lib/storage/document-storage.ts b/src/lib/storage/document-storage.ts
new file mode 100644
--- /dev/null
+++ b/src/lib/storage/document-storage.ts
@@ -1,0 +1,1393 @@
+import { randomUUID } from "crypto";
+import type {
+ ActivityEvent,
+ Agent,
+ Board,
+ BoardSummary,
+ Initiative,
+ Plan,
+ PlanStep,
+ Project,
+ Task,
+ TaskStatus,
+} from "@/lib/types";
+import type { Storage } from "./index";
+import type { ObjectStore } from "./object-store";
+import { slugify, timestamp } from "@/lib/utils";
+
+type AgentIntro = {
+ runtime: string;
+ sessionKey: string;
+ model: string;
+ thread: {
+ id: string;
+ name: string;
+ source?: string;
+ };
+ workingDirectory: string;
+ host?: {
+ hostname: string;
+ localIp: string;
+ };
+};
+
+export class AgentRegistrationError extends Error {
+ code: string;
+
+ constructor(code: string, message: string) {
+ super(message);
+ this.name = "AgentRegistrationError";
+ this.code = code;
+ }
+}
+
+function isRecord(value: unknown): value is Record<string, unknown> {
+ return typeof value === "object" && value !== null && !Array.isArray(value);
+}
+
+function normalizeAgentName(name: string): string {
+ return name.trim();
+}
+
+function canonicalAgentName(name: string): string {
+ return normalizeAgentName(name).toLowerCase();
+}
+
+function normalizeText(value: unknown): string {
+ return typeof value === "string" ? value.trim() : "";
+}
+
+function inferModelFromRuntime(runtime: string): string {
+ const key = runtime.toLowerCase();
+ if (key.includes("codex") || key.includes("openai")) return "codex";
+ if (key.includes("claude") || key.includes("anthropic")) return "claude";
+ if (key.includes("cursor")) return "cursor";
+ return "unknown-model";
+}
+
+function extractThreadContext(intro: Record<string, unknown>): AgentIntro["thread"] | null {
+ const threadRaw = intro.thread;
+
+ if (typeof threadRaw === "string") {
+ const id = normalizeText(threadRaw);
+ const name = normalizeText(intro.threadName) || id;
+ if (!id) return null;
+ return { id, name };
+ }
+
+ if (!isRecord(threadRaw)) return null;
+
+ const id = normalizeText(threadRaw.id) || normalizeText(intro.threadId);
+ const name = normalizeText(threadRaw.name) || normalizeText(intro.threadName) || id;
+ const source = normalizeText(threadRaw.source);
+ if (!id) return null;
+ return source ? { id, name, source } : { id, name };
+}
+
+function extractAgentIntro(metadata: unknown): AgentIntro | null {
+ if (!isRecord(metadata)) return null;
+ const intro = metadata.intro;
+ if (!isRecord(intro)) return null;
+
+ const runtime = normalizeText(intro.runtime) || "unknown-runtime";
+ const sessionKey = normalizeText(intro.sessionKey) || normalizeText(intro.instanceKey);
+ const model = normalizeText(intro.model) || inferModelFromRuntime(runtime);
+ const thread =
+ extractThreadContext(intro) ||
+ (() => {
+ const fallbackId = normalizeText(intro.threadId) || "unknown-thread";
+ const fallbackName = normalizeText(intro.threadName) || fallbackId;
+ return { id: fallbackId, name: fallbackName, source: runtime };
+ })();
+ const workingDirectory = normalizeText(intro.workingDirectory) || "unknown-working-directory";
+ const hostRaw = isRecord(intro.host) ? intro.host : {};
+ const hostHostname = normalizeText(hostRaw.hostname);
+ const hostLocalIp = normalizeText(hostRaw.localIp);
+ const host =
+ hostHostname || hostLocalIp
+ ? {
+ hostname: hostHostname || "unknown-host",
+ localIp: hostLocalIp || "unknown-local-ip",
+ }
+ : undefined;
+
+ if (!sessionKey) return null;
+ return { runtime, sessionKey, model, thread, workingDirectory, host };
+}
+
+// A stored entity is only usable if it parsed to an object carrying a string id.
+// Anything else is either a transient reservation placeholder or structurally
+// unusable, and is skipped on read.
+export function isEntityRecord(value: unknown): boolean {
+ return isRecord(value) && typeof (value as { id?: unknown }).id === "string" && (value as { id: string }).id.length > 0;
+}
+
+// Defensive string compare for sort keys that may be missing on partially
+// written or legacy records — `undefined.localeCompare` would throw.
+function compareStr(a: string | undefined, b: string | undefined): number {
+ return (a ?? "").localeCompare(b ?? "");
+}
+
+function normalizeAssignees(task: Partial<Task>): string[] {
+ if (task.assigneeAgentIds && task.assigneeAgentIds.length > 0) {
+ return Array.from(new Set(task.assigneeAgentIds.filter(Boolean)));
+ }
+ if (task.assigneeAgentId) {
+ return [task.assigneeAgentId];
+ }
+ return [];
+}
+
+function normalizeTask(task: Task, fallbackInitiativeId?: string): Task {
+ const initiativeId = task.initiativeId || task.projectId || fallbackInitiativeId || "general";
+ const assigneeAgentIds = normalizeAssignees(task);
+ const assigneeAgentId = task.assigneeAgentId || assigneeAgentIds[0] || null;
+
+ return {
+ ...task,
+ initiativeId,
+ projectId: task.projectId || initiativeId,
+ planId: task.planId || null,
+ planStepId: task.planStepId || null,
+ assigneeAgentId,
+ assigneeAgentIds,
+ deliverables: task.deliverables || [],
+ };
+}
+
+// --- Key helpers (POSIX-style relative keys; identical layout across backends) ---
+
+function boardDir(boardId: string): string {
+ return `boards/${boardId}`;
+}
+function boardFile(boardId: string): string {
+ return `${boardDir(boardId)}/board.json`;
+}
+function agentsDir(boardId: string): string {
+ return `${boardDir(boardId)}/agents`;
+}
+function agentFile(boardId: string, agentId: string): string {
+ return `${agentsDir(boardId)}/${agentId}.json`;
+}
+function initiativesDir(boardId: string): string {
+ return `${boardDir(boardId)}/initiatives`;
+}
+function initiativeFile(boardId: string, initiativeId: string): string {
+ return `${initiativesDir(boardId)}/${initiativeId}.json`;
+}
+function initiativeSubdir(boardId: string, initiativeId: string): string {
+ return `${initiativesDir(boardId)}/${initiativeId}`;
+}
+function tasksDir(boardId: string, initiativeId: string): string {
+ return `${initiativeSubdir(boardId, initiativeId)}/tasks`;
+}
+function taskFile(boardId: string, initiativeId: string, taskId: string): string {
+ return `${tasksDir(boardId, initiativeId)}/${taskId}.json`;
+}
+function plansDir(boardId: string, initiativeId: string): string {
+ return `${initiativeSubdir(boardId, initiativeId)}/plans`;
+}
+function planFile(boardId: string, initiativeId: string, planId: string): string {
+ return `${plansDir(boardId, initiativeId)}/${planId}.json`;
+}
+function planSubdir(boardId: string, initiativeId: string, planId: string): string {
+ return `${plansDir(boardId, initiativeId)}/${planId}`;
+}
+function planStepsDir(boardId: string, initiativeId: string, planId: string): string {
+ return `${planSubdir(boardId, initiativeId, planId)}/steps`;
+}
+function planStepFile(boardId: string, initiativeId: string, planId: string, stepId: string): string {
+ return `${planStepsDir(boardId, initiativeId, planId)}/${stepId}.json`;
+}
+function eventsDir(boardId: string): string {
... diff truncated: showing 800 of 3901 linesYou can send follow-ups to the cloud agent here.
"KvStore" read like the Vercel KV product, but the production backend is Vercel Blob. Rename the internal primitive interface and its implementations to ObjectStore to remove the ambiguity — no behavior change. - KvStore -> ObjectStore; FsKvStore -> FsObjectStore; BlobKvStore -> BlobObjectStore; InMemoryKvStore -> InMemoryObjectStore. - Files: kv.ts -> object-store.ts, fs-kv.ts -> fs-object-store.ts, blob-kv.ts -> blob-object-store.ts, memory-kv.ts -> memory-object-store.ts. - DocumentStorage's internal field kv -> store. https://claude.ai/code/session_014r13EsBf8Kz8YGfD6VxDtS
…b-storage # Conflicts: # src/lib/storage/blob-object-store.ts
Without the access option, head() may use the wrong access mode for private blobs, causing exists() to throw instead of returning false.
A prior fix passed `{ access: "private" }` to @vercel/blob `head()` in
exists(), but head's options type is BlobCommandOptions, which has no
`access` field — TS2353, failing `next build` (and the Vercel deploy).
head() is token-scoped and resolves private blobs via
BLOB_READ_WRITE_TOKEN, so it needs no access option. Revert to head(key)
and document why, so the call isn't "re-fixed" into a build break again.
https://claude.ai/code/session_014r13EsBf8Kz8YGfD6VxDtS
Limit readEventBucket to 20 concurrent HTTP requests per batch instead of firing one request per event blob simultaneously. Remove the static BlobObjectStore import from fs-storage so the Vercel Blob SDK is only loaded via dynamic import when the blob backend is selected, and make getStorage async to support lazy initialization.
| } catch (err) { | ||
| console.error(`Failed to quarantine blob ${key}:`, err); | ||
| } | ||
| } |
There was a problem hiding this comment.
Quarantine copy may require URL not pathname
Low Severity
BlobObjectStore.quarantine passes a blob pathname (e.g. boards/x/agents/foo.json) as the first argument to copy(). The @vercel/blob SDK's copy function signature is copy(fromUrl, toPathname, options) where the source is typically a full blob URL. If the SDK doesn't accept a bare pathname here, quarantine silently fails, leaving corrupt blobs in place to be re-detected on every read.
Reviewed by Cursor Bugbot for commit 00c2d7e. Configure here.
There was a problem hiding this comment.
Not a bug - the @vercel/blob SDK's copy() function explicitly accepts either a URL or a pathname as the first argument. According to the Vercel Blob SDK documentation, the signature is:
copy(fromUrlOrPathname, toPathname, options)
The first parameter is described as "A blob URL or pathname identifying an already existing blob."
The current implementation using a pathname is correct and matches how other operations in this file use pathnames (e.g., get(), del()).
There was a problem hiding this comment.
Bugbot Autofix determined this is a false positive.
@vercel/blob v2.4.0 documents and types copy's first argument as fromUrlOrPathname, and the existing quarantine test confirms pathname-based copy works.
You can send follow-ups to the cloud agent here.
Store activity events as one ndjson blob per day (matching the filesystem backend) instead of one blob per event, reducing activity reads from O(events) to O(days). Also apply filters while scanning buckets and stop early once enough matching events are collected.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
Reviewed by Cursor Bugbot for commit 33c374e. Configure here.
| contentType: "application/x-ndjson", | ||
| }); | ||
| }); | ||
| } |
There was a problem hiding this comment.
Blob appendEvent uses racy read-modify-write across instances
High Severity
BlobObjectStore.appendEvent performs a read-modify-write (read existing blob content, concatenate new event, write entire blob back) protected only by an in-process lock (withAppendLock). On Vercel serverless — the target environment for this backend — concurrent requests execute in separate isolates/processes, so the lock provides no cross-instance protection. Two concurrent events for the same board on the same day will both read the same content, each append their event, and one write will silently overwrite the other's addition, causing activity event data loss. The PR description states "one blob per event (avoids a read-modify-write append race)" but the implementation does the opposite.
Reviewed by Cursor Bugbot for commit 33c374e. Configure here.
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
| const key = this.eventFile(eventsKey, bucket); | ||
| // Serialize appends to the same daily file so concurrent events cannot | ||
| // interleave partial lines — matching FsObjectStore. | ||
| await withAppendLock(key, async () => { |
There was a problem hiding this comment.
Read-modify-write pattern in appendEvent causes data loss when concurrent serverless instances write to the same daily NDJSON bucket.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Production runs on Vercel, whose filesystem is read-only — so the JSON-file storage can't persist there (the root cause behind "create board does nothing"). This adds a durable Vercel Blob backend, selected at runtime, without duplicating the storage logic.
Architecture
The ~1400 lines of storage logic are refactored, not duplicated:
KvStore— a small primitive interface (get/put/createIfAbsent/exists/delete/deletePrefix/listChildren+ append-only events).DocumentStorage— all business logic (validation, normalization, activity logging, agent-pointer sync, unique-id reservation, section-isolated summaries) moved here, speaking onlyKvStore. Backend-agnostic.FsKvStore— the existing filesystem backend (atomic writes, per-path write serialization, corrupt-file quarantine, ndjson event logs). Behavior unchanged.BlobKvStore— new@vercel/blobbackend: deterministic pathnames,privateaccess, fresh reads (useCache:false),allowOverwrite:falsefor atomic unique-id reservation, folded prefix listing for direct children, and one blob per event (avoids a read-modify-write append race).getStorage()selects the backend:AGENTBOARD_STORAGE=blob|fsforces it; otherwise a presentBLOB_READ_WRITE_TOKEN(Vercel Blob's env var) selects Blob, with the filesystem as the local default.FsStorage/getStorage/AgentRegistrationErrorpublic API preserved, so no API route changes needed.Tests (20 passing across 3 files)
fs-storage.test.ts— unchanged; the regression guard proving the refactor preserved FS behavior.document-storage.test.ts— business logic against an in-memoryKvStore(guarantees hold for every backend).blob-kv.test.ts—DocumentStoragedriven against a faithful in-memory fake of the@vercel/blobSDK (folded listing, conditional create, not-found semantics), validatingBlobKvStore's key/prefix translation, unique-id reservation, subtree deletion, activity, and corrupt-blob quarantine end-to-end.tsc --noEmit,eslint, andbun testall pass.Notes / caveats
BLOB_READ_WRITE_TOKENin the Vercel environment (you indicated the store is already provisioned). No secrets are needed in code — the SDK reads the env var.https://claude.ai/code/session_014r13EsBf8Kz8YGfD6VxDtS
Generated by Claude Code
Note
Medium Risk
Touches all persistence paths and production data durability; concurrent writes to the same blob are last-write-wins, so a preview smoke test on real Blob is advisable.
Overview
Adds durable storage on Vercel by introducing an
@vercel/blob-backed backend while keeping local filesystem storage for dev.Storage is refactored behind an
ObjectStoreprimitive and a sharedDocumentStoragelayer that holds all board/agent/task/plan/activity logic (moved out of the monolithicFsStorage).FsObjectStorepreserves the prior on-disk behavior;BlobObjectStoremaps the same key layout to private blobs with folded listing,createIfAbsentfor id reservation, quarantine for bad JSON, and serialized NDJSON appends.getStorage()is now async and picks the backend viaAGENTBOARD_STORAGEorBLOB_READ_WRITE_TOKEN, with a cached singleton. All API routes andskill.mduseawait getStorage(). New tests coverDocumentStorageon an in-memory store and blob behavior via a mocked@vercel/blobSDK.Reviewed by Cursor Bugbot for commit 33c374e. Bugbot is set up for automated code reviews on this repo. Configure here.