refactor(storage): move the S3 protocol down to @internal/s3-protocol; add a disk-backed object store by wmadden-electric · Pull Request #159 · prisma/composer

wmadden-electric · 2026-07-22T18:45:13Z

# What a bucket looks like in local dev, after this PR — plain files on disk:
.prisma-composer/dev/buckets/uploads/
├── avatars/alice.png                  # the object itself — browse it, open it
└── .meta/avatars/alice.png.json      # {"contentType":"image/png","etag":"\"9f86d0…\""}

# Drop a file into the directory and it IS an object. The S3 wire protocol
# (SigV4 auth, presigned URLs, pagination) sits on top, unchanged.

The decision: move the S3 protocol implementation down a package layer, and give it a disk-backed store.

Context in one breath: we're building prisma-composer dev (ADR-0041, whose docs ride on this branch), which runs a whole Composer app locally with no cloud credentials. That work ships as a stack of PRs; this is the first layer, and it merges independently.

What moves, and why

Local dev needs a bucket emulator: a real S3 endpoint over local disk, because app code talks to buckets through real S3 SDKs and anything less than the actual wire protocol is a mock that lies. We already have a battle-tested implementation — the storage module serves S3 traffic with it in production today. The problem is where it lives. The codebase's packages are layered, and a lower layer may never import a higher one (CI enforces this). The protocol code sits in the modules layer, at the top; the dev machinery lives below it and cannot reach up.

So the protocol pieces move down, byte-identical, into a new bottom-layer package @internal/s3-protocol: the request handler, the SigV4 signature verifier (including presigned URLs), the ObjectStore storage interface, and the in-memory store. The storage module now imports them from there. Its public exports and behavior are unchanged, and its test bodies are untouched — only their import lines moved.

Beside the moved code lands fsStore, the new disk-backed ObjectStore shown above. Objects are ordinary files at their key paths, because inspectable state is half the value of local dev: you can open . a bucket, and a file you drop in is a real object (its metadata sidecar is written lazily on first read). Writes go temp-then-rename so a concurrent read never sees a half-written object. Key validation lives in the store itself, so no request can ever write outside its bucket directory. One shared key-pair minting function also moves in, so deploy credentials and future emulator credentials come from a single implementation.

Tested

The moved files verified byte-identical against their originals. The storage module's 47 tests pass unmodified. The disk store's contract is covered directly: dropped-file adoption, path-escape rejection, list pagination, delete pruning. A round-trip test drives the handler over fsStore with a real @aws-sdk/client-s3 client, including presigned GET and PUT (the AWS SDK is a test-only dependency of this package). Workspace: typecheck 66/66, tests 55/55, lint and dependency-layering clean in both import directions, no new type casts.

Alternatives considered

A second S3 implementation for dev — would drift from the one serving real traffic; the entire point is one implementation.
Importing the storage module from below — forbidden by the layering, and rightly: modules depend on the framework, never the reverse.
A filesystem fake without the wire protocol — real service code speaks S3 through real SDKs; a fake that skips SigV4 and presigning tests nothing.

🤖 Generated with Claude Code

…ocal providers Records the local-dev architecture: prisma-composer dev re-runs the deploy pipeline (Load, assemble, lower, Alchemy converge) against local implementations of the same resource types, declared on an optional dev field of ExtensionDescriptor. The field carries no nodes/provisions, so dev structurally cannot diverge the lowering. Adds the local-dev domain deep dive (process table, supervisor, substitution table, disk-backed S3 server, value sourcing, error surface), indexes both docs, points deploy-cli.md at the design, and marks the deferred.md dev-time sketch resolved. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Pins every name, type, behavior, and file placement for ADR-0041: DevDescriptor (no state — dev uses alchemy localState via LowerOptions), the framework-owned DevProcessRecord table, the eight local providers and their exact reconcile semantics, the @internal/s3-protocol extraction (layering forces the protocol pieces down to the lowering layer), the serviceAddress threading through ComputeSerialized/DeploymentProps, the dir() build adapter, the dev command pipeline/supervisor/watch loops, the value-sourcing policy (secret placeholders vs hard-error env params), and the full error surface. Aligns ADR-0041 and local-dev.md with the refinements the deep code read forced (standIns hook, per-Database prisma dev instances, Ctrl-C semantics, stage `dev`, port allocation). Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Converge-scoped (the alchemy child), session-scoped (service children and the bucket server, owned by the dev command via the process table and standIns), and machine-scoped daemons (prisma dev Postgres instances). Names why the bucket server is session-scoped rather than a daemon, and that standIns permits a detached daemon without a seam change. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…s; drop the standIns hook Operator decision: treat every local backing service the way prisma dev Postgres instances already work — long-lived daemons that survive dev sessions, the firebase/supabase emulator model. Providers ensure their emulator at reconcile and provision instances by communicating with it (the ORM CLI for Postgres; a loopback /_pcdev/ admin API on the new daemonized bucket emulator). This deletes the standIns hook entirely: DevDescriptor is providers/container/preflight/teardown. Pins the emulator daemon manager (machine-global registry, stable ports from 4300, version-skew restart, ready-file protocol), the emulator-main entry in @internal/s3-protocol, Ctrl-C leaving all emulators up, and --fresh as the only removal path. Vocabulary swept from stand-in to emulator across ADR-0041, local-dev.md, and the project spec/plan. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ervice processes Operator decision refining the emulator model: the target spins up one machine-scoped, multi-tenant emulator per node kind the topology uses — a Compute emulator (a small local counterpart of the platform compute service: deployment PUTs, child processes, crash backoff, logs), the ORM prisma dev Postgres emulator, and one bucket emulator — and providers provision isolated instances by communicating with them, wired together exactly as deployed. This deletes the framework-owned process table and the CLI supervisor: the dev command is a view through a new dev.attach hook (endpoints, merged logs, stop control), plus a topology-aware dev.emulators ensure hook. Ctrl-C stops the app services; emulators and data persist; --fresh removes app-scoped instances but never the shared daemons. Bucket namespacing via <app>--<name> physical names on the binding; per-bucket in-project data roots keep objects browsable. Daemon machinery consolidates into a new @internal/dev-emulators package; the plan grows to six slices. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Verification pass over the spec after the emulator-model rewrites, fixing six genuine gaps an implementer would otherwise have had to improvise: deployment PUT start rules (stopped/held services always start — an explicit converge clears held and undoes stop); per-daemon health paths (the bucket daemon root is the S3 wire); the prisma dev stop/start port-stability assumption turned into a pinned verification item with a pinned fallback (--db-port from a machine-global allocation file); the watch contract made adapter-declared (Bundle.watch, populated by node/ nextjs/dir, consumed by the CLI); bucket credential upsert keyed by accessKeyId plus physical-name length validation and lazy dir recreation; and DevProvidersInput threading — dev.providers(input) receives the resolved container and devDir, since local providers are emulator clients that must know which app they provision for (lower() already holds the deserialized container map). Plus API hygiene pins (path-segment validation, serialized state writes, no log rotation), a warm-start acceptance criterion, and plan updates (S2 carries Bundle.watch, S4 runs the port-stability check first). Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…wering/shared plane Step zero of the local-dev S1 slice: registers the new package as domain prisma-cloud, layer lowering, plane shared so it is importable by both extensions/control and modules/execution code. Verified with a scratch stub in each direction against pnpm lint:deps before building on it (per spec); the stubs are not part of this commit. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ernal/s3-protocol Moves the S3 wire-protocol pieces verbatim (git mv; only import paths change) out of the storage module into the new lowering-layer package, per local-dev spec S1. storage keeps pg-store.ts and storage-server.ts and now imports the moved pieces from @internal/s3-protocol; its public exports stay byte-compatible. Also moves mintKeyPair into sigv4.ts (from the target extensions s3-credentials-resource.ts) so deploy-time credential minting and the local dev bucket emulators future minting share one implementation; the extension re-exports it. Storages own handler/memory-store/sigv4 tests stay in place and now import the moved pieces from @internal/s3-protocol by package identifier instead of by relative path — their bodies are otherwise unmodified, and all 47 pass unchanged. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Implements the local-dev bucket emulators backing store (spec S1): object bytes at <bucketDir>/<key>, a JSON sidecar at <bucketDir>/.meta/<key>.json carrying {contentType, etag}, and temp-then-rename writes through <bucketDir>/.tmp/<uuid> so a concurrent read never observes a partial object. Unknown buckets and missing keys degrade gracefully on every read path (matching the handlers existing 404/empty-list shape); a malformed or escaping key throws on every operation, and an unknown bucket throws on put since there is nowhere to write the bytes. A file dropped on disk with no sidecar is adopted as a valid object with a lazily-written sidecar. Tests: fs-store contract tests directly against the ObjectStore (dropped-file adoption, path-escape and bucket-name rejection, list pagination, delete pruning empty parent dirs, and an observable temp-then-rename proof under concurrent reads), plus a handler round-trip over fsStore via a real @aws-sdk/client-s3 (incl. presigned GET/PUT), mirroring storages existing handler test but proving the disk-backed path specifically. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…dir, entry points into the copied tree Review finding on S2: the spec said { dir: <workDir>/bundle, entry }, which taken literally drops the wrapper main.mjs from the artifact and breaks boot. The implementation correctly followed the stronger "identical to node()" pin; align the prose with the real Bundle contract. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

S1 review verdicted both implementer gap-fills ACCEPTABLE-BUT-PIN: reads on an unknown or invalid-named bucket degrade to missing-key shapes (404/empty-200/204 through the handler) while a put throws and surfaces as the host 500; an invalid key throws on every operation when the bucket is known, with the store — not the handler — as the escape protection, and no thrown message ever naming a server directory. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review finding: the two as Record<string, unknown> casts plus the as Sidecar return cast pushed the pnpm lint:casts ratchet from 29 to 32. Real narrowing was trivial here — an isSidecar(value): value is Sidecar predicate using typeof/in checks (.agents/rules/type-predicates.mdc) replaces all three; no blindCast needed. pnpm lint:casts now reports current=29 merge-base=29 delta=0. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review nit: the old "maxKeys defaults to 1000" test only proved the default was >= the fixtures 3 keys. Exports DEFAULT_MAX_KEYS from fs-store.ts (internal-only — not re-exported from src/exports/) so the test can assert the literal value directly and prove that omitting maxKeys behaves identically to passing DEFAULT_MAX_KEYS explicitly, rather than merely happening to exceed a small fixture. A 1001-key disk fixture would have worked too but is not worth the runtime for what the exported constant already proves. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

coderabbitai · 2026-07-22T18:45:23Z

Warning

Review limit reached

@wmadden-electric, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 46 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 1721f1f8-dd1b-4d13-bf68-03a2754c4e03

📥 Commits

Reviewing files that changed from the base of the PR and between 46fa2c3 and 91a75e9.

⛔ Files ignored due to path filters (1)

pnpm-lock.yaml is excluded by !**/pnpm-lock.yaml

📒 Files selected for processing (30)

.drive/deferred.md
.drive/projects/local-dev/plan.md
.drive/projects/local-dev/spec.md
architecture.config.json
docs/design/10-domains/README.md
docs/design/10-domains/deploy-cli.md
docs/design/10-domains/local-dev.md
docs/design/90-decisions/ADR-0041-local-dev-runs-the-deploy-pipeline-against-local-providers.md
docs/design/90-decisions/README.md
packages/1-prisma-cloud/0-lowering/s3-protocol/package.json
packages/1-prisma-cloud/0-lowering/s3-protocol/src/__tests__/fs-store.test.ts
packages/1-prisma-cloud/0-lowering/s3-protocol/src/__tests__/handler-fs-store.test.ts
packages/1-prisma-cloud/0-lowering/s3-protocol/src/exports/index.ts
packages/1-prisma-cloud/0-lowering/s3-protocol/src/fs-store.ts
packages/1-prisma-cloud/0-lowering/s3-protocol/src/handler.ts
packages/1-prisma-cloud/0-lowering/s3-protocol/src/memory-store.ts
packages/1-prisma-cloud/0-lowering/s3-protocol/src/sigv4.ts
packages/1-prisma-cloud/0-lowering/s3-protocol/src/store.ts
packages/1-prisma-cloud/0-lowering/s3-protocol/tsconfig.json
packages/1-prisma-cloud/0-lowering/s3-protocol/tsdown.config.ts
packages/1-prisma-cloud/1-extensions/target/package.json
packages/1-prisma-cloud/1-extensions/target/src/s3-credentials-resource.ts
packages/1-prisma-cloud/2-shared-modules/storage/package.json
packages/1-prisma-cloud/2-shared-modules/storage/src/__tests__/handler.test.ts
packages/1-prisma-cloud/2-shared-modules/storage/src/__tests__/memory-store.test.ts
packages/1-prisma-cloud/2-shared-modules/storage/src/__tests__/sigv4.test.ts
packages/1-prisma-cloud/2-shared-modules/storage/src/pg-store.ts
packages/1-prisma-cloud/2-shared-modules/storage/src/storage-server.ts
packages/9-public/composer-prisma-cloud/package.json
tsconfig.depcruise.json

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch claude/local-dev-s1-s3-protocol

✨ Simplify code

Create PR with simplified code
Commit simplified code in branch claude/local-dev-s1-s3-protocol

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

pkg-pr-new · 2026-07-22T18:46:38Z

Open in StackBlitz

npm i https://pkg.pr.new/@prisma/composer@159

npm i https://pkg.pr.new/@prisma/composer-prisma-cloud@159

commit: 91a75e9

wmadden-electric and others added 13 commits July 22, 2026 18:19

wmadden-electric mentioned this pull request Jul 22, 2026

feat(core): build adapters declare their watch paths; fix symlink handling in node()'s directory form #158

Open

wmadden-electric changed the title ~~refactor(storage): extract the S3 protocol into @internal/s3-protocol + a disk-backed store~~ refactor(storage): extract the S3 protocol to @internal/s3-protocol; add a disk-backed object store Jul 22, 2026

wmadden-electric mentioned this pull request Jul 22, 2026

feat(dev): the local Compute and bucket emulator daemons #160

Open

wmadden-electric changed the title ~~refactor(storage): extract the S3 protocol to @internal/s3-protocol; add a disk-backed object store~~ refactor(storage): move the S3 protocol down to @internal/s3-protocol; add a disk-backed object store Jul 23, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

refactor(storage): move the S3 protocol down to @internal/s3-protocol; add a disk-backed object store#159

refactor(storage): move the S3 protocol down to @internal/s3-protocol; add a disk-backed object store#159
wmadden-electric wants to merge 13 commits into
mainfrom
claude/local-dev-s1-s3-protocol

wmadden-electric commented Jul 22, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 22, 2026

Review limit reached

Uh oh!

pkg-pr-new Bot commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

wmadden-electric commented Jul 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What moves, and why

Tested

Alternatives considered

Uh oh!

coderabbitai Bot commented Jul 22, 2026

Review limit reached

Uh oh!

pkg-pr-new Bot commented Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

wmadden-electric commented Jul 22, 2026 •

edited

Loading