Skip to content

The Postgres emulator starts servers one at a time#167

Merged
wmadden-electric merged 7 commits into
mainfrom
claude/local-dev-ghost-adoption-fix
Jul 24, 2026
Merged

The Postgres emulator starts servers one at a time#167
wmadden-electric merged 7 commits into
mainfrom
claude/local-dev-ghost-adoption-fix

Conversation

@wmadden-electric

@wmadden-electric wmadden-electric commented Jul 23, 2026

Copy link
Copy Markdown
Contributor
# Two servers, started at the same time, in one process:
startPrismaDevServer({ name: 'a', databasePort: 61234 })   ┐ both at once
startPrismaDevServer({ name: 'b', databasePort: 61235 })   ┘
→ PortNotAvailableError { port: 61235 }        # distinct, pinned ports — still fails

# The same two, one after the other:
→ both start fine.

The decision: the Postgres emulator daemon starts servers one at a time. @prisma/dev's start is not concurrency-safe within a process — simultaneous starts choose their ports without seeing each other — and the daemon was issuing exactly that pattern, because an app converges its databases in parallel.

Why this is the whole bug

prisma-composer dev runs an app's Postgres databases as named @prisma/dev servers hosted inside one long-lived daemon. When an app has two databases, the deploy engine converges both resources at once, so two ensure requests hit the daemon simultaneously and it called startPrismaDevServer twice in parallel. One of the two would lose the port race and be refused.

Everything else followed from that. A refused start left a half-started server holding its own name's lock, so the daemon's next attempt for that name was refused a second time — this time as "A Prisma Dev server with the name … is already running." A recovery path added for that message treated the name as a dead leftover and cleared it with @prisma/dev's internal deleteServer, which kills the recorded owner and removes the server's persisted data. On CI that produced, from this single cause: a migrated schema disappearing mid-run (relation "public.product" does not exist), the daemon dying with no JavaScript error at all (its own PGlite files deleted underneath it), and repeated "already running" failures that no amount of retrying could clear.

So this PR fixes the origin and removes the machinery that was generating collisions:

  • Starts are serialized. One at a time inside the daemon; a two-database app pays a couple of extra seconds on a cold start and nothing on a warm one.
  • The daemon stops choosing @prisma/dev's auxiliary ports. Only the database port is ours to pin, because endpoints recorded in deploy state reference it. The http/shadow/streams listeners are internal to @prisma/dev, and its own picker is the only one that consults its registry of other servers — ours could not, which is what produced the "belongs to another Prisma Dev server" refusals. Net −54 lines of port bookkeeping.
  • Nothing on the recovery path deletes anything. A name refused as "already running" means a live process holds that server's lock. Since the name is namespaced to this app and database, that server is the database being asked for: the daemon adopts its connection string, waiting out a holder that is still booting, and clears only a record whose owning process is gone (killServer, which never touches data). The app-scoped DELETE /apps/<app> remains the only caller that deletes.
  • Teardown cannot make the daemon kill itself. deleteServer signals whatever pid a server's record names, and for a server hosted inside the daemon that pid is the daemon's own. Teardown now waits for a close to settle before deleting, skips deletion rather than risk a self-kill if it never settles, and never signals a record naming its own process.

Tested

  • The two experiments in the opening block were run directly against @prisma/dev — they are what identified the cause, rather than a theory fitted to the logs afterwards.
  • New: two databases ensured concurrently both come up and answer select 1 — the exact pattern that used to race.
  • New: a live server registered under the daemon-derived name that the daemon has no record of is adopted, its database answers a query, and the daemon stays up. The foreign server runs in its own process, because a record names its host's pid and an in-process one would make teardown kill the test runner.
  • The full daemon suite, the local-target suite, and the store integration proof (four prisma-composer dev sessions, including --fresh and warm restart) pass on CI, along with all three deploy smokes and both canaries.

Alternatives considered

  • Keep retrying harder — the branch history is a record of this failing: wider port-retry windows, longer adoption patience, process-level crash guards. Each addressed a symptom the concurrency bug produced, and the failures kept moving.
  • Let @prisma/dev pick the database port too — deploy state records the port, so it has to stay stable across restarts; only the internal listeners can float.
  • Serialize in the provider instead of the daemon — providers run in a short-lived converge process, and the daemon is the shared thing that actually owns the servers; a lock anywhere else would not cover a second dev session.

🤖 Generated with Claude Code

… patience

The 'already running' refusal is thrown on ELOCKED (verified in
@prisma/dev's source: the stale-state parent error class has no throw
site) — a LIVE process holds the server's state lock. The previous
recovery treated an unreadable server dump as a dead ghost and called
the internal deleteServer, which kills the holder and deletes its state
AND persisted data. Under CI load this fired against live mid-boot
holders and produced three symptoms from one cause: a migrated
database's schema vanishing (relation does not exist — the data was
wiped while Alchemy's migrate resource saw no diff), the daemon dying
silently (pglite's files deleted under a running native instance), and
repeat 'already running' churn.

Recovery is now adopt-only: the live holder IS the goal, its dump can be
briefly unreadable mid-boot, so adoption retries with patience (8 x
400ms) and then fails the request visibly. Nothing in this path deletes
state or data anymore; deleteApp remains the only deleting caller.

Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

The Postgres emulator now reconciles persisted server records before startup and during name-conflict recovery. It adopts servers with live owning processes and recorded URLs, clears stale live records when owning processes are dead, and bounds polling retries while a server is booting. Adopted-server teardown avoids killing the daemon itself, waits for persisted close settlement, and deletes state only after settlement. A cross-process test verifies adoption, connectivity, daemon health, and cleanup.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed It summarizes the main behavior change: Postgres emulator startup is serialized.
Description check ✅ Passed It describes the same server-conflict and serialization fix reflected in the diff.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/local-dev-ghost-adoption-fix
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch claude/local-dev-ghost-adoption-fix

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@pkg-pr-new

pkg-pr-new Bot commented Jul 23, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@prisma/composer@167
npm i https://pkg.pr.new/@prisma/composer-prisma-cloud@167

commit: a96bb92

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/1-prisma-cloud/0-lowering/dev-emulators/src/postgres-main.ts`:
- Around line 482-510: Document the changed recovery behavior around the
isNameAlreadyTaken/adoption flow: truly orphaned locks that never expose an
adoptable server exhaust MAX_ADOPT_ATTEMPTS and return a 500 until DELETE
/apps/<app> is performed. Update the relevant on-call runbooks and alerts to
identify this condition and prescribe manual deletion, while preserving the
existing protection against deleting live server state.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 95bf69e2-ce28-47fb-91bc-c8662cedc2e0

📥 Commits

Reviewing files that changed from the base of the PR and between 3142847 and 0aeef0d.

📒 Files selected for processing (1)
  • packages/1-prisma-cloud/0-lowering/dev-emulators/src/postgres-main.ts

Comment thread packages/1-prisma-cloud/0-lowering/dev-emulators/src/postgres-main.ts Outdated
…eServer

The in-daemon servers' persisted state records THIS DAEMON's own pid,
and deleteServer's kill path targets that pid whenever the state still
claims the server is live. deleteApp closes each server first, but the
stopped-status write is asynchronous — when deleteServer read the state
before the close flushed, the daemon killed itself (the silent CI
death, racy by nature; verified live that getServerStatus reports
'running' with the daemon's own pid while up and 'not_running' after
close). Teardown now polls getServerStatus until the close has settled
(up to 10s) before deleting; if it never settles, the persisted data is
left in place rather than risking a self-kill.

Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
wmadden-electric and others added 5 commits July 23, 2026 23:08
…stale lock

CI showed the 3.2s window losing to reality: a sibling drift-test case
adopted successfully at 8.5s while this one gave up at 3.2s. A cold
server boot takes seconds, and a crashed holder's lock is only released
by proper-lockfile's ~10s stale threshold — 15s (30 x 500ms) outlasts
both.

Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ecovering from a refusal

My previous adoption path could never have worked, and a live probe of
@prisma/dev proves why: the 'already running' error's own `server`
accessor resolves to a ServerState whose own properties are
databaseDumpPath/exports/experimental/pgliteDataDirPath/close/
writeServerDump — there is no `database` field, so the duck-type check
rejected every candidate and adoption timed out every time (CI: the full
15s window, then a 500).

The same probe shows the right source: getServerStatus(name) reports
{status, pid, databasePort, exports.database.connectionString} — the URL
directly, plus who owns it. So the daemon now reconciles BEFORE starting
rather than recovering after a refusal:

- a live record whose pid is alive is the database being asked for (the
  name is namespaced to this app+database) — adopt its URL;
- a live record whose pid is gone is a dead owner's leftover (a daemon
  replaced for version skew, a crash) — killServer clears the process
  record, never the data, and the start proceeds;
- anything else starts normally.

The post-refusal path keeps the same logic for a lost race. deleteApp
kills adopted servers by record before its settled-close check, since
they have no handle to close. Regression test included: it creates the
mismatch first-hand by starting a server under the daemon-derived name
behind the daemon's back, then asserts the ensure adopts that exact URL,
the database answers a query, and the daemon is still alive.

Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r kill by our own pid

CI killed the test runner with SIGTERM (exit 143) at exactly the new
test: it started the foreign server INSIDE the bun test process, so the
server's record named the runner's pid, and teardown's killServer
signalled it. That is the same self-kill mode the daemon fix is about,
reproduced one level up — and a faithful test needs the foreign server
in its own process anyway, since that is what a leftover from a previous
daemon actually is. The test now spawns a host process and reads the
connection string from its stdout.

Teardown also refuses to killServer when the record names this daemon's
own pid — that record can only mean an in-process server whose handle we
lost, and signalling it would kill the daemon.

Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI evidence: the daemon's own listing showed no record for
pcdev-store-orders-database while that exact name was refused as
'already running'. The refusal means a live process holds the server's
lock, but the record it holds can say three different things, and only
two were handled: a live record with a live owner (adopt) and a live
record with a dead owner (clear it). The third — NO live record at all,
because the holder has not published one yet or its record was removed
while the lock lingers — fell through to an immediate retry, which
burned every attempt in milliseconds and surfaced a 500.

That case now waits for the holder to publish a record (then adopts it)
or to release the lock (then starts), polling up to 12s per attempt with
at most two retries, so a genuinely stuck name still fails visibly
inside the dev command's startup budget.

Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e cascade

Two experiments settle it. Two `startPrismaDevServer` calls issued
concurrently in one process fail with a port refusal even when their
database ports are distinct and pinned; the same two started in sequence
both succeed. `@prisma/dev`'s start is not concurrency-safe within a
process — simultaneous starts pick their ports without seeing each
other.

The daemon issued exactly that pattern: an app converges its databases in
parallel, so ensures for different names arrived at once. That produced
the port refusals, whose retries left half-started servers holding their
own name's lock, which surfaced as 'already running' — the failure whose
'recovery' deleted a live database's data and killed the daemon. Every
symptom chased on this branch traces back here.

Starts now run one at a time. The daemon also stops hand-picking
@prisma/dev's http/shadow/streams ports: only the database port is ours
to pin (deploy state references it), and its own picker is the only one
that consults its registry of other servers — ours could not, which is
what produced the 'belongs to another Prisma Dev server' refusals.
Regression test included: two databases ensured at once both come up and
answer queries.

Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@wmadden-electric wmadden-electric changed the title The postgres emulator adopts a locked server instead of deleting its data The Postgres emulator starts servers one at a time Jul 23, 2026
@wmadden-electric
wmadden-electric merged commit cb93583 into main Jul 24, 2026
18 checks passed
@wmadden-electric
wmadden-electric deleted the claude/local-dev-ghost-adoption-fix branch July 24, 2026 04:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants