The Postgres emulator starts servers one at a time#167
Conversation
… patience The 'already running' refusal is thrown on ELOCKED (verified in @prisma/dev's source: the stale-state parent error class has no throw site) — a LIVE process holds the server's state lock. The previous recovery treated an unreadable server dump as a dead ghost and called the internal deleteServer, which kills the holder and deletes its state AND persisted data. Under CI load this fired against live mid-boot holders and produced three symptoms from one cause: a migrated database's schema vanishing (relation does not exist — the data was wiped while Alchemy's migrate resource saw no diff), the daemon dying silently (pglite's files deleted under a running native instance), and repeat 'already running' churn. Recovery is now adopt-only: the live holder IS the goal, its dump can be briefly unreadable mid-boot, so adoption retries with patience (8 x 400ms) and then fails the request visibly. Nothing in this path deletes state or data anymore; deleteApp remains the only deleting caller. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
WalkthroughThe Postgres emulator now reconciles persisted server records before startup and during name-conflict recovery. It adopts servers with live owning processes and recorded URLs, clears stale live records when owning processes are dead, and bounds polling retries while a server is booting. Adopted-server teardown avoids killing the daemon itself, waits for persisted close settlement, and deletes state only after settlement. A cross-process test verifies adoption, connectivity, daemon health, and cleanup. 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
commit: |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/1-prisma-cloud/0-lowering/dev-emulators/src/postgres-main.ts`:
- Around line 482-510: Document the changed recovery behavior around the
isNameAlreadyTaken/adoption flow: truly orphaned locks that never expose an
adoptable server exhaust MAX_ADOPT_ATTEMPTS and return a 500 until DELETE
/apps/<app> is performed. Update the relevant on-call runbooks and alerts to
identify this condition and prescribe manual deletion, while preserving the
existing protection against deleting live server state.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 95bf69e2-ce28-47fb-91bc-c8662cedc2e0
📒 Files selected for processing (1)
packages/1-prisma-cloud/0-lowering/dev-emulators/src/postgres-main.ts
…eServer The in-daemon servers' persisted state records THIS DAEMON's own pid, and deleteServer's kill path targets that pid whenever the state still claims the server is live. deleteApp closes each server first, but the stopped-status write is asynchronous — when deleteServer read the state before the close flushed, the daemon killed itself (the silent CI death, racy by nature; verified live that getServerStatus reports 'running' with the daemon's own pid while up and 'not_running' after close). Teardown now polls getServerStatus until the close has settled (up to 10s) before deleting; if it never settles, the persisted data is left in place rather than risking a self-kill. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…stale lock CI showed the 3.2s window losing to reality: a sibling drift-test case adopted successfully at 8.5s while this one gave up at 3.2s. A cold server boot takes seconds, and a crashed holder's lock is only released by proper-lockfile's ~10s stale threshold — 15s (30 x 500ms) outlasts both. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ecovering from a refusal
My previous adoption path could never have worked, and a live probe of
@prisma/dev proves why: the 'already running' error's own `server`
accessor resolves to a ServerState whose own properties are
databaseDumpPath/exports/experimental/pgliteDataDirPath/close/
writeServerDump — there is no `database` field, so the duck-type check
rejected every candidate and adoption timed out every time (CI: the full
15s window, then a 500).
The same probe shows the right source: getServerStatus(name) reports
{status, pid, databasePort, exports.database.connectionString} — the URL
directly, plus who owns it. So the daemon now reconciles BEFORE starting
rather than recovering after a refusal:
- a live record whose pid is alive is the database being asked for (the
name is namespaced to this app+database) — adopt its URL;
- a live record whose pid is gone is a dead owner's leftover (a daemon
replaced for version skew, a crash) — killServer clears the process
record, never the data, and the start proceeds;
- anything else starts normally.
The post-refusal path keeps the same logic for a lost race. deleteApp
kills adopted servers by record before its settled-close check, since
they have no handle to close. Regression test included: it creates the
mismatch first-hand by starting a server under the daemon-derived name
behind the daemon's back, then asserts the ensure adopts that exact URL,
the database answers a query, and the daemon is still alive.
Signed-off-by: willbot <w.a.madden+machine@gmail.com>
Signed-off-by: Will Madden <madden@prisma.io>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…r kill by our own pid CI killed the test runner with SIGTERM (exit 143) at exactly the new test: it started the foreign server INSIDE the bun test process, so the server's record named the runner's pid, and teardown's killServer signalled it. That is the same self-kill mode the daemon fix is about, reproduced one level up — and a faithful test needs the foreign server in its own process anyway, since that is what a leftover from a previous daemon actually is. The test now spawns a host process and reads the connection string from its stdout. Teardown also refuses to killServer when the record names this daemon's own pid — that record can only mean an in-process server whose handle we lost, and signalling it would kill the daemon. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI evidence: the daemon's own listing showed no record for pcdev-store-orders-database while that exact name was refused as 'already running'. The refusal means a live process holds the server's lock, but the record it holds can say three different things, and only two were handled: a live record with a live owner (adopt) and a live record with a dead owner (clear it). The third — NO live record at all, because the holder has not published one yet or its record was removed while the lock lingers — fell through to an immediate retry, which burned every attempt in milliseconds and surfaced a 500. That case now waits for the holder to publish a record (then adopts it) or to release the lock (then starts), polling up to 12s per attempt with at most two retries, so a genuinely stuck name still fails visibly inside the dev command's startup budget. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e cascade Two experiments settle it. Two `startPrismaDevServer` calls issued concurrently in one process fail with a port refusal even when their database ports are distinct and pinned; the same two started in sequence both succeed. `@prisma/dev`'s start is not concurrency-safe within a process — simultaneous starts pick their ports without seeing each other. The daemon issued exactly that pattern: an app converges its databases in parallel, so ensures for different names arrived at once. That produced the port refusals, whose retries left half-started servers holding their own name's lock, which surfaced as 'already running' — the failure whose 'recovery' deleted a live database's data and killed the daemon. Every symptom chased on this branch traces back here. Starts now run one at a time. The daemon also stops hand-picking @prisma/dev's http/shadow/streams ports: only the database port is ours to pin (deploy state references it), and its own picker is the only one that consults its registry of other servers — ours could not, which is what produced the 'belongs to another Prisma Dev server' refusals. Regression test included: two databases ensured at once both come up and answer queries. Signed-off-by: willbot <w.a.madden+machine@gmail.com> Signed-off-by: Will Madden <madden@prisma.io> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The decision: the Postgres emulator daemon starts servers one at a time.
@prisma/dev's start is not concurrency-safe within a process — simultaneous starts choose their ports without seeing each other — and the daemon was issuing exactly that pattern, because an app converges its databases in parallel.Why this is the whole bug
prisma-composer devruns an app's Postgres databases as named@prisma/devservers hosted inside one long-lived daemon. When an app has two databases, the deploy engine converges both resources at once, so two ensure requests hit the daemon simultaneously and it calledstartPrismaDevServertwice in parallel. One of the two would lose the port race and be refused.Everything else followed from that. A refused start left a half-started server holding its own name's lock, so the daemon's next attempt for that name was refused a second time — this time as "A Prisma Dev server with the name … is already running." A recovery path added for that message treated the name as a dead leftover and cleared it with
@prisma/dev's internaldeleteServer, which kills the recorded owner and removes the server's persisted data. On CI that produced, from this single cause: a migrated schema disappearing mid-run (relation "public.product" does not exist), the daemon dying with no JavaScript error at all (its own PGlite files deleted underneath it), and repeated "already running" failures that no amount of retrying could clear.So this PR fixes the origin and removes the machinery that was generating collisions:
@prisma/dev's auxiliary ports. Only the database port is ours to pin, because endpoints recorded in deploy state reference it. The http/shadow/streams listeners are internal to@prisma/dev, and its own picker is the only one that consults its registry of other servers — ours could not, which is what produced the "belongs to another Prisma Dev server" refusals. Net −54 lines of port bookkeeping.killServer, which never touches data). The app-scopedDELETE /apps/<app>remains the only caller that deletes.deleteServersignals whatever pid a server's record names, and for a server hosted inside the daemon that pid is the daemon's own. Teardown now waits for a close to settle before deleting, skips deletion rather than risk a self-kill if it never settles, and never signals a record naming its own process.Tested
@prisma/dev— they are what identified the cause, rather than a theory fitted to the logs afterwards.select 1— the exact pattern that used to race.prisma-composer devsessions, including--freshand warm restart) pass on CI, along with all three deploy smokes and both canaries.Alternatives considered
@prisma/devpick the database port too — deploy state records the port, so it has to stay stable across restarts; only the internal listeners can float.devsession.🤖 Generated with Claude Code