Skip to content

Recover LMDB crash on startup#615

Open
satyakigh wants to merge 2 commits into
mainfrom
tst-lmdb
Open

Recover LMDB crash on startup#615
satyakigh wants to merge 2 commits into
mainfrom
tst-lmdb

Conversation

@satyakigh

@satyakigh satyakigh commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

Problem
The LMDB schema cache lives at a single machine-wide path shared by every IDE/language-server process. If opening that database trips a C-level assertion in the native lmdb library, the process is aborted - no JS catch/finally runs. The editor restarts the server, it aborts on the same corrupt data again, and the user is stuck in an infinite startup crash loop with no way for normal try/catch recovery to help.

How it fixes it - a cross-process startup "dirty bit":

  • New LMDBOwnershipTracker writes a per-PID marker file before the risky open owner.<pid>.opening and renames it to owner.<pid>.running once the open succeeds; both are removed on clean shutdown.
  • On the next startup, under a proper-lockfile cross-process lock, it scans markers and wipes the (presumed corrupt) data directory only when both: no marker belongs to a live process, and a dead owner is stuck in the opening phase - the signature of a mid-open crash. A dead running marker (an ordinary abrupt kill of a healthy process) never triggers a wipe, so reboots/force-quits preserve the cache.

Changes

  • LMDBStoreFactory refactored from a do-everything constructor into a two-phase construct → async initialize() lifecycle (env open, crash recovery, and background tasks now happen in initialize()), with cleaner tryOpen/recreateAfterCorruption recovery helpers.
  • initialize() added to the DataStoreFactory/DataStoreFactoryProvider interfaces; MemoryStore/FileStore implement it as no-ops; standalone.ts awaits it before constructing the server.

@satyakigh satyakigh requested a review from a team as a code owner June 29, 2026 21:25
@github-code-quality

github-code-quality Bot commented Jul 6, 2026

Copy link
Copy Markdown

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/vitest

The overall coverage in the branch remains at 90%, unchanged from the branch.

Show a code coverage summary of the most impacted files.
File efab55e 1811cb8 +/-
src/datastore/DataStore.ts 90% 87% -3%
src/datastore/F...StoreFactory.ts 75% 76% +1%
src/datastore/MemoryStore.ts 87% 88% +1%
src/datastore/L...StoreFactory.ts 66% 73% +7%
src/datastore/l...rshipTracker.ts 0% 93% +93%

Updated July 07, 2026 15:10 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

return value === OwnershipPhase.opening.toString() || value === OwnershipPhase.running.toString();
}

function isProcessAlive(pid: number): boolean {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PID recycling can silently defeat crash-loop recovery.

isProcessAlive(pid) returns true whenever any process currently holds that PID — it can't tell the crashed opener apart from an unrelated process that the OS later assigned the same PID. On a busy machine this is a real collision, not a corner case.

Sequence:

  1. Opener crashes mid-open, leaving owner.<pid>.opening.
  2. Before the next startup, the OS recycles <pid> to some unrelated live process.
  3. scanMarkers() counts that opening marker as a liveOwner, so liveOwners === 0 is false and the wipe is suppressed.
  4. The user stays stuck in exactly the corruption crash loop this feature exists to break.

writeOwnMarker already persists ${pid}\n${ISO-timestamp} into the marker body, but parseMarker only reads the filename, so the timestamp is currently dead weight. Reading it would close this gap: treat an opening marker older than some threshold (e.g. well beyond any plausible open duration) as a crashed startup regardless of isProcessAlive, since a genuine in-progress open is never that old. That both fixes the recycling case and puts the already-written timestamp to use.

Not blocking, but worth addressing since it's the one path where the safeguard silently no-ops.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a timestamp check so that a marker older than OPENING_MARKER_STALE_MS (5 min) cannot be a live opener, so its age proves a crash no matter what isProcessAlive says

kddejong
kddejong previously approved these changes Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants