Skip to content

test(d1): every vitest run leaks ~7.6k SQLite clones (~3.8 GB) to tmp — the exit sweep does not fire for pooled workers #10281

Description

@JSONbored

Problem

Every npx vitest run leaves ~7,600 SQLite clone files (~3.8 GB) in the OS temp directory, permanently.
Nothing ever removes them, so the cost is linear in how many times anyone runs the suite.

Measured on a clean, fully-passing run (1369 passed | 3 skipped, 0 failures) immediately after
emptying the temp directory:

clone files left behind by one run 7,577
total size ~3.8 GB
size of one clone 520,192 bytes
distinct worker PIDs that leaked 400

This is not a crash artifact — the run succeeded completely.

Cause

test/helpers/d1.ts:78-108. Every new TestD1Database() copies the migrated template to
${tmpdir()}/loopover-test-clone-<pid>-<n>-<rand>.sqlite3, and the only cleanup is a single
process.on("exit") sweep registered once per worker:

state.clonePaths.push(clonePath);
if (!state.exitSweepRegistered) {
  state.exitSweepRegistered = true;
  process.on("exit", () => {
    for (const path of state.clonePaths) {
      try { unlinkSync(path); } catch { /* best-effort tmp hygiene only */ }
    }
  });
}

With vitest's worker pool, that handler evidently does not run for the great majority of workers — 400
distinct PIDs left files behind in a single successful run. process.on("exit") is the wrong hook for a
pooled worker that is torn down rather than exiting normally, and the catch {} means a failure to unlink
is silent by design.

The obvious alternative — unlink immediately after opening — is explicitly ruled out by the file's own
comment at line 27 ("The clone file must NOT be unlinked while its database is open"), so the fix needs to
be tied to the database being closed, not to process exit.

Impact, which is not theoretical

This filled a 460 GB dev machine. /var/folders/…/T had accumulated 241.2 GB across 496,955
loopover-* files
, of which 483,844 were created that same day; the volume hit 100% with 2.0 GB free.

The failure mode is badly misleading. Once the disk fills, the suite does not report "no space" — it reports
hundreds of unrelated test failures. One run showed 1235 failed test files | 730 failed tests, another
518 failed | 267 failed, with assertion errors scattered across api.test.ts,
check-migrations-script.test.ts, selfhost-update-script.test.ts and others. The real cause appeared only
as a couple of buried lines:

Error: ENOSPC: no space left on device, mkdtemp '/var/folders/.../loopover-miner-init-wizard-XXXXXX'
Error: database or disk is full

Deleting only ${tmpdir()}/loopover-* recovered 244 GB and the same suite went green with 26,762
passed / 0 failed
and no code change. So this costs real debugging time on failures that look like genuine
regressions, and it will do so again for anyone who runs the suite often enough.

Deliverables

  • A clone is removed when its database is closed, rather than only at process exit — tied to the
    TestD1Database lifecycle so a pooled worker that never exits cleanly still cleans up.
  • A full npx vitest run leaves zero loopover-test-clone-* files behind. Worth asserting
    mechanically, since the current failure is silent and only shows up as unrelated red weeks later.
  • Decide whether the catch {} should stay silent. It is defensible for tmp hygiene, but it is also
    why 496k files accumulated with no signal.
  • Sweep stale clones on startup as a backstop, so an interrupted run (Ctrl-C, OOM) cannot accumulate
    indefinitely.
  • loopover-miner-init-wizard-* and loopover-cli-config-* temp dirs appear in the same directory and
    should be checked for the same pattern.

How this was found

Not by looking for it: repeated full-suite runs while working #10269 filled the disk, and the resulting mass
failures were initially indistinguishable from a real regression.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions