Skip to content

Make Vector Set recovery re-entrant so disk-based full sync sanitizes handles - #1996

Draft
tiagonapoli wants to merge 1 commit into
tiagonapoli/vs-1-disklessfrom
tiagonapoli/vs-2-recovery-reentrancy
Draft

Make Vector Set recovery re-entrant so disk-based full sync sanitizes handles#1996
tiagonapoli wants to merge 1 commit into
tiagonapoli/vs-1-disklessfrom
tiagonapoli/vs-2-recovery-reentrancy

Conversation

@tiagonapoli

Copy link
Copy Markdown
Collaborator

Stacked on #1 — review that one first; this PR targets tiagonapoli/vs-1-diskless, so its diff shows only this change.

Problem

VectorManager collects the contexts it sees during recovery into recoveredIndexes / recoveredMetadata, and ResumePostRecovery reconciles them against the recovered store. Both were released at the end of that method, on the assumption that a process recovers exactly once.

A replica does not. It re-recovers the store on every disk-based full sync, and nothing re-ran ResumePostRecovery on that path.

So the second recovery threw NullReferenceException out of RecoveredVectorSetIndexKey. That exception escaped ClearBitsOnPage — the per-record walk that fires OnDiskRead — so the walk aborted after the first record on the page. Every Vector Set record behind it kept the primary's raw DiskANN handle.

Instrumented recovery of three vector sets, before the fix:

CBOP page=0 startLA=0 endLA=16777216 until=156296 endOff=156296 snapFrom=64
  rec off=64   size=8224  ->  msf_            (context metadata)
  rec off=8288 size=88    ->  {vsdisk}dblarge (sanitized OK)
  EX System.NullReferenceException
     at VectorManager.RecoveredVectorSetIndexKey(LogRecord&) VectorManager.cs:384
     at GarnetRecordTriggers.OnRecoverySnapshotRead(LogRecord&)
     at TsavoriteKV.ClearBitsOnPage(...) Recovery.cs:1259

dbsmall and dbmedium were never visited, and came up on the replica holding IndexPtr == primary's pointer.

The half-finished recovery also leaked the checkpoint file handle, which is what made teardown hang ~60s deleting the directory. The disk-based fixture went 1m07s → 7s.

This path matters more than the diskless one in practice, since ReplicaDisklessSync defaults to false.

Fix

  • Recovery state is cleared rather than released, so a second recovery starts from a clean slate instead of a null reference.
  • A replica's disk-based recovery reconciles it via the new StoreWrapper.RecoverVectorSets(), mirroring what the startup path already did.

Validation

  • 16/16 pass with the fix.
  • All 4 disk-based tests fail without it.
  • Regressions green: Garnet.test.cluster.vectorsets 80/80, replication.disklesssync 28/28, replication 103/103.

… handles

VectorManager collects the contexts it sees during recovery into
recoveredIndexes/recoveredMetadata and ResumePostRecovery reconciles them
against the recovered store. Both were released at the end of that method, on
the assumption that a process recovers exactly once. A replica does not: it
re-recovers the store on every disk-based full sync, and nothing re-ran
ResumePostRecovery on that path.

The second recovery therefore threw NullReferenceException out of
RecoveredVectorSetIndexKey. That exception escaped ClearBitsOnPage, which is
the per-record walk that fires OnDiskRead, so the walk aborted after the first
record on the page. Every Vector Set record behind it kept the primary's raw
DiskANN handle, and the half-finished recovery also leaked the checkpoint file
handle, which is what made teardown hang for a minute deleting the directory.

The recovery state is now cleared rather than released, and a replica's
disk-based recovery reconciles it through StoreWrapper.RecoverVectorSets.

This path matters more than the diskless one in practice, since
ReplicaDisklessSync defaults to false.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9cdb19eb-7ffa-44cb-ab05-566f44d79d70
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant