Make Vector Set recovery re-entrant so disk-based full sync sanitizes handles - #1996
Draft
tiagonapoli wants to merge 1 commit into
Draft
Make Vector Set recovery re-entrant so disk-based full sync sanitizes handles#1996tiagonapoli wants to merge 1 commit into
tiagonapoli wants to merge 1 commit into
Conversation
… handles VectorManager collects the contexts it sees during recovery into recoveredIndexes/recoveredMetadata and ResumePostRecovery reconciles them against the recovered store. Both were released at the end of that method, on the assumption that a process recovers exactly once. A replica does not: it re-recovers the store on every disk-based full sync, and nothing re-ran ResumePostRecovery on that path. The second recovery therefore threw NullReferenceException out of RecoveredVectorSetIndexKey. That exception escaped ClearBitsOnPage, which is the per-record walk that fires OnDiskRead, so the walk aborted after the first record on the page. Every Vector Set record behind it kept the primary's raw DiskANN handle, and the half-finished recovery also leaked the checkpoint file handle, which is what made teardown hang for a minute deleting the directory. The recovery state is now cleared rather than released, and a replica's disk-based recovery reconciles it through StoreWrapper.RecoverVectorSets. This path matters more than the diskless one in practice, since ReplicaDisklessSync defaults to false. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 9cdb19eb-7ffa-44cb-ab05-566f44d79d70
This was referenced Jul 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
VectorManagercollects the contexts it sees during recovery intorecoveredIndexes/recoveredMetadata, andResumePostRecoveryreconciles them against the recovered store. Both were released at the end of that method, on the assumption that a process recovers exactly once.A replica does not. It re-recovers the store on every disk-based full sync, and nothing re-ran
ResumePostRecoveryon that path.So the second recovery threw
NullReferenceExceptionout ofRecoveredVectorSetIndexKey. That exception escapedClearBitsOnPage— the per-record walk that firesOnDiskRead— so the walk aborted after the first record on the page. Every Vector Set record behind it kept the primary's raw DiskANN handle.Instrumented recovery of three vector sets, before the fix:
dbsmallanddbmediumwere never visited, and came up on the replica holdingIndexPtr == primary's pointer.The half-finished recovery also leaked the checkpoint file handle, which is what made teardown hang ~60s deleting the directory. The disk-based fixture went 1m07s → 7s.
This path matters more than the diskless one in practice, since
ReplicaDisklessSyncdefaults tofalse.Fix
StoreWrapper.RecoverVectorSets(), mirroring what the startup path already did.Validation
Garnet.test.cluster.vectorsets80/80,replication.disklesssync28/28,replication103/103.