Skip to content

Keep the Vector Set replay channel alive across AofProcessor disposal - #1997

Draft
tiagonapoli wants to merge 1 commit into
tiagonapoli/vs-2-recovery-reentrancyfrom
tiagonapoli/vs-3-aof-replay
Draft

Keep the Vector Set replay channel alive across AofProcessor disposal#1997
tiagonapoli wants to merge 1 commit into
tiagonapoli/vs-2-recovery-reentrancyfrom
tiagonapoli/vs-3-aof-replay

Conversation

@tiagonapoli

Copy link
Copy Markdown
Collaborator

Stacked on #1996 (which is stacked on #1995) — review those first; this PR targets tiagonapoli/vs-2-recovery-reentrancy, so its diff shows only this change.

Problem

AofProcessor.Dispose() called VectorManager.ShutdownReplayTasks(), which is a permanent shutdown — it completes the replication replay channel writer and parks the replay tasks for good.

But the VectorManager belongs to the database, not to the processor, and a node constructs and disposes several AofProcessors over its lifetime. SingleDatabaseManager.ReplayAOF builds a transient one during startup AOF recovery.

So on any node started with Recover enabled — a normal production replica config — the channel was already closed before the node began serving. From that point HandleVectorSetAddReplication could no longer queue:

var queued = replicationReplayChannel.Writer.TryWrite(...);
if (!queued) replicationBlockEvent.Decrement();   // no log, no error

It discards whatever it cannot queue, silently. The node dropped every VADD replicated to it while continuing to report healthy AOF offsets — silent data loss with no signal at all.

Fix

Dispose() now only drains. VectorManager.Dispose() still performs the real shutdown when the database goes away.

Tests

Adds ClusterVectorSetRecoveryTests, which restarts the replica, the primary, and both nodes with tryRecover: true — the configuration that exposed this.

Note these tests also depend on #1996: without it, an aborted recovery leaves the store corrupt enough that a subsequent read dies with AccessViolationException in TsavoriteKV.TraceBackForKeyMatch, taking down the test host. That is why this is the top of the stack.

Validation

  • 19/19 pass with the fix (full suite, top of stack).
  • All 3 recovery tests fail without it.
  • Regressions green: Garnet.test.cluster.vectorsets 80/80, replication.disklesssync 28/28, replication 103/103. ShutdownReplayTasks() is left public but now has no callers.

AofProcessor.Dispose called VectorManager.ShutdownReplayTasks, which is a
permanent shutdown: it completes the replication replay channel writer and
parks the replay tasks for good.

The VectorManager, however, belongs to the database rather than to the
processor, and a node constructs and disposes several AofProcessors over its
lifetime. SingleDatabaseManager.ReplayAOF builds a transient one during
startup AOF recovery, so on any node started with Recover enabled the channel
was already closed before the node began serving. From that point
HandleVectorSetAddReplication could no longer queue, and it discards whatever
it cannot queue without logging, so the node silently dropped every VADD
replicated to it while reporting healthy AOF offsets.

Dispose now only drains. VectorManager.Dispose still performs the real
shutdown when the database goes away.

Adds ClusterVectorSetRecoveryTests, which restarts the replica, the primary,
and both nodes with tryRecover enabled - the configuration that exposed this.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 9cdb19eb-7ffa-44cb-ab05-566f44d79d70
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant