fix(cluster): freeze slot writes until topology update after migration (#3147) - #3576
Open
advisedy wants to merge 3 commits into
Open
fix(cluster): freeze slot writes until topology update after migration (#3147)#3576advisedy wants to merge 3 commits into
advisedy wants to merge 3 commits into
Conversation
Member
|
Your PR changes the priority from availability to consistency. There is no perfect solution here; this is fundamentally a trade-off. The problem is much more complex than it appears on the surface, and we may need to evaluate it together with the controller’s behavior and the actual use cases. If you do not have relevant experience in distributed systems design, it may be difficult for you to move this PR forward on your own. |
git-hulk
self-requested a review
August 5, 2026 01:57
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3147.
Problem
After a slot migration finishes on the data plane, the source notifies the
destination that import succeeded. The destination used to record the slot in
imported_slots_and could already serve ordinary clients, while clustertopology still pointed at the source until the controller applied
SETSLOT/SETNODES.If the destination’s success reply back to the source is lost, the source may
never finish
migrated_slots_bookkeeping. Topology still routes the slot tothe source, so the source can keep accepting ordinary traffic, and the
destination can already accept ordinary traffic via
imported_slots_. Bothnodes then look writable for the same slot and can diverge.
In the same window there is a related write-loss risk: if the source fails or
is replaced before the controller publishes the new topology, writes that still
target the old source may not land on the destination that already holds the
migrated data. Freeze reduces how much the data plane will accept in that gap;
it does not replace controller failover or topology publish.
Approach
For the dual-write case and this write-loss case, I think a "full" fix is
actually fairly complex. If we allow the unwritable window to cover the
whole interval from slot migration success until the topology is updated,
then first we can prevent dual-write caused by the destination becoming
writable early, and second we can prevent write loss when the source is
lost before the topology update. My understanding is that if we insist on
not widening that unwritable window, a better implementation would need
not only extra checks on this side about the destination's imported_slots
state, but also more controller-side handling when the source is lost.
That is more complicated than simply widening the unwritable window, so
this PR chooses the latter.
One more point: although the issue discussion said imported_slots_ still
had a reason to exist, under the approach of widening the unwritable
window I do not think it is meaningful anymore, so this PR removes it.
Changes
After migration success, the source rejects writes to slots in
migrated_slots_ with TRYAGAIN instead of MOVED, while reads on
the source remain available. importing_slot_range is cleared when
import succeeds, fails, or the import link drops. The
migrator forbidden range is released when no migrated slots remain after a
topology update. imported_slots_ and the related bookkeeping are removed.
AI
Integration test updates were drafted with help from an LLM. I re-reviewed
the generated test changes myself before including them in this PR.