Skip to content

fix(cluster): freeze slot writes until topology update after migration (#3147) - #3576

Open
advisedy wants to merge 3 commits into
apache:unstablefrom
advisedy:#3147
Open

fix(cluster): freeze slot writes until topology update after migration (#3147)#3576
advisedy wants to merge 3 commits into
apache:unstablefrom
advisedy:#3147

Conversation

@advisedy

@advisedy advisedy commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Fixes #3147.

Problem

After a slot migration finishes on the data plane, the source notifies the
destination that import succeeded. The destination used to record the slot in
imported_slots_ and could already serve ordinary clients, while cluster
topology still pointed at the source until the controller applied SETSLOT /
SETNODES.

If the destination’s success reply back to the source is lost, the source may
never finish migrated_slots_ bookkeeping. Topology still routes the slot to
the source, so the source can keep accepting ordinary traffic, and the
destination can already accept ordinary traffic via imported_slots_. Both
nodes then look writable for the same slot and can diverge.

In the same window there is a related write-loss risk: if the source fails or
is replaced before the controller publishes the new topology, writes that still
target the old source may not land on the destination that already holds the
migrated data. Freeze reduces how much the data plane will accept in that gap;
it does not replace controller failover or topology publish.

Approach

For the dual-write case and this write-loss case, I think a "full" fix is
actually fairly complex. If we allow the unwritable window to cover the
whole interval from slot migration success until the topology is updated,
then first we can prevent dual-write caused by the destination becoming
writable early, and second we can prevent write loss when the source is
lost before the topology update. My understanding is that if we insist on
not widening that unwritable window, a better implementation would need
not only extra checks on this side about the destination's imported_slots
state, but also more controller-side handling when the source is lost.
That is more complicated than simply widening the unwritable window, so
this PR chooses the latter.

One more point: although the issue discussion said imported_slots_ still
had a reason to exist, under the approach of widening the unwritable
window I do not think it is meaningful anymore, so this PR removes it.

Changes

After migration success, the source rejects writes to slots in
migrated_slots_ with TRYAGAIN instead of MOVED, while reads on
the source remain available. importing_slot_range is cleared when
import succeeds, fails, or the import link drops. The
migrator forbidden range is released when no migrated slots remain after a
topology update. imported_slots_ and the related bookkeeping are removed.

AI

Integration test updates were drafted with help from an LLM. I re-reviewed
the generated test changes myself before including them in this PR.

@jihuayu

jihuayu commented Aug 5, 2026

Copy link
Copy Markdown
Member

Your PR changes the priority from availability to consistency. There is no perfect solution here; this is fundamentally a trade-off. The problem is much more complex than it appears on the surface, and we may need to evaluate it together with the controller’s behavior and the actual use cases.

If you do not have relevant experience in distributed systems design, it may be difficult for you to move this PR forward on your own.

@git-hulk
git-hulk self-requested a review August 5, 2026 01:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Possible split brain on migration race condition

2 participants