meta: stop routing a shard to a server that is out of service - #124
Merged
Conversation
build_shards builds its placement candidates from servers whose state is Normal. The recorded owner of a shard is the one entry that reaches the topology without passing that filter: it is pushed straight in as a replica and taken verbatim as the primary. So for every shard a server owns, freezing or dropping that server changed nothing about where clients were told to read. The lever exists, the metaserver reports the server as frozen, and the routing table keeps pointing at it. The dropped case is the plainer one - a dropped server is not coming back, so naming it is simply false. The rest of the metaserver already agrees. finish_load rejects a frozen server, placement skips it, the divergence check counts only Normal servers as live, and retention refuses to forget a dropped server that still owns a shard, on the grounds that the route names it and forgetting the server strands the route. Every one of those knows a route to a non-serving server is a problem. The routing table itself did not. A shard whose owner is not serving is now reported with no primary and no replica entry for that server. It deliberately does not fall through to another candidate: the old code ended in .or_else(replicas.first()), and letting a stale route land there would nominate a server that has never loaded this shard, so a client that followed the nomination would read an empty shard and believe it. No primary is the honest answer - the shard exists and nothing is serving it. Two cases must keep working and are pinned by tests. A shard with no owner recorded yet still gets a proposed placement, which is how a new table is told where its shards should go. And a route naming a server the metaserver has no record of is kept, because a route can outlive the server record it names, and treating unknown as out of service would unroute shards the metaserver has simply not been told about yet. Six tests. Four fail on main: not routed to a frozen owner, not routed to a dropped owner, no stand-in primary for a stale route, and unfreezing putting the shard back. The other two hold the edges down and pass either way. This does not move the shard - that is rebalance's job, or an operator's. It only stops the metaserver naming a server it has itself marked out of service.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Freezing a server did not stop clients being sent to it
build_shardsbuilds its placement candidates from servers whose state isNormal. The recorded owner of a shard is the one entry that reaches the topology without passing that filter — it is pushed straight in as a replica, and taken verbatim as the primary.So for every shard a server owns, freezing or dropping that server changed nothing about where clients were told to read. The lever exists, the metaserver reports the server as frozen, and the routing table keeps pointing at it.
The dropped case is the plainer one: a dropped server is not coming back, and naming it is simply false. The rest of the metaserver already agrees —
finish_loadrejects a frozen server, placement skips it, the divergence check treats onlyNormalservers as live, and retention refuses to forget a dropped server that still owns a shard because "the route names it, so forgetting the server strands the route". Every one of those paths knows a route to a non-serving server is a problem. The routing table itself did not.The fix, and the two edges it has to not break
A shard whose owner is not serving is reported with no primary and no replica entry for that server.
It does not fall through to another candidate. The old code ended in
.or_else(|| replicas.first().cloned()), and letting a stale route land there would nominate a server that has never loaded this shard. A client that followed the nomination would read an empty shard and believe it. No primary is the honest answer: the shard exists, and nothing is currently serving it.Two cases must keep working, and both are pinned by tests:
Tests
6 new. Four of them fail on
mainexactly as you would expect:The other two — proposed placement for an unowned shard, and an unknown owner being kept — pass before and after; they are there to hold the edges down.
Verification:
cargo test -p temporalstore-rust --lib meta -- --test-threads=1— 258 passed, 0 failed.cargo check -p temporalstore-rust --all-targets— clean.What this is not
This does not move the shard. Getting it served again is rebalance's job, or an operator's. This only stops the metaserver naming a server it has itself marked out of service.
The full
--librun is 997 passed / 4 failed; those four (raft::tests::part1::raft_rejects_electing_stale_replica_until_it_catches_up,request_vote_higher_term_resets_prior_vote_before_decision,engine::tests::part3::recovery_validates_all_timestamped_kv_page_families,data_node::tests::part3::storage_manager_runtime_supports_stop_pause_resume_jitter_backoff_and_phase_flags) fail identically on a branch that touches none of those areas, so they are not from this change.