Skip to content

meta: stop routing a shard to a server that is out of service - #124

Merged
matrixarkai merged 1 commit into
mainfrom
oss/topology-skips-dead-owner
Aug 22, 2026
Merged

meta: stop routing a shard to a server that is out of service#124
matrixarkai merged 1 commit into
mainfrom
oss/topology-skips-dead-owner

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

Freezing a server did not stop clients being sent to it

build_shards builds its placement candidates from servers whose state is Normal. The recorded owner of a shard is the one entry that reaches the topology without passing that filter — it is pushed straight in as a replica, and taken verbatim as the primary.

So for every shard a server owns, freezing or dropping that server changed nothing about where clients were told to read. The lever exists, the metaserver reports the server as frozen, and the routing table keeps pointing at it.

The dropped case is the plainer one: a dropped server is not coming back, and naming it is simply false. The rest of the metaserver already agrees — finish_load rejects a frozen server, placement skips it, the divergence check treats only Normal servers as live, and retention refuses to forget a dropped server that still owns a shard because "the route names it, so forgetting the server strands the route". Every one of those paths knows a route to a non-serving server is a problem. The routing table itself did not.

The fix, and the two edges it has to not break

A shard whose owner is not serving is reported with no primary and no replica entry for that server.

It does not fall through to another candidate. The old code ended in .or_else(|| replicas.first().cloned()), and letting a stale route land there would nominate a server that has never loaded this shard. A client that followed the nomination would read an empty shard and believe it. No primary is the honest answer: the shard exists, and nothing is currently serving it.

Two cases must keep working, and both are pinned by tests:

  • A shard with no owner recorded yet still gets a proposed placement. That is how a new table is told where its shards should go, and it is a different situation from a route that names someone specific.
  • A route naming a server the metaserver has no record of is kept. A route can outlive the server record it names, and treating unknown as out-of-service would silently unroute shards the metaserver has simply not been told about yet.

Tests

6 new. Four of them fail on main exactly as you would expect:

test a_shard_is_not_routed_to_a_dropped_owner   ... FAILED
test a_shard_is_not_routed_to_a_frozen_owner    ... FAILED
test a_stale_route_does_not_get_a_stand_in_primary ... FAILED
test unfreezing_the_owner_puts_the_shard_back   ... FAILED

The other two — proposed placement for an unowned shard, and an unknown owner being kept — pass before and after; they are there to hold the edges down.

Verification:

  • cargo test -p temporalstore-rust --lib meta -- --test-threads=1258 passed, 0 failed.
  • cargo check -p temporalstore-rust --all-targets — clean.

What this is not

This does not move the shard. Getting it served again is rebalance's job, or an operator's. This only stops the metaserver naming a server it has itself marked out of service.

The full --lib run is 997 passed / 4 failed; those four (raft::tests::part1::raft_rejects_electing_stale_replica_until_it_catches_up, request_vote_higher_term_resets_prior_vote_before_decision, engine::tests::part3::recovery_validates_all_timestamped_kv_page_families, data_node::tests::part3::storage_manager_runtime_supports_stop_pause_resume_jitter_backoff_and_phase_flags) fail identically on a branch that touches none of those areas, so they are not from this change.

build_shards builds its placement candidates from servers whose state is
Normal. The recorded owner of a shard is the one entry that reaches the
topology without passing that filter: it is pushed straight in as a replica and
taken verbatim as the primary.

So for every shard a server owns, freezing or dropping that server changed
nothing about where clients were told to read. The lever exists, the metaserver
reports the server as frozen, and the routing table keeps pointing at it. The
dropped case is the plainer one - a dropped server is not coming back, so naming
it is simply false.

The rest of the metaserver already agrees. finish_load rejects a frozen server,
placement skips it, the divergence check counts only Normal servers as live, and
retention refuses to forget a dropped server that still owns a shard, on the
grounds that the route names it and forgetting the server strands the route.
Every one of those knows a route to a non-serving server is a problem. The
routing table itself did not.

A shard whose owner is not serving is now reported with no primary and no
replica entry for that server. It deliberately does not fall through to another
candidate: the old code ended in .or_else(replicas.first()), and letting a stale
route land there would nominate a server that has never loaded this shard, so a
client that followed the nomination would read an empty shard and believe it. No
primary is the honest answer - the shard exists and nothing is serving it.

Two cases must keep working and are pinned by tests. A shard with no owner
recorded yet still gets a proposed placement, which is how a new table is told
where its shards should go. And a route naming a server the metaserver has no
record of is kept, because a route can outlive the server record it names, and
treating unknown as out of service would unroute shards the metaserver has
simply not been told about yet.

Six tests. Four fail on main: not routed to a frozen owner, not routed to a
dropped owner, no stand-in primary for a stale route, and unfreezing putting the
shard back. The other two hold the edges down and pass either way.

This does not move the shard - that is rebalance's job, or an operator's. It
only stops the metaserver naming a server it has itself marked out of service.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 22, 2026 06:10
@matrixarkai
matrixarkai merged commit 500e8b1 into main Aug 22, 2026
6 checks passed
@matrixarkai
matrixarkai deleted the oss/topology-skips-dead-owner branch August 22, 2026 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants