Skip to content

meta: let a single shard be taken out of service - #119

Open
bjmeetsfo wants to merge 2 commits into
mainfrom
oss/per-shard-serving-state
Open

meta: let a single shard be taken out of service#119
bjmeetsfo wants to merge 2 commits into
mainfrom
oss/per-shard-serving-state

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

The problem

ShardLocation records where a shard lives and nothing about whether it should be served. So there is no way to take one shard out of service. The only lever is freezing its whole table, which stops every other shard in that table too.

That is the wrong granularity for the cases it comes up in: one shard is corrupt, or hot, or under investigation, and the rest of the table is fine.

What this adds

A state on ShardLocation, and three operations: POST /shards/freeze, /shards/unfreeze, /shards/drop.

A frozen shard keeps its owner entry — that is what lets unfreezing put it back where it was — but stops being served. Dropping removes the route outright rather than leaving a tombstone: a shard has no identity beyond where it is served from.

The field defaults to Normal, so shard entries written before this load as serving.

The part that matters: nothing may fight the operator

Freezing a shard is only useful if the rest of the metaserver respects it. Four paths read placement, and all of them now go through one serving_shard_owners view:

  • Topology — a frozen shard is handed to clients with no primary and no replicas.
  • Rebalance (both planners) — will not move it. Moving a shard an operator deliberately froze is the planner undoing a decision.
  • Divergence check — will not flag it. Its owner not serving it is exactly what freezing means, so flagging it would have the checker "repair" the operator's intent.
  • Orphan guard — a frozen shard does not count as something a conviction would strand.

A hole the tests caught

My first attempt filtered the frozen shard's recorded owner out of build_shards. That was not enough: the general candidate scan still picked a primary for it through the replicas.first() fallback, so the shard stayed routable and the topology test failed with primary: Some("node-a").

A shard that is not serving now gets no placement computed at all — the loop emits an empty entry and moves on. That is the honest semantic: the shard still occupies its key range, but nothing is serving it.

Tests

9 new tests: freezing one shard leaving the rest of the table serving (and the table itself untouched), unfreezing restoring it, the owner entry surviving a freeze, rebalance leaving it alone, the divergence check ignoring it, drop removing the route and reporting shard_not_found on a second attempt, unknown/unchanged rejections, the topology-version bump, and state surviving both a snapshot round trip and mutation-log replay.

Verification:

  • cargo test -p temporalstore-rust --lib meta -- --test-threads=1261 passed, 0 failed.
  • cargo test -p temporalstore-rust --lib client -- --test-threads=1 — 33 passed, 0 failed.
  • cargo test -p temporalstore-rust --bin metaserver -- --test-threads=1 — 18 passed, 0 failed.
  • cargo build -p temporalstore-rust --bin metaserver — clean.

No gate: the field defaults to serving, so nothing changes until an operator freezes something.

ShardLocation records where a shard lives and nothing about whether it should
be served, so there is no way to take one shard out of service. The only lever
is freezing its whole table, which stops every other shard in that table too -
the wrong granularity for the cases it comes up in, where one shard is corrupt
or hot or under investigation and the rest of the table is fine.

A state on ShardLocation, with freeze, unfreeze and drop. A frozen shard keeps
its owner entry, which is what lets unfreezing put it back where it was, but
stops being served. Dropping removes the route outright rather than leaving a
tombstone: a shard has no identity beyond where it is served from. The field
defaults to serving, so entries written before this load unchanged.

Freezing is only useful if the rest of the metaserver respects it, so the four
paths that read placement now go through one serving-only view. Topology hands
clients no primary and no replicas for it. Neither rebalance planner will move
it, because moving a shard an operator deliberately froze is the planner undoing
a decision. The divergence check will not flag it, because its owner not serving
it is exactly what freezing means. The orphan guard does not count it as
something a conviction would strand.

The tests caught a hole in the first attempt. Filtering the frozen shard's
recorded owner out of build_shards was not enough - the general candidate scan
still picked a primary through the replicas.first() fallback, so the shard
stayed routable. A shard that is not serving now gets no placement computed at
all.

9 tests, including one shard freezing while the rest of its table keeps serving,
rebalance and the divergence check both leaving it alone, and state surviving a
snapshot round trip and mutation-log replay.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 22, 2026 02:22
The new field was added to every ShardLocation literal in the tree, including
two in src/bin. In a binary `crate::` is that binary, not the library, so
`crate::meta::MetaEntityState` does not resolve there.

Caught by CI, not by me: I compiled the library and the metaserver binary, and
this crate has several more.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants