meta: let a single shard be taken out of service - #119
Open
bjmeetsfo wants to merge 2 commits into
Open
Conversation
ShardLocation records where a shard lives and nothing about whether it should be served, so there is no way to take one shard out of service. The only lever is freezing its whole table, which stops every other shard in that table too - the wrong granularity for the cases it comes up in, where one shard is corrupt or hot or under investigation and the rest of the table is fine. A state on ShardLocation, with freeze, unfreeze and drop. A frozen shard keeps its owner entry, which is what lets unfreezing put it back where it was, but stops being served. Dropping removes the route outright rather than leaving a tombstone: a shard has no identity beyond where it is served from. The field defaults to serving, so entries written before this load unchanged. Freezing is only useful if the rest of the metaserver respects it, so the four paths that read placement now go through one serving-only view. Topology hands clients no primary and no replicas for it. Neither rebalance planner will move it, because moving a shard an operator deliberately froze is the planner undoing a decision. The divergence check will not flag it, because its owner not serving it is exactly what freezing means. The orphan guard does not count it as something a conviction would strand. The tests caught a hole in the first attempt. Filtering the frozen shard's recorded owner out of build_shards was not enough - the general candidate scan still picked a primary through the replicas.first() fallback, so the shard stayed routable. A shard that is not serving now gets no placement computed at all. 9 tests, including one shard freezing while the rest of its table keeps serving, rebalance and the divergence check both leaving it alone, and state surviving a snapshot round trip and mutation-log replay.
The new field was added to every ShardLocation literal in the tree, including two in src/bin. In a binary `crate::` is that binary, not the library, so `crate::meta::MetaEntityState` does not resolve there. Caught by CI, not by me: I compiled the library and the metaserver binary, and this crate has several more.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
ShardLocationrecords where a shard lives and nothing about whether it should be served. So there is no way to take one shard out of service. The only lever is freezing its whole table, which stops every other shard in that table too.That is the wrong granularity for the cases it comes up in: one shard is corrupt, or hot, or under investigation, and the rest of the table is fine.
What this adds
A
stateonShardLocation, and three operations:POST /shards/freeze,/shards/unfreeze,/shards/drop.A frozen shard keeps its owner entry — that is what lets unfreezing put it back where it was — but stops being served. Dropping removes the route outright rather than leaving a tombstone: a shard has no identity beyond where it is served from.
The field defaults to
Normal, so shard entries written before this load as serving.The part that matters: nothing may fight the operator
Freezing a shard is only useful if the rest of the metaserver respects it. Four paths read placement, and all of them now go through one
serving_shard_ownersview:A hole the tests caught
My first attempt filtered the frozen shard's recorded owner out of
build_shards. That was not enough: the general candidate scan still picked a primary for it through thereplicas.first()fallback, so the shard stayed routable and the topology test failed withprimary: Some("node-a").A shard that is not serving now gets no placement computed at all — the loop emits an empty entry and moves on. That is the honest semantic: the shard still occupies its key range, but nothing is serving it.
Tests
9 new tests: freezing one shard leaving the rest of the table serving (and the table itself untouched), unfreezing restoring it, the owner entry surviving a freeze, rebalance leaving it alone, the divergence check ignoring it, drop removing the route and reporting
shard_not_foundon a second attempt, unknown/unchanged rejections, the topology-version bump, and state surviving both a snapshot round trip and mutation-log replay.Verification:
cargo test -p temporalstore-rust --lib meta -- --test-threads=1— 261 passed, 0 failed.cargo test -p temporalstore-rust --lib client -- --test-threads=1— 33 passed, 0 failed.cargo test -p temporalstore-rust --bin metaserver -- --test-threads=1— 18 passed, 0 failed.cargo build -p temporalstore-rust --bin metaserver— clean.No gate: the field defaults to serving, so nothing changes until an operator freezes something.