meta: a resource that comes back gets a fresh drop clock - #125
Merged
Conversation
bjmeetsfo
force-pushed
the
oss/revival-resets-retention-clock
branch
from
August 23, 2026 17:50
b34242b to
1029c46
Compare
stamp_dropped_since keeps the first time it is given, deliberately, so that re-dropping an already dropped resource cannot restart its retention clock. That is right for drop-then-drop and wrong across a revival, and re-registration never cleared the entry, so the tombstone outlived the drop it recorded. The consequence shows up on the second drop, which is where nobody looks. A node dropped in January, back an hour later, serving for six months, dropped again in July: retention reads the January stamp, computes an age of six months, and collects it on the next round with none of the grace the window exists to give. The grace period is what lets a node that comes back keep its identity, and it had already been spent. Re-registering a server or a proxy now clears its drop clock. The other direction still holds and has its own test: repeatedly dropping an already dropped server cannot hold off collection. One existing test did not test its own name. a_resource_that_comes_back_loses_its_tombstone_clock asserted only that the revived server was not collected, which the candidate filter already guarantees on its own since it requires state Dropped - the assertion passed with the clock untouched. It now checks the clock. Four new tests. Two fail on main: a server and a proxy that come back still carrying their old clock. The others are the consequence and the guard. Retention GC is off by default, so this changes nothing until it is switched on. Worth fixing anyway: the wrong value is being recorded now, into snapshots that outlive the setting.
Rebasing onto main crossed two changes that landed while this branch was open: the metadata counters became atomics, and server registration gained a hardware shape. This branch still had the old counter line and helpers that predate the field. Neither is a conflict of intent -- the counter is bumped either way, and a test server with no NUMA topology is what every other helper here declares.
bjmeetsfo
force-pushed
the
oss/revival-resets-retention-clock
branch
from
August 24, 2026 04:06
1029c46 to
5a28463
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A resource that comes back keeps the clock from the last time it left
stamp_dropped_sincekeeps the first time it is given:That is right for drop-then-drop. It is wrong across a revival, and re-registration never cleared the entry — so the tombstone outlived the drop it recorded.
The consequence shows up on the second drop, which is where nobody looks:
node-ais dropped in January. Clock stamped.node-acomes back an hour later and serves for six months.node-ais dropped again in July.The grace period is what lets a node that comes back keep its identity. Here it was already spent.
The fix
Re-registering a server or a proxy clears its drop clock, so the next drop starts a fresh one. Four lines across the two registration paths.
The other direction still holds and is pinned by its own test: repeatedly dropping an already dropped server cannot hold off collection.
A test that did not test its own name
a_resource_that_comes_back_loses_its_tombstone_clockasserted only that the revived server was not collected. That was already guaranteed by the candidate filter, which requires stateDropped— the assertion passed with the clock untouched. It now checks the clock, which is what its name claims.Tests
4 new plus that strengthened one. Two of the new ones fail on
main:The other two are the consequence (a revived server dropped again is not immediately eligible) and the guard (re-dropping without a revival still keeps the first clock).
Verification:
cargo test -p temporalstore-rust --lib meta -- --test-threads=1— 256 passed, 0 failed.cargo check -p temporalstore-rust --all-targets— clean.Retention GC is off by default, so this changes nothing until it is switched on. It is worth fixing anyway: the wrong value is being recorded now, and it is recorded into snapshots that outlive the setting.