Skip to content

meta: a resource that comes back gets a fresh drop clock - #125

Merged
bjmeetsfo merged 2 commits into
mainfrom
oss/revival-resets-retention-clock
Aug 24, 2026
Merged

meta: a resource that comes back gets a fresh drop clock#125
bjmeetsfo merged 2 commits into
mainfrom
oss/revival-resets-retention-clock

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

A resource that comes back keeps the clock from the last time it left

stamp_dropped_since keeps the first time it is given:

// Keep the first drop time: re-dropping an already dropped resource
// must not restart its retention clock.
state.dropped_since_ms.entry(key.to_string()).or_insert(now_ms);

That is right for drop-then-drop. It is wrong across a revival, and re-registration never cleared the entry — so the tombstone outlived the drop it recorded.

The consequence shows up on the second drop, which is where nobody looks:

  1. node-a is dropped in January. Clock stamped.
  2. node-a comes back an hour later and serves for six months.
  3. node-a is dropped again in July.
  4. Retention reads the January stamp, computes an age of six months, and collects it on the next round — with none of the grace the retention window exists to give.

The grace period is what lets a node that comes back keep its identity. Here it was already spent.

The fix

Re-registering a server or a proxy clears its drop clock, so the next drop starts a fresh one. Four lines across the two registration paths.

The other direction still holds and is pinned by its own test: repeatedly dropping an already dropped server cannot hold off collection.

A test that did not test its own name

a_resource_that_comes_back_loses_its_tombstone_clock asserted only that the revived server was not collected. That was already guaranteed by the candidate filter, which requires state Dropped — the assertion passed with the clock untouched. It now checks the clock, which is what its name claims.

Tests

4 new plus that strengthened one. Two of the new ones fail on main:

test a_server_that_comes_back_starts_a_fresh_drop_clock ... FAILED
test a_proxy_that_comes_back_starts_a_fresh_drop_clock  ... FAILED

The other two are the consequence (a revived server dropped again is not immediately eligible) and the guard (re-dropping without a revival still keeps the first clock).

Verification:

  • cargo test -p temporalstore-rust --lib meta -- --test-threads=1256 passed, 0 failed.
  • cargo check -p temporalstore-rust --all-targets — clean.

Retention GC is off by default, so this changes nothing until it is switched on. It is worth fixing anyway: the wrong value is being recorded now, and it is recorded into snapshots that outlive the setting.

@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 22, 2026 06:33
@bjmeetsfo
bjmeetsfo force-pushed the oss/revival-resets-retention-clock branch from b34242b to 1029c46 Compare August 23, 2026 17:50
supermengm and others added 2 commits August 23, 2026 21:04
stamp_dropped_since keeps the first time it is given, deliberately, so that
re-dropping an already dropped resource cannot restart its retention clock. That
is right for drop-then-drop and wrong across a revival, and re-registration
never cleared the entry, so the tombstone outlived the drop it recorded.

The consequence shows up on the second drop, which is where nobody looks. A node
dropped in January, back an hour later, serving for six months, dropped again in
July: retention reads the January stamp, computes an age of six months, and
collects it on the next round with none of the grace the window exists to give.
The grace period is what lets a node that comes back keep its identity, and it
had already been spent.

Re-registering a server or a proxy now clears its drop clock. The other
direction still holds and has its own test: repeatedly dropping an already
dropped server cannot hold off collection.

One existing test did not test its own name.
a_resource_that_comes_back_loses_its_tombstone_clock asserted only that the
revived server was not collected, which the candidate filter already guarantees
on its own since it requires state Dropped - the assertion passed with the clock
untouched. It now checks the clock.

Four new tests. Two fail on main: a server and a proxy that come back still
carrying their old clock. The others are the consequence and the guard.

Retention GC is off by default, so this changes nothing until it is switched on.
Worth fixing anyway: the wrong value is being recorded now, into snapshots that
outlive the setting.
Rebasing onto main crossed two changes that landed while this branch was open:
the metadata counters became atomics, and server registration gained a hardware
shape. This branch still had the old counter line and helpers that predate the
field.

Neither is a conflict of intent -- the counter is bumped either way, and a test
server with no NUMA topology is what every other helper here declares.
@bjmeetsfo
bjmeetsfo force-pushed the oss/revival-resets-retention-clock branch from 1029c46 to 5a28463 Compare August 24, 2026 04:06
@bjmeetsfo
bjmeetsfo merged commit 14b4b45 into main Aug 24, 2026
6 of 7 checks passed
@bjmeetsfo
bjmeetsfo deleted the oss/revival-resets-retention-clock branch August 24, 2026 04:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants