Skip to content

meta: make retention GC actually collect, and make it stick - #121

Merged
matrixarkai merged 1 commit into
mainfrom
oss/retention-collect-and-persist
Aug 22, 2026
Merged

meta: make retention GC actually collect, and make it stick#121
matrixarkai merged 1 commit into
mainfrom
oss/retention-collect-and-persist

Conversation

@bjmeetsfo

Copy link
Copy Markdown
Collaborator

Retention GC — the thing that stops meta state growing for the life of the cluster — did not work. Two independent reasons, one commit each.

1. A dropped table was never eligible to be forgotten

Servers and proxies are dropped through the shared state setter, which stamps a drop time. Tables have their own path, apply_delete_table, and it never stamped one.

Retention deliberately treats a missing drop time as "this tombstone predates the feature, leave it alone" — otherwise the first round after an upgrade would purge the entire history. So a table dropped through the only path anyone uses looked exactly like a pre-upgrade tombstone, and no dropped table was ever collected. Neither were the shard routes underneath it, which retention only purges along with their table.

One line, next to where the state is set. The stamp keeps the first drop time, so a repeated drop cannot restart the clock.

Worth knowing while reading this: add_table rejects a key that already exists, dropped included. So until retention actually collects a table, its name stays permanently unusable — dropping and re-creating a table was not possible.

2. A purge did not survive a restart

purge_expired_meta edited state directly and recorded no mutation.

The log still holds the register and the drop of everything the round forgot, so on the next start replay brought all of it back, tombstone timestamps included. The GC would forget it again, then again on the next boot. The state it exists to bound never actually shrank, and a snapshot exported after a restart carried resources that had already been collected.

The round now records what it applied. The outcome is recorded rather than the intent to run a round, because retention is computed from the wall clock — re-planning during replay would forget a different set. age_frozen_meta already had this shape, routing its drops through the ordinary recorded setters; this brings purge in line.

An empty round records nothing, which matters when the loop ticks every minute.

Tests

5 new, in meta/retention.rs:

  • a dropped table becomes eligible (fails on main: the plan comes back empty)
  • dropping twice does not restart the clock
  • a purge is not undone by a restart (fails on main: the server, proxy and table all come back)
  • a replayed purge forgets exactly what the live round forgot — a resource the round deliberately kept survives replay
  • an empty round writes nothing to the log

Verification:

  • cargo test -p temporalstore-rust --lib meta -- --test-threads=1257 passed, 0 failed.
  • cargo test -p temporalstore-rust --lib client -- --test-threads=1 and --bin metaserver — both green.
  • cargo build -p temporalstore-rust --bin metaserver — clean.

No new configuration. Retention GC is still off by default (TS_META_RETENTION_GC), because forgetting a resource is not reversible — this only means that when it is switched on, it does what it says.

Retention GC - the thing that stops meta state growing for the life of the
cluster - did not work, for two independent reasons.

A dropped table was never eligible. Servers and proxies are dropped through the
shared state setter, which stamps a drop time; tables have their own path,
apply_delete_table, and it never stamped one. Retention deliberately treats a
missing drop time as "this tombstone predates the feature, leave it alone",
because otherwise the first round after an upgrade would purge the entire
history. So a table dropped through the only path anyone uses looked exactly
like a pre-upgrade tombstone, and no dropped table was ever collected - nor were
the shard routes underneath it, which retention only purges along with their
table. The fix is one call, beside where the state is set; it keeps the first
drop time, so a repeated drop cannot restart the clock.

Worth knowing while reading this: add_table rejects a key that already exists,
dropped included. So until retention actually collects a table, its name stays
permanently unusable - dropping and re-creating a table was not possible.

And a purge did not survive a restart. purge_expired_meta edited state directly
and recorded no mutation, so the log still held the register and the drop of
everything the round forgot; replay brought all of it back, tombstone timestamps
included, and the GC forgot it again on every boot. The state it exists to bound
never actually shrank, and a snapshot exported after a restart carried resources
that had already been collected. The round now records what it applied. The
outcome is recorded rather than the intent to run a round, because retention is
computed from the wall clock and re-planning during replay would forget a
different set. age_frozen_meta already had this shape, routing its drops through
the ordinary recorded setters; this brings purge in line. An empty round records
nothing, which matters when the loop ticks every minute.

Five tests: a dropped table becomes eligible, dropping twice does not restart
its clock, a purge is not undone by a restart, a replayed purge forgets exactly
what the live round forgot, and an empty round writes nothing to the log.

Retention GC is still off by default (TS_META_RETENTION_GC), because forgetting
a resource is not reversible. This only means that when it is switched on, it
does what it says.
@bjmeetsfo
bjmeetsfo requested a review from superhaiou as a code owner August 22, 2026 03:12
@matrixarkai
matrixarkai merged commit 581ad0d into main Aug 22, 2026
6 checks passed
@matrixarkai
matrixarkai deleted the oss/retention-collect-and-persist branch August 22, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants