meta: make retention GC actually collect, and make it stick - #121
Merged
Conversation
Retention GC - the thing that stops meta state growing for the life of the cluster - did not work, for two independent reasons. A dropped table was never eligible. Servers and proxies are dropped through the shared state setter, which stamps a drop time; tables have their own path, apply_delete_table, and it never stamped one. Retention deliberately treats a missing drop time as "this tombstone predates the feature, leave it alone", because otherwise the first round after an upgrade would purge the entire history. So a table dropped through the only path anyone uses looked exactly like a pre-upgrade tombstone, and no dropped table was ever collected - nor were the shard routes underneath it, which retention only purges along with their table. The fix is one call, beside where the state is set; it keeps the first drop time, so a repeated drop cannot restart the clock. Worth knowing while reading this: add_table rejects a key that already exists, dropped included. So until retention actually collects a table, its name stays permanently unusable - dropping and re-creating a table was not possible. And a purge did not survive a restart. purge_expired_meta edited state directly and recorded no mutation, so the log still held the register and the drop of everything the round forgot; replay brought all of it back, tombstone timestamps included, and the GC forgot it again on every boot. The state it exists to bound never actually shrank, and a snapshot exported after a restart carried resources that had already been collected. The round now records what it applied. The outcome is recorded rather than the intent to run a round, because retention is computed from the wall clock and re-planning during replay would forget a different set. age_frozen_meta already had this shape, routing its drops through the ordinary recorded setters; this brings purge in line. An empty round records nothing, which matters when the loop ticks every minute. Five tests: a dropped table becomes eligible, dropping twice does not restart its clock, a purge is not undone by a restart, a replayed purge forgets exactly what the live round forgot, and an empty round writes nothing to the log. Retention GC is still off by default (TS_META_RETENTION_GC), because forgetting a resource is not reversible. This only means that when it is switched on, it does what it says.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Retention GC — the thing that stops meta state growing for the life of the cluster — did not work. Two independent reasons, one commit each.
1. A dropped table was never eligible to be forgotten
Servers and proxies are dropped through the shared state setter, which stamps a drop time. Tables have their own path,
apply_delete_table, and it never stamped one.Retention deliberately treats a missing drop time as "this tombstone predates the feature, leave it alone" — otherwise the first round after an upgrade would purge the entire history. So a table dropped through the only path anyone uses looked exactly like a pre-upgrade tombstone, and no dropped table was ever collected. Neither were the shard routes underneath it, which retention only purges along with their table.
One line, next to where the state is set. The stamp keeps the first drop time, so a repeated drop cannot restart the clock.
Worth knowing while reading this:
add_tablerejects a key that already exists, dropped included. So until retention actually collects a table, its name stays permanently unusable — dropping and re-creating a table was not possible.2. A purge did not survive a restart
purge_expired_metaedited state directly and recorded no mutation.The log still holds the register and the drop of everything the round forgot, so on the next start replay brought all of it back, tombstone timestamps included. The GC would forget it again, then again on the next boot. The state it exists to bound never actually shrank, and a snapshot exported after a restart carried resources that had already been collected.
The round now records what it applied. The outcome is recorded rather than the intent to run a round, because retention is computed from the wall clock — re-planning during replay would forget a different set.
age_frozen_metaalready had this shape, routing its drops through the ordinary recorded setters; this brings purge in line.An empty round records nothing, which matters when the loop ticks every minute.
Tests
5 new, in
meta/retention.rs:main: the plan comes back empty)main: the server, proxy and table all come back)Verification:
cargo test -p temporalstore-rust --lib meta -- --test-threads=1— 257 passed, 0 failed.cargo test -p temporalstore-rust --lib client -- --test-threads=1and--bin metaserver— both green.cargo build -p temporalstore-rust --bin metaserver— clean.No new configuration. Retention GC is still off by default (
TS_META_RETENTION_GC), because forgetting a resource is not reversible — this only means that when it is switched on, it does what it says.