Expire stale refs in ExpireSnapshots#3246
Conversation
alessandro-nori
force-pushed
the
anori/expire_refs
branch
3 times, most recently
from
73e24a0 to
8373d34
Compare
| MIN_SNAPSHOTS_TO_KEEP = "history.expire.min-snapshots-to-keep" | ||
| MIN_SNAPSHOTS_TO_KEEP_DEFAULT = 1 | ||
|
|
||
| MAX_REF_AGE_MS = "history.expire.max-ref-age-ms" |
There was a problem hiding this comment.
this is not used in this PR but I thought it could be useful for callers of ExpireSnapshot to get the default value for max-ref-age-ms from table properties (like in the Java implementation)
| for snapshot in self._transaction.table_metadata.snapshots: | ||
| if snapshot.timestamp_ms < expire_from and snapshot.snapshot_id not in protected_ids: | ||
| self._snapshot_ids_to_expire.add(snapshot.snapshot_id) | ||
| self._expire_older_than_ms = datetime_to_millis(dt) |
There was a problem hiding this comment.
moved most of the logic to _commit() to avoid order-dependence between older_than() and
remove_expired_refs() calls
| table_v2.catalog.commit_table.return_value = mock_response | ||
|
|
||
| # Remove any refs that protect the snapshots to be expired | ||
| table_v2.metadata = table_v2.metadata.model_copy( |
There was a problem hiding this comment.
removed because it duplicates old line 205?
alessandro-nori
force-pushed
the
anori/expire_refs
branch
5 times, most recently
from
b45c6fe to
dcfdd04
Compare
alessandro-nori
force-pushed
the
anori/expire_refs
branch
from
dcfdd04 to
ad0a92d
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
1 participant
Changes
remove_expired_refs(default_max_ref_age_ms)method onExpireSnapshots: removes branches/tags whose snapshot age exceeds theirmax_ref_age_ms(per-ref setting, with fallback to the method parameter);mainis never removedolder_than()is now lazy — threshold is evaluated in_commit()so that call order withremove_expired_refs()does not affect the result_get_protected_snapshot_ids()to exclude refs marked for removal, so their orphaned snapshots become eligible for age-based expiryMotivation
Mirrors Java's
RemoveSnapshots.computeRetainedRefs()behavior. Previously,ExpireSnapshotsnever removed stale refs even whenmax-ref-age-mswas configured on a branch or tag.This PR was AI-assisted.