OAK-12280: Provide a way to reconcile the offline GC result back #2977
Open
royteeuwen wants to merge 1 commit into
Open
OAK-12280: Provide a way to reconcile the offline GC result back #2977royteeuwen wants to merge 1 commit into
royteeuwen wants to merge 1 commit into
Conversation
royteeuwen
changed the title
After an offline `datastore --collect-garbage` (non mark-only) run, the BlobIdTracker used by online GC is now reconciled automatically: blob IDs deleted by this run are removed from the local tracker files and the SharedDataStore snapshot (the `*.refs` record). This stops online GC from repeatedly logging `DataStoreException: Record ... does not exist` for ghost blobs and failing to prune them from the tracker. The online sweep is also made tolerant of ghost blobs: `countDeleteChunk` returns -1 (or a DataStoreException is caught) and the id is removed from the tracker instead of failing, so pre-existing stale entries get cleaned up. The tracker lives at <repository-home>/blobids. The repository home is derived from the segment store path (its parent) and can be overridden with the new optional --repository-home. This replaces the earlier --blobids-path option, which pointed one level too deep: BlobIdTracker.build() appends "blobids", so passing the blobids dir resolved to <...>/blobids/blobids and left the live local tracker untouched (ghosts returned after a restart).
royteeuwen
force-pushed
the
fix/ghost-blob-gc-datastore-exception
branch
from
6f3a459 to
75c873c
Compare
Contributor
|
@royteeuwen The blobId Inline tracker was added as an experimental feature. The feature only solved the purpose if very few repos shared the data store and consistently enabled DSGC. Long-pending DSGC tasks or irregular schedules worsen the problem. So disabling this feature is the correct way to proceed here. In future releases, we are even planning to remove this feature. cc @amit-jain |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There is a blobid tracker used by the online GC, which should be reconciled by the offline GC run command. Make sure that if the file is not reconciled, the online GC still cleans up the blobs that no longer exist from the blobid tracker