Skip to content

HBASE-30291 Load backup HFile references once per cleaner run - #8539

Open
Sigma-Ma wants to merge 1 commit into
apache:masterfrom
Sigma-Ma:HBASE-30291-backup-hfile-cleaner-cycle-snapshot
Open

HBASE-30291 Load backup HFile references once per cleaner run#8539
Sigma-Ma wants to merge 1 commit into
apache:masterfrom
Sigma-Ma:HBASE-30291-backup-hfile-cleaner-cycle-snapshot

Conversation

@Sigma-Ma

@Sigma-Ma Sigma-Ma commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

JIRA: https://issues.apache.org/jira/browse/HBASE-30291

What changes were proposed in this pull request?

This pull request changes BackupHFileCleaner to load the referenced HFile names once in preClean() and reuse the resulting snapshot for all getDeletableFiles() calls in the same cleaner cycle.

The snapshot is published only after loading completes. If loading the references fails, the snapshot is marked unavailable and all files are kept for that cleaner cycle.

The existing previousCleaningCompletionTimestamp protection is preserved to prevent recently archived HFiles from being deleted before the next snapshot refresh.

Tests are added to verify that:

  • multiple getDeletableFiles() calls in one cleaner cycle load the references only once;
  • the snapshot is refreshed in the next cleaner cycle;
  • a failed refresh does not reuse a stale snapshot or make files deletable.

Why are the changes needed?

BackupHFileCleaner#getDeletableFiles() is invoked concurrently for each non-empty archive directory. It currently scans the backup system tables on every invocation to load the same set of backed-up tables and bulk-loaded HFile references.

On clusters with many regions, this causes tens of thousands of redundant system-table scans in a single cleaner cycle. The resulting I/O load can prevent the cleaner from keeping up, causing archived HFiles to accumulate.

Loading the references once per cleaner cycle removes these redundant scans without introducing a cross-cycle cache or changing the general CleanerChore framework.

How was this patch tested?

The following checks were run successfully:

mvn -pl hbase-backup -am \
  -Dtest=TestBackupHFileCleaner \
  -Dsurefire.failIfNoSpecifiedTests=false \
  -DskipITs \
  test

mvn -pl hbase-backup -DskipITs test

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant