Skip to content

[core] Support LocalKvDb merge and TTL compaction#8872

Open
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/local-kv-merge-ttl-compaction
Open

[core] Support LocalKvDb merge and TTL compaction#8872
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/local-kv-merge-ttl-compaction

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Add the generic compaction primitives needed to preserve RocksDB-style merge and non-strict TTL semantics in LocalKvDb.

Changes

  • Add an optional merge operator for adjacent logical records written by flush and compaction.
  • Add an optional expiration predicate evaluated during compaction.
  • Convert expired values to tombstones during partial compaction so older values cannot be resurrected.
  • Drop expired values during full compaction.
  • Apply expiration after record combining, allowing a merged value to refresh its effective TTL.
  • Rewrite a single existing run during full compaction when TTL cleanup is configured.

State-specific value envelopes and merge operators are intentionally left to the dependent state-backend PR.

Verification

  • mvn -q -pl paimon-common -Pfast-build -DwildcardSuites=none '-Dtest=LocalKvDbTest#testPartialCompactionDoesNotResurrectValueFilteredByExpiration+testCompactionMergesBeforeFilteringExpiredValues+testCompactionRemovesTombstones+testManualCompaction' test
  • mvn -q -pl paimon-common -Pfast-build -DwildcardSuites=none -Dtest=LocalKvDbTest test
  • mvn -q -pl paimon-common spotless:check -DskipTests
  • mvn -q -pl paimon-common -DskipTests compile

@JingsongLi
JingsongLi marked this pull request as ready for review July 27, 2026 16:25

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a blocking stale-value resurrection case when a merge operator collapses multiple physical keys while older runs still contain the consumed keys.

if (isTombstone(entry.getValue())) {
tombstoneCount++;
}
combiningWriter.put(entry.getKey(), entry.getValue());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RecordCombiningWriter can collapse multiple keys into the first key without writing tombstones for the consumed keys. If an older run still contains one of those keys (for example from bulkLoad), reads fall through to the stale value after flush. I reproduced this with old a-1/a-2, then new a-1/a-2: get(a-2) returns new-2 before flush and old-2 after flush. Partial compaction has the same risk when older runs remain. Please ensure every consumed key shadows older versions, or restrict combining to cases where no older run can remain, and add a regression test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants