[core] Support LocalKvDb merge and TTL compaction#8872
Conversation
leaves12138
left a comment
There was a problem hiding this comment.
There is a blocking stale-value resurrection case when a merge operator collapses multiple physical keys while older runs still contain the consumed keys.
| if (isTombstone(entry.getValue())) { | ||
| tombstoneCount++; | ||
| } | ||
| combiningWriter.put(entry.getKey(), entry.getValue()); |
There was a problem hiding this comment.
RecordCombiningWriter can collapse multiple keys into the first key without writing tombstones for the consumed keys. If an older run still contains one of those keys (for example from bulkLoad), reads fall through to the stale value after flush. I reproduced this with old a-1/a-2, then new a-1/a-2: get(a-2) returns new-2 before flush and old-2 after flush. Partial compaction has the same risk when older runs remain. Please ensure every consumed key shadows older versions, or restrict combining to cases where no older run can remain, and add a regression test.
Purpose
Add the generic compaction primitives needed to preserve RocksDB-style merge and non-strict TTL semantics in
LocalKvDb.Changes
State-specific value envelopes and merge operators are intentionally left to the dependent state-backend PR.
Verification
mvn -q -pl paimon-common -Pfast-build -DwildcardSuites=none '-Dtest=LocalKvDbTest#testPartialCompactionDoesNotResurrectValueFilteredByExpiration+testCompactionMergesBeforeFilteringExpiredValues+testCompactionRemovesTombstones+testManualCompaction' testmvn -q -pl paimon-common -Pfast-build -DwildcardSuites=none -Dtest=LocalKvDbTest testmvn -q -pl paimon-common spotless:check -DskipTestsmvn -q -pl paimon-common -DskipTests compile