Skip to content

[core] Add version-aware LocalKvDb range iterator#8871

Open
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/local-kv-range-iterator
Open

[core] Add version-aware LocalKvDb range iterator#8871
JingsongLi wants to merge 1 commit into
apache:masterfrom
JingsongLi:codex/local-kv-range-iterator

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Add an efficient, version-aware prefix/range iteration primitive to LocalKvDb for ListState and SetState.

Changes

  • Add lazy half-open range iteration over the memtable and overlapping SST files.
  • Merge all sources in key order with newest-version-wins semantics.
  • Suppress deleted keys through tombstones instead of exposing older values.
  • Select overlapping non-L0 SST files with binary search.
  • Hold a levels read snapshot while an iterator is open so concurrent compaction cannot delete its input files.
  • Expose collecting and callback-based range scan conveniences.

Verification

  • mvn -q -pl paimon-common -Pfast-build -DwildcardSuites=none '-Dtest=LocalKvDbTest#testRangeScanMergesMemTableAndLevels+testForEachInRangeAcrossMemTableAndSingleSst+testRangeIteratorDeduplicatesVersionsAcrossAllSources+testRangeIteratorCrossesSstBlocks,LocalKvDbAsyncCompactionTest#testAutomaticCompactionRunsInBackground+testRangeIteratorPreventsCompactionFromDeletingItsFiles' test
  • mvn -q -pl paimon-common spotless:check -DskipTests
  • mvn -q -pl paimon-common -DskipTests compile

@JingsongLi
JingsongLi marked this pull request as ready for review July 27, 2026 16:25

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is one blocking mutation path missing the range-iterator guard. I reproduced the bulk-load operation blocking until the iterator is closed.

}
}

private void ensureNoRangeIterator() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bulkLoad also mutates levels, but it does not call ensureNoRangeIterator(). rangeIterator() keeps the LsmLevels read lock until close, while bulkLoad() eventually calls levels.addFiles() and needs the write lock. With an iterator open on an empty DB, a concurrent bulk load blocks until close; invoking it on the iterator-owning thread deadlocks permanently because this lock cannot be upgraded. Please guard bulkLoad as well and add a regression test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants