Skip to content

[core] Support streaming LocalKvDb bulk load - #8870

Merged
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/local-kv-streaming-bulk-load
Jul 28, 2026
Merged

[core] Support streaming LocalKvDb bulk load#8870
JingsongLi merged 1 commit into
apache:masterfrom
JingsongLi:codex/local-kv-streaming-bulk-load

Conversation

@JingsongLi

Copy link
Copy Markdown
Contributor

Purpose

Support streaming initial bulk load for LocalKvDb without materializing all input entries.

Changes

  • Add a BulkLoadWriter API that writes sorted entries directly into bounded-size SST files.
  • Publish generated SST files atomically only after finish().
  • Require keys to be strictly increasing, so duplicate keys are rejected consistently with RocksDB SST ingestion.
  • Clean up all generated files when validation or writing fails.
  • Keep the existing iterator API as a checked wrapper around the streaming writer.

Verification

  • mvn -q -pl paimon-common -Pfast-build -DwildcardSuites=none '-Dtest=LocalKvDbTest#testBulkLoad+testBulkLoadMultipleSstFiles+testBulkLoadEmptyIterator+testStreamingBulkLoad+testStreamingBulkLoadRejectsDuplicateKeysAndCleansFiles+testBulkLoadValidatesEntryCount+testBulkLoadFailsOnUnsortedEntries+testBulkLoadFailsOnUnorderedSstRanges+testBulkLoadOrderCheckUsesConfiguredComparator' test
  • mvn -q -pl paimon-common spotless:check -DskipTests
  • mvn -q -pl paimon-common -DskipTests compile

@JingsongLi
JingsongLi marked this pull request as ready for review July 27, 2026 16:25

@leaves12138 leaves12138 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The streaming writer validates ordering/counts, publishes generated SSTs atomically, and aborts cleanly on failures or early close. LocalKvDbTest passes locally.

@JingsongLi
JingsongLi merged commit 59003b5 into apache:master Jul 28, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants