Skip to content

[WIP] ivfpq: Reuse list tables in batch search - #65

Open
shyjsarah wants to merge 3 commits into
apache:mainfrom
shyjsarah:codex/ivfpq-batch-table-reuse-20260728
Open

[WIP] ivfpq: Reuse list tables in batch search#65
shyjsarah wants to merge 3 commits into
apache:mainfrom
shyjsarah:codex/ivfpq-batch-table-reuse-20260728

Conversation

@shyjsarah

@shyjsarah shyjsarah commented Jul 28, 2026

Copy link
Copy Markdown

Summary

Reuse batch-local IVF-PQ list tables when many queries probe the same lists. The optimization applies to both filtered and unfiltered batch search and now exposes off, on, and auto runtime modes. The default is auto.

This PR is stacked on #64. Until #64 merges, GitHub also shows the filter-scanning commit.

Changes

  • Build the list-dependent part of residual L2 PQ tables once per active loaded list.
  • Build the query-dependent inner-product table once per active query in each loaded-list batch.
  • Combine the list and query components before scanning instead of rebuilding a full residual table for every query/list pair.
  • Apply the same optimization to filtered and unfiltered batch search.
  • Add IvfPqBatchTableReuseMode with:
    • off: always use the direct residual-table path.
    • on: force batch-local table reuse when the metric/layout prerequisites hold.
    • auto: require the existing batch-size and probe-reuse cost heuristic.
  • Default Rust, Java, JNI, and legacy FFI search parameters to auto.
  • Add Java VectorSearchParams.withIvfPqBatchTableReuse(...) overloads for enum and string values.
  • Keep ephemeral tables batch-local so they do not remain resident on the reader.

Testing

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets --locked -- -D warnings
  • cargo test --workspace --locked
    • core: 405 passed, 1 ignored
    • FFI: 4 passed
    • JNI: 3 passed
  • mvn -f java/pom.xml test
  • Verified filtered and unfiltered batch results against individual reader searches with floating-point tolerance.
  • Covered off, forced on, automatic activation, and automatic fallback cases.

Notes

  • Depends on [WIP] ivfpq: Optimize filtered batch scanning #64.
  • The precomputed and direct residual formulas are algebraically equivalent but can differ slightly due to floating-point accumulation order.
  • The Paimon-side vector_search option wiring is being kept in a separate local change and is not part of this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant