Skip to content

[WIP] ivfpq: Optimize filtered batch scanning - #64

Open
shyjsarah wants to merge 1 commit into
apache:mainfrom
shyjsarah:codex/ivfpq-filter-pushdown-20260727
Open

[WIP] ivfpq: Optimize filtered batch scanning#64
shyjsarah wants to merge 1 commit into
apache:mainfrom
shyjsarah:codex/ivfpq-filter-pushdown-20260727

Conversation

@shyjsarah

@shyjsarah shyjsarah commented Jul 28, 2026

Copy link
Copy Markdown

Summary

Optimize filtered IVF-PQ scanning by evaluating row-id filters once per relevant inverted list and sharing the resulting positions across all queries in a batch.

This is the filter-focused part split from the original combined PR. Batch-local IVF-PQ table reuse is moved to a follow-up WIP PR.

Changes

  • Materialize matching row positions once per probed or loaded inverted list instead of evaluating the filter for every query.
  • Evaluate in-memory filters only for lists actually probed by the current query batch.
  • Use sparse position scans for selective filters while retaining dense SIMD/full-scan paths for less selective filters.
  • Pass matching positions explicitly through 4-bit, transposed, batched, resident-reader, and streamed-reader scan paths.
  • Preserve exact residual filtering and existing public APIs.

Testing

  • cargo fmt --all -- --check
  • cargo clippy -p paimon-vindex-core --all-targets -- -D warnings
  • cargo test -p paimon-vindex-core --locked
  • Added coverage that batch readers evaluate each loaded row once.
  • Added coverage that in-memory batch search does not evaluate filters for unprobed lists.

Notes

@shyjsarah
shyjsarah force-pushed the codex/ivfpq-filter-pushdown-20260727 branch from dd1f7c5 to 353a1f1 Compare July 28, 2026 13:10
@shyjsarah shyjsarah changed the title [WIP] ivfpq: Optimize filtered batch search [WIP] ivfpq: Optimize filtered batch scanning Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant