Skip to content

fix(datafusion): apply dynamic options to vector search - #623

Open
shyjsarah wants to merge 2 commits into
apache:mainfrom
shyjsarah:fix/vector-search-dynamic-options
Open

fix(datafusion): apply dynamic options to vector search#623
shyjsarah wants to merge 2 commits into
apache:mainfrom
shyjsarah:fix/vector-search-dynamic-options

Conversation

@shyjsarah

@shyjsarah shyjsarah commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #622

SQLContext applies session-scoped Paimon dynamic options to ordinary table scans, but the automatically registered vector_search function loaded its table directly from the raw catalog. This caused supported read options such as blob-as-descriptor to be ignored when vector-search results were materialized.

Brief change log

  • pass the shared DynamicOptions instance when SQLContext registers vector_search
  • apply the current supported dynamic options to the loaded vector-search table
  • exclude time-travel selectors because vector search currently resolves ANN candidates from the latest snapshot; forwarding them could search one snapshot and materialize another
  • preserve the existing standalone register_vector_search and VectorSearchFunction::new APIs by using empty options there
  • add a regression test covering both blob-as-descriptor propagation and time-travel-option exclusion

Tests

  • cargo fmt --all -- --check
  • cargo check -p paimon-datafusion --all-features
  • cargo clippy -p paimon-datafusion --all-features --lib --tests --no-deps -- -D warnings
  • cargo test -p paimon-datafusion test_vector_search_applies_supported_session_dynamic_options --lib
  • cargo test -p paimon-datafusion --test read_tables vector_search_tests::test_vector_search_java_vindex_table

The full paimon-datafusion --lib run completed 311 tests successfully; 7 existing warehouse-fixture tests failed because the local PAIMON_TEST_WAREHOUSE did not contain partitioned_log_table / multi_partitioned_log_table.

API and Format

No public API or storage-format changes.

Documentation

No documentation changes are required; this makes vector_search honor supported session read options without implying time-travel support.

Comment thread crates/integrations/datafusion/src/sql_context.rs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] vector_search ignores session dynamic options

2 participants