fix: validate LanceDB vector dimensions before batch writes#2414
Closed
yinli-systems wants to merge 1 commit into
Closed
fix: validate LanceDB vector dimensions before batch writes#2414yinli-systems wants to merge 1 commit into
yinli-systems wants to merge 1 commit into
Conversation
Contributor
|
Fixed in PR #2442 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #2265 by validating each LanceDB document vector against the configured
vector_sizebefore constructing the PyArrow fixed-size-list column.When the embedding model returns vectors with a different dimension than the configured LanceDB index, the current code reaches
FixedSizeListArray.from_arrays(...)and fails later with a confusing row-count-style Arrow error such asColumn 1 named vector expected length 44 but got length 11. This change fails earlier with the exact document id, actual vector dimension, index name, and configured vector size.Tests
UV_CACHE_DIR=/private/tmp/uv-cache-graphrag uv run pytest -q tests/integration/vector_stores/test_lancedb.py -k "load_documents"UV_CACHE_DIR=/private/tmp/uv-cache-graphrag uv run pytest -q tests/integration/vector_stores/test_lancedb.pyUV_CACHE_DIR=/private/tmp/uv-cache-graphrag uv run ruff format --check packages/graphrag-vectors/graphrag_vectors/lancedb.py tests/integration/vector_stores/test_lancedb.pyUV_CACHE_DIR=/private/tmp/uv-cache-graphrag uv run ruff check packages/graphrag-vectors/graphrag_vectors/lancedb.py tests/integration/vector_stores/test_lancedb.py