feat: add Apache Pinot vector search client#757
Open
xiangfu0 wants to merge 1 commit intozilliztech:mainfrom
Open
feat: add Apache Pinot vector search client#757xiangfu0 wants to merge 1 commit intozilliztech:mainfrom
xiangfu0 wants to merge 1 commit intozilliztech:mainfrom
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: xiangfu0 The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
xiangfu0
force-pushed
the
feat/pinot-support
branch
from
1d51fbf to
9c9d434
Compare
Author
|
/assign @XuanYang-cn |
xiangfu0
force-pushed
the
feat/pinot-support
branch
2 times, most recently
from
0f01a75 to
4296e07
Compare
Adds a complete Apache Pinot client for VectorDBBench. Index types: HNSW (Lucene), IVF_FLAT, IVF_PQ, IVF_ON_DISK Metrics: L2, IP, COSINE Filters: NumGE, StrEqual Optional dep: pip install "vectordb-bench[pinot]" Parallel loading: thread_safe=True — each worker thread maintains its own row buffer and flushes to Pinot via a fresh HTTP session. Since Pinot's ingestFromFile is synchronous (blocks until HNSW index is built, ~6 min per 100K×768D segment), concurrent flushes across threads reduce load time significantly vs sequential flushing. Benchmark results: Small dataset (OpenAI 50K, 768D, L2): HNSW: 798 QPS, recall=1.000 IVF_FLAT: 800 QPS, recall=1.000 IVF_PQ: 795 QPS, recall=1.000 IVF_ON_DISK: 691 QPS, recall=1.000 Large dataset (Cohere 1M, 768D, COSINE): HNSW m=16: 74 QPS, recall=0.982 Filter benchmark (Cohere 1M, COSINE, HNSW m=32): 1% NumGE: 71 QPS, recall=0.977 99% NumGE: 97 QPS, recall=0.649 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
xiangfu0
force-pushed
the
feat/pinot-support
branch
from
4296e07 to
5568d1f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a complete Apache Pinot client for VectorDBBench.
thread_safe=True— each worker thread maintains its own row buffer and flushes to Pinot via a fresh HTTP session. Since Pinot'singestFromFileis synchronous (blocks until the HNSW index is built, ~6 min per 100K×768D segment), running concurrent flushes across threads reduces load time significantly compared to sequential flushing.pip install "vectordb-bench[pinot]"Benchmark Results
Small dataset (OpenAI 50K, 768D, L2)
Large dataset (Cohere 1M, 768D, COSINE)
Filter benchmark (Cohere 1M, COSINE, HNSW m=32)
Low recall on 99% filter is expected: the HNSW graph is built on the full dataset, so filtering to 1% of vectors at query time causes many graph neighbors to be pruned.
Test plan
make lintpasses on all modified filesmake unittestpassesdocker compose(see Pinot quickstart)vectordbbench pinot-hnsw --db-label testagainst a local Pinot cluster🤖 Generated with Claude Code