Skip to content

Commit 61b4e87

Browse files
committed
example(benchmark): add VIBE integration and performance evaluation suite
- Implement reproducible ANN benchmark pipeline comparing DEG configurations - Measure Pareto-optimal QPS vs Recall trade-offs across standardized datasets - Include plotting scripts and result serialization
1 parent cce966e commit 61b4e87

10 files changed

Lines changed: 2100 additions & 0 deletions

File tree

examples/README.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,5 +43,7 @@ uv sync --reinstall-package deglib
4343
- [`knng`](./knng/): k-Nearest Neighbor Graph (k-NNG) construction benchmark using EVP quantization and FP16 reranking (SISAP 2026 Challenge Task 1).
4444
- [`mips`](./mips/): Maximum Inner Product Search (MIPS) benchmark using $(d+1)$-dimensional $L_2$ transformation, FLAS pre-sorting, and SIMD FP16 inner products (SISAP 2026 Challenge Task 2).
4545
- [`static_data`](./static_data/): DEG paper search benchmark reproduction (Recall vs. QPS) on static datasets (`sift1m`, `deep1m`, `glove-100`, `audio`, `enron`).
46+
- [`vibe`](./vibe/): Vector Index Benchmark for Embeddings (VIBE) ANNS top-100 benchmark on modern embedding datasets (`agnews-mxbai`, `arxiv-nomic`, `landmark-dino`, `msmarco-qwen`, `gooaq-distilroberta`, `laion-clip`, `imagenet-align`, `imagenet-clip`, `yandex`, `yahoo-minilm`).
4647
- [`dynamic_data`](./dynamic_data/): DEG dynamic data streaming benchmark (`AddHalf`, `AddHalfRemoveAndAddOneAtATime`, `AddAllRemoveHalf`).
4748
- [`sliding_window`](./sliding_window/): DEG sliding window benchmark reproducing the dynamic continuous update experiment against DEG from the CleANN paper.
49+

examples/vibe/README.md

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
# VIBE ANNS Benchmark Reproduction with DEG
2+
3+
This example reproduces the Approximate Nearest Neighbor Search (ANNS) experiments from the **[VIBE (Vector Index Benchmark for Embeddings)](https://vector-index-bench.github.io/)** project using the Dynamic Exploration Graph (DEG).
4+
5+
VIBE provides realistic embedding datasets covering in-distribution and out-of-distribution (OOD) search across text, vision, and multi-modal models with exact pre-computed Top-100 ground truth neighbors.
6+
7+
## Supported VIBE Datasets
8+
9+
| Dataset Key | VIBE Name | Type | Size ($N$) | Dimension ($D$) | Metric |
10+
| :--- | :--- | :--- | :--- | :--- | :--- |
11+
| `agnews-mxbai` | AGNews-mxbai | In-Distribution | 120,000 | 1,024 | L2 (Euclidean) |
12+
| `arxiv-nomic` | ArXiv-nomic | In-Distribution | 2,000,000 | 768 | Inner Product |
13+
| `landmark-dino` | Landmark-dino | In-Distribution | 760,757 | 768 | Cosine |
14+
| `msmarco-qwen` | MSMARCO-qwen | In-Distribution | 8,841,823 | 1,024 | Inner Product |
15+
| `gooaq-distilroberta`| GooAQ-distilroberta | In-Distribution | 1,471,375 | 768 | Inner Product |
16+
| `laion-clip` | LAION-clip | Out-of-Distribution | 1,000,000 | 512 | Inner Product |
17+
| `imagenet-align` | ImageNet-align | Out-of-Distribution | 1,281,167 | 640 | Inner Product |
18+
| `imagenet-clip` | ImageNet-clip | In-Distribution | 1,281,167 | 512 | Inner Product |
19+
| `yandex` | Yandex-200 | Out-of-Distribution | 1,000,000 | 200 | Cosine |
20+
| `yahoo-minilm` | Yahoo-minilm | In-Distribution | 677,305 | 384 | Inner Product |
21+
22+
## Quick Start
23+
24+
### 1. Environment Setup
25+
26+
```bash
27+
cd examples/vibe
28+
uv sync
29+
```
30+
31+
### 2. Run Benchmark
32+
33+
Datasets are automatically downloaded on demand directly from the official [VIBE Hugging Face repository](https://huggingface.co/datasets/vector-index-bench/vibe) and stored in `D:/Data/VIBE` (or `~/.cache/vibe` / `VIBE_CACHE_DIR`).
34+
35+
```bash
36+
# Run on AGNews-mxbai (compact dataset ~120k vectors)
37+
uv run main.py --dataset agnews-mxbai
38+
39+
# Run on Yahoo-MiniLM (677k vectors)
40+
uv run main.py --dataset yahoo-minilm
41+
42+
# Run without GUI plot popups
43+
uv run main.py --dataset arxiv-nomic --no-show
44+
```
45+
46+
### 3. Command-Line Options
47+
48+
```bash
49+
uv run main.py --help
50+
```
51+
52+
- `--dataset`, `-d`: Dataset name to benchmark (e.g. `agnews-mxbai`, `arxiv-nomic`, `landmark-dino`, `msmarco-qwen`, `gooaq-distilroberta`, `laion-clip`, `imagenet-align`, `imagenet-clip`, `yandex`, `yahoo-minilm`).
53+
- `--cache-dir`, `-c`: Custom directory for dataset files and saved `.deg` graphs.
54+
- `--build-threads`, `-t`: Number of CPU threads for graph building (default: half of CPU cores, `threads // 2`).
55+
- `--k`: Graph degree $k$ (out-degree per vertex).
56+
- `--extend-k`: Exploration width during graph building.
57+
- `--eps`: Build $\varepsilon$ parameter.
58+
- `--anns-k`: Number of nearest neighbors to evaluate (default: 100).
59+
- `--rebuild-graph`: Force graph re-construction even if a cached graph file exists.
60+
- `--no-show`: Do not open interactive matplotlib plot window.

examples/vibe/config.yml

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
float:
2+
any:
3+
- base_args: ['@metric']
4+
constructor: DegANN
5+
disabled: false
6+
gpu: false
7+
ood: false
8+
singularity_image: deg
9+
module: vibe.algorithms.deg
10+
name: deg
11+
run_groups:
12+
base:
13+
args:
14+
k: [16, 24, 30, 40, 48]
15+
opt_target: ['LowLID', 'HighLID', 'StreamingData']
16+
extend_k: [60]
17+
build_eps: [0.1]
18+
improve_k: [0]
19+
improve_eps: [0.0]
20+
threads: [1]
21+
query_args:
22+
search_eps: [0.0, 0.01, 0.02, 0.04, 0.06, 0.08, 0.1, 0.12, 0.15, 0.2, 0.25, 0.3, 0.4, 0.5, 0.7, 1.0, 1.5, 2.0]
23+
24+
uint8:
25+
euclidean:
26+
- base_args: ['@metric']
27+
constructor: DegANN
28+
disabled: false
29+
gpu: false
30+
ood: false
31+
singularity_image: deg
32+
module: vibe.algorithms.deg
33+
name: deg
34+
run_groups:
35+
base:
36+
args:
37+
k: [16, 24, 30, 40, 48]
38+
opt_target: ['LowLID', 'HighLID', 'StreamingData']
39+
extend_k: [60]
40+
build_eps: [0.1]
41+
improve_k: [0]
42+
improve_eps: [0.0]
43+
threads: [1]
44+
query_args:
45+
search_eps: [0.0, 0.01, 0.02, 0.04, 0.06, 0.08, 0.1, 0.12, 0.15, 0.2, 0.25, 0.3, 0.4, 0.5, 0.7, 1.0, 1.5, 2.0]

0 commit comments

Comments
 (0)