Add DiskANN benchmark pipeline to GitHub Actions#857

Open

YuanyuanTian-hh wants to merge 31 commits intomainfrom

user/tianyuanyuan/add-benchmark-pipeline

Commits on Mar 19, 2026

Migrate DiskANN benchmark pipeline from ADO to GitHub Actions
Yuanyuan Tian (from Dev Box)
committed

Commits on Mar 20, 2026

Fix openai-100K distance metric and add gitignore patterns
Yuanyuan Tian (from Dev Box)
committed
Add push trigger to benchmarks workflow for pre-merge testing
Yuanyuan Tian (from Dev Box)
committed
Fix baseline run: use input config from current branch checkout
Yuanyuan Tian (from Dev Box)
committed
Fix markdown conversion: replace broken csvtomd with inline Python
Yuanyuan Tian (from Dev Box)
committed
Update benchmark configs: align build/search params to fix low recall
Yuanyuan Tian (from Dev Box)
committed

Commits on Mar 23, 2026

Fix recall_at: set to 100 to match groundtruth file K=100
Yuanyuan Tian (from Dev Box)
committed
Remove stale absolute contracts (qps/recall/total_time): calibrated for ADO mimir-enron, not applicable to public datasets on GitHub runners. Threshold calibration tracked in PBI.
Yuanyuan Tian (from Dev Box)
committed
remove comments
Yuanyuan Tian (from Dev Box)
committed
Fix build_ram_limit_gb: reduce 10->4 to fit GitHub runner RAM (7GB limit)
Yuanyuan Tian (from Dev Box)
committed
Fix wikipedia distance: cosine_normalized->cosine (vectors are not L2-normalized, metric is inner product)
Yuanyuan Tian (from Dev Box)
committed
Fix wikipedia distance: cosine->inner_product (groundtruth uses raw ip, not cosine similarity)
Yuanyuan Tian (from Dev Box)
committed

Commits on Mar 24, 2026

Align build threads
Yuanyuan Tian (from Dev Box)
committed
Speed up benchmarks: build threads 1->4, openai pq_chunks 384->192
Yuanyuan Tian (from Dev Box)
committed
Temp: disable concurrency cancellation for A/A batch testing
Yuanyuan Tian (from Dev Box)
committed

Commits on Mar 26, 2026

revert A/A test settings, update OpenAI config to SQ_1_2.0
Yuanyuan Tian (from Dev Box)
committed

Commits on Mar 27, 2026

Remove micro-benchmark-iai comments
Yuanyuan Tian (from Dev Box)
committed
use GitHub Release assets for benchmark datasets
Yuanyuan Tian (from Dev Box)
committed
extract csv-to-markdown into reusable script
Yuanyuan Tian (from Dev Box)
committed
calibrate contract thresholds from GitHub runner data
Yuanyuan Tian (from Dev Box)
committed
add daily A/A benchmark stability test with failure notification
Yuanyuan Tian (from Dev Box)
committed
widen mean_cpus threshold to 15% for shared-runner CPU noise
Yuanyuan Tian (from Dev Box)
committed

Commits on Mar 30, 2026

move benchmark datasets to separate repo (YuanyuanTian-hh/diskann-benchmark-data)
Yuanyuan Tian (from Dev Box)
committed
consolidate 3 benchmark scripts into single benchmark_validate.py
Yuanyuan Tian (from Dev Box)
committed

Commits on Mar 31, 2026

switch benchmark jobs to self-hosted 1ES runner pool (diskann-github)
Yuanyuan Tian (from Dev Box)
committed
replace gh CLI with curl for dataset downloads (gh not available on 1ES runners)
Yuanyuan Tian (from Dev Box)
committed
fix latency_95: read p95_latency instead of p999_latency from benchmark JSON
Yuanyuan Tian (from Dev Box)
committed

Commits on Apr 1, 2026

revert to ubuntu-latest runners, switch dataset source to BAB v0.4.0
Yuanyuan Tian (from Dev Box)
committed

Commits on Apr 2, 2026

address PR review: reduce search_list to 200, remove hardcoded Rust version, fix missing-field handling, clean up orphaned thresholds, switch data source to BAB v0.4.0
Yuanyuan Tian (from Dev Box)
committed
widen latency_95 threshold to 15% for shared-runner noise
Yuanyuan Tian (from Dev Box)
committed
replace push trigger with pull_request trigger targeting main
Yuanyuan Tian (from Dev Box)
committed