research: nightly 2026-07-16 — partition-aware diverse ANN retrieval (ADR-272)

[ruvnet]

Summary

• Adds crates/ruvector-diverse-retrieval (v0.1.0): a new standalone Rust crate implementing three ANN retrieval variants — TopKRetriever, MmrRetriever, and PartitionMmrRetriever.
• PartitionMmrRetriever extends standard MMR with a graph-cut-inspired partition layer: builds an ephemeral connectivity graph over the candidate pool, extracts connected components via Union-Find, and applies a per-selection penalty when a candidate shares its component with an already-chosen result.
• Delivers 3.25× more diverse results than TopK and 47% more diverse than plain MMR on the two-level hierarchical benchmark dataset; all 4 acceptance tests and 26 unit tests pass.

Key Files

File	Purpose
crates/ruvector-diverse-retrieval/src/partition_mmr.rs	Core PartitionMMR algorithm
crates/ruvector-diverse-retrieval/src/graph.rs	Union-Find + adaptive threshold estimation
crates/ruvector-diverse-retrieval/src/dataset.rs	Two-level hierarchical synthetic dataset
crates/ruvector-diverse-retrieval/src/main.rs	Benchmark binary
docs/adr/ADR-272-mincut-diverse-retrieval.md	Architecture decision record
docs/research/nightly/2026-07-16-mincut-diverse-retrieval/README.md	Full research document
docs/research/nightly/2026-07-16-mincut-diverse-retrieval/gist.md	SEO gist

Benchmark Results (cargo run --release, x86_64 Linux)

Dataset: 10 super-clusters × 6 sub-clusters × 50 vectors = 3,000 total, 64 dims.

Variant              Mean µs    p50 µs    p95 µs      QPS   MeanDiv   MeanRel
TopK (baseline)        271.3     256.2     359.1     3686    2.5742    2.2341
MMR (λ=0.5)            443.8     432.8     545.5     2253    5.6962    4.0169
PartitionMMR           725.8     713.5     855.8     1378    8.3766    6.2068

[PASS] PartitionMMR diversity ≥ TopK×1.15: ratio 3.254
[PASS] PartitionMMR diversity > MMR diversity: 8.377 > 5.696
[PASS] MMR diversity > TopK diversity: 5.696 > 2.574
[PASS] PartitionMMR rel ≤ MMR×2.0: ratio 1.545

Algorithm Design Note

The threshold for the connectivity graph must be estimated from the full candidate pool, not a nearest-N subset. Using only the nearest N candidates yields a sample dominated by intra-sub-cluster distances, producing a threshold too small to connect within-sub pairs → all singletons → partition penalty never fires. The full pool's bimodal distribution (intra-sub ≈ 2.83, inter-sub ≈ 13.57) gives a mean (~11.94) between the two modes, and 0.55 × mean ≈ 6.57 correctly connects within-sub pairs while leaving between-sub pairs separate.

Ecosystem Connections

Connects three RuVector themes:

1. Vector search — three retrieval variants with a common DiverseRetriever trait
2. Graph primitives — Union-Find from graph.rs mirrors the primitives in ruvector-mincut
3. Agent memory / ruFlo — diversity-aware recall is a direct improvement for ruFlo memory steps; proposed MCP tool vector/search_diverse with diversity_mode param

Test Plan

• cargo test -p ruvector-diverse-retrieval — 26 tests pass
• cargo run --release -p ruvector-diverse-retrieval — all 4 acceptance tests pass
• cargo fmt -p ruvector-diverse-retrieval — applied
• Integration with ruvector-coherence-hnsw (future PR)
• WASM wrapper ruvector-diverse-retrieval-wasm (future PR)
• MCP tool vector/search_diverse (future PR)

---

Generated by Claude Code