research(nightly): LSM-Segmented Vector Index — epoch-based three-tier HNSW for streaming inserts#541
Draft
ruvnet wants to merge 4 commits into
Draft
research(nightly): LSM-Segmented Vector Index — epoch-based three-tier HNSW for streaming inserts#541ruvnet wants to merge 4 commits into
ruvnet wants to merge 4 commits into
Conversation
…r index Introduces a new standalone crate implementing a synchronous, three-tier LSM-style vector index for streaming agent-memory workloads: - FlatSegment (hot): O(1) insert, O(n) linear scan - NswSegment (warm/cold): single-layer NSW proximity graph, O(log n) search - LsmVectorIndex: fan-out search across all tiers with inline compaction No background threads, no OS dependencies — suitable for no_std / WASM. Benchmark results at N=10K, dim=128 (release): Flat recall=1.000, NSW recall=0.575, LSM-NSW recall=0.627 Hot insert p50=0.0001ms (pure hot path) Implements ADR-196 Phase 0 PoC. https://claude.ai/code/session_01PUuV29jg91p2yBpFFC5wSe
Architecture Decision Record for the epoch-based three-tier HNSW streaming insert design. Covers: - Context (streaming agent-memory workload gaps) - Decision (three-tier LSM with synchronous compaction) - Consequences (positive: additive recall, O(1) insert; negative: 6.5x build cost) - Alternatives considered (IP-DiskANN, IVF, UBISS, full HNSW) - Implementation plan (Phase 0 PoC, Phase 1 hardening, Phase 2 integration) - Real benchmark evidence from cargo run --release - Failure modes and security considerations Status: Proposed. https://claude.ai/code/session_01PUuV29jg91p2yBpFFC5wSe
Comprehensive 24-section research document covering: - SOTA survey (LSM-VEC, UBISS, IP-DiskANN, Ada-IVF, SPFresh) - Three-variant benchmark with real measured results - Architecture design and tier lifecycle - Edge/WASM/no_std deployment implications - RVF temperature-tiering integration path - Applications (8 practical + 8 exotic) - Mermaid architecture diagram - Memory and performance mathematics - 10 footnoted references All benchmark numbers sourced from cargo run --release --bin benchmark. https://claude.ai/code/session_01PUuV29jg91p2yBpFFC5wSe
Public-facing Markdown gist summarising the LSM-NSW streaming vector index for discoverability. Covers: - The streaming insert problem vs batch ANN indexes - Three-tier design and synchronous compaction - Real benchmark table (Flat / NSW / LSM-NSW) - Why multi-tier fan-out improves recall (ensemble intuition) - WASM / no_std compatibility rationale - Honest tradeoff list (missing: deletes, thread safety, HNSW layers) - SOTA comparison table (6 systems) - Minimal usage code snippet https://claude.ai/code/session_01PUuV29jg91p2yBpFFC5wSe
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Nightly research branch for 2026-06-05. Implements and benchmarks a three-tier LSM-style vector index in Rust optimised for streaming agent-memory inserts on edge, WASM, and
no_stdtargets.crates/ruvector-lsm-index— new standalone crate:FlatSegment(hot),NswSegment(warm/cold),LsmVectorIndexwith synchronous compactiondocs/adr/ADR-196-lsm-vector-index.md— Architecture Decision Record (status: Proposed)docs/research/nightly/2026-06-05-lsm-vector-index/README.md— 24-section research document with SOTA survey and real benchmark resultsdocs/research/nightly/2026-06-05-lsm-vector-index/gist.md— SEO-optimised public summaryBenchmark Results (N=10,000 × 128d, release build, real measured)
Hot insert p50 = 0.0001 ms (pure flat append, no graph work).
LSM-NSW achieves higher recall than single NSW (0.627 vs 0.575) because fan-out across three independently-built graphs covers uncorrelated candidate pools — same intuition as ensemble methods.
Key Design Properties
no_std/ WASMwarm_capacity(not total n)SOTA Differentiation
Existing streaming ANN systems (LSM-VEC arXiv:2505.17152, UBISS arXiv:2602.00563, IP-DiskANN arXiv:2502.13826) all target billion-scale servers with background threads. None support
no_std, WASM, or edge appliance deployment. This PoC occupies an unoccupied niche.What's Deferred (Phase 1)
Arc<RwLock<>>concurrent read pathno_stdflat WASM target (flat stride indexing)Test Plan
cargo test -p ruvector-lsm-index— 10 unit tests pass (flat, nsw, lsm tiers)cargo run --release --bin benchmark -p ruvector-lsm-index— OVERALL: PASS ✓cargo build --workspace— clean release buildcargo fmt --check— formattedcargo run --releaseoutputhttps://claude.ai/code/session_01PUuV29jg91p2yBpFFC5wSe
Generated by Claude Code