research(nightly): temporal-decay-hnsw — recency-aware ANN for agent memory

[ruvnet]

Nightly Research: Temporal-Decay ANN for Agent Memory

Topic: Recency-aware nearest-neighbour search where effective distance is
modulated by vector insertion age: d_eff = d_raw × (1 + S × (1 − exp(−age/H)))

Slug: temporal-decay-hnsw
ADR: docs/adr/ADR-196-temporal-decay-hnsw.md
Research doc: docs/research/nightly/2026-06-03-temporal-decay-hnsw/README.md
Gist: docs/research/nightly/2026-06-03-temporal-decay-hnsw/gist.md

---

What This Adds

A new standalone Rust crate crates/ruvector-td-hnsw implementing three variants
of temporal-aware flat-index nearest-neighbour search:

Variant	Distance	Fresh Recall	QPS
Baseline	Raw L2²	0.095	553
TemporalDecay	L2² × weight(age)	1.000	540
CoherenceGated	Decay + age+distance gate	1.000	691

All numbers from cargo run --release -p ruvector-td-hnsw --bin td-benchmark.
Dataset: 10,000 vectors, 128 dims, 1,000 queries, 10% fresh distribution.

The Problem

Standard vector indexes rank by geometric distance only. With 90% stale memories,
90% of top-10 results are stale — proportional to the corpus distribution. Every
major vector database (Qdrant, Milvus, Weaviate, Pinecone) applies temporal decay
post-retrieval, after the ANN graph has already decided which candidates to
surface. This misses the gap: fresh but slightly farther vectors never enter the
candidate set.

The Solution

Modify the effective distance during candidate scoring:

let age_secs = now_secs - entry.timestamp_secs;
let weight = 1.0 + decay_strength * (1.0 - (-age_secs / half_life_secs).exp());
let d_eff = d_raw * weight;

Key Results

• Fresh recall: 0.095 → 1.000 (+953%) with TemporalDecay
• CoherenceGated: same recall, 20% faster than baseline (691 vs 553 QPS)
• TemporalDecay overhead: ~4% vs baseline
• All 9 unit tests pass, 1 doc-test passes
• Build: green (cargo build --release -p ruvector-td-hnsw)
• Acceptance: PASS

SOTA Gap

No existing vector database modifies the ANN distance function for temporal
preference. TANNS (ICDE 2025) is closest — hard validity gating — but does not
do soft recency weighting during traversal. This PoC fills that gap.

Ecosystem Fit

• ruvector-core: next step — integrate DecayConfig into HNSW greedy traversal
• ruFlo: auto-tune half_life_secs from session-level retrieval utility
• MCP: expose decay_config as a per-query parameter
• Cognitum Seed: DecayConfig::weight() is WASM-safe, no heap allocation
• RVF manifest: default_decay_config field for cognitive package portability

Files Changed

crates/ruvector-td-hnsw/
  Cargo.toml
  benches/td_bench.rs
  src/decay.rs          # DecayConfig, temporal_weight formula
  src/index.rs          # TdIndex, SearchResult, tests
  src/lib.rs            # public API
  src/main.rs           # benchmark binary (td-benchmark)
docs/adr/ADR-196-temporal-decay-hnsw.md
docs/research/nightly/2026-06-03-temporal-decay-hnsw/README.md
docs/research/nightly/2026-06-03-temporal-decay-hnsw/gist.md
Cargo.toml              # workspace member added

Research Loop

• Pass 1 (Discover): Scanned prior nightly branches; identified temporal decay as uncovered gap
• Pass 2 (Deepen): Located 13 peer-reviewed/arXiv sources; confirmed no system modifies ANN distance function for temporal preference
• Pass 3 (Critique): Validated flat-index approach is correct for PoC; HNSW integration is the clear next step

Benchmark Command

git checkout research/nightly/2026-06-03-temporal-decay-hnsw
cargo build --release -p ruvector-td-hnsw
cargo test -p ruvector-td-hnsw
cargo run --release -p ruvector-td-hnsw --bin td-benchmark

This branch should either become a production RuVector capability (starting with HNSW graph integration) or a falsified research path with useful evidence.

https://claude.ai/code/session_0153VnjXwTfTuLjmSg7WVgcc

---

Generated by Claude Code