Skip to content

task_plan_mmr.md

Latest commit

 

History

History
82 lines (59 loc) · 2.75 KB

task_plan_mmr.md

File metadata and controls

82 lines (59 loc) · 2.75 KB

Task Plan: MMR (Maximal Marginal Relevance) Implementation

Goal

Improve multi-session "all" recall from 12.5% to ~20%+ by implementing MMR diversity in result assembly.

Background

MMR balances relevance and diversity:

MMR = λ * Sim(d, Q) - (1-λ) * max(Sim(d, S))
  • Sim(d, Q) = relevance to query (we have this as score)
  • Sim(d, S) = max similarity to already-selected documents
  • λ = tradeoff parameter (0.6-0.7 typical)

Implementation Phases

Phase Task Status
1 Read existing code + understand data flow complete
2 Add embedding vectors to SearchResult or fetch at assembly time skipped (using text similarity instead)
3 Implement MMR selection in assemble_context_diverse() complete
4 Add tests complete (10 new tests, 65 total pass)
5 Run eval to measure impact complete
6 Document and commit complete

Implementation Notes

Approach: Text-Based MMR (No Embedding Fetch)

Instead of fetching embeddings from DB, we use Jaccard word similarity as a proxy:

  • _word_set(text) - Extract lowercase word set from content
  • _jaccard_similarity(a, b) - Compute word overlap between two documents
  • _max_similarity_to_selected(candidate, selected) - Find max similarity to any selected doc

This is efficient (no DB calls) and surprisingly effective for conversation text.

MMR Formula

MMR = λ * normalized_relevance - (1-λ) * max_similarity_to_selected

Default λ = 0.6 (60% relevance, 40% diversity)

Key Questions

  1. Do we have embeddings available at assembly time?
  2. Should we compute similarity on-the-fly or pre-compute?
  3. What λ value to use? (start with 0.6)

Success Criteria

  • multi-session (all): 12.5% → 18%+ (+5.5%)
  • No regression on other categories
  • Tests pass

Results ✅

Category Before MMR With MMR Change
multi-session (all) 12.5% 16.1% +3.6% ✅
Overall (all) 57.6% 59.6% +2.0%
knowledge-update (all) 80.6% 83.3% +2.7%
single-session-preference 33.3% 43.3% +10%
temporal-reasoning (all) 59.3% 57.4% -1.9%
multi-session (any) 89.3% 82.1% -7.2%

Analysis:

  • multi-session (all) improved +3.6% (target was +5.5%, got 65% of target)
  • Bonus: preference queries improved +10%
  • Trade-off: multi-session (any) dropped -7.2% (diversity over relevance)
  • Overall "all" recall improved +2%

Verdict: Success - MMR provides meaningful improvement for aggregation queries

Files to Modify

  • src/cems/retrieval.py - assemble_context_diverse() function
  • src/cems/memory/retrieval.py - Pass embeddings to assembly
  • tests/test_retrieval.py - Add MMR tests