fix: cap DocumentNDCGEvaluator score at 1.0 for duplicate retrievals

[otiscuilei]

What

DocumentNDCGEvaluator could return an NDCG score greater than 1.0, violating the 0.0 to 1.0 range promised in the class docstring and the individual_scores output description.

Why

calculate_dcg adds a relevance contribution for every retrieved document whose comparison value is present in the ground truth, without deduplicating. calculate_idcg, however, builds the ideal gain from the unique ground truth set. When the retrieved list contains the same relevant document more than once, its relevance is counted multiple times, so DCG exceeds IDCG and the resulting NDCG (dcg / idcg) climbs above 1.0.

Fix

Credit each ground truth comparison value at most once when computing DCG, by tracking the values already counted. A relevant document retrieved multiple times now contributes a single relevance term (the earliest, highest-discount position), keeping DCG bounded by IDCG and NDCG within [0.0, 1.0].

Before / after

from haystack import Document
from haystack.components.evaluators import DocumentNDCGEvaluator

evaluator = DocumentNDCGEvaluator()
result = evaluator.run(
    ground_truth_documents=[[Document(content="A", score=1.0)]],
    retrieved_documents=[[Document(content="A"), Document(content="A")]],
)
print(result["score"])
# Before: 1.6309297535714575   (score > 1.0, out of documented range)
# After:  1.0

Tests

Added test_run_with_duplicate_retrieved_document_does_not_exceed_one, which fails on the current code (score 1.63) and passes with the fix. The full test_document_ndcg.py suite (29 tests) passes.

[vercel]

@Otis0408 is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
haystack/components/evaluators
document_ndcg.py
Project Total

This report was generated by python-coverage-comment-action

[anakin87]

Hey @Otis0408, thank you for contributing.

I see you currently have 6 different PRs open in Haystack. I recommend pausing this work for a bit, as it may be difficult for the team to keep up with reviews.

Thank you again!

[otiscuilei]

Thanks for saying so directly, and sorry for the pile-up. That's on me, and I should have paced it better.

I've stopped opening new PRs here and won't add more until the queue clears.

If it would help the team, I'm happy to close the two most recent ones (#11958 and #11959) and re-open them later once you've had a chance to work through the rest. Just say the word and I'll do it. Equally happy to leave them and simply wait.

Thanks for the reviews so far.

[anakin87]

No need to close them. Thank you again!

[anakin87]

Thank you!

Looking into this topic, the IDCG computation also needs a fix for duplicates.

evaluator = DocumentNDCGEvaluator()
result = evaluator.run(
    ground_truth_documents=[[Document(content="A", score=1.0), Document(content="A", score=1.0)]],
    retrieved_documents=[[Document(content="A")]],
)
print(result["score"])  # 0.6131471927654584, expected 1.0

Could you take a look, fix and add a test?

[otiscuilei]

Good catch, thank you. You're right that calculate_idcg had the mirror of the same bug: it iterated every ground-truth document including duplicates, so a duplicated relevant document inflated the ideal ranking and kept NDCG below 1.0 even for a perfect retrieval. Meanwhile calculate_dcg now credits each comparison value at most once, so the two sides disagreed.

I updated calculate_idcg to collapse ground-truth documents by comparison value (keeping the highest relevance per value), mirroring calculate_dcg, so the ideal reflects the unique relevant set. Your example now returns 1.0. Added a regression test test_run_with_duplicate_ground_truth_document_can_still_reach_perfect_ndcg covering exactly that case; it fails on the previous code and passes now. Pushed.

[anakin87]

Thank you.

I notice that now the two functions treat duplicates differently: calculate_idcg keeps max, while calculate_dcg keeps last. I'd create a small helper function used by both that always uses max.

[otiscuilei]

Good point about the two functions resolving duplicates differently — fixed. Both calculate_dcg and calculate_idcg now build their relevance lookup through a shared _build_relevance_map helper that always keeps the highest score per comparison value, and I added a regression test covering duplicate ground truth documents with differing scores (it fails with the previous last-wins behavior in DCG). Sorry for the slow turnaround, and thanks for the careful review @anakin87.

[anakin87]

Thank you!

(I also pushed a small simplification)