Skip to content

Answer Relevancy metric returns 1.0 even if actual_output is empty #2385

@GeisaFaustino

Description

@GeisaFaustino

Describe the bug
Answer Relevancy metric returns 1.0 even if actual_output is empty

To Reproduce
Steps to reproduce the behavior:

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

answer_relevancy = AnswerRelevancyMetric(model=MODEL)

test_case = LLMTestCase(
    input="What if these shoes don't fit?",
    actual_output="" # an empty output
)

answer_relevancy.measure(test_case)
print(f"Score:{answer_relevancy.score}\nReason: {answer_relevancy.reason}")

Score:1
Reason: The score is 1.00 because the response is fully relevant and directly addresses the input without any irrelevant statements. Great job!

Expected behavior
For an empty output, the expected behavior would be a score of 0.0 or close to it, as the output provides no relevant information to address the input.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions