-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Open
Labels
Description
Describe the bug
Answer Relevancy metric returns 1.0 even if actual_output is empty
To Reproduce
Steps to reproduce the behavior:
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
answer_relevancy = AnswerRelevancyMetric(model=MODEL)
test_case = LLMTestCase(
input="What if these shoes don't fit?",
actual_output="" # an empty output
)
answer_relevancy.measure(test_case)
print(f"Score:{answer_relevancy.score}\nReason: {answer_relevancy.reason}")
Score:1
Reason: The score is 1.00 because the response is fully relevant and directly addresses the input without any irrelevant statements. Great job!
Expected behavior
For an empty output, the expected behavior would be a score of 0.0 or close to it, as the output provides no relevant information to address the input.