Skip to content

eval: set USER_ERROR blame when AOAI run fails with UserError code#46746

Draft
Copilot wants to merge 1 commit intomainfrom
copilot/update-error-blame-logic
Draft

eval: set USER_ERROR blame when AOAI run fails with UserError code#46746
Copilot wants to merge 1 commit intomainfrom
copilot/update-error-blame-logic

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 6, 2026

When an AOAI evaluation run ends in a non-completed status, the exception was always raised with blame=UNKNOWN. If the run's error.code is "UserError", the blame should reflect that the failure is attributable to the user, not the system.

Changes

  • _evaluate_aoai.py_get_single_run_results: safely retrieve run_results.error.code via nested getattr and assign ErrorBlame.USER_ERROR when the code matches "usererror" (case-insensitive); falls back to UNKNOWN for any other value, missing attribute, or None.
error_code = getattr(getattr(run_results, "error", None), "code", None)
blame = ErrorBlame.USER_ERROR if isinstance(error_code, str) and error_code.lower() == "usererror" else ErrorBlame.UNKNOWN
  • test_evaluate_aoai.pyTestGetSingleRunResultsBlame: five new unit tests covering:
    • "UserError" / "usererror" / "USERERROR" / "uSeReRrOr"USER_ERROR (parametrized)
    • Non-matching code (e.g. "SystemError") → UNKNOWN
    • Missing error attribute → UNKNOWN
    • error present but no code attribute → UNKNOWN
    • error=NoneUNKNOWN

…s + unit tests

Agent-Logs-Url: https://github.com/Azure/azure-sdk-for-python/sessions/85522f11-e665-4eb7-b1bc-05e48272e704

Co-authored-by: posaninagendra <2713981+posaninagendra@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants