Skip to content

Python: gen_ai.client.operation.duration is never recorded for failed chat or embedding calls #7818

Description

gen_ai.client.operation.duration is only recorded when a call succeeds. Every failed
chat call and every failed embedding call is missing from the metric.

The OpenTelemetry GenAI semantic conventions define this metric for failed operations too,
with error.type set to the class of the error. error.type is listed as conditionally
required, "if the operation ended in an error".

The framework already knows this. _capture_response in
python/packages/core/agent_framework/observability.py has a branch that copies
error.type from the attribute dictionary into the metric attributes:

if operation_duration_histogram and duration is not None:
    if OtelAttr.ERROR_TYPE in attributes:
        attrs[OtelAttr.ERROR_TYPE] = attributes[OtelAttr.ERROR_TYPE]
    operation_duration_histogram.record(duration, attributes=attrs)

No caller ever puts OtelAttr.ERROR_TYPE into that dictionary, and _capture_response is
only reached on the success path, so the branch is unreachable in production. The only
place that covers it is test_capture_response_with_error_type, which calls the helper
directly.

FunctionTool.invoke in _tools.py already does this correctly. It sets
attributes[OtelAttr.ERROR_TYPE] in its except block and records the duration histogram
in a finally block, so failed tool calls do reach
agent_framework.function.invocation.duration. The chat and embedding layers do not.

What this costs an operator: a dashboard built on gen_ai.client.operation.duration shows
only healthy traffic. A provider outage that makes every call time out after 30 seconds
looks like a drop in request volume, not a latency or error problem. There is no error rate
to alert on, because no error series exists.

Version

agent-framework-core 1.15.0, at commit d9d3fb6.

Note

The two tests I have written for this fail on main with Expected 'record' to have been called once. Called 0 times.

I am going to take this on.

Metadata

Metadata

Assignees

No one assigned

    Labels

    pythonUsage: [Issues, PRs], Target: PythontriageUsage: [Issues], Target: All issues that still need to be triaged

    Type

    No type

    Projects

    Status
    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions