gen_ai.client.operation.duration is only recorded when a call succeeds. Every failed
chat call and every failed embedding call is missing from the metric.
The OpenTelemetry GenAI semantic conventions define this metric for failed operations too,
with error.type set to the class of the error. error.type is listed as conditionally
required, "if the operation ended in an error".
The framework already knows this. _capture_response in
python/packages/core/agent_framework/observability.py has a branch that copies
error.type from the attribute dictionary into the metric attributes:
if operation_duration_histogram and duration is not None:
if OtelAttr.ERROR_TYPE in attributes:
attrs[OtelAttr.ERROR_TYPE] = attributes[OtelAttr.ERROR_TYPE]
operation_duration_histogram.record(duration, attributes=attrs)
No caller ever puts OtelAttr.ERROR_TYPE into that dictionary, and _capture_response is
only reached on the success path, so the branch is unreachable in production. The only
place that covers it is test_capture_response_with_error_type, which calls the helper
directly.
FunctionTool.invoke in _tools.py already does this correctly. It sets
attributes[OtelAttr.ERROR_TYPE] in its except block and records the duration histogram
in a finally block, so failed tool calls do reach
agent_framework.function.invocation.duration. The chat and embedding layers do not.
What this costs an operator: a dashboard built on gen_ai.client.operation.duration shows
only healthy traffic. A provider outage that makes every call time out after 30 seconds
looks like a drop in request volume, not a latency or error problem. There is no error rate
to alert on, because no error series exists.
Version
agent-framework-core 1.15.0, at commit d9d3fb6.
Note
The two tests I have written for this fail on main with Expected 'record' to have been called once. Called 0 times.
I am going to take this on.
gen_ai.client.operation.durationis only recorded when a call succeeds. Every failedchat call and every failed embedding call is missing from the metric.
The OpenTelemetry GenAI semantic conventions define this metric for failed operations too,
with
error.typeset to the class of the error.error.typeis listed as conditionallyrequired, "if the operation ended in an error".
The framework already knows this.
_capture_responseinpython/packages/core/agent_framework/observability.pyhas a branch that copieserror.typefrom the attribute dictionary into the metric attributes:No caller ever puts
OtelAttr.ERROR_TYPEinto that dictionary, and_capture_responseisonly reached on the success path, so the branch is unreachable in production. The only
place that covers it is
test_capture_response_with_error_type, which calls the helperdirectly.
FunctionTool.invokein_tools.pyalready does this correctly. It setsattributes[OtelAttr.ERROR_TYPE]in itsexceptblock and records the duration histogramin a
finallyblock, so failed tool calls do reachagent_framework.function.invocation.duration. The chat and embedding layers do not.What this costs an operator: a dashboard built on
gen_ai.client.operation.durationshowsonly healthy traffic. A provider outage that makes every call time out after 30 seconds
looks like a drop in request volume, not a latency or error problem. There is no error rate
to alert on, because no error series exists.
Version
agent-framework-core1.15.0, at commitd9d3fb6.Note
The two tests I have written for this fail on
mainwithExpected 'record' to have been called once. Called 0 times.I am going to take this on.