feat(adk): expose gen_ai.client.token.usage Prometheus metric by braghettos · Pull Request #2149 · kagent-dev/kagent

braghettos · 2026-07-04T13:45:00Z

What

Adds GenAI token-usage instrumentation to the ADK agent, exposing the OTel-semconv metric gen_ai.client.token.usage via the native Prometheus client library (github.com/prometheus/client_golang) — matching kagent's existing Go metrics approach on the controller side. First of the three observability gaps in #2148.

Updated per review (#2149 review): switched from the initial OTLP MeterProvider to the Prometheus client library, keeping OTel-semconv naming.

How

pkg/telemetry/metrics.go — a gen_ai_client_token_usage histogram (Prometheus name for the semconv metric gen_ai.client.token.usage) with labels, following GenAI semantic conventions (semconv 1.40.0), mapped to Prometheus naming (dots → underscores):
- gen_ai_token_type — input / output
- gen_ai_request_model — e.g. gpt-4o
- gen_ai_provider_name — e.g. openai, anthropic
- gen_ai_operation_name — chat (the chat-completion operation that produces these tokens)
pkg/a2a/server/server.go — serves the metrics at /metrics on the agent's A2A mux (excluded from request tracing).
pkg/a2a/executor.go — records token counts from the A2A event loop where UsageMetadata is surfaced: input = PromptTokenCount, output = CandidatesTokenCount + ThoughtsTokenCount.
cmd/main.go / pkg/config — resolve the request model + provider labels from the agent config.

Semconv alignment

Addresses the naming points raised on #1412: uses gen_ai.provider.name (not gen_ai.system), includes gen_ai.request.model, and sets gen_ai.operation.name (chat) rather than omitting it. Naming targets semconv 1.40.0.

Testing

pkg/telemetry/metrics_test.go — asserts the histogram records one series per token type, skips zero counts, and that the /metrics handler serves the series with the expected labels (via prometheus/client_golang/testutil + httptest).
go build ./adk/..., go test -race -skip 'TestE2E.*' ./adk/..., and golangci-lint run all pass.

Notes

prometheus/client_golang is already a dependency; no new direct requirements (only a transitive godebug from the test util).
The two remaining gaps from [FEATURE] Close three observability gaps: GenAI token metrics, controller reconcile Events, and a controller log→OTLP bridge (all gated default-OFF) #2148 (reconcile Events feat(core): emit Kubernetes Events on Agent/ModelConfig/MCPServerTool/RemoteMCPServer reconciles #2150, controller log→OTLP bridge feat(core): add controller log→OTLP bridge (gated OTEL_LOGGING_ENABLED) #2151) are unaffected.

Refs: #2148

Copilot

Pull request overview

Adds an OTLP-backed OpenTelemetry metrics pipeline to the ADK and emits the GenAI semantic-convention metric gen_ai.client.token.usage (gated behind OTEL_METRICS_ENABLED) using token usage already surfaced by the A2A executor loop.

Changes:

Introduces an OTLP MeterProvider + periodic reader/exporter and a package-level RecordTokenUsage(...) helper for gen_ai.client.token.usage.
Wires metrics initialization/shutdown into ADK telemetry init (alongside existing tracing/logging setup).
Plumbs model/provider labels from agent config into the executor so token usage points include gen_ai.request.model and gen_ai.provider.name.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
go/go.mod	Promotes OTLP metric exporter + otel metric SDK deps to direct requirements.
go/adk/pkg/telemetry/tracing.go	Initializes MeterProvider when metrics are enabled and composes shutdown.
go/adk/pkg/telemetry/metrics.go	Implements OTLP metrics pipeline and GenAI token-usage histogram + recorder.
go/adk/pkg/telemetry/metrics_test.go	Adds unit tests for recording/no-op behavior.
go/adk/pkg/config/config_usage.go	Exposes model-name helper for telemetry labeling.
go/adk/pkg/a2a/executor.go	Records token usage from `UsageMetadata` during the A2A event loop.
go/adk/cmd/main.go	Derives model/provider labels from agent config and passes them to the executor.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+		meterProvider, mpErr := newMeterProvider(ctx, telemetryResource)
+		if mpErr != nil {
+			return nil, true, mpErr
+		}


+		if instErr := initGenAIMetrics(); instErr != nil {
+			return nil, true, instErr
+		}


+// genAIMetrics holds the GenAI metric instruments. It is recorded by the A2A
+// executor when the agent runtime reports token usage. When metrics are
+// disabled (the default), instrumentInstance is nil and the public Record*
+// helpers are cheap no-ops, keeping the runtime byte-identical when off.


krisztianfekete

Thanks for opening this PR!

Can you please follow the recommendations in this issue (you're umbrella issue references this actually...) comment: #1412 (review), and use the official Prometheus client library for instrumentation?

Add GenAI token-usage instrumentation to the ADK agent using the native Prometheus client library (github.com/prometheus/client_golang), matching kagent's existing Go metrics approach on the controller side. - pkg/telemetry/metrics.go: a gen_ai_client_token_usage histogram (Prometheus name for the OTel semconv metric gen_ai.client.token.usage) with labels gen_ai_token_type (input/output), gen_ai_request_model, gen_ai_provider_name and gen_ai_operation_name, following GenAI semantic conventions (semconv 1.40.0). Exposed via MetricsHandler. - pkg/a2a/server/server.go: serve the metrics at /metrics on the agent's A2A mux (excluded from request tracing). - pkg/a2a/executor.go: record token counts from the A2A event loop where UsageMetadata is surfaced (input = PromptTokenCount, output = CandidatesTokenCount + ThoughtsTokenCount). - cmd/main.go / pkg/config: resolve the request model and provider labels from the agent config. Replaces the earlier OTLP MeterProvider approach per maintainer review, keeping OTel-semconv naming. prometheus/client_golang is already a dependency; no new direct requirements. Refs: kagent-dev#2148 Signed-off-by: Diego Braga <diego.braga86@gmail.com>

braghettos · 2026-07-06T19:15:15Z

Thanks for the review, @krisztianfekete! I've reworked the PR to use the native Prometheus client library (prometheus/client_golang) for instrumentation instead of the OTLP MeterProvider, matching the approach kagent already uses on the controller side.

Kept it OTel-semconv compliant (targeting semconv 1.40.0). Specifically, addressing the points from the #1412 review:

gen_ai.provider.name (not gen_ai.system)
gen_ai.request.model is included
gen_ai.operation.name is set to chat (the chat-completion operation these token counts come from), rather than being omitted

The metric is a gen_ai_client_token_usage histogram (dots→underscores for Prometheus) exposed at /metrics on the agent's A2A server, with labels gen_ai_token_type / gen_ai_request_model / gen_ai_provider_name / gen_ai_operation_name. prometheus/client_golang was already a dependency, so there are no new direct requirements. The Copilot comments about MeterProvider shutdown paths are moot now that the OTel metrics pipeline is gone.

Happy to adjust the metric buckets, the operation-name handling, or add the response model / other semconv attributes if you'd prefer. Thanks!

krisztianfekete

Thanks for the changes, added some comments!

Couple of additional recommendations:

Since you wanted to go with the OTel SDK, you might want to push metrics. You can add the OTel Go bridge to enable optionally pushing metrics to an otlp endpoint. This can be a follow-up.
Please make sure to include steps (or even better, docs) in the PR on how did you test this (with screenshots) and how users should configure it to leverage the feature.

krisztianfekete · 2026-07-07T12:19:02Z

+	if agentConfig == nil || agentConfig.Model == nil {
+		return "", ""
+	}
+	return config.ModelName(agentConfig.Model), agentConfig.Model.GetType()


While some of these are semconv compliant, not all. Let's use an enum for this to align with the well-known names.

krisztianfekete · 2026-07-07T12:22:46Z

+		// Record GenAI token usage on the events that carry it. Output combines
+		// candidate + reasoning tokens.
+		if um := adkEvent.UsageMetadata; um != nil {
+			input := int64(um.PromptTokenCount)
+			output := int64(um.CandidatesTokenCount) + int64(um.ThoughtsTokenCount)
+			telemetry.RecordTokenUsage(e.modelName, e.providerName, input, output)
+		}


This seems to be correct today, but can you please make sure we don't double count partial events?

We should also add a test with multiple LLM streaming sessions and asserts the histogram observations equal the number of LLM calls (not the number of stream chunks).

krisztianfekete · 2026-07-07T12:23:40Z

+		Name: metricGenAIClientTokenUsage,
+		Help: "Measures the number of input and output tokens used by GenAI requests.",
+		// Token-count buckets spanning short prompts to large-context requests.
+		Buckets: []float64{1, 4, 16, 64, 256, 1024, 4096, 16384, 65536, 262144, 1048576},


Please use the semconv recommended bucket layout.

krisztianfekete · 2026-07-07T12:24:47Z

+	metricGenAIClientTokenUsage = "gen_ai_client_token_usage"
+
+	labelGenAITokenType     = "gen_ai_token_type"
+	labelGenAIRequestModel  = "gen_ai_request_model"


Please also add gen_ai.response.model

krisztianfekete · 2026-07-07T12:26:20Z

+	labelGenAIRequestModel  = "gen_ai_request_model"
+	labelGenAIProviderName  = "gen_ai_provider_name"
+	labelGenAIOperationName = "gen_ai_operation_name"
+


Can you please add error.type for failures as per the spec?

krisztianfekete · 2026-07-07T12:28:27Z

 	mux := http.NewServeMux()
 	RegisterHealthEndpoints(mux)
+	// Expose GenAI token-usage (and standard Go/process) metrics for scraping.
+	mux.Handle("/metrics", telemetry.MetricsHandler())


I guess we'd need to expose metrics on the controller generated objects as well.

@krisztianfekete

…ror.type, scrape annotations Addresses @krisztianfekete's review on kagent-dev#2149: - provider label mapped to semconv well-known gen_ai.provider.name values via SemconvProviderName (gemini_vertex_ai->gcp.vertex_ai, bedrock->aws.bedrock, azure_openai->azure.ai.openai, etc.); unknown types pass through. - add gen_ai.response.model (falls back to request model) and error.type labels. - use the semconv-recommended histogram bucket layout. - guard against double-counting streaming partial events (record once per LLM call, not per chunk) + test asserting observations == LLM calls. - controller: annotate Go-runtime agent pods for Prometheus discovery (prometheus.io/scrape|port|path) so the /metrics endpoint is scrapeable. - docs/genai-token-metrics.md: what is emitted, how to scrape, how to verify. The OTel Prometheus->OTLP push bridge is left as a follow-up per the review. Refs: kagent-dev#2148 Signed-off-by: Diego Braga <diego.braga86@gmail.com>

braghettos · 2026-07-07T12:53:52Z

Thanks for the detailed review @krisztianfekete! Pushed changes addressing all points:

Provider enum (main.go): added telemetry.SemconvProviderName() mapping kagent model types to well-known gen_ai.provider.name values — gemini_vertex_ai/gemini_anthropic → gcp.vertex_ai, bedrock → aws.bedrock, azure_openai → azure.ai.openai, gemini → gcp.gemini; openai/anthropic unchanged; unknown types (ollama, sap_ai_core, custom) pass through. Unit-tested.
gen_ai.response.model: added, sourced from the event's ModelVersion, falling back to the request model when the provider doesn't report one.
error.type: added as a label, set from the event's ErrorCode on failed requests, empty otherwise.
Bucket layout: switched to the semconv-recommended explicit buckets for gen_ai.client.token.usage.
Double-counting partial/streaming events: usage is now recorded only on non-partial events (extracted into recordTokenUsage). Added TestExecutor_StreamingUsageNotDoubleCounted which feeds multiple LLM calls, each with several partial chunks that also carry usage, and asserts observations == number of LLM calls (not chunks).
Expose on controller-generated objects: the controller now annotates Go-runtime agent pods with prometheus.io/scrape|port|path so the /metrics endpoint is discoverable by Prometheus/collector pod-SD (Python runtime pods are left un-annotated since they aren't instrumented yet). Tested in runtime_test.go.
Docs: added docs/genai-token-metrics.md — what's emitted, the labels, how to scrape (incl. an OTel Collector prometheus receiver snippet), and how to verify. (Screenshot placeholder left in the doc — I don't currently have a cluster with a scrape target wired to grab one; happy to add once I do.)
OTLP push bridge: left as a follow-up per your note.

The three earlier Copilot comments referenced the removed OTLP MeterProvider, so they're moot after the Prometheus rewrite.

krisztianfekete · 2026-07-07T13:07:04Z

Docs: added docs/genai-token-metrics.md — what's emitted, the labels, how to scrape (incl. an OTel Collector prometheus receiver snippet), and how to verify. (Screenshot placeholder left in the doc — I don't currently have a cluster with a scrape target wired to grab one; happy to add once I do.)

Thank you, will re-review once you had a chance to test it.

github-actions Bot added the enhancement New feature or request label Jul 4, 2026

This was referenced Jul 4, 2026

feat(core): emit Kubernetes Events on Agent/ModelConfig/MCPServerTool/RemoteMCPServer reconciles #2150

Open

feat(core): add controller log→OTLP bridge (gated OTEL_LOGGING_ENABLED) #2151

Open

braghettos marked this pull request as ready for review July 4, 2026 16:11

braghettos requested review from EItanya and supreme-gg-gg as code owners July 4, 2026 16:11

Copilot AI review requested due to automatic review settings July 4, 2026 16:11

braghettos requested review from ilackarms and yuval-k as code owners July 4, 2026 16:11

Copilot started reviewing on behalf of braghettos July 4, 2026 16:11 View session

Copilot AI reviewed Jul 4, 2026

View reviewed changes

krisztianfekete suggested changes Jul 6, 2026

View reviewed changes

braghettos force-pushed the observability-token-metrics branch from 55e63ba to fa6939a Compare July 6, 2026 19:14

braghettos changed the title ~~feat(adk): emit gen_ai.client.token.usage GenAI metric (gated OTEL_METRICS_ENABLED)~~ feat(adk): expose gen_ai.client.token.usage Prometheus metric Jul 6, 2026

github-actions Bot added enhancement New feature or request and removed enhancement New feature or request labels Jul 6, 2026

krisztianfekete suggested changes Jul 7, 2026

View reviewed changes

braghettos requested review from iplay88keys and jmhbh as code owners July 7, 2026 12:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(adk): expose gen_ai.client.token.usage Prometheus metric#2149

feat(adk): expose gen_ai.client.token.usage Prometheus metric#2149
braghettos wants to merge 2 commits into
kagent-dev:mainfrom
braghettos:observability-token-metrics

braghettos commented Jul 4, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

krisztianfekete left a comment

Uh oh!

braghettos commented Jul 6, 2026

Uh oh!

krisztianfekete left a comment

Uh oh!

krisztianfekete Jul 7, 2026

Uh oh!

krisztianfekete Jul 7, 2026

Uh oh!

krisztianfekete Jul 7, 2026

Uh oh!

krisztianfekete Jul 7, 2026

Uh oh!

krisztianfekete Jul 7, 2026

Uh oh!

krisztianfekete Jul 7, 2026

Uh oh!

braghettos commented Jul 7, 2026

Uh oh!

krisztianfekete commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

braghettos commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

How

Semconv alignment

Testing

Notes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

krisztianfekete left a comment

Choose a reason for hiding this comment

Uh oh!

braghettos commented Jul 6, 2026

Uh oh!

krisztianfekete left a comment

Choose a reason for hiding this comment

Uh oh!

krisztianfekete Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

krisztianfekete Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

krisztianfekete Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

krisztianfekete Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

krisztianfekete Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

krisztianfekete Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

braghettos commented Jul 7, 2026

Uh oh!

krisztianfekete commented Jul 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

braghettos commented Jul 4, 2026 •

edited

Loading