Skip to content

Force-flush trace spans before the A2A response completes for substrate#2176

Open
iplay88keys wants to merge 2 commits into
mainfrom
iplay88keys/substrate-updates
Open

Force-flush trace spans before the A2A response completes for substrate#2176
iplay88keys wants to merge 2 commits into
mainfrom
iplay88keys/substrate-updates

Conversation

@iplay88keys

Copy link
Copy Markdown
Contributor

Description

On Agent Substrate, the session actor is gVisor-checkpointed as soon as the A2A response body closes. Spans still buffered in the SDK's BatchSpanProcessor (default ~5s flush interval) freeze into the snapshot and only export when the session is next resumed — so a session's last message never produces a trace, and every other message's trace is delayed until the next chat. Both ADK executors now force-flush the tracer provider before the response completes.

Go ADK

  • New telemetry.ForceFlush(ctx): flushes the global tracer provider's batch processor with a detached 3s timeout (the request context is already canceled when deferred cleanup runs); no-op for providers without ForceFlush (tracing disabled).
  • a2a/executor.go calls it in the execute defer, after the root invocation span ends so the root span is included in the flush.

Python ADK

  • New kagent.core.tracing.force_flush(timeout_millis=3000): no-op when the provider has no force_flush; exporter errors are logged, never raised.
  • _agent_executor.py calls it in a finally on execute() via asyncio.to_thread, so the blocking gRPC export cannot stall the event loop.

A collector outage costs at most one bounded 3s flush per invocation, after response events are already published — it never adds user-visible chat latency.

Also included: the grafana-mcp and querydoc helm image helpers now append the optional image.name segment when set (compact drops it when absent, so charts using the default two-segment registry/repository values render unchanged).

Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
Signed-off-by: Jeremy Alvis <jeremy.alvis@solo.io>
@iplay88keys iplay88keys marked this pull request as ready for review July 7, 2026 21:12
@iplay88keys iplay88keys requested review from a team and supreme-gg-gg as code owners July 7, 2026 21:12
Copilot AI review requested due to automatic review settings July 7, 2026 21:12

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR ensures OpenTelemetry spans are force-flushed before an A2A response completes on Agent Substrate (to avoid buffered spans being checkpoint-frozen and exported only on the next resume), and updates Helm helper templates to optionally include an image.name segment when constructing image references.

Changes:

  • Add Go telemetry.ForceFlush(ctx) and invoke it after the root invocation span ends in the Go A2A executor.
  • Add Python kagent.core.tracing.force_flush() and invoke it at the end of the Python ADK executor via asyncio.to_thread to avoid blocking the event loop.
  • Update grafana-mcp and querydoc Helm image helpers to include an optional image.name segment in the image path.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
python/packages/kagent-core/tests/test_tracing_configure.py Adds unit tests for the new Python tracing force-flush helper behavior.
python/packages/kagent-core/src/kagent/core/tracing/_utils.py Introduces force_flush() to flush buffered spans with a bounded timeout and swallow exporter errors.
python/packages/kagent-core/src/kagent/core/tracing/init.py Exports force_flush from the tracing package public API.
python/packages/kagent-adk/src/kagent/adk/_agent_executor.py Calls tracing force-flush in execute() cleanup using asyncio.to_thread.
helm/tools/querydoc/templates/_helpers.tpl Extends image reference construction to include optional image.name.
helm/tools/grafana-mcp/templates/_helpers.tpl Extends image reference construction to include optional image.name.
go/adk/pkg/telemetry/tracing.go Adds ForceFlush helper that detaches from request cancellation and flushes spans with a 3s timeout.
go/adk/pkg/telemetry/tracing_test.go Adds test ensuring buffered spans are exported on flush and that no-op providers don’t panic.
go/adk/pkg/a2a/executor.go Flushes spans after ending the invocation span in deferred executor cleanup.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

# body closes, freezing any unexported spans into the snapshot.
# Run in a thread so the blocking gRPC export (bounded by the
# flush timeout) doesn't stall the event loop.
await asyncio.to_thread(force_flush_tracing)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants