Skip to content

feat(google-genai): add util-backed 2.x instrumentation#250

Draft
sipercai wants to merge 1 commit into
mainfrom
feat/google-genai-observability
Draft

feat(google-genai): add util-backed 2.x instrumentation#250
sipercai wants to merge 1 commit into
mainfrom
feat/google-genai-observability

Conversation

@sipercai

Copy link
Copy Markdown
Collaborator

Description

This PR adds LoongSuite-native instrumentation for the Google Gen AI Python SDK, backed by opentelemetry-util-genai.

It instruments the Google Gen AI 2.x synchronous and asynchronous APIs for:

  • generate_content
  • generate_content_stream
  • embed_content

The implementation records LLM and embedding spans, request/response model metadata, response IDs, finish reasons, token usage, captured messages, streaming TTFT, embedding dimensions, errors, and GenAI client metrics.

The package includes opt-in live tests and sanitized VCR coverage. Public CI replays cassettes with --vcr-record=none and does not require provider credentials.

Fixes # (N/A)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • PRE_COMMIT_HOME="$(mktemp -d)" tox -e precommit
  • tox -e generate-workflows
  • tox -c tox-loongsuite.ini -e py312-test-loongsuite-instrumentation-google-genai-oldest,py312-test-loongsuite-instrumentation-google-genai-latest
  • Real Google Gen AI 2.14.0 provider smoke with CMS/ARMS readback

Does This PR Require a Core Repo Change?

  • Yes. - Link to PR:
  • No.

Checklist:

See contributing.md for styleguide, changelog guidelines, and more.

  • Followed the style guidelines of this project
  • Changelogs have been updated
  • Unit tests have been added
  • Documentation has been updated

Validation Evidence

Spec and Scope

  • Linked issue/spec: N/A; direct feature implementation.
  • Approved spec/comment: add Google Gen AI SDK 2.x instrumentation and keep the change scoped to the provider SDK.
  • Changed surface: one new Google Gen AI instrumentation package, bootstrap registration, documentation, sanitized VCR fixtures, and package-scoped tox/workflow registration.

Local Checks

Check Command Result Notes
Static readiness python "$PIPELINE_SKILL_DIR/scripts/check_loongsuite_pr_readiness.py" --repo . pass No admission errors or warnings.
Precommit PRE_COMMIT_HOME="$(mktemp -d)" tox -e precommit pass Fresh hook cache; all hooks passed and the post-run tree was stable.
Focused tests tox -c tox-loongsuite.ini -e py312-test-loongsuite-instrumentation-google-genai-oldest,py312-test-loongsuite-instrumentation-google-genai-latest pass Oldest: 18 passed/7 skipped; latest: 24 passed/1 live-opt-in skipped.
Workflow generation tox -e generate-workflows pass Generated lint and test workflows are up to date.
Change detector .github/scripts/detect_loongsuite_changes.py with the PR file list pass full=false; package set contains only loongsuite-instrumentation-google-genai.
Claude review three-round local review/fix/re-review gate pass No unresolved P0/P1 findings.
Privacy scan cassette and branch-diff credential/path scan pass No real API key, bearer token, username, local path, private trace ID, or CMS endpoint was found.

Real E2E Matrix

Scenario Status Command or Demo Evidence
non-streaming pass bounded Google Gen AI 2.14.0 live demo Real Gemini response produced a generate_content LLM span with response and token attributes.
streaming pass synchronous and asynchronous live streaming calls Both spans completed with merged output, usage, and TTFT.
concurrency pass two simultaneous live provider calls Each call used an isolated trace and captured its own response content.
agent/tool/ReAct N/A provider SDK surface The SDK exposes function declarations and model tool-call parts, but does not own an agent execution lifecycle or AGENT/STEP/TOOL spans.
tool-heavy N/A provider SDK surface Multi-tool orchestration and tool execution are outside this model/embedding SDK instrumentation.
error path pass real invalid-model request The provider exception was preserved and the LLM span recorded ERROR with error.type=ClientError.

Additional live coverage included direct multi-turn generation, forced function calling, synchronous/asynchronous embeddings, and asynchronous non-streaming generation.

Telemetry and Weaver

Check Status Command or Artifact Notes
Span tree / span kinds pass in-memory exporter plus fresh CMS/ARMS readback 11 isolated traces and 23 spans were read back; the SDK emitted 10 LLM and 2 EMBEDDING spans with the expected parent-child structure.
Content capture modes pass focused tests plus live SPAN_ONLY demo Live spans contained request/response messages; the shared util retains default NO_CONTENT behavior.
Metrics pass local OTel reader plus CMS/ARMS PromQL readback Client duration and token metrics were emitted; backend call count, duration, and token series were present.
Concurrency isolation pass bounded two-call live smoke No cross-trace parent or content mixing was observed.
Weaver live-check blocked weaver registry live-check -r <registry>/model --advice-profile loongsuite-genai ... The current advice registry fails before sample evaluation because tool_call_index is defined twice. The 12-span JSON sample is available for rerun after the registry is fixed.

CI

  • GitHub checks: pending PR creation.
  • Known unrelated failures: none at publication time.
  • Follow-up needed: inspect generated PR checks after creation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants