feat(observability): introduce minimal tracing implementation #4105

diegomarquezp · 2026-02-05T20:37:48Z

Summary

This PR introduces a new tracing mechanism in GAX that allows recording traces using OpenTelemetry. It provides a way of recording spans and attributes, following the existing ApiTracer class pattern with a few tracing-specific additions. The implementation is meant to be extensible to support other implementations.

New Classes

TracingRecorder: An interface for recording spans and attributes; can be implemented by observability frameworks.
OpenTelemetryTracingRecorder: An implementation of TracingRecorder that uses the OpenTelemetry API.
AppCentricTracer: An ApiTracer implementation that delegates span management to a TracingRecorder.
AppCentricTracerFactory: A factory for creating TracingTracer instances.
ApiTracerContext: A context object that carries information (like EndpointContext's server address property) used to infer common attributes for all tracers.
TraceSpan: A handle returned by TracingRecorder to manage the lifecycle of a specific span (ending it, recording errors, or setting attributes).

Approach

Connecting Tracer with Recorder

The implementation aims to decouple AppCentricTracer from TracingRecorder. When a tracer starts an operation or an attempt, it requests a TraceSpan from the recorder. This handle allows the tracer to update the span (e.g., adding attributes or recording errors) to keep AppCentricTracer separated from specific recorder implementations (like OpenTelemetry's Span object).

Attribute Inference via `ApiTracerContext`

To provide a source of Span Attributes that are common to all operations, we introduced ApiTracerContext. This context is passed to the ApiTracerFactory and contains information such as serverAddress (provided by EndpointContext). It is operated by ClientContext.
Initially, only serverAddress is contained in this class and it's meant to obtain the server.address attribute.

Integration Tests

A new integration test, ITOtelTracing, was added to the java-showcase module:

It validates that the expected spans (operation and attempt spans) are recorded with the correct names, parent-child relationships, and attributes (including server.address and gcp.client.language).

Note on java-bigtable downstream check

Since SkipTrailersTest mocks the tracer factory, the EndpointContext call to apiTracerFactory.withContext() returns a null factory, causing a null pointer exception when building the client context.
We expect the test to be adjusted with this change with the next release.

Confirmation in Cloud Trace

…-tracing-impl

This reverts commit b7b9e31.

…-tracing-impl

gemini-code-assist · 2026-02-05T20:38:19Z

Summary of Changes

Hello @diegomarquezp, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the observability capabilities of GAX by integrating a robust tracing system based on OpenTelemetry. The design emphasizes extensibility and clear separation of concerns, allowing GAX to record detailed RPC lifecycle events as spans and attributes. It introduces a mechanism to automatically enrich these traces with contextual information, providing deeper insights into client-server interactions.

Highlights

OpenTelemetry Integration: Introduces a new tracing mechanism in GAX that records traces using OpenTelemetry, providing an extensible way to monitor RPC lifecycles.
New Tracing Abstractions: Defines core interfaces ('TracingRecorder', 'SpanHandle') and implementations ('OpenTelemetryTracingRecorder', 'TracingTracer', 'TracingTracerFactory') to decouple tracing logic from specific observability frameworks.
Context-Based Attribute Inference: Implements 'ApiTracerContext' to carry information (like 'EndpointContext') for inferring common span attributes, such as 'server.address' and 'gcp.client.language'.
Comprehensive Testing: Adds new unit tests for the tracing components and integration tests ('ITOtelTracing') in 'java-showcase' to validate span recording, relationships, and attribute propagation across gRPC and HTTP/JSON clients.

Changelog

gax-java/gax/src/main/java/com/google/api/gax/rpc/ClientContext.java
- Integrated 'ApiTracerContext' to pass endpoint information to the 'ApiTracerFactory' for attribute inference.
gax-java/gax/src/main/java/com/google/api/gax/rpc/EndpointContext.java
- Added 'resolvedServerAddress()' method and logic to extract the server address from the resolved endpoint string.
gax-java/gax/src/main/java/com/google/api/gax/tracing/ApiTracerContext.java
- New class: Introduced 'ApiTracerContext' to hold common attributes for 'ApiTracer' instances, initially containing 'EndpointContext'.
gax-java/gax/src/main/java/com/google/api/gax/tracing/ApiTracerFactory.java
- Added a default 'withContext' method to allow 'ApiTracerFactory' implementations to incorporate 'ApiTracerContext' for attribute generation.
gax-java/gax/src/main/java/com/google/api/gax/tracing/OpenTelemetryTracingRecorder.java
- New class: Implemented 'TracingRecorder' using the OpenTelemetry API for creating, managing, and ending spans, including setting attributes and recording errors.
gax-java/gax/src/main/java/com/google/api/gax/tracing/TracingRecorder.java
- New interface: Defined 'TracingRecorder' and 'SpanHandle' to abstract span recording operations and lifecycle management.
gax-java/gax/src/main/java/com/google/api/gax/tracing/TracingTracer.java
- New class: Implemented 'ApiTracer' to use 'TracingRecorder' for span management, handling operation and attempt spans, and applying default and inferred attributes.
gax-java/gax/src/main/java/com/google/api/gax/tracing/TracingTracerFactory.java
- New class: Implemented 'ApiTracerFactory' to construct 'TracingTracer' instances, propagating 'TracingRecorder' and handling context-based attribute injection.
gax-java/gax/src/test/java/com/google/api/gax/rpc/EndpointContextTest.java
- Added a test to verify the correct parsing and resolution of server addresses from various endpoint formats.
gax-java/gax/src/test/java/com/google/api/gax/tracing/OpenTelemetryTracingRecorderTest.java
- New unit tests for 'OpenTelemetryTracingRecorder' covering span creation, kind assignment, attribute setting, error recording, and OpenTelemetry context management.
gax-java/gax/src/test/java/com/google/api/gax/tracing/TracingTracerFactoryTest.java
- New unit tests for 'TracingTracerFactory' validating tracer instantiation, attribute propagation, and 'ApiTracerContext' integration.
gax-java/gax/src/test/java/com/google/api/gax/tracing/TracingTracerTest.java
- New unit tests for 'TracingTracer' ensuring correct span lifecycle management for operations and attempts, and attribute handling.
java-showcase/gapic-showcase/src/test/java/com/google/showcase/v1beta1/it/ITOtelTracing.java
- New integration tests ('ITOtelTracing') to verify end-to-end OpenTelemetry tracing for gRPC and HTTP/JSON clients, including span naming, hierarchy, and attribute correctness.

Activity

This pull request introduces a new feature and includes several new files and tests. There is no specific human activity (comments, reviews, or progress updates) recorded in the provided context.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a new OpenTelemetry-based tracing system into GAX. Key changes include adding ApiTracerContext to provide endpoint information to tracers, enhancing EndpointContext to extract and store server addresses, and implementing TracingRecorder, TracingTracer, and TracingTracerFactory to integrate OpenTelemetry for operation and attempt spans. Review comments highlight the need to address thread-safety in TracingTracer by using ConcurrentHashMap for attributes, improve the robustness of EndpointContext's server address parsing using java.net.URL, remove a potentially problematic default implementation in TracingRecorder to enforce explicit parent span handling, standardize span naming conventions in TracingTracerFactory to align with OpenTelemetry, and correct the copyright year across several new files.

gax-java/gax/src/main/java/com/google/api/gax/tracing/TracingTracer.java

gax-java/gax/src/main/java/com/google/api/gax/tracing/TraceRecorder.java

gax-java/gax/src/main/java/com/google/api/gax/tracing/OpenTelemetryTraceRecorder.java

gax-java/gax/src/main/java/com/google/api/gax/rpc/EndpointContext.java

gax-java/gax/src/main/java/com/google/api/gax/tracing/TracingRecorder.java

gax-java/gax/src/main/java/com/google/api/gax/tracing/AppCentricTracerFactory.java

gax-java/gax/src/test/java/com/google/api/gax/tracing/OpenTelemetryTraceRecorderTest.java

gax-java/gax/src/test/java/com/google/api/gax/tracing/AppCentricTracerFactoryTest.java

gax-java/gax/src/test/java/com/google/api/gax/tracing/AppCentricTracerTest.java

java-showcase/gapic-showcase/src/test/java/com/google/showcase/v1beta1/it/ITOtelTracing.java

gax-java/gax/src/main/java/com/google/api/gax/tracing/AppCentricTracerFactory.java

gax-java/gax/src/main/java/com/google/api/gax/tracing/TracingTracer.java

blakeli0 · 2026-02-06T20:22:07Z

gax-java/gax/src/main/java/com/google/api/gax/tracing/TracingRecorder.java

+    return () -> {};
+  }
+
+  interface SpanHandle {


Do we have to create a separate interface? Can this be flattened?

Discussed offline. We still need a reference to the Span created earlier in the request. Rename this to something like GaxSpan that wraps Span from Opentelemetry.

Renamed to GaxSpan

After renaming some classes to AppCentric* and since other classes are note prefixed with Gax*, a name like TraceSpan to indicate "it's the span of a trace" could work for our purposes. WDYT?

blakeli0 · 2026-02-06T20:24:05Z

gax-java/gax/src/main/java/com/google/api/gax/tracing/OpenTelemetryTracingRecorder.java

+  }
+
+  @Override
+  public SpanHandle startSpan(String name, Map<String, String> attributes) {


Are there any attributes specific to the start of Span? If not, I think we may want to make them a field of the tracer, instead of always passing in as a parameter.

resend_count may be the only attribute specific to the start of the span.

I think a more accurate name here would be createSpan to make more clear the need of the attributes field. Starting the span is more of a detail of the recorder implementation.

A field of the tracer

TracingTracer keeps most attributes as a field, and this recorder class is provided by the user. I think due to resend_count we may want to keep the attributes as a parameter.

gax-java/gax/src/main/java/com/google/api/gax/tracing/TracingRecorder.java

gax-java/gax/src/main/java/com/google/api/gax/rpc/ClientContext.java

gax-java/gax/src/main/java/com/google/api/gax/tracing/OpenTelemetryTracingRecorder.java

- Rename SpanHandle to GaxSpan - Simplify adding op and attempt attributes - Use HashMap instead of ConcurrentHashMap - Remove error handling (for now)

…ivate

diegomarquezp · 2026-02-10T20:16:31Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a new tracing mechanism based on OpenTelemetry, abstracting the implementation behind a TraceRecorder interface. While the integration tests provide good coverage, a significant thread-safety issue was identified in the AppCentricTracer class. A shared HashMap is modified concurrently in the constructor, which could lead to map corruption and Denial of Service in multi-threaded environments. Additionally, there is a potential bug in endpoint parsing for IPv6 addresses.

gax-java/gax/src/main/java/com/google/api/gax/tracing/AppCentricTracer.java

gax-java/gax/src/main/java/com/google/api/gax/rpc/EndpointContext.java

…icTracer.java Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

…om/googleapis/sdk-platform-java into observability/initial-tracing-impl

diegomarquezp · 2026-02-10T20:29:32Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a new tracing mechanism based on OpenTelemetry, which is a great addition for observability. However, a security concern has been identified regarding how server addresses are parsed for tracing attributes. The current implementation may leak sensitive information (such as credentials or tokens) from the endpoint URL into the tracing system if the URL contains userinfo, paths, or query parameters. A suggestion has been provided to improve the parsing logic to strip these sensitive components. Additionally, feedback has been provided to improve consistency, specifically regarding span naming conventions across different span types and enhancing integration tests for consistent attribute verification across different transport protocols.

gax-java/gax/src/main/java/com/google/api/gax/rpc/EndpointContext.java

gax-java/gax/src/main/java/com/google/api/gax/tracing/AppCentricTracerFactory.java

java-showcase/gapic-showcase/src/test/java/com/google/showcase/v1beta1/it/ITOtelTracing.java

gax-java/gax/src/main/java/com/google/api/gax/rpc/EndpointContext.java

gax-java/gax/src/main/java/com/google/api/gax/tracing/AppCentricTracer.java

sonarqubecloud · 2026-02-11T18:57:19Z

Quality Gate passed for 'gapic-generator-java-root'

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
94.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

sonarqubecloud · 2026-02-11T19:01:04Z

Quality Gate passed for 'java_showcase_integration_tests'

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
92.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

diegomarquezp added 10 commits February 4, 2026 14:34

feat: introduce minimal implementation of OTel tracing

7082247

chore: avoid new public methods

bfe6a6d

feat: add api tracer context

ea6cb18

Merge remote-tracking branch 'origin/main' into observability/initial…

97bec08

…-tracing-impl

chore: concise comment

097f701

fix: use internal span kind for operations

f547e5a

chore: remove unnecessary inScope implementation

b7b9e31

Revert "chore: remove unnecessary inScope implementation"

556c84c

This reverts commit b7b9e31.

test: add test for inScope()

0359b7d

Merge remote-tracking branch 'origin/main' into observability/initial…

402cb89

…-tracing-impl

product-auto-label bot added the size: xl Pull request size is extra large. label Feb 5, 2026

gemini-code-assist bot reviewed Feb 5, 2026

View reviewed changes

diegomarquezp added 6 commits February 5, 2026 15:45

chore: use suggested server address resolution impl

420278b

fix: use concurrent hash map

6de9fb6

chore: remove default impl for startSpan with parent

a61dacf

chore: add opentelemery-context to tests

b000d52

test: increase coverage for TracingTracerTest

9ce0c34

deps: include opentelemetry context in gax

9c6c737

diegomarquezp marked this pull request as ready for review February 5, 2026 23:54

blakeli0 reviewed Feb 6, 2026

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/tracing/TracingRecorder.java Outdated Show resolved Hide resolved

blakeli0 reviewed Feb 6, 2026

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/tracing/TracingRecorder.java Outdated Show resolved Hide resolved

blakeli0 reviewed Feb 6, 2026

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/rpc/ClientContext.java Show resolved Hide resolved

blakeli0 reviewed Feb 6, 2026

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/tracing/OpenTelemetryTracingRecorder.java Outdated Show resolved Hide resolved

diegomarquezp added 4 commits February 9, 2026 12:00

chore: simplify and remove error handling

e77de58

chore: review refactor

8f07f61

- Rename SpanHandle to GaxSpan - Simplify adding op and attempt attributes - Use HashMap instead of ConcurrentHashMap - Remove error handling (for now)

chore: make TracingTracerFactory(recorder, opAtts, atAtts) package pr…

2cfcd68

…ivate

chore: remove unnecessary inScope

2415c6b

product-auto-label bot added the size: l Pull request size is large. label Feb 10, 2026

product-auto-label bot removed the size: xl Pull request size is extra large. label Feb 10, 2026

diegomarquezp added 6 commits February 10, 2026 12:39

chore: rename startSpan to createSpan

0f5e9b5

chore: use server address instead of endpoint context

0c3bd22

chore: rename classes

bdeb291

chore: add javadoc for tracer

5c07a3c

chore: rename to TraceSpan, improve javadocs

ff1d45d

chore: format

043ce62

gemini-code-assist bot reviewed Feb 10, 2026

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/tracing/AppCentricTracer.java Outdated Show resolved Hide resolved

gax-java/gax/src/main/java/com/google/api/gax/rpc/EndpointContext.java Show resolved Hide resolved

diegomarquezp and others added 3 commits February 10, 2026 15:20

Update gax-java/gax/src/main/java/com/google/api/gax/tracing/AppCentr…

dbc5f41

…icTracer.java Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

chore: handle ipv6 in endpoint context

b4bce0d

Merge branch 'observability/initial-tracing-impl' of https://github.c…

6e95cf4

…om/googleapis/sdk-platform-java into observability/initial-tracing-impl

product-auto-label bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Feb 10, 2026

gemini-code-assist bot reviewed Feb 10, 2026

View reviewed changes

chore: add language tests

e873904

blakeli0 reviewed Feb 10, 2026

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/rpc/EndpointContext.java Show resolved Hide resolved

blakeli0 reviewed Feb 10, 2026

View reviewed changes

gax-java/gax/src/main/java/com/google/api/gax/tracing/AppCentricTracer.java Outdated Show resolved Hide resolved

chore: AppCentricTracer to implement interface

68943b0

feat(observability): introduce minimal tracing implementation #4105

Are you sure you want to change the base?

feat(observability): introduce minimal tracing implementation #4105

Uh oh!

Conversation

diegomarquezp commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

New Classes

Approach

Connecting Tracer with Recorder

Attribute Inference via ApiTracerContext

Integration Tests

Note on java-bigtable downstream check

Confirmation in Cloud Trace

Uh oh!

gemini-code-assist bot commented Feb 5, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

blakeli0 Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

blakeli0 Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

diegomarquezp Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

diegomarquezp Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

blakeli0 Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

diegomarquezp Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

diegomarquezp commented Feb 10, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

diegomarquezp commented Feb 10, 2026

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sonarqubecloud bot commented Feb 11, 2026

Quality Gate passed for 'gapic-generator-java-root'

diegomarquezp commented Feb 5, 2026 •

edited

Loading

Attribute Inference via `ApiTracerContext`