feat: add initial opentelemetry tracing to big query HTTP requests by ldetmer · Pull Request #4126 · googleapis/java-bigquery

ldetmer · 2026-03-02T20:08:04Z

This feature adds the ability to enable open telemetry tracing on all HTTP requests.

It reuses the existing client options setting enableOpenTelemetryTracing to enable it, and then wraps the existing HttpRequestInitializer to intercept and add tracing.

This PR only contains the initial basic general/http attributes. Separate PRs will contain additional attributes.

Tested via sample test program and validated attributes show up in cloud trace:
https://screenshot.googleplex.com/AQJp4Nbb6oVbgAk

gemini-code-assist · 2026-03-02T20:08:24Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the BigQuery client library by adding initial OpenTelemetry tracing capabilities to its HTTP requests. The changes enable the client to automatically generate and enrich OpenTelemetry spans for each HTTP call, providing deeper insights into the performance and behavior of interactions with the BigQuery service. This feature is conditionally applied, ensuring that tracing is only active when explicitly enabled.

Highlights

OpenTelemetry Tracing Integration: Introduced initial OpenTelemetry tracing for BigQuery HTTP requests, allowing for detailed observability of network calls.
HttpTracingRequestInitializer: Added a new HttpTracingRequestInitializer class that wraps existing HTTP request initializers to inject OpenTelemetry spans, capturing request and response details.
Telemetry Attributes: Defined common OpenTelemetry attribute keys for GCP client information, HTTP request/response details, and error/exception handling within the BigQueryTelemetryTracer utility.
Error and Exception Handling: Implemented logic within the tracing initializer to record error status codes, exception types, and messages in OpenTelemetry spans for failed HTTP requests.
Comprehensive Testing: Included both unit and integration tests to ensure the correct functioning of the HTTP tracing, verifying attribute capture for successful, error, and exceptional scenarios.

Changelog

google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java
- Modified to conditionally wrap the HttpRequestInitializer with HttpTracingRequestInitializer if OpenTelemetry tracing is enabled and a tracer is provided.
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java
- Added a new class to intercept HTTP requests and create OpenTelemetry spans.
- Implemented logic to capture HTTP method, URL, host, port, and request/response body sizes.
- Included mechanisms to record response status codes, error messages, and exceptions within the spans.
google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/telemetry/BigQueryTelemetryTracer.java
- Added a new utility class to define standard OpenTelemetry attribute keys for GCP client, error, and server information.
- Provided a helper method for creating SpanBuilder instances with default BigQuery client attributes.
google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/spi/v2/HttpTracingIntegrationTest.java
- Added an integration test to validate HTTP tracing with a real HTTP server, confirming span creation and attribute population for successful requests.
google-cloud-bigquery/src/test/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializerTest.java
- Added unit tests for HttpTracingRequestInitializer to verify correct attribute setting for success and error responses.
- Tested exception recording when the original unsuccessful response handler throws an IOException.
- Confirmed that the delegate HttpRequestInitializer is called as expected.

Activity

The pull request was opened by ldetmer.
The author used a standard pull request template, outlining prerequisites such as opening an issue, passing tests, maintaining code coverage, and updating documentation.
The PR aims to add initial OpenTelemetry tracing to BigQuery HTTP requests, as indicated by the title.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces OpenTelemetry tracing for BigQuery HTTP requests, enhancing observability. However, the current implementation has security concerns, as it leaks potentially sensitive information (PII, identifiers, and query details) into telemetry span attributes via full URLs and raw error messages. Furthermore, a critical bug allows spans to be ended multiple times, potentially corrupting telemetry data, and a separate issue in HttpBigQueryRpc.java leads to attribute loss. Addressing these security vulnerabilities and bugs, along with improving adherence to OpenTelemetry semantic conventions and completing TODO items, is crucial for accurate and secure telemetry.

...d-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java

gemini-code-assist · 2026-03-02T20:13:15Z

...d-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java

+  public static final AttributeKey<String> HTTP_REQUEST_METHOD =
+          AttributeKey.stringKey("http.request.method");
+  public static final AttributeKey<String> URL_FULL = AttributeKey.stringKey("url.full");
+  public static final AttributeKey<String> URL_TEMPLATE = AttributeKey.stringKey("url.template");
+  public static final AttributeKey<String> URL_DOMAIN = AttributeKey.stringKey("url.domain");
+  public static final AttributeKey<Long> HTTP_RESPONSE_STATUS_CODE =
+          AttributeKey.longKey("http.response.status_code");
+  public static final AttributeKey<Long> HTTP_REQUEST_RESEND_COUNT =
+          AttributeKey.longKey("http.request.resend_count");
+  public static final AttributeKey<Long> HTTP_REQUEST_BODY_SIZE =
+          AttributeKey.longKey("http.request.body.size");
+  public static final AttributeKey<Long> HTTP_RESPONSE_BODY_SIZE =
+          AttributeKey.longKey("http.response.body.size");


The OpenTelemetry attribute keys being defined here are based on an older version of the semantic conventions. The conventions have been stabilized and it's recommended to use the new key names to ensure compatibility with standard OpenTelemetry tooling.

For example:

http.request.method should be http.method

http.response.status_code should be http.status_code

http.request.body.size should be http.request_content_length

http.response.body.size should be http.response_content_length

Consider adding a dependency on io.opentelemetry:opentelemetry-semconv and using the constants from io.opentelemetry.semconv.trace.attributes.SemanticAttributes to stay aligned with the specification. If you prefer to avoid the extra dependency, please update the string keys manually.

public static final AttributeKey<String> HTTP_REQUEST_METHOD = AttributeKey.stringKey("http.method"); public static final AttributeKey<String> URL_FULL = AttributeKey.stringKey("url.full"); public static final AttributeKey<String> URL_TEMPLATE = AttributeKey.stringKey("url.template"); public static final AttributeKey<String> URL_DOMAIN = AttributeKey.stringKey("url.domain"); public static final AttributeKey<Long> HTTP_RESPONSE_STATUS_CODE = AttributeKey.longKey("http.status_code"); public static final AttributeKey<Long> HTTP_REQUEST_RESEND_COUNT = AttributeKey.longKey("http.request.resend_count"); public static final AttributeKey<Long> HTTP_REQUEST_BODY_SIZE = AttributeKey.longKey("http.request_content_length"); public static final AttributeKey<Long> HTTP_RESPONSE_BODY_SIZE = AttributeKey.longKey("http.response_content_length");

FYI I doubled, checked and it looks like what we're using is correct:

https://opentelemetry.io/docs/specs/semconv/http/http-spans/

...d-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java

...loud-bigquery/src/main/java/com/google/cloud/bigquery/telemetry/BigQueryTelemetryTracer.java

blakeli0 · 2026-03-04T22:33:09Z

...loud-bigquery/src/main/java/com/google/cloud/bigquery/telemetry/BigQueryTelemetryTracer.java

+
+  // Common GCP Attributes
+  public static final AttributeKey<String> GCP_CLIENT_SERVICE =
+      AttributeKey.stringKey("gcp.client.service");


It's OK to hardcode these keys in Bigquery for now, we may want to use them from Gax once they are stabilized.

blakeli0 · 2026-03-04T22:34:25Z

...d-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java

+ * HttpRequestInitializer that wraps a delegate initializer, intercepts all HTTP requests, adds
+ * OpenTelemetry tracing and then invokes delegate interceptor.
+ */
+@InternalApi


Let's mark all new classes as @BetaApi so we can easily change them before GA.

blakeli0 · 2026-03-04T22:40:07Z

google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java

+
+    String resolvedBigQueryRootUrl = options.getResolvedApiaryHost("bigquery");
+
+    if (options.isOpenTelemetryTracingEnabled() && options.getOpenTelemetryTracer() != null) {


I think we need to put more thoughts about how to enable this feature.

This seems to be reusing the existing options which may or may not be a good practice. Because 1. Existing customers will also automatically get new Spans which they may not want. 2. Customers will get two different Spans which are not related at this moment.

Can we understand more about the use cases of the existing Spans?

+1 to Blake's concerns. I'm leaning against reusing the existing options for the reasons he listed. Would it be possible to have something like isHttpTelemetryTracingEnabled() to focus on the network-level tracing?

blakeli0 · 2026-03-04T22:54:06Z

...d-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java

+  }
+
+  @VisibleForTesting
+  static String sanitizeUrlFull(String url) {


This seems a lot of logic and could easily make mistakes. Did Rust do similar things?

this is Rusts's logic, so in function I would say is comparable. I will try to come up with a simpler/idiomatic way to scrub the query params

can you use (a clone of) request.getUrl() instead of string manip?

blakeli0 · 2026-03-04T23:02:35Z

...d-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java

+  }
+
+  private static void addExceptionToSpan(IOException e, Span span) {
+    span.recordException(e);


What exactly does recordException do? Does it automatically creates a few attributes? It seems like a good practice, but I want to make sure it does not conflict with existing attributes.

Same question for setStatus below.

recordException adds an event type "exception" to a span (does not affect attributes). See screenshot.

For setStatus, it sets the overall status of the span (also not an attribute), default is Unset. See screenshot.

Let me know if you want me to flag either of these for confirmation from Wes.

westarle

Just a couple q's about lifecycle of initializers; I had imagined a more "direct" instrumentation around execute().

westarle · 2026-03-05T20:33:07Z

...d-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java

+            addExceptionToSpan(e, span);
+            throw e;
+          } finally {
+            span.end();


can this happen twice if the request is retried?

westarle · 2026-03-05T20:33:42Z

...d-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java

+
+    Span span = createHttpTraceSpan(httpMethod, url, host, port);
+
+    HttpResponseInterceptor originalInterceptor = request.getResponseInterceptor();


are exactly one of these guaranteed to run to close the span?

westarle · 2026-03-05T20:36:13Z

...d-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java

+  }
+
+  @VisibleForTesting
+  static String sanitizeUrlFull(String url) {


can you use (a clone of) request.getUrl() instead of string manip?

jinseopkim0 · 2026-03-05T18:27:30Z

...d-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java

+            addExceptionToSpan(e, span);
+            throw e;
+          } finally {
+            span.end();


Will this result in ending the span before a retry can occur?

jinseopkim0 · 2026-03-05T18:32:10Z

...d-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java

+    String host = request.getUrl().getHost();
+    Integer port = request.getUrl().getPort();
+
+    Span span = createHttpTraceSpan(httpMethod, url, host, port);


Since the span is started here during initialize(), what happens if an exception occurs in the client before the request actually executes? Is there a risk that the response handlers are never reached and span.end() is never called, leading to a span leak?

jinseopkim0 · 2026-03-05T18:40:31Z

...d-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpTracingRequestInitializer.java

+    } catch (Exception ex) {
+      // Ignore
+    }
+    span.setAttribute(BigQueryTelemetryTracer.STATUS_MESSAGE, errorMessage);


Are there any data privacy concerns with exporting the error message? I assume BigQuery error messages could potentially contain sensitive details like SQL query snippets, table names, or row data.

jinseopkim0 · 2026-03-05T23:07:53Z

google-cloud-bigquery/src/main/java/com/google/cloud/bigquery/spi/v2/HttpBigQueryRpc.java

+
+    String resolvedBigQueryRootUrl = options.getResolvedApiaryHost("bigquery");
+
+    if (options.isOpenTelemetryTracingEnabled() && options.getOpenTelemetryTracer() != null) {


+1 to Blake's concerns. I'm leaning against reusing the existing options for the reasons he listed. Would it be possible to have something like isHttpTelemetryTracingEnabled() to focus on the network-level tracing?

feat: add initial opentelemetry tracing to big query HTTP requests

f5fa8ea

product-auto-label bot added size: l Pull request size is large. api: bigquery Issues related to the googleapis/java-bigquery API. labels Mar 2, 2026

gemini-code-assist bot reviewed Mar 2, 2026

View reviewed changes

ldetmer added 5 commits March 2, 2026 16:08

formated code

4faaa82

cleaned up code + added suggestions from gemini

4202fb3

added handling of full.url redacted content

0372db5

remove nullable annotation

de0ee8a

simplified logic

68704bd

ldetmer marked this pull request as ready for review March 4, 2026 16:56

ldetmer requested review from a team as code owners March 4, 2026 16:56

blakeli0 reviewed Mar 4, 2026

View reviewed changes

jinseopkim0 self-requested a review March 5, 2026 16:20

westarle reviewed Mar 5, 2026

View reviewed changes

jinseopkim0 requested changes Mar 5, 2026

View reviewed changes


		String resolvedBigQueryRootUrl = options.getResolvedApiaryHost("bigquery");

		if (options.isOpenTelemetryTracingEnabled() && options.getOpenTelemetryTracer() != null) {


		Span span = createHttpTraceSpan(httpMethod, url, host, port);

		HttpResponseInterceptor originalInterceptor = request.getResponseInterceptor();

Conversation

ldetmer commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot commented Mar 2, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist bot Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

westarle left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ldetmer commented Mar 2, 2026 •

edited

Loading