Skip to content

chore(spanner): support built-in metrics for ResultSet streaming RPCs - #6302

Open
olavloite wants to merge 2 commits into
googleapis:mainfrom
olavloite:spanner-streaming-resultset-metrics
Open

chore(spanner): support built-in metrics for ResultSet streaming RPCs#6302
olavloite wants to merge 2 commits into
googleapis:mainfrom
olavloite:spanner-streaming-resultset-metrics

Conversation

@olavloite

Copy link
Copy Markdown
Contributor

Adds OpenTelemetry built-in metrics instrumentation for streaming operations (ExecuteStreamingSql and StreamingRead) in Cloud Spanner ResultSet.

  • Records attempt_count, attempt_latencies, gfe_latencies, and afe_latencies across all initial attempts and stream restarts.
  • Records end-to-end operation_count and operation_latencies spanning the entire query lifecycle until stream completion or permanent failure.
  • Preserves attempt response headers across background stream trailer drains (seen_last) and early drops.
  • Standardizes metric method attribute formatting to "Spanner.<MethodName>" (Spanner.ExecuteStreamingSql, Spanner.StreamingRead) matching Java and Go reference clients.
  • Parses single and multiple server-timing response headers for GFE and AFE timings.

Adds OpenTelemetry built-in metrics instrumentation for streaming operations (`ExecuteStreamingSql` and `StreamingRead`) in Cloud Spanner `ResultSet`.
- Records `attempt_count`, `attempt_latencies`, `gfe_latencies`, and `afe_latencies` across all initial attempts and stream restarts.
- Records end-to-end `operation_count` and `operation_latencies` spanning the entire query lifecycle until stream completion or permanent failure.
- Preserves attempt response headers across background stream trailer drains (`seen_last`) and early drops.
- Standardizes metric `method` attribute formatting to `"Spanner.<MethodName>"` (`Spanner.ExecuteStreamingSql`, `Spanner.StreamingRead`) matching Java and Go reference clients.
- Parses single and multiple `server-timing` response headers for GFE and AFE timings.
@olavloite
olavloite requested review from a team as code owners August 7, 2026 12:01
@product-auto-label product-auto-label Bot added the api: spanner Issues related to the Spanner API. label Aug 7, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces attempt and operation metrics tracking for Spanner streaming queries and reads, including parsing server-timing headers for GFE/AFE latencies and normalising method names. The review feedback highlights several critical gaps in error handling and metrics accuracy: stream initialization failures are currently not recorded (and incorrectly reported as successful upon drop), stale headers from previous attempts are not cleared during retries, and complete connection failures prior to ResultSet creation are missed entirely by the metrics recorder.

Comment thread src/spanner/src/result_set.rs
Comment thread src/spanner/src/result_set.rs
Comment thread src/spanner/src/read_only_transaction.rs Outdated
Comment thread src/spanner/src/batch_read_only_transaction.rs Outdated
Comment thread src/spanner/src/batch_read_only_transaction.rs Outdated
@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.20972% with 31 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.15%. Comparing base (3fb8e6d) to head (3740e16).
⚠️ Report is 9 commits behind head on main.

Files with missing lines Patch % Lines
src/spanner/src/observability/metrics.rs 94.69% 12 Missing ⚠️
src/spanner/src/result_set.rs 98.77% 10 Missing ⚠️
src/spanner/src/batch_read_only_transaction.rs 82.50% 7 Missing ⚠️
src/spanner/src/server_streaming/stream.rs 83.33% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6302      +/-   ##
==========================================
- Coverage   96.26%   96.15%   -0.11%     
==========================================
  Files         282      282              
  Lines       72762    73739     +977     
==========================================
+ Hits        70044    70905     +861     
- Misses       2718     2834     +116     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@olavloite
olavloite force-pushed the spanner-streaming-resultset-metrics branch from 7f24278 to 3e80e52 Compare August 7, 2026 12:53
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements built-in metrics collection for Spanner streaming operations, tracking attempt and operation counts, latencies, and GFE/AFE latencies parsed from response headers. It updates ResultSet to record these metrics during stream initialization, execution, retries, and upon being dropped, and adds comprehensive unit and integration tests to verify the metrics recording behavior. The review feedback suggests extending the stream auto-trait assertions to explicitly verify the Sync trait, ensuring compatibility with thread-safe types.

Comment thread src/spanner/src/server_streaming/stream.rs Outdated
@olavloite
olavloite force-pushed the spanner-streaming-resultset-metrics branch from 3e80e52 to 499440c Compare August 7, 2026 13:51
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements attempt and operation metrics tracking for Spanner streaming RPCs (ExecuteStreamingSql and StreamingRead), including parsing GFE/AFE latencies from response headers and recording metrics on completion or failure. While the implementation is well-tested, the review feedback highlights a critical blind spot: initial stream creation failures (when the initial send().await fails before a ResultSet is instantiated) are currently not recorded in the metrics. To ensure complete observability, these initial failure paths in both batch_read_only_transaction.rs and the retry macro in read_only_transaction.rs should be updated to record attempt and operation metrics.

Comment thread src/spanner/src/read_only_transaction.rs Outdated
Comment thread src/spanner/src/batch_read_only_transaction.rs Outdated
Comment thread src/spanner/src/batch_read_only_transaction.rs Outdated
@olavloite
olavloite force-pushed the spanner-streaming-resultset-metrics branch from 499440c to b471d09 Compare August 10, 2026 05:51
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces comprehensive observability improvements to the Spanner client, specifically by adding support for recording GFE and AFE latencies via server-timing headers and normalizing gRPC method names for metrics. Key changes include updating ResultSet to track attempt and operation durations, refactoring retry logic to capture these metrics, and adding extensive test coverage for the new observability features. The reviewer provided valuable feedback on reducing code duplication by extracting shared logic into helper functions for both the streaming RPC execution and error handling, as well as suggestions for simplifying match arm logic in the metrics module.

Comment thread src/spanner/src/batch_read_only_transaction.rs Outdated
Comment thread src/spanner/src/observability/metrics.rs
Comment thread src/spanner/src/read_only_transaction.rs
@olavloite
olavloite force-pushed the spanner-streaming-resultset-metrics branch from b471d09 to 3740e16 Compare August 10, 2026 09:32
@olavloite

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements built-in metrics collection for Spanner streaming operations, including ExecuteStreamingSql, StreamingRead, and BatchWrite. It updates ResultSet and the server streaming builders to capture response headers (specifically parsing 'server-timing' for GFE and AFE latencies) and track attempt and operation durations. Additionally, it introduces gRPC method name normalization, disables metrics for plaintext endpoints, and adds comprehensive unit and integration tests to verify metrics recording under various success, retry, and failure scenarios. I have no further feedback to provide as the implementation is robust and well-tested.

@olavloite
olavloite requested review from sakthivelmanii and removed request for sakthivelmanii August 10, 2026 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: spanner Issues related to the Spanner API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant