chore(spanner): support built-in metrics for ResultSet streaming RPCs - #6302
chore(spanner): support built-in metrics for ResultSet streaming RPCs#6302olavloite wants to merge 2 commits into
Conversation
Adds OpenTelemetry built-in metrics instrumentation for streaming operations (`ExecuteStreamingSql` and `StreamingRead`) in Cloud Spanner `ResultSet`. - Records `attempt_count`, `attempt_latencies`, `gfe_latencies`, and `afe_latencies` across all initial attempts and stream restarts. - Records end-to-end `operation_count` and `operation_latencies` spanning the entire query lifecycle until stream completion or permanent failure. - Preserves attempt response headers across background stream trailer drains (`seen_last`) and early drops. - Standardizes metric `method` attribute formatting to `"Spanner.<MethodName>"` (`Spanner.ExecuteStreamingSql`, `Spanner.StreamingRead`) matching Java and Go reference clients. - Parses single and multiple `server-timing` response headers for GFE and AFE timings.
There was a problem hiding this comment.
Code Review
This pull request introduces attempt and operation metrics tracking for Spanner streaming queries and reads, including parsing server-timing headers for GFE/AFE latencies and normalising method names. The review feedback highlights several critical gaps in error handling and metrics accuracy: stream initialization failures are currently not recorded (and incorrectly reported as successful upon drop), stale headers from previous attempts are not cleared during retries, and complete connection failures prior to ResultSet creation are missed entirely by the metrics recorder.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #6302 +/- ##
==========================================
- Coverage 96.26% 96.15% -0.11%
==========================================
Files 282 282
Lines 72762 73739 +977
==========================================
+ Hits 70044 70905 +861
- Misses 2718 2834 +116 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
7f24278 to
3e80e52
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements built-in metrics collection for Spanner streaming operations, tracking attempt and operation counts, latencies, and GFE/AFE latencies parsed from response headers. It updates ResultSet to record these metrics during stream initialization, execution, retries, and upon being dropped, and adds comprehensive unit and integration tests to verify the metrics recording behavior. The review feedback suggests extending the stream auto-trait assertions to explicitly verify the Sync trait, ensuring compatibility with thread-safe types.
3e80e52 to
499440c
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements attempt and operation metrics tracking for Spanner streaming RPCs (ExecuteStreamingSql and StreamingRead), including parsing GFE/AFE latencies from response headers and recording metrics on completion or failure. While the implementation is well-tested, the review feedback highlights a critical blind spot: initial stream creation failures (when the initial send().await fails before a ResultSet is instantiated) are currently not recorded in the metrics. To ensure complete observability, these initial failure paths in both batch_read_only_transaction.rs and the retry macro in read_only_transaction.rs should be updated to record attempt and operation metrics.
499440c to
b471d09
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces comprehensive observability improvements to the Spanner client, specifically by adding support for recording GFE and AFE latencies via server-timing headers and normalizing gRPC method names for metrics. Key changes include updating ResultSet to track attempt and operation durations, refactoring retry logic to capture these metrics, and adding extensive test coverage for the new observability features. The reviewer provided valuable feedback on reducing code duplication by extracting shared logic into helper functions for both the streaming RPC execution and error handling, as well as suggestions for simplifying match arm logic in the metrics module.
b471d09 to
3740e16
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements built-in metrics collection for Spanner streaming operations, including ExecuteStreamingSql, StreamingRead, and BatchWrite. It updates ResultSet and the server streaming builders to capture response headers (specifically parsing 'server-timing' for GFE and AFE latencies) and track attempt and operation durations. Additionally, it introduces gRPC method name normalization, disables metrics for plaintext endpoints, and adds comprehensive unit and integration tests to verify metrics recording under various success, retry, and failure scenarios. I have no further feedback to provide as the implementation is robust and well-tested.
Adds OpenTelemetry built-in metrics instrumentation for streaming operations (
ExecuteStreamingSqlandStreamingRead) in Cloud SpannerResultSet.attempt_count,attempt_latencies,gfe_latencies, andafe_latenciesacross all initial attempts and stream restarts.operation_countandoperation_latenciesspanning the entire query lifecycle until stream completion or permanent failure.seen_last) and early drops.methodattribute formatting to"Spanner.<MethodName>"(Spanner.ExecuteStreamingSql,Spanner.StreamingRead) matching Java and Go reference clients.server-timingresponse headers for GFE and AFE timings.