chore(spanner): [WIP] implement built-in attempt metrics and server-timing recording

[olavloite]

Do not review: Work in progress, the Gax changes will be split into a separate pull request.

• Implement SpannerMetricsInterceptor to record attempt latencies, attempt count, and server-timing metrics (GFE and AFE latencies) on RPC attempt completion.
• Inject x-goog-spanner-enable-afe-server-timing request header on outgoing attempts, respecting SPANNER_DISABLE_AFE_SERVER_TIMING.
• Normalize the method metric attribute to "Spanner." with zero-allocation static matching across operation and attempt metrics.
• Fix GFE and AFE connectivity error recording to prevent false positive AFE errors on non-DirectPath traffic.
• Disable Cloud Monitoring exporter initialization for plaintext mock/test endpoints, Omni, and emulator environments.

[gemini-code-assist]

Code Review

This pull request extends the AttemptInterceptor trait with lifecycle hooks (on_attempt_start and on_attempt_complete) to track unary RPC attempts, and integrates these hooks into the Spanner client to collect experimental builtin metrics (such as GFE/AFE latencies and connectivity errors). Feedback on the changes highlights an opportunity to avoid unnecessary double indirection (Arc<Vec<...>>) when metrics are disabled, as well as a bug in the method name normalization fallback logic when handling custom methods prefixed with Spanner/.

[codecov]

Codecov Report

❌ Patch coverage is 97.55435% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.16%. Comparing base (516e38e) to head (2ed3509).

Files with missing lines	Patch %	Lines
src/spanner/src/observability/metrics.rs	96.57%	17 Missing ⚠️
src/gax-internal/src/attempt_interceptor.rs	99.30%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6311      +/-   ##
==========================================
- Coverage   96.27%   96.16%   -0.11%     
==========================================
  Files         283      283              
  Lines       73110    73733     +623     
==========================================
+ Hits        70384    70905     +521     
- Misses       2726     2828     +102     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[olavloite]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces lifecycle hooks (on_attempt_start and on_attempt_complete) to the AttemptInterceptor trait in the GAX library, allowing unary RPC attempts to be tracked. These hooks are integrated into the gRPC client and utilized in the Spanner client to collect detailed attempt-level metrics, such as GFE/AFE latencies and connectivity errors, when the _experimental-builtin-metrics feature is enabled. Feedback on the changes suggests simplifying the fallback pattern matching in normalize_method_name by utilizing the already-cleaned method name prefix instead of the raw trimmed string.

[olavloite]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces lifecycle hooks (on_attempt_start and on_attempt_complete) to the AttemptInterceptor trait to track unary RPC attempts, and integrates them into the gRPC client and Spanner metrics interceptor to record attempt-level metrics. Feedback focuses on optimizing the gRPC client's on_attempt_complete method to take and return the result by value, enabling zero-allocation header extraction by destructuring the Response instead of cloning its metadata. Additionally, it is recommended to document a limitation in the composite Vec interceptor where custom Instant values returned by individual interceptors are discarded.

[olavloite]

/gemini review

[gemini-code-assist]

Code Review

This pull request enhances the AttemptInterceptor trait in gax-internal with on_attempt_start and on_attempt_complete lifecycle hooks to enable detailed tracking of unary RPC attempts, and integrates these hooks into the spanner crate via a new SpannerMetricsInterceptor to record GFE/AFE latencies and connectivity errors. The review feedback correctly identifies a bug where GFE connectivity errors would be falsely recorded on successful RPCs if the server-timing header is missing; the reviewer suggests restricting this metric to failed attempts and provides a corresponding fix for the associated unit test.