chore(gax-internal): add attempt start and complete lifecycle hooks to AttemptInterceptor

[olavloite]

Extend the internal AttemptInterceptor trait with on_attempt_start and on_attempt_complete callbacks to allow tracking per-attempt latency, metadata, and response headers across unary gRPC attempts.

Bump google-cloud-gax-internal version to 0.7.17.

Needed for #6311

[gemini-code-assist]

Code Review

This pull request bumps the version of google-cloud-gax-internal to 0.7.17 and introduces new lifecycle hooks, on_attempt_start and on_attempt_complete, to the AttemptInterceptor trait. These hooks enable tracking of unary RPC attempt durations, response headers, and outcomes. The gRPC Client has been updated to invoke these hooks, and comprehensive unit and integration tests have been added to verify the new behavior. Additionally, several .unwrap() calls in tests were replaced with .expect() to align with the style guide. I have no feedback to provide.

[codecov]

Codecov Report

❌ Patch coverage is 99.57082% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 96.28%. Comparing base (516e38e) to head (32ce2f8).

Files with missing lines	Patch %	Lines
src/gax-internal/src/attempt_interceptor.rs	99.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6315      +/-   ##
==========================================
+ Coverage   96.27%   96.28%   +0.01%     
==========================================
  Files         283      283              
  Lines       73110    73313     +203     
==========================================
+ Hits        70384    70587     +203     
  Misses       2726     2726              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[coryan]

to allow tracking per-attempt latency,

We already have a way to do that.... Look at the implementation of transport-level metrics. This may be a good place to start:

https://github.com/googleapis/google-cloud-rust/blob/main/src/gax-internal/src/observability/client_signals/with_transport_metric.rs

Can you examine the other approach (maybe you already did) and let us know why it does not work in your case?

[olavloite]

to allow tracking per-attempt latency,

We already have a way to do that.... Look at the implementation of transport-level metrics. This may be a good place to start:

https://github.com/googleapis/google-cloud-rust/blob/main/src/gax-internal/src/observability/client_signals/with_transport_metric.rs

Can you examine the other approach (maybe you already did) and let us know why it does not work in your case?

Yeah, looked into that possibility (sorry, I should have added that directly to the PR description). The short answer is 'While not utterly impossible, it would require major changes to WithTransportMetric'. The main reasons are:

1. For Spanner, we need to capture some relatively specific metrics and also include header values that are returned by Spanner, so we can compare end-to-end latency with the latency that was measured by the GFE. This allows us to make a reasonable judgement call on whether end-to-end latency is coming from Spanner (i.e. the GFE is also reporting high latency), or whether the end-to-end latency is caused by something 'after' the GFE (network, client, ...).
2. We need to capture metrics per attempt, and not only per overall RPC. Meaning: We need a hook into the gRPC retry loop.

---

Or to quote a verbose friend of mine:

1. Why WithTransportMetric Cannot Be Used As-Is

WithTransportMetric and TransportMetric are designed specifically for standard Google Cloud Client Signals (T4 transport signals):

1. Hardcoded Metric Definition:
   • TransportMetric is hardcoded to emit gcp.client.attempt.duration in seconds (s) with 16 fixed bucket boundaries to the global opentelemetry::global::meter_provider().
   • Spanner Built-In Metrics requires emitting 8 distinct metrics (attempt_latencies, operation_latencies, gfe_latencies, afe_latencies, attempt_count, operation_count, gfe_connectivity_error_count, afe_connectivity_error_count) in milliseconds (ms), with Spanner's explicit 50-bucket distribution (0.0ms to 3,200,000.0ms), under a custom monitored resource (spanner_instance_client).
2. Missing Response Headers in RequestRecorder:
   • Spanner Built-In Metrics requires measuring GFE and AFE latencies and connectivity error counts by parsing the server-timing headers returned in gRPC response metadata.
   • WithTransportMetric reads from RequestRecorder (tokio::task_local! storing TransportSnapshot), which captures URL, status code, and duration, but does not capture gRPC response metadata/headers on successful attempts.

---

2. What Would It Take to Adapt WithTransportMetric?

If we tried to route Spanner Built-In Metrics through WithTransportMetric / RequestRecorder, it would require significant architectural changes across GAX:

1. Refactoring RequestRecorder:
   • Extend RequestRecorder and TransportSnapshot in gax-internal to capture and store gRPC response HeaderMap metadata on every attempt.
   • Update grpc.rs to extract response headers from tonic::Response on every attempt and write them into the task-local RequestRecorder.
2. Pluggable Metric Emitters in GAX:
   • WithTransportMetric currently only calls TransportMetric.with_recorder_ok() and with_recorder_error().
   • It would need to be generalized to support custom metric listeners or service-specific metric pipelines.
3. Retry Loop Visibility:
   • Intermediate attempt results are encapsulated within grpc::Client::execute. To notify Spanner of intermediate attempt outcomes (for attempt-level metric recording), GAX still needs to provide an attempt-level hook/callback during the retry loop.

---

Summary Comparison

Requirement	WithTransportMetric / RequestRecorder	AttemptInterceptor Lifecycle Hooks
Response Headers (server-timing)	❌ Not captured in TransportSnapshot	✅ Directly passed as Option<&HeaderMap>
Spanner Metric Suite (8 instruments)	❌ Hardcoded to gcp.client.attempt.duration	✅ Delegated directly to Spanner's Observability
Custom Monitored Resource (spanner_instance_client)	❌ Hardcoded to generic OTel resource conventions	✅ Managed by Spanner's SdkMeterProvider
Intermediate Attempt Notification	❌ Encapsulated inside GAX retry loop	✅ Direct callback per attempt