feat(core-amqp): refresh CBS tokens before they expire

[Johnathan W (j7nw4r)]

Summary

The AMQP connection cached a CBS access token for each audience and never looked at it again. A cache hit returned the stored token with no expiry test, and the store used emplace, which does not overwrite. A client that outlived one token lifetime, about 60 to 90 minutes for Microsoft Entra ID, received an amqp:unauthorized-access detach and could not recover without a process restart.

This change is the token part of #7254. The link rebuild, the receiver rebuild, and the retry classification stay out of it.

Motivation

ConnectionImpl::AuthenticateAudience returned a cached token with no test of ExpiresOn, so authentication was a one time event for the life of the connection. The .NET client refreshes with a seven minute buffer (AmqpConnectionScope.cs:73,736); the C++ client had no equivalent.

The failure needed no fault on the network, and it is terminal on the first attempt. EventHubsExceptionFactory::IsErrorTransient does not list amqp:unauthorized-access, so the producer treats an expired token as not transient and stops instead of retrying.

Changes

Three layers, all in azure-core-amqp.

The cache now refuses a token with less than 30 seconds of life left and authenticates again. The store assigns instead of emplace. Both transports compile this part.

A refresh thread replaces each token seven minutes before it expires, behind #if ENABLE_UAMQP. It releases the token mutex for the credential call and the CBS operation, since both go to the network. It keeps a weak pointer to the session and promotes it only for one refresh. It discards its result when a caller stored a newer token, drops the cache entry on failure, and keeps a minimum interval of 20 seconds between passes so a short lived token cannot make it spin. A second narrow mutex serializes the CBS operation itself, because uAMQP names CBS links after the node and two attaches with one name violate AMQP 1.0 section 2.6.1. Callers take the token mutex then that one; the thread takes that one only while it holds no token mutex, so the order stays acyclic.

An ownership rule makes the thread safe to shut down. A session holds its connection, so the promoted pointer can be the last reference to both, and releasing it ran both destructors on the refresh thread. That gave two failures: on the normal path the destructor detached the thread and then destroyed the token mutex, which the detached thread locked on the next line; on the early return the pointer died under the token mutex, which the destructor then locked from the thread already holding it. The destructor asserts catch neither, because Event Hubs over uAMQP never calls Connection::Open or Connection::Close, both being inside #if ENABLE_RUST_AMQP.

The fix moves the token mutex, the condition variable, the stop flag, and the two token maps into a TokenRefreshState block that the connection and the thread co-own through a shared_ptr, so the state outlives a connection that dies under the thread. A ReleaseOutsideLock holder releases the promoted session with the lock free on every exit path, including the early return and an exception. The thread body is static and takes a raw connection pointer, and the refresh returns true when a release can have destroyed the connection, so the thread leaves without touching it again.

Three behaviours are not visible in the diff. The destructor is the only shutdown path on the Event Hubs uAMQP configuration, so there is no Close path to review. A caller authenticating a new audience can wait behind an in flight refresh, bounded at 60 seconds by the refresh context. The thread does not restart after an unexpected error; the near expiry test on the cache is the fallback.

One scaling note: ProducerClient creates one connection for each partition ID, so a 32 partition producer runs 33 connections and 33 refresh threads. Each carries its own mutexes, so partitions never queue behind each other.

Test plan

Unit tests in connection_tests.cpp cover the expiry rules, including a default constructed ExpiresOn and a year 9999 value, which must not throw. Nine more cover the ownership rule: seven drive the ReleaseOutsideLock holder across release, restore, destructor, exception, repeat release, and shared ownership; two cover the state block.

Five _LIVEONLY_ tests in producer_client_test.cpp run against a real namespace, with a credential decorator that reports a short life for a real token so a refresh happens in seconds. They cover a refresh that reaches the service, continuity past two token lifetimes, a clean shutdown, no repeat requests for a normal token, and eight threads calling across a refresh. Together they add about eight minutes to a live run.

Both ownership failures were reproduced before the fix with a throwaway harness that forces the interleaving without a network. Each hung for 15 seconds and each passes after the fix. AddressSanitizer reports nothing, because the faulting call sits inside an uninstrumented libc function.

Validation

• azure-core-amqp unit tests, uAMQP configuration: 129 of 129 pass.
• The Event Hubs library and test target compile against the changed private headers.
• clang-format 11 is clean on every file touched.
• Live tests against a real Event Hubs namespace, run against the ownership fix: 5 of 5 pass, in 331 seconds.

Follow up

The cleanup close in PutTokenForAudience uses the caller's context. A cancelled context makes MessageSenderImpl::Close throw before it clears the management client's open flag, and that client stops the process when destroyed. Only direct users of Connection::Close over uAMQP reach it, which excludes Event Hubs. A fresh context with a short deadline fixes it; a bare uncancellable one does not, because MessageSenderImpl never completes its close queue when a sender enters the error state. That missing completion can hang any sender close.

The concurrent call test drives about 40 management operations each second through one client, against a documented limit of 50 for each consumer group, and a throttle arrives as amqp:not-allowed, which is classified as not transient. A slower cadence, a StatusCode check, and a progress watchdog would keep its purpose without the flake.

Mock server tests for the refresh are absent. mock_amqp_server.hpp rejects a second attach on a link name it holds and does not restart its loop after the last detach, so it cannot serve a CBS re-attach. It is shared infrastructure and excluded on macOS, so that belongs in a change validated where those tests run.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
7 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.