feat(rest): add SigV4 auth manager for the REST catalog

[plusplusjiajia]

Which issue does this PR close?

Follow-up to #2838, split from the #2815 prototype per review.

What changes are included in this PR?

SigV4 as a SigV4AuthManager wrapping a delegate auth manager, per #2838's design:

• The delegate authenticates first (e.g. an OAuth2 bearer); its Authorization is relocated to Original-Authorization (Java convention) and included in the signature, then the request is SigV4-signed.
• SigV4Signer follows AWS canonical-request rules with Java parity (path normalization + double encoding, all-headers-minus-blacklist, IcebergRest/StandardAws payload-hash modes).
• Properties match Java's AwsProperties (rest.signing-region, rest.signing-name default execute-api, rest.access-key-id/rest.secret-access-key/rest.session-token, single-source credential resolution). Delegate defaults to oauth2; an injected signer is never rebuilt from server properties.

Three behavior changes worth calling out, all for Java parity:

• Authentication is applied after extra headers so a SigV4 session signs the final header set. A configured header.authorization still wins, because the OAuth2 session now only sets the header when absent — as Java's OAuth2Util.AuthSession does with putIfAbsent.
• A signer-generated header that conflicts with a caller's (x-amz-date, x-amz-content-sha256, x-amz-security-token) is relocated to Original-* rather than dropped, matching Java's updateRequestHeaders. Relocated values are marked sensitive.
• The default client does not follow redirects when SigV4 is configured or an auth manager is injected: a signed request can't be transparently re-followed, and reqwest's cross-origin strip list doesn't know Original-Authorization or x-amz-security-token.

[dannycjones]

There's a PR open for SigV4 signing, is this picking up from that one? #2311

I ask as its had a few rounds of feedback already.

[plusplusjiajia]

There's a PR open for SigV4 signing, is this picking up from that one? #2311

I ask as its had a few rounds of feedback already.

@dannycjones Thanks for the pointer — I'd missed #2311, just took a look and compared the two. Mine isn't based on it: it follows Iceberg Java's RESTSigV4AuthSessionRESTSigV4AuthSession(apache/iceberg#11995) and the merged iceberg-cpp version(apache/iceberg-cpp#616). The main difference I see is the base64-encoded x-amz-content-sha256 convention (the Java behavior), which #2311's hex-only signing doesn't cover and which some REST servers require.
I'm keen to help get SigV4 support landed either way. Since #2311 is further along, I'm happy to fold the base64 support into it rather than duplicate — or carry this one forward if you'd prefer. Whatever helps move it forward.

[CTTY]

I haven't thought quite clearly about this part yet, my general intuition is that it would be better to start from something like a AuthManager so we have a clean interface before diving into the specific implementation.

Would be happy to hear more thoughts on this

[CTTY]

I think we need a more general design rather than just a signer. some authentication mechanism is token-based.

[plusplusjiajia]

@CTTY I dug into how Java structures this, and I think your point is well taken: OAuth2 token handling is hardcoded inside HttpClient today, and this PR adds a second, parallel mechanism that is mutually exclusive with token auth. In Java the two compose — SigV4AuthManager wraps a delegate session, relocates its Authorization header to X-Iceberg-Authorization, then signs — so SigV4-over-OAuth2 is a real combination the current design can't express.

Is something along these lines what you had in mind, mirroring Java's AuthManager/AuthSession?

#[async_trait]
pub trait AuthManager: Debug + Send + Sync {
    /// Session used for catalog-level requests.
    async fn catalog_session(
        &self,
        props: &HashMap<String, String>,
    ) -> Result<Arc<dyn AuthSession>>;
    // room to grow, matching Java: init_session() for the config
    // handshake, table_session() for table-scoped auth, close().
}

#[async_trait]
pub trait AuthSession: Debug + Send + Sync {
    /// Applies authentication to an outgoing request (headers, signing, ...).
    async fn authenticate(&self, request: &mut reqwest::Request) -> Result<()>;
}

with NoopAuthManager / OAuth2Manager (existing logic extracted, behavior unchanged) / SigV4AuthManager (wrapping a delegate, Java-style) as the initial implementations, selected via rest.auth.type or injected through the builder.

If that matches your intuition, my instinct would be to land the interface plus the OAuth2 extraction as a small standalone refactor first, then rework this PR on top as the SigV4 implementation — which would also give #2311 a common landing spot.

[plusplusjiajia]

Prototype is up as a draft PR: #2815 — full AuthManager/AuthSession shape with Noop/OAuth2/SigV4 managers; details and known simplifications in the PR description.

[CTTY]

I didn't know this detail until this PR. Thanks for capturing this!

[plusplusjiajia]

Update: this PR will be reworked in place as a SigV4AuthManager on top of #2838 (implementation ready locally, tracking that PR's review). Will push once #2838 merges.

[plusplusjiajia]

Pushed the rework as planned — stacked on #2838 for now (see the note in the description; only the last commit is this PR). Will rebase to a single commit once #2838 merges.