How to switch model provider per round without recreating Agent?

[1448376744 (soul-soft)]

Hi team,

I'm trying to implement a feature where I need to switch between different model providers (e.g., OpenAI, Azure, Anthropic, etc.) before each round of conversation, rather than using a fixed model throughout the entire session.

My initial thought was to create a new AIAgent instance per request, each with its own IChatClient pointing to the desired provider/model. However, I'm concerned about the overhead—creating a new Agent each time feels heavy, especially if I need to do this frequently (e.g., for every user message in a chat session).

Is there a more lightweight way to achieve this in MAF?

Specifically, I'm looking for something like:

Dynamically changing the IChatClient or ModelId per request without rebuilding the entire Agent pipeline.

Or some middleware/interceptor pattern that allows me to intercept each request and route it to a different underlying model client.

I've looked into ConfigureOptionsChatClient for switching ModelId, but that seems to work only within the same provider client. If I want to switch between different providers (e.g., OpenAI vs. Anthropic), do I need a different approach? Maybe a custom IChatClient wrapper that acts as a router?

Any guidance or code examples would be greatly appreciated!

Thanks in advance.

[1448376744 (soul-soft)]

Does ChatClientAgentRunOptions.ChatClientFactory break the existing pipeline?

[Casper Kristiansson (CasperKristiansson)]

ChatClientFactory = _ => otherClient can change the pipeline, yes. The factory receives the agent's already-configured client, so returning a raw provider client discards the decorators around the original one. You would need to build every replacement client with the same middleware yourself.

The cleaner fit for this is RoutePersistingRoutingChatClient, which was merged just after the 1.18.0 release. Put the provider clients behind that router, then create the agent once from the router:

var router = new RoutePersistingRoutingChatClient(
    new Dictionary<string, IChatClient>
    {
        ["openai"] = openAiClient,
        ["azure"] = azureClient,
        ["anthropic"] = anthropicClient,
    });

var agent = router.AsAIAgent(new ChatClientAgentOptions
{
    ChatHistoryProvider = new InMemoryChatHistoryProvider(),
});

var session = await agent.CreateSessionAsync();

router.SetActiveRoute(session, providerForThisTurn);
var reply = await agent.RunAsync(message, session);

That keeps one agent and one session. The agent's normal middleware wraps the router once, so shared agent-level decorators stay in place. The selected route is stored in session state, and you can change it before every turn.

[1448376744 (soul-soft)]

Thank you so much for the detailed and incredibly timely response!

This is exactly what I was looking for. The RoutePersistingRoutingChatClient seems to be a perfect fit for my requirement to switch providers per round while keeping the agent and session intact. I also appreciate you pointing out the pitfall of using ChatClientFactory and losing the existing pipeline decorators.

I have two quick follow-up questions, if you don't mind:

Release Timeline: Since this PR was merged just after the 1.18.0 release, do you have an estimate for when it will be included in an official stable NuGet package? (e.g., in the upcoming 1.19.0 release).

Temporary Workaround: Is there a recommended workaround or a way to use this feature now (like using a nightly/preview build or a specific source code reference) while waiting for the official release?

Thanks again for your help on this!

[Ritikesh Nimje (Hrithikesh-Computer)]

Following up on the context question above: there's a related but distinct problem once routing happens mid-session capability negotiation.

The agent/session is created once, but SetActiveRoute can select a provider with a different capability set. For example, suppose the current route supports structured output + tool calling + vision, while the next route doesn't support one of those.

A few questions about the intended contract:

If a selected route lacks a capability the agent/request requires (e.g. structured output), is the expectation that the router rejects the route, the caller performs the capability check, or the request is simply passed through and allowed to fail at the provider?
Is there a capability declaration on IChatClient that a routing layer can inspect before selecting a route, or is capability compatibility entirely an application-level concern?
Are capabilities expected to be negotiated per turn, or are they effectively assumed to be stable for the lifetime of the Agent/Session?
For tools specifically, if one provider supports a tool/function-calling contract that another provider doesn't, is the application expected to maintain a provider-compatible tool configuration for every possible route?

The underlying question for me is whether "same Agent + same Session + different provider" is intended to mean interchangeable execution, or whether the abstraction assumes that callers only route between providers with compatible capability profiles.

If it's the latter, that seems like an important constraint to make explicit, because routing between providers with similar capabilities can work perfectly in testing while a route change across a capability boundary fails only at runtime.