Realtime Client Proposal

[tarekgh]

Realtime Client Abstraction Proposal

This proposal outlines a unified abstraction for Realtime model clients within the Microsoft.Extensions.AI.Abstractions library. The goal is to provide a consistent and provider-agnostic interface for interacting with Realtime AI systems, making it easier for developers to integrate, use, and switch between different model implementations.

Realtime models typically operate over bidirectional streaming, enabling clients to send input and receive output concurrently. This allows the model server to begin generating results while still processing incoming data.

Interacting with realtime models usually involves creating a session or connection through which input and output are exchanged as streams. Sessions may maintain state across multiple interactions, enabling richer, more context-aware responses.

Realtime models can accept various types of input, such as text, audio, or images and typically produce output in the form of text or audio.

Proposed Interface

IRealtimeClient

Defined in IRealtimeClient.cs

This is the primary interface for interacting with realtime models. Applications use it to create sessions and manage realtime connections. Below is an example of how application code might look:

// OpenAIRealtimeClient is an example implementation of IRealtimeClient
IRealtimeClient realtimeClient = new OpenAIRealtimeClient(apiKey);

IRealtimeSession

Defined in IRealtimeSession.cs

After creating an IRealtimeClient, you can use it to create an instance of IRealtimeSession, which represents an individual session with the realtime model. A session enables sending input and receiving output as streams. Here’s an example of how the application code might look:

IRealtimeSession realtimeSession = await realtimeClient.CreateSessionAsync();

The application can then send input and receive output through the session using a mechanism similar to the example shown below:

// Create channel for client messages
clientMessageChannel = Channel.CreateUnbounded<RealtimeClientMessage>();
streamingCancellationTokenSource = new CancellationTokenSource();

// Start background task to process server messages
_ = Task.Run(async () =>
{
    try
    {
        await foreach (var serverMessage in realtimeSession.GetStreamingResponseAsync(
            clientMessageChannel.Reader.ReadAllAsync(streamingCancellationTokenSource.Token),
            streamingCancellationTokenSource.Token))
        {
            ProcessServerMessage(serverMessage); // application method to handle server messages
        }
    }
    catch (OperationCanceledException)
    {
        // Expected when cancelling
    }
    catch (Exception ex)
    {
        HandleError(ex); // application method to handle errors
    }
}, streamingCancellationTokenSource.Token);

The application sends messages to the model by creating instances of RealtimeClientMessage and writing them to the clientMessageChannel. RealtimeClientMessage is a base type with specialized derived types representing the different message categories supported by realtime models.

Similarly, the application receives RealtimeServerMessage instances from the model. This is also a base type with multiple derived message types representing messages the server may emit.

The abstraction defines the following common client and server message types, with the expectation that more can be added over time.

Client Message Types

Type	Purpose
RealtimeClientConversationItemCreateMessage	Creates a new conversation item (text, audio, image, etc.).
RealtimeClientInputAudioBufferAppendMessage	Appends audio data to an existing audio buffer in the conversation.
RealtimeClientInputAudioBufferCommitMessage	Commits the appended audio data in the buffer for processing.
RealtimeClientResponseCreateMessage	Requests generation of a new response within the conversation.

Server Message Types

Type	Purpose
RealtimeServerErrorMessage	Represents an error sent by the model server.
RealtimeServerInputAudioTranscriptionMessage	Represents messages related to input audio transcription results.
RealtimeServerOutputTextAudioMessage	Represents generated output text and audio from the model.
RealtimeServerResponseCreatedMessage	Indicates a new response item has been created on the server.

Here are examples of how to send messages to the model:

//
// Send a message to create a new audio conversation item
//
await clientMessageChannel.Writer.WriteAsync(new RealtimeClientInputAudioBufferAppendMessage(
    audioContent: new DataContent($"data:audio/pcm;base64,{Convert.ToBase64String(resampledAudio)}")));
await clientMessageChannel.Writer.WriteAsync(new RealtimeClientInputAudioBufferCommitMessage());

//
// Create conversation item with text
//
var contentItem = new RealtimeContentItem(new TextContent(text), id: null, role: ChatRole.User);
await clientMessageChannel.Writer.WriteAsync(new RealtimeClientConversationItemCreateMessage(item: contentItem));
await clientMessageChannel.Writer.WriteAsync(new RealtimeClientResponseCreateMessage());

//
// Create conversation item with image
//
var contentItem = new RealtimeContentItem(new DataContent($"data:{mimeType};base64,{base64Image}"), id: null, role: ChatRole.User);
await clientMessageChannel.Writer.WriteAsync(new RealtimeClientConversationItemCreateMessage(item: contentItem));
await clientMessageChannel.Writer.WriteAsync(new RealtimeClientResponseCreateMessage());

Here are examples of receiving messages from the model:

private void ProcessServerMessage(RealtimeServerMessage serverMessage)
{
    Invoke(new Action(() =>
    {
        switch (serverMessage)
        {
            case RealtimeServerOutputTextAudioMessage audioMessage:
                if (audioMessage.Type == RealtimeServerMessageType.OutputAudioDelta && audioMessage.Text != null)
                {
                    // PlayAudioChunk(audioMessage.Text);
                }
                else if (audioMessage.Type == RealtimeServerMessageType.OutputAudioTranscriptionDelta && audioMessage.Text != null)
                {
                    // WriteTranscript(audioMessage.Text);
                }
                else if (audioMessage.Type == RealtimeServerMessageType.OutputAudioTranscriptionDone)
                {
                    // output transcription is done
                }
                break;

            case RealtimeServerInputAudioTranscriptionMessage transcriptionMessage:
                if (transcriptionMessage.Type == RealtimeServerMessageType.InputAudioTranscriptionCompleted &&
                    transcriptionMessage.Transcription != null)
                {
                    // WriteUserInputTranscription($"You: {transcriptionMessage.Transcription}\n");
                }
                break;

            case RealtimeServerErrorMessage errorMessage:
                // WriteError($"Error: {errorMessage.Error?.Message}");
                break;

            case RealtimeServerResponseCreatedMessage responseMessage:
                if (responseMessage.Usage != null)
                {

                    // WriteUsage($"Usage - Input: {responseMessage.Usage.InputTokenCount}, Output: {responseMessage.Usage.OutputTokenCount}\n");
                }
                break;
        }
   }));
}

Important Notes

• The abstraction includes a mechanism for updating session settings after creation. This is done through the UpdateAsync method on the IRealtimeSession interface. The RealtimeSessionOptions type represents the updatable settings, and providers may extend this type to include provider-specific options.
• The abstraction is designed to be extensible. Clients can create RealtimeClientMessage instances that contain raw data for scenarios not covered by the predefined message types. The RealtimeClientMessage.RawRepresentation property supports this behavior.
• Likewise, servers can send RealtimeServerMessage instances with raw data for scenarios outside the predefined message types. Applications can access this through the RealtimeServerMessage.RawRepresentation property.
• Several additional types were introduced, such as LogProbability, NoiseReductionOptions, PromptTemplate, RealtimeAudioFormat, SemanticVoiceActivityDetection, ServerVoiceActivityDetection, ToolChoiceMode, and TranscriptionOptions. If we choose to keep some or all of these, they may need to be moved into more appropriate folders. For now, they are grouped together to simplify review.
• The proposal contains the types RealtimeClientMessageType and RealtimeServerMessageType enumerations to categorize the different message types. I am on the fence about including these; they may not be necessary.
• Many implementation details across these types can be discussed during the review process.
• Naming is often challenging; the current names reflect initial choices and can be revised as needed.
• XML documentation has been added for public types and members, though there is room for refinement.
• An example implementation based on this abstraction is available in the OpenAIRealtimeClient file. This is not part of the proposal but serves to validate the design. The implementation is not fully complete but supports the major scenarios. A demo application is also provided to illustrate practical usage.
• This proposal does not address related topics such as delegate wrappers, dependency injection extensions, or telemetry support. These can be added later after the core abstraction is finalized.
• As I am still ramping up in this area, there may be gaps or missing details in the proposal. I would greatly appreciate any feedback or suggestions to help refine and improve it.

[ericstj]

It seems very intentional that Chat is omitted from the name. Is this consistent with the way the models/providers are exposing real-time support?

[tarekgh]

Right, I don't think the Chat term is used much with the realtime models. Here is snippet example of one of the python's implemenatin:

from openai_realtime import RealtimeClient

client = RealtimeClient(api_key="...", model="gpt-realtime")
client.send_text("Hello")
for event in client:
    print(event)

[ericstj]

Elsewhere we call this MediaType

[tarekgh]

Just wondering if the MediaType is broader and not scoped to audio formats?

[ericstj]

This all looks very different from the existing chat client APIs, is it necessarily so? I was imagining there would be an onramp from existing APIs to real-time. As it is, it seems a complete rewrite?

[tarekgh]

This all looks very different from the existing chat client APIs, is it necessarily so? I was imagining there would be an onramp from existing APIs to real-time. As it is, it seems a complete rewrite?

Yes, it is different because the chat client is designed around a traditional request–response model. As mentioned in the description, realtime models operate over bidirectional streaming, allowing clients to send input and receive output at the same time. This enables the server to start generating results while it is still processing incoming data.

For example, Realtime models support Voice Activity Detection (VAD), which can determine when to begin responding even before the entire audio stream is received. Think of it as a natural voice conversation: you can interrupt the model while it is speaking, and the server can start responding as soon as it detects a brief pause in your speech.

If you have a better approach for supporting Realtime models, I’d be happy to explore it.

[ericstj]

I'm just trying to think about how this fits into existing samples / templates and how we teach people about it. It sounds like it's not a "grow into" type of technology but instead an "early fork".

If an early fork, then I'd still expect most of the different tasks we already have to be possible in real-time. I wonder if it would help to look at a table of the tasks, which types are involved for Chat and which types are involved for Realtime. In some cases maybe we can use similar or same types (Options, Content?), but in others we might need new types. I would expect Realtime to be a superset of functionality in most cases. Is it ever possible to derive from those existing types?

Additionally - we have infrastructure around IChatClient - like function calling - how does that work in RealTime and can we reuse the infrastructure we already have?

[tarekgh]

In some cases maybe we can use similar or same types (Options, Content?), but in others we might need new types

That’s exactly the approach I followed in the proposal. I reused existing types wherever they naturally fit, such as ErrorContent, AITool, AIContent, UsageDetails, etc. I also introduced several types that can apply to both chat and realtime models, including LogProbability, PromptTemplate, ToolChoiceMode, and TranscriptionOptions. We can certainly decide to delay adding some of these, but I’ve noticed that newer OpenAI models like GPT-5.1 are already using similar concepts, so it felt appropriate to include them.

The proposal also defines realtime-specific types, such as VoiceActivityDetection, ServerVoiceActivityDetection, SemanticVoiceActivityDetection, and RealtimeSessionOptions, to cover functionality unique to realtime scenarios.

Additionally - we have infrastructure around IChatClient - like function calling - how does that work in RealTime and can we reuse the infrastructure we already have?

I believe the use of AITool and function calls should work the same way with Realtime models. For function calls, the process ultimately extracts the function name and parameters from our types, and then sends that information to the model.

[ericstj]

I believe the use of AITool and function calls should work the same way with Realtime models. For function calls, the process ultimately extracts the function name and parameters from our types, and then sends that information to the model.

Can you have a look at how function calling works today in https://github.com/dotnet/extensions/blob/e7fac9d9885b12ea2aacf75875802cc4571ee2ca/src/Libraries/Microsoft.Extensions.AI/ChatCompletion/FunctionInvokingChatClient.cs. We have these and other infrastructure built up around IChatClient - are we going to need parallel types around IRealtimeSession?

[tarekgh]

Can you have a look at how function calling works today in https://github.com/dotnet/extensions/blob/e7fac9d9885b12ea2aacf75875802cc4571ee2ca/src/Libraries/Microsoft.Extensions.AI/ChatCompletion/FunctionInvokingChatClient.cs. We have these and other infrastructure built up around IChatClient - are we going to need parallel types around IRealtimeSession?

I’ll look into that. My initial thought is that we shouldn’t need a parallel infrastructure; function calls could be handled like any other conversation item (similar to text, audio, etc.). The client would initiate a function call through a client message using FunctionCallContent, and the model would return the results in a server message. That said, I’m not an expert yet, so I may be overlooking something. I’ll spend more time reviewing this area.

[tarekgh]

Here are the open issues currently being tracked in the proposal:

• Supporting similar mechanism as FunctionInvokingChatClient with the ChatClient. Look more to the other tools like MCP in general too. Note, Realtime models are full duplex streaming which differ than ChatClient in this matter.
• Need to have DelegatingRealtimeSession?
• Do we need a Realtime builder as we have it with the ChtClientBuilder? should the builder be for the Session instead?
• For consistency, use IList instead of IEnumerable in the places like IEnumerable or similar places.
• Add a server message corresponds to response.output_item.added which can be used to handle the tool calls.
• Should we have object? GetService(Type serviceType, object? serviceKey = null); on IRealtimeSession?

I’ll continue adding items to the list as we explore the proposal in more depth.

[stephentoub]

Have you tried implementing this on multiple providers?

[stephentoub]

Why is this needed? How does it get used?

[tarekgh]

I've removed it. Initially, I thought it might be useful if having a session object to check the connection status, but I've removed it for now until we find a need for it.

[stephentoub]

Can I call this multiple times on the same IRealtimeSession instance?

[tarekgh]

No, this method cannot be called concurrently on the same session instance. The provider's implementation should throw an exception if multiple calls are attempted. However, calling it sequentially should be fine, though I don't anticipate this being a common use case. I have added a remark to the docs for that.

[stephentoub]

Is this something that developers need in the 90% case? Is this generic across all providers?

[tarekgh]

This is typically used by AI engineers for guardrails and testing. It helps with confidence scoring or probability distribution between tokens. I’m going to remove it from the proposal for now until we identify a need for it, at which point we can reintroduce it.

By the way, Log Probability is supported by other providers, such as Gemini Flash models. However, there are differences in the supported fields between providers. For example, OpenAI uses bytes to generate the result, whereas Gemini uses token IDs instead.

[stephentoub]

This doesn't seem specific to real-time. We should think through whether we need a representation for this that applies to IChatClient as well, for example. Or if it's actually needed at all... with IChatClient, devs that need this from the underlying provider can break glass, using RawRepresentationFactory or similar.

[tarekgh]

I removed it for now. We can bring it back later if we need to.

[stephentoub]

How does this relate to the concrete subtypes like RealtimeClientInputAudioBufferAppendMessage ?

[tarekgh]

I had initially to be able of using the same subtype for multiple events, but this is not the case any more so, I have removed RealtimeClientMessageType.

[stephentoub]

Why are these needed? Can't these be modeled using the same AITool-derived types we already have?

[tarekgh]

Fixed it.

[stephentoub]

It's always one and only one?

[tarekgh]

You are correct, this should be an array of contents. I'll fix that.

[stephentoub]

Could this just be ErrorContent as part of another message?

[tarekgh]

I prefer to keep this as a separate error message because it includes additional properties, such as ErrorEventId and Parameter, which cannot necessarily be added to other messages or to the ErrorContent itself. Please let me know if you still feel differently.

[stephentoub]

These are important enough to model like this rather than as part of AdditionalCounts?

[tarekgh]

@stephentoub I don't know where we draw the line for adding such new property or just using the AdditionalCounts? Recently, I am seeing we have added CachedInputTokenCount, why decided as a new property and not a part of AdditionalCounts? I am fine either way but will be good if we have a clear guidance when using AdditionalCounts and when add a new property.

[stephentoub]

Commonality across providers. Individual providers have been putting cached token information into AdditonalCounts for a while, but now that most providers have this notion and expose it, we elevated it to public property.

[tarekgh]

The new properties look supported in multiple providers. The difference is in Gemini Flash models it is derived from modality. Would that be enough to introduce the properties or just add them to the AdditonalCounts for now and we can consider exposing them later if we want to?

OpenAI:

"usage": {
  "input_token_details": {
    "text_tokens": 100,
    "audio_tokens": 400
  }
}

Gemini:

"usage_metadata": {
  "prompt_tokens_details": [
    { "modality": "TEXT", "token_count": 100 },
    { "modality": "AUDIO", "token_count": 400 }
  ]
}

[stephentoub]

I assume this is because of the source generator? Everyone else would get those same warnings as well. The experimental properties on serializable types should be marked as [JsonIgnore] as long as they're experimental.

[tarekgh]

Thanks. Using [JsonIgnore] fixed the issue.

[tarekgh]

Have you tried implementing this on multiple providers?

My plan is to implement this for the Gemini Flash 2.5 model. I'll update the proposal according to the finding from this implementation.

[tarekgh]

Closing this as we opened the official PR at dotnet#7285