Describe the bug
When a kernel function returns ImageContent, FunctionCallsProcessor.ProcessFunctionResult() serializes it to JSON. This prevents connectors from using native image formats in tool results, even when the underlying API supports it (Gemini 3+, Anthropic Claude).
The model receives "{ \"Data\": \"base64...\", \"MimeType\": \"image/png\" }" as text instead of an actual image it can "see".
To Reproduce
-
Create a kernel function that returns ImageContent:
[KernelFunction]
public ImageContent CaptureScreenshot()
{
var imageBytes = TakeScreenshot();
return new ImageContent(imageBytes, "image/png");
}
-
Enable auto function calling with Gemini 3 connector
-
Inspect the API request - observe that functionResponse.response contains serialized JSON text instead of functionResponse.parts[].inlineData
Expected behavior
Connectors that support images in tool results should receive the original ImageContent object and convert it to their native format:
- Gemini 3+:
functionResponse.parts[].inlineData
- Anthropic:
ImageBlockParam in ToolResultBlockParam.Content
- OpenAI/others: Clear error message (API does not support images in tool results)
Platform
- Language: C#
- Source: main branch
- AI model: Google Gemini 3 (gemini-2.5-pro, gemini-2.5-flash)
- IDE: any
- OS: any
Additional context
| API |
Image Tool Results |
Native Format |
| Google Gemini 3+ |
✅ Supported |
functionResponse.parts[].inlineData |
| Anthropic Claude |
✅ Supported |
ImageBlockParam in ToolResultBlockParam.Content |
| Google Gemini 2.5 and earlier |
❌ Not supported |
- |
| OpenAI |
❌ Not supported |
- |
Related: PR #13419 (Anthropic Connector multimodal support)
Describe the bug
When a kernel function returns
ImageContent,FunctionCallsProcessor.ProcessFunctionResult()serializes it to JSON. This prevents connectors from using native image formats in tool results, even when the underlying API supports it (Gemini 3+, Anthropic Claude).The model receives
"{ \"Data\": \"base64...\", \"MimeType\": \"image/png\" }"as text instead of an actual image it can "see".To Reproduce
Create a kernel function that returns
ImageContent:Enable auto function calling with Gemini 3 connector
Inspect the API request - observe that
functionResponse.responsecontains serialized JSON text instead offunctionResponse.parts[].inlineDataExpected behavior
Connectors that support images in tool results should receive the original
ImageContentobject and convert it to their native format:functionResponse.parts[].inlineDataImageBlockParaminToolResultBlockParam.ContentPlatform
Additional context
functionResponse.parts[].inlineDataImageBlockParaminToolResultBlockParam.ContentRelated: PR #13419 (Anthropic Connector multimodal support)