Skip to content

Fixtures can't express tool-call-before-text ("tool-first") ordering; recorder collapses block order #274

Description

@tombeckenham

Fixtures can't express tool-call-before-text ("tool-first") ordering; recorder collapses block order

Summary

aimock always streams text before tool calls within a single assistant message, and there's no fixture knob to change that. The recorder makes it worse: it collapses the upstream stream into a normalized { content, toolCalls } fixture, so even recording a real provider's tool-first stream discards the ordering and replays it text-first.

This blocks testing any SDK behavior that depends on a provider streaming a tool_use / tool_call block before any text in the same message (e.g. AG-UI parentMessageId binding, where a tool-first stream must not change the assistant message id mid-stream).

Reproduction / evidence (v1.27.0)

A fixture with both content and toolCalls:

{ "response": { "content": "Let me check.", "toolCalls": [{ "name": "get_weather", "arguments": "{\"city\":\"SF\"}" }] } }

always streams the text block first, then the tool call, on every provider handler:

  • OpenAI chat-completionsbuildContentWithToolCallsChunks (helpers.js) pushes reasoning → text deltas → tool-call deltas.
  • AnthropicbuildClaudeContentWithToolCallsStreamEvents (messages.js) emits the text content block, content_block_stop, then the tool_use blocks.
  • Gemini / Ollama combined builders follow the same order.

There is no fixture field that reverses or interleaves this ordering.

The recorder can't recover it either: proxyAndRecord (recorder.js) runs the upstream response through collapseStreamingResponse(...) and persists fixtureResponse = { content, toolCalls }. The relative order of blocks in the original wire stream is lost at record time, so playback re-synthesizes text-first regardless of what the real provider sent.

Impact

SDK test suites that need a tool-first stream currently have to bypass aimock with a hand-crafted SSE mount (mock.mount(...)). We've now done this twice for shapes aimock can't model:

  1. server_tool_use blocks (Anthropic web_fetch).
  2. tool-call-before-text ordering (this issue).

Each hand-crafted mount re-implements a provider's wire format, which is exactly what aimock exists to avoid.

Proposed fix

Let a fixture express ordered blocks, streamed in array order across all providers:

{
  "response": {
    "blocks": [
      { "type": "toolCall", "name": "get_weather", "arguments": "{\"city\":\"SF\"}" },
      { "type": "text", "text": "Let me check the weather." }
    ]
  }
}

A minimal "toolCallsFirst": true flag on the existing { content, toolCalls } shape would unblock the common case if a full block model is too large a change. Ideally the recorder would also preserve block ordering when collapsing a stream.

Version

@copilotkit/aimock@1.27.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions