Anthropic server-side tool content blocks (`server_tool_use`, `web_search_tool_result`) dropped in streaming instrumentation

[braintrust-bot]

Summary

Anthropic's server-side tools (web_search, code_execution, etc.) produce server_tool_use and web_search_tool_result content blocks in responses. The auto-instrumentation plugin's streaming aggregation silently drops these blocks because finalizeContentBlock only handles text and tool_use types explicitly. For server_tool_use, the accumulated input_json_delta text is pushed to fallbackTextDeltas (corrupting the text output with raw JSON), and the block itself is deleted. For web_search_tool_result, the block is silently deleted with no trace.

In non-streaming mode, the raw content array passes through extractOutput and these blocks survive via the { type: string } catch-all in AnthropicOutputContentBlock. The gap is specifically in the streaming aggregation path.

What instrumentation is missing

Plugin streaming aggregation (js/src/instrumentation/plugins/anthropic-plugin.ts)

finalizeContentBlock (lines 255–302) has three explicit paths:

1. isToolUseContentBlock → matches only type: "tool_use", NOT type: "server_tool_use"
2. isTextContentBlock → matches only type: "text"
3. Fallback → pushes accumulated text to fallbackTextDeltas and deletes the block

When a server_tool_use block streams:

• content_block_start saves the block with { type: "server_tool_use", id, name, input: {} }
• content_block_delta with input_json_delta IS accumulated (this delta type is recognized)
• content_block_stop → finalizeContentBlock → falls through to fallback → accumulated JSON pushed to fallbackTextDeltas, block deleted

When a web_search_tool_result block streams:

• content_block_start saves the full block with search results
• No deltas expected for this block type
• content_block_stop → finalizeContentBlock → no accumulated text → block silently deleted

Vendor types (js/src/vendor-sdk-types/anthropic.ts)

AnthropicOutputContentBlock (lines 85–93) only explicitly models text and tool_use. While the { type: string } catch-all technically accepts server tool blocks, the type narrowing helpers (isToolUseContentBlock, isTextContentBlock) don't recognize them, which is what causes the streaming deletion.

Braintrust docs status

Braintrust Anthropic integration docs mention streaming and tool use support but do not specifically mention server-side tools (not_found).

• https://www.braintrust.dev/docs/instrument/wrap-providers

Upstream reference

• Anthropic web search tool docs: https://platform.claude.com/docs/en/docs/build-with-claude/tool-use/web-search-tool
• Anthropic code execution tool docs: https://platform.claude.com/docs/en/docs/build-with-claude/tool-use/code-execution-tool
• Server-side tools are stable, production features (not beta). Supported on all current Claude models.
• Response content block types: server_tool_use (the tool invocation), web_search_tool_result (search results with URLs, titles, encrypted content)
• Streaming format: server_tool_use blocks stream with input_json_delta deltas (same as regular tool_use), web_search_tool_result blocks arrive complete in content_block_start
• Usage includes server_tool_use.web_search_requests count

Local files inspected

• js/src/instrumentation/plugins/anthropic-plugin.ts (lines 255–302: finalizeContentBlock, lines 174–193: delta handling)
• js/src/vendor-sdk-types/anthropic.ts (lines 85–93: AnthropicOutputContentBlock)
• js/src/wrappers/anthropic.ts (streaming handler)
• e2e/scenarios/anthropic-instrumentation/scenario.impl.mjs (no server tool test cases)