Skip to content

fix(ai-anthropic): populate cachedInputTokens in streaming responses#6288

Merged
tim-smart merged 2 commits into
Effect-TS:mainfrom
davidgoli:fix-anthropic-streaming-cached-input-tokens
Jun 24, 2026
Merged

fix(ai-anthropic): populate cachedInputTokens in streaming responses#6288
tim-smart merged 2 commits into
Effect-TS:mainfrom
davidgoli:fix-anthropic-streaming-cached-input-tokens

Conversation

@davidgoli

@davidgoli davidgoli commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Summary

The Anthropic streaming response builder (makeStreamResponse) populates inputTokens, outputTokens, and totalTokens, but never copies cache_read_input_tokens into the normalized usage. As a result usage.cachedInputTokens is always undefined for streamText / streamObject, even when prompt caching is active and the API reports a non-zero cache read at message_start. The non-streaming path already maps this field, so streaming and non-streaming disagree.

Fix

Set cachedInputTokens from cache_read_input_tokens at message_start — where input_tokens is already read — mirroring the non-streaming mapping. The output payload shape is unchanged; only the previously-missing field is now populated.

case "message_start": {
  usage.inputTokens = event.message.usage.input_tokens
  usage.cachedInputTokens = event.message.usage.cache_read_input_tokens ?? undefined
  metaUsage = event.message.usage
  // ...
}

event.message.usage is Generated.BetaUsage, which already declares cache_read_input_tokens, so this matches the existing non-streaming mapping (?? undefined).

Notes

I didn't add a streaming test — packages/ai/anthropic has no test harness for the language model adapter (no test/ dir, and the sibling ai-openai package has none either), so there's no pattern yet for mocking the Anthropic SSE stream. Happy to follow up with a focused stream test once the maintainers point at the preferred mocking shape.

Cache writes (cache_creation_input_tokens) are out of scope — @effect/ai's core Usage type has no corresponding field.

A changeset is included (@effect/ai-anthropic, patch).

@davidgoli davidgoli requested a review from IMax153 as a code owner June 23, 2026 17:20
@github-project-automation github-project-automation Bot moved this to Discussion Ongoing in PR Backlog Jun 23, 2026
@changeset-bot

changeset-bot Bot commented Jun 23, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 512af66

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
@effect/ai-anthropic Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

The streaming response builder set inputTokens/outputTokens/totalTokens but
never copied cache_read_input_tokens, so cachedInputTokens was always
undefined for streamText/streamObject even when prompt caching was active.
The non-streaming path already maps it; mirror that at message_start, where
the Anthropic API reports the cache-read count.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@davidgoli davidgoli force-pushed the fix-anthropic-streaming-cached-input-tokens branch from bf21ede to 1ca7194 Compare June 23, 2026 17:27
Comment thread .changeset/anthropic-streaming-cached-input-tokens.md Outdated
@tim-smart tim-smart deployed to release-queue June 24, 2026 01:30 — with GitHub Actions Active
@tim-smart tim-smart enabled auto-merge (squash) June 24, 2026 01:31
@tim-smart tim-smart merged commit cfa2bbf into Effect-TS:main Jun 24, 2026
12 checks passed
@github-project-automation github-project-automation Bot moved this from Discussion Ongoing to Done in PR Backlog Jun 24, 2026
@github-actions github-actions Bot mentioned this pull request Jun 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants