Skip to content

llm: surface prompt-cache usage (cache_creation/cache_read tokens) in UsageInfo + llm_call audit #295

Description

@initializ-mk

Context

Live validation of compression.cache_hints (anthropic provider) required standing up an external recording proxy between forge and api.anthropic.com, because forge itself gives operators no way to see whether prompt caching is working.

Gap

The anthropic provider parses only input_tokens / output_tokens from the API response usage block. Anthropic also returns:

  • cache_creation_input_tokens — tokens written to the prompt cache (billed at 1.25x)
  • cache_read_input_tokens — tokens served from cache (billed at 0.1x)

Both are currently dropped on the floor (forge-core/llm/providers/anthropic.go, non-streaming Usage struct and the streaming message_start event). Consequences:

  • An operator enabling compression.cache_hints cannot verify it does anything — llm_call audit events show identical fields whether the cache hits, misses, or the hints are silently malformed.
  • Cost dashboards built on the audit stream over-estimate spend: cache-read tokens are counted (via input_tokens context) as if they were full-rate.

Proposal

  1. Parse both fields in the anthropic provider (non-streaming response and streaming message_start), and populate two new optional fields on llm.UsageInfo (e.g. CacheCreationTokens, CacheReadTokens) — zero-valued for providers that don't report them.
  2. Stamp them on the llm_call audit event when non-zero, e.g. cache_creation_input_tokens, cache_read_input_tokens — additive fields, no breaking change to the audit contract.
  3. Optionally roll them up on invocation_complete (e.g. cache_read_tokens_total) next to the existing compression_saved_tokens_total, so a single event answers "what did compression + caching save this invocation".
  4. OpenAI equivalent for parity: usage.prompt_tokens_details.cached_tokens is already returned on cache hits and is likewise unparsed today.

Evidence from the validation run

An external proxy observed what forge could not: call 1 wrote 3,447 tokens to cache, calls 2–4 each read 3,447 back (cache_read_input_tokens: 3447, cache_creation: 0) — none of it visible in forge's own logs or audit stream.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions