Skip to content

ai(observability): PostHog prices cached prompt tokens as fresh input — cache tiers are summed away before capture #10263

Description

@JSONbored

Part of epic #8286 (Phase 3 — AI observability). Follow-on to #10251 and #10246.

Problem

After #10251, $ai_input_tokens carries the correct total prompt size for the CLI providers — the three Anthropic tiers summed. But PostHog has no way to tell how much of that total was a cache read, and cache reads bill at roughly a tenth of fresh input.

So PostHog prices the whole figure at the fresh-input rate, and its derived $ai_input_cost_usd over-states the input component by several multiples on a cached review. $ai_total_cost_usd is unaffected — it is sent directly from the CLI's own total_cost_usd and remains authoritative.

Fix

Retain the individual tiers alongside the summed total and emit them:

  • $ai_cache_read_input_tokens
  • $ai_cache_creation_input_tokens
  • $ai_cache_reporting_exclusive: false

The false is the load-bearing part. PostHog documents two conventions:

Exclusive counting (Anthropic/Claude) — cache tokens are separate from $ai_input_tokens. 100 input + 50 cached → $ai_input_tokens is 100.
Inclusive counting (OpenAI and most others) — cache tokens are included. Same example → $ai_input_tokens is 150.

Post-#10251 this deployment sends the inclusive shape, so the flag must say so. Setting it true against an inclusive count makes PostHog add the cache on top of a figure that already contains it — exactly the 3.4× over-billing reported in PostHog/posthog-js#3615.

Leaving the flag unset is probably fine today, since PostHog auto-detects from $ai_provider and ours reads claude-code / claude-cli / agent-sdk rather than anthropic, so it would fall through to inclusive. But relying on an unrecognised provider name to land on the right default is fragile — a future rename, or PostHog widening its detection, would silently flip the pricing.

Scope

  • ORB: src/selfhost/ai.ts mergeUsage currently discards the tiers after summing them (totalInputTokens); CliUsage/AiUsage would need to retain them, and capturePostHogAiGeneration to emit them.
  • Miner: the same, through MinerAiGenerationRecord and captureMinerPostHogAiGeneration.

Deliverables

  • Retain cacheReadInputTokens / cacheCreationInputTokens on the usage types without changing what inputTokens means (it stays the inclusive sum from fix(observability): count the prompt-cache tiers as input tokens for the CLI providers #10251).
  • Emit both properties plus $ai_cache_reporting_exclusive: false from both PostHog paths.
  • Omit all three — the flag included, absent rather than false — for a provider that reports no cache tiers. A local ollama model has no cache-accounting convention to declare.
  • Verify against a real event afterwards that $ai_input_cost_usd drops to a plausible figure rather than tracking $ai_total_cost_usd.
  • Add a cache-hit-rate panel to the AI observability dashboard (id 1934420) once the data lands — cache read ÷ total input is a directly actionable spend signal.

Metadata

Metadata

Assignees

Labels

maintainer-onlyOwner-only work — yields no Gittensor points.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions