Add automatic provider prompt caching and cache-hit metrics

[willccbb]

Summary

• Automatically apply prompt-cache defaults for supported providers inferred from endpoint URL and client type
• Surface cache-hit usage in token accounting so input_tokens reflects non-cache-hit prompt tokens
• Keep rollout scheduling unchanged and remove pre-firing logic
• Update eval display, TUI, docs, and tests for the new cache accounting shape

Testing

• Focused unit tests for prompt-cache policy inference, provider request mutation, usage parsing, and serialized output fallback
• Broader client/runtime regression tests for OpenAI, Anthropic, and eval lifecycle paths
• Lint and formatting checks passed

---

Note

Medium Risk
Medium risk because it changes request construction for Anthropic/OpenRouter (injecting cache_control) and alters token/usage accounting (input_tokens now excludes cache hits), which can affect downstream metrics and cost/usage reporting.

Overview
Adds automatic prompt caching behavior for supported providers by inferring provider from api_base_url + client_type, and applying the correct request mutation (Anthropic top-level cache_control, OpenRouter extra_body.cache_control, OpenAI implicit no-op) via a new prompt_cache_utils hook wired into Client.get_response().

Introduces a prompt_cache opt-out flag (default true) plumbed from endpoint registry and eval TOML into ClientConfig/EndpointClientConfig, with TOML validation and precedence rules.

Surfaces cache-hit accounting end-to-end by adding cached_input_tokens to Usage/TokenUsage, updating OpenAI/Anthropic usage parsing to split cached vs uncached prompt tokens, and propagating the new field through state saving, metrics, CLI/TUI display, and documentation/tests.

^{Reviewed by Cursor Bugbot for commit 10e0030. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add automatic prompt caching and cache-hit metrics for OpenAI, Anthropic, and OpenRouter

• Adds a new prompt_cache_utils.py module that detects the provider from client config/URL and injects provider-specific cache control payloads into requests for Anthropic and OpenRouter before each API call.
• All OpenAI, Anthropic, and OpenRouter clients now parse cache hit tokens from native responses and expose them as cached_input_tokens on Usage, subtracting them from reported input_tokens to avoid double-counting.
• Adds CachedInputTokensMetric and propagates cached_input_tokens through StateUsageTracker, TokenUsage, RolloutOutput, and metadata so cache hits are tracked end-to-end.
• Prompt caching can be disabled per-endpoint or globally via prompt_cache = false in TOML config or ClientConfig; invalid non-boolean values raise a ValueError.
• Behavioral Change: input_tokens in Usage and TokenUsage now excludes cache-hit tokens; consumers relying on input_tokens for total prompt size should add cached_input_tokens.

^{Macroscope summarized 10e0030.}

[macroscopeapp]

Approvability

Verdict: Needs human review

This PR introduces a new feature that automatically enables prompt caching for supported providers by default, modifying API requests and changing how token usage is calculated/reported. New features with runtime behavior changes enabled by default warrant human review.

^{You can customize Macroscope's approvability policy. Learn more.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 10e0030. Configure here.}

[cursor]

Skills files not updated after eval changes

Low Severity

Changes to verifiers/scripts/eval.py (adding prompt_cache config option and build_prompt_cache_enabled) and docs/evaluation.md (documenting prompt caching behavior) are listed as triggers for skills updates. No corresponding updates to skills/evaluate-environments/SKILL.md or other affected skill files appear in this PR.

Additional Locations (1)

• docs/evaluation.md#L169-L171

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit 10e0030. Configure here.}