Skip to content

fix(translation): decode OpenRouter cache write usage#141

Merged
ayushag-nv merged 1 commit into
mainfrom
fix/openrouter-cache-write-usage
Jul 24, 2026
Merged

fix(translation): decode OpenRouter cache write usage#141
ayushag-nv merged 1 commit into
mainfrom
fix/openrouter-cache-write-usage

Conversation

@nachiketb-nvidia

@nachiketb-nvidia nachiketb-nvidia commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • decode OpenRouter's current prompt_tokens_details.cache_write_tokens field in buffered OpenAI Chat responses
  • decode the same field from streaming usage chunks
  • retain cache_creation_tokens as a compatibility alias

Root cause

OpenRouter reports cache writes as cache_write_tokens, while the OpenAI Chat codecs only recognized cache_creation_tokens. Cache-write tokens were therefore left in the uncached input count and omitted from normalized cache-creation usage.

Validation

  • cargo fmt --all -- --check
  • cargo test -p switchyard-translation
  • cargo clippy -p switchyard-translation --all-targets -- -D warnings

Summary by CodeRabbit

  • Bug Fixes

    • Improved token usage reporting for OpenAI-compatible responses and streaming events.
    • Cache-write token counts are now recognized consistently, including support for legacy cache-creation fields.
    • Corrected input, cached, cache-creation, and output token totals in translated usage data.
  • Tests

    • Added coverage confirming equivalent handling of both cache-write field formats.

Signed-off-by: nachiketb <nachiketb@nvidia.com>
@ayushag-nv
ayushag-nv marked this pull request as ready for review July 24, 2026 23:51
@ayushag-nv
ayushag-nv requested a review from a team as a code owner July 24, 2026 23:51
@ayushag-nv
ayushag-nv merged commit 9a43930 into main Jul 24, 2026
14 of 15 checks passed
@ayushag-nv
ayushag-nv deleted the fix/openrouter-cache-write-usage branch July 24, 2026 23:51
@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 004d2a67-e264-4e8c-9ef7-c3c8c0bf5f20

📥 Commits

Reviewing files that changed from the base of the PR and between 5349735 and dd02faa.

📒 Files selected for processing (4)
  • crates/switchyard-translation/src/codecs/openai_chat/buffered.rs
  • crates/switchyard-translation/src/codecs/openai_chat/stream.rs
  • crates/switchyard-translation/tests/response_translation.rs
  • crates/switchyard-translation/tests/stream_translation.rs

Walkthrough

OpenAI Chat usage normalization now prefers cache_write_tokens and falls back to cache_creation_tokens in buffered and streaming paths. Tests cover cache read, cache creation, input, and output token mappings for both field names.

Changes

OpenAI cache usage

Layer / File(s) Summary
Cache-write token normalization
crates/switchyard-translation/src/codecs/openai_chat/buffered.rs, crates/switchyard-translation/src/codecs/openai_chat/stream.rs, crates/switchyard-translation/tests/response_translation.rs, crates/switchyard-translation/tests/stream_translation.rs
Buffered and streaming usage decoding prefer cache_write_tokens, fall back to cache_creation_tokens, and test normalized cache read, cache creation, input, and output token values.

Estimated code review effort: 2 (Simple) | ~10 minutes

Poem

A bunny found tokens tucked under a leaf,
Cache-write carrots, fresh and brief.
Legacy crumbs still lead the way,
Read and output count the day.
Hop, hop—usage blooms anew!


Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants