fix(deepinfra): derive tiered cache prices from the published multiplier - #5280
Open
xyzs996 wants to merge 1 commit into
Open
fix(deepinfra): derive tiered cache prices from the published multiplier#5280xyzs996 wants to merge 1 commit into
xyzs996 wants to merge 1 commit into
Conversation
DeepInfra publishes each tier's cached price twice: as the structured rate_per_input_token_cached multiplier, and restated as an absolute price inside the free-text `full` string. The tiered path read the string, the flat path read the multiplier, so one field had two derivations. They disagree on ByteDance/Seed-2.0-mini, whose string prices cached input above 128K at $0.2/Mtok - identical to its fresh input price - where the multiplier gives $0.04/Mtok. Every other published tier segment agrees. Also emit cache_write per tier. Both cache rates are multipliers on the segment's input price, so a tier that doubles input doubles them too; pricing cache_write only on the base tier would bill a long-context request at the short-context write rate.
Contributor
|
No actionable findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
deepinfra.tsderivescache_readtwo different ways depending on which branch ofbuildCostruns:inputCost * rate_per_input_token_cached(the structured multiplier)fullstringBoth are published in the same payload, and normally they agree. Across every tiered model DeepInfra serves today — 7 models, 17 tier segments — 16 segments agree exactly and one doesn't:
fullsays cachedrate * inputQwen/Qwen3-MaxQwen/Qwen3-Max-ThinkingQwen/Qwen3.7-MaxByteDance/Seed-2.0-proByteDance/Seed-2.0-codeByteDance/Seed-1.8ByteDance/Seed-2.0-miniByteDance/Seed-2.0-miniThe live string is:
rate_per_input_token_cachedis0.2for all seven of these models, so the second segment's cached price should be0.2 * 0.2 = $0.04. What ships today iscache_read = 0.2— 5x too high, and exactly equal to the tier's own input price, i.e. cached input priced identically to fresh input. It's also the same0.2asSeed-2.0-pro's top tier, whose input price is 5x higher.Two independent tells that the string is the wrong source here, not the multiplier: a cached rate can't equal the input rate it discounts, and 16 of 17 segments say the multiplier is authoritative.
What this changes
1. One derivation for
cache_read. Tier segments now compute it assegment input * rate_per_input_token_cached, the same way the flat path already does, andconsole.warnwhen the string disagrees — so a genuine future rate change surfaces instead of silently overwriting. When no multiplier is published the stated string value is still used, so nothing regresses for a model that only publishes the string.2.
cache_writeis now emitted per tier. It was computed only for the base tier:Same reasoning as
cache_read: the file's own comment says these rates are "multipliers applied to the input price", and a context tier that doubles input therefore doubles both cache prices. Leavingcache_writeoff the tiers bills a long-context request at the short-context write rate. This one is latent, not live — no tiered DeepInfra model publishesrate_per_input_token_cache_writetoday, so the shipped data is unchanged by it. I've fixed it anyway because the field is already read on the flat path, so the moment DeepInfra populates it on a tiered model the data would go quietly wrong.Data change is one line:
Seed-2.0-mini's 128K tiercache_read0.2 → 0.04.Tests
New
packages/core/test/deepinfra.test.ts, following thebaseten.test.tsshape. Five cases, all built from live payloads:Seed-2.0-pro) — unchanged output, so the fix isn't a blanket rewriteSeed-2.0-mini) — pins0.04, plus an invariant that a tier'scache_readstays below itsinputcache_writescales per tierQwen3.7-Max, 32K + 128K) — the multi-tier shape isn't exercised anywhere todayVerification
The 4 failures are pre-existing on
devat08324a0and identical before and after this change (snapshot entrypoint is self-contained,snapshot exports providers, models, generatedAt, and a default catalog,catalog generation > repository open-weight model metadata includes weights links,DeepInfra preserves live modalities for new base models). Baseline is 221 pass / 4 fail; this adds the 5 new passes and no new failures.Prices re-read from
https://api.deepinfra.com/models/list?type=text-generationon 2026-08-22.One thing I did not do
I didn't report the
$0.2 cachedstring to DeepInfra as a typo, because I can't tell from outside whether it's a formatting slip or a real rate. If it's real, this PR is wrong for that one model and the warning is the thing that would tell you — it fires on exactly that segment and names both numbers. Happy to invert the precedence (trust the string, warn on mismatch) if you'd rather the free text stay authoritative; the helper is one branch either way.