Skip to content

fix(deepinfra): derive tiered cache prices from the published multiplier - #5280

Open
xyzs996 wants to merge 1 commit into
anomalyco:devfrom
xyzs996:deepinfra-tiered-cache-prices
Open

fix(deepinfra): derive tiered cache prices from the published multiplier#5280
xyzs996 wants to merge 1 commit into
anomalyco:devfrom
xyzs996:deepinfra-tiered-cache-prices

Conversation

@xyzs996

@xyzs996 xyzs996 commented Aug 22, 2026

Copy link
Copy Markdown

The problem

deepinfra.ts derives cache_read two different ways depending on which branch of buildCost runs:

  • flat pricinginputCost * rate_per_input_token_cached (the structured multiplier)
  • tiered pricing → the absolute number parsed out of the free-text full string

Both are published in the same payload, and normally they agree. Across every tiered model DeepInfra serves today — 7 models, 17 tier segments — 16 segments agree exactly and one doesn't:

model segment input full says cached rate * input
Qwen/Qwen3-Max base / 32K / 128K 1.2 / 2.4 / 3 0.24 / 0.48 / 0.6 0.24 / 0.48 / 0.6 ✅
Qwen/Qwen3-Max-Thinking base / 32K / 128K 1.2 / 2.4 / 3 0.24 / 0.48 / 0.6 0.24 / 0.48 / 0.6 ✅
Qwen/Qwen3.7-Max base / 32K / 128K 2.5 / 5 / 6.25 0.5 / 1 / 1.25 0.5 / 1 / 1.25 ✅
ByteDance/Seed-2.0-pro base / 128K 0.5 / 1 0.10 / 0.20 0.10 / 0.20 ✅
ByteDance/Seed-2.0-code base / 128K 0.5 / 1 0.10 / 0.20 0.10 / 0.20 ✅
ByteDance/Seed-1.8 base / 128K 0.25 / 0.5 0.05 / 0.1 0.05 / 0.1 ✅
ByteDance/Seed-2.0-mini base 0.1 0.02 0.02 ✅
ByteDance/Seed-2.0-mini 128K 0.2 0.2 0.04

The live string is:

$0.10 in $0.40 out $0.02 cached <= 128K, $0.2 in $0.80 out $0.2 cached

rate_per_input_token_cached is 0.2 for all seven of these models, so the second segment's cached price should be 0.2 * 0.2 = $0.04. What ships today is cache_read = 0.25x too high, and exactly equal to the tier's own input price, i.e. cached input priced identically to fresh input. It's also the same 0.2 as Seed-2.0-pro's top tier, whose input price is 5x higher.

Two independent tells that the string is the wrong source here, not the multiplier: a cached rate can't equal the input rate it discounts, and 16 of 17 segments say the multiplier is authoritative.

What this changes

1. One derivation for cache_read. Tier segments now compute it as segment input * rate_per_input_token_cached, the same way the flat path already does, and console.warn when the string disagrees — so a genuine future rate change surfaces instead of silently overwriting. When no multiplier is published the stated string value is still used, so nothing regresses for a model that only publishes the string.

2. cache_write is now emitted per tier. It was computed only for the base tier:

cache_write: cacheWriteRate == null ? undefined : round(base.input * cacheWriteRate),
tiers: tiered.tiers.map((tier) => ({
  input: round(tier.input),
  output: round(tier.output),
  cache_read: ...,          // <- no cache_write
})),

Same reasoning as cache_read: the file's own comment says these rates are "multipliers applied to the input price", and a context tier that doubles input therefore doubles both cache prices. Leaving cache_write off the tiers bills a long-context request at the short-context write rate. This one is latent, not live — no tiered DeepInfra model publishes rate_per_input_token_cache_write today, so the shipped data is unchanged by it. I've fixed it anyway because the field is already read on the flat path, so the moment DeepInfra populates it on a tiered model the data would go quietly wrong.

Data change is one line: Seed-2.0-mini's 128K tier cache_read 0.2 → 0.04.

Tests

New packages/core/test/deepinfra.test.ts, following the baseten.test.ts shape. Five cases, all built from live payloads:

  • agreeing case (Seed-2.0-pro) — unchanged output, so the fix isn't a blanket rewrite
  • disagreeing case (Seed-2.0-mini) — pins 0.04, plus an invariant that a tier's cache_read stays below its input
  • no multiplier published — falls back to the string
  • cache_write scales per tier
  • three-band string (Qwen3.7-Max, 32K + 128K) — the multi-tier shape isn't exercised anywhere today

Verification

bun test                # 226 pass, 4 fail
bun run validate        # exit 0

The 4 failures are pre-existing on dev at 08324a0 and identical before and after this change (snapshot entrypoint is self-contained, snapshot exports providers, models, generatedAt, and a default catalog, catalog generation > repository open-weight model metadata includes weights links, DeepInfra preserves live modalities for new base models). Baseline is 221 pass / 4 fail; this adds the 5 new passes and no new failures.

Prices re-read from https://api.deepinfra.com/models/list?type=text-generation on 2026-08-22.

One thing I did not do

I didn't report the $0.2 cached string to DeepInfra as a typo, because I can't tell from outside whether it's a formatting slip or a real rate. If it's real, this PR is wrong for that one model and the warning is the thing that would tell you — it fires on exactly that segment and names both numbers. Happy to invert the precedence (trust the string, warn on mismatch) if you'd rather the free text stay authoritative; the helper is one branch either way.

DeepInfra publishes each tier's cached price twice: as the structured
rate_per_input_token_cached multiplier, and restated as an absolute price
inside the free-text `full` string. The tiered path read the string, the
flat path read the multiplier, so one field had two derivations.

They disagree on ByteDance/Seed-2.0-mini, whose string prices cached input
above 128K at $0.2/Mtok - identical to its fresh input price - where the
multiplier gives $0.04/Mtok. Every other published tier segment agrees.

Also emit cache_write per tier. Both cache rates are multipliers on the
segment's input price, so a tier that doubles input doubles them too;
pricing cache_write only on the base tier would bill a long-context
request at the short-context write rate.
@github-actions

Copy link
Copy Markdown
Contributor

No actionable findings.

@github-actions github-actions Bot added the reviewer: ready Automated review found no actionable items label Aug 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

reviewer: ready Automated review found no actionable items

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant