Skip to content

fix(core): make cache read cost fallback to input token cost#17130

Open
benjamincburns wants to merge 1 commit intoanomalyco:devfrom
benjamincburns:fix/17121_cost_calculation_bug
Open

fix(core): make cache read cost fallback to input token cost#17130
benjamincburns wants to merge 1 commit intoanomalyco:devfrom
benjamincburns:fix/17121_cost_calculation_bug

Conversation

@benjamincburns
Copy link

Issue for this PR

Closes #17121

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

When setting up the model object, the cost calculation for cache reads was falling back to 0. For providers that charge the same price for input caching, this caused the calculated cost to be dramatically lower than it should be. Changed the fallback to use the input token cost instead of 0 in both fromModelsDevModel() and the config provider section.

How did you verify your code works?

Added unit tests, also ran opencode via bun run dev serve on the same project, with the same prompt and same initial starting conditions as described in #17121. I checked the context view and validated that the cost shown roughly matches what Together.AI charged.

Screenshots / recordings

Proof of manual testing:
image

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

If you do not follow this template your PR will be automatically rejected.

When setting up the model object, the cost calculation for cache reads was
falling back to 0. For providers that charge the same price for input
caching, this caused the calculated cost to be dramatically lower than
it should be. Changed the fallback to use the input token cost instead
of 0 in both fromModelsDevModel() and the config provider section.

fixes anomalyco#17121
@benjamincburns
Copy link
Author

Failure in e2e job doesn't appear to be related to anything that I did - are these expected to fail on runs from outside contributors due to missing GITHUB_TOKEN or some other similar constraint? If not, maybe it just needs a rerun? 🤞

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CLI stats dramatically underestimate Together.AI costs for models that don't support input caching

1 participant