diff --git a/chat/overview.mdx b/chat/overview.mdx index 7f4ebdc..7a5a121 100644 --- a/chat/overview.mdx +++ b/chat/overview.mdx @@ -167,7 +167,7 @@ The longer the conversation, the more tokens it uses. The maximum conversation l | `frequency_penalty` | Penalizes tokens based on how often they've appeared so far, reducing repetition. Range: -2.0 to 2.0. Default: `0` | | `response_format` | See [Structured Outputs](/chat/structured-outputs) | | `tools`, `tool_choice` | See [Tool Calling](/chat/tool-calling) | -| `service_tier` | Priority inference for tagged models. See [Service Tier](#service-tier) below. | +| `service_tier` | Select a service tier (`"priority"` or `"flex"`) for tagged models. See [Service Tier](#service-tier) below. | | `reasoning_effort` | Controls reasoning depth for reasoning models. See [Reasoning Models](/chat/reasoning). | @@ -178,6 +178,10 @@ For the complete parameter reference, see the [API reference](/api-reference/cha ## Service tier +Set the optional `service_tier` parameter to run a request on a non-standard tier. Two tiers are available on tagged models: **priority** (faster, at a premium) and **flex** (cheaper, best-effort). Leave `service_tier` unset for standard real-time scheduling and pricing. + +### Priority + Set `service_tier` to `"priority"` to request priority inference on supported models. Priority requests get faster time-to-first-token and higher throughput during peak demand. @@ -220,7 +224,51 @@ curl "https://api.deepinfra.com/v1/openai/chat/completions" \ -The response includes a `service_tier` field confirming which tier was actually used. Not all models support priority tiers — check the model page for availability. If a model doesn't support priority inference, the request is served at the standard tier and billed at the standard price; no error is returned. +### Flex + +Set `service_tier` to `"flex"` to run Chat Completions requests at a lower cost in exchange for slower response times and occasional resource unavailability. It's ideal for non-production or lower-priority tasks such as model evaluations, data enrichment, and asynchronous workloads. When a model is busy, a flex request may wait up to 10 minutes for available capacity before it runs or is rejected with an HTTP 429, so use it for work you can retry. + + + Flex inference is billed at a 20% discount off the model's standard per-token price. + + + + +```python Python +response = client.chat.completions.create( + model="deepseek-ai/DeepSeek-V3", + messages=[{"role": "user", "content": "Hello!"}], + extra_body={"service_tier": "flex"}, +) +``` + +```javascript JavaScript +const response = await openai.chat.completions.create({ + model: "deepseek-ai/DeepSeek-V3", + messages: [{ role: "user", content: "Hello!" }], + service_tier: "flex", +}); +``` + +```bash cURL +curl "https://api.deepinfra.com/v1/openai/chat/completions" \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer $DEEPINFRA_TOKEN" \ + -d '{ + "model": "deepseek-ai/DeepSeek-V3", + "service_tier": "flex", + "messages": [ + { + "role": "user", + "content": "Hello!" + } + ] + }' +``` + + + +The response includes a `service_tier` field confirming which tier was actually used. Not all models support these tiers — check the model page for availability. If a model doesn't support the requested tier, the request is served at the standard tier and billed at the standard price; no error is returned. ## Max output tokens