Skip to content

openai-compat.md

Latest commit

 

History

History
208 lines (167 loc) · 8.12 KB

openai-compat.md

File metadata and controls

208 lines (167 loc) · 8.12 KB
Raw

OpenAI-compatible API (Step 3.4)

The gateway speaks the OpenAI REST dialect so any OpenAI client SDK can use AgentContextOS as a drop-in base_url. Three routes:

Route Purpose
POST /v1/embeddings Embed text via the wired Embedder SPI
POST /v1/chat/completions Chat completion with retrieval pre-fetch (RAG)
GET /v1/models List the wired LLM + embedder under OpenAI's model-list shape

The wire models live in rag_core.openai_types so SDKs and downstream services can import the request/response shapes without depending on the FastAPI app.

Overview

The value-add over a bare LLM proxy is retrieval pre-fetch: before a chat completion reaches the LLM, the gateway runs the Phase 2 retrieval pipeline (understanding → router → optional rerank / pack) over the conversation's last user turn and injects the retrieved context as a system message. A vanilla OpenAI client gets RAG "for free" (retrieval is on by default); a RAG-aware client tunes it through the non-standard rag request field.

Errors use the OpenAI envelope ({"error": {message, type, param, code}}) so client SDKs parse them — not the platform's GatewayError envelope used by the native /v1/query routes.

Usage

POST /v1/embeddings

curl -s localhost:8000/v1/embeddings \
  -H 'authorization: Bearer demo' \
  -d '{"input": ["hello world", "second text"], "model": "text-embedding-3-small"}'
{
  "object": "list",
  "data": [
    {"object": "embedding", "embedding": [0.0, 0.0, ...], "index": 0},
    {"object": "embedding", "embedding": [0.0, 0.0, ...], "index": 1}
  ],
  "model": "noop-embedder",
  "usage": {"prompt_tokens": 5, "total_tokens": 5}
}
  • input accepts a string or a list of strings. Token-id array forms (list[int]) are not supported (HTTP 400) — the gateway embeds text.
  • encoding_format: "base64" returns each vector as a base64-encoded little-endian float32 blob (OpenAI's binary form) instead of a number array.
  • model and dimensions are accepted for client compatibility but are informational: the wired Embedder controls the model + output dimension and reports its canonical model id in model.

Using the official client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="demo")
client.embeddings.create(model="text-embedding-3-small", input=["a", "b"])

POST /v1/chat/completions

curl -s localhost:8000/v1/chat/completions \
  -H 'authorization: Bearer demo' \
  -d '{
        "model": "rag-gateway",
        "messages": [{"role": "user", "content": "What is reciprocal rank fusion?"}],
        "rag": {"top_k": 5, "corpus_ids": ["docs"]}
      }'
{
  "id": "chatcmpl-…",
  "object": "chat.completion",
  "created": 1780000000,
  "model": "noop-llm",
  "choices": [
    {"index": 0, "message": {"role": "assistant", "content": "…[1]"}, "finish_reason": "stop"}
  ],
  "usage": {"prompt_tokens": 42, "completion_tokens": 18, "total_tokens": 60},
  "rag": {
    "enabled": true,
    "shape": "semantic",
    "bm25_fallback": false,
    "chunks_n": 5,
    "citations": [ … ]
  }
}

The non-standard rag request field (RagOptions) mirrors the retrieval knobs of /v1/query:

Field Default Meaning
enabled true Run retrieval + inject context. false = plain LLM proxy
corpus_ids [] Restrict retrieval to these corpora (empty = all visible)
top_k 10 Max chunks from retrieval (pre-rerank)
rerank true Run the cross-encoder reranker
pack true Apply the ContextPacker budget-fit + reorder
pack_budget 2048 Token budget for the packer
query null Override the retrieval query (default: last user message)

The response carries retrieval metadata back under the non-standard rag field (RagAnnotations) — citations, the query-shape the router picked, and whether the BM25-only fallback fired. OpenAI clients ignore it; RAG-aware clients read it.

Streaming

Set "stream": true to receive Server-Sent Events. The stream emits a role-only opening chunk, one chat.completion.chunk per content token, a terminal chunk carrying finish_reason: "stop" (plus usage and the rag annotations), then the literal data: [DONE] sentinel.

curl -N localhost:8000/v1/chat/completions \
  -H 'authorization: Bearer demo' \
  -d '{"messages": [{"role": "user", "content": "hi"}], "stream": true}'

GET /v1/models

Returns the wired LLM + embedder ids plus a generic rag-gateway alias, under OpenAI's {"object": "list", "data": [{"object": "model", "id": …}]} shape. Many clients call this on startup to validate the endpoint.

Authentication

Identity is header-driven (the OpenAI wire shape has no tenant field):

  • Authorization: Bearer <token> → resolved to a Principal via the wired Auth backend. X-Tenant-Id selects the tenant (defaults to the gateway's default tenant).
  • Dev mode (the default NoopAuth wiring): the routes are creds-free — a request with no auth gets an anonymous principal under the default tenant, so the demo is curl-able like /v1/query. The optional OpenAI user field is used as the anonymous principal id.
  • Production (a real Auth backend wired): a request that arrives without a resolved principal is rejected with 401.

Errors

Condition HTTP error.type
Bad input (empty/array input, etc.) 400 invalid_request_error
Pydantic validation failure 422 (FastAPI default)
Missing / invalid auth 401 authentication_error
ACL denied 403 permission_error
Quota / rate limit 429 rate_limit_error
Downstream retrieval / LLM failure 502 api_error

Internals

make_openai_router() in apps/gateway/src/rag_gateway/openai_compat.py builds the routes. They read the same injectable dependencies off app.state that /v1/query does (understanding / retrieval_router / reranker / packer / llm / policy_engine) plus a new top-level embedder (build_default_embedder()), so REST, gRPC, MCP, and the OpenAI surface all share one wiring.

Context injection inserts a single system message — the numbered context block plus a citation instruction — immediately before the last user turn, so it primes that turn without clobbering the caller's own system prompt.

Token counts for embeddings usage (and streaming chat usage) come from the packer's TiktokenCounter with a whitespace fallback; non-streaming chat usage reports the LLM's own input_tokens / output_tokens.

Governance

Both routes consult the PolicyEngine:

  • Chat runs the egress_text decision over the retrieved context before every LLM call (the same consult /v1/query's answer generation makes). A deny drops the context (the answer proceeds without it, annotations show chunks_n: 0); a transform substitutes redacted chunks.
  • Embeddings runs a quota_check over the input token count before the embed; a deny returns 429.

This keeps the module inside the PolicyEngine coverage linter naturally.

Extension points

  • Real embedder / LLM — wire production backends via build_app(embedder=…, llm=…); the routes report their model ids.
  • dimensions enforcement — a Matryoshka-capable embedder honours the requested dimension through its own config; the route passes the field through unchanged.
  • Multi-part content / tool calls — the v0 surface supports string message content only; richer OpenAI message shapes are a later add.

See also