Docs audit — llm gateway (routing, tiers, fallbacks) — 2026-08-09

[SebConejo]

Audited: mnfst/manifest routing/gateway surface (merged PRs over the last 30 days, plus the full current code path for routing, tiers, and fallbacks: #2468, #2573, #2600, #2639, #2158, #2546, #2417, #2374 among others), against the docs at 0db4da1.

Scope: one page, llm-gateway.mdx (https://manifest.build/docs/llm-gateway). Findings target only this page.

Findings

1 · [WRONG] The doc says a direct model ID on the Anthropic /v1/messages endpoint always routes through your tiers. It does not: a real model ID is a direct call on all three endpoints.

Page: https://manifest.build/docs/llm-gateway (Routing, the "Route a specific model" step)

What actually happens:

1. You send a request with a concrete model ID (not auto).
2. Manifest treats it as a direct call on every proxy endpoint: /v1/chat/completions, /v1/responses, and /v1/messages.
3. It forwards straight to that model's provider, with no tier lookup and no fallbacks.
4. The response carries X-Manifest-Tier: direct. Your default and custom tiers do not apply, even on /v1/messages.

Fix: replace the two endpoint sentences with one that lists all three endpoints as honoring a direct model ID. Suggested: "A direct model ID works on all three endpoints: /v1/chat/completions, /v1/responses, and the Anthropic /v1/messages."

Evidence (for the agent)

• llm-gateway.mdx:87 — "Direct model IDs work on the OpenAI-format endpoints (/v1/chat/completions and /v1/responses). Requests to the Anthropic /v1/messages endpoint always route through your default or custom tiers."
• proxy.service.ts:899-916 — the explicit-route branch (requestedModel !== 'auto' → resolveExplicitModel) runs before any endpoint-mode check; code comment at :900: "Every public proxy surface treats a concrete model as an explicit route." No /v1/messages-specific force-routing path exists.
• Impact: an Anthropic-SDK user who sends a concrete model to /v1/messages expecting tier routing and fallbacks gets a direct forward with neither. Their fallback config silently does not apply.

2 · [WRONG] The doc invents a "424 exception" to fallback and says an exhausted chain returns HTTP 424. Neither is true.

Page: https://manifest.build/docs/llm-gateway (Fallback, "What triggers a fallback" and "Response headers")

What actually happens:

1. Any provider response with status 400 or above triggers a fallback. There is no exception.
2. Manifest tries the next model in the tier's chain.
3. If every model fails, the response keeps the primary model's real error status (for example 500 or 529), not 424.
4. You detect an exhausted chain from the X-Manifest-Fallback-Exhausted: true header and the body code fallback_exhausted, not from a status code.

Options:

1. Rewrite both spots: state ">= 400 triggers a fallback" with no exception, and state that exhaustion keeps the last provider's real status and is marked by the X-Manifest-Fallback-Exhausted: true header plus fallback_exhausted body code. Also drop the "424 fallback exhausted" frontmatter keyword. Recommended.
2. Delete only the "with one exception: 424..." clause and change "returns 424" to "keeps the failing provider's status," leaving the header sentence.

Evidence (for the agent)

• llm-gateway.mdx:97 — "Any HTTP status code >= 400 triggers a fallback, with one exception: 424 (Failed Dependency) does not, since that's the status Manifest itself returns when the entire chain is exhausted. Retrying it would loop forever."
• llm-gateway.mdx:138 — "When the chain is exhausted, X-Manifest-Fallback-Exhausted: true is set and the request returns 424."
• llm-gateway.mdx:6 — frontmatter keyword "424 fallback exhausted".
• fallback-status-codes.ts:1-3 — shouldTriggerFallback is literally return status >= 400;, no 424 branch in production routing code (424 appears only in tests, one asserting it is no longer a sentinel).
• proxy.service.ts:1258-1265 — comment "All fallbacks exhausted — preserve the primary provider's real HTTP status"; response rebuilt with status: primaryStatus.
• proxy-response-handler.ts:403-410 — on exhaustion res.status(errorStatus) then sets X-Manifest-Fallback-Exhausted: true; body code is fallback_exhausted (or the provider's code).
• Impact: a client that detects chain exhaustion by looking for HTTP 424 never sees it; a reader designing around the "424 is not retried" rule reasons about behavior that does not exist.
• Suggested wording — line 97: "Any HTTP status code >= 400 triggers a fallback." Line 138 (last sentence): "When the chain is exhausted, X-Manifest-Fallback-Exhausted: true is set with body code fallback_exhausted, and the response keeps the primary model's real error status (for example 500 or 529)."

3 · [WRONG] The doc says a blocking hard-limit rule returns HTTP 429. It returns HTTP 200 with the block text as the assistant reply.

Page: https://manifest.build/docs/llm-gateway (Hard limits)

What actually happens:

1. Your blocking limit rule is over its threshold for the period.
2. Manifest stops the request before any provider is called, so nothing is spent.
3. It returns an HTTP 200 response whose assistant message carries the [🦚 Manifest M200] block text.
4. Your agent receives a normal 200 completion, not an error status. HTTP 429 belongs to the rate-limit codes (M201/M202/M203); HTTP 402 to the free-plan quota (M204).

Fix: change the sentence introducing the code block so the block is described as an HTTP 200 response whose assistant message carries the M200 text, not "HTTP 429." Suggested: "the gateway returns M200 as an HTTP 200 response whose assistant message carries the block text, so your agent keeps flowing instead of throwing:"

Evidence (for the agent)

• llm-gateway.mdx:142 — "the gateway rejects the request with M200 and HTTP 429:"
• proxy.service.ts:258-260 — enforceLimits(...) returns the M200 message, then buildFriendlyResponse(..., 'limit_exceeded', 'M200').
• proxy-friendly-response.ts:115-119 — that response is built with status: 200 (an assistant-message stub).
• error-codes.ts:48-59 — HTTP 429 is M201/M202/M203 (rate limits); M204 (free-plan quota) is HTTP 402.
• Impact: a user who wires error handling to catch 429 for hard-limit blocks treats the block as a successful completion and never triggers their limit-hit logic.

4 · [MISSING-CASE] The doc says any unlisted model ID returns M302. A model whose provider is connected and enabled is forwarded to the provider instead.

Page: https://manifest.build/docs/llm-gateway (Routing, "Route a specific model")

What actually happens:

1. You send a concrete model ID that is not in GET /v1/models.
2. If it maps to a connected, enabled provider route, Manifest forwards it to that provider (it may still succeed).
3. Only when no connected provider route can be identified does Manifest return M302.

This is low severity: the undocumented path yields success, not a broken request, and Seb already accepted the simpler M302 framing on self-hosted.mdx (decisions.md, issue #49 finding 3).

Options:

1. Leave "unlisted model ID -> M302" as an intentional simplification. Legitimate given the negligible harm.
2. Soften the absolute: "If you send a model ID that has no connected provider route, Manifest returns M302: Model not available." Recommended if any change is made: removes the false universal, adds no new concept.
3. Document the forwarded-model case explicitly. Most complete, but likely too much detail for this page.

Evidence (for the agent)

• llm-gateway.mdx:89 — "If you send an unlisted model ID, Manifest returns M302: Model not available. Send auto to use routing."
• proxy.service.ts:981-1002 and resolveConnectedExplicitModel at :1009-1049 (PR #2639) — an uncatalogued but connected/unambiguous model is forwarded to the provider; M302 (return null) fires only when no connected route can be identified.
• Impact: a user may avoid sending a new/uncatalogued model that would in fact succeed. Harm is a missed capability, not a failure.

5 · [IMAGE] The fallback configuration steps describe the Routing page and its drag-to-reorder control in prose only; a screenshot is missing.

Page: https://manifest.build/docs/llm-gateway (Fallback, "Configuration")

The image goes right after the Configuration <Steps> block (before the <Tip>). It must show the dashboard Routing page with one tier expanded: its primary model, the ordered fallback list, and the drag handles used to reorder them.

Evidence (for the agent)

• llm-gateway.mdx:114-124 — the <Steps> "Open Routing -> Select a tier -> Add up to 5 fallback models. Drag to reorder" describe a dashboard UI entirely in prose, no visual.
• frontend RoutingDefaultTierSection.tsx:164 — the Routing page renders the tier's primary model plus its "up to 5" ordered fallback list; a cloud-dashboard flow with no code a user can inspect.
• Impact: a first-time user must locate the Routing page, identify a tier, and find the drag-to-reorder fallback control from text alone.

---

Reply with the finding number and the chosen option
(e.g. "3 → option 2", or "3: other idea, let's discuss").
PRs are then made manually with the agent, never by the harness.
Rejected findings: reply "N → reject" and they will never be re-raised.