From 98a7a3065ca4da1fce4e690f54d9cee050d6a806 Mon Sep 17 00:00:00 2001 From: "Anna.Zhdan" Date: Mon, 2 Feb 2026 19:23:05 +0100 Subject: [PATCH 01/11] Add custom-llm-endpoint.mdx --- docs/rfds/custom-llm-endpoint.mdx | 291 ++++++++++++++++++++++++++++++ 1 file changed, 291 insertions(+) create mode 100644 docs/rfds/custom-llm-endpoint.mdx diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx new file mode 100644 index 00000000..1e60479a --- /dev/null +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -0,0 +1,291 @@ +--- +title: "Custom LLM Endpoint Configuration" +--- + +- Author(s): [@anna239](https://github.com/anna239) + +> **Note:** This RFD is very preliminary and intended to start a dialog about this feature. The proposed design may change significantly based on feedback and further discussion. + +## Elevator pitch + +> What are you proposing to change? + +Add the ability for clients to pass a custom LLM endpoint URL and authentication credentials to agents during initialization. This allows organizations to route all LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. + +## Status quo + +> How do things work today and what problems does this cause? Why would we change things? + +Currently, agents are configured with their own LLM endpoints and credentials, typically through environment variables or configuration files. This creates problems for: + +- **Client proxies**: Clients want to route agent traffic through their own proxies, f.i. for setting additional headers or logging +- **Enterprise deployments**: Organizations want to route LLM traffic through their own proxies for compliance, logging, or cost management +- **Self-hosted models**: Users running local LLM servers (vLLM, Ollama, etc.) cannot easily redirect agent traffic +- **API gateways**: Organizations using LLM gateways for rate limiting, caching, or multi-provider routing + +## Shiny future + +> How will things play out once this feature exists? + +Clients will be able to: +1. Pass a custom LLM endpoint URL and authentication header +2. Have all agent LLM requests automatically routed through that endpoint + +## Implementation details and plan + +> Tell me more about your implementation. What is your detailed implementation plan? + +We present three alternative approaches for discussion. + +### Alternative A: Pass in `initialize` request + +Add a new optional `llmEndpoint` property to `InitializeRequest`: + +```typescript +interface InitializeRequest { + // ... existing fields ... + + /** + * Custom LLM endpoint configuration. + * When provided, the agent should route all LLM requests through this endpoint. + * This configuration is per-process and should not be persisted to disk. + */ + llmEndpoint?: LlmEndpointConfig | null; +} + +interface LlmEndpointConfig { + /** Base URL for LLM API requests (e.g., "https://llm-proxy.corp.example.com/v1") */ + url: string; + + /** + * Value for the Authorization header. + * Will be sent as-is (e.g., "Bearer sk-..." or "Basic ..."). + */ + authHeader: string; + + /** Extension metadata */ + _meta?: Record; +} + +interface InitializeResponse { + // ... existing fields ... + + /** + * Echoed back when the agent accepts the custom LLM endpoint. + * If present in the response, the agent confirms it will use this endpoint. + * If absent (and was provided in request), the agent did not accept it. + */ + llmEndpoint?: LlmEndpointConfig | null; +} +``` + +#### JSON Schema Additions + +```json +{ + "$defs": { + "LlmEndpointConfig": { + "description": "Configuration for a custom LLM endpoint. This configuration is per-process and should not be persisted to disk.", + "properties": { + "url": { + "type": "string", + "description": "Base URL for LLM API requests." + }, + "authHeader": { + "type": ["string"], + "description": "Value for the Authorization header. Sent as-is." + }, + "_meta": { + "additionalProperties": true, + "type": ["object", "null"] + } + }, + "required": ["url", "authHeader"], + "type": "object" + } + } +} +``` + +#### Example Exchange + +**Initialize Request:** +```json +{ + "jsonrpc": "2.0", + "id": 1, + "method": "initialize", + "params": { + "clientInfo": { + "name": "MyIDE", + "version": "1.0.0" + }, + "protocolVersion": "2025-01-01", + "llmEndpoint": { + "url": "https://llm-gateway.corp.example.com/v1", + "authHeader": "Bearer temporary-token-abc123" + } + } +} +``` + +**Initialize Response (endpoint accepted):** +```json +{ + "jsonrpc": "2.0", + "id": 1, + "result": { + "protocolVersion": "2025-01-01", + "agentInfo": { + "name": "MyAgent", + "version": "2.0.0" + }, + "agentCapabilities": { + "sessionCapabilities": {} + }, + "llmEndpoint": { + "url": "https://llm-gateway.corp.example.com/v1", + "authHeader": "Bearer temporary-token-abc123" + } + } +} +``` + +**Initialize Response (endpoint not accepted):** +```json +{ + "jsonrpc": "2.0", + "id": 1, + "result": { + "protocolVersion": "2025-01-01", + "agentInfo": { + "name": "MyAgent", + "version": "2.0.0" + }, + "agentCapabilities": { + "sessionCapabilities": {} + } + } +} +``` + +#### Behavior + +1. **Confirmation via response**: When the agent accepts the `llmEndpoint`, it MUST echo it back in the `InitializeResponse`. If `llmEndpoint` is absent from the response, the client should assume the agent did not accept the custom endpoint configuration. + +2. **Per-process scope**: The `llmEndpoint` configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. + +3. **All LLM requests**: When `llmEndpoint` is accepted, the agent should route ALL LLM API requests through the specified URL, using the provided `authHeader` for authentication. + +4. **No mid-execution changes**: The endpoint cannot be changed after initialization. If a different endpoint is needed, a new agent process must be started. + +5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoint` from the response or return an error during initialization. + +### Alternative B: Dedicated `setLlmEndpoint` method + +Instead of passing the endpoint in `initialize`, introduce a dedicated method that can be called after initialization but before session creation. + +```typescript +interface SetLlmEndpointRequest { + /** Base URL for LLM API requests */ + url: string; + + /** Value for the Authorization header */ + authHeader: string; + + /** Extension metadata */ + _meta?: Record; +} + +interface SetLlmEndpointResponse { + /** Whether the endpoint was accepted */ + accepted: boolean; + + /** Extension metadata */ + _meta?: Record; +} +``` + +**Example flow:** +1. Client calls `initialize` +2. Client calls `setLlmEndpoint` with URL and auth header +3. Agent confirms acceptance +4. Client creates session + +**Trade-offs:** +- (+) Cleaner separation of concerns - initialization vs configuration +- (+) Could potentially allow changing endpoint between sessions (if desired) +- (-) Different sessions might want different endpoints, but we want a single endpoint per process + +### Alternative C: Registry-declared environment variables + +Declare in the Agent Registry which environment variables or command-line arguments should be used to pass the URL and token when starting the agent process. + +```json +{ + "name": "MyAgent", + "command": "my-agent", + "llmEndpointConfig": { + "urlEnvVar": "LLM_BASE_URL", + "authEnvVar": "LLM_AUTH_TOKEN" + } +} +``` + +Or with command-line arguments: + +```json +{ + "name": "MyAgent", + "command": "my-agent", + "llmEndpointConfig": { + "urlArg": "--llm-url", + "authArg": "--llm-token" + } +} +``` + +The client would then start the agent process with: +```bash +LLM_BASE_URL="https://gateway.example.com/v1" LLM_AUTH_TOKEN="Bearer sk-..." my-agent +# or +my-agent --llm-url "https://gateway.example.com/v1" --llm-token "Bearer sk-..." +``` + +**Trade-offs:** +- (+) Agents in the Registry can be marked as supporting custom endpoints +- (+) Easier to implement for agents that already read these values from environment variables +- (+) No protocol changes needed - uses existing process spawning mechanisms +- (+) Works even before ACP connection is established +- (-) Requires Registry support +- (-) Less dynamic - configuration is fixed at process start + +## Open questions + +### How should model availability be handled? + +When a custom endpoint is provided, it may only support a subset of models. For example, a self-hosted vLLM server might only have `llama-3-70b` available, while the agent normally advertises `claude-3-opus`, `gpt-4`, etc. + +### Should the agent advertise support for custom endpoints? + +Should there be a capability flag indicating the agent supports `llmEndpoint`? This would let clients know whether passing this configuration will have any effect. + +## Frequently asked questions + +> What questions have arisen over the course of authoring this document? + +### Why not pass endpoint when selecting a model? + +One option would be to pass the endpoint URL and credentials when the user selects a model (e.g., in `session/new` or a model selection method). + +Many agents throw authentication errors before the model selection happens. This makes the flow unreliable. + +### What if the agent doesn't support custom endpoints? + +If the agent doesn't support custom endpoints, it should omit `llmEndpoint` from the `InitializeResponse`. The client can then detect this by comparing the request and response: +- If `llmEndpoint` was sent but not returned, the agent did not accept it +- The client can then decide whether to proceed (using agent's default endpoint) or abort + +## Revision history + +- 2026-02-02: Initial draft - preliminary proposal to start discussion From 8c26ce79191724de00709bed4ad2ac64d47b2d57 Mon Sep 17 00:00:00 2001 From: "Anna.Zhdan" Date: Mon, 2 Feb 2026 19:32:26 +0100 Subject: [PATCH 02/11] make authHeader optional --- docs/rfds/custom-llm-endpoint.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 1e60479a..5d7b8d14 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -61,7 +61,7 @@ interface LlmEndpointConfig { * Value for the Authorization header. * Will be sent as-is (e.g., "Bearer sk-..." or "Basic ..."). */ - authHeader: string; + authHeader?: string | null; /** Extension metadata */ _meta?: Record; @@ -92,7 +92,7 @@ interface InitializeResponse { "description": "Base URL for LLM API requests." }, "authHeader": { - "type": ["string"], + "type": ["string", "null"], "description": "Value for the Authorization header. Sent as-is." }, "_meta": { @@ -100,7 +100,7 @@ interface InitializeResponse { "type": ["object", "null"] } }, - "required": ["url", "authHeader"], + "required": ["url"], "type": "object" } } @@ -175,7 +175,7 @@ interface InitializeResponse { 2. **Per-process scope**: The `llmEndpoint` configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. -3. **All LLM requests**: When `llmEndpoint` is accepted, the agent should route ALL LLM API requests through the specified URL, using the provided `authHeader` for authentication. +3. **All LLM requests**: When `llmEndpoint` is accepted, the agent should route ALL LLM API requests through the specified URL, using the provided `authHeader` for authentication (if provided). 4. **No mid-execution changes**: The endpoint cannot be changed after initialization. If a different endpoint is needed, a new agent process must be started. @@ -191,7 +191,7 @@ interface SetLlmEndpointRequest { url: string; /** Value for the Authorization header */ - authHeader: string; + authHeader?: string | null; /** Extension metadata */ _meta?: Record; From 6b8332f257c6ac80df78174175f8f05be2420907 Mon Sep 17 00:00:00 2001 From: "Anna.Zhdan" Date: Tue, 3 Feb 2026 18:08:08 +0100 Subject: [PATCH 03/11] support for multiple providers --- docs/rfds/custom-llm-endpoint.mdx | 166 ++++++++++++++---------------- 1 file changed, 79 insertions(+), 87 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 5d7b8d14..a824f022 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -10,7 +10,7 @@ title: "Custom LLM Endpoint Configuration" > What are you proposing to change? -Add the ability for clients to pass a custom LLM endpoint URL and authentication credentials to agents during initialization. This allows organizations to route all LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. +Add the ability for clients to pass custom LLM endpoint URLs and authentication credentials to agents during initialization, with support for multiple providers. This allows clients to route LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. ## Status quo @@ -28,29 +28,33 @@ Currently, agents are configured with their own LLM endpoints and credentials, t > How will things play out once this feature exists? Clients will be able to: -1. Pass a custom LLM endpoint URL and authentication header -2. Have all agent LLM requests automatically routed through that endpoint +1. Pass custom LLM endpoint URLs and authentication headers for different providers +2. Have agent LLM requests automatically routed through the appropriate endpoint based on provider ## Implementation details and plan > Tell me more about your implementation. What is your detailed implementation plan? -We present three alternative approaches for discussion. +We present two alternative approaches for discussion. ### Alternative A: Pass in `initialize` request -Add a new optional `llmEndpoint` property to `InitializeRequest`: +Add a new optional `llmEndpoints` property to `InitializeRequest` that maps provider identifiers to endpoint configurations: ```typescript +/** Well-known LLM provider identifiers */ +type LlmProvider = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; + interface InitializeRequest { // ... existing fields ... /** - * Custom LLM endpoint configuration. - * When provided, the agent should route all LLM requests through this endpoint. + * Custom LLM endpoint configurations per provider. + * When provided, the agent should route LLM requests to the appropriate endpoint + * based on the provider being used. * This configuration is per-process and should not be persisted to disk. */ - llmEndpoint?: LlmEndpointConfig | null; + llmEndpoints?: Record | null; } interface LlmEndpointConfig { @@ -71,11 +75,11 @@ interface InitializeResponse { // ... existing fields ... /** - * Echoed back when the agent accepts the custom LLM endpoint. - * If present in the response, the agent confirms it will use this endpoint. - * If absent (and was provided in request), the agent did not accept it. + * Echoed back with the providers the agent accepts. + * Only includes providers that the agent will actually use. + * If absent (and was provided in request), the agent did not accept any endpoints. */ - llmEndpoint?: LlmEndpointConfig | null; + llmEndpoints?: Record | null; } ``` @@ -85,7 +89,7 @@ interface InitializeResponse { { "$defs": { "LlmEndpointConfig": { - "description": "Configuration for a custom LLM endpoint. This configuration is per-process and should not be persisted to disk.", + "description": "Configuration for a custom LLM endpoint.", "properties": { "url": { "type": "string", @@ -102,6 +106,13 @@ interface InitializeResponse { }, "required": ["url"], "type": "object" + }, + "LlmEndpoints": { + "description": "Map of provider identifiers to endpoint configurations. This configuration is per-process and should not be persisted to disk.", + "type": "object", + "additionalProperties": { + "$ref": "#/$defs/LlmEndpointConfig" + } } } } @@ -121,15 +132,21 @@ interface InitializeResponse { "version": "1.0.0" }, "protocolVersion": "2025-01-01", - "llmEndpoint": { - "url": "https://llm-gateway.corp.example.com/v1", - "authHeader": "Bearer temporary-token-abc123" + "llmEndpoints": { + "anthropic": { + "url": "https://llm-gateway.corp.example.com/anthropic/v1", + "authHeader": "Bearer anthropic-token-abc123" + }, + "openai": { + "url": "https://llm-gateway.corp.example.com/openai/v1", + "authHeader": "Bearer openai-token-xyz789" + } } } } ``` -**Initialize Response (endpoint accepted):** +**Initialize Response (endpoints accepted):** ```json { "jsonrpc": "2.0", @@ -143,15 +160,19 @@ interface InitializeResponse { "agentCapabilities": { "sessionCapabilities": {} }, - "llmEndpoint": { - "url": "https://llm-gateway.corp.example.com/v1", - "authHeader": "Bearer temporary-token-abc123" + "llmEndpoints": { + "anthropic": { + "url": "https://llm-gateway.corp.example.com/anthropic/v1", + "authHeader": "Bearer anthropic-token-abc123" + } } } } ``` -**Initialize Response (endpoint not accepted):** +In this example, the agent only uses Anthropic, so it only echoes back that provider's configuration. + +**Initialize Response (endpoints not accepted):** ```json { "jsonrpc": "2.0", @@ -171,35 +192,32 @@ interface InitializeResponse { #### Behavior -1. **Confirmation via response**: When the agent accepts the `llmEndpoint`, it MUST echo it back in the `InitializeResponse`. If `llmEndpoint` is absent from the response, the client should assume the agent did not accept the custom endpoint configuration. +1. **Confirmation via response**: When the agent accepts any `llmEndpoints`, it MUST echo back the ones it will use in the `InitializeResponse`. If `llmEndpoints` is absent from the response, the client should assume the agent did not accept any custom endpoint configurations. -2. **Per-process scope**: The `llmEndpoint` configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. +2. **Per-process scope**: The `llmEndpoints` configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. -3. **All LLM requests**: When `llmEndpoint` is accepted, the agent should route ALL LLM API requests through the specified URL, using the provided `authHeader` for authentication (if provided). +3. **Provider-based routing**: The agent should route LLM requests to the appropriate endpoint based on the provider. If the agent uses a provider not in the provided map, it uses its default endpoint for that provider. -4. **No mid-execution changes**: The endpoint cannot be changed after initialization. If a different endpoint is needed, a new agent process must be started. +4. **No mid-execution changes**: The endpoints cannot be changed after initialization. If different endpoints are needed, a new agent process must be started. -5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoint` from the response or return an error during initialization. +5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoints` from the response or return an error during initialization. -### Alternative B: Dedicated `setLlmEndpoint` method +### Alternative B: Dedicated `setLlmEndpoints` method -Instead of passing the endpoint in `initialize`, introduce a dedicated method that can be called after initialization but before session creation. +Instead of passing the endpoints in `initialize`, introduce a dedicated method that can be called after initialization but before session creation. ```typescript -interface SetLlmEndpointRequest { - /** Base URL for LLM API requests */ - url: string; - - /** Value for the Authorization header */ - authHeader?: string | null; +interface SetLlmEndpointsRequest { + /** Map of provider identifiers to endpoint configurations */ + endpoints: Record; /** Extension metadata */ _meta?: Record; } -interface SetLlmEndpointResponse { - /** Whether the endpoint was accepted */ - accepted: boolean; +interface SetLlmEndpointsResponse { + /** Map of accepted provider endpoints */ + accepted: Record; /** Extension metadata */ _meta?: Record; @@ -208,59 +226,23 @@ interface SetLlmEndpointResponse { **Example flow:** 1. Client calls `initialize` -2. Client calls `setLlmEndpoint` with URL and auth header -3. Agent confirms acceptance +2. Client calls `setLlmEndpoints` with provider -> endpoint map +3. Agent confirms which providers were accepted 4. Client creates session **Trade-offs:** - (+) Cleaner separation of concerns - initialization vs configuration -- (+) Could potentially allow changing endpoint between sessions (if desired) -- (-) Different sessions might want different endpoints, but we want a single endpoint per process - -### Alternative C: Registry-declared environment variables - -Declare in the Agent Registry which environment variables or command-line arguments should be used to pass the URL and token when starting the agent process. - -```json -{ - "name": "MyAgent", - "command": "my-agent", - "llmEndpointConfig": { - "urlEnvVar": "LLM_BASE_URL", - "authEnvVar": "LLM_AUTH_TOKEN" - } -} -``` +- (+) Could potentially allow changing endpoints between sessions (if desired) +- (-) If we want a single set of endpoints per process, this is kind of confusing -Or with command-line arguments: - -```json -{ - "name": "MyAgent", - "command": "my-agent", - "llmEndpointConfig": { - "urlArg": "--llm-url", - "authArg": "--llm-token" - } -} -``` - -The client would then start the agent process with: -```bash -LLM_BASE_URL="https://gateway.example.com/v1" LLM_AUTH_TOKEN="Bearer sk-..." my-agent -# or -my-agent --llm-url "https://gateway.example.com/v1" --llm-token "Bearer sk-..." -``` +## Open questions -**Trade-offs:** -- (+) Agents in the Registry can be marked as supporting custom endpoints -- (+) Easier to implement for agents that already read these values from environment variables -- (+) No protocol changes needed - uses existing process spawning mechanisms -- (+) Works even before ACP connection is established -- (-) Requires Registry support -- (-) Less dynamic - configuration is fixed at process start +### How should provider identifiers be standardized? -## Open questions +We need to define a standard set of provider identifiers (e.g., `"anthropic"`, `"openai"`, `"google"`, `"amazon"`). Should this be: +- A fixed enum in the protocol specification? +- An extensible set with well-known values and support for custom strings? +- Defined in a separate registry/document that can be updated independently? ### How should model availability be handled? @@ -268,7 +250,7 @@ When a custom endpoint is provided, it may only support a subset of models. For ### Should the agent advertise support for custom endpoints? -Should there be a capability flag indicating the agent supports `llmEndpoint`? This would let clients know whether passing this configuration will have any effect. +Should there be a capability flag indicating the agent supports `llmEndpoints`? This would let clients know whether passing this configuration will have any effect. ## Frequently asked questions @@ -280,11 +262,21 @@ One option would be to pass the endpoint URL and credentials when the user selec Many agents throw authentication errors before the model selection happens. This makes the flow unreliable. +### Why not use environment variables or command-line arguments? + +One option would be to pass endpoint configuration via environment variables (like `OPENAI_API_BASE`) or command-line arguments when starting the agent process. + +This approach has significant drawbacks: +- With multiple providers, the configuration becomes complex JSON that is awkward to pass via command-line arguments +- Environment variables may be logged or visible to other processes, creating security concerns +- Requires knowledge of agent-specific variable names or argument formats +- No standardized way to confirm the agent accepted the configuration + ### What if the agent doesn't support custom endpoints? -If the agent doesn't support custom endpoints, it should omit `llmEndpoint` from the `InitializeResponse`. The client can then detect this by comparing the request and response: -- If `llmEndpoint` was sent but not returned, the agent did not accept it -- The client can then decide whether to proceed (using agent's default endpoint) or abort +If the agent doesn't support custom endpoints, it should omit `llmEndpoints` from the `InitializeResponse`. The client can then detect this by comparing the request and response: +- If `llmEndpoints` was sent but not returned, the agent did not accept any +- The client can then decide whether to proceed (using agent's default endpoints) or abort ## Revision history From 42ddd4cd192883794c6a35093886e65810a71526 Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Wed, 4 Mar 2026 22:14:34 +0200 Subject: [PATCH 04/11] docs: revise custom LLM endpoint RFD to use dedicated setLlmEndpoints method --- docs/rfds/custom-llm-endpoint.mdx | 238 ++++++++++++++++-------------- 1 file changed, 124 insertions(+), 114 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index a824f022..28ba843e 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -2,7 +2,7 @@ title: "Custom LLM Endpoint Configuration" --- -- Author(s): [@anna239](https://github.com/anna239) +- Author(s): [@anna239](https://github.com/anna239), [@xtmq](https://github.com/xtmq) > **Note:** This RFD is very preliminary and intended to start a dialog about this feature. The proposed design may change significantly based on feedback and further discussion. @@ -10,7 +10,7 @@ title: "Custom LLM Endpoint Configuration" > What are you proposing to change? -Add the ability for clients to pass custom LLM endpoint URLs and authentication credentials to agents during initialization, with support for multiple providers. This allows clients to route LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. +Add the ability for clients to pass custom LLM endpoint URLs and authentication credentials to agents via a dedicated `setLlmEndpoints` method, with support for multiple providers. This allows clients to route LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. ## Status quo @@ -28,58 +28,120 @@ Currently, agents are configured with their own LLM endpoints and credentials, t > How will things play out once this feature exists? Clients will be able to: -1. Pass custom LLM endpoint URLs and authentication headers for different providers -2. Have agent LLM requests automatically routed through the appropriate endpoint based on provider +1. Discover whether an agent supports custom LLM endpoints via capabilities during initialization +2. Perform agent configuration, including authorization based on this knowledge +3. Pass custom LLM endpoint URLs and headers for different providers via a dedicated method +4. Have agent LLM requests automatically routed through the appropriate endpoint based on provider ## Implementation details and plan > Tell me more about your implementation. What is your detailed implementation plan? -We present two alternative approaches for discussion. +### Intended flow -### Alternative A: Pass in `initialize` request +The design uses a two-step approach: capability discovery during initialization, followed by endpoint configuration via a dedicated method. This enables the following flow: -Add a new optional `llmEndpoints` property to `InitializeRequest` that maps provider identifiers to endpoint configurations: +```mermaid +sequenceDiagram + participant Client + participant Agent -```typescript -/** Well-known LLM provider identifiers */ -type LlmProvider = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; + Client->>Agent: initialize + Note right of Agent: Agent reports capabilities,
including llmEndpoints support + Agent-->>Client: initialize response
(agentCapabilities.llmEndpoints = true) + + Note over Client: Client sees llmEndpoints capability.
Performs configuration / authorization
based on this knowledge. + + Client->>Agent: setLlmEndpoints + Agent-->>Client: setLlmEndpoints response
(accepted providers) + + Note over Client,Agent: Ready for session setup + Client->>Agent: session/new +``` -interface InitializeRequest { +1. **Initialization**: The client calls `initialize`. The agent responds with its capabilities, including an `llmEndpoints` flag indicating support for custom endpoint configuration. +2. **Client-side decision**: The client inspects the capability. If the agent supports `llmEndpoints`, the client can perform authorization, resolve credentials, or configure endpoints accordingly. If the agent does not support it, the client falls back to a different authorization and configuration strategy. +3. **Endpoint configuration**: The client calls `setLlmEndpoints` with provider-to-endpoint mappings. The agent responds with the subset of providers it accepted. +4. **Session creation**: The client proceeds to create a session. + +### Capability advertisement + +The agent advertises support for custom LLM endpoints via a new capability flag in `agentCapabilities`: + +```typescript +interface AgentCapabilities { // ... existing fields ... - /** - * Custom LLM endpoint configurations per provider. - * When provided, the agent should route LLM requests to the appropriate endpoint - * based on the provider being used. - * This configuration is per-process and should not be persisted to disk. - */ - llmEndpoints?: Record | null; + /** Whether the agent supports custom LLM endpoint configuration via setLlmEndpoints */ + llmEndpoints?: boolean; } +``` + +**Initialize Response example:** +```json +{ + "jsonrpc": "2.0", + "id": 0, + "result": { + "protocolVersion": 1, + "agentInfo": { + "name": "MyAgent", + "version": "2.0.0" + }, + "agentCapabilities": { + "llmEndpoints": true, + "sessionCapabilities": {} + } + } +} +``` + +### `setLlmEndpoints` method + +A dedicated method that can be called after initialization but before session creation. + +```typescript +/** Well-known LLM provider identifiers */ +type LlmProvider = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; interface LlmEndpointConfig { /** Base URL for LLM API requests (e.g., "https://llm-proxy.corp.example.com/v1") */ url: string; /** - * Value for the Authorization header. - * Will be sent as-is (e.g., "Bearer sk-..." or "Basic ..."). + * Additional HTTP headers to include in LLM API requests. + * Each entry is a header name mapped to its value. + * Common use cases include Authorization, custom routing, or tracing headers. */ - authHeader?: string | null; + headers?: Record | null; /** Extension metadata */ _meta?: Record; } -interface InitializeResponse { - // ... existing fields ... +interface SetLlmEndpointsRequest { + /** + * Custom LLM endpoint configurations per provider. + * When provided, the agent should route LLM requests to the appropriate endpoint + * based on the provider being used. + * This configuration is per-process and should not be persisted to disk. + */ + endpoints: Record; + + /** Extension metadata */ + _meta?: Record; +} +interface SetLlmEndpointsResponse { /** * Echoed back with the providers the agent accepts. * Only includes providers that the agent will actually use. - * If absent (and was provided in request), the agent did not accept any endpoints. + * If empty, the agent did not accept any endpoints. */ - llmEndpoints?: Record | null; + accepted: Record; + + /** Extension metadata */ + _meta?: Record; } ``` @@ -95,9 +157,12 @@ interface InitializeResponse { "type": "string", "description": "Base URL for LLM API requests." }, - "authHeader": { - "type": ["string", "null"], - "description": "Value for the Authorization header. Sent as-is." + "headers": { + "type": ["object", "null"], + "description": "Additional HTTP headers to include in LLM API requests.", + "additionalProperties": { + "type": "string" + } }, "_meta": { "additionalProperties": true, @@ -120,120 +185,66 @@ interface InitializeResponse { #### Example Exchange -**Initialize Request:** +**setLlmEndpoints Request:** ```json { "jsonrpc": "2.0", - "id": 1, - "method": "initialize", + "id": 2, + "method": "setLlmEndpoints", "params": { - "clientInfo": { - "name": "MyIDE", - "version": "1.0.0" - }, - "protocolVersion": "2025-01-01", - "llmEndpoints": { + "endpoints": { "anthropic": { "url": "https://llm-gateway.corp.example.com/anthropic/v1", - "authHeader": "Bearer anthropic-token-abc123" + "headers": { + "Authorization": "Bearer anthropic-token-abc123", + "X-Request-Source": "my-ide" + } }, "openai": { "url": "https://llm-gateway.corp.example.com/openai/v1", - "authHeader": "Bearer openai-token-xyz789" + "headers": { + "Authorization": "Bearer openai-token-xyz789" + } } } } } ``` -**Initialize Response (endpoints accepted):** +**setLlmEndpoints Response:** ```json { "jsonrpc": "2.0", - "id": 1, + "id": 2, "result": { - "protocolVersion": "2025-01-01", - "agentInfo": { - "name": "MyAgent", - "version": "2.0.0" - }, - "agentCapabilities": { - "sessionCapabilities": {} - }, - "llmEndpoints": { + "accepted": { "anthropic": { "url": "https://llm-gateway.corp.example.com/anthropic/v1", - "authHeader": "Bearer anthropic-token-abc123" + "headers": { + "Authorization": "Bearer anthropic-token-abc123", + "X-Request-Source": "my-ide" + } } } } } ``` -In this example, the agent only uses Anthropic, so it only echoes back that provider's configuration. - -**Initialize Response (endpoints not accepted):** -```json -{ - "jsonrpc": "2.0", - "id": 1, - "result": { - "protocolVersion": "2025-01-01", - "agentInfo": { - "name": "MyAgent", - "version": "2.0.0" - }, - "agentCapabilities": { - "sessionCapabilities": {} - } - } -} -``` +In this example, the agent only uses Anthropic, so it only accepts that provider's configuration. #### Behavior -1. **Confirmation via response**: When the agent accepts any `llmEndpoints`, it MUST echo back the ones it will use in the `InitializeResponse`. If `llmEndpoints` is absent from the response, the client should assume the agent did not accept any custom endpoint configurations. - -2. **Per-process scope**: The `llmEndpoints` configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. - -3. **Provider-based routing**: The agent should route LLM requests to the appropriate endpoint based on the provider. If the agent uses a provider not in the provided map, it uses its default endpoint for that provider. - -4. **No mid-execution changes**: The endpoints cannot be changed after initialization. If different endpoints are needed, a new agent process must be started. - -5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoints` from the response or return an error during initialization. - -### Alternative B: Dedicated `setLlmEndpoints` method - -Instead of passing the endpoints in `initialize`, introduce a dedicated method that can be called after initialization but before session creation. - -```typescript -interface SetLlmEndpointsRequest { - /** Map of provider identifiers to endpoint configurations */ - endpoints: Record; +1. **Capability discovery**: The agent MUST advertise `llmEndpoints: true` in `agentCapabilities` if it supports the `setLlmEndpoints` method. Clients SHOULD check this capability before calling the method. - /** Extension metadata */ - _meta?: Record; -} +2. **Timing**: The `setLlmEndpoints` method MUST be called after `initialize` and before `session/new`. Calling it during an active session is undefined behavior. All subsequent sessions will use the newly configured endpoints. -interface SetLlmEndpointsResponse { - /** Map of accepted provider endpoints */ - accepted: Record; +3. **Confirmation via response**: The agent MUST respond with the `accepted` map containing only the providers it will actually use. If the agent accepts none, the `accepted` map SHOULD be empty. - /** Extension metadata */ - _meta?: Record; -} -``` +4. **Per-process scope**: The endpoint configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. -**Example flow:** -1. Client calls `initialize` -2. Client calls `setLlmEndpoints` with provider -> endpoint map -3. Agent confirms which providers were accepted -4. Client creates session +5. **Provider-based routing**: The agent should route LLM requests to the appropriate endpoint based on the provider. If the agent uses a provider not in the provided map, it uses its default endpoint for that provider. -**Trade-offs:** -- (+) Cleaner separation of concerns - initialization vs configuration -- (+) Could potentially allow changing endpoints between sessions (if desired) -- (-) If we want a single set of endpoints per process, this is kind of confusing +6. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should not advertise the `llmEndpoints` capability. ## Open questions @@ -248,14 +259,14 @@ We need to define a standard set of provider identifiers (e.g., `"anthropic"`, ` When a custom endpoint is provided, it may only support a subset of models. For example, a self-hosted vLLM server might only have `llama-3-70b` available, while the agent normally advertises `claude-3-opus`, `gpt-4`, etc. -### Should the agent advertise support for custom endpoints? - -Should there be a capability flag indicating the agent supports `llmEndpoints`? This would let clients know whether passing this configuration will have any effect. - ## Frequently asked questions > What questions have arisen over the course of authoring this document? +### Why not pass endpoints in the `initialize` request? + +Passing endpoints directly in `initialize` would require the client to have already resolved credentials and configured endpoints before knowing whether the agent supports this feature. In practice, the client needs to inspect the agent's capabilities first to decide its authorization strategy — for example, whether to route through a corporate proxy or use direct credentials. A dedicated method after initialization solves this chicken-and-egg problem and keeps capability negotiation separate from endpoint configuration. + ### Why not pass endpoint when selecting a model? One option would be to pass the endpoint URL and credentials when the user selects a model (e.g., in `session/new` or a model selection method). @@ -274,10 +285,9 @@ This approach has significant drawbacks: ### What if the agent doesn't support custom endpoints? -If the agent doesn't support custom endpoints, it should omit `llmEndpoints` from the `InitializeResponse`. The client can then detect this by comparing the request and response: -- If `llmEndpoints` was sent but not returned, the agent did not accept any -- The client can then decide whether to proceed (using agent's default endpoints) or abort +If the agent doesn't support custom endpoints, it will not advertise `llmEndpoints: true` in `agentCapabilities` during initialization. The client can detect this and choose an alternative authorization and configuration strategy, or proceed using the agent's default endpoints. ## Revision history +- 2026-03-04: Revised to use dedicated `setLlmEndpoints` method with capability advertisement - 2026-02-02: Initial draft - preliminary proposal to start discussion From f7548d9b805d85257570fbecfe70b26fab0d45cd Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Thu, 5 Mar 2026 10:26:42 +0200 Subject: [PATCH 05/11] docs: update custom LLM endpoint RFD to include provider-based capabilities --- docs/rfds/custom-llm-endpoint.mdx | 79 +++++++++++++++---------------- 1 file changed, 39 insertions(+), 40 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 28ba843e..773b5eda 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -48,32 +48,50 @@ sequenceDiagram Client->>Agent: initialize Note right of Agent: Agent reports capabilities,
including llmEndpoints support - Agent-->>Client: initialize response
(agentCapabilities.llmEndpoints = true) + Agent-->>Client: initialize response
(agentCapabilities.llmEndpoints.providers) - Note over Client: Client sees llmEndpoints capability.
Performs configuration / authorization
based on this knowledge. + Note over Client: Client sees supported providers.
Performs configuration / authorization
based on this knowledge. Client->>Agent: setLlmEndpoints - Agent-->>Client: setLlmEndpoints response
(accepted providers) + Agent-->>Client: setLlmEndpoints response Note over Client,Agent: Ready for session setup Client->>Agent: session/new ``` -1. **Initialization**: The client calls `initialize`. The agent responds with its capabilities, including an `llmEndpoints` flag indicating support for custom endpoint configuration. -2. **Client-side decision**: The client inspects the capability. If the agent supports `llmEndpoints`, the client can perform authorization, resolve credentials, or configure endpoints accordingly. If the agent does not support it, the client falls back to a different authorization and configuration strategy. -3. **Endpoint configuration**: The client calls `setLlmEndpoints` with provider-to-endpoint mappings. The agent responds with the subset of providers it accepted. +1. **Initialization**: The client calls `initialize`. The agent responds with its capabilities, including an `llmEndpoints` object listing supported providers. +2. **Client-side decision**: The client inspects the supported providers list. If the agent lists providers in `llmEndpoints.providers`, the client can perform authorization, resolve credentials, or configure endpoints for those specific providers. If `llmEndpoints` is absent or the providers list is empty, the client falls back to a different authorization and configuration strategy. +3. **Endpoint configuration**: The client calls `setLlmEndpoints` with endpoint configurations for the supported providers. 4. **Session creation**: The client proceeds to create a session. ### Capability advertisement -The agent advertises support for custom LLM endpoints via a new capability flag in `agentCapabilities`: +The agent advertises support for custom LLM endpoints and lists its supported providers via a new `llmEndpoints` capability in `agentCapabilities`: ```typescript +/** Well-known LLM provider identifiers */ +type LlmProvider = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; + +interface LlmEndpointsCapability { + /** + * Map of supported provider identifiers. + * The client should only configure endpoints for providers listed here. + */ + providers: Record; + + /** Extension metadata */ + _meta?: Record; +} + interface AgentCapabilities { // ... existing fields ... - /** Whether the agent supports custom LLM endpoint configuration via setLlmEndpoints */ - llmEndpoints?: boolean; + /** + * Custom LLM endpoint support. + * If present with a non-empty providers map, the agent supports the setLlmEndpoints method. + * If absent or providers is empty, the agent does not support custom endpoints. + */ + llmEndpoints?: LlmEndpointsCapability; } ``` @@ -89,7 +107,12 @@ interface AgentCapabilities { "version": "2.0.0" }, "agentCapabilities": { - "llmEndpoints": true, + "llmEndpoints": { + "providers": { + "anthropic": {}, + "openai": {} + } + }, "sessionCapabilities": {} } } @@ -101,9 +124,6 @@ interface AgentCapabilities { A dedicated method that can be called after initialization but before session creation. ```typescript -/** Well-known LLM provider identifiers */ -type LlmProvider = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; - interface LlmEndpointConfig { /** Base URL for LLM API requests (e.g., "https://llm-proxy.corp.example.com/v1") */ url: string; @@ -133,13 +153,6 @@ interface SetLlmEndpointsRequest { } interface SetLlmEndpointsResponse { - /** - * Echoed back with the providers the agent accepts. - * Only includes providers that the agent will actually use. - * If empty, the agent did not accept any endpoints. - */ - accepted: Record; - /** Extension metadata */ _meta?: Record; } @@ -216,35 +229,21 @@ interface SetLlmEndpointsResponse { { "jsonrpc": "2.0", "id": 2, - "result": { - "accepted": { - "anthropic": { - "url": "https://llm-gateway.corp.example.com/anthropic/v1", - "headers": { - "Authorization": "Bearer anthropic-token-abc123", - "X-Request-Source": "my-ide" - } - } - } - } + "result": {} } ``` -In this example, the agent only uses Anthropic, so it only accepts that provider's configuration. - #### Behavior -1. **Capability discovery**: The agent MUST advertise `llmEndpoints: true` in `agentCapabilities` if it supports the `setLlmEndpoints` method. Clients SHOULD check this capability before calling the method. +1. **Capability discovery**: The agent MUST list its supported providers in `agentCapabilities.llmEndpoints.providers` if it supports the `setLlmEndpoints` method. Clients SHOULD only send endpoint configurations for providers listed there. 2. **Timing**: The `setLlmEndpoints` method MUST be called after `initialize` and before `session/new`. Calling it during an active session is undefined behavior. All subsequent sessions will use the newly configured endpoints. -3. **Confirmation via response**: The agent MUST respond with the `accepted` map containing only the providers it will actually use. If the agent accepts none, the `accepted` map SHOULD be empty. - -4. **Per-process scope**: The endpoint configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. +3. **Per-process scope**: The endpoint configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. -5. **Provider-based routing**: The agent should route LLM requests to the appropriate endpoint based on the provider. If the agent uses a provider not in the provided map, it uses its default endpoint for that provider. +4. **Provider-based routing**: The agent should route LLM requests to the appropriate endpoint based on the provider. If the agent uses a provider not in the provided map, it uses its default endpoint for that provider. -6. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should not advertise the `llmEndpoints` capability. +5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoints` from capabilities or return an empty providers map. ## Open questions @@ -285,7 +284,7 @@ This approach has significant drawbacks: ### What if the agent doesn't support custom endpoints? -If the agent doesn't support custom endpoints, it will not advertise `llmEndpoints: true` in `agentCapabilities` during initialization. The client can detect this and choose an alternative authorization and configuration strategy, or proceed using the agent's default endpoints. +If the agent doesn't support custom endpoints, `llmEndpoints` will be absent from `agentCapabilities` (or its `providers` map will be empty). The client can detect this and choose an alternative authorization and configuration strategy, or proceed using the agent's default endpoints. ## Revision history From a9c4fadf341c54da7ce5d534657d00a88f598e1a Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Thu, 5 Mar 2026 19:50:56 +0200 Subject: [PATCH 06/11] docs: update custom LLM endpoint RFD - wording --- docs/rfds/custom-llm-endpoint.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 773b5eda..dd98eaf8 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -237,7 +237,7 @@ interface SetLlmEndpointsResponse { 1. **Capability discovery**: The agent MUST list its supported providers in `agentCapabilities.llmEndpoints.providers` if it supports the `setLlmEndpoints` method. Clients SHOULD only send endpoint configurations for providers listed there. -2. **Timing**: The `setLlmEndpoints` method MUST be called after `initialize` and before `session/new`. Calling it during an active session is undefined behavior. All subsequent sessions will use the newly configured endpoints. +2. **Timing**: The `setLlmEndpoints` method MUST be called after `initialize` and before `session/new`. Calling this MAY NOT affect currently running sessions. Agents MUST apply these settings to any sessions created or loaded after this has been called. 3. **Per-process scope**: The endpoint configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. From 4091d1dd94ba030f3d8f39090f216d2fa04cae8c Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Sat, 7 Mar 2026 15:24:14 +0200 Subject: [PATCH 07/11] docs: rename llm provider to llm protocol in custom LLM endpoint RFD --- docs/rfds/custom-llm-endpoint.mdx | 59 +++++++++++++++++-------------- 1 file changed, 32 insertions(+), 27 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index dd98eaf8..6647fafc 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -10,7 +10,7 @@ title: "Custom LLM Endpoint Configuration" > What are you proposing to change? -Add the ability for clients to pass custom LLM endpoint URLs and authentication credentials to agents via a dedicated `setLlmEndpoints` method, with support for multiple providers. This allows clients to route LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. +Add the ability for clients to pass custom LLM endpoint URLs and authentication credentials to agents via a dedicated `setLlmEndpoints` method, with support for multiple LLM protocols. This allows clients to route LLM requests through their own infrastructure (proxies, gateways, or self-hosted models) without agents needing to know about this configuration in advance. ## Status quo @@ -30,8 +30,8 @@ Currently, agents are configured with their own LLM endpoints and credentials, t Clients will be able to: 1. Discover whether an agent supports custom LLM endpoints via capabilities during initialization 2. Perform agent configuration, including authorization based on this knowledge -3. Pass custom LLM endpoint URLs and headers for different providers via a dedicated method -4. Have agent LLM requests automatically routed through the appropriate endpoint based on provider +3. Pass custom LLM endpoint URLs and headers for different LLM protocols via a dedicated method +4. Have agent LLM requests automatically routed through the appropriate endpoint based on the LLM protocol ## Implementation details and plan @@ -48,9 +48,9 @@ sequenceDiagram Client->>Agent: initialize Note right of Agent: Agent reports capabilities,
including llmEndpoints support - Agent-->>Client: initialize response
(agentCapabilities.llmEndpoints.providers) + Agent-->>Client: initialize response
(agentCapabilities.llmEndpoints.protocols) - Note over Client: Client sees supported providers.
Performs configuration / authorization
based on this knowledge. + Note over Client: Client sees supported protocols.
Performs configuration / authorization
based on this knowledge. Client->>Agent: setLlmEndpoints Agent-->>Client: setLlmEndpoints response @@ -59,25 +59,30 @@ sequenceDiagram Client->>Agent: session/new ``` -1. **Initialization**: The client calls `initialize`. The agent responds with its capabilities, including an `llmEndpoints` object listing supported providers. -2. **Client-side decision**: The client inspects the supported providers list. If the agent lists providers in `llmEndpoints.providers`, the client can perform authorization, resolve credentials, or configure endpoints for those specific providers. If `llmEndpoints` is absent or the providers list is empty, the client falls back to a different authorization and configuration strategy. -3. **Endpoint configuration**: The client calls `setLlmEndpoints` with endpoint configurations for the supported providers. +1. **Initialization**: The client calls `initialize`. The agent responds with its capabilities, including an `llmEndpoints` object listing supported LLM protocols. +2. **Client-side decision**: The client inspects the supported protocols list. If the agent lists protocols in `llmEndpoints.protocols`, the client can perform authorization, resolve credentials, or configure endpoints for those specific protocols. If `llmEndpoints` is absent or the protocols list is empty, the client falls back to a different authorization and configuration strategy. +3. **Endpoint configuration**: The client calls `setLlmEndpoints` with endpoint configurations for the supported protocols. 4. **Session creation**: The client proceeds to create a session. ### Capability advertisement -The agent advertises support for custom LLM endpoints and lists its supported providers via a new `llmEndpoints` capability in `agentCapabilities`: +The agent advertises support for custom LLM endpoints and lists its supported LLM protocols via a new `llmEndpoints` capability in `agentCapabilities`: ```typescript -/** Well-known LLM provider identifiers */ -type LlmProvider = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; +/** + * Well-known LLM protocol identifiers. + * Each identifier represents an API compatibility level, not a specific vendor. + * For example, "openai" means any endpoint implementing the OpenAI-compatible API + * (including proxies, gateways, and self-hosted servers like vLLM or Ollama). + */ +type LlmProtocol = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; interface LlmEndpointsCapability { /** - * Map of supported provider identifiers. - * The client should only configure endpoints for providers listed here. + * Map of supported LLM protocol identifiers. + * The client should only configure endpoints for protocols listed here. */ - providers: Record; + protocols: Record; /** Extension metadata */ _meta?: Record; @@ -88,8 +93,8 @@ interface AgentCapabilities { /** * Custom LLM endpoint support. - * If present with a non-empty providers map, the agent supports the setLlmEndpoints method. - * If absent or providers is empty, the agent does not support custom endpoints. + * If present with a non-empty protocols map, the agent supports the setLlmEndpoints method. + * If absent or protocols is empty, the agent does not support custom endpoints. */ llmEndpoints?: LlmEndpointsCapability; } @@ -108,7 +113,7 @@ interface AgentCapabilities { }, "agentCapabilities": { "llmEndpoints": { - "providers": { + "protocols": { "anthropic": {}, "openai": {} } @@ -141,12 +146,12 @@ interface LlmEndpointConfig { interface SetLlmEndpointsRequest { /** - * Custom LLM endpoint configurations per provider. + * Custom LLM endpoint configurations per LLM protocol. * When provided, the agent should route LLM requests to the appropriate endpoint - * based on the provider being used. + * based on the protocol being used. * This configuration is per-process and should not be persisted to disk. */ - endpoints: Record; + endpoints: Record; /** Extension metadata */ _meta?: Record; @@ -186,7 +191,7 @@ interface SetLlmEndpointsResponse { "type": "object" }, "LlmEndpoints": { - "description": "Map of provider identifiers to endpoint configurations. This configuration is per-process and should not be persisted to disk.", + "description": "Map of LLM protocol identifiers to endpoint configurations. This configuration is per-process and should not be persisted to disk.", "type": "object", "additionalProperties": { "$ref": "#/$defs/LlmEndpointConfig" @@ -235,21 +240,21 @@ interface SetLlmEndpointsResponse { #### Behavior -1. **Capability discovery**: The agent MUST list its supported providers in `agentCapabilities.llmEndpoints.providers` if it supports the `setLlmEndpoints` method. Clients SHOULD only send endpoint configurations for providers listed there. +1. **Capability discovery**: The agent MUST list its supported protocols in `agentCapabilities.llmEndpoints.protocols` if it supports the `setLlmEndpoints` method. Clients SHOULD only send endpoint configurations for protocols listed there. 2. **Timing**: The `setLlmEndpoints` method MUST be called after `initialize` and before `session/new`. Calling this MAY NOT affect currently running sessions. Agents MUST apply these settings to any sessions created or loaded after this has been called. 3. **Per-process scope**: The endpoint configuration applies to the entire agent process lifetime. It should not be stored to disk or persist beyond the process. -4. **Provider-based routing**: The agent should route LLM requests to the appropriate endpoint based on the provider. If the agent uses a provider not in the provided map, it uses its default endpoint for that provider. +4. **Protocol-based routing**: The agent should route LLM requests to the appropriate endpoint based on the LLM protocol. If the agent uses a protocol not in the provided map, it uses its default endpoint for that protocol. -5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoints` from capabilities or return an empty providers map. +5. **Agent discretion**: If an agent cannot support custom endpoints (e.g., uses a proprietary API), it should omit `llmEndpoints` from capabilities or return an empty protocols map. ## Open questions -### How should provider identifiers be standardized? +### How should protocol identifiers be standardized? -We need to define a standard set of provider identifiers (e.g., `"anthropic"`, `"openai"`, `"google"`, `"amazon"`). Should this be: +We need to define a standard set of LLM protocol identifiers (e.g., `"anthropic"`, `"openai"`, `"google"`, `"amazon"`). Should this be: - A fixed enum in the protocol specification? - An extensible set with well-known values and support for custom strings? - Defined in a separate registry/document that can be updated independently? @@ -284,7 +289,7 @@ This approach has significant drawbacks: ### What if the agent doesn't support custom endpoints? -If the agent doesn't support custom endpoints, `llmEndpoints` will be absent from `agentCapabilities` (or its `providers` map will be empty). The client can detect this and choose an alternative authorization and configuration strategy, or proceed using the agent's default endpoints. +If the agent doesn't support custom endpoints, `llmEndpoints` will be absent from `agentCapabilities` (or its `protocols` map will be empty). The client can detect this and choose an alternative authorization and configuration strategy, or proceed using the agent's default endpoints. ## Revision history From d5fbf9676247defe7a77cd90b3a37b0482deb8f5 Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Sat, 7 Mar 2026 15:35:39 +0200 Subject: [PATCH 08/11] docs: make LlmProtocol an open string type with well-known values in custom LLM endpoint RFD --- docs/rfds/custom-llm-endpoint.mdx | 47 +++++++++++++++++++++++++------ 1 file changed, 39 insertions(+), 8 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 6647fafc..08dc1a0e 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -70,12 +70,14 @@ The agent advertises support for custom LLM endpoints and lists its supported LL ```typescript /** - * Well-known LLM protocol identifiers. - * Each identifier represents an API compatibility level, not a specific vendor. + * LLM protocol identifier representing an API compatibility level, not a specific vendor. * For example, "openai" means any endpoint implementing the OpenAI-compatible API * (including proxies, gateways, and self-hosted servers like vLLM or Ollama). + * + * Well-known values: "anthropic", "openai", "google", "amazon". + * Custom protocol identifiers are allowed for regional or emerging LLM APIs not covered by the well-known set. */ -type LlmProtocol = "anthropic" | "openai" | "google" | "amazon" | "will be added later"; +type LlmProtocol = string; interface LlmEndpointsCapability { /** @@ -252,12 +254,9 @@ interface SetLlmEndpointsResponse { ## Open questions -### How should protocol identifiers be standardized? +### ~~How should protocol identifiers be standardized?~~ Resolved -We need to define a standard set of LLM protocol identifiers (e.g., `"anthropic"`, `"openai"`, `"google"`, `"amazon"`). Should this be: -- A fixed enum in the protocol specification? -- An extensible set with well-known values and support for custom strings? -- Defined in a separate registry/document that can be updated independently? +Protocol identifiers are plain strings with a set of well-known values (`"anthropic"`, `"openai"`, `"google"`, `"amazon"`). Custom identifiers are allowed for regional or emerging LLM APIs not covered by the well-known set. Agents advertise the protocol identifiers they understand; clients match against them. ### How should model availability be handled? @@ -267,6 +266,38 @@ When a custom endpoint is provided, it may only support a subset of models. For > What questions have arisen over the course of authoring this document? +### Why is `LlmProtocol` a plain string instead of a fixed enum? + +The protocol wire format uses plain strings to keep the specification stable and avoid blocking adoption of new LLM APIs. A fixed enum would require a spec update every time a new protocol emerges. + +Instead, well-known protocol identifiers are provided by SDKs as convenience constants: + +**TypeScript** — an open string type with predefined values: +```typescript +type LlmProtocol = string & {}; + +const LlmProtocols = { + Anthropic: "anthropic", + OpenAI: "openai", + Google: "google", + Amazon: "amazon", +} as const; +``` + +**Kotlin** — a string wrapper with predefined companion objects: +```kotlin +value class LlmProtocol(val value: String) { + companion object { + val Anthropic = LlmProtocol("anthropic") + val OpenAI = LlmProtocol("openai") + val Google = LlmProtocol("google") + val Amazon = LlmProtocol("amazon") + } +} +``` + +This way SDK users get discoverability and autocomplete for well-known protocols while remaining free to use any custom string for protocols not yet in the predefined set. + ### Why not pass endpoints in the `initialize` request? Passing endpoints directly in `initialize` would require the client to have already resolved credentials and configured endpoints before knowing whether the agent supports this feature. In practice, the client needs to inspect the agent's capabilities first to decide its authorization strategy — for example, whether to route through a corporate proxy or use direct credentials. A dedicated method after initialization solves this chicken-and-egg problem and keeps capability negotiation separate from endpoint configuration. From 09a8de0ee3813701fb0595932f2224b414a65484 Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Sat, 7 Mar 2026 17:58:16 +0200 Subject: [PATCH 09/11] docs: resolve model availability open question in custom LLM endpoint RFD --- docs/rfds/custom-llm-endpoint.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 08dc1a0e..b1b22cf5 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -258,9 +258,9 @@ interface SetLlmEndpointsResponse { Protocol identifiers are plain strings with a set of well-known values (`"anthropic"`, `"openai"`, `"google"`, `"amazon"`). Custom identifiers are allowed for regional or emerging LLM APIs not covered by the well-known set. Agents advertise the protocol identifiers they understand; clients match against them. -### How should model availability be handled? +### ~~How should model availability be handled?~~ Resolved -When a custom endpoint is provided, it may only support a subset of models. For example, a self-hosted vLLM server might only have `llama-3-70b` available, while the agent normally advertises `claude-3-opus`, `gpt-4`, etc. +Model availability is not a concern of this RFD. Most agents can discover available models from the configured endpoint at runtime. Agents that cannot should report an error if a user selects a model not supported by the custom endpoint. ## Frequently asked questions From fd3aa2c95cb643827e099080c811c2fd29f1a5b5 Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Sat, 7 Mar 2026 18:13:48 +0200 Subject: [PATCH 10/11] docs: update revision history in custom LLM endpoint RFD --- docs/rfds/custom-llm-endpoint.mdx | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index b1b22cf5..8249656f 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -4,8 +4,6 @@ title: "Custom LLM Endpoint Configuration" - Author(s): [@anna239](https://github.com/anna239), [@xtmq](https://github.com/xtmq) -> **Note:** This RFD is very preliminary and intended to start a dialog about this feature. The proposed design may change significantly based on feedback and further discussion. - ## Elevator pitch > What are you proposing to change? @@ -324,5 +322,6 @@ If the agent doesn't support custom endpoints, `llmEndpoints` will be absent fro ## Revision history +- 2026-03-07: Rename "provider" to "protocol" to reflect API compatibility level; make `LlmProtocol` an open string type with well-known values; resolve open questions on identifier standardization and model availability - 2026-03-04: Revised to use dedicated `setLlmEndpoints` method with capability advertisement - 2026-02-02: Initial draft - preliminary proposal to start discussion From 8e93892a1fc50379e2ca03675703a49800c361c6 Mon Sep 17 00:00:00 2001 From: Evgeniy Stepanov Date: Mon, 9 Mar 2026 10:41:04 +0200 Subject: [PATCH 11/11] docs: formatting for custom LLM endpoint RFD --- docs/rfds/custom-llm-endpoint.mdx | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/rfds/custom-llm-endpoint.mdx b/docs/rfds/custom-llm-endpoint.mdx index 8249656f..3b6a9532 100644 --- a/docs/rfds/custom-llm-endpoint.mdx +++ b/docs/rfds/custom-llm-endpoint.mdx @@ -26,6 +26,7 @@ Currently, agents are configured with their own LLM endpoints and credentials, t > How will things play out once this feature exists? Clients will be able to: + 1. Discover whether an agent supports custom LLM endpoints via capabilities during initialization 2. Perform agent configuration, including authorization based on this knowledge 3. Pass custom LLM endpoint URLs and headers for different LLM protocols via a dedicated method @@ -101,6 +102,7 @@ interface AgentCapabilities { ``` **Initialize Response example:** + ```json { "jsonrpc": "2.0", @@ -204,6 +206,7 @@ interface SetLlmEndpointsResponse { #### Example Exchange **setLlmEndpoints Request:** + ```json { "jsonrpc": "2.0", @@ -230,6 +233,7 @@ interface SetLlmEndpointsResponse { ``` **setLlmEndpoints Response:** + ```json { "jsonrpc": "2.0", @@ -271,6 +275,7 @@ The protocol wire format uses plain strings to keep the specification stable and Instead, well-known protocol identifiers are provided by SDKs as convenience constants: **TypeScript** — an open string type with predefined values: + ```typescript type LlmProtocol = string & {}; @@ -283,6 +288,7 @@ const LlmProtocols = { ``` **Kotlin** — a string wrapper with predefined companion objects: + ```kotlin value class LlmProtocol(val value: String) { companion object { @@ -311,6 +317,7 @@ Many agents throw authentication errors before the model selection happens. This One option would be to pass endpoint configuration via environment variables (like `OPENAI_API_BASE`) or command-line arguments when starting the agent process. This approach has significant drawbacks: + - With multiple providers, the configuration becomes complex JSON that is awkward to pass via command-line arguments - Environment variables may be logged or visible to other processes, creating security concerns - Requires knowledge of agent-specific variable names or argument formats