diff --git a/crates/switchyard/README.md b/crates/switchyard/README.md index f172686f9..ba50b355f 100644 --- a/crates/switchyard/README.md +++ b/crates/switchyard/README.md @@ -15,6 +15,9 @@ Decision API. It adds routing-aware LLM execution intercepts to the Relay runtime while preserving Relay ownership of provider credentials, target bindings, dispatch, retries, fallbacks, and observability. +NeMo Relay 0.6.0 uses a separately running Switchyard Decision API from the +`topic/nemo-relay-integration` branch. + Install it from crates.io, or build it from the NeMo Relay source checkout with the optional CLI feature while the Switchyard Decision API contract and service/library boundary are still evolving. @@ -79,9 +82,7 @@ The current service setup is documented in the pinned topic-branch commit, local configuration, compatibility smoke test, and trajectory workflow. -Translation is already in-process through Switchyard's Rust translation -library. A future in-process `DecisionProvider` can replace the HTTP Decision API -call without changing the Relay-owned dispatch and observability boundary. +Translation runs in-process through Switchyard's Rust translation library. ## Configuration and Registration @@ -118,6 +119,7 @@ outside tracked configuration files. For more information, refer to the following resources: +- [Switchyard 0.6.0 setup and validation guide](https://docs.nvidia.com/nemo/relay/v0.6.0/configure-plugins/switchyard/about) +- [Switchyard configuration reference](https://docs.nvidia.com/nemo/relay/v0.6.0/configure-plugins/switchyard/configuration) - [Switchyard integration examples](../../examples/switchyard/README.md) -- [NeMo Relay documentation](https://docs.nvidia.com/nemo/relay) - [Switchyard repository](https://github.com/NVIDIA-NeMo/Switchyard) diff --git a/docs/configure-plugins/switchyard/about.mdx b/docs/configure-plugins/switchyard/about.mdx index fd21aa072..321a4d43d 100644 --- a/docs/configure-plugins/switchyard/about.mdx +++ b/docs/configure-plugins/switchyard/about.mdx @@ -1,9 +1,11 @@ --- title: "Switchyard (Experimental)" sidebar-title: "Switchyard (Experimental)" -description: "Route Relay LLM requests through the experimental Switchyard Decision API integration." +description: "Set up and validate the experimental Switchyard Decision API integration for NeMo Relay 0.6.0." position: 5 --- +import { MermaidStyles } from "@/components/MermaidStyles"; + {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} @@ -12,48 +14,169 @@ SPDX-License-Identifier: Apache-2.0 */} > change, and the current deployment requires a separately running Switchyard > Decision API service. -The `nemo-relay-switchyard` plugin uses the -[Switchyard](https://github.com/NVIDIA-NeMo/Switchyard) Decision API to select -backends for supported LLM requests. Relay validates each decision, translates -requests and responses in process through Switchyard's -`switchyard-translation` library, and dispatches directly to the selected -Relay-owned target. Relay also owns provider credentials, retries, trusted -fallbacks, and observability. The separately running Switchyard service owns -routing decisions. +> **NeMo Relay 0.6.0 architecture:** This release uses the external Switchyard +> Decision API from the +> [`topic/nemo-relay-integration`](https://github.com/NVIDIA-NeMo/Switchyard/tree/topic/nemo-relay-integration) +> branch. + +Switchyard is an LLM routing decision engine. The `nemo-relay-switchyard` +plugin asks Switchyard which configured target should handle an LLM request. +Relay validates the decision and performs the authenticated provider request. +For example, a routing profile can send a simple prompt to a lower-cost model +and a complex prompt to a more capable model. + +## Architecture + +The following diagram shows the NeMo Relay 0.6.0 service boundary: + + + +```mermaid +flowchart LR + Agent["Agent"] -->|LLM request| Relay["NeMo Relay"] + Relay -->|routing request| Decision["Switchyard Decision API"] + Decision -->|backend decision| Relay + Relay -->|authenticated provider request| Backend["Selected LLM backend"] + Relay -->|ATOF events| History["Switchyard ATOF accumulator"] + History -. routing history .-> Decision + + class Agent grey-lightest; + class Relay green-lightest; + class Decision yellow-lightest; + class History yellow-lightest; + class Backend teal-lightest; +``` + +The components divide responsibility as follows: + +| Component | Responsibilities | +| --- | --- | +| Switchyard | Selects a backend and accumulates ATOF routing history for history-based profiles. | +| NeMo Relay | Owns provider credentials, target bindings, decision validation, protocol translation, dispatch, retries, trusted fallback, and observability. | + +Relay does not start or supervise the Switchyard service. Start Switchyard +before Relay activates the plugin. Relay derives the service's root `/health` +URL from `decision_api_url` and fails activation unless the endpoint returns +`{"status":"ok"}`. ## Prerequisites -The current integration has two runtime pieces: +Install the following prerequisites before you start: -1. A Relay build with the optional `switchyard` feature. -2. A Switchyard service built from the pinned - [`topic/nemo-relay-integration` revision](https://github.com/NVIDIA-NeMo/Switchyard/tree/topic/nemo-relay-integration). +| Requirement | Version or Value | +| --- | --- | +| NeMo Relay | Tag `0.6.0` | +| Rust | `1.96.1` | +| Switchyard branch | `topic/nemo-relay-integration` | +| Switchyard commit | `8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8` | +| Local commands | `git`, `python3`, and `curl` | -The current Relay examples pin Switchyard commit -[`8f9db9a6`](https://github.com/NVIDIA-NeMo/Switchyard/commit/8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8). -Treat that revision as part of this experimental compatibility boundary; update -it deliberately when testing a different Switchyard topic commit. +Keep the repositories in the following sibling layout. The validation script +uses this layout by default: -The plugin does not start or supervise the Switchyard service. Start a compatible -service before Relay activates the plugin. Relay derives the service's root -`/health` URL from `decision_api_url` and fails activation unless the endpoint -returns `{"status":"ok"}`. After activation, unavailable Decision API calls fail -open to the configured same-protocol fallback. +```text +/ +├── NeMo-Relay/ +└── Switchyard-topic-nemo-relay-integration/ +``` -Profiles that depend on ATOF history, including StageRouter, must use -`context_mode = "atof_required"` and name an enabled Relay ATOF HTTP stream sink -with `atof_endpoint_name`. Relay fails startup validation when that sink is -missing or invalid. Relay pushes ATOF events to the sink; a local ATOF file sink -alone does not populate Switchyard's accumulator. +## Build and Run the Compatibility Test -Build the opt-in CLI feature with: +Use the following procedure to clone and pin both repositories: + +```bash +mkdir relay-switchyard-0.6 +cd relay-switchyard-0.6 + +git clone --branch 0.6.0 \ + https://github.com/NVIDIA/NeMo-Relay.git \ + NeMo-Relay + +git clone --branch topic/nemo-relay-integration \ + https://github.com/NVIDIA-NeMo/Switchyard.git \ + Switchyard-topic-nemo-relay-integration + +git -C Switchyard-topic-nemo-relay-integration checkout --detach \ + 8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8 + +cd NeMo-Relay +``` + +The Switchyard plugin is excluded from default Relay CLI builds. Build the CLI +with the optional feature explicitly: ```bash cargo build -p nemo-relay-cli --features switchyard ``` -The default CLI build does not include the plugin or the Switchyard translation -library. +Run the real-service compatibility test from the NeMo Relay repository root: + +```bash +examples/switchyard/run-real-e2e.sh +``` + +The script prints the verified Switchyard revision and ends with the following +line when routing succeeds: + +```text +real Switchyard E2E passed: ['provider/weak', 'provider/strong', 'provider/strong'] +``` + +The test starts these local processes: + +| Process | Port | Readiness Check or Purpose | +| --- | --- | --- | +| Switchyard Decision API | `4000` | Serves `/health`, routing decisions, and ATOF ingestion. | +| NeMo Relay | `4041` | Serves `/healthz` and the OpenAI-compatible gateway. | +| Deterministic provider | `4101` | Records the model selected for each provider request. | + +The first request has no accumulated ATOF history and routes to +`provider/weak`. The script then sends session and tool events to Relay. Relay +exports those events to Switchyard's ATOF accumulator, so the next buffered +request and the final streaming request route to `provider/strong`. The test +also validates the streamed response before reporting success. + +## Configuration Walkthrough + +The compatibility test uses +[`real-e2e-plugins.toml`](https://github.com/NVIDIA/NeMo-Relay/blob/release/0.6/examples/switchyard/real-e2e-plugins.toml) +and +[`real-e2e-profiles.yaml`](https://github.com/NVIDIA/NeMo-Relay/blob/release/0.6/examples/switchyard/real-e2e-profiles.yaml). +The following settings establish the routing boundary: + +| Setting | Purpose | +| --- | --- | +| `decision_api_url` | Points Relay to the separately running Switchyard Decision API. | +| `decision_profile_id` | Selects the Switchyard routing profile. The profile maps its semantic targets to backend IDs. | +| `targets.` | Binds each returned backend ID to a Relay-owned model, protocol, endpoint, base URL, and credentials. | +| `mode = "enforce"` | Applies a valid Switchyard decision and dispatches to the selected target. | +| `mode = "observe_only"` | Records the hypothetical decision but dispatches once to the trusted same-protocol default. | +| `default_targets` | Defines trusted same-protocol fallbacks for unavailable or invalid decisions and provider failures. | +| `context_mode = "atof_required"` | Requires stable identity and accumulated ATOF history for the selected profile. | +| `atof_endpoint_name` | Selects exactly one named Relay ATOF HTTP stream sink that sends events to Switchyard. | + +For history-based routing, a local ATOF JSONL file is not sufficient. +Switchyard must receive events through the named HTTP sink at +`/v1/atof/events` so its accumulator can provide the history used by the +routing profile. Relay rejects a missing, duplicate, disabled, or invalid +named sink during startup validation. + +For the complete option reference, refer to +[Switchyard Configuration](./configuration.mdx). + +## Troubleshooting + +Use the following table to diagnose common compatibility-test failures: + +| Symptom | Cause and Resolution | +| --- | --- | +| `Switchyard worktree not found: ...` | Place the Switchyard checkout next to `NeMo-Relay` as `Switchyard-topic-nemo-relay-integration`, or set `SWITCHYARD_ROOT` to its absolute path. | +| `Switchyard checkout mismatch: expected ..., found ...` | Check out commit `8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8`. Override `SWITCHYARD_EXPECTED_COMMIT` only when deliberately testing a different contract. | +| A build uses Rust `1.96.0` or reports an unsupported toolchain | Run `rustc --version`, install Rust `1.96.1`, and allow the repository's `rust-toolchain.toml` to select it. | +| `timed out waiting for http://127.0.0.1:4000/health` | Review `switchyard.log`. Confirm the pinned checkout builds and port `4000` is free. | +| `timed out waiting for http://127.0.0.1:4041/healthz` or `process ... exited before http://127.0.0.1:4041/healthz became ready` | Review `relay.log`. Confirm the CLI was built with `--features switchyard`, the Switchyard health check passed, and port `4041` is free. | +| The provider cannot start or requests fail | Review `upstream.log` and confirm port `4101` is free. | +| `E2E logs preserved in ...` | The script preserves its temporary directory after a runtime failure and prints the exact path. Inspect `switchyard.log`, `relay.log`, `upstream.log`, and the captured request outputs there. Successful runs remove the directory. | ## Capabilities @@ -73,8 +196,9 @@ The integration provides the following capabilities: For more information, refer to the following pages: - [Switchyard Configuration](./configuration.mdx) - is the complete option and deployment reference. Review its + provides the complete option and deployment reference. Review its [experimental limitations](./configuration.mdx#experimental-limitations) before adopting the integration. -- [Switchyard integration examples](https://github.com/NVIDIA/NeMo-Relay/tree/main/examples/switchyard) - documents the pinned service worktree and manual compatibility smoke tests. +- [Switchyard integration examples](https://github.com/NVIDIA/NeMo-Relay/tree/release/0.6/examples/switchyard) + contains the versioned configuration and validation scripts used by this + guide. diff --git a/docs/configure-plugins/switchyard/configuration.mdx b/docs/configure-plugins/switchyard/configuration.mdx index 370dbea3b..54bce55c0 100644 --- a/docs/configure-plugins/switchyard/configuration.mdx +++ b/docs/configure-plugins/switchyard/configuration.mdx @@ -7,9 +7,12 @@ position: 2 {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} -> **Experimental:** This page describes the current service-based Switchyard -> integration. Its configuration and service boundary can change as Switchyard -> develops an in-process Decision Provider. +> **Experimental NeMo Relay 0.6.0 integration:** This release calls a separately +> running Switchyard Decision API. + +Start with the +[Switchyard 0.6.0 setup and validation guide](./about.mdx) +before using this page as the complete configuration reference. The Switchyard component connects Relay's CLI gateway to a separately running [Switchyard](https://github.com/NVIDIA-NeMo/Switchyard) Decision API. Relay @@ -341,8 +344,7 @@ Consider the following experimental limitations before adopting the plugin: [`8f9db9a6`](https://github.com/NVIDIA-NeMo/Switchyard/commit/8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8) on `topic/nemo-relay-integration`. Testing another revision is deliberate and can expose contract drift. -- The Decision API and service/library boundary are experimental and can change - when Switchyard introduces an in-process Decision Provider. +- The Decision API and service/library boundary are experimental. - Nonportable provider extensions in configurations that can translate across protocols, malformed decisions, service failures, pre-dispatch translation failures, buffered-response translation failures, and target drift fail open @@ -353,4 +355,4 @@ Consider the following experimental limitations before adopting the plugin: Run `nemo-relay doctor` to validate component configuration before startup. For the pinned local service and manual compatibility workflow, refer to the -[Switchyard integration examples](https://github.com/NVIDIA/NeMo-Relay/tree/main/examples/switchyard). +[Switchyard integration examples](https://github.com/NVIDIA/NeMo-Relay/tree/release/0.6/examples/switchyard). diff --git a/examples/switchyard/README.md b/examples/switchyard/README.md index 91535945e..87a2cff68 100644 --- a/examples/switchyard/README.md +++ b/examples/switchyard/README.md @@ -5,13 +5,17 @@ SPDX-License-Identifier: Apache-2.0 # Switchyard Integration Examples -These examples exercise the experimental Relay integration with a separately running Switchyard -Decision API service and the in-process Switchyard translation library. They are manual, local -validation workflows rather than production startup orchestration. +These examples exercise the experimental NeMo Relay 0.6.0 integration with a separately running +Switchyard Decision API service and the in-process Switchyard translation library. They are +manual, local validation workflows rather than production startup orchestration. + +For the canonical architecture, setup, configuration, validation, and troubleshooting workflow, +refer to the +[Switchyard 0.6.0 setup and validation guide](https://docs.nvidia.com/nemo/relay/v0.6.0/configure-plugins/switchyard/about). ## Required Switchyard Revision -The scripts default to the latest commit currently pinned for the public topic branch: +The NeMo Relay 0.6.0 scripts require the following public topic branch and commit: ```text https://github.com/NVIDIA-NeMo/Switchyard/tree/topic/nemo-relay-integration @@ -58,6 +62,12 @@ a successful run. examples/switchyard/run-real-e2e.sh ``` +A successful run ends with: + +```text +real Switchyard E2E passed: ['provider/weak', 'provider/strong', 'provider/strong'] +``` + ### Hermes and Ollama Trajectory `run-hermes-ollama-smoke.sh` runs a fixed multi-query trajectory through Hermes, Relay, Ollama, @@ -99,4 +109,6 @@ configuration and refuses activation unless it reports `{"status":"ok"}`. Trajectory scripts write to `artifacts/` by default. Set `SWITCHYARD_TRAJECTORY_DIR` to choose a shareable output directory. On failure, logs are preserved and include the verified Switchyard revision. Do not place API keys or bearer tokens in configuration files; use environment variables -or an untracked secrets file. +or an untracked secrets file. For service readiness failures, port conflicts, and the exact +temporary log locations used by `run-real-e2e.sh`, refer to the +[canonical Switchyard 0.6.0 guide](https://docs.nvidia.com/nemo/relay/v0.6.0/configure-plugins/switchyard/about#troubleshooting).