From 968992a0e5f835850acb9a1460c704f2427e6d04 Mon Sep 17 00:00:00 2001 From: Bryan Bednarski Date: Mon, 20 Jul 2026 17:36:23 -0600 Subject: [PATCH] docs: add canonical Switchyard 0.6 guide (#503) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #### Overview Provide one canonical, versioned setup and validation guide for the experimental Switchyard integration in NeMo Relay 0.6.0. The guide makes the external Decision API boundary explicit and presents the release-specific setup in one place. - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details - Expand the existing Fern Switchyard overview with a plain-language introduction, architecture diagram, component ownership table, exact prerequisites, and sibling checkout layout. - Add a copyable build and real-service E2E workflow pinned to Switchyard commit `8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8`, including the expected weak/strong/strong routing sequence. - Document the E2E ports, health checks, ATOF-informed routing behavior, configuration boundary, and common preflight or startup failures. - Keep the configuration page as the exhaustive option reference while adding the NeMo Relay 0.6 external-service warning and versioned source links. - Point the Switchyard crate and example READMEs to the canonical Fern guide. Validation: - `just docs` - `just docs-linkcheck` - Focused pre-commit checks for all four changed files, including external link validation - All-files pre-commit checks with the three attribution generators skipped; all remaining hooks passed - Pinned real-service E2E: `real Switchyard E2E passed: ['provider/weak', 'provider/strong', 'provider/strong']` - Missing checkout, mismatched commit, Relay readiness timeout, and preserved-log failure paths The all-files Rust attribution generator rewrites the release branch's checked-in `ATTRIBUTIONS-Rust.md` despite no Cargo or lockfile changes. That unrelated generated drift is excluded from this documentation-only PR. Breaking changes: none. #### Where should the reviewer start? Start with `docs/configure-plugins/switchyard/about.mdx`, especially the NeMo Relay 0.6 architecture boundary, copyable compatibility workflow, and troubleshooting table. Then review the narrower cross-links in the configuration page and two READMEs. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Relates to: [RELAY-515](https://linear.app/nvidia/issue/RELAY-515) ## Summary by CodeRabbit - **Documentation** - Updated Switchyard integration guidance for NeMo Relay 0.6.0’s service-based Decision API workflow. - Expanded the architecture overview, setup/build steps, and configuration/validation requirements. - Improved compatibility testing instructions, including expected smoke-test output and health/readiness expectations. - Added troubleshooting details for common failures like service readiness issues and port conflicts, with updated log references. - Refreshed integration example and resource links to the 0.6 release. Authors: - Bryan Bednarski (https://github.com/bbednarski9) Approvers: - Will Killian (https://github.com/willkill07) URL: https://github.com/NVIDIA/NeMo-Relay/pull/503 --- crates/switchyard/README.md | 10 +- docs/configure-plugins/switchyard/about.mdx | 190 +++++++++++++++--- .../switchyard/configuration.mdx | 14 +- examples/switchyard/README.md | 22 +- 4 files changed, 188 insertions(+), 48 deletions(-) diff --git a/crates/switchyard/README.md b/crates/switchyard/README.md index f172686f9..ba50b355f 100644 --- a/crates/switchyard/README.md +++ b/crates/switchyard/README.md @@ -15,6 +15,9 @@ Decision API. It adds routing-aware LLM execution intercepts to the Relay runtime while preserving Relay ownership of provider credentials, target bindings, dispatch, retries, fallbacks, and observability. +NeMo Relay 0.6.0 uses a separately running Switchyard Decision API from the +`topic/nemo-relay-integration` branch. + Install it from crates.io, or build it from the NeMo Relay source checkout with the optional CLI feature while the Switchyard Decision API contract and service/library boundary are still evolving. @@ -79,9 +82,7 @@ The current service setup is documented in the pinned topic-branch commit, local configuration, compatibility smoke test, and trajectory workflow. -Translation is already in-process through Switchyard's Rust translation -library. A future in-process `DecisionProvider` can replace the HTTP Decision API -call without changing the Relay-owned dispatch and observability boundary. +Translation runs in-process through Switchyard's Rust translation library. ## Configuration and Registration @@ -118,6 +119,7 @@ outside tracked configuration files. For more information, refer to the following resources: +- [Switchyard 0.6.0 setup and validation guide](https://docs.nvidia.com/nemo/relay/v0.6.0/configure-plugins/switchyard/about) +- [Switchyard configuration reference](https://docs.nvidia.com/nemo/relay/v0.6.0/configure-plugins/switchyard/configuration) - [Switchyard integration examples](../../examples/switchyard/README.md) -- [NeMo Relay documentation](https://docs.nvidia.com/nemo/relay) - [Switchyard repository](https://github.com/NVIDIA-NeMo/Switchyard) diff --git a/docs/configure-plugins/switchyard/about.mdx b/docs/configure-plugins/switchyard/about.mdx index fd21aa072..321a4d43d 100644 --- a/docs/configure-plugins/switchyard/about.mdx +++ b/docs/configure-plugins/switchyard/about.mdx @@ -1,9 +1,11 @@ --- title: "Switchyard (Experimental)" sidebar-title: "Switchyard (Experimental)" -description: "Route Relay LLM requests through the experimental Switchyard Decision API integration." +description: "Set up and validate the experimental Switchyard Decision API integration for NeMo Relay 0.6.0." position: 5 --- +import { MermaidStyles } from "@/components/MermaidStyles"; + {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} @@ -12,48 +14,169 @@ SPDX-License-Identifier: Apache-2.0 */} > change, and the current deployment requires a separately running Switchyard > Decision API service. -The `nemo-relay-switchyard` plugin uses the -[Switchyard](https://github.com/NVIDIA-NeMo/Switchyard) Decision API to select -backends for supported LLM requests. Relay validates each decision, translates -requests and responses in process through Switchyard's -`switchyard-translation` library, and dispatches directly to the selected -Relay-owned target. Relay also owns provider credentials, retries, trusted -fallbacks, and observability. The separately running Switchyard service owns -routing decisions. +> **NeMo Relay 0.6.0 architecture:** This release uses the external Switchyard +> Decision API from the +> [`topic/nemo-relay-integration`](https://github.com/NVIDIA-NeMo/Switchyard/tree/topic/nemo-relay-integration) +> branch. + +Switchyard is an LLM routing decision engine. The `nemo-relay-switchyard` +plugin asks Switchyard which configured target should handle an LLM request. +Relay validates the decision and performs the authenticated provider request. +For example, a routing profile can send a simple prompt to a lower-cost model +and a complex prompt to a more capable model. + +## Architecture + +The following diagram shows the NeMo Relay 0.6.0 service boundary: + + + +```mermaid +flowchart LR + Agent["Agent"] -->|LLM request| Relay["NeMo Relay"] + Relay -->|routing request| Decision["Switchyard Decision API"] + Decision -->|backend decision| Relay + Relay -->|authenticated provider request| Backend["Selected LLM backend"] + Relay -->|ATOF events| History["Switchyard ATOF accumulator"] + History -. routing history .-> Decision + + class Agent grey-lightest; + class Relay green-lightest; + class Decision yellow-lightest; + class History yellow-lightest; + class Backend teal-lightest; +``` + +The components divide responsibility as follows: + +| Component | Responsibilities | +| --- | --- | +| Switchyard | Selects a backend and accumulates ATOF routing history for history-based profiles. | +| NeMo Relay | Owns provider credentials, target bindings, decision validation, protocol translation, dispatch, retries, trusted fallback, and observability. | + +Relay does not start or supervise the Switchyard service. Start Switchyard +before Relay activates the plugin. Relay derives the service's root `/health` +URL from `decision_api_url` and fails activation unless the endpoint returns +`{"status":"ok"}`. ## Prerequisites -The current integration has two runtime pieces: +Install the following prerequisites before you start: -1. A Relay build with the optional `switchyard` feature. -2. A Switchyard service built from the pinned - [`topic/nemo-relay-integration` revision](https://github.com/NVIDIA-NeMo/Switchyard/tree/topic/nemo-relay-integration). +| Requirement | Version or Value | +| --- | --- | +| NeMo Relay | Tag `0.6.0` | +| Rust | `1.96.1` | +| Switchyard branch | `topic/nemo-relay-integration` | +| Switchyard commit | `8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8` | +| Local commands | `git`, `python3`, and `curl` | -The current Relay examples pin Switchyard commit -[`8f9db9a6`](https://github.com/NVIDIA-NeMo/Switchyard/commit/8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8). -Treat that revision as part of this experimental compatibility boundary; update -it deliberately when testing a different Switchyard topic commit. +Keep the repositories in the following sibling layout. The validation script +uses this layout by default: -The plugin does not start or supervise the Switchyard service. Start a compatible -service before Relay activates the plugin. Relay derives the service's root -`/health` URL from `decision_api_url` and fails activation unless the endpoint -returns `{"status":"ok"}`. After activation, unavailable Decision API calls fail -open to the configured same-protocol fallback. +```text +/ +├── NeMo-Relay/ +└── Switchyard-topic-nemo-relay-integration/ +``` -Profiles that depend on ATOF history, including StageRouter, must use -`context_mode = "atof_required"` and name an enabled Relay ATOF HTTP stream sink -with `atof_endpoint_name`. Relay fails startup validation when that sink is -missing or invalid. Relay pushes ATOF events to the sink; a local ATOF file sink -alone does not populate Switchyard's accumulator. +## Build and Run the Compatibility Test -Build the opt-in CLI feature with: +Use the following procedure to clone and pin both repositories: + +```bash +mkdir relay-switchyard-0.6 +cd relay-switchyard-0.6 + +git clone --branch 0.6.0 \ + https://github.com/NVIDIA/NeMo-Relay.git \ + NeMo-Relay + +git clone --branch topic/nemo-relay-integration \ + https://github.com/NVIDIA-NeMo/Switchyard.git \ + Switchyard-topic-nemo-relay-integration + +git -C Switchyard-topic-nemo-relay-integration checkout --detach \ + 8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8 + +cd NeMo-Relay +``` + +The Switchyard plugin is excluded from default Relay CLI builds. Build the CLI +with the optional feature explicitly: ```bash cargo build -p nemo-relay-cli --features switchyard ``` -The default CLI build does not include the plugin or the Switchyard translation -library. +Run the real-service compatibility test from the NeMo Relay repository root: + +```bash +examples/switchyard/run-real-e2e.sh +``` + +The script prints the verified Switchyard revision and ends with the following +line when routing succeeds: + +```text +real Switchyard E2E passed: ['provider/weak', 'provider/strong', 'provider/strong'] +``` + +The test starts these local processes: + +| Process | Port | Readiness Check or Purpose | +| --- | --- | --- | +| Switchyard Decision API | `4000` | Serves `/health`, routing decisions, and ATOF ingestion. | +| NeMo Relay | `4041` | Serves `/healthz` and the OpenAI-compatible gateway. | +| Deterministic provider | `4101` | Records the model selected for each provider request. | + +The first request has no accumulated ATOF history and routes to +`provider/weak`. The script then sends session and tool events to Relay. Relay +exports those events to Switchyard's ATOF accumulator, so the next buffered +request and the final streaming request route to `provider/strong`. The test +also validates the streamed response before reporting success. + +## Configuration Walkthrough + +The compatibility test uses +[`real-e2e-plugins.toml`](https://github.com/NVIDIA/NeMo-Relay/blob/release/0.6/examples/switchyard/real-e2e-plugins.toml) +and +[`real-e2e-profiles.yaml`](https://github.com/NVIDIA/NeMo-Relay/blob/release/0.6/examples/switchyard/real-e2e-profiles.yaml). +The following settings establish the routing boundary: + +| Setting | Purpose | +| --- | --- | +| `decision_api_url` | Points Relay to the separately running Switchyard Decision API. | +| `decision_profile_id` | Selects the Switchyard routing profile. The profile maps its semantic targets to backend IDs. | +| `targets.` | Binds each returned backend ID to a Relay-owned model, protocol, endpoint, base URL, and credentials. | +| `mode = "enforce"` | Applies a valid Switchyard decision and dispatches to the selected target. | +| `mode = "observe_only"` | Records the hypothetical decision but dispatches once to the trusted same-protocol default. | +| `default_targets` | Defines trusted same-protocol fallbacks for unavailable or invalid decisions and provider failures. | +| `context_mode = "atof_required"` | Requires stable identity and accumulated ATOF history for the selected profile. | +| `atof_endpoint_name` | Selects exactly one named Relay ATOF HTTP stream sink that sends events to Switchyard. | + +For history-based routing, a local ATOF JSONL file is not sufficient. +Switchyard must receive events through the named HTTP sink at +`/v1/atof/events` so its accumulator can provide the history used by the +routing profile. Relay rejects a missing, duplicate, disabled, or invalid +named sink during startup validation. + +For the complete option reference, refer to +[Switchyard Configuration](./configuration.mdx). + +## Troubleshooting + +Use the following table to diagnose common compatibility-test failures: + +| Symptom | Cause and Resolution | +| --- | --- | +| `Switchyard worktree not found: ...` | Place the Switchyard checkout next to `NeMo-Relay` as `Switchyard-topic-nemo-relay-integration`, or set `SWITCHYARD_ROOT` to its absolute path. | +| `Switchyard checkout mismatch: expected ..., found ...` | Check out commit `8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8`. Override `SWITCHYARD_EXPECTED_COMMIT` only when deliberately testing a different contract. | +| A build uses Rust `1.96.0` or reports an unsupported toolchain | Run `rustc --version`, install Rust `1.96.1`, and allow the repository's `rust-toolchain.toml` to select it. | +| `timed out waiting for http://127.0.0.1:4000/health` | Review `switchyard.log`. Confirm the pinned checkout builds and port `4000` is free. | +| `timed out waiting for http://127.0.0.1:4041/healthz` or `process ... exited before http://127.0.0.1:4041/healthz became ready` | Review `relay.log`. Confirm the CLI was built with `--features switchyard`, the Switchyard health check passed, and port `4041` is free. | +| The provider cannot start or requests fail | Review `upstream.log` and confirm port `4101` is free. | +| `E2E logs preserved in ...` | The script preserves its temporary directory after a runtime failure and prints the exact path. Inspect `switchyard.log`, `relay.log`, `upstream.log`, and the captured request outputs there. Successful runs remove the directory. | ## Capabilities @@ -73,8 +196,9 @@ The integration provides the following capabilities: For more information, refer to the following pages: - [Switchyard Configuration](./configuration.mdx) - is the complete option and deployment reference. Review its + provides the complete option and deployment reference. Review its [experimental limitations](./configuration.mdx#experimental-limitations) before adopting the integration. -- [Switchyard integration examples](https://github.com/NVIDIA/NeMo-Relay/tree/main/examples/switchyard) - documents the pinned service worktree and manual compatibility smoke tests. +- [Switchyard integration examples](https://github.com/NVIDIA/NeMo-Relay/tree/release/0.6/examples/switchyard) + contains the versioned configuration and validation scripts used by this + guide. diff --git a/docs/configure-plugins/switchyard/configuration.mdx b/docs/configure-plugins/switchyard/configuration.mdx index 370dbea3b..54bce55c0 100644 --- a/docs/configure-plugins/switchyard/configuration.mdx +++ b/docs/configure-plugins/switchyard/configuration.mdx @@ -7,9 +7,12 @@ position: 2 {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 */} -> **Experimental:** This page describes the current service-based Switchyard -> integration. Its configuration and service boundary can change as Switchyard -> develops an in-process Decision Provider. +> **Experimental NeMo Relay 0.6.0 integration:** This release calls a separately +> running Switchyard Decision API. + +Start with the +[Switchyard 0.6.0 setup and validation guide](./about.mdx) +before using this page as the complete configuration reference. The Switchyard component connects Relay's CLI gateway to a separately running [Switchyard](https://github.com/NVIDIA-NeMo/Switchyard) Decision API. Relay @@ -341,8 +344,7 @@ Consider the following experimental limitations before adopting the plugin: [`8f9db9a6`](https://github.com/NVIDIA-NeMo/Switchyard/commit/8f9db9a6a47f848cdff1d262276ba25a8ae9cbc8) on `topic/nemo-relay-integration`. Testing another revision is deliberate and can expose contract drift. -- The Decision API and service/library boundary are experimental and can change - when Switchyard introduces an in-process Decision Provider. +- The Decision API and service/library boundary are experimental. - Nonportable provider extensions in configurations that can translate across protocols, malformed decisions, service failures, pre-dispatch translation failures, buffered-response translation failures, and target drift fail open @@ -353,4 +355,4 @@ Consider the following experimental limitations before adopting the plugin: Run `nemo-relay doctor` to validate component configuration before startup. For the pinned local service and manual compatibility workflow, refer to the -[Switchyard integration examples](https://github.com/NVIDIA/NeMo-Relay/tree/main/examples/switchyard). +[Switchyard integration examples](https://github.com/NVIDIA/NeMo-Relay/tree/release/0.6/examples/switchyard). diff --git a/examples/switchyard/README.md b/examples/switchyard/README.md index 91535945e..87a2cff68 100644 --- a/examples/switchyard/README.md +++ b/examples/switchyard/README.md @@ -5,13 +5,17 @@ SPDX-License-Identifier: Apache-2.0 # Switchyard Integration Examples -These examples exercise the experimental Relay integration with a separately running Switchyard -Decision API service and the in-process Switchyard translation library. They are manual, local -validation workflows rather than production startup orchestration. +These examples exercise the experimental NeMo Relay 0.6.0 integration with a separately running +Switchyard Decision API service and the in-process Switchyard translation library. They are +manual, local validation workflows rather than production startup orchestration. + +For the canonical architecture, setup, configuration, validation, and troubleshooting workflow, +refer to the +[Switchyard 0.6.0 setup and validation guide](https://docs.nvidia.com/nemo/relay/v0.6.0/configure-plugins/switchyard/about). ## Required Switchyard Revision -The scripts default to the latest commit currently pinned for the public topic branch: +The NeMo Relay 0.6.0 scripts require the following public topic branch and commit: ```text https://github.com/NVIDIA-NeMo/Switchyard/tree/topic/nemo-relay-integration @@ -58,6 +62,12 @@ a successful run. examples/switchyard/run-real-e2e.sh ``` +A successful run ends with: + +```text +real Switchyard E2E passed: ['provider/weak', 'provider/strong', 'provider/strong'] +``` + ### Hermes and Ollama Trajectory `run-hermes-ollama-smoke.sh` runs a fixed multi-query trajectory through Hermes, Relay, Ollama, @@ -99,4 +109,6 @@ configuration and refuses activation unless it reports `{"status":"ok"}`. Trajectory scripts write to `artifacts/` by default. Set `SWITCHYARD_TRAJECTORY_DIR` to choose a shareable output directory. On failure, logs are preserved and include the verified Switchyard revision. Do not place API keys or bearer tokens in configuration files; use environment variables -or an untracked secrets file. +or an untracked secrets file. For service readiness failures, port conflicts, and the exact +temporary log locations used by `run-real-e2e.sh`, refer to the +[canonical Switchyard 0.6.0 guide](https://docs.nvidia.com/nemo/relay/v0.6.0/configure-plugins/switchyard/about#troubleshooting).