feat(routing): add context-window fallback retry to v2 random-routing

[elyasmnvidian]

Random-routing now retries once on a context-window overflow, matching what stage_router and llm_routing already do.

Before, if the randomly selected target rejected a request because the prompt was too long, the random-routing profile returned that ContextWindowExceeded error straight to the client. Now it retries the request once against a fallback target, and only fails if that target also overflows — in which case it returns ContextPoolExhausted instead of the raw overflow.

New optional config key on the random-routing profile (serve --config):

• fallback_target_on_evict: the target id to retry after an overflow. Defaults to the strong target. build() rejects any value that is not the profile's strong or weak target id.

Tests in the diff:

• run_retries_fallback_target_after_context_overflow drives run() through a weak-target overflow and checks the second backend call lands on the strong target and its response is returned.
• build_rejects_fallback_target_not_matching_strong_or_weak checks the config rejects an unknown fallback id.

How to try it: add fallback_target_on_evict: strong under a type: random-routing profile whose weak target has a small context window, then send a prompt that overflows the weak target — the response comes back from the strong target instead of erroring.
Audit follow-up in this branch: after a fallback switches tiers, the x-model-router-rationale header now names the overflow-and-retry instead of reprinting the original random-draw explanation (which described the wrong tier). The retry test asserts the corrected rationale. The serve --config docs for type: random-routing now show the optional fallback_target_on_evict key.

[coderabbitai]

Walkthrough

Changes

Random routing fallback

Layer / File(s)	Summary
Fallback configuration contract crates/switchyard-components-v2/src/profiles/random_routing.rs	Adds an optional fallback target, defaults it to the strong target, validates configured targets, and stores the resolved value.
Overflow retry execution crates/switchyard-components-v2/src/profiles/random_routing.rs	Adds target lookup and request-rewrite helpers, then retries once on ContextWindowExceeded and returns ContextPoolExhausted after a second overflow.
Retry and configuration validation crates/switchyard-components-v2/src/profiles/random_routing.rs, crates/switchyard-components-v2/tests/random_routing_profile.rs	Adds overflow test backends, retry-routing coverage, invalid-target validation, and updated configuration fixtures.

Estimated code review effort: 4 (Complex) | ~45 minutes

Poem

I’m a rabbit with a fallback route,
When context clouds make callers pout.
Strong target waits beside the gate,
One retry hops before too late.
Invalid paths are turned away—
Neat little burrows saved today!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding context-window fallback retry to v2 random-routing.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/switchyard-components-v2/src/profiles/random_routing.rs`:
- Around line 164-192: Add concise Rust doc comments to the private helpers
`record_success` and `record_error`, describing their shared role in recording
routing statistics and, for `record_success`, wrapping response usage/stream
handling. Match the brief explanatory style used by `backend_for_target`,
`tier_for_target`, `fallback_processed_request`, and `call_selected`.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: a9084d28-b522-47cd-beae-d5b912158485

📥 Commits

Reviewing files that changed from the base of the PR and between 2fa6b87 and a5ccb16.

📒 Files selected for processing (2)

• crates/switchyard-components-v2/src/profiles/random_routing.rs
• crates/switchyard-components-v2/tests/random_routing_profile.rs

[elyasmnvidian]

Live before/after — random-routing now recovers from a context overflow

Config: type: random-routing with strong = azure/anthropic/claude-opus-4-8, weak = nvidia/meta/llama-3.2-1b-instruct (131k context), strong_probability: 0.0 (always weak first). Endpoints point at the live NVIDIA gateway. Then POST a ~300k-token prompt, which overflows the weak target.

Before (main) — two problems:

1. The config is rejected outright, because fallback_target_on_evict is a new key:

   SwitchyardConfigError: profile rr: failed to parse random-routing profile config:
   unknown field `fallback_target_on_evict`, expected one of `strong`, `weak`, `strong_probability`, `rng_seed`
   

2. With the accepted config (no fallback key), the overflow goes straight back to the client:

   HTTP 500  {"error":{"code":"server_error","message":"context window exceeded on target weak ..."}}
   

After (this branch) — with fallback_target_on_evict: strong, the same overflow retries the strong target and returns a real completion:

HTTP 200
x-model-router-selected-model: azure/anthropic/claude-opus-4-8
x-model-router-selected-tier:  strong
x-model-router-rationale:      selected target weak exceeded its context window; retried fallback target strong

response.model:              azure/anthropic/claude-opus-4-8
response.usage.total_tokens: 440007

The rationale header now names the overflow-and-retry instead of reprinting the (now-wrong) random-draw text — an audit follow-up in this push, asserted in run_retries_fallback_target_after_context_overflow.

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-106/
Built to branch gh-pages at 2026-07-22 00:16 UTC. Preview will be ready when the GitHub Pages deployment is complete.