Skip to content

fix(launch): tier backend formats come from the preset, not launcher overrides#96

Open
ryan-lempka wants to merge 1 commit into
mainfrom
rlempka/switch-687-preset-owns-tier-formats
Open

fix(launch): tier backend formats come from the preset, not launcher overrides#96
ryan-lempka wants to merge 1 commit into
mainfrom
rlempka/switch-687-preset-owns-tier-formats

Conversation

@ryan-lempka

@ryan-lempka ryan-lempka commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

The launcher route builder rebuilt every deterministic tier with caller-supplied backend_format / strong_backend_format params, discarding the preset's per-tier formats. The preset's strong tier is a Claude model, so launch codex and launch openclaw forced it onto OpenAI format, which strips cache_control and silently disables prompt caching.

The preset now owns tier formats: strong is AUTO (resolves to Anthropic by model prefix; probes otherwise), weak and classifier stay OPENAI. Both override params are deleted. launch claude is unchanged (it already overrode to AUTO); switchyard serve is untouched (YAML formats come from the config schema, not the preset). Launchers never exposed a user-facing format flag, so no capability is removed; explicit per-tier formats remain available via --routing-profiles.

How tested

  • uv run ruff check . clean
  • uv run mypy switchyard clean
  • uv run pytest tests/ green (1998 passed; builder tests now assert formats are preset-owned)
  • New e2e regression guard tests/test_launch_preset_cache_e2e.py: real builder output, real profile build, real HTTP upstream — asserts the strong tier's wire body carries cache_control ephemeral markers on the zero-flag launcher path, and the weak tier's does not.

Checklist

  • One class per file; filename = snake_case of the primary class.
  • New public symbols exported from switchyard/__init__.py.__all__ if intended for downstream use. (none added)
  • Unit tests added for new components / bug fixes.
  • README / --help updated if customer-facing surface changed. (no surface change)

@ryan-lempka
ryan-lempka requested a review from a team as a code owner July 20, 2026 17:56
@ryan-lempka
ryan-lempka force-pushed the rlempka/switch-687-preset-owns-tier-formats branch from decb81e to e6bc8f0 Compare July 20, 2026 17:57
@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

Deterministic routing now takes per-tier wire formats from the selected preset. The strong tier defaults to AUTO, while weak and classifier tiers remain OPENAI; launch commands and tests use the updated builder contract.

Changes

Deterministic routing format ownership

Layer / File(s) Summary
Preset format contract
switchyard/lib/profiles/deterministic_routing_presets.py, tests/test_deterministic_routing_profile.py
The coding-agent preset assigns AUTO to the strong tier and retains OPENAI for weak and classifier tiers, with updated assertions.
Route builder propagation
switchyard/cli/routing/route_builder.py, tests/test_launch_route_builder.py
The builder removes format override parameters and applies preset formats to all deterministic targets; related default, override, and rejection tests are updated.
Launch command integration
switchyard/cli/launch_command.py
Claude, Codex, and OpenClaw deterministic launch paths stop passing explicit format overrides.

Estimated code review effort: 2 (Simple) | ~10 minutes

Poem

I’m a rabbit with routes in a neat little row,
Strong hops on AUTO wherever you go.
Weak and classifier keep OPENAI bright,
Presets now guide every launch just right.
Sproing, the formats are tidy tonight!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: deterministic launch tier backend formats now come from the preset instead of launcher overrides.

Comment @coderabbitai help to get the list of available commands.

@ryan-lempka
ryan-lempka force-pushed the rlempka/switch-687-preset-owns-tier-formats branch 2 times, most recently from 99ef5f9 to ac3b173 Compare July 20, 2026 19:04
…overrides

Signed-off-by: Ryan Lempka <rlempka@nvidia.com>
@ryan-lempka
ryan-lempka force-pushed the rlempka/switch-687-preset-owns-tier-formats branch from ac3b173 to 6f5c10b Compare July 21, 2026 21:35

@elyasmnvidian elyasmnvidian left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes a real problem: launch codex and launch openclaw were sending the Claude strong model over OpenAI Chat Completions, which drops cache_control and quietly turns off prompt caching. Moving the format onto the preset (strong = AUTO) fixes it, and launch claude still behaves exactly as before. I checked the resolver: anthropic/claude-opus-4.7 matches the anthropic/ prefix, so AUTO picks Anthropic Messages with no startup probe and keeps the cache markers. Small, clean change. GitHub now shows it merges cleanly into main after the rebase.

One thing to confirm before merge. It lives in a file this PR does not change, so I can't leave it on a line: switchyard/lib/profiles/plan_execute_presets.py line 76 has the same bug you're fixing here. The plan-execute planner is anthropic/claude-opus-4.6 (a Claude model) but is pinned to format=BackendFormat.OPENAI, so the plan-execute path drops cache_control on its planner exactly like codex/openclaw did. If switchyard launch ... --plan-execute ships with these defaults, its planner loses prompt caching too. Can you confirm whether that path ships? If it does, the planner tier needs the same AUTO treatment.

While tracing that: build_plan_execute_config in route_builder.py still takes a backend_format argument, but nothing in the app calls it anymore (only tests do), so that argument and its docstring now describe code that never runs. Worth deleting in a follow-up.

Two smaller notes are on the lines themselves.

id="strong",
model=strong_model,
format=strong_backend_format or backend_format,
format=preset.strong.format,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small heads-up on the override case. When someone runs launch codex --model <a non-Claude model>, the strong tier now takes format=preset.strong.format, which is AUTO. Before this change it was a fixed OPENAI. AUTO makes one /v1/chat/completions probe to the gateway at startup; on the NVIDIA gateway that probe answers yes quickly, so the model still ends up on OpenAI format — same result, one extra round-trip at launch. Not a correctness problem, just flagging that override launches now pay for a probe they used to skip. Fine to leave as-is.

],
})
ctx = ProxyContext()
response = await backend._backends["strong"].call(ctx, request)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This calls the strong backend directly, so the test skips the classifier and router that decide which tier a request goes to. It proves that once the strong tier is picked, it sends Anthropic Messages with cache_control — exactly the bug you're guarding against, so this is good. It does not prove that a normal incoming request actually gets routed to the strong tier. If you want the guard to cover the full path, send the request through the deterministic routing backend instead of reaching into _backends["strong"]. Optional — the current test already locks in the format fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants