fix(launch): tier backend formats come from the preset, not launcher overrides#96
fix(launch): tier backend formats come from the preset, not launcher overrides#96ryan-lempka wants to merge 1 commit into
Conversation
decb81e to
e6bc8f0
Compare
WalkthroughDeterministic routing now takes per-tier wire formats from the selected preset. The strong tier defaults to ChangesDeterministic routing format ownership
Estimated code review effort: 2 (Simple) | ~10 minutes Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Comment |
99ef5f9 to
ac3b173
Compare
…overrides Signed-off-by: Ryan Lempka <rlempka@nvidia.com>
ac3b173 to
6f5c10b
Compare
elyasmnvidian
left a comment
There was a problem hiding this comment.
This fixes a real problem: launch codex and launch openclaw were sending the Claude strong model over OpenAI Chat Completions, which drops cache_control and quietly turns off prompt caching. Moving the format onto the preset (strong = AUTO) fixes it, and launch claude still behaves exactly as before. I checked the resolver: anthropic/claude-opus-4.7 matches the anthropic/ prefix, so AUTO picks Anthropic Messages with no startup probe and keeps the cache markers. Small, clean change. GitHub now shows it merges cleanly into main after the rebase.
One thing to confirm before merge. It lives in a file this PR does not change, so I can't leave it on a line: switchyard/lib/profiles/plan_execute_presets.py line 76 has the same bug you're fixing here. The plan-execute planner is anthropic/claude-opus-4.6 (a Claude model) but is pinned to format=BackendFormat.OPENAI, so the plan-execute path drops cache_control on its planner exactly like codex/openclaw did. If switchyard launch ... --plan-execute ships with these defaults, its planner loses prompt caching too. Can you confirm whether that path ships? If it does, the planner tier needs the same AUTO treatment.
While tracing that: build_plan_execute_config in route_builder.py still takes a backend_format argument, but nothing in the app calls it anymore (only tests do), so that argument and its docstring now describe code that never runs. Worth deleting in a follow-up.
Two smaller notes are on the lines themselves.
| id="strong", | ||
| model=strong_model, | ||
| format=strong_backend_format or backend_format, | ||
| format=preset.strong.format, |
There was a problem hiding this comment.
Small heads-up on the override case. When someone runs launch codex --model <a non-Claude model>, the strong tier now takes format=preset.strong.format, which is AUTO. Before this change it was a fixed OPENAI. AUTO makes one /v1/chat/completions probe to the gateway at startup; on the NVIDIA gateway that probe answers yes quickly, so the model still ends up on OpenAI format — same result, one extra round-trip at launch. Not a correctness problem, just flagging that override launches now pay for a probe they used to skip. Fine to leave as-is.
| ], | ||
| }) | ||
| ctx = ProxyContext() | ||
| response = await backend._backends["strong"].call(ctx, request) |
There was a problem hiding this comment.
This calls the strong backend directly, so the test skips the classifier and router that decide which tier a request goes to. It proves that once the strong tier is picked, it sends Anthropic Messages with cache_control — exactly the bug you're guarding against, so this is good. It does not prove that a normal incoming request actually gets routed to the strong tier. If you want the guard to cover the full path, send the request through the deterministic routing backend instead of reaching into _backends["strong"]. Optional — the current test already locks in the format fix.
Summary
The launcher route builder rebuilt every deterministic tier with caller-supplied
backend_format/strong_backend_formatparams, discarding the preset's per-tier formats. The preset's strong tier is a Claude model, solaunch codexandlaunch openclawforced it onto OpenAI format, which stripscache_controland silently disables prompt caching.The preset now owns tier formats: strong is
AUTO(resolves to Anthropic by model prefix; probes otherwise), weak and classifier stayOPENAI. Both override params are deleted.launch claudeis unchanged (it already overrode toAUTO);switchyard serveis untouched (YAML formats come from the config schema, not the preset). Launchers never exposed a user-facing format flag, so no capability is removed; explicit per-tier formats remain available via--routing-profiles.How tested
uv run ruff check .cleanuv run mypy switchyardcleanuv run pytest tests/green (1998 passed; builder tests now assert formats are preset-owned)tests/test_launch_preset_cache_e2e.py: real builder output, real profile build, real HTTP upstream — asserts the strong tier's wire body carriescache_controlephemeral markers on the zero-flag launcher path, and the weak tier's does not.Checklist
snake_caseof the primary class.switchyard/__init__.py.__all__if intended for downstream use. (none added)--helpupdated if customer-facing surface changed. (no surface change)