Skip to content

fix(cli): harness-arg forwarding, provider/env-key resolution, serve errors, proxy bypass#107

Open
elyasmnvidian wants to merge 4 commits into
mainfrom
emehtabuddin/qa-cli
Open

fix(cli): harness-arg forwarding, provider/env-key resolution, serve errors, proxy bypass#107
elyasmnvidian wants to merge 4 commits into
mainfrom
emehtabuddin/qa-cli

Conversation

@elyasmnvidian

@elyasmnvidian elyasmnvidian commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

CLI fixes. Rebased onto main after #98 ("serve libsy algorithms without profiles") landed — #98 added its own clean duplicate-model-id message, so that fix is dropped here.

  • launch <agent> -- <args> forwards documented harness args after a single --. Before, main() stripped the first -- wherever it was, so launch claude -- --version made argparse reject --version and exit 2. Now only a -- that comes before the subcommand token is stripped, so the harness -- survives into argparse.REMAINDER. Same fix covers codex/openclaw.
  • configure and configure --list-models honor the saved default_provider again (--provider no longer defaults to openrouter at the argparse level), and non-interactive configure resolves an API key from OPENROUTER_API_KEY / NVIDIA_API_KEY / OPENAI_API_KEY or secrets.json instead of demanding --api-key.
  • serve with neither --config nor a saved bundle points at --config PATH first, and names --routing-profiles PATH as the legacy route bundle.
  • Local 127.0.0.1 readiness and health probes (wait_for_proxy_ready, ProxyHealthMonitor, and verify's round-trip) bypass env proxies, so a configured HTTP_PROXY no longer breaks the loopback checks. The upstream backend probe still honors env proxies.

Regression tests cover each behavior.

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

Changes

Configure defaults

Layer / File(s) Summary
Provider and API-key resolution
switchyard/cli/configure_command.py, switchyard/cli/switchyard_cli.py, tests/test_launch_claude.py, tests/test_user_config.py
Configure now honors saved providers and environment-backed API keys when CLI values are absent, with updated parser and regression tests.

Loopback health checks

Layer / File(s) Summary
Shared local health transport
switchyard/cli/launchers/launcher_runtime.py
Launcher readiness probes use a proxy-free opener, and annotation behavior is adjusted.
Proxy monitor and verification probes
switchyard/cli/launchers/proxy_health_monitor.py, switchyard/server/verify.py, tests/test_launcher_proxy_bypass.py
Monitoring and verification loopback requests bypass configured proxies, with dedicated coverage.

CLI robustness

Layer / File(s) Summary
Serve diagnostics and duplicate registration handling
switchyard/cli/switchyard_cli.py, tests/test_serve_profile_config.py
Serve messaging is updated and duplicate registrations now exit with concise diagnostics.
Argument separator forwarding
switchyard/cli/switchyard_cli.py, tests/test_launch_claude.py
Separators after subcommands are preserved for forwarded launcher arguments.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Poem

I’m a rabbit with a proxy-free nose,
Hopping through configs wherever it goes.
Saved keys bloom, separators stay,
Duplicate IDs hop safely away.
Health checks thump on localhost bright—
CLI burrows now run just right.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main CLI fixes: harness arg forwarding, provider/API-key resolution, serve error handling, and proxy bypasses.

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@switchyard/cli/configure_command.py`:
- Around line 425-437: Restrict the environment key and base-URL candidates
passed by the configure flow around resolve_provider_connectivity() to the
selected provider, so provider nvidia cannot consume an OPENROUTER_API_KEY (and
vice versa). Preserve provider/key consistency before assigning
prompt_default_api_key, and add a regression test covering mixed environment
variables with --provider nvidia.

In `@switchyard/cli/switchyard_cli.py`:
- Around line 1463-1470: Update the argument scanning around _SUBCOMMANDS,
sep_idx, and cmd_idx to skip values consumed by global options such as
--routing-profiles before identifying the real subcommand. Ensure a value equal
to a subcommand name is not treated as the command, then remove only a "--"
separator occurring before that resolved subcommand while preserving separators
after it.

In `@tests/test_launcher_proxy_bypass.py`:
- Around line 30-33: Update the proxy setup in the affected test to use a
test-owned proxy stub or dynamically allocated local port instead of hard-coded
127.0.0.1:9. Ensure the endpoint reliably returns a connection error for the
negative proxy case, and apply the dynamically selected endpoint to both
HTTP_PROXY and http_proxy.

In `@tests/test_user_config.py`:
- Around line 346-358: Update the test setup before saving the user
configuration by calling _clear_provider_env_vars(monkeypatch). Place it after
setting SWITCHYARD_CONFIG_DIR and before save_user_config so
resolve_provider_connectivity() cannot use ambient provider credentials instead
of the saved Nvidia credentials.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8cc12026-f435-4d2f-8989-8c77ebe7ef06

📥 Commits

Reviewing files that changed from the base of the PR and between 2fa6b87 and 2e74ffb.

📒 Files selected for processing (9)
  • switchyard/cli/configure_command.py
  • switchyard/cli/launchers/launcher_runtime.py
  • switchyard/cli/launchers/proxy_health_monitor.py
  • switchyard/cli/switchyard_cli.py
  • switchyard/server/verify.py
  • tests/test_launch_claude.py
  • tests/test_launcher_proxy_bypass.py
  • tests/test_serve_profile_config.py
  • tests/test_user_config.py

Comment thread switchyard/cli/configure_command.py
Comment thread switchyard/cli/switchyard_cli.py
Comment on lines +30 to +33
monkeypatch.setenv("HTTP_PROXY", "http://127.0.0.1:9")
monkeypatch.setenv("http_proxy", "http://127.0.0.1:9")
monkeypatch.delenv("NO_PROXY", raising=False)
monkeypatch.delenv("no_proxy", raising=False)

@coderabbitai coderabbitai Bot Jul 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Use a test-owned or dynamically allocated proxy endpoint.

The test assumes 127.0.0.1:9 is unused. If a service is listening there, the pre-fix path may not fail deterministically. Start a local proxy stub that returns an error, or allocate an unused port for the negative proxy case.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tests/test_launcher_proxy_bypass.py` around lines 30 - 33, Update the proxy
setup in the affected test to use a test-owned proxy stub or dynamically
allocated local port instead of hard-coded 127.0.0.1:9. Ensure the endpoint
reliably returns a connection error for the negative proxy case, and apply the
dynamically selected endpoint to both HTTP_PROXY and http_proxy.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test asserts wait_for_proxy_ready returns True, which only happens if the loopback probe ignores the proxy and reaches the health server. Port 9 (discard) won't answer as a working HTTP forward proxy, so if the bypass ever regressed the probe would fail through :9 and the test would go red. What the test checks doesn't depend on :9 being unbound, so I'll keep the fixed port rather than spin up a stub proxy thread.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elyasmnvidian, understood. The test will catch the expected regression as long as port 9 is the normal discard service—or otherwise not an HTTP forward proxy. My concern is limited to the atypical, host-dependent case where another process provides a functioning proxy on that fixed port; using an owned endpoint would remove that dependency. I’ll leave the decision with you.

Comment thread tests/test_user_config.py
@elyasmnvidian

Copy link
Copy Markdown
Contributor Author

Live before/after — CLI behavior

These are CLI-behavior fixes, so the reproduction is command output rather than token usage. Ran the same commands on main and this branch.

1) serve --config with a duplicate model id (cfg.yaml has a profile dup and a target dup):

# main
_switchyard_rust.SwitchyardDuplicateRegistrationError: model "dup" is already registered
  ...raw traceback (10+ frames)...

# this branch
serve --config: model "dup" is already registered

2) serve with no config and no saved bundle:

# main
serve: no routing-profiles given. Pass --routing-profiles PATH or run `switchyard configure --routing-profiles PATH` to save one.

# this branch
serve: no config given. Pass --config PATH for a v2 profile config (recommended), or --routing-profiles PATH for a legacy route bundle.

3) Harness args after a single --launch claude --dry-run ... -- --harness-flag foo:

# main
switchyard: error: unrecognized arguments: --harness-flag        (the single -- was stripped)

# this branch
forwarded args: --harness-flag foo                               (forwarded to the harness)

The doubled--- workaround is no longer needed.

The saved-provider, non-interactive env-key, and HTTP_PROXY loopback-bypass fixes are covered by the regression tests added in this PR (test_user_config.py, test_launcher_proxy_bypass.py).

Audit follow-up in this push: ConfigureRequest.provider is now typed str | None (the CLI can leave it unset), and the --provider help text names the real resolution order (saved default_provider, else openrouter).

…, clarify serve errors

Signed-off-by: Elyas Mehtabuddin <emehtabuddin@nvidia.com>
(cherry picked from commit 967f72b)
Signed-off-by: Elyas Mehtabuddin <emehtabuddin@nvidia.com>
(cherry picked from commit aad16ca)
@elyasmnvidian

Copy link
Copy Markdown
Contributor Author

Reworked onto the new main (post-#98) — live before/after

#98 landed and rewrote the serve --config path, adding its own clean duplicate-model-id message. So I dropped this PR's duplicate-id fix (now redundant) and rebased the rest. The two behaviors below are still broken on current main (e09b5b03) and fixed here:

1) serve with no config — leads with the wrong flag on main:

# main
serve: no routing-profiles given. Pass --routing-profiles PATH or run `switchyard configure --routing-profiles PATH` to save one.

# this branch
serve: no config given. Pass --config PATH for a v2 profile config (recommended), or --routing-profiles PATH for a legacy route bundle.

2) Harness args after a single --launch claude --dry-run ... -- --harness-flag foo:

# main
switchyard: error: unrecognized arguments: --harness-flag     (the single -- was stripped)

# this branch
forwarded args: --harness-flag foo                            (forwarded to the harness)

The saved-provider, non-interactive env-key, and HTTP_PROXY loopback-bypass fixes are covered by the regression tests in this PR.

Signed-off-by: Elyas Mehtabuddin <emehtabuddin@nvidia.com>
Signed-off-by: Elyas Mehtabuddin <emehtabuddin@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant