bug: Copilot CLI 1.0.21 added a startup model validation step: when `COPILOT_MODEL` is set

[jaroslawgajewski]

Root Cause

Copilot CLI 1.0.21 added a startup model validation step: when COPILOT_MODEL
is set, the CLI calls GET /models to validate the requested model name before
execution. The /models endpoint does not accept classic PATs (ghp_*).

Workflows that do not set COPILOT_MODEL (i.e., use the default model)
are unaffected — they never trigger the validation call.

Affected Scope

• Trigger: COPILOT_MODEL is set in the agent environment.
• Token type: Classic PAT (ghp_*) used as COPILOT_GITHUB_TOKEN.
• Unaffected: Workflows without COPILOT_MODEL; OAuth or fine-grained tokens.

Reproduction

1. Use gh-aw v0.67.3 with COPILOT_GITHUB_TOKEN set to a classic PAT.
2. Set COPILOT_MODEL=gpt-5.4 (or any model) in the agent environment.
3. Trigger any workflow. Observe exit code 1 on the "Execute GitHub Copilot CLI" step.

Expected Behaviour

Model validation should either:

• Accept classic PATs (parity with previous CLI versions), OR
• Fail gracefully with a clear diagnostic pointing to the token type issue.

Prior Versions

gh-aw v0.67.2 (Copilot CLI 1.0.20) did not perform model validation on
startup. This regression was introduced silently in v0.67.3.

Environment

• gh-aw CLI: v0.67.3
• AWF: v0.25.16
• Copilot CLI: 1.0.21
• GitHub MCP Server: v0.32.0
• MCP Gateway: v0.2.16
• COPILOT_GITHUB_TOKEN type: classic PAT

[jaroslawgajewski]

possibly will address also #25494

[raye-deng]

This regression description is excellent, especially the way you narrowed it to the COPILOT_MODEL path plus classic PATs.

We've run into the same class of failure in AI-assisted pipelines: everything looks healthy until one extra validation call hits a newer endpoint or auth assumption, and then the workflow dies before any useful work starts. The frustrating part is that traditional code-quality checks usually stay green because the break is in the runtime contract between tool, token type, and API behavior.

What helped us was adding a PR-stage compatibility check for exactly these "API drift" edges, things like auth-mode assumptions, model-name validation, and package/version availability, alongside normal static analysis. It ends up complementing SonarQube well because Sonar catches code smells, but not these AI-tooling contract mismatches.

If it's useful to compare approaches, we've been testing that with OCR here:

npx @opencodereview/cli scan . --sla L1

It won't fix this specific token regression by itself, but it has been useful for catching the same family of silent AI-pipeline breakages earlier.