fix(agent): refresh the gateway token so long runs stop 401ing - #1054
Open
Twixes wants to merge 3 commits into
Open
fix(agent): refresh the gateway token so long runs stop 401ing#1054Twixes wants to merge 3 commits into
Twixes wants to merge 3 commits into
Conversation
Wizard access tokens expire after an hour, but the agent runs as one long-lived subprocess whose env is fixed at spawn, so the token can't be replaced once it goes stale. Any run past the hour dies on a 401. Hand the SDK a `settings.apiKeyHelper` script instead of a fixed token. It refreshes when the token is nearly out and prints whatever is currently valid, so the access-token window stays at an hour rather than widening.
🧙 Wizard CIRun the Wizard CI and test your changes against wizard-workbench example apps by replying with a GitHub comment using one of the following commands: Test all apps:
Test all apps in a directory:
Test an individual app:
Show more apps
Results will be posted here when complete. |
…s a string The helper was a template literal written to a temp dir at runtime, which meant no typecheck, no lint, and hand-escaped source. Make it a real module with its own tsdown entry, shipped as dist/auth-helper.js. It gets its state file path from an env var now, since it no longer sits beside it. Typing the refresh response immediately caught an untyped `response.json()` the string had hidden.
Member
|
okay i totally get where this is coming from, but i'm worried about changing the core auth mechanics. i think this might be annoying to handle for different OSs like windows. i also don't know how i feel about refresh tokens living on disk either i think we can have token refreshes and keep everything in memory if we create a little token server that talks to each subprocess |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Wizard access tokens are good for exactly one hour. The agent runs as a single long-lived subprocess, and its env is fixed at spawn time, so once that token goes stale there's no way to hand it a new one. Any run that crosses the hour mark dies on a 401 from the LLM gateway.
The easiest way to hit it is a
wizard_askthat nobody answers. The prompt times out after 30 minutes, so the agent sits idle straight through its own token's expiry and only finds out when it resumes. Two unanswered prompts is enough to get there, and the run is dead by then.It's also confusing when it lands. All our local diagnostics come back clean (no settings conflict, no stored Claude login), so the auth-error screen falls through to its generic branch and starts advising about key types and scopes, none of which is the actual problem. People reasonably read that as "I passed the wrong key".
Changes
Rather than bake a token into the subprocess env, hand the SDK
settings.apiKeyHelper: a small script it re-runs on its own cadence. The script refreshes when the token is nearly out and prints whatever is currently valid.The thing I like about this over the obvious alternative: it keeps the access-token window at an hour instead of widening it. We could have just given the wizard's OAuth app a longer TTL server-side, but that means a longer-lived bearer token sitting on a laptop, and these tokens carry write scopes on the user's project (feature flags, dashboards, insights). Short-and-rotating is the better trade.
Details worth knowing:
Credentials.dist/auth-helper.js), since the SDK execs it as a standalone process. It gets its state file path from an env var on the agent subprocess, because a shebang can't portably take arguments.Test plan
src/lib/agent/__tests__/rotating-credential.test.tsruns the real helper as a subprocess against a local token endpoint, covering both branches: refresh when the token is near expiry (and persist the rotated refresh token so the spent one is never reused), pass through untouched when it's healthy. Takes about half a second.Full suite green at 1718 tests, typecheck at parity with main.
Known gaps, happy to follow up
Authorizationheader for the whole run, so it has exactly the same expiry exposure that this PR fixes for model calls.SIGKILL'd wizard leaves it in/tmp, and refresh tokens are good for 30 days versus one hour for the access token. Worth weighing against the "short-lived credential" argument above. Moving refresh into core (the parent process is alive for the whole run and could answer the helper over a socket) would keep it in memory and fix the two gaps above at the same time.The first two both fall out naturally if we later move to a session-owned "give me a currently-valid token" accessor, which is probably the right end state. This PR deliberately doesn't go there.
LLM context
Co-authored with Claude Code. The root cause was confirmed by reading the actual OAuth token row for a failing run (created and expires exactly 3600s apart) and matching it against the gateway's 401s, rather than inferred from the client side. Worth stating because the gateway currently logs nothing about why auth failed, which made this much harder to pin down than it should have been.