fix(agent): Align model defaults and eval contracts#964
Conversation
Use the production base, handoff, and auxiliary models across runtime configuration, evals, tests, and docs. Keep deterministic behavior out of model judges, enforce the reply deadline, and warm sandbox snapshots before timed eval cases.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Use five bounded attempts with capped exponential backoff so transient Cloudflare API timeouts do not abort the eval suite.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Resolve the reply promise or abort signal based on which settles first, preserving the hard deadline without discarding a successful boundary completion.
Give the production-aligned base model enough time for tool-backed turns while retaining a bounded per-reply deadline.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a012500. Configure here.
| return await Promise.race([operation(), abortPromise]); | ||
| } finally { | ||
| removeAbortListener(); | ||
| } |
There was a problem hiding this comment.
Abort race drops losing rejection
Medium Severity
raceWithAbort returns whichever of the agent run or abort settles first, but does not settle or swallow the loser. When the deadline wins, a later rejection from executeAgentRun becomes an unhandled rejection after the timed-out eval has already failed.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit a012500. Configure here.
There was a problem hiding this comment.
I verified this is a false positive. Promise.race attaches fulfillment and rejection handlers to every input promise immediately; when the abort promise wins, a later rejection from executeAgentRun is consumed by the rejection handler already installed by Promise.race and does not become unhandled. I also reproduced the winner/late-reject case under an unhandledRejection listener with no event emitted, so no code change is needed here.
— Claude Code
Allow cleanup and rubric judging to complete after the 60-second reply budget expires.
github-actions
Bot
added
risk: high
1 participant
Runtime and eval configuration now match the production model lineup: Grok 4.5
for standard turns, Claude Haiku 4.5 for auxiliary work, and GPT-5.6-sol for
handoffs. Config tests and public reference docs carry the same defaults.
Eval coverage now reserves rubric judges for nondeterministic visible replies.
Deterministic tool calls, attachments, delivery, and persistence use typed
selectors with assertions at each case, while duplicate fixture-only cases and
their unused reply injection hooks are removed.
Sandbox-backed suites prepare their dependency snapshots in
beforeAll, whichmatches production warmup and keeps cold snapshot construction outside the
60-second reply budget. Agent execution races that deadline directly so a
successful boundary completion is not discarded after returning. Invocation-scoped
Redis coordinates are installed before stateful worker modules load so local and
CI eval runs share the intended state.
Cloudflare Quick Tunnel allocation now uses five bounded attempts with capped
exponential backoff so transient API timeouts do not abort the suite immediately.