Skip to content

fix: 首字停摆自动重试 — init 后首个 token 停摆时自动重发 query#265

Open
lishuceo wants to merge 2 commits into
mainfrom
fix/first-token-stall-retry
Open

fix: 首字停摆自动重试 — init 后首个 token 停摆时自动重发 query#265
lishuceo wants to merge 2 commits into
mainfrom
fix/first-token-stall-retry

Conversation

@lishuceo

Copy link
Copy Markdown
Owner

背景

土豆儿等 bot 偶发 Query idle timeout after 300s with no activity (total elapsed: 303s)。查日志确认失败模式:

idle timeout | lastResetSource=msg:system:init | hasToolActivity=false | idleTimeoutMs=300000 | elapsedMs=301129

即:收到 system:init 后,首个 assistant token 因网关/API 瞬时抖动迟迟不到,300s 后空闲超时直接判失败 —— 让用户白等 5 分钟还收到报错。这是可重试的瞬时故障,不该当作终态错误。

改动

  • executor 跟踪两个新状态:
    • firstTokenSeen —— assistant 消息 / stream_event / 工具活动任一到达即置位
    • timeoutKind —— 区分 idle(空闲,可能首字停摆)与 hard(硬性总超时,不重试)
  • catch 分支识别「idle 超时 + 从未收到首个 token」组合,带退避(1.5s)递归重发 query。保留 resume(会话状态未推进,重发安全),上限取 config。
  • 新增 CLAUDE_FIRST_TOKEN_STALL_RETRIES 配置(默认 2,设 0 关闭)。

只在「首字停摆」时重试 —— 一旦有任何 token / 工具活动产出,即视为普通空闲超时,不重试(避免重复已产出的工作)。

测试

src/claude/__tests__/executor.test.ts 新增 4 个回归用例(fake timers 模拟停摆):

  • 停摆一次后重试成功
  • 超过上限后返回 idle timeout 错误(初次 + 2 次重试 = 3 次 query)
  • 首个 token 已到达的普通空闲超时不重试
  • firstTokenStallRetries=0 时关闭重试

全量回归:1688 passed(唯一失败 memory/quality.test.ts > 当前工作重点 在 base 上同样失败,是本工作区缺 DASHSCOPE_API_KEY 导致的环境问题,与本 PR 无关)。

🤖 Generated with Claude Code

lishuceo and others added 2 commits June 17, 2026 16:11
收到 system:init 后,首个 assistant token 因网关/API 瞬时抖动迟迟不到,
300s 后空闲超时直接判失败,让用户白等 5 分钟还收到报错。这是可重试的
瞬时故障,不该当作终态错误。

- executor 跟踪 firstTokenSeen(assistant 消息 / stream_event / 工具活动
  任一到达即置位)与 timeoutKind(idle / hard)
- catch 分支识别「idle 超时 + 从未收到首个 token」组合,带退避递归重发
  query(保留 resume,会话状态未推进),上限取 config
- 新增 CLAUDE_FIRST_TOKEN_STALL_RETRIES 配置(默认 2,设 0 关闭)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
用 fake timers 模拟 init 后停摆,覆盖四个场景:
- 停摆一次后重试成功
- 超过 firstTokenStallRetries 上限后返回 idle timeout 错误
- 首个 token 已到达的普通空闲超时不重试
- firstTokenStallRetries=0 时关闭重试

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Claude finished @lishuceo's task in 7m 13s —— View job


Review: 首字停摆自动重试 (PR #265)

✅ Approved — clean, well-scoped fix with solid regression coverage.

  • Read changed files & understand context
  • Review executor.ts retry logic
  • Review config.ts changes
  • Review test coverage
  • Post review feedback

What the PR does

Distinguishes idle vs hard timeout and tracks firstTokenSeen (set on assistant / stream_event / canUseTool). In the catch branch, an idle timeout with no token ever received is treated as a transient first-token stall and the query is re-sent (with backoff, up to CLAUDE_FIRST_TOKEN_STALL_RETRIES, default 2).

Correctness — looks good ✅

  • firstTokenSeen gating is sound. It's set in all three places a token/activity could surface (assistant L1064, stream_event L1183, canUseTool L849). Resuming is only safe when the session truly hasn't advanced, and this guard enforces exactly that. Re-sending with resume after a no-token stall is the correct call.
  • Timer & resource cleanup before recursion is correct. clearTimeout(idleTimer) / clearTimeout(hardTimer) (L1195–1196), runningQueries.delete (L1197), and q.close() (L1233) all run before the recursive execute(), so no leaked timers, stale map entries, or orphaned subprocesses.
  • Retry-count arithmetic is right. stallRetries < maxStallRetries yields initial + N retries = N+1 queries; the "gives up after 2" test confirms 3 total calls.
  • Doesn't shadow the message-size retry (separate error class, no counter interaction) and _firstTokenStallRetries is properly documented as internal-only.

Observations (non-blocking, no change required)

  • 🟡 Worst-case wait grows ~3× (confidence ~70, design trade-off not a bug). With defaults (CLAUDE_TIMEOUT=300, retries=2), a persistently-down gateway now fails after 300 + 300 + 300 ≈ 15 min instead of 5. Acceptable given stalls are assumed rare/transient and a retry usually recovers fast — but worth being aware of for the always-failing case. If it ever bites, a shorter idle timeout on the detection attempt (vs. full 300s each round) would bound it. Not needed now.
  • ℹ️ The stream_event → firstTokenSeen branch (L1183) is effectively defensive: includePartialMessages isn't set on the query, so stream_event likely never fires. Harmless, and good to keep as a safety net.

Tests — comprehensive ✅

The 4 fake-timer cases cover the matrix well: stall-then-recover, exhaust-retries, no-retry-after-first-token (the key guard), and retries=0 disable. The config mock was updated consistently. I couldn't execute the suite here (test-run command not permitted in this job), but the test logic matches the implementation; the author reports 1688 passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant