test/benchmark: turn bench_agents.py into an adaptive stress test - #1436
Open
sufubao wants to merge 5 commits into
Open
test/benchmark: turn bench_agents.py into an adaptive stress test#1436sufubao wants to merge 5 commits into
sufubao wants to merge 5 commits into
Conversation
给 bench_agents.py 加了一键健壮性/性能测试套件,并修了几个会导致结果失真的 bug。 Bug 修复: - 流式 token 字段捕获:原来只读 delta.content / reasoning_content,thinking 模型输出在 reasoning 字段时会整条流 0 token,误报 "no tokens"。改为通过 pydantic model_extra 兜底读取 reasoning / reasoning_content / text 等任意字段。 - 并发重复派发:fire_one 只检查 s.pending 就 create_task,但 pending 在 do_turn 里才置位;catch-up 突发时同一 session 会被重复选中(饱和区最关键处失真)。 改为 fire_one 里先占位再调度。 - rr_idx 污染:do_turn 的 finally 里不该自增发射游标 rr_idx,会与 fire_one 的 游标推进冲突,移除。 - ITL 计算:原仅取 token1→token2 单点采样,改为全部相邻间隔的均值,并补上汇总输出。 新增能力: - --suite:一键跑 smoke -> QPS 扫描(找饱和拐点) -> 过载 -> idle-then-burst, 可选 --full 追加长稳阶段。每阶段落 *.log,结尾输出 markdown 总表 report.md。 - 实时进度 reporter(--report-sec):每行打印 fired/blocked/ok/err/pend/achQPS 及当前 TTFT/TPOT,便于长跑观察。 - 默认端口改为 8000;文件头补完整用法说明(单次 bench + 套件两种模式、关键参数、 输出判读)。
原来 history 里追加的是硬编码桩 "(reply)",下一轮 prefix 几乎不增长, 不像真实多轮 agent。改为累计本轮流式真实输出(content + reasoning)写回 history,使每轮 prefix 真实增长,模拟真实 agent 调用。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
test/benchmark/bench_agents.py用多 session、独立长 system prompt 和持续增长的多轮历史模拟 agent 流量。本次把原来的固定健壮性/性能套件改为真正的自适应压力测试:持续递增负载,找到饱和点后保持压力,便于暴露过载、超时和长时间运行问题。压力测试模式
--stress:从--stress-start-qps开始,按--stress-step-qps逐级增加到--stress-max-qps。--stress-step-sec;一旦达到饱和条件就停止递增,并在该 QPS 持续运行--stress-hold-sec。--stress-min-achieved-ratio;--stress-max-error-rate;--stress-max-blocked-rate。<out>/<phase>.log;最终生成<out>/report.md,标出首个饱和 QPS、失败率、blocked 比例、TTFT 等数据。--quick将 ramp/hold 时长缩短为四分之一,便于快速验证流程。一并保留/修复
reasoning/reasoning_content/text等字段中的流式输出。用法
改动范围
仅修改
test/benchmark/bench_agents.py,不影响服务端运行时逻辑。