Add pinned Kimi Vendor Verifier smoke / 添加固定版本的 Kimi Vendor Verifier 冒烟评估 - #2547
Add pinned Kimi Vendor Verifier smoke / 添加固定版本的 Kimi Vendor Verifier 冒烟评估#2547adibarra wants to merge 29 commits into
Conversation
中文：添加基于 Kimi Vendor Verifier 原生实现的工具调用评估
|
Thanks for the contribution! Please reach out to respective companies' CODEOWNER to fill in the latest PR_REVIEW_CHECKLIST.md before pinging core maintainer on Slack for review. In order for the signoff PR check bot to trigger, you must follow the PR_REVIEW_CHECKLIST.md template correctly, including the phrase For PR verification, add the PR authors are responsible for ensuring that after merging, all GitHub Action jobs fully pass. A lot of the time, failures are just flakes and simply re-running the failed jobs will fix it. See GitHub's docs on re-running failed jobs 感谢你的贡献！请联系相应公司的 CODEOWNER 填写最新的 PR_REVIEW_CHECKLIST.md，然后再在 Slack 上联系核心维护者进行审阅。为了触发 signoff PR 检查机器人，你必须正确遵循 PR_REVIEW_CHECKLIST.md 模板，包括保留英文语句 如需进行 PR 验证，请为此 PR 添加 PR 作者有责任确保合并后所有 GitHub Action 任务完全通过。 很多时候失败只是偶发抖动（flake），重新运行失败的任务即可解决。参见 GitHub 关于重新运行失败任务的文档 |
中文：精简工具调用评估实现并接入工作流选择路径
中文：合并最新主分支以完成发布准备
中文：隔离 Kimi 验证器依赖并明确供应商适配边界，同时加入执行超时和通用结果格式标识。
中文：合并最新 main，确保发布前验证基于当前代码。
adibarra
changed the title
中文：区分失败结果产物写入成功与写入失败，同时保留原始安装失败退出码。
中文：将 agentic 评估矩阵的投机解码模式传递给启动器，避免 MTP 配置静默回退到 STP。
中文：合并最新 main，准备发布。
中文：补全纯评估工作流的结果收集依赖，避免所有评估任务成功后工作流仍显示失败。
The collector dependency change did not affect GitHub’s workflow conclusion and added unrelated orchestration scope. 中文：恢复纯评估结果收集的原有范围。该依赖调整未改变 GitHub 工作流结论，且扩大了无关改动范围。
中文：合并最新 main，准备进入评审。
|
Claude finished @adibarra's task in 6m 48s —— View job PR Review
LGTM - no blocking issues found Verified end-to-end: the always-set |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Beyond the inline findings, I also checked the multi-node rejection path in _run_kimi_tool_call_schema_eval (benchmarks/benchmark_lib.sh): when IS_MULTINODE is true it returns early without writing a zero-score compatibility artifact, unlike the setup-failure paths. This is a real asymmetry but not a bug in this PR's scope — run_eval's scenario dispatch never routes multi-node jobs to kimi-vendor, so the path is unreachable in practice today.
Extended reasoning...
I reviewed the multi-node rejection branch in _run_kimi_tool_call_schema_eval because it looked structurally similar to the other early-return failure paths that do write a zero-score compatibility artifact via _write_kimi_vendor_integration_error. This one instead returns exit code 2 with only a stderr message and no artifact. Tracing the call graph, run_eval's framework dispatch is single-node only for the kimi-vendor case in this PR (Phase 1 is explicitly scoped to single-node), so there's currently no caller that reaches this branch with IS_MULTINODE=true. I'm noting it as examined-and-ruled-out rather than filing it, since it doesn't affect current CI behavior.
中文：修正验证器失败样本数、共享格式契约、路径复用及双语文档链接。
中文：保留结果收集器测试的现有格式，仅添加格式契约测试。
中文：在启动失败时保留验证器计划样本数，同时将有效样本数记录为零。
中文：合并最新 main，并保留工具调用评估元数据。
中文：保留可配置的 eval 调度行为，并修正失败返回码、Bash 兼容性和 eval 文档。
中文：强化验证器评审路径。
|
Claude finished @adibarra's task in 6m 17s —— View job Re-review (new changes since 2026-08-10T23:01Z review)
Re-review: one 🟡 warning — the latest commit re-broke the bilingual doc line anchors that an earlier commit had fixed (inline comment on Everything else in the new commits checks out:
Not verified locally: the focused pytest suites and |
| 5. `run_eval` and artifact staging run. | ||
|
|
||
| Relevant implementation: [context setup](../benchmarks/benchmark_lib.sh#L853-L888), [eval dispatch and failure policy](../benchmarks/benchmark_lib.sh#L1537-L1654), and [workflow inputs](../.github/workflows/benchmark-tmpl.yml#L162-L185). | ||
| Relevant implementation: [context setup](../benchmarks/benchmark_lib.sh#L1049-L1078), [eval dispatch and failure policy](../benchmarks/benchmark_lib.sh#L1789-L1908), and [workflow inputs](../.github/workflows/benchmark-tmpl.yml#L79-L97). |
There was a problem hiding this comment.
🟡 WARNING: The latest commit (294e39d51) re-broke the bilingual line anchors this PR previously fixed. That commit inserted 13 lines into benchmarks/benchmark_lib.sh (the EVAL_SUITE guard at L1818–L1829, inside run_eval) without refreshing the docs, so every benchmark_lib.sh anchor pointing past L1815 in this file and docs/eval-agentx-procedures_zh.md is now 13 lines early.
Why it matters: The links land on the wrong code — e.g. "run_eval batching" (#L1824-L1885) now starts at the esac of the new EVAL_SUITE guard and cuts off before the deferred-failure tail, and "trace resolution" (#L2008-L2087) starts mid-pip install list instead of at resolve_trace_source() (now L2021). This also regresses a finding from the previous review that had been addressed, and AGENTS.md requires docs to track the implementation in both languages.
Fix: Shift these 8 anchors by +13 in both docs/eval-agentx-procedures.md and docs/eval-agentx-procedures_zh.md (same lines/values in each):
| Doc line | Current | Correct |
|---|---|---|
| 100 | #L1789-L1908 |
#L1789-L1921 |
| 124 | #L1824-L1885 |
#L1837-L1898 |
| 176 | #L2089-L2113 |
#L2102-L2126 |
| 209 | #L2251-L2253 |
#L2264-L2266 |
| 213 | #L2008-L2087 |
#L2021-L2100 |
| 213 | #L2089-L2255 |
#L2102-L2268 |
| 245 | #L2305-L2345 |
#L2318-L2358 |
| 294 | #L2221-L2245 |
#L2234-L2258 |
(Anchors at or before L1815 — #L1049-L1078, #L1080-L1162, #L1163-L1222 — are still correct.)
There was a problem hiding this comment.
LGTM — the one new finding (EVAL_SUITE leaking across repeated run_eval calls in the same sourced shell) is a nit with a trivial workaround and no CI impact, since every workflow job runs run_eval once in a fresh process.
Extended reasoning...
This run's bug hunt surfaced a single nit: a plain (non-local) export EVAL_SUITE=... in run_swebench_eval/run_kimi_vendor_eval leaks into the calling shell, which can trip the new EVAL_SUITE guard in run_eval on a second call within the same sourced shell. This only affects interactive/manual multi-call workflows, not CI (each CI job runs run_eval once in a fresh process), and is easily worked around with unset EVAL_SUITE.
The rest of the diff is the same Kimi Vendor Verifier smoke-eval integration already covered by prior review passes on this PR (dispatch, dependency isolation, timeout/cleanup, result projection, thresholding, reuse identity, and STP/MTP forwarding), plus a ruled-out candidate issue (zero-score artifact handling for unsupported-suite/multinode failures) that was checked and is not a real bug. Given the change is CI/tooling scoped, opt-in behind new workflow inputs with unchanged defaults for existing jobs, and the only new finding is a low-severity nit, I don't think this warrants blocking approval.
| local out_dir="${EVAL_RESULT_DIR:-$(mktemp -d /tmp/eval_out-XXXXXX)}" | ||
| local task_name="${SWEBENCH_TASK_NAME:-swebench_lite}" | ||
| export EVAL_SUITE="${EVAL_SUITE:-$task_name}" | ||
| local gen_dir |
There was a problem hiding this comment.
🟡 run_swebench_eval() and run_kimi_vendor_eval() both do a plain (non-local) export EVAL_SUITE=..., which leaks into the calling shell after the function returns. Because run_eval() now rejects any set EVAL_SUITE for a non-kimi-vendor framework, a later run_eval --framework lm-eval (or a second --framework swebench) call in the same sourced shell fails immediately with "EVAL_SUITE is only supported with EVAL_FRAMEWORK=kimi-vendor" before dispatching, breaking the documented "source benchmark_lib.sh once, run_eval repeatedly" workflow. Fix by resetting/unsetting EVAL_SUITE at the top of run_eval, or scoping the export to the duration of the dispatched call.
Extended reasoning...
What the bug is. run_eval() in benchmarks/benchmark_lib.sh adds a new guard: if [ -n "${EVAL_SUITE:-}" ] && [ "$framework" != "kimi-vendor" ]; then ... return 2; fi. This is meant to stop a stray EVAL_SUITE override from silently being ignored by non-kimi frameworks. However, both run_swebench_eval() (export EVAL_SUITE="${EVAL_SUITE:-$task_name}") and run_kimi_vendor_eval() (export EVAL_SUITE="$eval_suite") use a plain, non-local export. Bash exports made this way persist in the current shell (or any shell that sourced benchmark_lib.sh) even after the function that set them returns.
How it manifests. In the documented interactive workflow — source benchmarks/benchmark_lib.sh once, then issue multiple run_eval calls in the same shell/process (see docs/eval-agentx-procedures.md and utils/evals/EVALS.md) — the first call to run_eval --framework swebench (or --framework kimi-vendor) leaves EVAL_SUITE set afterward (e.g. EVAL_SUITE=swebench_lite). Any subsequent run_eval call with a different (or even the same) non-kimi framework in that shell then hits the new guard and fails with rc=2 and "ERROR: EVAL_SUITE is only supported with EVAL_FRAMEWORK=kimi-vendor" — before ever dispatching to the underlying runner. Notably, this even blocks a second run_eval --framework swebench call in the same shell, and the error message is confusing since the user never set EVAL_SUITE themselves.
Step-by-step proof:
source benchmarks/benchmark_lib.shrun_eval --framework swebench --port 8888— this callsrun_swebench_eval(), which executesexport EVAL_SUITE="${EVAL_SUITE:-swebench_lite}". This export is not scoped to the function; it persists in the shell even if the swebench run itself later fails, because the export happens before any dataset/generation logic.- Back at the shell prompt,
EVAL_SUITE=swebench_liteis now exported globally. run_eval --framework lm-eval --port 8888is issued next (e.g. to run a GSM8K smoke after a SWE-bench smoke).run_eval()'s guard seesEVAL_SUITEis non-empty andframework != kimi-vendor, prints the error, and returns 2 —run_lm_evalis never invoked.- The same failure recurs for a second
run_eval --framework swebenchcall in that shell, since the guard fires for any framework other thankimi-vendoronceEVAL_SUITEis set.
Why nothing currently catches it. The guard and both leaking exports are new in this PR (the diff adds export EVAL_SUITE="${EVAL_SUITE:-$task_name}" inside run_swebench_eval, the new run_kimi_vendor_eval export, and the new guard in run_eval), so this is a PR-introduced interaction, not a pre-existing issue. utils/evals/test_run_eval_dispatch.py tests each run_eval/run_kimi_vendor_eval/run_swebench_eval invocation in an isolated subprocess (subprocess.run(["bash", "-c", ...])), so no test exercises two sequential run_eval calls sharing one shell/process, and the leak never surfaces in CI.
Impact and fix. In CI this is low-impact: every workflow job runs run_eval exactly once in a fresh process, so the score gate and sweep are unaffected. The impact is confined to interactive/manual usage that follows the documented "source once, call run_eval repeatedly" pattern, and is trivially worked around with unset EVAL_SUITE. The straightforward fix is to reset/unset EVAL_SUITE at the top of run_eval() before framework dispatch (or to scope the export in run_swebench_eval/run_kimi_vendor_eval to only the duration of the call, e.g. via a subshell or by passing the value directly to the dispatched function instead of exporting it into the caller's shell).
中文：限定评估套件状态的作用域，并修正双语 AgentX 文档中的源码行号链接。
中文：将发生不可纠正 NVLink 错误的 b300-017 节点排除在 Slurm 分配之外。
中文：让 B300 启动器将 SALLOC_EXCLUDE 传递给 salloc，避免调度到已知故障节点。
中文：启用多节点 Kimi 验证器
中文：强化 Kimi 评估运行时故障处理
中文：捕获 Kimi 工具调用响应诊断数据。
This reverts commit 134906e.
中文：采集确定性的 Kimi 工具调用诊断数据。
中文：启用 Kimi 结构化工具调用约束。
中文：格式化 Kimi 配方回归测试。
中文：重试 Kimi 验证器的临时下载失败。
1 participant
Summary
tests/tool_call_json_schemasuite in non-streaming and streaming modes. The adapter pins upstream commitb9ed3a6665bdff2c943246f7d2903cd003d6ddd6.eval-framework: kimi-vendorandeval-suite: kimi_tool_call_schemaworkflow inputs. Existing jobs keeplm-evaland GSM8K defaults.inferencex-eval-v1results, use exact-match as the primary score, enforce a1.0threshold, aggregate the suite identity, and preserve it during reusable-artifact validation.n_eff: 0; valid two-mode reports reportn_eff: 2.spec-decodinginto agentic eval jobs so STP and MTP configurations launch their matching benchmark scripts.Scope
perf-changelog.yaml, model configurations, images, and Kimi benchmark scripts are unchanged by this PR.Validation
128 passed.actionlint, andgit diff --checkpassed.2 passed.58858593fran distinct STP and MTP Kimi K3 FP4 vLLM jobs. Both jobs, collection, and score validation succeeded. The aggregate contains two rows,spec_decoding: noneandspec_decoding: mtp, each withem_strict: 1.0andn_eff: 2.中文说明
概要
tests/tool_call_json_schema测试集，覆盖非流式和流式两种模式。适配器固定到上游提交b9ed3a6665bdff2c943246f7d2903cd003d6ddd6。eval-framework: kimi-vendor和eval-suite: kimi_tool_call_schema。现有任务继续默认使用lm-eval和 GSM8K。inferencex-eval-v1结果，以精确匹配率作为主评分，设置1.0阈值，在聚合及可复用产物校验中保留测试集标识。n_eff: 0，有效的双模式报告记录n_eff: 2。spec-decoding传给 Agentic 评估任务，确保 STP 和 MTP 配置启动对应的基准测试脚本。范围
perf-changelog.yaml、模型配置、镜像或 Kimi 基准测试脚本。验证
128 passed。actionlint和git diff --check均通过。2 passed。58858593f上完成真实 B300 验证，分别运行 STP 和 MTP 的 Kimi K3 FP4 vLLM 任务。两个任务、结果收集和分数校验均成功。聚合结果包含spec_decoding: none与spec_decoding: mtp两行，两者均为em_strict: 1.0、n_eff: 2。