Aggregate E2E normalized interactivity in AgentX CI results / 在 AgentX CI 结果中聚合端到端归一化交互性

[Duyi-Wang]

Summary

• Compute each valid profiling request E2EL/OSL ratio directly from profile_export.jsonl during AgentX aggregation.
• Apply the existing linear percentile policy to seconds per output token, then invert the matching mean/P50/P75/P90/P95 values into output tokens/s/user.
• Emit the derived metric at request_metrics.latency.e2e_norm_intvty in compact bmk_agentic_* JSON artifacts.
• Pair E2EL and OSL within the same request record, reject missing, nonpositive, and nonfinite samples, and preserve an empty {} object when no sample is valid.
• Keep full_response_intvty unchanged and do not add P99 or duplicate top-level aliases.

Motivation

The E2E normalized interactivity formula and Pareto policy were documented in #2502, but current CI results do not contain the metric. Consumers therefore have to download the much larger raw agentic_* artifact and reparse profile_export.jsonl. Computing it in the existing aggregation pass makes P75/P90 available from the compact result artifact without changing artifact upload workflows. The calculation follows the reference semantics in SemiAnalysisAI/InferenceX-app#638.

Output schema

{
  "request_metrics": {
    "latency": {
      "e2e_norm_intvty": {
        "mean": 33.33333,
        "p50": 33.33333,
        "p75": 28.57143,
        "p90": 26.31579,
        "p95": 25.64103,
        "std": 12.5
      }
    }
  }
}

Validation

• python -m pytest -q utils/agentic — 56 passed
• git diff --check
• python -m compileall -q utils/agentic/aggregation

中文说明

• 在 AgentX 聚合阶段直接读取 profile_export.jsonl，为每个有效的性能分析请求计算 E2EL/OSL。
• 先对“秒/输出 Token”使用仓库现有的线性插值策略计算 percentile，再对对应的 mean/P50/P75/P90/P95 求倒数，得到单位为 output tokens/s/user 的指标。
• 将派生指标写入紧凑的 bmk_agentic_* JSON，路径为 request_metrics.latency.e2e_norm_intvty。
• E2EL 与 OSL 必须来自同一条 request record；跳过字段缺失、非正数或非有限值样本；没有有效样本时保留稳定的空对象 {}。
• full_response_intvty 保持不变，不增加 P99，也不复制顶层 alias。

背景

#2502 已定义端到端归一化交互性公式与帕累托策略，但当前 CI 结果尚未包含该指标。消费者因此必须下载体积大得多的 agentic_* 原始产物，并重新解析 profile_export.jsonl。本 PR 在现有聚合流程中完成计算，使 P75/P90 可直接从小体积结果产物读取，无需修改 artifact 上传 workflow。计算语义与 SemiAnalysisAI/InferenceX-app#638 的参考实现一致。

验证

• python -m pytest -q utils/agentic — 56 项测试通过
• git diff --check
• python -m compileall -q utils/agentic/aggregation

[claude]

Claude finished @Duyi-Wang's task in 3m 29s —— View job

---

Review of PR #2544

• Gather context and read the PR diff
• Review metric computation logic (request_metrics.py, aggregation_common.py)
• Review tests
• Run test suite (blocked — pytest is not in my allowed tools for this run; verified the arithmetic by hand instead, see below)
• Post review feedback

LGTM - no blocking issues found

Verification notes:

• The test expectations match the repo's linear-interpolation percentile helper exactly: ratios [0.02, 0.04] → mean 1/0.03 ≈ 33.33, p75 1/0.035 ≈ 28.57, p90 1/0.038 ≈ 26.32, and pstdev([50, 25]) = 12.5 — consistent with the PR's schema example.
• Inverting per-percentile (rather than taking percentiles of inverted rates) matches the existing intvty / full_response_intvty policy in _interactivity_stats, and correctly yields p50 ≥ p75 ≥ p90 ≥ p95 slow-tail semantics.
• The to_int OverflowError fix in aggregation_common.py:50 is load-bearing, not incidental: without it, int(float("inf")) from the new nonfinite-OSL test records would crash compute_workload_stats.
• Same-record pairing, nonpositive/nonfinite rejection, and the empty-{} fallback are all covered by tests; warmup/error record exclusion is exercised via the modified test_processor_surfaces_request_accounting (changed E2E values on excluded records don't affect the metric).
• The added math.isfinite filter in _ms_to_s slightly tightens existing ttft/e2el/itl stats (previously inf passed the > 0 check) — an improvement, not a regression.
• No downstream schema consumers break: utils/compare_results.py iterates a fixed key list on the non-agentic path.

I could not execute python -m pytest -q utils/agentic because Bash approval for pytest was not granted in this run — if you want CI-independent confirmation, please rely on the PR's CI checks or add pytest to the action's --allowedTools.

---

[claude]

LGTM — small, well-tested additive metric in the AgentX aggregation pipeline.

What was reviewed: the new _e2e_normalized_interactivity_stats E2EL/OSL pairing and percentile-then-invert math (verified by hand against the new test fixtures), the missing/nonpositive/nonfinite sample filtering, and the empty-{} fallback path. Also checked that e2e_norm_intvty is additive to the schema (existing intvty/full_response_intvty keys and top-level aliases are untouched) and that no other code in this repo reads the request-metrics schema keys, so there's no forward-compat break.

Extended reasoning...

Overview

This PR adds a new derived metric, request_metrics.latency.e2e_norm_intvty (E2E-normalized interactivity, i.e. output tokens/s/user), to the AgentX CI aggregation pipeline. It reads E2EL and OSL from the same profiling record in profile_export.jsonl, computes a per-request seconds-per-output-token ratio, applies the repo's existing linear percentile interpolation to that ratio, and inverts mean/P50/P75/P90/P95 into the new metric. It reuses the existing _interactivity_stats helper (previously only used for ITL-based intvty), so the percentile/invert logic itself is not new — only its application to the E2EL/OSL ratio is. Two small defensive fixes are bundled in: to_int now also catches OverflowError, and _ms_to_s now filters non-finite values via math.isfinite. The bulk of the diff is new unit tests covering missing-field pairing, non-positive/non-finite sample rejection, and the empty-object fallback when no sample is valid.

Security risks

None. This is CI result aggregation/reporting code operating on locally-produced benchmark JSON files — no user input, network calls, auth, or credential handling involved.

Level of scrutiny

Low-to-moderate is appropriate: this is non-production, CI-artifact-generation code, purely additive to the output schema (no existing keys changed or removed), and covered by 8 new targeted unit tests plus the existing 56-test suite (per the PR description). I manually verified the percentile/inversion arithmetic against the new test fixtures (e.g. ratios 0.02/0.04 → mean 1/0.03, p75 1/0.035, p90 1/0.038, std 12.5) and it is correct.

Other factors

No other code in this repository consumes the specific new schema key, so there's no risk of breaking existing downstream readers within this repo (the PR notes a separate consumer in SemiAnalysisAI/InferenceX-app#638, which is out of scope for this review). There are no outstanding review comments to address, and this is a first-pass review with no prior approvals from me on this PR.