perf: DVFS/thermal-aware stable measurement (`--stable`) + run-context capture for cross-run comparability

[DingmaomaoBJTU]

Summary

winml perf currently reports latency from a single contiguous back-to-back loop (warmup + iterations), surfacing Avg/P50/P90/P99/Min/Max/Std. On NPU (and any DVFS/thermal-managed device) this makes results noisy and hard to reproduce or compare across runs — the exact pain a model owner raised:

"The time measures I was getting were quite noisy and could change quite a bit over time (maybe depending on the machine temperature)… Using the min duration over multiple runs was more stable than the average, but still varying."

This issue proposes two complementary directions to make winml perf numbers trustworthy and comparable.

---

Direction A — perf --stable: DVFS/thermal-aware robust measurement

Problem

A single contiguous loop captures every sample under one thermal/DVFS state. If the chip heats up mid-run, the whole batch drifts. There is also no signal telling the user whether a given measurement is trustworthy, and no robust point estimate beyond raw min/mean.

Why plain min (which the owner found "more stable") is not enough as the headline:

• Single-sample fragility — one lucky scheduling event defines it; high estimator variance.
• Monotonic in sample count — more iterations can only lower min, so runs with different --iterations aren't comparable.
• Sustained throttling raises the floor itself — once the whole run is throttled, even min drifts (exactly the owner's "min still varying").

Proposed design

Add an opt-in --stable mode that measures in rounds instead of one contiguous block, with optional inter-round cooldown, and reports a robust estimator + a measurement-quality verdict.

Measurement model

• warmup once, then rounds × iters_per_round with optional cooldown_ms between rounds.
• Segment samples per round; compute round_medians = [median(round_i)].

Robust estimator

robust_latency = min(round_medians)          # per-round median kills within-round jitter;
                                             # min across rounds picks the least-throttled thermal state
round_cov      = std(round_medians) / mean(round_medians)

This beats both plain min (no averaging) and mean (optimistic-bias-free but throttle-sensitive): each round's median removes batch jitter, and the inter-round min approaches the unthrottled compute floor.

Convergence early-stop

stop when round_cov ≤ stability_target (default 0.03)
     AND rounds_run ≥ min_rounds (default 5)
     AND target met for `patience` consecutive rounds (default 2)
else continue up to max_rounds (default 12)

Measurement-quality grade (direct trustworthiness signal)

Grade	Condition
STABLE	round_cov ≤ target
NOISY	target < round_cov ≤ 2×target
UNTRUSTWORTHY	round_cov > 2×target (suggest closing background load / adding cooldown)

Proposed CLI (fully backward-compatible; defaults preserve current behavior)

winml perf -m model.onnx --device npu --stable
  --rounds 12            # max rounds
  --min-rounds 5
  --iters-per-round 30
  --cooldown-ms 500
  --stability-target 0.03

Integration points (grounded in current code)

• session/stats.py::PerfStats — unchanged (still the flat sample recorder).
• New pure module session/stability.py — estimator + CoV + grading + convergence (unit-testable with injected samples, no hardware needed).
• New _run_rounds_loop() alongside _run_simple_loop() in commands/perf.py; dispatch from _run_benchmark() on config.stable.
• Extend BenchmarkConfig / BenchmarkResult + to_dict() with a stable block:

  "stable": {
    "robust_latency_ms": 33.4,
    "round_medians_ms": [...],
    "round_cov": 0.021,
    "rounds_run": 7,
    "converged": true,
    "quality": "STABLE",
    "iters_per_round": 30,
    "cooldown_ms": 500
  }

• Text output adds two lines under the existing table (Stable estimate: / Measurement:).

Scope

• Fixes: measurement noise from DVFS/thermal drift — turns "one run, one number" into a robust estimate with a confidence verdict.
• Does NOT: lower the model's real latency, or eliminate genuine sustained thermal throttling (it reports UNTRUSTWORTHY and recommends cooldown instead).

---

Direction B — Record run-context / confounder metrics for cross-run comparability

Problem

Two perf runs are only comparable if the machine was in a comparable state. Today --monitor (HWMonitor) captures during-run NPU/GPU utilization, device memory (local/shared/peak), CPU% (mean/peak), and RAM (used/peak) — but:

• it is opt-in and oriented at a live display, not recorded as comparability metadata by default;
• it does not capture temperature, clock/DVFS frequency state, or pre-run background load — precisely the confounders that explain run-to-run drift.

As torchbenchmark's "Using a low-noise machine" notes: interrupts, context switches, and clock-frequency scaling all conspire to make results variable — so you must understand the noise level of your setup before drawing conclusions. Capturing that context is the prerequisite for comparability.

Proposed design

Attach a run-context fingerprint to every perf result (not just --monitor), capturing the environment that affects latency so two runs can be judged comparable — or flagged as not.

Capture (best-effort, degrade gracefully when a metric is unavailable):

• Thermal: SoC/NPU temperature at start and end of the run (thermal headroom / observed throttle). (new — not currently captured)
• Clock/DVFS: CPU/NPU clock or performance-state where queryable. (new)
• Pre-run background load: idle-baseline CPU% and free RAM sampled before benchmarking starts (was the box quiet?). (new)
• During-run load (reuse existing HWMonitor): mean/peak CPU%, RAM, device mem, adapter utilization.
• Static environment: OS/driver/EP/SDK versions, device name, power/AC state.

Comparability gate — when a baseline result is provided (or via the auto-config A/B harness), flag pairs as not comparable when context diverges beyond thresholds, e.g. ΔstartTemp > X°C, pre-run background CPU > Y%, or a different power state. This prevents drawing conclusions from runs taken under different machine states.

Result JSON — extend with a run_context block:

"run_context": {
  "temp_start_c": 46.2, "temp_end_c": 58.9,
  "clock_state": "...", "power_state": "AC",
  "prerun_cpu_pct": 3.1, "prerun_free_ram_mb": 21440,
  "during": { "mean_cpu_pct": ..., "peak_ram_mb": ..., "adapter_util_pct": ... },
  "env": { "os": "...", "driver": "...", "ep": "...", "sdk": "..." },
  "comparable_to_baseline": true
}

Integration points

• Extend session/monitor/hw_monitor.py (+ _pdh.py) with temperature/clock sampling where the platform exposes it; add a lightweight pre-run idle sampler.
• Emit run_context from commands/perf.py::_collect_results for every run (cheap fields always; monitor-derived fields when --monitor or --stable).
• Optional --fail-on-incomparable for CI / auto-config sweeps.

Scope

• Fixes: makes cross-run comparisons defensible by recording (and gating on) the machine state; explains why two runs differ instead of leaving it a mystery.
• Does NOT: replace --stable — B records/【judges】 the environment, A makes the measurement itself robust. They compose: --stable produces a trustworthy number, run_context certifies two such numbers were taken under comparable conditions.

---

Why both, and how they relate to auto-config

The auto-config research POC (research/autoconfig/) already relies on an interleaved A/B protocol (KB finding npu-007: DVFS thermal noise requires session-level round averaging) to make its config sweeps reliable. That logic currently lives in ad-hoc research scripts. Landing A (--stable) and B (run_context) in winml perf lets auto-config call the CLI directly instead of re-implementing noise handling per sweep — promoting a validated research protocol into the product.

Prior art / non-goals

Direction A does not invent a new protocol — every mechanism has established precedent in mature CPU micro-benchmark tooling. The novelty is bringing these battle-tested noise-handling techniques into an ML-on-NPU inference CLI, where they are almost universally absent.

Where each mechanism comes from:

Mechanism (this proposal)	Prior art	What they do
Rounds / interleaved measurement (vs one contiguous block)	Google Benchmark --benchmark_enable_random_interleaving, JMH forks, pyperf multi-process	Google Benchmark shuffles execution across repetitions specifically to average out CPU freq-scaling / thermal / background noise
median robust estimator	PyTorch torch.utils.benchmark.Timer, Criterion.rs, nanobench	Timer uses median to resist outliers and reports IQR as a spread/stability signal
Min across rounds	torchbenchmark	Its docs state min-over-runs is more stable than average — matching the owner's own observation
Convergence early-stop	Timer blocked_autorange, Criterion, hyperfine	Adaptively grows sampling until a target precision / min run-time is reached
Stability grade / warning	pyperf, hyperfine, Criterion	pyperf prints WARNING: the benchmark result may be unstable when std% exceeds a threshold; hyperfine warns statistical outliers were detected, consider re-running… closing background programs

The only mildly novel piece is composing median + min into a two-level min-of-round-medians estimator tuned for single-sided thermal noise; each layer individually is standard practice.

The competitive gap (why this is worth doing): mature noise-handling lives in CPU micro-benchmark libraries, but the direct peers for ML inference benchmarking do not implement it:

Tool	DVFS/thermal-aware protocol
onnxruntime_perf_test	❌ warmup + repeat + percentiles only
NVIDIA trtexec	❌ relies on manually locking GPU clocks, not in-tool
MLPerf Inference	⚠️ enforced via run rules, not adaptive estimation
Olive	❌ delegates to ORT, no DVFS handling

So --stable closes a real gap: the owner's noise pain is exactly the space these ML CLIs leave unaddressed. Implementation should reuse these validated designs (interleaving à la Google Benchmark, std%/CoV warning à la pyperf, adaptive stop à la Timer, background-interference hint à la hyperfine → which also feeds Direction B) rather than invent from scratch.

Suggested rollout

1. session/stability.py pure functions + pytest (no hardware).
2. Wire --stable into perf.py (driver + CLI + output + result schema).
3. Add temperature/clock/pre-run sampling to HWMonitor; emit run_context from every run.
4. Comparability gate + --fail-on-incomparable; auto-config sweeps switch to --stable.

Out of scope (future)

• --cooldown-until-temp (block until NPU cools below a threshold) — needs cross-vendor thermal APIs.
• Automated machine tuning (à la torchbenchmark) — document a "low-noise setup" checklist first.