feat: normalize benchmark mode to AverageTime under CodSpeed#5
Conversation
COD-2446 Normalize `BenchmarkMode` to `SampleTime`
JMH benchmarks can use different For CodSpeed’s Java walltime support, the benchmark mode should be normalized instead of only warning and continuing. A first step is to override unsupported modes to This should preserve plug-and-play behavior while avoiding incompatible result shapes. |
There was a problem hiding this comment.
Pull request overview
This PR updates the JMH runner to ensure that when CodSpeed instrumentation is active, benchmarks always execute using Mode.SampleTime, regardless of whether the mode was set via CLI (-bm), Mode.All, or per-benchmark annotations, and emits an informational message when an override occurs.
Changes:
- Normalize CLI-specified benchmark modes (
-bm) toSampleTimeunder CodSpeed instrumentation. - Normalize
Mode.Allexpansion and any non-SampleTimeannotation modes toSampleTimeunder CodSpeed instrumentation. - Print informational messages when normalization occurs to make the override visible to users.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
jmh-fork/jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java
Outdated
Show resolved
Hide resolved
jmh-fork/jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java
Outdated
Show resolved
Hide resolved
jmh-fork/jmh-core/src/main/java/org/openjdk/jmh/runner/Runner.java
Outdated
Show resolved
Hide resolved
Merging this PR will improve performance by 38.73%
Performance Changes
Comparing Footnotes
|
469a91f to
0d2d059
Compare
166853c to
17a1e5c
Compare
GuillaumeLagrange
left a comment
There was a problem hiding this comment.
LGTM, although we need to clear up the Average/Mean discussion that we had in the daily this morning before merging this
jmh-fork/jmh-core/src/main/java/io/codspeed/result/CodSpeedResultCollector.java
Outdated
Show resolved
Hide resolved
|
@GuillaumeLagrange I added a new CodSpeed benchmark mode which now saves |
When CodSpeed instrumentation is active, all benchmark modes (Throughput, SampleTime, SingleShotTime, Mode.All, and CLI -bm overrides) are now normalized to AverageTime. This ensures consistent result shapes that match how criterion (codspeed-rust) collects walltime data: each round has a variable iteration count and a measured total wall time. Also simplifies CodSpeedResultCollector to only handle AverageTime, removing the SampleTime histogram expansion and Throughput back- calculation paths. Resolves COD-2446
… iteration Add Mode.CodSpeed, an internal benchmark mode that runs identically to AverageTime but preserves the raw (ops, durationNs) pair per iteration in a new CodSpeedResult class. This eliminates the lossy floating-point roundtrip that the previous approach required (score = time/ops, then back-calculate time = score * ops). When CodSpeed instrumentation is detected, the Runner automatically overrides all declared benchmark modes to CodSpeed and deduplicates by benchmark name so multi-mode declarations run exactly once. CodSpeedResultCollector is simplified to only handle Mode.CodSpeed, reading raw values directly from CodSpeedResult with no back-calculation.
199003b to
8cb09f5
Compare
Summary
AverageTimewhen CodSpeed instrumentation is active-bmflag overrides,Mode.Allexpansion, and annotation-level modes (e.g.@BenchmarkMode(Mode.Throughput))Resolves COD-2446