Skip to content

feat: normalize benchmark mode to AverageTime under CodSpeed#5

Merged
not-matthias merged 3 commits intomainfrom
cod-2446-normalize-benchmarkmode-to-sampletime
Apr 7, 2026
Merged

feat: normalize benchmark mode to AverageTime under CodSpeed#5
not-matthias merged 3 commits intomainfrom
cod-2446-normalize-benchmarkmode-to-sampletime

Conversation

@not-matthias
Copy link
Copy Markdown
Member

@not-matthias not-matthias commented Mar 31, 2026

Summary

  • Normalizes all benchmark modes to AverageTime when CodSpeed instrumentation is active
  • Handles three paths: CLI -bm flag overrides, Mode.All expansion, and annotation-level modes (e.g. @BenchmarkMode(Mode.Throughput))
  • Prints an informational message when normalization occurs so users know their mode was overridden

Resolves COD-2446

@linear
Copy link
Copy Markdown

linear bot commented Mar 31, 2026

COD-2446 Normalize `BenchmarkMode` to `SampleTime`

JMH benchmarks can use different BenchmarkMode values such as average time, sample time, and throughput. The current implementation derives statistics from the selected mode results, which means results can’t be merged reliably across modes.

For CodSpeed’s Java walltime support, the benchmark mode should be normalized instead of only warning and continuing. A first step is to override unsupported modes to SampleTime, since it provides the most precise basis for the statistics currently reported.

This should preserve plug-and-play behavior while avoiding incompatible result shapes.

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the JMH runner to ensure that when CodSpeed instrumentation is active, benchmarks always execute using Mode.SampleTime, regardless of whether the mode was set via CLI (-bm), Mode.All, or per-benchmark annotations, and emits an informational message when an override occurs.

Changes:

  • Normalize CLI-specified benchmark modes (-bm) to SampleTime under CodSpeed instrumentation.
  • Normalize Mode.All expansion and any non-SampleTime annotation modes to SampleTime under CodSpeed instrumentation.
  • Print informational messages when normalization occurs to make the override visible to users.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Mar 31, 2026

Merging this PR will improve performance by 38.73%

⚡ 2 improved benchmarks
✅ 123 untouched benchmarks
⏩ 9 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
insertionSort[100] 17.2 µs 12.4 µs +38.73%
timSort[1000] 168.4 µs 147.2 µs +14.45%

Comparing cod-2446-normalize-benchmarkmode-to-sampletime (8cb09f5) with main (4b7a767)

Open in CodSpeed

Footnotes

  1. 9 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@not-matthias not-matthias force-pushed the cod-2446-normalize-benchmarkmode-to-sampletime branch from 469a91f to 0d2d059 Compare April 1, 2026 10:45
@not-matthias not-matthias marked this pull request as ready for review April 1, 2026 11:00
@not-matthias not-matthias force-pushed the cod-2446-normalize-benchmarkmode-to-sampletime branch from 166853c to 17a1e5c Compare April 1, 2026 11:06
@not-matthias not-matthias changed the base branch from main to cod-2447-remove-fork0-limitation-from-jmh April 1, 2026 11:06
@not-matthias not-matthias changed the title feat: normalize benchmark mode to SampleTime under CodSpeed feat: normalize benchmark mode to AverageTime under CodSpeed Apr 1, 2026
Copy link
Copy Markdown

@GuillaumeLagrange GuillaumeLagrange left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, although we need to clear up the Average/Mean discussion that we had in the daily this morning before merging this

@not-matthias
Copy link
Copy Markdown
Member Author

@GuillaumeLagrange I added a new CodSpeed benchmark mode which now saves iters_per_round and times_per_round_ns rather than back-calculating it from the avg value.

Copy link
Copy Markdown

@GuillaumeLagrange GuillaumeLagrange left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

When CodSpeed instrumentation is active, all benchmark modes
(Throughput, SampleTime, SingleShotTime, Mode.All, and CLI -bm
overrides) are now normalized to AverageTime. This ensures consistent
result shapes that match how criterion (codspeed-rust) collects
walltime data: each round has a variable iteration count and a
measured total wall time.

Also simplifies CodSpeedResultCollector to only handle AverageTime,
removing the SampleTime histogram expansion and Throughput back-
calculation paths.

Resolves COD-2446
… iteration

Add Mode.CodSpeed, an internal benchmark mode that runs identically to
AverageTime but preserves the raw (ops, durationNs) pair per iteration
in a new CodSpeedResult class. This eliminates the lossy floating-point
roundtrip that the previous approach required (score = time/ops, then
back-calculate time = score * ops).

When CodSpeed instrumentation is detected, the Runner automatically
overrides all declared benchmark modes to CodSpeed and deduplicates
by benchmark name so multi-mode declarations run exactly once.

CodSpeedResultCollector is simplified to only handle Mode.CodSpeed,
reading raw values directly from CodSpeedResult with no back-calculation.
@not-matthias not-matthias force-pushed the cod-2446-normalize-benchmarkmode-to-sampletime branch from 199003b to 8cb09f5 Compare April 7, 2026 15:52
@not-matthias not-matthias changed the base branch from cod-2447-remove-fork0-limitation-from-jmh to main April 7, 2026 15:52
@not-matthias not-matthias merged commit 8cb09f5 into main Apr 7, 2026
16 checks passed
@not-matthias not-matthias deleted the cod-2446-normalize-benchmarkmode-to-sampletime branch April 7, 2026 17:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants