Skip to content

fix: Exclude setup time from benchmark in walltime mode#114

Merged
adriencaccia merged 2 commits intoCodSpeedHQ:masterfrom
toby-coleman:fix/pedantic-walltime-mode
Apr 8, 2026
Merged

fix: Exclude setup time from benchmark in walltime mode#114
adriencaccia merged 2 commits intoCodSpeedHQ:masterfrom
toby-coleman:fix/pedantic-walltime-mode

Conversation

@toby-coleman
Copy link
Copy Markdown
Contributor

@toby-coleman toby-coleman commented Apr 6, 2026

Suggested fix for #113:

  • It appears that the issue is limited to walltime mode.
  • Small fix to the ordering, plus a test.

Generated with AI Agent (GPT 5.3 Codex in Copilot)

Fix #113

Copy link
Copy Markdown
Member

@adriencaccia adriencaccia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, it was indeed not intended.

Thanks for the contribution @toby-coleman!

@codspeed-hq
Copy link
Copy Markdown

codspeed-hq bot commented Apr 8, 2026

Merging this PR will degrade performance by 27.07%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 22 improved benchmarks
❌ 16 regressed benchmarks
✅ 239 untouched benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
WallTime test_sum_of_squares[sum_of_squares_sum_comprehension_product] 193.9 µs 199.1 µs -2.59%
WallTime test_multiprocessing_map[10000] 85.2 ms 116.9 ms -27.07%
WallTime test_make_lowpass 5.3 µs 6.1 µs -11.79%
WallTime test_sum_squares_slow 361.7 µs 322.3 µs +12.21%
WallTime test_tcp_connection[google.com-443] 1.1 ms 1.2 ms -4.69%
WallTime test_make_peak 6.4 µs 6.6 µs -2.27%
WallTime test_iir_filter_process 3 µs 3.1 µs -4.11%
WallTime test_hamilton_cycle[graph0] 25 µs 26.7 µs -6.17%
WallTime test_generate_all_combinations[5-4] 10.3 µs 11.5 µs -10.53%
WallTime test_fs_read[1000] 827.6 ns 803.3 ns +3.03%
WallTime test_threadpool_map[10] 2.6 ms 2.5 ms +2.02%
WallTime test_process_creation[cat /dev/null] 5.4 ms 5 ms +9.15%
WallTime test_noop_pass 468.2 ns 481.6 ns -2.79%
WallTime test_multiprocessing_map[100] 58.3 ms 56.4 ms +3.33%
WallTime test_tcp_connection[8.8.8.8-53] 659.2 µs 680.6 µs -3.15%
WallTime test_make_highshelf 7.7 µs 7.5 µs +2.33%
WallTime test_sum_of_squares[sum_of_squares_sum_labmda_product] 274.8 µs 295.8 µs -7.09%
WallTime test_process_creation[echo hello] 2.5 ms 2.1 ms +14.89%
WallTime test_array_alloc[100] 1.2 µs 1.2 µs -5.03%
WallTime test_tcp_connection[1.1.1.1-53] 958.8 µs 774.9 µs +23.73%
... ... ... ... ... ...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.


Comparing toby-coleman:fix/pedantic-walltime-mode (4a1f0f8) with master (aa267f3)

Open in CodSpeed

@adriencaccia adriencaccia force-pushed the fix/pedantic-walltime-mode branch from 8637946 to b61640c Compare April 8, 2026 13:49
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@adriencaccia adriencaccia merged commit 48a4822 into CodSpeedHQ:master Apr 8, 2026
39 of 40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unexpected behaviour from benchmark.pedantic

2 participants