ci(benchmark): Add help task configuration benchmark on PRs#1244
Draft
runningcode wants to merge 2 commits into
Draft
ci(benchmark): Add help task configuration benchmark on PRs#1244runningcode wants to merge 2 commits into
runningcode wants to merge 2 commits into
Conversation
Add a pull_request workflow that benchmarks this project's `help` task with the configuration cache disabled (2 warm-ups, 5 builds), comparing the PR base commit against the head commit in a single gradle-profiler run via the git-checkout mutator. This is separate from the existing duckduckgo benchmark build: it profiles configuration time of this project rather than the runtime cost of applying the plugin to a sample app, and it runs automatically on every PR. Results are uploaded as an artifact and summarized as a sticky PR comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributor
|
| Scenario | Mean build time |
|---|---|
Base (help base) |
1778 ms |
PR (help PR) |
1777 ms |
| Difference | ✅ -1 ms (-0.1%) |
gradle-profiler does not write summary rows (mean, min, max) to benchmark.csv; those only appear in the HTML report. The comment parser looked for a non-existent `mean` row and always fell back to "Could not parse benchmark results". Average the `measured build #N` rows instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
pull_requestworkflow that benchmarks this project'shelptask with the configuration cache disabled, comparing the PR base commit against the head commit.What it does
helptask with--no-configuration-cache, 2 warm-ups + 5 builds.git-checkoutmutator to benchmark the PR base and head commits in a single run (help_basevshelp_prscenarios).How it differs from the existing benchmark
The existing
Benchmark build speedworkflow profiles the runtime cost of applying the plugin to the duckduckgo sample app and isworkflow_dispatchonly. This one profiles the configuration time of this project itself and runs automatically on every PR.Notes
base.sha/head.sha) withfetch-depth: 0so both are reachable from the PR merge commit.helpbuild time (dominated by configuration with the config cache off); the HTML artifact has the full breakdown.Files
.github/workflows/benchmark-help-config.yml— the workflowscripts/benchmark/help-config-cache.scenarios— gradle-profiler scenariosscripts/benchmark/help-config-comment.py— rendersbenchmark.csvinto the comment table🤖 Generated with Claude Code