Skip to content

fix(benchmarks): rebuild every branch-versioned module before an A/B run#465

Open
DemchaAV wants to merge 1 commit into
developfrom
fix/ab-bench-installs-all-branch-modules
Open

fix(benchmarks): rebuild every branch-versioned module before an A/B run#465
DemchaAV wants to merge 1 commit into
developfrom
fix/ab-bench-installs-all-branch-modules

Conversation

@DemchaAV

Copy link
Copy Markdown
Owner

Why

The A/B scripts installed only the engine module, but benchmarks/pom.xml resolves graph-compose-render-pdf and graph-compose-templates — plus the templates tests-classifier jar, which is deliberately never published — at ${graphcompose.version}, i.e. the branch's own version.

So both sides of an A/B took those two jars from ~/.m2. When the compared branches share a version, which is the normal case for two branches cut from develop, they took the same ones: any change in the PDF backend or the templates module measured a delta of exactly zero. A stale backend jar built against the other branch's core could also throw NoSuchMethodError mid-run instead of reporting a number.

What

scripts/ab-bench.sh and scripts/run-benchmarks.ps1 now install render-pdf and templates per branch alongside the engine, each guarded by the presence of its pom so a pre-split layout — where the root build produces them — still works. This mirrors the install sequence the CI performance job already uses (.github/workflows/ci.yml:346-361).

graph-compose-fonts is left alone deliberately: it tracks an independent version line, so it does not change with the branch under test.

Tests

Both scripts parse clean (bash -n, PowerShell AST parser).

The install logic itself was exercised end to end across the layout boundary while investigating engine performance: 8 build-and-measure cycles over v1.8.0, v1.9.1, v2.0.0, v2.1.0 and develop, in a detached worktree. v1.8.0 predates the module split and has no render-pdf/ or templates/ directory, so it takes the guarded-skip path; the 2.x refs take the install path. All eight produced benchmark output.

scripts/ab-bench.sh itself was not run end to end here — its own CI smoke (ab-bench-smoke.yml) covers that and fires on changes to the script.

The A/B scripts installed only the engine, but the benchmark module resolves
render-pdf and templates — plus the templates tests-classifier jar, which is
never published — at the branch's own version. Both sides therefore took those
two jars from ~/.m2, and whenever the compared branches share a version, which
is the normal case for two branches cut from develop, they took the *same*
ones: any change in the PDF backend or the templates module measured a delta of
exactly zero. A stale backend jar built against the other branch's core could
also throw NoSuchMethodError mid-run instead of reporting a number.

Both modules are now installed per branch, guarded by the presence of their
pom so a pre-split layout, where the root build produces them, still works.
Mirrors the install sequence the CI performance job already uses.
@DemchaAV

Copy link
Copy Markdown
Owner Author

The ab-bench.sh smoke (main vs develop) check fired on this change and passed, so the script now runs end to end with the added per-branch installs — the verification the description deferred to CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant