feat: swap to instrumented __coverage image in dedicated nightly coverage runs#117
Open
gustavolira wants to merge 4 commits into
Open
feat: swap to instrumented __coverage image in dedicated nightly coverage runs#117gustavolira wants to merge 4 commits into
gustavolira wants to merge 4 commits into
Conversation
…rage runs
A nightly run can only collect browser coverage if RHDH deploys the
instrumented __coverage plugin image. Today the image swap only happens
in PR mode (when a PR OCI URL is present); nightly runs deploy the plain
released image, so window.__coverage__ is never created.
Add the swap to the nightly OCI-direct resolution branch (plugins not in
default.packages.yaml): for a frontend plugin, replace the image tag with
its __coverage variant.
Gating is deliberately conservative so the FUNCTIONAL nightly cannot
break:
- E2E_NIGHTLY_COVERAGE (explicit opt-in), NOT the ambient
E2E_COLLECT_COVERAGE. run-e2e.sh defaults E2E_COLLECT_COVERAGE=true, so
gating on it would make every functional nightly deploy __coverage
images — and since those are built non-fatally by the overlay release
publish, a missing one would break the deployment. Requiring the
explicit opt-in keeps the functional nightly's resolution byte-identical
to today (the env check short-circuits before any fs access).
- coverage-anchors/ presence (the same rolled-out signal the overlay
release publish uses to build the __coverage variant), so a plugin
without one is never pointed at a tag that does not exist.
- frontend-plugin role only.
The {{inherit}} branch (DPDY plugins) is untouched — those resolve to the
RHDH catalog image, which we cannot instrument.
Adds tests for: opted-in swap, no swap in the functional nightly even
with E2E_COLLECT_COVERAGE on, no swap when not rolled out, no swap for
non-frontend roles.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Review follow-ups on the nightly coverage swap: - Extract toCoverageImageRef() so the `:(tag)__coverage` rewrite (and the greedy-up-to-! regex) lives in one documented place instead of being duplicated across PR mode and nightly resolution. - Hoist the workspace-root resolution to a single workspaceRoot const (was computed twice). - Document the new behavior per the repo convention: a changelog entry, an E2E_NIGHTLY_COVERAGE row in the environment-variables reference, and a coverage-swap section in plugin-metadata-resolution. No behavior change; full suite still passes (yarn check + tests green). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a test proving a DPDY plugin resolves to {{inherit}} and is never
swapped to a __coverage tag even with E2E_NIGHTLY_COVERAGE on — the core
safety guarantee, since {{inherit}} points at RHDH's catalog image, which
we can't instrument. Also fix a sentence fragment in the toCoverageImageRef
comment.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The PR adds a new feature (E2E_NIGHTLY_COVERAGE swap), so bump the minor version per the repo's version-check gate. 2.1.0 is published, so move its changelog section to released and open a new [2.2.0] - Current section for this change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A nightly run can only collect browser coverage if RHDH deploys the instrumented
__coverageplugin image. Today the image swap only happens in PR mode (plugin-metadata.ts, when a PR OCI URL is present). Nightly runs deploy the plain released image, sowindow.__coverage__is never created and no coverage is collected.This is the consumer side of the overlay's coverage producer (rhdh-plugin-export-overlays#2645), which builds the
bs_X__Y__coveragevariant of released images.What
In the nightly OCI-direct resolution branch (plugins not in
default.packages.yaml), for a frontend plugin, swap the image tag to its__coveragevariant — the same:(tag)→:(tag)__coveragerewrite already used in PR mode.Safety — the functional nightly cannot break
The gating is deliberately conservative:
E2E_NIGHTLY_COVERAGE(explicit opt-in), NOT the ambientE2E_COLLECT_COVERAGE.run-e2e.shdefaultsE2E_COLLECT_COVERAGE=true, so gating on it would make every functional nightly deploy__coverageimages — and since those are built non-fatally by the overlay release publish, a missing one would break the deployment. Requiring the explicit opt-in keeps the functional nightly's resolution byte-identical to today (the env check short-circuits before anyfsaccess).coverage-anchors/presence — the same rolled-out signal the overlay release publish uses to build the__coveragevariant — so a plugin without one is never pointed at a tag that doesn't exist.frontend-pluginrole only.The
{{inherit}}branch (DPDY plugins) is untouched — those resolve to the RHDH catalog image, which we can't instrument.So: the functional nightly (no
E2E_NIGHTLY_COVERAGE) behaves exactly as before. Only a dedicated coverage run that sets the flag (and ensures the images exist) gets the swap.Tests
plugin-metadata.nightly.test.tsadds:__coveragetag, preserving the!pathE2E_COLLECT_COVERAGEoncoverage-anchors/)yarn check(typecheck + lint + prettier) and the full suite pass (86/86).🤖 Generated with Claude Code