feat(ci): run the GARM end-to-end test on ProdStack (ISD-5876) - #317
Draft
cbartz wants to merge 11 commits into
Draft
feat(ci): run the GARM end-to-end test on ProdStack (ISD-5876)#317cbartz wants to merge 11 commits into
cbartz wants to merge 11 commits into
Conversation
Extracts what the end-to-end suite has to reuse from the integration suite rather than forking it: the configurator and image-builder deploys become parameterised helpers, GARM login is split out of first-run, and the GitHub App env is described by a GithubAppEnv record so a second app can be addressed without duplicating the accessors. Adds dispatch_workflow and wait_for_completion, ported from the equivalents in github-runner-operator, and widens the diagnostics redaction: it now takes sentinel values -- the credential strings themselves -- so juju debug-log and show-unit output is filtered on what a credential actually is rather than on a guessed key name. test_redaction.py guards that, since it is the only thing that stops a later diagnostics change from regressing it. github_app_private_key accepts a pasted PEM as well as a base64 blob, matching how the key is stored in the repository secret.
Adds an e2e-test backend pinned to the private-endpoint runner and a charms/tests/e2e/ suite, kept separate from the integration suite so opcli does not discover the end-to-end test as part of the pull-request matrix. Restricting charms_integration.yaml to 'integration-test-ci:*' is what keeps it off the merge gate: the filter defaults to empty, which would schedule every backend in spread.yaml, ProdStack included, on every pull request. concierge-e2e.yaml.tmpl adds a MetalLB addon whose range the workflow fills in with the runner's own address. traefik-k8s needs a LoadBalancer service microk8s cannot otherwise satisfy, and that address is what makes GARM's callback and metadata URLs reachable from a runner VM on the tenant.
…suite spread.yaml and tox.ini already carried these three onward, but nothing put them into the environment in the first place, so the fixtures saw them empty. The image name is required and fails fast when absent, since a blank one surfaces much later as an image lookup that finds nothing. The flavor and proxy are written only when set: the fixtures supply their own defaults, and an exported empty string is not the same as absent -- it would win over the default rather than yield to it.
…n path The image name, flavor and proxy were never exported before, so nothing has run with them present. Assert the image name reaches pytest as well, since a value that loads but does not arrive is the failure this is meant to catch. The trigger is temporary and comes out before review.
cbartz
force-pushed
the
feat/garm-e2e-implementation-ISD-5876
branch
from
August 21, 2026 13:06
eec4f9a to
4369aed
Compare
Deploys GARM behind a traefik ingress with the configurator holding real tenant credentials, waits for a runner to register, dispatches garm_e2e_test_run.yaml at the scale set's label and asserts the run concludes successfully -- which is only reachable if GARM authenticated to OpenStack, booted a VM from the published image, and the runner registered, claimed the job and exited clean. Waiting on runner_status rather than the provider's status: the VM reports running well before the agent inside it has registered, so following the provider's view dispatches into a label with nothing listening. Instances are matched by scale set rather than taken from the model at large. The dispatch wait outlasts the dispatched job's own ceiling, so a wedged runner surfaces as that job timing out rather than as this wait giving up first, and it covers queue time, which timeout-minutes does not span. Teardown waits for the drain instead of sleeping through it: GARM rejects the delete while the scale set still owns instances, and a rejected delete is a VM left running on the tenant.
Adds the build job the suite deploys from and swaps the direct tox call for opcli spread run, which is what supplies the charm_paths and resource-image fixtures and fetches the build's artifacts. The e2e job waits for the build rather than provisioning alongside it, as charm-ci's reusable workflow does: overlapping them only holds the private-endpoint runner idle until the artifacts exist, and that runner is scarce. concierge is pointed at a MetalLB pool of this host's own address. traefik-k8s asks for a LoadBalancer service microk8s cannot satisfy unaided, and that address is what lets a runner VM on the tenant reach GARM's callback and metadata URLs. Runners left on the tenant are deleted whatever the outcome, matched on the garm-controller-id GARM stamps on every server it creates so the sweep cannot reach anything else. A fixture cannot cover this: it does not run if the model or the runner dies mid-test, which is exactly when VMs are left behind. No juju debug-log or status dump on failure, unlike charm-ci's template: both replay charm output carrying the rendered GARM config, and so the tenant password. The suite's own diagnostics go through the sentinel redactor.
traefik-k8s has no 2 track: juju refused the deploy with "charm or bundle not found for channel 2/stable". Charmhub publishes 1.0 and latest only.
…tor exists GARM reports "Waiting for garm-configurator relation" until the configurator is related, and the fixture that deploys the configurator depends on this one, so waiting for active here could only ever time out. Wait for traefik to serve and for GARM's hook to settle instead: that is what says the ingress has been taken up, which is all this fixture needs before it checks the controller URLs.
The repository's labels are managed and run-e2e was deleted from the repo, so a label gate cannot be relied on here. Gate on the development branch instead, which is self-limiting and needs nothing set up by hand. Runs for a ref now cancel any earlier one still going: several full runs queueing against a single private-endpoint runner helps nobody. Still temporary, and still removed before review.
The charm's restart() returns before starting the workload while no configurator has supplied provider configs, so GARM's API is not listening when the ingress relation is made. Asking it for controller-info there could only ever retry against a refused connection. Relating the ingress and checking what it produced are now separate: the fixture relates and waits for traefik, and the scale set fixture makes the assertion once the configurator has brought the workload up -- the first moment GARM is serving, and still before any runner has been asked for.
…ress The assertion read traefik's unit address, which is its pod IP. GARM advertises the address MetalLB handed traefik, so the check compared two addresses that were never going to match even when everything was wired correctly. Read the address traefik reports serving on instead, and fail clearly if that cannot be parsed rather than silently comparing against nothing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Runs the full chain against ProdStack: charm starts → configurator delivers config →
GARM API reachable → provider authenticates to OpenStack → VM created from the published
image → runner registers with GitHub → job scheduled → picked up → exits clean.
Landed so far:
parameterised helpers, GARM login splits out of first-run, and a
GithubAppEnvrecordlets a second GitHub App be addressed without duplicating accessors. Adds
dispatch_workflow/wait_for_completion, and widens diagnostics redaction to filter onsentinel values — the credential strings themselves — rather than guessed key names,
guarded by
test_redaction.py.e2e-testspread backend on the private-endpoint runner, acharms/tests/e2e/suite, thegarm-e2etox environment, the MetalLB conciergetemplate, and the
spread-jobs-includefilter that keeps ProdStack off the merge gate.Still to come: the end-to-end test and fixtures, the driver's build job and real spread
run, and the orphan-VM cleanup.
Test plan
opcli spread jobsresolves both backends; the e2e job lands on the private-endpointrunner.
opcli spread jobs --include 'integration-test-ci:*'returns 8 jobs, none of them thee2e suite — the "not a merge gate" acceptance criterion.
Breaking changes
None yet.