Add staging-push workflow to push stagings to ghcr - #7
Conversation
e228694 to
f2e92d2
Compare
9333098 to
bf3412e
Compare
staging-push is a reusable workflow that publishes a run's artifacts to an OCI registry. The caller gives a full package path and globs; every matched artifact becomes an untagged child of one OCI index, and the index is the only tagged manifest. A set publishes atomically -- the children are unreachable by name until the index lands -- and the registry enforces the index's references. Children are told apart by their file name and, when the artifact carried staging-meta.json, by com.opendaq.* descriptor annotations; `platform` is left empty so no docker-style client believes it can auto-select between compilers. The workflow authenticates with github.token and needs only packages: write -- package names are constructible, so no PAT is required. test-staging now publishes hello-core and hello-module -- stagings and installers, four packages -- to the test channel through staging-push, and manylinux builds a DEB so its staging has a matching installer.
bf3412e to
e483851
Compare
JakaMohorko
left a comment
JakaMohorko
left a comment
There was a problem hiding this comment.
Looks generally good. Hard to evaluate, however, without seeing it used in the openDAQ main or module repositories.
| shell: python3 {0} | ||
| env: | ||
| STAGINGS: ${{ inputs.staging-packages }} | ||
| TOKEN: ${{ github.token }} |
There was a problem hiding this comment.
Are we ever going to run this on fork PRs? I think the token is not available in that case.
There was a problem hiding this comment.
On a fork PR the token is read-only, so it can't push to our container registry namespace. A couple of ways to handle it:
- The registry namespace is owner-relative -- the path is
ghcr.io/<owner>/...where<owner>is${{github.repository_owner}}-- so it can be set up so each repo publishes to its own namespace (a fork's own runs would then publish to the fork's namespace, with the fork's own token). - Alternatively, disable push on fork PRs (gate the publish job, or a separate PR-triggered workflow without it) -- fork PRs then run
pull+build / testsandnever push.
For the pull side either way:
- Make the staging packages public, so fork PRs can pull them anonymously (public ghcr needs no token).
- Make the
oras logininstaging-pullnon-fatal (warn + continue) -- trusted contexts still log in for private pulls, and fork PRs fall back to anonymous pull of the public packages.
We'll have to validate this on a real fork: anonymous pull of our public packages from a fork PR, staging-pull continuing past an absent/failed login, and repository_owner behaving as expected in the fork context.
There was a problem hiding this comment.
Also worth noting: the reusable workflow -- the build engine -- only deals with GitHub artifacts, that's its isolated interface, it never touches the registry. staging-pull bridges the two: it pulls the packages from ghcr and re-uploads them as artifacts, which the build engine then consumes like any other.
So the fork handling stays confined to staging-pull; the build engine doesn't care where the artifacts came from, it just consumes them.
| packages: write | ||
| steps: | ||
| - name: Download stagings | ||
| uses: actions/download-artifact@v7 |
There was a problem hiding this comment.
Are we going to publish all artifacts downloaded?
There was a problem hiding this comment.
No, we don't publish everything downloaded, only the artifacts matching the artifacts globs of each staging-packages entry:
dirs = {d for g in staging["artifacts"] for d in glob.glob(f".artifacts/{g}") if os.path.isdir(d)}download-artifact grabs the whole run's artifacts into .artifacts/ for convenience, and the script then filters by those globs -- anything unmatched is ignored (and if nothing matches, the job fails). The selection lives in the input (several entries, each with its own globs), which doesn't map cleanly to download-artifact's single pattern, so it's simpler to download all and filter in the script.
Add org.opencontainers.image.created to the index annotations so a consumer can read each staging's age straight from the registry.
2 participants
Brief
Add
staging-push, a reusable workflow that takes a run's build artifacts and pushes them to an OCI registry (ghcr).Description
It's a pure publish primitive: it knows a package only as a path, and pushes whatever artifacts it was handed there.
A package is its path. A container registry can hold several variants of one package under a single name — the mechanism Docker uses to serve one image across platforms;
staging-pushuses it to serve a staging's per-platform builds. The path's tag points at an OCI index (manifest list) — the only tagged manifest — and each build is an untagged child of it, reachable only through the index. Schematically, the index looks like this:Children are told apart by their image title — the file name of the published tarball (
org.opencontainers.image.title) — which on its own is enough to pick a variant: read the index, choose an entry, and pull it by digest, a single request.Resolving by build settings is optional, and driven by a
staging-meta.jsonshipped alongside the artifact. It has two fields —settingsandcustom, key sets in the spirit of Conan — which the workflow turns intocom.opendaq.<key>andcom.opendaq.custom.<key>annotations on the child's descriptor. A consumer can then select by those (os, arch, compiler, …) instead of by the file name.The workflow pushes the children first — they land untagged — and then the index that lists them, so the index only ever references what already exists in the registry.
The caller drives this through the workflow input — for each package, a path and a list of artifact globs; every artifact whose name matches a glob is published under the path as a child.
test-stagingpublishes hello-core's staging and installer like this: