Skip to content

fix(release): repair four gates that reported success without checking - #338

Open
loispostula wants to merge 1 commit into
jrr/release-commands-opsfrom
lp/release-gate-fixes
Open

fix(release): repair four gates that reported success without checking#338
loispostula wants to merge 1 commit into
jrr/release-commands-opsfrom
lp/release-gate-fixes

Conversation

@loispostula

Copy link
Copy Markdown
Member

Stacks on #337. Review of the release command family found four gates that
report success without checking the thing they exist to check, plus the
supporting fixes. No published object shape changes.

The four

1. bundle-linux has never succeeded. cargo-deb's default output is
target_dir_base.join("debian") with no target triple (config.rs
default_deb_output_dir; the triple is appended only by
target_dependent_path, which the deb output does not go through). So
--target does not move the artifact, and reading a tripled path is an
unconditional ENOENT. bundle-linux exits nonzero, sign-linux and publish
skip. This means the end-to-end Linux run described in #336 cannot have
happened — worth re-checking what else that run was thought to cover.

Fixed by passing -o explicitly. The per-triple directory is kept (cargo-deb's
default is shared by every package in the workspace), cleared first, and
exactly one .deb must emerge rather than taking the first by sort order.

2. The production check-run gate is satisfied by its own job.
conclusion is null until status reaches completed, and only four explicit
failure conclusions counted as bad, so a queued or running check read as green.
This command runs inside facts, whose own check run is attached to the tagged
commit, so the count was never zero and the no-checks refusal was unreachable.
Net effect: tag any ancestor of main whose CI never ran and it publishes.

3. manifest_base redirection. resolve built the manifest URL from a
field inside channels.json, which is written by the promotion credential —
whose documented invariant is that it cannot write artifacts. Point it
anywhere and every digest check downstream is satisfied against the attacker's
own manifest; a "kind": "authenticode" entry skips client signature
verification entirely. Resolution now uses the client's own production base;
promote re-asserts the field on every write instead of only at creation.

4. probe-store cannot fail on the property it gates. Its only If-Match
coverage asserted that a successful CAS update succeeded — which a store that
ignores the precondition also does. A stale-ETag write is now attempted and
must be refused.

⚠️ This may legitimately fail against the deployed MinIO. That is the
point: if If-Match is not enforced, publication must not be enabled. Nobody has
run this against the real endpoint yet.

⚠️ Breaking change

--required-checks is now mandatory unless --skip-check-runs is passed.
fslabs/fsl_libs#4230's companion branch supplies it via
vars.RELEASE_REQUIRED_CHECKS, defaulting to test.

Failure is deliberately considered only for the named checks. Measured on
fsl_libs main: 20 check runs on one commit, including a failed docker_build
and several cancelled by concurrency. Refusing on any bad run anywhere would
block every release. That same commit carries two runs named test — one
failed, one successful — because the postsubmit and the nightly both attach to
main's SHA, so the best outcome per name wins.

Also

  • publish reads back the detached signatures, which are immutable once
    written but carry no manifest digest and so were never verified. Artifacts
    stay compared against the digest the manifest records, not one recomputed
    from the bytes just sent, which would be self-consistent by construction.
  • key_from_url takes the last bucket segment: a base URL containing the
    bucket name yielded a doubled prefix, surfacing only after every artifact was
    already immutable.
  • assert_monotonic parses the candidate version before the comparison loop.
    An empty published set skipped validation entirely, so a bad first version
    committed once and then bricked every future production publish permanently.
  • healthcheck distinguishes a definite 404 from a transport failure via a
    typed status error rather than matching on message text, and treats only
    404 as absent. A 403 against an anonymously readable bucket is a broken read
    policy; calling it healthy let the workflow close its own tracking issue
    during an outage.
  • license_macos is omitted when absent rather than serialized as null,
    which jq -r renders as the literal string "null".
  • sign-linux feeds the passphrase over --passphrase-fd 0 instead of the
    argv (readable in /proc), never echoes gpg's arguments into an error, and
    writes stdin concurrently with draining stdout so a full pipe buffer cannot
    deadlock the signing job.

Testing

cargo test: 339 passed (324 baseline + 15). cargo fmt --check and
cargo clippy --all-targets clean.

The check-run gate is extracted to a pure function so the still-running,
satisfied-by-itself and unrelated-failure cases are covered directly, including
a regression test built from the measured shape of fsl_libs main.

Not exercised: the bundle-linux fix needs a real cargo-deb run, and
probe-store needs the deployed store.

Review of the release command family surfaced four defects that each let a
failure through silently, plus the supporting should-fixes. All sit in the
#336/#337 stack and none change the published object shapes.

bundle-linux could never have succeeded. cargo-deb's default output is
`target_dir_base.join("debian")` with no target triple (config.rs
`default_deb_output_dir`; the triple is appended only by
`target_dependent_path`, which the deb output does not go through), so
`--target` does not move the artifact and reading a tripled path was an
unconditional ENOENT. `-o` is now passed explicitly at the per-triple
directory, which is cleared first and from which exactly one .deb must
emerge: the previous take-the-first-by-sort-order would ship a stale build.

The production check-run gate was satisfied by the job it runs inside.
`conclusion` is null until `status` reaches `completed`, so a queued or
running check read as green, and this command's own check run is attached to
the tagged commit, making the count-based no-checks refusal unreachable. The
gate now takes named `--required-checks` and demands completed-and-success
for each.

BREAKING: `--required-checks` is mandatory unless `--skip-check-runs` is
passed. fsl_libs supplies it via `vars.RELEASE_REQUIRED_CHECKS`, defaulting
to `test`.

Failure is considered only for the required names. A commit on main routinely
carries unrelated bad check runs: measured on fsl_libs main, 20 check runs
including a failed docker_build and several cancelled by concurrency.
Refusing on any bad run anywhere would block every release. The same commit
carries two runs named `test`, one failed and one successful, because the
postsubmit and the nightly both attach to main's SHA, so the best outcome per
name wins.

resolve built the manifest URL from `channels.manifest_base`. channels.json is
written by the promotion credential, whose stated invariant is that it cannot
write artifacts; sourcing the manifest location from inside it hands over
exactly that power, and a `kind: authenticode` entry then skips client
signature verification on every platform. Resolution now uses the client's own
production base, promote re-asserts the field on every write rather than only
at creation, and the docs state it is informational.

probe-store could not fail on the property it gates. Its only If-Match
coverage asserted that a successful CAS update succeeded, which a store
ignoring the precondition also does. A stale-ETag write is now attempted and
must be refused. Note this may legitimately FAIL against the deployed MinIO,
which is the point: publication must not be enabled until it passes.

Also: publish verifies detached signatures, which are immutable once written
but carry no manifest digest and so were never read back; artifacts stay
compared against the digest the MANIFEST records rather than one recomputed
from the bytes just sent, which would be self-consistent by construction.
key_from_url takes the last bucket segment, since a base URL containing the
bucket name yielded a doubled prefix that surfaced only after the artifacts
were immutable. assert_monotonic parses the candidate version before the
comparison loop, which an empty published set skipped entirely, permanently
bricking future publishes. healthcheck distinguishes a definite 404 from a
transport failure via a typed status error rather than matching on message
text, and treats only 404 as absent: 403 against an anonymously readable
bucket is a broken read policy, and calling it healthy let the workflow close
its own tracking issue during an outage. license_macos is omitted when absent
rather than serialized as null, which jq -r renders as the string "null".
sign-linux feeds the passphrase over --passphrase-fd 0 instead of the argv,
never echoes gpg's arguments into an error, and writes stdin concurrently with
draining stdout so a full pipe buffer cannot deadlock the signing job.

15 new tests, 339 passing. The check-run gate is extracted to a pure function
so the still-running, satisfied-by-itself and unrelated-failure cases are
covered directly.

Signed-off-by: Loïs Postula <admin@conceptions.postu.la>
@fslabs-bot

fslabs-bot Bot commented Aug 19, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from loispostula. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@fslabs-bot fslabs-bot Bot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant