Skip to content

fix: sign and notarize macos binary (CON-787) - #321

Merged
justinwlin merged 3 commits into
mainfrom
justinlin/con-787-sign-notarize-macos
Aug 5, 2026
Merged

fix: sign and notarize macos binary (CON-787)#321
justinwlin merged 3 commits into
mainfrom
justinlin/con-787-sign-notarize-macos

Conversation

@justinwlin

@justinwlin justinwlin commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

macOS binaries ship with only the ad-hoc signature the Go linker emits — TeamIdentifier unset, Signature=adhoc, spctl verdict rejected. Gatekeeper refuses to run them; the brew cask path fails loudest, as "the application is damaged and can't be opened."

All four macOS install paths (brew formula, brew cask, install.sh, runpodctl update) fetch runpodctl-darwin-all.tar.gz, so signing that one artifact fixes all four.

Changes

  • .goreleaser.ymlquill sign-and-notarize post-hook on universal_binaries, after lipo merges the per-arch builds. output: true so quill's notary submission id and rejection reasons reach the job log; GoReleaser swallows hook stdout otherwise.
  • release.yml — install quill (pinned to v0.7.1), pass five QUILL_* secrets to GoReleaser, and bound the job at timeout-minutes: 45.

Snapshots ad-hoc sign and skip notary submission (--dry-run={{ .IsSnapshot }}), so local builds need no credentials. Job stays on ubuntu-latest — quill signs and notarizes Mach-O from Linux.

The timeout matters because quill waits for a conclusive notary verdict with no bound of its own: a stalled Apple submission would otherwise hold the release-${ref} concurrency lock for the default 6h and block any retry.

Verification

Snapshot build passes; hooks inherit job env; the real universal binary signs valid against the production cert

goreleaser release --snapshot --clean → exit 0, hook fires on the merged binary and its output is now visible:

• universal binaries
  • creating from 2 binaries    id=default binary=dist/default_darwin_all/runpodctl
  • running hook                hook=quill sign-and-notarize "dist/default_darwin_all/runpodctl" --dry-run=true --ad-hoc=true
    │ [0000]  WARN only ad-hoc signing ...
    │ [0000]  WARN DRY RUN: skipping notarization...

GoReleaser hooks inherit the job environment — verified with a sentinel variable in an isolated project, since that is how the five secrets reach quill:

• running hook   hook=sh -c 'echo "HOOK SEES SENTINEL=[$SENTINEL_VAR]"'
  │ HOOK SEES SENTINEL=[leaked-into-hook]

Same universal artifact signed with the real Developer ID .p12:

Authority=Developer ID Application: Runpod, Inc. (YM4PQ356FQ)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
TeamIdentifier=YM4PQ356FQ

codesign --verify --deep --strict:
  valid on disk
  satisfies its Designated Requirement

Not yet exercised: the live Apple notary round-trip, which only runs on a real tag. Watch the first tagged release.

Notes

  • Signing is not retroactive — existing v2.3.0 artifacts stay unsigned. A new tag is required after merge.
  • Certificate is Developer ID Application, G2 Sub-CA, valid to 2031-07-31. Previous Sub-CA certs would have expired 2027-02-01.
  • Fails closed: notarization failure aborts during the build stage, before the GitHub release or tap PR exist. No half-published release.
  • upx stays scoped to linux_amd64 — it rewrites the Mach-O and would destroy the signature.
  • Bare binaries in .tar.gz can't be stapled (xcrun stapler is .app/.dmg/.pkg only), so Gatekeeper does an online ticket lookup. Expected.
  • The legacy per-arch darwin binaries remain unsigned. Only pre-change clients self-update via those; current update.go and install.sh both use darwin-all. Unchanged by this PR.

Does not fix: Homebrew tap trust (HOMEBREW_REQUIRE_TAP_TRUST) or self-update checksum verification (API-318). Both orthogonal.

CON-787

@justinwlin
justinwlin marked this pull request as ready for review July 30, 2026 23:48
the darwin universal binary ships with only the ad-hoc signature the go
linker emits, so gatekeeper rejects it with "no usable signature". all
four macos install paths (brew formula, brew cask, install.sh, runpodctl
update) fetch runpodctl-darwin-all.tar.gz, so signing that one artifact
fixes all of them.

sign after lipo merges the per-arch builds — signing before the merge
produces an invalid signature. snapshots ad-hoc sign and skip notary
submission, so local builds stay credential-free.

quill signs and notarizes mach-o from linux, so the job stays on
ubuntu-latest.
three operability fixes on the signing pipeline:

- output: true on the quill hook. goreleaser swallows hook stdout, which
  is where the notary submission id and any rejection reason go —
  notarization is the most failure-prone step here and was silent.
- pin quill to v0.7.1. the installer piped into sudo sh took whatever was
  latest at release time, so an upstream regression could break signing
  with no change on our side, and releases were not reproducible.
- timeout-minutes: 45 on the release job. quill waits for a conclusive
  notary verdict with no bound of its own; a stalled apple submission
  would hold the release-${ref} concurrency lock for the default 6h and
  block any retry.
@justinwlin
justinwlin force-pushed the justinlin/con-787-sign-notarize-macos branch from f0960bb to 4b44233 Compare August 4, 2026 17:32

@lukepiette lukepiette left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Verified the pipeline end to end rather than just the diff: the quill hook runs post-lipo and pre-archive, so the release checksums, brew formula, and cask sha256 are all computed from the signed artifact (no checksum-mismatch bug); a signing or notary failure fails the whole run before anything publishes; secrets are env-scoped to the goreleaser step, never on argv, and unreachable from fork PRs (no pull_request_target in the repo, and all five secrets already exist in repo settings); UPX stays scoped to linux so it cannot rewrite the signed Mach-O; not stapling is correct since bare Mach-O binaries cannot be stapled and quill zips for submission internally.

Comments, in priority order (none blocking):

  1. Harden the quill install. curl https://get.anchore.io/quill | sudo sh pipes an unverified installer to root in the same job that later receives the Developer ID .p12 and notary key in env. The quill version is pinned but the installer is not; a compromised installer is a direct exfiltration path for the signing identity. Fetch the quill release tarball from GitHub and verify a sha256 pinned in the workflow instead.
  2. The timeout comment is inverted. quill v0.7.1 bounds its own notary wait: DefaultStatus() is Wait: true, PollSeconds: 10, TimeoutSeconds: 900, and PollStatus wraps the poll loop in context.WithTimeout — a stalled submission fails at ~15 min on its own, so the 6h-lock scenario the comment fears cannot happen. The actual risk is the opposite: first-time Apple submissions routinely exceed 15 minutes, which would fail a release Apple would have accepted. Keep timeout-minutes: 45 as belt and braces, fix the comment, and consider a committed .quill.yaml with status: { timeout-seconds: 2400 } (or document that re-running is cheap — resubmission of an already-scanned hash is fast). Watch the first tag; it is the live notary test.
  3. Minor: the legacy per-arch darwin binaries still ship unsigned (disclosed, only pre-change self-updaters fetch them); goreleaser-action version: latest is now part of the signing chain and worth pinning in the same spirit as the quill pin; local goreleaser snapshot now requires quill on the developer's machine (ad-hoc mode, no credentials) — worth a README line.

@justinwlin
justinwlin force-pushed the justinlin/con-787-sign-notarize-macos branch from e4c3940 to 1c2a1fc Compare August 5, 2026 17:17
review follow-ups:

- install quill from the pinned release tarball with a pinned sha256, into a
  scratch dir, instead of piping get.anchore.io to root. this job holds the
  developer id p12 and the notary key, so an unverified installer running as
  root is an exfiltration path for the signing identity. the scratch dir keeps
  the download out of the workspace, where a leftover file would fail
  goreleaser's dirty-tree check with an unrelated error.
- the timeout comment was backwards. quill does bound its own notary wait
  (notary.PollStatus wraps the loop in context.WithTimeout, 900s default), so a
  stall cannot hold the concurrency lock. the comment now says so, and also
  records why that 900s must not be raised: quill mints one app store connect
  jwt with exp = timeout + 2m and never refreshes it, and apple rejects notary
  tokens with a lifetime over 20 minutes, so a longer wait guarantees a 401.
- pin goreleaser to v2.17.1 now that it drives the signing chain.
- keep the install vars out of the QUILL_* prefix quill itself reads.
- README: what is and is not signed, that local snapshots need quill on PATH
  (ad-hoc, no credentials), that a notary timeout means a full re-scan on
  re-run rather than a resume, and that the binary is not stapled.
@justinwlin
justinwlin force-pushed the justinlin/con-787-sign-notarize-macos branch from 1c2a1fc to 3a3c60b Compare August 5, 2026 17:30
@justinwlin
justinwlin merged commit 05ebe21 into main Aug 5, 2026
1 check passed
@justinwlin
justinwlin deleted the justinlin/con-787-sign-notarize-macos branch August 5, 2026 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants