fix: scope apt regen to dispatched channel (stop cross-channel index clobber) — #3218#5
Merged
Merged
Conversation
…x clobber The publish workflow syncs only pool/<dispatched-channel>/ from R2, but regenerate-apt-metadata.sh hardcoded DISTRIBUTIONS="stable bleeding-edge" and regenerated BOTH channels every run. The non-dispatched channel had an empty local pool, so apt-ftparchive emitted an empty Packages, and the upload step's `find dists` pushed that empty index to R2 — clobbering the other channel's good index. In practice every bleeding-edge snapshot publish wiped the stable index minutes after each stable release (the .deb stays in the R2 pool; only the dists Packages index dies). This is the root cause of #3218 (a recurrence of #2838). The per-channel regen that #2838 was meant to introduce was absent from the live script. Fix: regen reads CHANNELS (default both, for manual full rebuilds when both pools are present locally); the workflow passes CHANNELS=<dispatched channel> so a publish touches only its own channel's dists. The other channel's R2 index is left untouched. Refs: wheels-dev/wheels#3218, wheels-dev/wheels#2838 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
bpamiri
added a commit
that referenced
this pull request
Jun 19, 2026
…218 follow-up) (#6) The cross-channel clobber (apt-wheels#5) was only half the #3218 story. The other half is CDN caching: Cloudflare auto-caches by file extension, so a published Packages.gz got edge-cached and kept being served after the next publish rewrote it. Its hash no longer matched the freshly-regenerated Release, so `apt-get update` rejected it with "File has unexpected size (29 != 1404)" — exactly the symptom users keep reporting. The uncompressed Packages/Release are extension-less (served DYNAMIC, never cached), which is why only the .gz broke and why a cache-buster query masked it (different cache key -> origin). Fix: - Upload apt metadata (Release, Release.gpg, InRelease, Packages, Packages.gz) with `Cache-Control: no-store` so the edge never caches a stale index again. This alone prevents recurrence regardless of purge permissions. - Upload .deb pool files with `public, max-age=31536000, immutable` — they're version-stamped and never change, so long edge caching is correct and fast. - Add a best-effort Cloudflare purge step that evicts the just-published URLs, to unstick copies cached BEFORE no-store existed (the current legacy entry). Resolves the wheels.dev zone at runtime; warns and continues (does not fail the publish) if the token lacks Zone.Cache-Purge scope. Refs: wheels-dev/wheels#3218, #5 Signed-off-by: Peter Amiri <peter@alurium.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
bpamiri
added a commit
to wheels-dev/wheels
that referenced
this pull request
Jun 19, 2026
…gression guard) (#3234) * ci: add index-integrity probe to distribution install smoke (#3218 regression guard) The daily install-smoke installs the CLI per channel and asserts the version, but that only samples once at 14:00 UTC. The #3218 cross-channel clobber empties the stable apt index for most of the day (a bleeding-edge snapshot wipes it minutes after each stable publish; it's only briefly populated), so a clobber can land outside the install legs' sample window and slip past them — and when they do fail, "apt install failed" doesn't name the cause. Add a fast, container-free `index-integrity` job that probes the PUBLISHED apt and yum dist indexes directly: stable must be non-empty and name the current GA (per arch for apt), and bleeding-edge must stay non-empty (the clobber was bidirectional — apt-wheels#5 scopes regen per-channel to prevent both directions). An empty/missing/stale index fails with a message that names the regression. Reliability: fetches use `curl --retry ... --retry-all-errors` so a transient blip can't false-red the guardian, and all grep checks feed from a here-string (`grep -q PAT <<<"$body"`) rather than `printf | grep -q` — under `set -o pipefail` the latter false-fails when grep -q early-exits and SIGPIPEs the upstream printf of a 100KB index. Verified: 3/3 green against the live indexes, and a negative test (bogus version) correctly fails. Refs: #3218, wheels-dev/apt-wheels#5 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <petera@pai.com> * ci: probe plain (edge) URLs + hash-consistency, mirroring apt/dnf exactly Two corrections to the index-integrity job after finding the real #3218 failure mode is CDN edge-cache staleness, not just the origin clobber: 1. Drop the `?cb=` cache-buster. apt/dnf fetch the plain URLs that hit Cloudflare's edge; a cache-buster hit R2 origin instead, so the probe went green while real `apt install` failed on a stale edge-cached Packages.gz. Test the SAME plain URLs clients use. 2. Verify hash-consistency, exactly what the package managers verify: parse the SHA256 the (plain) Release records for each binary-<arch>/Packages.gz, fetch the (plain) Packages.gz, and compare — this is precisely the "File has unexpected size" check that failed in #3218. yum: verify the served primary.xml.gz matches the hash repomd references. Then gunzip and confirm the content lists the GA (catches an empty-but-internally-consistent index). Binary correctness: gzip blobs are fetched to a temp file and hashed/gunzipped from the file — never captured into a shell variable, since `$(...)` is text-only and silently corrupts binary (strips trailing newlines, drops NULs), which would compute a bogus hash. Paired with apt-wheels#6 / yum-wheels#6 (set `no-store` on metadata so the edge stops caching it). Verified against the live indexes: correctly RED on the currently-stale apt stable amd64 edge, GREEN on arm64 / bleeding-edge / yum. Refs: #3218, wheels-dev/apt-wheels#6, wheels-dev/yum-wheels#6 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <petera@pai.com> --------- Signed-off-by: Peter Amiri <petera@pai.com> Co-authored-by: Peter Amiri <petera@pai.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
apt install wheelson the stable channel fails with "Unable to locate package wheels" — the stablePackagesindex is empty (0 bytes), reported in wheels-dev/wheels#3218 (a recurrence of #2838). The.debis present in the R2 pool (pool/stable/w/wheels/wheels_4.0.5_all.deb→ HTTP 200); only thedists/stable/.../Packagesindex is empty.Root cause (cross-channel clobber)
Confirmed from the live run timeline:
Publish 4.0.5 (stable)Publish 4.0.6-snapshot.2205 (bleeding-edge)Publish 4.0.6-snapshot.2208 (bleeding-edge)Three-step interaction:
pool/${CHANNEL}/from R2 (channel-filtered).regenerate-apt-metadata.shhardcodedDISTRIBUTIONS="stable bleeding-edge"and regenerated both channels every run. The non-dispatched channel's local pool is empty (onlymkdir -p'd), soapt-ftparchiveemits an emptyPackages.find dists -type fand pushes all dists to R2 → the empty index overwrites the other channel's good one.So every bleeding-edge snapshot publish (several per day) wipes the stable index minutes after each stable release. The per-channel regen that #2838 was meant to introduce is absent from the live script (only the arm64 fix from #4 is present).
Fix
regenerate-apt-metadata.sh:DISTRIBUTIONS="${CHANNELS:-stable bleeding-edge}"— default unchanged for manual full rebuilds; the workflow now scopes it.wheels-released.yml: regen step passesCHANNELS=<dispatched channel>. A publish now regenerates and uploads only its own channel's dists; the other channel's R2 index is untouched.CHANNELSis passed viaenv:and the channel value is already validated against a strictstable|bleeding-edgeallowlist in the Resolve inputs step — no injection surface.After merge
Dispatch
channel=stable, version=4.0.5(workflow_dispatch) to repopulate the stable index, then verifyapt install wheelsresolves on both amd64 and arm64.Refs: wheels-dev/wheels#3218, wheels-dev/wheels#2838