fix: stop edge-caching apt metadata (no-store) + purge on publish — #3218 follow-up#6
Merged
Merged
Conversation
…218 follow-up) The cross-channel clobber (apt-wheels#5) was only half the #3218 story. The other half is CDN caching: Cloudflare auto-caches by file extension, so a published Packages.gz got edge-cached and kept being served after the next publish rewrote it. Its hash no longer matched the freshly-regenerated Release, so `apt-get update` rejected it with "File has unexpected size (29 != 1404)" — exactly the symptom users keep reporting. The uncompressed Packages/Release are extension-less (served DYNAMIC, never cached), which is why only the .gz broke and why a cache-buster query masked it (different cache key -> origin). Fix: - Upload apt metadata (Release, Release.gpg, InRelease, Packages, Packages.gz) with `Cache-Control: no-store` so the edge never caches a stale index again. This alone prevents recurrence regardless of purge permissions. - Upload .deb pool files with `public, max-age=31536000, immutable` — they're version-stamped and never change, so long edge caching is correct and fast. - Add a best-effort Cloudflare purge step that evicts the just-published URLs, to unstick copies cached BEFORE no-store existed (the current legacy entry). Resolves the wheels.dev zone at runtime; warns and continues (does not fail the publish) if the token lacks Zone.Cache-Purge scope. Refs: wheels-dev/wheels#3218, #5 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <peter@alurium.com>
bpamiri
pushed a commit
to wheels-dev/wheels
that referenced
this pull request
Jun 19, 2026
…ctly Two corrections to the index-integrity job after finding the real #3218 failure mode is CDN edge-cache staleness, not just the origin clobber: 1. Drop the `?cb=` cache-buster. apt/dnf fetch the plain URLs that hit Cloudflare's edge; a cache-buster hit R2 origin instead, so the probe went green while real `apt install` failed on a stale edge-cached Packages.gz. Test the SAME plain URLs clients use. 2. Verify hash-consistency, exactly what the package managers verify: parse the SHA256 the (plain) Release records for each binary-<arch>/Packages.gz, fetch the (plain) Packages.gz, and compare — this is precisely the "File has unexpected size" check that failed in #3218. yum: verify the served primary.xml.gz matches the hash repomd references. Then gunzip and confirm the content lists the GA (catches an empty-but-internally-consistent index). Binary correctness: gzip blobs are fetched to a temp file and hashed/gunzipped from the file — never captured into a shell variable, since `$(...)` is text-only and silently corrupts binary (strips trailing newlines, drops NULs), which would compute a bogus hash. Paired with apt-wheels#6 / yum-wheels#6 (set `no-store` on metadata so the edge stops caching it). Verified against the live indexes: correctly RED on the currently-stale apt stable amd64 edge, GREEN on arm64 / bleeding-edge / yum. Refs: #3218, wheels-dev/apt-wheels#6, wheels-dev/yum-wheels#6 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <petera@pai.com>
bpamiri
added a commit
to wheels-dev/yum-wheels
that referenced
this pull request
Jun 19, 2026
…218 parallel) (#6) Mirror of apt-wheels#6 for the rpm side. Cloudflare auto-caches by extension, so a published repomd.xml / *-primary.xml.gz can be edge-cached and served stale after the next publish rewrites it, leaving dnf with a repomd that references checksums the served metadata no longer matches. repodata changes every publish -> upload it with `Cache-Control: no-store` so the edge never serves a stale index. .rpm packages are version-stamped/immutable -> `public, max-age=1y, immutable`. Plus a best-effort Cloudflare purge step (resolves the wheels.dev zone at runtime; warns + continues if the token lacks Zone.Cache-Purge scope) to evict copies cached before no-store existed. yum-wheels was not the repo that triggered #3218 (its upload was already channel-scoped, so it never had the apt clobber), but it shared the same extension-based edge-cache exposure on metadata — hardening it now so the rpm channel can't develop the same stale-metadata failure. Refs: wheels-dev/wheels#3218, wheels-dev/apt-wheels#6 Signed-off-by: Peter Amiri <peter@alurium.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
bpamiri
added a commit
to wheels-dev/wheels
that referenced
this pull request
Jun 19, 2026
…gression guard) (#3234) * ci: add index-integrity probe to distribution install smoke (#3218 regression guard) The daily install-smoke installs the CLI per channel and asserts the version, but that only samples once at 14:00 UTC. The #3218 cross-channel clobber empties the stable apt index for most of the day (a bleeding-edge snapshot wipes it minutes after each stable publish; it's only briefly populated), so a clobber can land outside the install legs' sample window and slip past them — and when they do fail, "apt install failed" doesn't name the cause. Add a fast, container-free `index-integrity` job that probes the PUBLISHED apt and yum dist indexes directly: stable must be non-empty and name the current GA (per arch for apt), and bleeding-edge must stay non-empty (the clobber was bidirectional — apt-wheels#5 scopes regen per-channel to prevent both directions). An empty/missing/stale index fails with a message that names the regression. Reliability: fetches use `curl --retry ... --retry-all-errors` so a transient blip can't false-red the guardian, and all grep checks feed from a here-string (`grep -q PAT <<<"$body"`) rather than `printf | grep -q` — under `set -o pipefail` the latter false-fails when grep -q early-exits and SIGPIPEs the upstream printf of a 100KB index. Verified: 3/3 green against the live indexes, and a negative test (bogus version) correctly fails. Refs: #3218, wheels-dev/apt-wheels#5 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <petera@pai.com> * ci: probe plain (edge) URLs + hash-consistency, mirroring apt/dnf exactly Two corrections to the index-integrity job after finding the real #3218 failure mode is CDN edge-cache staleness, not just the origin clobber: 1. Drop the `?cb=` cache-buster. apt/dnf fetch the plain URLs that hit Cloudflare's edge; a cache-buster hit R2 origin instead, so the probe went green while real `apt install` failed on a stale edge-cached Packages.gz. Test the SAME plain URLs clients use. 2. Verify hash-consistency, exactly what the package managers verify: parse the SHA256 the (plain) Release records for each binary-<arch>/Packages.gz, fetch the (plain) Packages.gz, and compare — this is precisely the "File has unexpected size" check that failed in #3218. yum: verify the served primary.xml.gz matches the hash repomd references. Then gunzip and confirm the content lists the GA (catches an empty-but-internally-consistent index). Binary correctness: gzip blobs are fetched to a temp file and hashed/gunzipped from the file — never captured into a shell variable, since `$(...)` is text-only and silently corrupts binary (strips trailing newlines, drops NULs), which would compute a bogus hash. Paired with apt-wheels#6 / yum-wheels#6 (set `no-store` on metadata so the edge stops caching it). Verified against the live indexes: correctly RED on the currently-stale apt stable amd64 edge, GREEN on arm64 / bleeding-edge / yum. Refs: #3218, wheels-dev/apt-wheels#6, wheels-dev/yum-wheels#6 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Peter Amiri <petera@pai.com> --------- Signed-off-by: Peter Amiri <petera@pai.com> Co-authored-by: Peter Amiri <petera@pai.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The other half of #3218
apt-wheels#5 fixed the cross-channel clobber (origin was writing an empty
Packages). But users still can'tapt install wheels, because of a second, independent cause: CDN caching.Cloudflare auto-caches by file extension. A published
Packages.gzgets edge-cached and keeps being served after the next publish rewrites it — its hash no longer matches the freshly-regeneratedRelease, soapt-get updaterejects it:The uncompressed
Packages/Release/InReleaseare extension-less → Cloudflare serves themDYNAMIC(never cached), which is why only the.gzbroke, and why a?cb=cache-buster masked it (different cache key → origin). Verified live: plainPackages.gz=cf-cache-status: HIT, 29 bytes (empty, from a prior clobber);?cb==MISS, 1404 bytes (correct).Fix
Release,Release.gpg,InRelease,Packages,Packages.gz) uploaded withCache-Control: no-store— the edge never caches a stale index again. This prevents recurrence on its own, regardless of purge permissions..debpool files uploaded withpublic, max-age=31536000, immutable— version-stamped and immutable, so long edge caching is correct and fast.no-storeexisted (the current legacy stale entry). Resolves thewheels.devzone at runtime; warns and continues (never fails the publish) if the token lacksZone.Cache-Purgescope.After merge
Dispatch
channel=stable, version=4.0.5to re-upload with the new headers + purge, thenapt install wheelsworks on a clean box (will verify). The purge step's log says whether the token has zone scope; if not,no-store+ the legacy TTL still resolve it, and a one-time dashboard purge is the fallback.Refs: wheels-dev/wheels#3218, #5