Skip to content

Fix tag-based manifest pulls 404ing while digest-based pulls succeed - #2438

Open
wussh wants to merge 2 commits into
pulp:mainfrom
wussh:fix-2417-tag-manifest-404
Open

Fix tag-based manifest pulls 404ing while digest-based pulls succeed#2438
wussh wants to merge 2 commits into
pulp:mainfrom
wussh:fix-2417-tag-manifest-404

Conversation

@wussh

@wussh wussh commented Jul 26, 2026

Copy link
Copy Markdown

Summary

  • ManifestResponse.from_tag rejected requests based on the client Accept header, raising ManifestNotFound when the tag's manifest media type wasn't explicitly listed. from_manifest (the digest lookup path) never had this check, so the identical content served 200 by digest and 404 by tag — most visible for manifest-list/OCI-index tags whose media type clients often omit from Accept.
  • Regression from Omit redirects to content-app #1974 (8479d527), which moved manifest serving from a content-app redirect (that ignored Accept) into the registry API view.
  • from_tag now mirrors from_manifest: it always serves the tagged manifest, only rewriting schema v1 to the signed content type.

Fixes #2417

Test plan

  • Reworked test_pull_content.py::test_api_performes_schema_conversion (previously asserted a 4xx with the comment "I don't understand what this is testing") into test_api_serves_tag_regardless_of_accept_header, which pulls the same manifest by tag and by digest with a narrow Accept header and asserts identical 200 responses.
  • Reproduced end-to-end on the live domain-scoped, on-demand, follow-latest environment that originally reported Domain-scoped container registry: tag-based manifest fetch 404s, digest-based works (pulpcore 3.113.0 / pulp_container 2.28.0) #2417 (Domains enabled, S3-backed storage, pulpcore 3.113.0 / pulp_container 2.28.0). Pulled the exact live registry_api.py out of the running pulp-api pod, applied only the from_tag diff from this PR on top of it (no unrelated version drift), wrote it back into the running container, and reloaded gunicorn workers (no pod restart). Results:
    • Before patch: tag pull 404, digest pull 200 (matches the issue exactly).
    • After patch: tag pull 200, digest pull 200, tags/list 200, and the tag/digest response bodies are byte-identical.
    • Verified pod stayed healthy throughout, then reverted the pod back to the original unpatched image and re-confirmed the original 404 baseline — no lasting changes were left on the cluster.

@wussh

wussh commented Jul 26, 2026

Copy link
Copy Markdown
Author

Verified this fix end-to-end against the live cluster that originally hit #2417 (Domains + on-demand + follow-latest distribution, S3 storage, pulpcore 3.113.0 / pulp_container 2.28.0):

  • Before: GET /v2/default/quay/prometheus/prometheus/manifests/latest404; the same manifest by digest → 200.
  • After (applying only this PR's from_tag diff to the live registry_api.py, no unrelated changes, then reloading gunicorn workers): tag pull → 200, digest pull → 200, tags/list200, and the tag/digest response bodies are byte-identical.
  • Cluster was left in its original (pre-fix) state afterward — no permanent changes.

@gerrod3

gerrod3 commented Jul 28, 2026

Copy link
Copy Markdown
Contributor

@wussh Did you see my comment on the original issue? I'm not sure we should be removing the accept header check. In fact we should probably add it to the digest route for consistency. I went and fix some bugs with the accept header check #2424

Comment thread lint_requirements.txt Outdated
wussh added a commit to wussh/pulp_container that referenced this pull request Jul 31, 2026
- error_code: use the CON00XX naming convention (CON0001) jobselko
  and gerrod3 agreed on, instead of PLP_CONTAINER_BUILD_0001.
- CHANGES fragment: use single backticks, since they render as
  markdown.
- cdomain_factory: adopt gerrod3's version of the namespace cleanup
  from pulp#2424 (suppress errors per-namespace/guard instead of letting
  one failure abort the rest, and wait on the delete task via
  monitor_task).
- Drop the ruff<0.16 pin: no longer needed after rebasing on main,
  where the 0.16 lint findings were already fixed elsewhere.
@wussh
wussh force-pushed the fix-2417-tag-manifest-404 branch from 806b30a to 5fae027 Compare July 31, 2026 10:28
@wussh

wussh commented Jul 31, 2026

Copy link
Copy Markdown
Author

Addressed the mechanical review feedback:

  • Dropped the ruff<0.16 pin — rebased on main, where the 0.16 lint findings were already fixed elsewhere, so it's no longer needed.
  • ContainerBuildError.error_code renamed to CON0001 per the naming convention you and @jobselko settled on.
  • Changelog fragment now uses single backticks (renders as markdown).
  • cdomain_factory now uses your version of the namespace cleanup from Fix 404 on tag fetch due to overly strict Accept header matching #2424 (suppress + monitor_task) instead of mine.

Still not addressed: your point about the accept-header check itself (removing it from from_tag vs. adding it to the digest route for consistency, and #2424 fixing the underlying matching bugs). Wanted to get the mechanical stuff out of the way first — let me know how you'd like to proceed on that part, happy to rework or close this in favor of #2424 if that's the better fix.

@dralley

dralley commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Squash the commits please

@gerrod3 gerrod3 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking pretty good. Few changes and we should be good to merge.

Comment thread CHANGES/2417.bugfix Outdated
@@ -0,0 +1 @@
Fixed tag-based manifest pulls returning 404 when the client's ``Accept`` header did not list the tagged manifest's media type, while the identical digest-based pull succeeded.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Single tick marks.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed — single backticks now, both here and in the docstring.

Comment thread pulp_container/app/utils.py Outdated
"""
Return whether a manifest media type satisfies an Accept header.

Supports exact media type matches and the ``*/*`` wildcard used by some clients.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Single tick marks.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

Comment thread pulp_container/app/utils.py Outdated
Comment on lines +78 to +81
for accepted in accepted_media_types:
if accepted in ("*/*", media_type):
return True
return False

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this I think we can do this:

Suggested change
for accepted in accepted_media_types:
if accepted in ("*/*", media_type):
return True
return False
return accepted in ("*/*", *accepted_media_types)

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied, with one tweak: your literal snippet references accepted outside the loop it was defined in, and checking media_type against a tuple of ("*/*", *accepted_media_types) doesn't actually catch the wildcard case (it only matches if media_type itself equals "*/*", which it never does). Collapsed the loop into return any(accepted in ("*/*", media_type) for accepted in accepted_media_types) instead — same semantics as before, just a one-liner. Pushed in 01f7832.

@wussh

wussh commented Aug 6, 2026

Copy link
Copy Markdown
Author

@gerrod3 addressed all three review comments in 01f7832 — single backticks in the changelog and docstring, and is_media_type_accepted collapsed to a one-liner (see inline reply for a small correction to the suggested snippet, which had an unbound variable and missed the wildcard case).

Also confirming the earlier lint_requirements.txt comment is moot now — that file no longer diverges from main after the rebase.

@gerrod3

gerrod3 commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

@wussh Can you squash and repush? Hopefully github actions are working again, then I can approve and merge.

The Accept-header comparison in from_tag() did not strip parameters
(e.g. q-values) from each entry and had no support for the */*
wildcard, so a client sending a qualified Accept header for the
correct media type was incorrectly rejected with a 404. Fixed the
parsing in get_accepted_media_types() and added is_media_type_accepted()
to perform the comparison correctly, restoring the negotiation check
gerrod3 flagged as silently dropped.

Also raises a proper PulpException subclass (ContainerBuildError)
instead of a bare Exception when building or pushing an OCI image
fails, so the error isn't sanitized away by pulpcore in a future
release.
@wussh
wussh force-pushed the fix-2417-tag-manifest-404 branch from 5320f75 to ab61e13 Compare August 8, 2026 08:57
@wussh

wussh commented Aug 8, 2026

Copy link
Copy Markdown
Author

Squashed into a single commit and repushed — ab61e13. Also dropped some stray CHANGES.md/pyproject.toml noise that had crept in from an earlier rebase (unrelated bot commits, verified they weren't part of this fix's diff).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Domain-scoped container registry: tag-based manifest fetch 404s, digest-based works (pulpcore 3.113.0 / pulp_container 2.28.0)

4 participants