Skip to content

OCPBUGS-85519: Add Agent IRI credentials to global pull secret#6067

Open
sadasu wants to merge 1 commit into
openshift:mainfrom
sadasu:fix-iri-pull-secret
Open

OCPBUGS-85519: Add Agent IRI credentials to global pull secret#6067
sadasu wants to merge 1 commit into
openshift:mainfrom
sadasu:fix-iri-pull-secret

Conversation

@sadasu
Copy link
Copy Markdown
Contributor

@sadasu sadasu commented May 19, 2026

The IRI credentials were being added to /var/lib/kubelet/config.json which gave only kubelet the necessary credentials to pull images from the IRI registry.
Fix now adds the IRI credentials to the global pull secret so that other components like the MCO could also pull images from this internal registry.

- What I did

- How to verify it

- Description for the changelog

Summary by CodeRabbit

  • Bug Fixes
    • Ensure IRI registry credentials are merged into the cluster-wide global pull secret so rendered configurations reliably include required IRI auth and avoid missing-credential failures.
  • Behavior Change
    • Template rendering now assumes the global pull secret contains IRI credentials and no longer merges them during render.
  • Tests
    • Updated tests to validate rendering when IRI credentials come from the global pull secret.

@openshift-merge-bot
Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels May 19, 2026
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@sadasu: This pull request references Jira Issue OCPBUGS-85519, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

The IRI credentials were being added to /var/lib/kubelet/config.json which gave only kubelet the necessary credentials to pull images from the IRI registry.
Fix now adds the IRI credentials to the global pull secret so that other components like the MCO could also pull images from this internal registry.

- What I did

- How to verify it

- Description for the changelog

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 19, 2026

Walkthrough

This PR refactors IRI registry credential handling from the template controller to the internalreleaseimage controller. Helper functions are exported for reuse, global pull secret syncing is implemented in the internalreleaseimage controller, and template controller logic that merged credentials at render time is removed.

Changes

IRI Credentials Migration to InternalReleaseImage Controller

Layer / File(s) Summary
Export IRI secret helper functions
pkg/controller/common/iri_secret_merger.go
ExtractIRICredentials and MergeIRIRegistryCredentialsIntoPullSecret are renamed from unexported to exported; callers in NewIRISecretMerger, NewIRISecretMergerFromObjects, and Merge are updated to use the public names.
Add global pull secret syncing to InternalReleaseImage controller
pkg/controller/internalreleaseimage/internalreleaseimage_controller.go
Controller gains a kubeClient field wired in the constructor. New syncGlobalPullSecret helper is called during syncInternalReleaseImage to fetch the configured global pull secret, validate its type, extract IRI credentials, merge them into the dockerconfigjson payload, and update the secret when changes are detected.
Remove IRI merging from template controller
pkg/controller/template/template_controller.go
iriMerger field is removed from Controller struct; initialization of IRISecretMerger is removed; runtime merge call and error handling are eliminated and replaced with comments indicating IRI credentials are already present in the global pull secret.
Update template controller tests
pkg/controller/template/template_controller_test.go
Unused imports are removed; old test that merged credentials at render time is replaced with new test that verifies rendering when the global pull secret already contains IRI registry auth entries for quay.io, api-int.example.com:22625, and localhost:22625; assertions validate all expected hosts and their base64-encoded auth values.

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 11 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Test Structure And Quality ❓ Inconclusive Custom check specifies Ginkgo criteria (It blocks, BeforeEach/AfterEach), but PR contains only standard Go unit tests using testing.T, not Ginkgo BDD tests. Criteria not applicable. Clarify if check applies to standard Go unit tests, or verify if there are Ginkgo tests in this PR to review.
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title clearly and specifically describes the main change: exporting IRI credential handling functions and syncing Agent IRI credentials to the global pull secret instead of only the kubelet config.
Docstring Coverage ✅ Passed Docstring coverage is 87.50% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed PR modifies test files using standard Go testing framework, not Ginkgo. No Ginkgo tests with dynamic names are present in the modified code.
Microshift Test Compatibility ✅ Passed No new Ginkgo e2e tests were added in this PR. The modified test file contains standard Go unit tests using fake clients, not Ginkgo e2e tests.
Single Node Openshift (Sno) Test Compatibility ✅ Passed This PR adds no Ginkgo e2e tests. All changes are to controller logic and standard Go unit tests that use fake clients - not applicable to SNO compatibility check.
Topology-Aware Scheduling Compatibility ✅ Passed PR introduces no scheduling constraints, pod affinity, or topology assumptions. Changes are limited to credential merging logic within existing controllers.
Ote Binary Stdout Contract ✅ Passed PR changes contain no direct stdout writes, fmt.Print calls, or process-level code execution at module initialization. All logging uses klog configured to stderr.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed This PR does not add new Ginkgo e2e tests. Only standard Go unit tests in pkg/controller/template/template_controller_test.go were modified. The check is not applicable.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 19, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: sadasu
Once this PR has been reviewed and has the lgtm label, please assign yuqi-zhang for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sadasu sadasu force-pushed the fix-iri-pull-secret branch from eba73aa to 21b8d9e Compare May 19, 2026 20:43
@openshift-ci-robot
Copy link
Copy Markdown
Contributor

@sadasu: This pull request references Jira Issue OCPBUGS-85519, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (5.0.0) matches configured target version for branch (5.0.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
Details

In response to this:

The IRI credentials were being added to /var/lib/kubelet/config.json which gave only kubelet the necessary credentials to pull images from the IRI registry.
Fix now adds the IRI credentials to the global pull secret so that other components like the MCO could also pull images from this internal registry.

- What I did

- How to verify it

- Description for the changelog

Summary by CodeRabbit

  • Bug Fixes
  • Improved IRI registry credential management by syncing credentials to the global pull secret earlier in the process, ensuring credentials are available when needed for configuration rendering.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/controller/internalreleaseimage/internalreleaseimage_controller.go`:
- Around line 414-417: The reconcile only calls syncGlobalPullSecret but the
controller doesn't requeue on all resources that can stale the global pull
secret; update the controller's watches (SetupWithManager / Setup) so Reconcile
is triggered when the cluster pull secret (the Kubernetes Secret used as the
global pull secret) is created/updated/deleted, when ControllerConfig changes
(spec.pullSecret and spec.dns.baseDomain), and when any resource that can rotate
credentials (e.g., DockerRegistryKey) changes in addition to the existing
IRI-owned secrets. Concretely: add a watch for the cluster/global pull Secret
(by name/namespace or label) to enqueue affected InternalReleaseImage objects,
add a watch on ControllerConfig and filter to requeue when Spec.PullSecret or
Spec.DNS.BaseDomain changes, and ensure DockerRegistryKey and the two IRI-owned
secret watches remain; keep syncGlobalPullSecret as-is and rely on these
additional watches to ensure it runs whenever the global pull secret can become
stale.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0a3af203-7768-4e8d-b9bc-30a62398657d

📥 Commits

Reviewing files that changed from the base of the PR and between 75c3b83 and 21b8d9e.

📒 Files selected for processing (4)
  • pkg/controller/common/iri_secret_merger.go
  • pkg/controller/internalreleaseimage/internalreleaseimage_controller.go
  • pkg/controller/template/template_controller.go
  • pkg/controller/template/template_controller_test.go

Comment on lines +414 to +417
// Sync IRI credentials into the global pull secret
if err := ctrl.syncGlobalPullSecret(cconfig, iriRegistryCredentialsSecret); err != nil {
return fmt.Errorf("could not sync global pull secret: %w", err)
}
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Requeue on all inputs that can stale the global pull secret.

syncGlobalPullSecret only runs during an IRI reconcile, but this controller still requeues only on DockerRegistryKey changes and the two IRI-owned secrets. If the cluster pull secret is rotated/replaced, or ControllerConfig.Spec.PullSecret / the DNS base domain changes, the global secret can stay missing the IRI auth entries until some unrelated event happens.

Also applies to: 422-491

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/controller/internalreleaseimage/internalreleaseimage_controller.go`
around lines 414 - 417, The reconcile only calls syncGlobalPullSecret but the
controller doesn't requeue on all resources that can stale the global pull
secret; update the controller's watches (SetupWithManager / Setup) so Reconcile
is triggered when the cluster pull secret (the Kubernetes Secret used as the
global pull secret) is created/updated/deleted, when ControllerConfig changes
(spec.pullSecret and spec.dns.baseDomain), and when any resource that can rotate
credentials (e.g., DockerRegistryKey) changes in addition to the existing
IRI-owned secrets. Concretely: add a watch for the cluster/global pull Secret
(by name/namespace or label) to enqueue affected InternalReleaseImage objects,
add a watch on ControllerConfig and filter to requeue when Spec.PullSecret or
Spec.DNS.BaseDomain changes, and ensure DockerRegistryKey and the two IRI-owned
secret watches remain; keep syncGlobalPullSecret as-is and rely on these
additional watches to ensure it runs whenever the global pull secret can become
stale.

The IRI credentials were being added to /var/lib/kubelet/config.json
which gave only kubelet the necessary credentials to pull images from
the IRI registry.
Fix now adds the IRI credentials to the global pull secret so that
other components like the MCO could also pull images from this
internal registry.
@sadasu sadasu force-pushed the fix-iri-pull-secret branch from 21b8d9e to a2a957e Compare May 20, 2026 14:55
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/controller/template/template_controller.go`:
- Around line 584-586: The Secret filter currently hard-codes the name
"pull-secret" so changes to a differently named global pull secret won't
re-enqueue the template controller; update filterSecret to compare the incoming
Secret's name against the configured value from ControllerConfig.Spec.PullSecret
(accessed via the controller's config instance, e.g., c.config.Spec.PullSecret
or controllerConfig variable) and namespace as appropriate, falling back to
"pull-secret" only if Spec.PullSecret is empty; ensure you reference the
controller's config in filterSecret (filterSecret,
ControllerConfig.Spec.PullSecret) so the watch reacts to the actual configured
secret name.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: eed7f0d9-2394-4efe-ac5d-d3d662b75431

📥 Commits

Reviewing files that changed from the base of the PR and between 21b8d9e and a2a957e.

📒 Files selected for processing (4)
  • pkg/controller/common/iri_secret_merger.go
  • pkg/controller/internalreleaseimage/internalreleaseimage_controller.go
  • pkg/controller/template/template_controller.go
  • pkg/controller/template/template_controller_test.go

Comment on lines +584 to +586
// IRI registry credentials are now managed directly in the global pull secret
// by the IRI controller, so no merging is needed here. The global pull secret
// already contains IRI credentials when read above.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Don't assume the watched global pull secret is always named pull-secret.

With render-time merging gone, this path only converges when the global pull-secret update re-enqueues the template controller. filterSecret still hard-codes "pull-secret" instead of following ControllerConfig.Spec.PullSecret, so a differently named configured secret will not trigger a rerender and /var/lib/kubelet/config.json can stay stale until some unrelated enqueue.

Suggested change near filterSecret
 func (ctrl *Controller) filterSecret(secret *corev1.Secret) {
-	if secret.Name == "pull-secret" || secret.Name == ctrlcommon.InternalReleaseImageAuthSecretName {
+	if secret.Name == ctrlcommon.InternalReleaseImageAuthSecretName {
 		ctrl.enqueueController()
 		klog.Infof("Re-syncing ControllerConfig due to secret %s change", secret.Name)
+		return
+	}
+
+	cfg, err := ctrl.ccLister.Get(ctrlcommon.ControllerConfigName)
+	if err == nil && cfg.Spec.PullSecret != nil &&
+		secret.Namespace == cfg.Spec.PullSecret.Namespace &&
+		secret.Name == cfg.Spec.PullSecret.Name {
+		ctrl.enqueueController()
+		klog.Infof("Re-syncing ControllerConfig due to secret %s change", secret.Name)
 	}
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/controller/template/template_controller.go` around lines 584 - 586, The
Secret filter currently hard-codes the name "pull-secret" so changes to a
differently named global pull secret won't re-enqueue the template controller;
update filterSecret to compare the incoming Secret's name against the configured
value from ControllerConfig.Spec.PullSecret (accessed via the controller's
config instance, e.g., c.config.Spec.PullSecret or controllerConfig variable)
and namespace as appropriate, falling back to "pull-secret" only if
Spec.PullSecret is empty; ensure you reference the controller's config in
filterSecret (filterSecret, ControllerConfig.Spec.PullSecret) so the watch
reacts to the actual configured secret name.

@sadasu
Copy link
Copy Markdown
Contributor Author

sadasu commented May 26, 2026

/retest-required

@openshift-ci
Copy link
Copy Markdown
Contributor

openshift-ci Bot commented May 26, 2026

@sadasu: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants