Skip to content

"MCO-2489: Raise minor_min to 4.20.32 for 4.21" - #10047

Open
shahsahil264 wants to merge 9 commits into
openshift:masterfrom
shahsahil264:OCPBUGS-83862-combined
Open

"MCO-2489: Raise minor_min to 4.20.32 for 4.21"#10047
shahsahil264 wants to merge 9 commits into
openshift:masterfrom
shahsahil264:OCPBUGS-83862-combined

Conversation

@shahsahil264

@shahsahil264 shahsahil264 commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Combined PR: raises minor_min from 4.20.15 to 4.20.25 for future 4.21.z builds, and adds conditional update risk for all existing 4.21.z GA releases (4.21.0 through 4.21.25).

Cherry-picks the build-suggestions change from #9494 (by @aksjadha) and adds the companion blocked-edge files requested by @sdodson and @wking.

Replaces #9494 and #10046.


Update by @hongkailiu

  • Claim AutoSizingReservedDisabledSilently for upgrade

    • 4.19 -> 4.20.z (z<32)
    • 4.20 -> 4.21.*: After merging, we will have to extend the risk to all future releases for 4.21 as the fix is not shipped to 4.21 (and there is no plan of doing it).
  • Raise minor_min to 4.20.32 for 4.21. It helps reduce the number of 4.20 clusters that carries over the issue to 4.21.

Bug

OCPBUGS-83862: During upgrade from pre-4.20.25, auto-sizing-disabled MachineConfigs override KubeletConfig autosizing settings on custom MachineConfigPool nodes. Fixed in 4.20.25 via openshift/machine-config-operator#5885.

Risk details

  • from: ^4[.]20[.](1?[0-9]|2[0-4])[+].*$ — matches 4.20.0 through 4.20.24
  • PromQL: Scoped to clusters with KubeletConfigs via apiserver_storage_objects (per Trevor's recommendation — 26k clusters without KubeletConfigs won't see the warning)
  • 26 blocked-edge files: one per GA 4.21.z release (4.21.0 through 4.21.25)

aksjadha and others added 2 commits July 21, 2026 11:40
….21.0-25

Warn clusters with KubeletConfigs upgrading from 4.20.24 or earlier to
any existing 4.21.z that auto-sizing-disabled MachineConfigs may override
KubeletConfig autosizing settings on custom MachineConfigPool nodes.

Uses PromQL matching on apiserver_storage_objects to scope the risk to
clusters that have KubeletConfigs, per Trevor's recommendation.

The fix (openshift/machine-config-operator#5885) shipped in 4.20.25.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Signed-off-by: Sahil Shah <sahshah@redhat.com>
@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Jul 21, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@shahsahil264: This pull request references Jira Issue OCPBUGS-83862, which is invalid:

  • expected the bug to be open, but it isn't
  • expected the bug to target either version "5.0." or "openshift-5.0.", but it targets "4.20.z" instead
  • expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is Closed (Done) instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Combined PR: raises minor_min from 4.20.15 to 4.20.25 for future 4.21.z builds, and adds conditional update risk for all existing 4.21.z GA releases (4.21.0 through 4.21.25).

Cherry-picks the build-suggestions change from #9494 (by @aksjadha) and adds the companion blocked-edge files requested by @sdodson and @wking.

Replaces #9494 and #10046.

Bug

OCPBUGS-83862: During upgrade from pre-4.20.25, auto-sizing-disabled MachineConfigs override KubeletConfig autosizing settings on custom MachineConfigPool nodes. Fixed in 4.20.25 via openshift/machine-config-operator#5885.

Risk details

  • from: ^4[.]20[.](1?[0-9]|2[0-4])[+].*$ — matches 4.20.0 through 4.20.24
  • PromQL: Scoped to clusters with KubeletConfigs via apiserver_storage_objects (per Trevor's recommendation — 26k clusters without KubeletConfigs won't see the warning)
  • 26 blocked-edge files: one per GA 4.21.z release (4.21.0 through 4.21.25)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Jul 21, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: b3005b81-74f4-4849-afb4-0991e82c183b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@shahsahil264

Copy link
Copy Markdown
Contributor Author

/assign @wking

Comment thread build-suggestions/4.21.yaml Outdated
@@ -1,5 +1,5 @@
default:
minor_min: 4.20.15
minor_min: 4.20.25

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Poking into the implementation, I don't think 4.20.25 is sufficient, and I've reported the issue as OCPBUGS-99650 . I don't think we should be raising the floor until we know when it's actually fixed. And given the low impact rate and the number of clusters who would be updated by a floor-raise, it might be too late in the 4.20.z lifecycle to declare at all?

@DavidHurta DavidHurta Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should be raising the floor until we know when it's actually fixed.

Agree.

And given the low impact rate and the number of clusters who would be updated by a floor-raise, it might be too late in the 4.20.z lifecycle to declare at all?

Yes, it's a larger ask. Definitely something to ponder about a little bit before introducing.

@@ -0,0 +1,13 @@
to: 4.21.9
from: ^4[.]20[.](1?[0-9]|2[0-4])[+].*$
url: https://redhat.atlassian.net/browse/OCPBUGS-83862

@DavidHurta DavidHurta Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, the https://redhat.atlassian.net/browse/OCPBUGS-83862 has a security level assigned.

This will hinder the url's purpose, making it less useful for folks interested in more information. Meaning, we can either create a new impact statement jira with public-facing information, or the bug can be evaluated to see whether the assigned security level is justified or whether it can be open to the public (assuming the bug's description is helpful for the conditional risk).

@DavidHurta DavidHurta Jul 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, not that actionable because of #10047 (comment), but it is something to address before merging if we decide to merge the PR.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

83862 seems open to everyone now.

I will probably still create "a new impact statement jira" to save the energy of our users on jumping between two bugs OCPBUGS-83862 and OCPBUGS-99650.

I will update the link soon.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hongkailiu

Copy link
Copy Markdown
Member

/hold

We will wait until the fix is available for main and all previous releases to 4.20.
Then probably declare the risks without bumping minor_min for 4.21.
The relevant discussion on slack.

@openshift-ci openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 3, 2026
…SizingReservedDisabledSilently

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@hongkailiu
hongkailiu force-pushed the OCPBUGS-83862-combined branch from 59c4131 to cdc2204 Compare August 5, 2026 18:26
hongkailiu and others added 3 commits August 5, 2026 14:30
…utoSizingReservedDisabledSilently

Update url to MCO-2489, name to AutoSizingReservedDisabledSilently,
revise message, and extend from range to 4.20.0-31 since 4.20.25
only partially fixed the issue (complete fix in 4.20.32).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Extend the AutoSizingReservedDisabledSilently risk to block upgrades
from 4.19 to 4.20.0-31, with fixedIn 4.20.32.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This reduces the number of 4.20 clusters that carries over the
risk AutoSizingReservedDisabledSilently.

Every 4.20.z (0<=z<32) to 4.21 upgrade does that.

4.21 has no fix for the risk and needs to fix manually. We want to avoid
it if possible.
@hongkailiu
hongkailiu force-pushed the OCPBUGS-83862-combined branch from cdc2204 to cbbe361 Compare August 5, 2026 18:32
@hongkailiu

hongkailiu commented Aug 5, 2026

Copy link
Copy Markdown
Member

/retitle "MCO-2489: Raise minor_min to 4.20.32 for 4.21 and block upgrades for AutoSizingReservedDisabledSilently"

@openshift-ci openshift-ci Bot changed the title OCPBUGS-83862: Raise minor_min to 4.20.25 and add conditional update risk for 4.21.0-25 "MCO-2489: Raise minor_min to 4.20.32 for 4.21 and block upgrades for AutoSizingReservedDisabledSilently" Aug 5, 2026
@openshift-ci-robot openshift-ci-robot removed jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 5, 2026
@openshift-ci-robot

openshift-ci-robot commented Aug 5, 2026

Copy link
Copy Markdown

@shahsahil264: This pull request references MCO-2489 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the spike to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

Combined PR: raises minor_min from 4.20.15 to 4.20.25 for future 4.21.z builds, and adds conditional update risk for all existing 4.21.z GA releases (4.21.0 through 4.21.25).

Cherry-picks the build-suggestions change from #9494 (by @aksjadha) and adds the companion blocked-edge files requested by @sdodson and @wking.

Replaces #9494 and #10046.

Bug

OCPBUGS-83862: During upgrade from pre-4.20.25, auto-sizing-disabled MachineConfigs override KubeletConfig autosizing settings on custom MachineConfigPool nodes. Fixed in 4.20.25 via openshift/machine-config-operator#5885.

Risk details

  • from: ^4[.]20[.](1?[0-9]|2[0-4])[+].*$ — matches 4.20.0 through 4.20.24
  • PromQL: Scoped to clusters with KubeletConfigs via apiserver_storage_objects (per Trevor's recommendation — 26k clusters without KubeletConfigs won't see the warning)
  • 26 blocked-edge files: one per GA 4.21.z release (4.21.0 through 4.21.25)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

….26-28

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@hongkailiu

Copy link
Copy Markdown
Member

Currently, the pull both bumps minor_min and claim the risk.

If we claim the risk, we have to claim the risk for each 4.21 future releases. It is because there is no fix version for 4.21 clusters. We have to fix it manually. It cannot be done by a patch to 4.21 because it is technically difficult.

It is not cool but I do not have a better solution for it.

Another choice is we do not claim the risk at all. Maybe there are not many of clusters that enables autoSizingReserved on custom MCPs. 4.20 and 4.21 are there for a while, do we have complains of users about it?

I believe in any case we want minor_min: 4.20.32 to reduce the number of carryovers.

hongkailiu and others added 2 commits August 17, 2026 09:33
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This will avoid extending the risk for each future 4.21 release as there
won't be a fix for it. The downside is that the users do not know about.
It is probably fine as the amount of impacted clusters by the risk is
relatively low.
@hongkailiu

hongkailiu commented Aug 18, 2026

Copy link
Copy Markdown
Member

/retitle "MCO-2489: Raise minor_min to 4.20.32 for 4.21"

@openshift-ci openshift-ci Bot changed the title "MCO-2489: Raise minor_min to 4.20.32 for 4.21 and block upgrades for AutoSizingReservedDisabledSilently" "MCO-2489: Raise minor_min to 4.20.32 for 4.21" Aug 18, 2026
@hongkailiu

Copy link
Copy Markdown
Member

/label tide/merge-method-squash

@openshift-ci openshift-ci Bot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Aug 18, 2026
@ngopalak-redhat

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 18, 2026
@openshift-ci

openshift-ci Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ngopalak-redhat, shahsahil264
Once this PR has been reviewed and has the lgtm label, please ask for approval from wking. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci

openshift-ci Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

@shahsahil264: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants