[release-4.22] OCPBUGS-85124: use --delete-if-present for karg removal

[openshift-cherrypick-robot]

This is an automated cherry-pick of #5914

/assign isabella-janssen

/cherrypick release-4.21 release-4.20 release-4.19 release-4.18

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f69e77f8-0755-4e97-8b34-3a3ca1c66b36

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci-robot]

@openshift-cherrypick-robot: Jira Issue OCPBUGS-75894 has been cloned as Jira Issue OCPBUGS-85124. Will retitle bug to link to clone.

WARNING: Unexpected sprint field type []interface {} on source issue. Please update sprint manually on clone.

/retitle [release-4.22] OCPBUGS-85124: use --delete-if-present for karg removal

Details

In response to this:

This is an automated cherry-pick of #5914

/assign isabella-janssen

/cherrypick release-4.21 release-4.20 release-4.19 release-4.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-85124, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
• release note type set to "Release Note Not Required"
• dependent bug Jira Issue OCPBUGS-75894 is in the state MODIFIED, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
• dependent Jira Issue OCPBUGS-75894 targets the "5.0.0" version, which is one of the valid target versions: 5.0.0
• bug has dependents

Requesting review from QA contact:
/cc @tlbueno

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This is an automated cherry-pick of #5914

/assign isabella-janssen

/cherrypick release-4.21 release-4.20 release-4.19 release-4.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

@openshift-cherrypick-robot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-op-ocl-part2	37e6da3	link	false	/test e2e-gcp-op-ocl-part2
ci/prow/e2e-gcp-op-ocl	37e6da3	link	false	/test e2e-gcp-op-ocl

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[djoshy]

/lgtm

/label backport-risk-assessed

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aws-ovn
/test e2e-aws-ovn-upgrade
/test e2e-gcp-op-part1
/test e2e-gcp-op-part2
/test e2e-gcp-op-single-node
/test e2e-hypershift

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: djoshy, openshift-cherrypick-robot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [djoshy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ptalgulk01]

Pre-merge tested:

Environment Setup:
OCP version: 4.22.0-0-2026-05-18-140819-test-ci-ln-dcttjrt-latest
Platform: AWS

Steps:

1. Create a custom MCP
   result :

cat custom-mcp.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
  name: infra
spec:
  machineConfigSelector:
    matchExpressions:
      - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]}
  nodeSelector:
    matchLabels:
      node-role.kubernetes.io/infra: ""
      
oc label node ip-10-0-117-53.us-west-2.compute.internal  node-role.kubernetes.io/infra=

2. Apply kernel args based MC
   result:

oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1                
kind: MachineConfig                                                                                                                 
metadata:                                                       
  labels:                                                                                                                           
    machineconfiguration.openshift.io/role: infra              
  name: 99-karg-test                                                                                                                
spec:
  kernelArguments:                                                                                                                  
    - rcutree.nohz_full_patience_delay=1000                     
    - nmi_watchdog=0
EOF
machineconfig.machineconfiguration.openshift.io/99-karg-test created

4. wait for MCP update to complete and check changes are applied
   result:

oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
infra    rendered-infra-be1a4d9e613845b96f581ed30b54cf1d    True      False      False      1              1                   1                     0                      5m38s
master   rendered-master-233ff04eebd27bc4b7ebadc306352176   True      False      False      3              3                   3                     0                      43m
worker   rendered-worker-a60f9724409adaaeadb668b24d8d8031   True      False      False      2              2                   2                     0                      43m

oc debug node/ip-10-0-117-53.us-west-2.compute.internal -- chroot /host rpm-ostree kargs | grep rcutree
Starting pod/ip-10-0-117-53us-west-2computeinternal-debug-x27v5 ...
To use host binaries, run `chroot /host`

Removing debug pod ...
rw $ignition_firstboot  ostree=/ostree/boot.1/rhcos/cbd960afdfe78d311205728e3bc57b5e2cfecf402752b542345564ab64c96491/0 ignition.platform.id=aws console=tty0 console=ttyS0,115200n8 root=UUID=0018adf4-3000-4da3-976c-b3ef8b4d5b1f rw rootflags=prjquota boot=UUID=8fdfd281-6f43-489f-acd3-eef67817369c systemd.unified_cgroup_hierarchy=1 cgroup_no_v1="all" rcutree.nohz_full_patience_delay=1000 nmi_watchdog=0

Removing debug pod ...

5. Create config_drift by adding changes into node and reboot the node
   result:

oc debug node/ip-10-0-117-53.us-west-2.compute.internal --  chroot /host rpm-ostree kargs --delete=rcutree.nohz_full_patience_delay=1000
Starting pod/ip-10-0-117-53us-west-2computeinternal-debug-7h96w ...
To use host binaries, run `chroot /host`
Staging deployment...done
Changes queued for next boot. Run "systemctl reboot" to start a reboot

Removing debug pod ...

oc debug node/ip-10-0-117-53.us-west-2.compute.internal --  chroot /host systemctl reboot

6. Wait for MCP degrade
   result:

oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
infra    rendered-infra-be1a4d9e613845b96f581ed30b54cf1d    False     True       True       1              0                   0                     1                      18m
master   rendered-master-233ff04eebd27bc4b7ebadc306352176   True      False      False      3              3                   3                     0                      56m
worker   rendered-worker-a60f9724409adaaeadb668b24d8d8031   True      False      False      2              2                   2                     0                      56m

oc get mcp infra -o yaml
  - lastTransitionTime: "2026-05-18T15:18:33Z"
    message: 'Node ip-10-0-117-53.us-west-2.compute.internal is reporting: "Node ip-10-0-117-53.us-west-2.compute.internal
      upgrade failure. unexpected on-disk state validating against rendered-infra-be1a4d9e613845b96f581ed30b54cf1d:
      missing expected kernel arguments: [rcutree.nohz_full_patience_delay=1000]",
      Node ip-10-0-117-53.us-west-2.compute.internal is reporting: "unexpected on-disk
      state validating against rendered-infra-be1a4d9e613845b96f581ed30b54cf1d: missing
      expected kernel arguments: [rcutree.nohz_full_patience_delay=1000]"'
    reason: 1 nodes are reporting degraded status on sync
    status: "True"
    type: NodeDegraded

7. Apply second MC
   result:

oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1                                                                                    
kind: MachineConfig                                                                                                                 
metadata:                                                       
  labels:                                                                                                                           
    machineconfiguration.openshift.io/role: infra              
  name: 99-karg-test2                                        
spec:
  kernelArguments:                                                                                                                  
    - rcutree.nohz_full_patience_delay=2000                     
    - nmi_watchdog=0
EOF

8. Point the new rendered to failed node and touch the /run/machine-config-daemon-force
   result:

oc annotate node ip-10-0-117-53.us-west-2.compute.internal machineconfiguration.openshift.io/desiredConfig=rendered-infra-rendered-infra-c061083723b875e12f5dbc07f8bba848 --overwrite
node/ip-10-0-117-53.us-west-2.compute.internal annotated

oc debug node/ip-10-0-117-53.us-west-2.compute.internal --  chroot /host touch /run/machine-config-daemon-force

9. Check MCP is no more degrade and no error logs error: No key 'rcutree.nohz_full_patience_delay' found in MCD
   result:

oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
infra    rendered-infra-c061083723b875e12f5dbc07f8bba848    True      False      False      1              1                   1                     0                      37m
master   rendered-master-233ff04eebd27bc4b7ebadc306352176   True      False      False      3              3                   3                     0                      75m
worker   rendered-worker-a60f9724409adaaeadb668b24d8d8031   True      False      False      2              2                   2                     0                      75m

(Note Not to add in polarion) tested using this TC too

  [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:42365][OTP] add real time kernel argument [Disruptive]                                                                                            
  [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:42364][OTP] add selinux kernel argument [Disruptive]                                                                                              
  [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:67825][OTP] Use duplicated kernel arguments [Disruptive] 
    [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:72136][OTP] Reject MCs with ignition containing kernelArguments [Disruptive]                                                                      
  [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:53668][OTP] when FIPS and realtime kernel are both enabled node should NOT be degraded [Disruptive]                                               
  [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:67787][OTP] switch kernel type to 64k-pages for clusters with arm64 nodes [Disruptive]                                                            
  [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:67788][OTP] kernel type 64k-pages is not supported on non-arm64 nodes [Disruptive]     

/label qe-approved
/verified by @ptalgulk01

[openshift-ci-robot]

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-85124, which is valid.

7 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.22.0) matches configured target version for branch (4.22.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
• release note type set to "Release Note Not Required"
• dependent bug Jira Issue OCPBUGS-75894 is in the state Verified, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
• dependent Jira Issue OCPBUGS-75894 targets the "5.0.0" version, which is one of the valid target versions: 5.0.0
• bug has dependents

Requesting review from QA contact:
/cc @tlbueno

Details

In response to this:

This is an automated cherry-pick of #5914

/assign isabella-janssen

/cherrypick release-4.21 release-4.20 release-4.19 release-4.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@ptalgulk01: This PR has been marked as verified by @ptalgulk01.

Details

In response to this:

Pre-merge tested:

Environment Setup:
OCP version: 4.22.0-0-2026-05-18-140819-test-ci-ln-dcttjrt-latest
Platform: AWS

Steps:

1. Create a custom MCP
   result :

cat custom-mcp.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
 name: infra
spec:
 machineConfigSelector:
   matchExpressions:
     - {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,infra]}
 nodeSelector:
   matchLabels:
     node-role.kubernetes.io/infra: ""
     
oc label node ip-10-0-117-53.us-west-2.compute.internal  node-role.kubernetes.io/infra=

2. Apply kernel args based MC
   result:

oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1                
kind: MachineConfig                                                                                                                 
metadata:                                                       
 labels:                                                                                                                           
   machineconfiguration.openshift.io/role: infra              
 name: 99-karg-test                                                                                                                
spec:
 kernelArguments:                                                                                                                  
   - rcutree.nohz_full_patience_delay=1000                     
   - nmi_watchdog=0
EOF
machineconfig.machineconfiguration.openshift.io/99-karg-test created

4. wait for MCP update to complete and check changes are applied
   result:

oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
infra    rendered-infra-be1a4d9e613845b96f581ed30b54cf1d    True      False      False      1              1                   1                     0                      5m38s
master   rendered-master-233ff04eebd27bc4b7ebadc306352176   True      False      False      3              3                   3                     0                      43m
worker   rendered-worker-a60f9724409adaaeadb668b24d8d8031   True      False      False      2              2                   2                     0                      43m

oc debug node/ip-10-0-117-53.us-west-2.compute.internal -- chroot /host rpm-ostree kargs | grep rcutree
Starting pod/ip-10-0-117-53us-west-2computeinternal-debug-x27v5 ...
To use host binaries, run `chroot /host`

Removing debug pod ...
rw $ignition_firstboot  ostree=/ostree/boot.1/rhcos/cbd960afdfe78d311205728e3bc57b5e2cfecf402752b542345564ab64c96491/0 ignition.platform.id=aws console=tty0 console=ttyS0,115200n8 root=UUID=0018adf4-3000-4da3-976c-b3ef8b4d5b1f rw rootflags=prjquota boot=UUID=8fdfd281-6f43-489f-acd3-eef67817369c systemd.unified_cgroup_hierarchy=1 cgroup_no_v1="all" rcutree.nohz_full_patience_delay=1000 nmi_watchdog=0

Removing debug pod ...

5. Create config_drift by adding changes into node and reboot the node
   result:

oc debug node/ip-10-0-117-53.us-west-2.compute.internal --  chroot /host rpm-ostree kargs --delete=rcutree.nohz_full_patience_delay=1000
Starting pod/ip-10-0-117-53us-west-2computeinternal-debug-7h96w ...
To use host binaries, run `chroot /host`
Staging deployment...done
Changes queued for next boot. Run "systemctl reboot" to start a reboot

Removing debug pod ...

oc debug node/ip-10-0-117-53.us-west-2.compute.internal --  chroot /host systemctl reboot

6. Wait for MCP degrade
   result:

oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
infra    rendered-infra-be1a4d9e613845b96f581ed30b54cf1d    False     True       True       1              0                   0                     1                      18m
master   rendered-master-233ff04eebd27bc4b7ebadc306352176   True      False      False      3              3                   3                     0                      56m
worker   rendered-worker-a60f9724409adaaeadb668b24d8d8031   True      False      False      2              2                   2                     0                      56m

oc get mcp infra -o yaml
 - lastTransitionTime: "2026-05-18T15:18:33Z"
   message: 'Node ip-10-0-117-53.us-west-2.compute.internal is reporting: "Node ip-10-0-117-53.us-west-2.compute.internal
     upgrade failure. unexpected on-disk state validating against rendered-infra-be1a4d9e613845b96f581ed30b54cf1d:
     missing expected kernel arguments: [rcutree.nohz_full_patience_delay=1000]",
     Node ip-10-0-117-53.us-west-2.compute.internal is reporting: "unexpected on-disk
     state validating against rendered-infra-be1a4d9e613845b96f581ed30b54cf1d: missing
     expected kernel arguments: [rcutree.nohz_full_patience_delay=1000]"'
   reason: 1 nodes are reporting degraded status on sync
   status: "True"
   type: NodeDegraded

7. Apply second MC
   result:

oc create -f - << EOF
apiVersion: machineconfiguration.openshift.io/v1                                                                                    
kind: MachineConfig                                                                                                                 
metadata:                                                       
 labels:                                                                                                                           
   machineconfiguration.openshift.io/role: infra              
 name: 99-karg-test2                                        
spec:
 kernelArguments:                                                                                                                  
   - rcutree.nohz_full_patience_delay=2000                     
   - nmi_watchdog=0
EOF

8. Point the new rendered to failed node and touch the /run/machine-config-daemon-force
   result:

oc annotate node ip-10-0-117-53.us-west-2.compute.internal machineconfiguration.openshift.io/desiredConfig=rendered-infra-rendered-infra-c061083723b875e12f5dbc07f8bba848 --overwrite
node/ip-10-0-117-53.us-west-2.compute.internal annotated

oc debug node/ip-10-0-117-53.us-west-2.compute.internal --  chroot /host touch /run/machine-config-daemon-force

9. Check MCP is no more degrade and no error logs error: No key 'rcutree.nohz_full_patience_delay' found in MCD
   result:

oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
infra    rendered-infra-c061083723b875e12f5dbc07f8bba848    True      False      False      1              1                   1                     0                      37m
master   rendered-master-233ff04eebd27bc4b7ebadc306352176   True      False      False      3              3                   3                     0                      75m
worker   rendered-worker-a60f9724409adaaeadb668b24d8d8031   True      False      False      2              2                   2                     0                      75m

(Note Not to add in polarion) tested using this TC too

 [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:42365][OTP] add real time kernel argument [Disruptive]                                                                                            
 [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:42364][OTP] add selinux kernel argument [Disruptive]                                                                                              
 [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:67825][OTP] Use duplicated kernel arguments [Disruptive] 
   [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:72136][OTP] Reject MCs with ignition containing kernelArguments [Disruptive]                                                                      
 [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:53668][OTP] when FIPS and realtime kernel are both enabled node should NOT be degraded [Disruptive]                                               
 [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:67787][OTP] switch kernel type to 64k-pages for clusters with arm64 nodes [Disruptive]                                                            
 [sig-mco][Suite:openshift/machine-config-operator/longduration][Serial][Disruptive] MCO kernel [PolarionID:67788][OTP] kernel type 64k-pages is not supported on non-arm64 nodes [Disruptive]     

/label qe-approved
/verified by @ptalgulk01

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci-robot]

@openshift-cherrypick-robot: Jira Issue Verification Checks: Jira Issue OCPBUGS-85124
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-85124 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

This is an automated cherry-pick of #5914

/assign isabella-janssen

/cherrypick release-4.21 release-4.20 release-4.19 release-4.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-cherrypick-robot]

@openshift-cherrypick-robot: new pull request created: #6058

Details

In response to this:

This is an automated cherry-pick of #5914

/assign isabella-janssen

/cherrypick release-4.21 release-4.20 release-4.19 release-4.18

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-merge-robot]

Fix included in release 4.22.0-0.nightly-2026-05-19-052151