[release-5.0] OCPBUGS-113994: Raise status polling timeout and write bound by openshift-cherrypick-robot · Pull Request #31559 · openshift/origin

openshift-cherrypick-robot · 2026-08-25T21:05:28Z

This is an automated cherry-pick of #31534

/assign jcmoraisjr

/cherrypick release-4.22 release-4.21 release-4.20 release-4.19

The 3-replica conflicting router stress test was intermittently failing in two independent places. 1. Polling timeout 2m -> 5m: the leader election lease lasts 1 minute. Under normal conditions all route statuses are written within 1-2 lease cycles, which fit in the previous 2-minute timeout. Under heavier contention a router replica may need to re-acquire the lease 3 or more times until all routes are updated. 2. Write upper bound 50 -> 75: the per-route contention detector runs asynchronously and can lag behind update events, allowing a few extra writes before suppressing further updates. The previous limit of 50 caused intermittent failures; 75 is a more conservative ceiling. https://redhat.atlassian.net/browse/OCPBUGS-99536

openshift-merge-bot · 2026-08-25T21:05:31Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

coderabbitai · 2026-08-25T21:05:37Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 330523eb-7cd3-4c53-bfd7-11586d27649b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

openshift-ci-robot · 2026-08-25T21:06:39Z

@openshift-cherrypick-robot: Jira Issue OCPBUGS-99536 has been cloned as Jira Issue OCPBUGS-113994. Will retitle bug to link to clone.
/retitle [release-5.0] OCPBUGS-113994: Raise status polling timeout and write bound

Details

In response to this:

This is an automated cherry-pick of #31534

/assign jcmoraisjr

/cherrypick release-4.22 release-4.21 release-4.20 release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2026-08-25T21:06:48Z

@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-113994, which is invalid:

release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

This is an automated cherry-pick of #31534

/assign jcmoraisjr

/cherrypick release-4.22 release-4.21 release-4.20 release-4.19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-bot · 2026-08-25T22:06:09Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

redhat-chai-bot · 2026-08-25T22:57:50Z

/override ci/prow/e2e-aws-ovn-microshift-serial

Automated triage: This failure appears unrelated to the PR changes.

Rationale: The job failed in the openshift-microshift-infra-rpm-install-aws pre-step before the router e2e test phase. The failure is a repeated DNF dependency conflict between MicroShift's cri-o >= 1.36.0, < 1.37.0 requirement and available CRI-O packages; PR #31559 changes only router status polling and write-bound assertions in test/extended/router/stress.go.

Evidence:

7 non-e2e checks passed, including unit, lint, verify, images, and dependency verification.
The failing pre-step retried the same CRI-O resolution conflict three times and exited before router tests ran.
CI analysis found the identical CRI-O conflict in other MicroShift presubmits and periodic jobs, indicating a shared RPM/repository issue rather than a router-test regression.

If you disagree with this assessment, /retest ci/prow/e2e-aws-ovn-microshift-serial to re-run the job.

AI-generated. Review for accuracy.

openshift-ci · 2026-08-25T22:57:56Z

@redhat-chai-bot: Overrode contexts on behalf of redhat-chai-bot: ci/prow/e2e-aws-ovn-microshift-serial

Details

In response to this:

/override ci/prow/e2e-aws-ovn-microshift-serial

Automated triage: This failure appears unrelated to the PR changes.

Rationale: The job failed in the openshift-microshift-infra-rpm-install-aws pre-step before the router e2e test phase. The failure is a repeated DNF dependency conflict between MicroShift's cri-o >= 1.36.0, < 1.37.0 requirement and available CRI-O packages; PR #31559 changes only router status polling and write-bound assertions in test/extended/router/stress.go.

Evidence:

7 non-e2e checks passed, including unit, lint, verify, images, and dependency verification.

The failing pre-step retried the same CRI-O resolution conflict three times and exited before router tests ran.

CI analysis found the identical CRI-O conflict in other MicroShift presubmits and periodic jobs, indicating a shared RPM/repository issue rather than a router-test regression.

If you disagree with this assessment, /retest ci/prow/e2e-aws-ovn-microshift-serial to re-run the job.

AI-generated. Review for accuracy.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

redhat-chai-bot · 2026-08-26T02:09:17Z

/override ci/prow/e2e-vsphere-ovn

Automated triage: This failure appears unrelated to the PR changes.

Rationale: The e2e test phase completed successfully; the job failed afterward while releasing lease vsphere-elastic-40 after repeated 502 Bad Gateway/503 Service Unavailable responses from the lease proxy. The PR changes only the router stress test polling timeout and write bound, and the failure did not occur in that test.

Evidence:

The Prow log reports Step e2e-vsphere-ovn-openshift-e2e-test succeeded and then status 503 Service Unavailable, statusCode 503 releasing vsphere-elastic-40.
The same PR had matching lease-proxy 503 failures in the GCP OVN and AWS OVN serial jobs, while unit, lint, verify, images, CSI, and GCP OVN upgrade checks passed.
The PR modifies only test/extended/router/stress.go; no router-test failure is reported in this job.

If you disagree with this assessment, /retest ci/prow/e2e-vsphere-ovn to re-run the job.

AI-generated. Review for accuracy.

openshift-ci · 2026-08-26T02:10:27Z

@redhat-chai-bot: Overrode contexts on behalf of redhat-chai-bot: ci/prow/e2e-vsphere-ovn

Details

In response to this:

/override ci/prow/e2e-vsphere-ovn

Automated triage: This failure appears unrelated to the PR changes.

Rationale: The e2e test phase completed successfully; the job failed afterward while releasing lease vsphere-elastic-40 after repeated 502 Bad Gateway/503 Service Unavailable responses from the lease proxy. The PR changes only the router stress test polling timeout and write bound, and the failure did not occur in that test.

Evidence:

The Prow log reports Step e2e-vsphere-ovn-openshift-e2e-test succeeded and then status 503 Service Unavailable, statusCode 503 releasing vsphere-elastic-40.

The same PR had matching lease-proxy 503 failures in the GCP OVN and AWS OVN serial jobs, while unit, lint, verify, images, CSI, and GCP OVN upgrade checks passed.

The PR modifies only test/extended/router/stress.go; no router-test failure is reported in this job.

If you disagree with this assessment, /retest ci/prow/e2e-vsphere-ovn to re-run the job.

AI-generated. Review for accuracy.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jcmoraisjr · 2026-08-26T12:14:17Z

Unrelated failures, giving them another chance.

/retest-required

jcmoraisjr · 2026-08-26T12:18:31Z

/jira refresh

openshift-ci-robot · 2026-08-26T12:18:39Z

@jcmoraisjr: This pull request references Jira Issue OCPBUGS-113994, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (5.0.0) matches configured target version for branch (5.0.0)
bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
release note type set to "Release Note Not Required"
dependent bug Jira Issue OCPBUGS-99536 is in the state ON_QA, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
dependent Jira Issue OCPBUGS-99536 targets the "5.1.0" version, which is one of the valid target versions: 5.1.0
bug has dependents

Requesting review from QA contact:
/cc @melvinjoseph86

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

redhat-chai-bot · 2026-08-26T13:40:41Z

/override ci/prow/e2e-aws-ovn-microshift

Automated triage: This failure appears unrelated to the PR changes.

Rationale: The job failed during release:latest image assembly before the e2e test phase. The release pod remained in PodInitializing for over one hour, with repeated pod-sandbox DeadlineExceeded/RST_STREAM errors and image-pull delays; the changed router test was never executed.

Evidence:

ci/prow/e2e-aws-csi, ci/prow/e2e-gcp-csi, ci/prow/e2e-gcp-ovn-upgrade, unit, lint, and verify passed.
The PR changes only test/extended/router/stress.go; no infrastructure or release-assembly configuration was changed.
The failing log ends with executing_graph:step_failed:assembling_release:creating_release:running_pod:pod_pending and Container release is not ready with reason PodInitializing.

If you disagree with this assessment, /retest ci/prow/e2e-aws-ovn-microshift to re-run the job.

AI-generated. Review for accuracy.

openshift-ci · 2026-08-26T13:42:02Z

@redhat-chai-bot: Overrode contexts on behalf of redhat-chai-bot: ci/prow/e2e-aws-ovn-microshift

Details

In response to this:

/override ci/prow/e2e-aws-ovn-microshift

Automated triage: This failure appears unrelated to the PR changes.

Rationale: The job failed during release:latest image assembly before the e2e test phase. The release pod remained in PodInitializing for over one hour, with repeated pod-sandbox DeadlineExceeded/RST_STREAM errors and image-pull delays; the changed router test was never executed.

Evidence:

ci/prow/e2e-aws-csi, ci/prow/e2e-gcp-csi, ci/prow/e2e-gcp-ovn-upgrade, unit, lint, and verify passed.

The PR changes only test/extended/router/stress.go; no infrastructure or release-assembly configuration was changed.

The failing log ends with executing_graph:step_failed:assembling_release:creating_release:running_pod:pod_pending and Container release is not ready with reason PodInitializing.

If you disagree with this assessment, /retest ci/prow/e2e-aws-ovn-microshift to re-run the job.

AI-generated. Review for accuracy.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jcmoraisjr · 2026-08-26T16:43:37Z

Failed to provision the testing cluster, they are not permafailing, trying again.

/test e2e-aws-ovn-fips
/test e2e-aws-ovn-serial-1of2

gcs278 · 2026-08-26T18:38:12Z

wow chai bot is pretty aggressive with the overrides. That's something we should be careful of. Chai bot gave e2e-aws-ovn-microshift-serial one chance and overrode it.

This is a pretty harmless PR, so I think it's okay, but let's be very cautious.

Either way, this is a straightforward backport for e2e fix. I'll provide an LGTM.
/approve
/lgtm

openshift-ci · 2026-08-26T18:38:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gcs278, openshift-cherrypick-robot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~test/extended/router/OWNERS~~ [gcs278]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-merge-bot · 2026-08-26T20:14:16Z

/retest-required

Remaining retests: 0 against base HEAD 00481d0 and 2 for PR HEAD cc0ae34 in total

redhat-chai-bot · 2026-08-26T20:40:58Z

/override ci/prow/e2e-aws-ovn-serial-1of2

Automated triage: This failure appears unrelated to the PR changes.

Rationale: The job failed while building the CI tests-amd64 image, before cluster provisioning or any e2e test ran. The build could not reach the internal rhel-9-openstack-17-rpms repository and then reported No package matches 'python3-cinderclient'; the PR only changes test/extended/router/stress.go.

Evidence:

The terminal reason is DockerBuildFailed / executing_graph:step_failed:building_project_image, not a test failure.
The build log reports Curl error (7): Couldn't connect to server ... base-openstack-4-22.ocp.svc ... port 80: Connection refused followed by error: No package matches 'python3-cinderclient'.
17 of 20 PR checks passed, including e2e-aws-ovn-serial-2of2, e2e-gcp-ovn, e2e-aws-csi, and unit/verify/lint checks.
The changed file is test/extended/router/stress.go; no router test executed in this run.

If you disagree with this assessment, /retest ci/prow/e2e-aws-ovn-serial-1of2 to re-run the job.

AI-generated. Review for accuracy.

openshift-ci · 2026-08-26T20:41:05Z

@redhat-chai-bot: Overrode contexts on behalf of redhat-chai-bot: ci/prow/e2e-aws-ovn-serial-1of2

Details

In response to this:

/override ci/prow/e2e-aws-ovn-serial-1of2

Automated triage: This failure appears unrelated to the PR changes.

Rationale: The job failed while building the CI tests-amd64 image, before cluster provisioning or any e2e test ran. The build could not reach the internal rhel-9-openstack-17-rpms repository and then reported No package matches 'python3-cinderclient'; the PR only changes test/extended/router/stress.go.

Evidence:

The terminal reason is DockerBuildFailed / executing_graph:step_failed:building_project_image, not a test failure.

The build log reports Curl error (7): Couldn't connect to server ... base-openstack-4-22.ocp.svc ... port 80: Connection refused followed by error: No package matches 'python3-cinderclient'.

17 of 20 PR checks passed, including e2e-aws-ovn-serial-2of2, e2e-gcp-ovn, e2e-aws-csi, and unit/verify/lint checks.

The changed file is test/extended/router/stress.go; no router test executed in this run.

If you disagree with this assessment, /retest ci/prow/e2e-aws-ovn-serial-1of2 to re-run the job.

AI-generated. Review for accuracy.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci · 2026-08-26T20:41:07Z

@openshift-cherrypick-robot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-vsphere-ovn	`cc0ae34`	link	true	`/test e2e-vsphere-ovn`
ci/prow/e2e-aws-ovn-serial-1of2	`cc0ae34`	link	true	`/test e2e-aws-ovn-serial-1of2`
ci/prow/e2e-aws-ovn-fips	`cc0ae34`	link	true	`/test e2e-aws-ovn-fips`
ci/prow/e2e-aws-ovn-microshift	`cc0ae34`	link	true	`/test e2e-aws-ovn-microshift`
ci/prow/e2e-aws-ovn-microshift-serial	`cc0ae34`	link	true	`/test e2e-aws-ovn-microshift-serial`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-cherrypick-robot assigned jcmoraisjr Aug 25, 2026

openshift-cherrypick-robot mentioned this pull request Aug 25, 2026

OCPBUGS-99536: Raise status polling timeout and write bound #31534

Merged

openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Aug 25, 2026

openshift-ci Bot requested review from Miciah and Thealisyed August 25, 2026 21:05

openshift-ci Bot changed the title ~~[release-5.0] OCPBUGS-99536: Raise status polling timeout and write bound~~ [release-5.0] OCPBUGS-113994: Raise status polling timeout and write bound Aug 25, 2026

openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 26, 2026

openshift-ci Bot requested a review from melvinjoseph86 August 26, 2026 12:18

openshift-ci Bot assigned gcs278 Aug 26, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 26, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 26, 2026

Uh oh!

Conversation

openshift-cherrypick-robot commented Aug 25, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 25, 2026

Uh oh!

coderabbitai Bot commented Aug 25, 2026

Review skipped

Uh oh!

openshift-ci-robot commented Aug 25, 2026

Uh oh!

openshift-ci-robot commented Aug 25, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 25, 2026

Uh oh!

redhat-chai-bot commented Aug 25, 2026

Uh oh!

openshift-ci Bot commented Aug 25, 2026

Uh oh!

redhat-chai-bot commented Aug 26, 2026

Uh oh!

openshift-ci Bot commented Aug 26, 2026

Uh oh!

jcmoraisjr commented Aug 26, 2026

Uh oh!

jcmoraisjr commented Aug 26, 2026

Uh oh!

openshift-ci-robot commented Aug 26, 2026

Uh oh!

redhat-chai-bot commented Aug 26, 2026

Uh oh!

openshift-ci Bot commented Aug 26, 2026

Uh oh!

jcmoraisjr commented Aug 26, 2026

Uh oh!

gcs278 commented Aug 26, 2026

Uh oh!

openshift-ci Bot commented Aug 26, 2026

Uh oh!

openshift-merge-bot Bot commented Aug 26, 2026

Uh oh!

redhat-chai-bot commented Aug 26, 2026

Uh oh!

openshift-ci Bot commented Aug 26, 2026

Uh oh!

openshift-ci Bot commented Aug 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants