OCPBUGS-86473: Consolidate audit log must-gather tests to reduce parallel downloads and master node CPU pressure by xueqzhan · Pull Request #31200 · openshift/origin

xueqzhan · 2026-05-20T17:26:53Z

Summary by CodeRabbit

Tests
- Enhanced audit-logs must-gather test with clearer validation steps and consolidated coverage.
- Now verifies apiserver audit directories were processed strictly sequentially and validates OAuth audit entries include audit IDs.
- Removed redundant standalone tests whose checks are now covered in the consolidated flow.
- Adjusted must-gather invocations in related tests to rely on default gather behavior while preserving output assertions.

openshift-merge-bot · 2026-05-20T17:26:56Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

coderabbitai · 2026-05-20T17:27:16Z

Walkthrough

Consolidates audit-logs validations into a single must-gather test: adds an explicit validation step, checks apiserver audit directories/lock.log for sequential execution, validates OAuth gzip audit files contain "auditID", removes now-redundant standalone tests, and simplifies two must-gather invocations to omit the explicit gather command.

Changes

Audit Logs Test Refactoring

Layer / File(s)	Summary
Audit-logs validation and added checks `test/extended/cli/mustgather.go`	Adds an explicit `g.By` step label for audit validation, integrates sequential apiserver checks by walking apiserver audit subdirectories and inspecting `lock.log`, and validates OAuth audit gzip files by asserting the presence of `auditID`.
Removed redundant must-gather tests `test/extended/cli/mustgather.go`	Removes the separate tests that previously validated sequential apiserver execution and login-attempt OAuth audit logs; those checks are moved into the consolidated audit-logs test.
Simplified must-gather invocations in other tests `test/extended/cli/mustgather.go`	Update two tests to call `oc adm must-gather` with only `--dest-dir` (remove explicit `-- /usr/bin/gather_audit_logs`) while keeping existing `must-gather.logs`/output assertions.

🎯 3 (Moderate) | ⏱️ ~20 minutes

Caution

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

Ignore

❌ Failed checks (1 error, 3 warnings)

Check name	Status	Explanation	Resolution
Stable And Deterministic Test Names	❌ Error	Two g.By() steps use dynamic filepath variables (lines 209, 294) that change between runs, violating the stable test naming requirement.	Replace g.By(path) with static descriptive strings like g.By("Validating audit file format") that describe what is being tested.
Docstring Coverage	⚠️ Warning	Docstring coverage is 25.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.
Test Structure And Quality	⚠️ Warning	Multiple assertions lack meaningful failure messages (lines 42, 48, 95, 210, 225, 229, etc.) per requirement `#4`, violating the check's assertion message requirement.	Add descriptive messages to all Expect() assertions lacking them, e.g., change `o.Expect(err).NotTo(o.HaveOccurred())` to `o.Expect(err).NotTo(o.HaveOccurred(), "failed to [action]")`.
Single Node Openshift (Sno) Test Compatibility	⚠️ Warning	New test file mustgather.go has 3 tests with multi-node assumptions: one expects masters directory; two access workerNodeList[0] without SNO guards.	Add [Skipped:SingleReplicaTopology] labels to affected tests or add exutil.IsSingleNode() guards to skip on SNO.

✅ Passed checks (8 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Microshift Test Compatibility	✅ Passed	PR extends existing test already protected with [apigroup:config.openshift.io][apigroup:oauth.openshift.io] tags. MicroShift CI auto-skips these. No new unprotected tests added.
Topology-Aware Scheduling Compatibility	✅ Passed	PR only modifies test/extended/cli/mustgather.go, an E2E test file with no deployment manifests, operator code, or scheduling constraints. Check not applicable to test infrastructure changes.
Ote Binary Stdout Contract	✅ Passed	File has no process-level stdout writes: no main/init/TestMain/BeforeSuite functions, no fmt.Print/log writes at module level, all code properly contained in Ginkgo callbacks.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	Hardcoded "127.0.0.1" in OAuth token URIs is never accessed; test logic is IP-family agnostic and works identically on IPv4, IPv6, and dual-stack clusters.
Title check	✅ Passed	The PR title references OCPBUGS-86473 and mentions consolidating audit log tests to reduce parallel downloads and CPU pressure, which aligns with the actual changes that refactor must-gather audit log tests and remove redundant test flows.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

openshift-ci · 2026-05-20T17:30:59Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: xueqzhan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [xueqzhan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

xueqzhan · 2026-05-20T17:42:58Z

/payload-job periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-rhcos9-techpreview

openshift-ci · 2026-05-20T17:43:17Z

@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-rhcos9-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/57d1de60-5473-11f1-8018-90fc888065f7-0

xueqzhan · 2026-05-20T17:43:42Z

/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview

openshift-ci · 2026-05-20T17:43:50Z

@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/71dac150-5473-11f1-8c56-bf533e7f0877-0

openshift-merge-bot · 2026-05-20T17:50:22Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

xueqzhan · 2026-05-20T17:51:44Z

/payload-aggregate periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-rhcos9-techpreview 5

openshift-ci · 2026-05-20T17:51:48Z

@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-ci-5.0-e2e-aws-ovn-rhcos9-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/91671ef0-5474-11f1-98a8-f4e80f2c9b13-0

xueqzhan · 2026-05-20T17:52:28Z

/payload-aggregate periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview 4

openshift-ci · 2026-05-20T17:52:35Z

@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/ab855ea0-5474-11f1-8777-eccceea9d0ed-0

xueqzhan · 2026-05-21T00:40:11Z

/payload-aggregate periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview 6

openshift-ci · 2026-05-21T00:40:14Z

@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/a02bb390-54ad-11f1-84f9-af2b2ba73bbf-0

xueqzhan · 2026-05-21T20:50:06Z

/payload-aggregate periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview 6

openshift-ci · 2026-05-21T20:50:09Z

@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/a6785b50-5556-11f1-8ae8-0bd1020826e7-0

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/extended/cli/mustgather.go`:
- Around line 294-295: The current code calls result.Flakef unconditionally when
lock.log exists, causing false flakes; update the logic around the
result.Flakef(apiserver, lockLog) call so it only reports a flake when the
captured lockLog actually contains content (e.g., check len(lockLog) > 0 or
strings.TrimSpace(string(lockLog)) != "" before calling result.Flakef). Locate
the call to result.Flakef and the lockLog variable in mustgather.go and wrap the
Flakef invocation in that conditional, returning nil (or continuing) when
lockLog is empty.
- Line 299: Replace the incorrect Gomega assertion
o.Expect(seen.HasAll(expectedAuditSubDirs...), o.BeTrue()) with the proper
matcher call by invoking .To(o.BeTrue()) on the Expect result (i.e.
o.Expect(seen.HasAll(expectedAuditSubDirs...)).To(o.BeTrue())), and update the
lock log handling so result.Flakef(...) is only called when lockLog actually
indicates the “lock was still held” condition (gate the flake report on a
specific substring/marker in lockLog such as the known "lock was still held"
message or other reliable indicator rather than any presence of lock.log).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: f74e07b4-39e8-4915-b499-6b698bcc1dfc

📥 Commits

Reviewing files that changed from the base of the PR and between f1b4d90 and f3601f5.

📒 Files selected for processing (1)

test/extended/cli/mustgather.go

openshift-merge-bot · 2026-05-21T21:17:06Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

openshift-ci-robot · 2026-05-26T19:03:22Z

@xueqzhan: This pull request references Jira Issue OCPBUGS-86473, which is invalid:

expected the bug to target the "5.0.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary by CodeRabbit

Tests

Enhanced audit-logs must-gather test with clearer validation steps and consolidated coverage.

Now verifies apiserver audit directories were processed strictly sequentially and validates OAuth audit entries include audit IDs.

Removed redundant standalone tests whose checks are now covered in the consolidated flow.

Adjusted must-gather invocations in related tests to rely on default gather behavior while preserving output assertions.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

xueqzhan · 2026-05-26T19:07:05Z

/payload-aggregate periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview 4

openshift-ci · 2026-05-26T19:07:11Z

@xueqzhan: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command

periodic-ci-openshift-release-main-nightly-5.0-e2e-aws-ovn-upgrade-fips-rhcos9-techpreview

See details on https://pr-payload-tests.ci.openshift.org/runs/ci/1643bc80-5936-11f1-95e8-b00b9e9fdb68-0

openshift-merge-bot · 2026-05-26T19:24:09Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

Try to remove a crashing pod to test cpu situation in tp jobs

f1b4d90

openshift-ci Bot requested review from p0lyn0mial and sjenning May 20, 2026 17:30

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 20, 2026

coderabbitai Bot approved these changes May 20, 2026

View reviewed changes

openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label May 20, 2026

revert the change to see the differences

79d3d08

Refactor must gather test to minimize parallel load on the apiserver

f3601f5

coderabbitai Bot requested changes May 21, 2026

View reviewed changes

Comment thread test/extended/cli/mustgather.go Outdated

Comment thread test/extended/cli/mustgather.go Outdated

Refactor must gather audit log tests to avoid repeated audit download

de95822

xueqzhan changed the title ~~Try to remove a crashing pod to test cpu situation in tp jobs~~ OCPBUGS-86473: Consolidate audit log must-gather tests to reduce parallel downloads and master node CPU pressure May 26, 2026

openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels May 26, 2026

Conversation

xueqzhan commented May 20, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

openshift-merge-bot Bot commented May 20, 2026

Uh oh!

coderabbitai Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Pre-merge checks failed

❌ Failed checks (1 error, 3 warnings)

Uh oh!

openshift-ci Bot commented May 20, 2026

Uh oh!

xueqzhan commented May 20, 2026

Uh oh!

openshift-ci Bot commented May 20, 2026

Uh oh!

xueqzhan commented May 20, 2026

Uh oh!

openshift-ci Bot commented May 20, 2026

Uh oh!

openshift-merge-bot Bot commented May 20, 2026

Uh oh!

xueqzhan commented May 20, 2026

Uh oh!

openshift-ci Bot commented May 20, 2026

Uh oh!

xueqzhan commented May 20, 2026

Uh oh!

openshift-ci Bot commented May 20, 2026

Uh oh!

xueqzhan commented May 21, 2026

Uh oh!

openshift-ci Bot commented May 21, 2026

Uh oh!

xueqzhan commented May 21, 2026

Uh oh!

openshift-ci Bot commented May 21, 2026

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

openshift-merge-bot Bot commented May 21, 2026

Uh oh!

openshift-ci-robot commented May 26, 2026

Summary by CodeRabbit

Uh oh!

xueqzhan commented May 26, 2026

Uh oh!

openshift-ci Bot commented May 26, 2026

Uh oh!

openshift-merge-bot Bot commented May 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

xueqzhan commented May 20, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented May 20, 2026 •

edited

Loading