[DO-NOT-MERGE] Regression tests for quay 3.12.19

[nindsimonv]

PR intended to perform regression tests for quay 3.12.19

Summary by CodeRabbit

This is a [DO-NOT-MERGE] pull request that adds comprehensive CI-Operator regression test configurations for Quay 3.12.19 across multiple OpenShift versions (OCP 4.12 through 4.16).

What's being added:

The PR introduces 16 new CI configuration files under ci-operator/config/quay/quay-tests/ that establish a regression test matrix for Quay 3.12.19. For each OpenShift version (4.12, 4.13, 4.14, 4.15, and 4.16), three test variants are configured:

1. Operator regression tests - Validate the Quay operator deployment and functionality with dedicated catalog source setup and ODF storage provisioning
2. General regression tests - Include ACS violations checks, E2E tests with virtual builder setups, and operator tests with multiple test case filters
3. Upgrade regression tests - Test upgrading Quay from version 3.12 to 3.12.19 with variations across different cluster configurations (e.g., catalog source only, ODF variations)

Key configuration details:

Each test variant specifies:

• Base OpenShift images and build root settings aligned with the target OCP version
• Nightly OCP release candidates for testing
• Consistent resource requests/limits (200m CPU request, 500Mi-700Mi memory request, 8Gi memory limit)
• AWS-based cluster profiles (aws-quay-qe) for test execution
• Scheduled cron jobs (yearly or daily) with specific environment variables, operator channels (stable-3.12), and ODF operator configurations
• Pinned Quay index image digests to ensure version consistency

This establishes a repeatable regression testing framework to validate Quay 3.12.19 compatibility across the current and upcoming OpenShift versions.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 79ce7cab-db3f-4232-84eb-bcb4e2a41273

📥 Commits

Reviewing files that changed from the base of the PR and between d865dc5 and 01f787e.

⛔ Files ignored due to path filters (1)

• ci-operator/jobs/quay/quay-tests/quay-quay-tests-master-periodics.yaml is excluded by !ci-operator/jobs/**

📒 Files selected for processing (15)

• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-412-quay-operator-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-412-quay-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-412-quay-upgrade-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-413-quay-operator-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-413-quay-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-413-quay-upgrade-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-414-quay-operator-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-414-quay-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-414-quay-upgrade-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-415-quay-operator-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-415-quay-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-415-quay-upgrade-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-416-quay-operator-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-416-quay-regression.yaml
• ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-416-quay-upgrade-regression.yaml

---

Walkthrough

Fifteen new CI-operator YAML configurations are added under ci-operator/config/quay/quay-tests/, covering three test categories—operator-regression (allns), full-regression (ACS + E2E + operator), and upgrade-regression—for each of OCP versions 4.12 through 4.16. Each file defines base images, a nightly release target, global resource defaults, scheduled yearly test jobs with pinned Quay image digests and cucushift rehearsal workflows, and generated metadata.

Changes

Quay CI Regression Configurations (OCP 4.12–4.16)

Layer / File(s)	Summary
Operator-regression configs (allns, OCP 4.12–4.16) ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-412-quay-operator-regression.yaml, ...ocp-413-quay-operator-regression.yaml, ...ocp-414-quay-operator-regression.yaml, ...ocp-415-quay-operator-regression.yaml, ...ocp-416-quay-operator-regression.yaml	Five new files, one per OCP version, each defining builder/tools base images, a nightly OCP release target, global resource defaults (8Gi memory limit, 700Mi/200m requests), and a single yearly scheduled Quay-Allns-High operator test job with a pinned QUAY index image digest and three cucushift step refs (catalogsource, ODF storage, operator test).
Full-regression configs (ACS + E2E + operator, OCP 4.12–4.16) ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-412-quay-regression.yaml, ...ocp-413-quay-regression.yaml, ...ocp-414-quay-regression.yaml, ...ocp-415-quay-regression.yaml, ...ocp-416-quay-regression.yaml	Five new files, one per OCP version, each pinning cli and upi-installer base images to the corresponding OCP version, a golang-1.18 build root, and two to four scheduled test entries: an ACS violations check (daily), a Quay E2E virtual builder test (yearly), and a single-namespace operator test (yearly) with Quay-High|Quay-Medium filter and 8h timeout.
Upgrade-regression configs (three upgrade workflows, OCP 4.12–4.16) ci-operator/config/quay/quay-tests/quay-quay-tests-master__ocp-412-quay-upgrade-regression.yaml, ...ocp-413-quay-upgrade-regression.yaml, ...ocp-414-quay-upgrade-regression.yaml, ...ocp-415-quay-upgrade-regression.yaml, ...ocp-416-quay-upgrade-regression.yaml	Five new files, one per OCP version, each defining three yearly upgrade workflows (quay312-ocpNNN-upgrade, cso-quay312-ocpNNN-upgrade, qbo-quay312-ocpNNN-upgrade) with distinct compute sizing, ODF/Quay operator channels, upgrade testcase names, Quay version 3.12.19, and ordered test step refs using the cucushift-installer-rehearse-aws-ipi workflow.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 15

✅ Passed checks (15 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately reflects the primary purpose of the changeset: adding regression test configurations for Quay 3.12.19 across multiple OCP versions.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The PR adds only CI operator configuration YAML files with static, deterministic test names (e.g., quay312-ocp412-operator-test-allns). It contains no Ginkgo test code, so the check for Ginkgo test...
Test Structure And Quality	✅ Passed	PR contains no Ginkgo test code. The check for Ginkgo test structure (It blocks, BeforeEach/AfterEach, timeouts, assertion messages) is not applicable to this PR which adds CI-Operator YAML configs...
Microshift Test Compatibility	✅ Passed	PR adds only CI operator configuration YAML files, not Ginkgo e2e test code. No new test definitions (It(), Describe(), etc.) are added.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PR adds only CI-Operator YAML config files for Quay regression tests, not Ginkgo e2e test code. Check requires actual Ginkgo tests (It(), Describe(), etc.) and is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	PR adds only CI-Operator test configuration files for Quay regression testing, not deployment manifests, operator code, or controllers. Custom check is not applicable to CI configuration files.
Ote Binary Stdout Contract	✅ Passed	This PR contains only CI operator YAML configuration files (15 additions), not Go source code. The OTE Binary Stdout Contract check applies only to Go code; it is not applicable to CI configuration...
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	PR adds only CI-Operator YAML configuration files, not Ginkgo e2e tests. The check for IPv6/disconnected network compatibility is not applicable as no new Ginkgo test code is being added.
No-Weak-Crypto	✅ Passed	All 15 files are YAML CI-Operator configurations with no weak crypto, custom crypto, or insecure secret comparison implementations.
Container-Privileges	✅ Passed	No privilege escalation flags (privileged: true, hostPID, hostNetwork, hostIPC, SYS_ADMIN, allowPrivilegeEscalation, or root runAsUser) found in any of the 15 CI-Operator configuration files added...
No-Sensitive-Data-In-Logs	✅ Passed	No sensitive data (passwords, tokens, API keys, PII, session IDs, internal hostnames, or customer data) found in the 15 newly added CI-Operator YAML configuration files for Quay 3.12.19 regression...

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-merge-bot]

@nindsimonv, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

couldn't prepare candidate: couldn't checkout base SHA 7d9315a75a388bc78bd756911ea03bc647b5a156: error checking out "7d9315a75a388bc78bd756911ea03bc647b5a156": exit status 128 fatal: unable to read tree (7d9315a75a388bc78bd756911ea03bc647b5a156)

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nindsimonv
Once this PR has been reviewed and has the lgtm label, please assign lizhang19817 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• ci-operator/config/quay/quay-tests/OWNERS
• ci-operator/jobs/quay/quay-tests/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@nindsimonv: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[nindsimonv]

/pj-rehearse periodic-ci-quay-quay-tests-master-ocp-416-quay-operator-regression-quay312-ocp416-operator-test-allns periodic-ci-quay-quay-tests-master-ocp-416-quay-upgrade-regression-qbo-quay312-ocp416-upgrade periodic-ci-quay-quay-tests-master-ocp-416-quay-regression-quay-acs-violations-check-quay312 periodic-ci-quay-quay-tests-master-ocp-416-quay-upgrade-regression-quay312-ocp416-upgrade periodic-ci-quay-quay-tests-master-ocp-416-quay-upgrade-regression-cso-quay312-ocp416-upgrade periodic-ci-quay-quay-tests-master-ocp-416-quay-regression-quay-e2e-tests-quay312-ocp416-virtual-builder periodic-ci-quay-quay-tests-master-ocp-416-quay-regression-quay312-ocp416-operator-test-singlens

[openshift-merge-bot]

@nindsimonv: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[nindsimonv]

/pj-rehearse periodic-ci-quay-quay-tests-master-ocp-416-quay-regression-quay-e2e-tests-quay312-ocp416-virtual-builder periodic-ci-quay-quay-tests-master-ocp-416-quay-regression-quay312-ocp416-operator-test-singlens periodic-ci-quay-quay-tests-master-ocp-416-quay-upgrade-regression-qbo-quay312-ocp416-upgrade periodic-ci-quay-quay-tests-master-ocp-416-quay-operator-regression-quay312-ocp416-operator-test-allns

[openshift-merge-bot]

@nindsimonv: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

[nindsimonv]

/pj-rehearse periodic-ci-quay-quay-tests-master-ocp-416-quay-operator-regression-quay312-ocp416-operator-test-allns periodic-ci-quay-quay-tests-master-ocp-416-quay-regression-quay312-ocp416-operator-test-singlens periodic-ci-quay-quay-tests-master-ocp-416-quay-upgrade-regression-qbo-quay312-ocp416-upgrade

[openshift-merge-bot]

@nindsimonv: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.