MCO-2222: /automate-test skill for generating e2e tests from specifications#5864
Conversation
New Claude skill that generates MCO e2e test cases in test/extended-priv/ from test specifications provided as a text file. Includes code review learning from past PRs and follows all MCO test framework conventions. Future integration with Polarion (MCO-2220) and Jira (MCO-2221) planned. Signed-off-by: HarshwardhanPatil07 <harshpat@redhat.com>
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
✅ Files skipped from review due to trivial changes (1)
WalkthroughAdds two documentation files describing an AI-assisted workflow and command for generating Machine Config Operator (MCO) extended-privilege e2e tests, including input modes, test-generation conventions, review-pattern extraction, and build/verification steps. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~12 minutes Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 inconclusive)
✅ Passed checks (11 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Review rate limit: 0/1 reviews remaining, refill in 60 minutes.Comment |
|
/retitle MCO-2222: /automate-test skill for generating e2e tests from specifications |
openshift-ci
Bot
changed the title
|
@HarshwardhanPatil07: This pull request references MCO-2222 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/valid-reference
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
.claude/skills/mco-automate-test-workflow.md (1)
64-87: Remove unused gomega import from template.Line 69 imports gomega with alias
o, but this alias is never used in the new file structure template (lines 74-86). The template only uses ginkgo (g),exutil, andlogger.♻️ Proposed fix
import ( g "github.com/onsi/ginkgo/v2" - o "github.com/onsi/gomega" exutil "github.com/openshift/machine-config-operator/test/extended-priv/util" logger "github.com/openshift/machine-config-operator/test/extended-priv/util/logext" )Note: If gomega assertions are expected in test bodies, add it back with a comment explaining its usage, or add an example assertion in the template.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.claude/skills/mco-automate-test-workflow.md around lines 64 - 87, The gomega import is unused in the template: remove the alias import o "github.com/onsi/gomega" from the import block (leaving g, exutil, logger) or alternatively reintroduce gomega by adding a short example assertion in the test body (e.g., using o.Expect) with a comment; update the import list accordingly to eliminate the unused symbol `o`..claude/commands/automate-test.md (1)
15-20: Clarify initialization and error handling for the workflow.The workflow references
review_patterns_mco.md(line 17) but doesn't explain how this memory file is created or initialized on first run. Additionally, step 2 mentions fetching patterns from merged PRs viagh apibut provides no fallback behavior if GitHub CLI is unavailable or authentication fails. Step 5 also lacks guidance on handling build failures.📋 Suggested workflow clarifications
Add to step 2:
-2. Learn from previous code reviews — check memory (`review_patterns_mco.md`), fetch new patterns from merged PRs via `gh api` on `openshift/machine-config-operator`. +2. Learn from previous code reviews — check memory (`review_patterns_mco.md`, create empty if missing), fetch new patterns from merged PRs via `gh api` on `openshift/machine-config-operator`. Fall back to built-in conventions if `gh` is unavailable or fails.Add to step 5:
-5. Build with `make machine-config-tests-ext` and verify the test appears in listing. Fix any build errors. +5. Build with `make machine-config-tests-ext` and verify the test appears in listing. If build errors occur, analyze and fix iteratively.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In @.claude/commands/automate-test.md around lines 15 - 20, Clarify initialization and error handling in the workflow: document how review_patterns_mco.md is created/seeded on first run (e.g., generate an empty file or copy a template if missing) and where that happens in the workflow, add explicit fallback behavior around the gh api fetch (detect gh presence and auth, retry/backoff, and fall back to a cached local copy or skip fetch with a clear warning) and reference the fetch step that calls gh api, and expand step 5 (the make machine-config-tests-ext build) to include concrete failure handling instructions (capture build output, retry, run go test or make vet, and provide guidance on surfacing and fixing compile/test errors) so the workflow is deterministic and recovers sensibly from missing tools or build failures.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.claude/skills/mco-automate-test-workflow.md:
- Around line 24-34: Add concrete implementation details for the "Code Review
Learning" workflow: define the memory file format and location by specifying
review_patterns_mco.md as a YAML with a "Last Analyzed PR" numeric field and a
"Patterns" list (each entry includes pattern, source PR number, and timestamp);
add initialization logic to create review_patterns_mco.md with Last Analyzed PR:
0 when missing; replace the vague fetch step with explicit gh/GraphQL commands
(use gh api graphql with pullRequests(first:100, after:$cursor, states:MERGED,
orderBy:{field:UPDATED_AT,direction:DESC}) and iterate pagination with the
returned endCursor until null, filtering file paths for test/extended-priv/),
and use REST endpoints gh api /repos/:owner/:repo/pulls/{number}/comments and
/repos/:owner/:repo/pulls/{number}/reviews to fetch inline comments and PR-level
reviews; implement deduplication by normalizing pattern keys (e.g., lowercase
trimmed message + file/path context), grouping identical keys and keeping the
entry with the highest PR number or newest timestamp ("newer wins"), and write
the updated YAML back to review_patterns_mco.md after processing each batch.
---
Nitpick comments:
In @.claude/commands/automate-test.md:
- Around line 15-20: Clarify initialization and error handling in the workflow:
document how review_patterns_mco.md is created/seeded on first run (e.g.,
generate an empty file or copy a template if missing) and where that happens in
the workflow, add explicit fallback behavior around the gh api fetch (detect gh
presence and auth, retry/backoff, and fall back to a cached local copy or skip
fetch with a clear warning) and reference the fetch step that calls gh api, and
expand step 5 (the make machine-config-tests-ext build) to include concrete
failure handling instructions (capture build output, retry, run go test or make
vet, and provide guidance on surfacing and fixing compile/test errors) so the
workflow is deterministic and recovers sensibly from missing tools or build
failures.
In @.claude/skills/mco-automate-test-workflow.md:
- Around line 64-87: The gomega import is unused in the template: remove the
alias import o "github.com/onsi/gomega" from the import block (leaving g,
exutil, logger) or alternatively reintroduce gomega by adding a short example
assertion in the test body (e.g., using o.Expect) with a comment; update the
import list accordingly to eliminate the unused symbol `o`.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: d8884198-619f-4ce5-820a-01230b058a43
📒 Files selected for processing (2)
.claude/commands/automate-test.md.claude/skills/mco-automate-test-workflow.md
|
/hold |
openshift-ci
Bot
added
the
do-not-merge/hold
|
/unhold |
openshift-ci
Bot
removed
the
do-not-merge/hold
|
/verified by @ptalgulk01 here |
openshift-ci-robot
added
the
verified
|
@HarshwardhanPatil07: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Also I think at the end we should try to verify each test step is covered so that we dont miss any step in between. |
Address reviewer feedback: - Convention openshift#14: skip tests on SNO/Compact when multiple nodes needed - Generation rule for IsCompactOrSNOCluster skip at test body start - Post-generation verification to ensure 1:1 spec step coverage - Helper functions must be appended at end of file, not inserted between existing ones Signed-off-by: HarshwardhanPatil07 <harshpat@redhat.com>
openshift-ci-robot
removed
the
verified
|
@HarshwardhanPatil07: This pull request references MCO-2222 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@HarshwardhanPatil07: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/verified by @ptalgulk01 here |
openshift-ci-robot
added
the
verified
|
@HarshwardhanPatil07: This PR has been marked as verified by DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
Pipeline controller notification No second-stage tests were triggered for this PR. This can happen when:
Use |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: HarshwardhanPatil07, isabella-janssen The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
…est-skill MCO-2222: /automate-test skill for generating e2e tests from specifications
4 participants
Future integration with Polarion (MCO-2220) and Jira (MCO-2221) planned.
- What I did
New Claude skill that generates MCO e2e test cases in test/extended-priv/ from test specifications provided as a text file. Includes code review learning from past PRs and follows all MCO test framework conventions.
The skill includes:
- How to verify it
- Description for the changelog
Add /automate-test Claude skill for generating e2e tests from text file specifications.
Summary by CodeRabbit