Skip to content

Commit 5e815a5

Browse files
committed
Add a skill for reviewing OpenShift ci results
We run periodic CI jobs every week for OpenShift/RHCOS compliance content. This commit adds a skill and script to parse the latest CI results from Prow and builds a compliance report from them. The idea is that we can use this to help keep the assertion files up-to-date in the project by reducing the overall hops required to get to the most recent CI runs and checking the results. The fetch_results.py script can actually be used standalone, which is useful for assessing gaps in testing coverage, or changing rules. But, it can also be fed into the skill so that an agent can help diagnose rules that have been refactored recently, and help update assertion files. Assisted-By: Claude
1 parent d37c4c4 commit 5e815a5

2 files changed

Lines changed: 565 additions & 0 deletions

File tree

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
---
2+
description: Check the latest periodic CI compliance results for a given profile and OCP version.
3+
---
4+
5+
Check the latest weekly compliance CI job results for a specific profile and OCP version, comparing actual scan results against the local assertion file.
6+
7+
Arguments: $ARGUMENTS
8+
9+
Expected arguments: `<product> <profile> <ocp-version> [arm]`
10+
11+
- `<product>` — The product to check. Either `ocp4` or `rhcos4`.
12+
- `<profile>` — The compliance profile to check. Examples: `cis`, `stig`, `moderate`, `high`, `e8`, `pci-dss`, `pci-dss-4-0`, `cis-node`, `stig-node`, `moderate-node`, `high-node`, `pci-dss-node`, `pci-dss-node-4-0`.
13+
- `<ocp-version>` — The OpenShift version (e.g., `4.16`, `4.18`, `4.21`).
14+
- `[arm]` — (Optional) Pass `arm` to check the ARM architecture job. Defaults to x86.
15+
16+
For example:
17+
- `check-ci ocp4 cis 4.18`
18+
- `check-ci ocp4 stig-node 4.16`
19+
- `check-ci rhcos4 e8 4.18 arm`
20+
21+
---
22+
23+
## Step 1: Run the fetch script
24+
25+
Run the Python script at `.claude/skills/ocp/check-ci/fetch_results.py` with the provided arguments:
26+
27+
```bash
28+
python3 .claude/skills/ocp/check-ci/fetch_results.py <product> <profile> <ocp-version> [arm]
29+
```
30+
31+
The script will:
32+
1. Load the matching assertion file from `tests/assertions/ocp4/`
33+
2. Query the Prow API for the latest periodic job (platform or node, depending on the profile)
34+
3. Download the initial (pre-remediation) and final (post-remediation) scan results
35+
4. Compare each rule in the assertion file against the actual scan results
36+
5. Print a summary
37+
38+
Note: The periodic CI jobs run ALL platform rules or ALL node rules for a given OCP version — not per-profile. The script filters the combined results down to only the rules listed in the profile's assertion file.
39+
40+
## Step 2: Present the results
41+
42+
Present the script output to the user. If there are mismatches or missing rules, highlight them and suggest next steps:
43+
44+
- **Mismatches** (expected status differs from actual): The assertion file may need updating, or the rule itself needs investigation. Show the rule name, expected status, and actual status.
45+
- **Missing from results** (rule listed in the assertion file but not found in the scan): The rule may have been removed, renamed, or excluded from the scan. Check whether the rule still exists in the repository.
46+
47+
## Step 3: Offer to update the assertion file
48+
49+
If there are mismatches, offer to update the assertion file at `tests/assertions/ocp4/<product>-<profile>-<version>.yml`. When updating:
50+
51+
- For **mismatches**: Update the `default_result` (and `result_after_remediation` if final results are available) to match the actual scan result.
52+
- For **missing rules**: Ask the user whether to remove them from the assertion file or keep them.
53+
54+
Do NOT automatically update the file — always show the proposed changes and wait for approval.
55+
56+
### Important: profile membership
57+
58+
The periodic CI jobs run every rule in the product, not just the rules for a specific profile. If the user wants to know whether rules are **missing from the assertion file** (i.e., rules that should be tested for this profile but aren't listed), that requires checking the profile and control files in this repository — not the scan results. The profile and control files define which rules belong to a profile:
59+
60+
- Profile files: `products/ocp4/profiles/<profile-name>.profile`
61+
- Control files referenced by profiles: `controls/` or `products/ocp4/controls/`
62+
63+
If the user asks about missing coverage, use those files to determine which rules should be in the assertion file.

0 commit comments

Comments
 (0)