Skip to content

WIP: Ensure Platform Prometheus targets are not scraped with insecure_skip_verify#31373

Open
machine424 wants to merge 1 commit into
openshift:mainfrom
machine424:skipCAA
Open

WIP: Ensure Platform Prometheus targets are not scraped with insecure_skip_verify#31373
machine424 wants to merge 1 commit into
openshift:mainfrom
machine424:skipCAA

Conversation

@machine424

@machine424 machine424 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Summary by CodeRabbit

  • Tests
    • Added coverage to verify Prometheus scrape settings do not disable TLS certificate verification.
    • The new check inspects the configured scrape jobs and fails if any job is set to skip TLS CA verification.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 9, 2026
@machine424

Copy link
Copy Markdown
Contributor Author

/pipeline required

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 9, 2026
@coderabbitai

coderabbitai Bot commented Jul 9, 2026

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 7a1e0961-e2d2-4421-bd4f-95c41ed8b7fe

📥 Commits

Reviewing files that changed from the base of the PR and between c9e60d5 and 3e2b3ca.

📒 Files selected for processing (1)
  • test/extended/prometheus/prometheus.go

Walkthrough

Adds a new Ginkgo test in the Prometheus extended test suite that fetches scrape configuration from the Prometheus config API endpoint, parses the embedded YAML, and fails if any scrape config disables TLS CA verification via insecure_skip_verify.

Changes

Prometheus TLS Verification Test

Layer / File(s) Summary
Add TLS CA verification test
test/extended/prometheus/prometheus.go
New test fetches /api/v1/status/config using existing auth, parses YAML, and asserts no scrape config sets tls_config.insecure_skip_verify: true.

Estimated code review effort: 1 (Trivial) | ~5 minutes

🚥 Pre-merge checks | ✅ 13 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Test Structure And Quality ⚠️ Warning The new Ginkgo It block has several bare Expect(err).NotTo(HaveOccurred()) checks with no diagnostic context, which violates the assertion-message guideline. Add meaningful messages to the new error assertions (fetch, unmarshal, sanity check) so failures identify which step broke.
Microshift Test Compatibility ⚠️ Warning The new test depends on openshift-monitoring/prometheus-k8s via PrometheusRouteURL and service-account token helpers, and it has no MicroShift label/guard. Add a [Skipped:MicroShift] label or an exutil.IsMicroShiftCluster() g.Skip() guard around this suite/test if it is not meant to run on MicroShift.
✅ Passed checks (13 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: preventing Prometheus scrapes from using insecure_skip_verify.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed The new Ginkgo spec title is a fixed string, and no Ginkgo title calls in the file use fmt.Sprintf or concatenation.
Single Node Openshift (Sno) Test Compatibility ✅ Passed The new test only reads Prometheus config and checks TLS flags; it doesn't count nodes, schedule pods, or assume HA/multi-node behavior.
Topology-Aware Scheduling Compatibility ✅ Passed Only a read-only Prometheus config test was added; no manifests, controllers, or scheduling constraints were introduced or modified.
Ote Binary Stdout Contract ✅ Passed The PR only adds an It block; no stdout writes were added in main/init/suite setup, and the new code uses only test-case-scoped logic.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed New It only reads Prometheus’s cluster route and parses YAML; no hardcoded IPv4, host:port interpolation, or external/public connectivity.
No-Weak-Crypto ✅ Passed The PR only adds a test asserting Prometheus scrape configs do not use insecure_skip_verify; no weak crypto, custom crypto, or secret comparisons were introduced.
Container-Privileges ✅ Passed Only a Go test was added; no container/K8s manifests or securityContext fields were changed, and the new test just reads Prometheus config.
No-Sensitive-Data-In-Logs ✅ Passed The new test only logs progress and scrape-config job names; it doesn’t print the fetched YAML, bearer token, or secret values.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot requested review from simonpasquier and slashpai July 9, 2026 11:58
@openshift-ci

openshift-ci Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: machine424

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
test/extended/prometheus/prometheus.go (1)

146-169: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Brittle substring match on TLS error text.

strings.Contains(curlErrMsg, "certificate required") couples the etcd pass condition to the exact wording emitted by whatever TLS backend curl is built against. If the backend (OpenSSL vs. others) or curl version phrases the mTLS handshake alert differently, this branch silently falls through to the retry/timeout path. Consider matching a broader signal (e.g. statusCode == 0 && err != nil for the etcd/etcd job) or documenting the assumed curl/TLS backend.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/extended/prometheus/prometheus.go` around lines 146 - 169, The etcd mTLS
exception in the polling closure is too dependent on the exact curl error text,
so update the `wait.PollUntilContextTimeout` callback to avoid
`strings.Contains(curlErrMsg, "certificate required")` as the deciding signal.
In the `targetNs == "openshift-etcd"` and `targetJob == "etcd"` branch, prefer a
broader condition based on the existing `statusCode == 0` and `err != nil`
checks (or otherwise make the TLS assumption explicit in the test), so the
`scrapeErr` logic in this block remains stable across curl/TLS backend
differences.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@test/extended/prometheus/prometheus.go`:
- Around line 146-169: The etcd mTLS exception in the polling closure is too
dependent on the exact curl error text, so update the
`wait.PollUntilContextTimeout` callback to avoid `strings.Contains(curlErrMsg,
"certificate required")` as the deciding signal. In the `targetNs ==
"openshift-etcd"` and `targetJob == "etcd"` branch, prefer a broader condition
based on the existing `statusCode == 0` and `err != nil` checks (or otherwise
make the TLS assumption explicit in the test), so the `scrapeErr` logic in this
block remains stable across curl/TLS backend differences.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: b47b2f75-8acc-4c30-8669-d0dc97e770d5

📥 Commits

Reviewing files that changed from the base of the PR and between 965e8a0 and c9e60d5.

📒 Files selected for processing (2)
  • test/extended/prometheus/prometheus.go
  • test/extended/util/prometheus/helpers.go

@openshift-ci openshift-ci Bot added the ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review label Jul 9, 2026
@machine424

Copy link
Copy Markdown
Contributor Author

/pipeline required

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-ci

openshift-ci Bot commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

@machine424: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gcp-ovn 3e2b3ca link true /test e2e-gcp-ovn
ci/prow/e2e-aws-ovn-fips 3e2b3ca link true /test e2e-aws-ovn-fips
ci/prow/e2e-vsphere-ovn-upi 3e2b3ca link true /test e2e-vsphere-ovn-upi
ci/prow/e2e-aws-ovn-serial-1of2 3e2b3ca link true /test e2e-aws-ovn-serial-1of2
ci/prow/e2e-vsphere-ovn 3e2b3ca link true /test e2e-vsphere-ovn
ci/prow/e2e-metal-ipi-ovn-ipv6 3e2b3ca link true /test e2e-metal-ipi-ovn-ipv6
ci/prow/e2e-aws-ovn-serial-2of2 3e2b3ca link true /test e2e-aws-ovn-serial-2of2

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-trt

openshift-trt Bot commented Jul 9, 2026

Copy link
Copy Markdown

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New Test Risks for sha: 3e2b3ca

Job Name New Test Risk
pull-ci-openshift-origin-main-e2e-aws-ovn-fips High - "[sig-instrumentation][Late] Platform Prometheus targets should not skip TLS CA verification [Suite:openshift/conformance/parallel]" is a new test that failed 1 time(s) against the current commit
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-1of2 High - "[sig-instrumentation][Late] Platform Prometheus targets should not skip TLS CA verification [Suite:openshift/conformance/parallel]" is a new test that failed 1 time(s) against the current commit
pull-ci-openshift-origin-main-e2e-aws-ovn-serial-2of2 High - "[sig-instrumentation][Late] Platform Prometheus targets should not skip TLS CA verification [Suite:openshift/conformance/parallel]" is a new test that failed 1 time(s) against the current commit
pull-ci-openshift-origin-main-e2e-gcp-ovn High - "[sig-instrumentation][Late] Platform Prometheus targets should not skip TLS CA verification [Suite:openshift/conformance/parallel]" is a new test that failed 1 time(s) against the current commit
pull-ci-openshift-origin-main-e2e-metal-ipi-ovn-ipv6 High - "[sig-instrumentation][Late] Platform Prometheus targets should not skip TLS CA verification [Suite:openshift/conformance/parallel]" is a new test that failed 1 time(s) against the current commit
pull-ci-openshift-origin-main-e2e-vsphere-ovn High - "[sig-instrumentation][Late] Platform Prometheus targets should not skip TLS CA verification [Suite:openshift/conformance/parallel]" is a new test that failed 1 time(s) against the current commit
pull-ci-openshift-origin-main-e2e-vsphere-ovn-upi High - "[sig-instrumentation][Late] Platform Prometheus targets should not skip TLS CA verification [Suite:openshift/conformance/parallel]" is a new test that failed 1 time(s) against the current commit

New tests seen in this PR at sha: 3e2b3ca

  • "[sig-instrumentation][Late] Platform Prometheus targets should not skip TLS CA verification [Suite:openshift/conformance/parallel]" [Total: 7, Pass: 0, Fail: 7, Flake: 0]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. ready-for-human-review Indicates a PR has been reviewed by automated tools and is ready for human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant