Skip to content

feat(openshift): add mTLS for Prometheus metrics#3473

Open
jkhelil wants to merge 1 commit into
tektoncd:mainfrom
jkhelil:SRVKP-8172
Open

feat(openshift): add mTLS for Prometheus metrics#3473
jkhelil wants to merge 1 commit into
tektoncd:mainfrom
jkhelil:SRVKP-8172

Conversation

@jkhelil

@jkhelil jkhelil commented Jun 4, 2026

Copy link
Copy Markdown
Member

Secure component metrics endpoints with mutual TLS on OpenShift. Each reconciler syncs the Prometheus client CA bundle from kube-system/extension-apiserver-authentication into the component namespace as a metrics-client-ca ConfigMap.

OpenShift's serving-cert controller is triggered via annotation on each metrics Service to provision a per-component TLS Secret. Two new manifest transformers (InjectMetricsServingCert, ApplyMetricsTLS) wire the Secret and ConfigMap as volumes and inject METRICS_PROMETHEUS_TLS_* env vars so that the knative/pkg prometheus.Server enables mTLS with require client auth.

ServiceMonitor resources are updated with scheme: https, scrapeClass tls-client-certificate-auth, and tlsConfig.serverName. The existing UpdateServiceMonitorTargetNamespace transformer is extended to also patch the namespace segment inside serverName at runtime.

Relates-To: SRVKP-8172

Assisted-by: Claude Sonnet 4.6 (via Cursor)

Changes

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

See the contribution guide for more details.

Release Notes

Tekton Operator now enables mTLS between Prometheus and all Tekton
component metrics endpoints on OpenShift. Each component namespace
receives a synced client-CA ConfigMap and a serving-cert Secret so
that the knative/pkg prometheus.Server can enforce mutual TLS on its
/metrics endpoint.

@tekton-robot tekton-robot added the release-note-none Denotes a PR that doesnt merit a release note. label Jun 4, 2026
@tekton-robot tekton-robot requested review from khrm and pramodbindal June 4, 2026 13:03
@tekton-robot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please ask for approval from jkhelil after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@tekton-robot tekton-robot added the size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. label Jun 4, 2026
@aiagentforlab-source

Copy link
Copy Markdown

/kind feature

@tekton-robot tekton-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Jun 4, 2026
@tekton-robot tekton-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed release-note-none Denotes a PR that doesnt merit a release note. labels Jun 4, 2026
@jkhelil jkhelil changed the title feat(openshift): add mTLS for Prometheus metrics WI feat(openshift): add mTLS for Prometheus metrics Jun 5, 2026
@jkhelil jkhelil changed the title WI feat(openshift): add mTLS for Prometheus metrics WIP feat(openshift): add mTLS for Prometheus metrics Jun 5, 2026
@tekton-robot tekton-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 5, 2026
@jkhelil jkhelil changed the title WIP feat(openshift): add mTLS for Prometheus metrics feat(openshift): add mTLS for Prometheus metrics Jun 5, 2026
@tekton-robot tekton-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 5, 2026
@jkhelil jkhelil requested review from Copilot, enarha and infernus01 June 9, 2026 10:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds OpenShift-specific mutual TLS support for Prometheus scraping of Tekton component metrics endpoints by annotating metrics Services for serving cert issuance, mounting TLS materials into workloads, and updating ServiceMonitors to scrape over HTTPS.

Changes:

  • Introduces OpenShift manifest transformers to inject serving-cert annotations, mount metrics TLS Secret + client-CA ConfigMap, and set METRICS_PROMETHEUS_TLS_* env vars.
  • Updates OpenShift component reconcilers to apply the new metrics mTLS transformers to Deployments/StatefulSets and patch ServiceMonitors.
  • Extends ServiceMonitor namespace rewriting to also patch the namespace segment in tlsConfig.serverName.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 13 comments.

Show a summary per file
File Description
pkg/reconciler/openshift/tektontrigger/extension.go Applies metrics mTLS transformers to Triggers controller.
pkg/reconciler/openshift/tektonresult/extension.go Applies metrics mTLS transformers to Results watcher/API.
pkg/reconciler/openshift/tektonpruner/extension.go Applies metrics mTLS transformers to Pruner controller.
pkg/reconciler/openshift/tektonpipeline/extension.go Applies metrics mTLS transformers to Pipelines controllers and refines monitoring transforms.
pkg/reconciler/openshift/tektonconfig/extension.go Ensures the metrics client CA ConfigMap is synced into the target namespace.
pkg/reconciler/openshift/tektonchain/extension.go Applies metrics mTLS transformers to Chains metrics Service / controller.
pkg/reconciler/openshift/openshiftpipelinesascode/extension.go Applies metrics mTLS transformers and updates PAC ServiceMonitors for HTTPS scraping.
pkg/reconciler/openshift/common/transformer.go Extends ServiceMonitor namespace transformer to also update tlsConfig.serverName.
pkg/reconciler/openshift/common/metricstls.go New: common transformers/helpers for metrics mTLS (Service annotation/port rename, workload mounts/env, ServiceMonitor TLS wiring).
pkg/reconciler/openshift/common/metricsca.go New: sync logic for Prometheus client CA into component namespaces.
cmd/openshift/operator/kodata/openshift-monitoring/00-monitoring.yaml Updates Pipelines controller ServiceMonitor endpoint to HTTPS with TLS config.
cmd/openshift/operator/kodata/openshift-monitoring/01-trigger-monitoring.yaml Updates Triggers ServiceMonitor endpoint to HTTPS with TLS config.
cmd/openshift/operator/kodata/openshift-monitoring/02-chains-monitoring.yaml Updates Chains ServiceMonitor endpoint to HTTPS with TLS config.
cmd/openshift/operator/kodata/openshift-monitoring/03-pipeline-webhook-monitoring.yaml Explicitly sets webhook ServiceMonitor scheme to HTTP.
cmd/openshift/operator/kodata/openshift-monitoring/05-results-monitoring.yaml Updates Results ServiceMonitors endpoints to HTTPS with TLS config.
cmd/openshift/operator/kodata/openshift-monitoring/06-pruner-monitoring.yaml Updates Pruner ServiceMonitor endpoint to HTTPS with TLS config.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread pkg/reconciler/openshift/common/metricstls.go Outdated
Comment thread pkg/reconciler/openshift/common/metricstls.go Outdated
Comment thread pkg/reconciler/openshift/common/metricstls.go
Comment thread pkg/reconciler/openshift/common/transformer.go Outdated
Comment thread pkg/reconciler/openshift/common/transformer.go Outdated
Comment on lines +253 to +261
// Add the client-CA ConfigMap volume.
clientCAVol := corev1.Volume{
Name: metricsClientCAVolume,
VolumeSource: corev1.VolumeSource{
ConfigMap: &corev1.ConfigMapVolumeSource{
LocalObjectReference: corev1.LocalObjectReference{
Name: MetricsClientCAConfigMap,
},
},

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ordering guarantee is the fix. EnsureMetricsClientCA runs in TektonConfig.PreReconcile, which executes before any child component (TektonPipeline, TektonResult, etc.) is reconciled and before their pods are created. The ConfigMap is always present by the time pods are scheduled.

If the ConfigMap is deleted after install, TektonConfig's periodic reconciliation recreates it. In the small window between deletion and recreation, a pod restart will block — which is intentional. A pod without the client-CA bundle cannot perform mTLS client verification at all, so failing loudly (pod unschedulable) is preferable to starting silently with broken mTLS. This is consistent with how other mandatory ConfigMaps (e.g. CA bundles) are handled throughout the operator.

Comment thread pkg/reconciler/openshift/common/metricstls.go Outdated
Comment thread pkg/reconciler/openshift/tektonresult/extension.go Outdated
@jkhelil jkhelil force-pushed the SRVKP-8172 branch 2 times, most recently from 44e4a21 to 07c10f6 Compare June 15, 2026 09:30

@infernus01 infernus01 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@tekton-robot tekton-robot added the lgtm Indicates that a PR is ready to be merged. label Jun 16, 2026
Secure component metrics endpoints with mutual TLS on OpenShift.
Each reconciler syncs the Prometheus client CA bundle from
kube-system/extension-apiserver-authentication into the component
namespace as a metrics-client-ca ConfigMap.

OpenShift's serving-cert controller is triggered via annotation on
each metrics Service to provision a per-component TLS Secret.
Two new manifest transformers (InjectMetricsServingCert,
ApplyMetricsTLS) wire the Secret and ConfigMap as volumes and inject
METRICS_PROMETHEUS_TLS_* env vars so that the knative/pkg
prometheus.Server enables mTLS with require client auth.

ServiceMonitor resources are updated with scheme: https, scrapeClass
tls-client-certificate-auth, and tlsConfig.serverName. The existing
UpdateServiceMonitorTargetNamespace transformer is extended to also
patch the namespace segment inside serverName at runtime.

Relates-To: SRVKP-8172

Signed-off-by: Jawed khelil <jkhelil@redhat.com>
Assisted-by: Claude Sonnet 4.6 (via Cursor)
Co-authored-by: Cursor <cursoragent@cursor.com>
@tekton-robot tekton-robot removed the lgtm Indicates that a PR is ready to be merged. label Jun 16, 2026
@tekton-robot

Copy link
Copy Markdown
Contributor

New changes are detected. LGTM label has been removed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind/feature Categorizes issue or PR as related to a new feature. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants