k8s: add relabel config, relabeled rules, alerting health and AlertingRule CRD support

Add remaining k8s layer capabilities:
- AlertRelabelConfig CRUD operations for OpenShift AlertRelabelConfig CRs
- RelabeledRules: applies Prometheus relabel configs to derive effective
  alert rules with source/management labels
- AlertingHealth: route reachability checks for platform and user-workload
  Prometheus/Alertmanager endpoints
- AlertingRule CRUD operations for OpenShift AlertingRule CRs
- ExternalManagement label helpers

Also adds supporting leaf packages:
- pkg/managementlabels: management label constants and helpers
- pkg/classification: alert classification validation
- pkg/alert_rule: alert rule ID generation and parsing

Signed-off-by: avlitman <alitman@redhat.com>
Signed-off-by: Shirly Radco <sradco@redhat.com>
Signed-off-by: machadovilaca <machadovilaca@gmail.com>
Co-authored-by: Cursor AI Agent <cursor-ai@users.noreply.github.com>
Made-with: Cursor

Author: sradco

docs/classification-proposal-summary.md

@@ -0,0 +1,66 @@
+# Classification Overrides: Proposed Change Summary
+
+## What is changing
+
+Replace ConfigMap-based classification storage with labels stored directly on the rule resource, using AlertRelabelConfig (ARC) CRs only for rules that cannot be directly modified.
+
+## Before and After
+
+| Rule type | Management | Before (ConfigMap) | After (Labels + ARC) |
+|---|---|---|---|
+| User-defined PrometheusRule | Unmanaged | ConfigMap in plugin namespace | Labels directly on the PrometheusRule |
+| User-defined PrometheusRule | Operator-managed | ConfigMap in plugin namespace | ARC in `openshift-user-workload-monitoring` |
+| User-defined PrometheusRule | GitOps-managed | ConfigMap in plugin namespace | Block (user adds labels in Git) |
+| Platform via AlertingRule | Unmanaged | ConfigMap in plugin namespace | Labels directly on the AlertingRule |
+| Platform via AlertingRule | Operator-managed | ConfigMap in plugin namespace | ARC in `openshift-monitoring` |
+| Platform via AlertingRule | GitOps-managed | ConfigMap in plugin namespace | Block (user adds labels in Git) |
+| Platform without AlertingRule | Operator-managed | ConfigMap in plugin namespace | ARC in `openshift-monitoring` |
+
+### Storage comparison
+
+| Aspect | Before | After |
+|---|---|---|
+| Where classification is stored | ConfigMaps in plugin namespace | Labels on the rule itself (or ARC for operator-managed) |
+| New k8s interfaces | `ConfigMapInterface` (3 methods) | None |
+| New files | 5 files | 0 (reuses existing label update paths) |
+| Classification visibility | API response only | Everywhere (kubectl, Prometheus, Alertmanager) |
+| ARC model | N/A | Per-alert-rule (same as existing label changes) |
+| Provenance tracking | Implicit (separate data store) | `openshift_io_alert_rule_classification_managed_by` label |
+
+## Pros
+
+1. **Simpler implementation** -- removes 5 files and the ConfigMap subsystem; classification reuses the existing label update code paths
+2. **Classification visible beyond the API** -- labels on the rule are visible to kubectl, Prometheus, Alertmanager, and external tools (Grafana, PagerDuty)
+3. **No new k8s interfaces** -- no `ConfigMapInterface`, no ConfigMap mock in tests
+4. **Consistent with label changes** -- classification follows the exact same branching as other label updates (direct for editable rules, ARC for operator-managed, block for GitOps)
+5. **Clear provenance** -- `managed_by` label explicitly tracks whether the classification was set via the API
+6. **Zero migration** -- new feature, no existing data to migrate; existing clusters with custom ARCs or alerts are unaffected
+
+## Cons
+
+1. **`_from` (dynamic derivation) limited** -- `componentFrom`/`layerFrom` is only supported for ARC-based rules (operator-managed platform). Directly editable rules support only static component/layer. This covers the main use case (CVO-style platform alerts).
+2. **RBAC addition for user-workload ARCs** -- operator-managed user-defined PrometheusRules require a Role + RoleBinding for ARCs in `openshift-user-workload-monitoring` (one namespace, not cluster-wide)
+3. **GitOps-managed rules blocked** -- users must add classification labels in Git. ARC side-channel for GitOps rules can be added later if requested.
+4. **No provenance distinction for external label setters** -- if someone sets `openshift_io_alert_rule_component` via kubectl or their own ARC (without `managed_by`), the API treats it as "rule-defined." Functionally correct but the UI won't show it as a "user override."
+
+## RBAC changes needed
+
+A Role + RoleBinding in `openshift-user-workload-monitoring` for the monitoring-plugin ServiceAccount:
+
+```yaml
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  name: monitoring-plugin-arc-manager
+  namespace: openshift-user-workload-monitoring
+rules:
+  - apiGroups: ["monitoring.openshift.io"]
+    resources: ["alertrelabelconfigs"]
+    verbs: ["get", "list", "watch", "create", "update", "delete"]
+```
+
+This covers operator-managed user-defined PrometheusRules. All other rule types require no new RBAC.
+
+## Impact on existing clusters
+
+None. Classification is only activated when a user explicitly classifies a rule via the API. Existing ARCs, PrometheusRules, and alerts are unaffected.

pkg/alert_rule/alert_rule.go

@@ -0,0 +1,83 @@
+package alertrule
+
+import (
+	"crypto/sha256"
+	"encoding/base64"
+	"fmt"
+	"sort"
+	"strings"
+	"unicode/utf8"
+
+	"github.com/openshift/monitoring-plugin/pkg/classification"
+	"github.com/openshift/monitoring-plugin/pkg/managementlabels"
+	monitoringv1 "github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1"
+)
+
+func GetAlertingRuleId(alertRule *monitoringv1.Rule) string {
+	var name string
+	var kind string
+	if alertRule.Alert != "" {
+		name = alertRule.Alert
+		kind = "alert"
+	} else if alertRule.Record != "" {
+		name = alertRule.Record
+		kind = "record"
+	} else {
+		return ""
+	}
+
+	expr := normalizeExpr(alertRule.Expr.String())
+	forDuration := ""
+	if alertRule.For != nil {
+		forDuration = strings.TrimSpace(string(*alertRule.For))
+	}
+
+	labelsBlock := normalizedBusinessLabelsBlock(alertRule.Labels)
+
+	// Canonical payload is intentionally derived from rule spec (expr/for/labels) and identity (kind/name),
+	// and excludes annotations and openshift_io_* provenance/system labels.
+	canonicalPayload := strings.Join([]string{kind, name, expr, forDuration, labelsBlock}, "\n---\n")
+
+	// Generate SHA256 hash
+	hash := sha256.Sum256([]byte(canonicalPayload))
+
+	return "rid_" + base64.RawURLEncoding.EncodeToString(hash[:])
+}
+
+func normalizeExpr(expr string) string {
+	// Collapse consecutive whitespace so cosmetic formatting changes do not churn ids.
+	return strings.Join(strings.Fields(strings.TrimSpace(expr)), " ")
+}
+
+func normalizedBusinessLabelsBlock(in map[string]string) string {
+	if len(in) == 0 {
+		return ""
+	}
+
+	lines := make([]string, 0, len(in))
+	for k, v := range in {
+		key := strings.TrimSpace(k)
+		if key == "" {
+			continue
+		}
+		if strings.HasPrefix(key, "openshift_io_") || key == managementlabels.AlertNameLabel {
+			// Skip system labels
+			continue
+		}
+		if !classification.ValidatePromLabelName(key) {
+			continue
+		}
+		if v == "" {
+			// Align with specHash behavior: drop empty values
+			continue
+		}
+		if !utf8.ValidString(v) {
+			continue
+		}
+
+		lines = append(lines, fmt.Sprintf("%s=%s", key, v))
+	}
+
+	sort.Strings(lines)
+	return strings.Join(lines, "\n")
+}

pkg/k8s/alert_relabel_config.go

@@ -0,0 +1,99 @@
+package k8s
+
+import (
+	"context"
+	"fmt"
+
+	osmv1 "github.com/openshift/api/monitoring/v1"
+	osmv1client "github.com/openshift/client-go/monitoring/clientset/versioned"
+	"k8s.io/apimachinery/pkg/api/errors"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/fields"
+	"k8s.io/client-go/tools/cache"
+)
+
+type alertRelabelConfigManager struct {
+	clientset   *osmv1client.Clientset
+	arcInformer cache.SharedIndexInformer
+}
+
+func newAlertRelabelConfigManager(ctx context.Context, clientset *osmv1client.Clientset) (*alertRelabelConfigManager, error) {
+	arcInformer := cache.NewSharedIndexInformer(
+		alertRelabelConfigListWatchForAllNamespaces(clientset),
+		&osmv1.AlertRelabelConfig{},
+		0,
+		cache.Indexers{},
+	)
+
+	arcm := &alertRelabelConfigManager{
+		clientset:   clientset,
+		arcInformer: arcInformer,
+	}
+
+	go arcm.arcInformer.Run(ctx.Done())
+
+	cache.WaitForNamedCacheSync("AlertRelabelConfig informer", ctx.Done(),
+		arcm.arcInformer.HasSynced,
+	)
+
+	return arcm, nil
+}
+
+func alertRelabelConfigListWatchForAllNamespaces(clientset *osmv1client.Clientset) *cache.ListWatch {
+	return cache.NewListWatchFromClient(clientset.MonitoringV1().RESTClient(), "alertrelabelconfigs", "", fields.Everything())
+}
+
+func (arcm *alertRelabelConfigManager) List(ctx context.Context, namespace string) ([]osmv1.AlertRelabelConfig, error) {
+	arcs := arcm.arcInformer.GetStore().List()
+
+	alertRelabelConfigs := make([]osmv1.AlertRelabelConfig, 0, len(arcs))
+	for _, item := range arcs {
+		arc, ok := item.(*osmv1.AlertRelabelConfig)
+		if !ok {
+			continue
+		}
+		alertRelabelConfigs = append(alertRelabelConfigs, *arc)
+	}
+
+	return alertRelabelConfigs, nil
+}
+
+func (arcm *alertRelabelConfigManager) Get(ctx context.Context, namespace string, name string) (*osmv1.AlertRelabelConfig, bool, error) {
+	arc, err := arcm.clientset.MonitoringV1().AlertRelabelConfigs(namespace).Get(ctx, name, metav1.GetOptions{})
+	if err != nil {
+		if errors.IsNotFound(err) {
+			return nil, false, nil
+		}
+
+		return nil, false, err
+	}
+
+	return arc, true, nil
+}
+
+func (arcm *alertRelabelConfigManager) Create(ctx context.Context, arc osmv1.AlertRelabelConfig) (*osmv1.AlertRelabelConfig, error) {
+	created, err := arcm.clientset.MonitoringV1().AlertRelabelConfigs(arc.Namespace).Create(ctx, &arc, metav1.CreateOptions{})
+	if err != nil {
+		return nil, fmt.Errorf("failed to create AlertRelabelConfig %s/%s: %w", arc.Namespace, arc.Name, err)
+	}
+
+	return created, nil
+}
+
+func (arcm *alertRelabelConfigManager) Update(ctx context.Context, arc osmv1.AlertRelabelConfig) error {
+	_, err := arcm.clientset.MonitoringV1().AlertRelabelConfigs(arc.Namespace).Update(ctx, &arc, metav1.UpdateOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to update AlertRelabelConfig %s/%s: %w", arc.Namespace, arc.Name, err)
+	}
+
+	return nil
+}
+
+func (arcm *alertRelabelConfigManager) Delete(ctx context.Context, namespace string, name string) error {
+	err := arcm.clientset.MonitoringV1().AlertRelabelConfigs(namespace).Delete(ctx, name, metav1.DeleteOptions{})
+	if err != nil {
+		return fmt.Errorf("failed to delete AlertRelabelConfig %s: %w", name, err)
+	}
+
+	return nil
+}

pkg/k8s/alerting_health.go

@@ -0,0 +1,127 @@
+package k8s
+
+import (
+	"context"
+	"fmt"
+	"strings"
+	"sync"
+
+	"gopkg.in/yaml.v2"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/fields"
+	"k8s.io/client-go/kubernetes"
+	"k8s.io/client-go/tools/cache"
+)
+
+const (
+	clusterMonitoringConfigMap = "cluster-monitoring-config"
+	clusterMonitoringConfigKey = "config.yaml"
+)
+
+type clusterMonitoringConfig struct {
+	EnableUserWorkload bool `yaml:"enableUserWorkload"`
+}
+
+// clusterMonitoringConfigManager watches the cluster-monitoring-config ConfigMap
+// via an informer and caches the parsed enableUserWorkload value so that
+// AlertingHealth never needs a live API call.
+type clusterMonitoringConfigManager struct {
+	informer cache.SharedIndexInformer
+
+	mu      sync.RWMutex
+	enabled bool
+	err     error
+}
+
+func newClusterMonitoringConfigManager(ctx context.Context, clientset *kubernetes.Clientset) (*clusterMonitoringConfigManager, error) {
+	informer := cache.NewSharedIndexInformer(
+		cache.NewListWatchFromClient(
+			clientset.CoreV1().RESTClient(),
+			"configmaps",
+			ClusterMonitoringNamespace,
+			fields.OneTermEqualSelector("metadata.name", clusterMonitoringConfigMap),
+		),
+		&corev1.ConfigMap{},
+		0,
+		cache.Indexers{},
+	)
+
+	m := &clusterMonitoringConfigManager{
+		informer: informer,
+	}
+
+	_, err := informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
+		AddFunc: func(obj interface{}) {
+			cm, ok := obj.(*corev1.ConfigMap)
+			if !ok {
+				return
+			}
+			m.handleUpdate(cm)
+		},
+		UpdateFunc: func(_, newObj interface{}) {
+			cm, ok := newObj.(*corev1.ConfigMap)
+			if !ok {
+				return
+			}
+			m.handleUpdate(cm)
+		},
+		DeleteFunc: func(_ interface{}) {
+			m.mu.Lock()
+			defer m.mu.Unlock()
+			m.enabled = false
+			m.err = nil
+		},
+	})
+	if err != nil {
+		return nil, fmt.Errorf("failed to add event handler to cluster-monitoring-config informer: %w", err)
+	}
+
+	go informer.Run(ctx.Done())
+
+	cache.WaitForNamedCacheSync("ClusterMonitoringConfig informer", ctx.Done(),
+		informer.HasSynced,
+	)
+
+	return m, nil
+}
+
+func (m *clusterMonitoringConfigManager) handleUpdate(cm *corev1.ConfigMap) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+
+	raw, ok := cm.Data[clusterMonitoringConfigKey]
+	if !ok || strings.TrimSpace(raw) == "" {
+		m.enabled = false
+		m.err = nil
+		return
+	}
+
+	var cfg clusterMonitoringConfig
+	if err := yaml.Unmarshal([]byte(raw), &cfg); err != nil {
+		m.enabled = false
+		m.err = fmt.Errorf("parse cluster monitoring config.yaml: %w", err)
+		return
+	}
+
+	m.enabled = cfg.EnableUserWorkload
+	m.err = nil
+}
+
+func (m *clusterMonitoringConfigManager) userWorkloadEnabled() (bool, error) {
+	m.mu.RLock()
+	defer m.mu.RUnlock()
+	return m.enabled, m.err
+}
+
+// AlertingHealth returns alerting route health and UWM enablement status.
+func (c *client) AlertingHealth(ctx context.Context) (AlertingHealth, error) {
+	health := c.prometheusAlerts.alertingHealth(ctx)
+
+	enabled, err := c.clusterMonitoringConfig.userWorkloadEnabled()
+	if err != nil {
+		return health, fmt.Errorf("failed to determine user workload enablement: %w", err)
+	}
+	health.UserWorkloadEnabled = enabled
+
+	return health, nil
+}