[26.2] Improved cluster health summary

modules/get-started/pages/release-notes/redpanda.adoc

@@ -7,6 +7,11 @@ This topic includes new content added in version {page-component-version}. For a
 * xref:cloud-data-platform:get-started:whats-new-cloud.adoc[]
 * xref:cloud-data-platform:get-started:cloud-overview.adoc#redpanda-cloud-vs-self-managed-feature-compatibility[Redpanda Cloud vs Self-Managed feature compatibility]
 
+== Cluster health metrics
+
+Redpanda can now export a cluster health summary as Prometheus metrics, so you can alert on overall cluster health, such as brokers down, leaderless partitions, or a missing controller, directly from your metrics system instead of polling xref:reference:rpk/rpk-cluster/rpk-cluster-health.adoc[`rpk cluster health`]. The metrics mirror the output of `rpk cluster health` and are disabled by default.
+
+To enable them, set xref:reference:properties/cluster-properties.adoc#health_monitor_metrics_enabled[`health_monitor_metrics_enabled`] to `true` and restart your brokers. See xref:manage:monitoring.adoc#cluster-health[Cluster health] for the available metrics and query patterns.
 
 == Kafka 4.x client compatibility
 

modules/manage/partials/monitor-health.adoc

@@ -10,7 +10,7 @@ endif::[]
 
 This section provides guidelines and example queries using Redpanda's public metrics to optimize your system's performance and monitor its health.
 
-To help detect and mitigate anomalous system behaviors, capture baseline metrics of your healthy system at different stages (at start-up, under high load, in steady state) so you can set thresholds and alerts according to those baselines.
+To help detect and mitigate anomalous system behaviors, capture baseline metrics of your healthy system at different stages (at startup, under high load, in steady state) so you can set thresholds and alerts according to those baselines.
 
 [TIP]
 ====
@@ -428,6 +428,146 @@ Leaderless partitions can be caused by unresponsive brokers. When an alert on `r
 
 Redpanda's Raft implementation exchanges periodic status RPCs between a broker and its peers. The xref:reference:public-metrics-reference.adoc#redpanda_node_status_rpcs_timed_out[`redpanda_node_status_rpcs_timed_out`] gauge increases when a status RPC times out for a peer, which indicates that a peer may be unresponsive and may lead to problems with partition replication that Raft manages. Monitor for non-zero values of this gauge, and correlate it with any logged errors or changes in partition replication.
 
+ifndef::env-cloud[]
+[[cluster-health]]
+=== Cluster health
+
+A healthy cluster has all brokers responding, a leader for every partition, and an elected controller. Redpanda continuously assembles these signals into a cluster health summary. Redpanda can export this summary as a set of <<available-metrics,Prometheus metrics>>, so you can alert on overall cluster health from your metrics system instead of polling xref:reference:rpk/rpk-cluster/rpk-cluster-health.adoc[`rpk cluster health`] or the link:/api/doc/admin/operation/operation-get_cluster_health_overview[health overview Admin API].
+
+These metrics are disabled by default. When enabled, every broker periodically assembles its own view of the health overview and exposes it on the public metrics endpoint, prefixed with `redpanda_cluster_health_`. For most of these metrics, a value of `0` means healthy and any value greater than `0` indicates a problem.
+
+NOTE: Cluster health metrics require Redpanda version 26.2 or later.
+
+==== Enable cluster health metrics
+
+Set xref:reference:properties/cluster-properties.adoc#health_monitor_metrics_enabled[`health_monitor_metrics_enabled`] to `true`. This property requires a broker restart to take effect.
+
+ifndef::env-kubernetes[]
+[,bash]
+----
+rpk cluster config set health_monitor_metrics_enabled true
+----
+endif::[]
+
+ifdef::env-kubernetes[]
+[tabs]
+======
+Helm + Operator::
++
+--
+.`redpanda-cluster.yaml`
+[,yaml]
+----
+apiVersion: cluster.redpanda.com/v1alpha2
+kind: Redpanda
+metadata:
+  name: redpanda
+spec:
+  chartRef: {}
+  clusterSpec:
+    config:
+      cluster:
+        health_monitor_metrics_enabled: true
+----
+
+```bash
+kubectl apply -f redpanda-cluster.yaml --namespace <namespace>
+```
+
+--
+Helm::
++
+--
+[tabs]
+====
+--values::
++
+.`health-metrics.yaml`
+[,yaml]
+----
+config:
+  cluster:
+    health_monitor_metrics_enabled: true
+----
++
+```bash
+helm upgrade --install redpanda redpanda/redpanda --namespace <namespace> --create-namespace \
+--values health-metrics.yaml --reuse-values
+```
+
+--set::
++
+[,bash]
+----
+helm upgrade --install redpanda redpanda/redpanda \
+  --namespace <namespace> \
+  --create-namespace \
+  --set config.cluster.health_monitor_metrics_enabled=true
+----
+
+====
+--
+======
+endif::[]
+
+By default, each broker refreshes the metrics every 100 seconds. This interval is 10 times the value of xref:reference:properties/cluster-properties.adoc#health_monitor_max_metadata_age[`health_monitor_max_metadata_age`]. Enabling the metrics causes a modest increase in CPU usage and network traffic.
+
+==== Available metrics
+
+Redpanda emits two groups of metrics on the public metrics endpoint, both prefixed `redpanda_cluster_health_` and reported once per broker. The freshness metrics (`redpanda_cluster_health_metadata_age_seconds` and `redpanda_cluster_health_refreshes_total`) appear as soon as you enable the feature, so you can confirm that refreshes are running. The overview metrics, which are all the others, appear on a broker after it assembles a usable view of the cluster, typically a few seconds after startup.
+
+[cols="2a,1,3"]
+|===
+|Metric |Type |Description
+
+|xref:reference:public-metrics-reference.adoc#redpanda_cluster_health_unhealthy_reasons[`redpanda_cluster_health_unhealthy_reasons`] |gauge |Number of reasons the cluster is currently unhealthy. A value of `0` means healthy.
+|xref:reference:public-metrics-reference.adoc#redpanda_cluster_health_no_elected_controller[`redpanda_cluster_health_no_elected_controller`] |gauge |Reports `1` if no controller is elected, otherwise `0`.
+|xref:reference:public-metrics-reference.adoc#redpanda_cluster_health_nodes_down[`redpanda_cluster_health_nodes_down`] |gauge |Number of brokers not responding to liveness checks.
+|xref:reference:public-metrics-reference.adoc#redpanda_cluster_health_leaderless_partitions[`redpanda_cluster_health_leaderless_partitions`] |gauge |Number of partitions without a leader.
+|xref:reference:public-metrics-reference.adoc#redpanda_cluster_health_under_replicated_partitions[`redpanda_cluster_health_under_replicated_partitions`] |gauge |Number of partitions that are not fully replicated.
+|xref:reference:public-metrics-reference.adoc#redpanda_cluster_health_high_disk_usage_nodes[`redpanda_cluster_health_high_disk_usage_nodes`] |gauge |Number of brokers past the storage-space alert threshold.
+|xref:reference:public-metrics-reference.adoc#redpanda_cluster_health_no_health_report[`redpanda_cluster_health_no_health_report`] |gauge |Reports `1` if the most recent refresh attempt failed, otherwise `0`.
+|xref:reference:public-metrics-reference.adoc#redpanda_cluster_health_cluster_size[`redpanda_cluster_health_cluster_size`] |gauge |Number of known cluster members. Informational.
+|xref:reference:public-metrics-reference.adoc#redpanda_cluster_health_nodes_in_recovery_mode[`redpanda_cluster_health_nodes_in_recovery_mode`] |gauge |Number of brokers in recovery mode. Informational; does not affect the health verdict.
+|xref:reference:public-metrics-reference.adoc#redpanda_cluster_health_bytes_in_cloud_storage[`redpanda_cluster_health_bytes_in_cloud_storage`] |gauge |Bytes stored in object storage. Informational; `0` if Tiered Storage isn't enabled.
+|xref:reference:public-metrics-reference.adoc#redpanda_cluster_health_metadata_age_seconds[`redpanda_cluster_health_metadata_age_seconds`] |gauge |Seconds since the last successful refresh on the broker. Reports a very large value before the first refresh so that staleness alerts fire.
+|xref:reference:public-metrics-reference.adoc#redpanda_cluster_health_refreshes_total[`redpanda_cluster_health_refreshes_total`] |counter |Number of successful refreshes on the broker.
+|===
+
+NOTE: The informational metrics (`cluster_size`, `nodes_in_recovery_mode`, and `bytes_in_cloud_storage`) don't affect whether the cluster is considered healthy.
+
+==== Metric aggregation across brokers
+
+Every broker exposes its own copy of these metrics, so decide how to combine them.
+
+Aggregate across all brokers, for example with `max()`:
+
+[,promql]
+----
+max(redpanda_cluster_health_nodes_down)
+----
+
+Use `max()`, not `sum()`. Every broker reports its own copy, so `sum()` returns roughly the broker count times the real value. Aggregation is fine for coarse cluster-health alerting, but it's noisier during controller elections and broker restarts, and a stale or partitioned broker still contributes its last cached view.
+
+For a lower-noise signal, read the metric only from the current controller leader by joining against the Raft `leader_for` gauge for the controller partition:
+
+[,promql]
+----
+redpanda_cluster_health_unhealthy_reasons
+  and on(instance)
+    redpanda_raft_leader_for{namespace="redpanda",topic="controller",partition="0"} == 1
+----
+
+This join returns a value only while some broker is the controller leader. If no broker is the leader, for example when the cluster has no controller at all, the query returns an empty result, which an alert configured to fire above zero would not catch. Pair the join with a separate alert that fires when no broker reports itself as the controller leader:
+
+[,promql]
+----
+sum(redpanda_raft_leader_for{namespace="redpanda",topic="controller",partition="0"}) == 0
+----
+
+A controller election creates a brief window where no broker is the leader, and the joined series drops out. To avoid spurious alerts during these normal elections, configure the alerts to fire only after the metric has been non-zero for at least one minute (for example, with a Prometheus `for` duration of `1m`).
+endif::[]
+
 [[consumers]]
 === Consumer group lag
 

modules/reference/pages/public-metrics-reference.adoc

@@ -92,6 +92,104 @@ endif::[]
 
 ---
 
+ifndef::env-cloud[]
+=== redpanda_cluster_health_bytes_in_cloud_storage
+
+Bytes stored in object storage. Informational; `0` if Tiered Storage isn't enabled.
+
+*Type*: gauge
+
+---
+
+=== redpanda_cluster_health_cluster_size
+
+Number of known cluster members. Informational.
+
+*Type*: gauge
+
+---
+
+=== redpanda_cluster_health_high_disk_usage_nodes
+
+Number of brokers past the storage-space alert threshold.
+
+*Type*: gauge
+
+---
+
+=== redpanda_cluster_health_leaderless_partitions
+
+Number of partitions without a leader.
+
+*Type*: gauge
+
+---
+
+=== redpanda_cluster_health_metadata_age_seconds
+
+Seconds since the last successful refresh on the broker. Reports a very large value before the first refresh so that staleness alerts fire.
+
+*Type*: gauge
+
+---
+
+=== redpanda_cluster_health_no_elected_controller
+
+Reports `1` if no controller is elected, otherwise `0`.
+
+*Type*: gauge
+
+---
+
+=== redpanda_cluster_health_no_health_report
+
+Reports `1` if the most recent refresh attempt failed, otherwise `0`.
+
+*Type*: gauge
+
+---
+
+=== redpanda_cluster_health_nodes_down
+
+Number of brokers not responding to liveness checks.
+
+*Type*: gauge
+
+---
+
+=== redpanda_cluster_health_nodes_in_recovery_mode
+
+Number of brokers in recovery mode. Informational; does not affect the health verdict.
+
+*Type*: gauge
+
+---
+
+=== redpanda_cluster_health_refreshes_total
+
+Number of successful refreshes on the broker.
+
+*Type*: counter
+
+---
+
+=== redpanda_cluster_health_under_replicated_partitions
+
+Number of partitions that are not fully replicated.
+
+*Type*: gauge
+
+---
+
+=== redpanda_cluster_health_unhealthy_reasons
+
+Number of reasons the cluster is currently unhealthy. A value of `0` means healthy.
+
+*Type*: gauge
+
+---
+endif::[]
+
 === redpanda_cluster_latest_cluster_metadata_manifest_age
 
 The amount of time in seconds since the last time Redpanda uploaded metadata files to Tiered Storage for your cluster. A value of `0` indicates metadata has not yet been uploaded.

modules/reference/pages/rpk/rpk-cluster/rpk-cluster-health.adoc

@@ -13,6 +13,13 @@ A cluster is considered healthy when the following conditions are met:
 * All partitions have leaders
 * The cluster controller is present
 
+ifndef::env-cloud[]
+[NOTE]
+====
+The same health fields are also available as Prometheus metrics. See xref:manage:monitoring.adoc#cluster-health[Cluster health] for how to enable and query them.
+====
+endif::[]
+
 == Usage
 
 [,bash]