*: fix metrics high cardinality risks#4581
Open
pinebit wants to merge 2 commits into
Open
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #4581 +/- ##
==========================================
+ Coverage 57.34% 57.38% +0.03%
==========================================
Files 245 245
Lines 33554 33556 +2
==========================================
+ Hits 19243 19256 +13
+ Misses 11875 11854 -21
- Partials 2436 2446 +10 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
pinebit
force-pushed
the
pinebit/metrics-cardinality-fix
branch
from
4f552cc to
56b0a62
Compare
KaloyanTanev
reviewed
Jul 6, 2026
| if state.ClusterHash != "" { | ||
| activeConnsCounter.WithLabelValues(state.Name, state.ClusterHash).Set(0) | ||
| } | ||
| deletePeerMetrics(state.Name) |
Collaborator
There was a problem hiding this comment.
Wouldn't that delete historical metrics for a peer if it goes offline for a minute?
Collaborator
Author
There was a problem hiding this comment.
Historical data is safe, deleting a series from the client process only removes it from the live /metrics output. Everything Prometheus already scraped stays in its TSDB, so dashboards querying past ranges are unaffected.
Delete relay_p2p_* series when peers disconnect, delete scheduler validator series no longer reported by the beacon node, fix the high-cardinality health check to compare series count instead of label count, and bound the User-Agent metric label value. ResetGaugeVec now matches exact leading label values and no longer panics on label values containing the separator. category: bug ticket: none Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pinebit
force-pushed
the
pinebit/metrics-cardinality-fix
branch
from
56b0a62 to
d3c35ff
Compare
Keep the PR minimal per review feedback: validator series cardinality is bounded by the cluster validator count, so the sweep is not needed to prevent unbounded growth. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes high-cardinality risks found while reviewing all Prometheus metrics against the code that populates them:
cmd/relay: delete allrelay_p2p_*series of a peer when it disconnects. The relay accepts connections from arbitrary peers and clusters (open gater) and never deleted series, so a long-running relay accumulated(peer, peer_cluster)series indefinitely.app/health: the high-cardinality check compared the number of label keys per series (~10) against100 * numValidators, so it never fired. It now compares the series count per metric family.core/validatorapi: truncate the untrusted User-Agent header to 128 bytes and sanitize it to valid UTF-8 before using it as a label value.app/promauto:ResetGaugeVecno longer panics on label values containing|andResetnow matches exact leading label values instead of substrings anywhere, which could delete series of an unrelated peer whose labels contained the given value as a substring.category: bug
ticket: none
🤖 Generated with Claude Code