Skip to content

fix: alloy instance duplication bug#162

Open
amdove wants to merge 1 commit intomainfrom
fix-alloy-blackbox-instances
Open

fix: alloy instance duplication bug#162
amdove wants to merge 1 commit intomainfrom
fix-alloy-blackbox-instances

Conversation

@amdove
Copy link
Contributor

@amdove amdove commented Mar 4, 2026

Description

Problem
Healthcheck alerts were firing multiple times for the same failing endpoint, with different instance labels corresponding to different Alloy pod node hostnames (e.g., ip-10-85-131-143.us-east-2.compute.internal and ip-10-85-21-223.us-east-2.compute.internal).

Root Cause
When Alloy runs prometheus.exporter.blackbox in clustering mode, each pod's exporter sets the instance label to its node hostname. This creates duplicate metric series that trigger separate alerts for the same health check failure.
This is a known Alloy limitation: grafana/alloy#1009

Solution
Added a prometheus.relabel "blackbox" component that normalizes the instance label to the static value "blackbox" before forwarding to Mimir. This ensures all blackbox metrics from all Alloy pods have identical labels, allowing Mimir to deduplicate them and preventing duplicate alerts.

Category of change

  • Bug fix (non-breaking change which fixes an issue)
  • Version upgrade (upgrading the version of a service or product)
  • New feature (non-breaking change which adds functionality)
  • Build: a code change that affects the build system or external dependencies
  • Performance: a code change that improves performance
  • Refactor: a code change that neither fixes a bug nor adds a feature
  • Documentation: documentation changes
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist

  • I have reviewed my own diff and added inline comments on lines I want reviewers to focus on or that I am uncertain about

@amdove amdove marked this pull request as ready for review March 4, 2026 00:10
@amdove amdove requested a review from a team as a code owner March 4, 2026 00:10
@amdove amdove requested a review from t-margheim March 4, 2026 00:10
Copy link
Collaborator

@stevenolen stevenolen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fantastic find!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants