docs: propose severity floor for error-ratio alerts - #354
Merged
Conversation
Add an enhancement proposal for the remaining half of #326: gate ControllerReconcileErrorRatioCritical on an absolute error-rate floor (~1 error/minute) in addition to the 50% ratio, so a sparse controller failing a handful of permanently-failing objects fires a latched warning instead of paging on-call. The warning tier keeps no floor, so broken-but-idle controllers stay visible. Key changes: - Measured cases from #326 motivate the floor and its default value - Fixture plan pins both sides of the floor in promtool tests - Explicitly rejects the minimum-denominator guard with the #326 data
Contributor
Author
|
Runbook companion: #355 |
scotwells
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposal for the remaining half of #326:
ControllerReconcileErrorRatioCriticalpages with the same urgency for a controller retrying three orphaned objects as for one failing hundreds of reconciles of live traffic, because the ratio has no notion of volume.This enhancement gates the critical tier on an absolute error-rate floor (~1 error/minute over the 30m window) in addition to the 50% ratio. Below the floor the same condition fires the warning tier, which keeps no floor: a broken-but-idle controller stays visible and latched without paging. The doc derives the floor from the cases measured on #326, plans promtool fixtures pinning both sides of it, and records why a minimum-denominator guard was rejected.
Docs-only; the rule, fixture, and runbook changes follow once this is agreed.