From 846af14882dc88009d012229a44a7d309f4fd0a0 Mon Sep 17 00:00:00 2001 From: Evan Vetere Date: Wed, 12 Aug 2026 12:48:56 -0400 Subject: [PATCH] docs: note severity floor in controller runbook Document the absolute error-rate floor on the critical reconcile-error alert proposed for #326. Critical now requires both a >50% error ratio and roughly one error per minute sustained; the warning tier keeps no floor. A sparse controller failing a handful of orphaned objects pins its ratio at 100% but latches at warning instead of paging on-call. Key changes: - Critical "Meaning" gains the error-rate floor condition - Explain why a sparse permanently-failing controller warns, not pages - Point latched 100%-ratio warnings at the existing orphan guidance --- docs/runbooks/controller-health.md | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/docs/runbooks/controller-health.md b/docs/runbooks/controller-health.md index a6e48520..bef3bdf4 100644 --- a/docs/runbooks/controller-health.md +++ b/docs/runbooks/controller-health.md @@ -81,6 +81,11 @@ Common error classes: attempts are returning errors, sustained for 15 minutes. The reconciler is struggling to write its objects and some intent is not reaching its target. +This tier has no minimum error-rate floor. A sparse controller failing all of +a tiny workload — a few permanently-failing objects and nothing else in its +queue — fires here at a 100% ratio and latches, rather than escalating to the +critical tier (see below). + **Impact.** Whatever the controller manages is not receiving updates. The last successfully written state continues to run. @@ -96,8 +101,19 @@ writes succeed again, though the alert itself holds for up to 30 minutes longer ## ControllerReconcileErrorRatioCritical **Meaning (critical).** More than 50% of the named controller's reconcile -attempts are failing, sustained for 10 minutes. The reconciler has effectively -stopped applying changes. +attempts are failing *and* its error rate exceeds roughly one error per minute, +both over the 30-minute window and sustained for 10 minutes. The reconciler has +effectively stopped applying changes to a workload large enough to page for. + +**A sparse, permanently-failing controller does not reach this tier — by +design.** A controller retrying a handful of orphaned objects at +controller-runtime's capped backoff pins its ratio at 100% but stays far below +the error-rate floor, so it fires `ControllerReconcileErrorRatioHigh` and +latches there instead of paging on-call +([datum-cloud/compute#194](https://github.com/datum-cloud/compute/issues/194) +is the canonical case). Treat a latched warning with a ~100% ratio and a tiny +error rate as likely orphaned data — see the orphan guidance under +[Shared diagnosis](#shared-diagnosis). **Impact.** Treat as an active outage for anything this controller programs. No updates are being applied; consumers see whatever was in place before the errors