[factory] Discovery overload backoff ignores the advertised Retry-After and discards a whole sweep on one 429

[khaliqgant]

Summary

relayfile answers an overloaded workspace DO with a 429 in milliseconds, carrying Retry-After: 5. Factory responds by sleeping up to 300 seconds, latching on a single 429 from anywhere in a sweep, discarding the entire sweep even if everything else succeeded, and clearing the ratchet only after a fully clean sweep. The result is that Factory recovers from a transient dependency blip far more slowly than the dependency asks it to, and probes for recovery only once every five minutes.

Observed in production 2026-08-20 while the cloud Factory was unable to dispatch.

Measured

lastError     : "workspace durable object is busy; retry after the advertised delay"
lastDurationMs: 307050  (~300s cap + work)
                211729  (~160s -> 300s region)
                025775  (~20s rung)
consecutiveFailures: 3 -> 5 -> 8

Those durations are not time spent waiting on relayfile. relayfile returns the 429 immediately and its Worker has no retry loop on this path — the 429 passes straight through. The minutes are Factory's own sleep.

Mechanism

• discoveryOverloadBackoffMs = max(retryAfterSeconds * 1000, 5000 * 2^(consecutiveOverloads - 1)), capped at DISCOVERY_OVERLOAD_BACKOFF_MAX_MS = 5 * 60_000.
• Ladder: 5s, 10s, 20s, 40s, 80s, 160s, 300s and then pinned at the cap.
• consecutiveOverloads is durable and resets to 0 only inside completeDiscoverySweep — i.e. only after a fully successful sweep.
• #discoveryOverloadError ??= error (src/orchestrator/factory.ts:3278) latches on the first 429 from any relayfile call in the sweep.
• if (this.#discoveryOverloadError) throw (src/orchestrator/factory.ts:2136) then discards the whole sweep, including work that already succeeded.

Net effect: the dependency asks for a 5-second retry; Factory waits up to 300. Once pinned at the cap it probes the workspace once per five minutes, so it is slow to notice that the dependency recovered — and a single unlucky probe re-arms another five minutes. A fix deployed upstream at 13:35:33Z was still not reflected in Factory's health seven minutes later purely because of this.

Why it matters beyond slow recovery

consecutiveFailures crossing the threshold marks readinessReconcile degraded, which is the signal operators and monitors read. So a transient upstream blip presents as a sustained Factory outage, and the ratchet keeps it there long after the cause has cleared. It also interacts badly with long sweeps: the longer a sweep is, the more likely it contains at least one transient 429, and one is enough to discard all of it.

Proposed fix

1. Respect the advertised Retry-After. It is already parsed. Backing off to 300s when the dependency asked for 5s is not backpressure, it is a self-imposed outage. Cap the ladder near the advertised value, or treat Retry-After as authoritative when present.
2. Do not discard a whole sweep for one latched 429. Per-item overload should skip that item and continue — this is the same principle as A single non-skippable dispatch error aborts the whole run-once pass, wedging all dispatch #292/fix(orchestrator): skip per-item dispatch failures instead of aborting the run-once pass (#292) #293, which fixed exactly this shape for dispatch errors. A sweep that pulled 40 issues and hit one 429 on the 39th should not throw away the other 39.
3. Decay consecutiveOverloads on partial progress, not only on a fully clean sweep. Requiring perfection to clear a ratchet means the ratchet rarely clears under sustained mild load.
4. Log the 429's reason, not just its message. relayfile returns four distinct reason codes (inflight_limit, oldest_inflight_age, router_inflight_limit, durable_object_overloaded) that all share one message string. relayfileOverload() already parses details.reason / data.reason. Surfacing it is the difference between "the DO is busy" and knowing which limiter fired — during this incident that ambiguity was the single biggest obstacle to diagnosis.

Tests

• must-fire: given a 429 carrying Retry-After: 5, the next probe occurs in ~5s, not ~300s. Fails today.
• must-fire: a sweep in which one item 429s still dispatches the remaining eligible items.
• must-not-fire: genuine sustained overload still backs off — the fix must not become "hammer the dependency every 5 seconds forever". Keep a ceiling, just a sane one, and prove the ceiling still applies when Retry-After is absent.
• must-not-fire: consecutiveOverloads still escalates under real repeated overload; partial-progress decay must not make the ratchet meaningless.

Provenance

Root-caused during the 2026-08-20 cloud Factory outage by the relayfile-workspace-do-0820 investigation lane, whose findings also established that the upstream trigger was relayfile-cloud#166 (unbounded provider-generation alarm work) and that D1 is not in this path at all — the workspace DO uses per-object SQLite (state.storage.sql), not the D1 binding.