You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
spawn() is bounded, but not by a single budget. When invocation polling times out, the compensating release runs with a fresh full budget, so a spawn that is supposed to cost at most one budget can cost two.
Raised by cubic (confidence 9) on #307 at src/fleet/relay-fleet-client.ts:268, after that PR's final commit:
When invocation polling times out, spawn() waits for a compensating release with a fresh full budget, so the supposedly bounded spawn can take two budgets. Start cleanup as…
Why this was not merge-blocking
#307 ended a five-day dispatch outage in which a single sweep ran 62 minutes with consecutiveFailures: 0 — an unbounded hang that reported perfect health. Against that, 2N is a bounded worst case and the two guarantees #307 exists to establish both hold:
a spawn cannot hang forever, and
a placement Relay accepts after the local deadline is released rather than orphaned.
The cleanup path also legitimately needs some budget — it cannot inherit zero, or the compensating release becomes a no-op and we trade a bounded overrun for a leaked worker. So the fix is a cap, not simply passing the remainder.
What to change
Bound the total spawn operation, cleanup included, rather than bounding each phase separately:
Derive the cleanup budget from an overall ceiling — e.g. min(remaining, cleanupCap) — so the worst case is N + cleanupCap, not 2N.
Make the ceiling explicit and named, so the guarantee is stated rather than emergent.
Confirm the two budgets do not compound per-spec across the spawn loop in #dispatchUnlocked. A multi-spec dispatch taking 2Nper spec would be a materially worse bound than 2N overall, and that distinction should be settled by a test rather than by reading.
Tests
must-fire: a spawn whose polling times out and whose cleanup is slow completes within the overall ceiling, not 2N. Fails before the change.
Summary
spawn()is bounded, but not by a single budget. When invocation polling times out, the compensating release runs with a fresh full budget, so a spawn that is supposed to cost at most one budget can cost two.Raised by cubic (confidence 9) on #307 at
src/fleet/relay-fleet-client.ts:268, after that PR's final commit:Why this was not merge-blocking
#307 ended a five-day dispatch outage in which a single sweep ran 62 minutes with
consecutiveFailures: 0— an unbounded hang that reported perfect health. Against that,2Nis a bounded worst case and the two guarantees #307 exists to establish both hold:The cleanup path also legitimately needs some budget — it cannot inherit zero, or the compensating release becomes a no-op and we trade a bounded overrun for a leaked worker. So the fix is a cap, not simply passing the remainder.
What to change
Bound the total spawn operation, cleanup included, rather than bounding each phase separately:
min(remaining, cleanupCap)— so the worst case isN + cleanupCap, not2N.#dispatchUnlocked. A multi-spec dispatch taking2Nper spec would be a materially worse bound than2Noverall, and that distinction should be settled by a test rather than by reading.Tests
2N. Fails before the change.LatePlacementReleasedErrorwas added for in fix(orchestrator): reap a lifecycle that took a batch slot and never placed an agent (#303) #304.Context
Follow-up to #307 (fixes #306). Related: #303, #304.