Skip to content

[factory] Bound total spawn time including cleanup: a timed-out spawn can consume two budgets, not one #308

Description

@khaliqgant

Summary

spawn() is bounded, but not by a single budget. When invocation polling times out, the compensating release runs with a fresh full budget, so a spawn that is supposed to cost at most one budget can cost two.

Raised by cubic (confidence 9) on #307 at src/fleet/relay-fleet-client.ts:268, after that PR's final commit:

When invocation polling times out, spawn() waits for a compensating release with a fresh full budget, so the supposedly bounded spawn can take two budgets. Start cleanup as…

Why this was not merge-blocking

#307 ended a five-day dispatch outage in which a single sweep ran 62 minutes with consecutiveFailures: 0 — an unbounded hang that reported perfect health. Against that, 2N is a bounded worst case and the two guarantees #307 exists to establish both hold:

  • a spawn cannot hang forever, and
  • a placement Relay accepts after the local deadline is released rather than orphaned.

The cleanup path also legitimately needs some budget — it cannot inherit zero, or the compensating release becomes a no-op and we trade a bounded overrun for a leaked worker. So the fix is a cap, not simply passing the remainder.

What to change

Bound the total spawn operation, cleanup included, rather than bounding each phase separately:

  1. Derive the cleanup budget from an overall ceiling — e.g. min(remaining, cleanupCap) — so the worst case is N + cleanupCap, not 2N.
  2. Make the ceiling explicit and named, so the guarantee is stated rather than emergent.
  3. Confirm the two budgets do not compound per-spec across the spawn loop in #dispatchUnlocked. A multi-spec dispatch taking 2N per spec would be a materially worse bound than 2N overall, and that distinction should be settled by a test rather than by reading.

Tests

  • must-fire: a spawn whose polling times out and whose cleanup is slow completes within the overall ceiling, not 2N. Fails before the change.
  • must-not-fire: a cleanup that needs a normal, non-trivial amount of time still completes — the cap must not make the compensating release a no-op, which would reintroduce the orphaned-worker path LatePlacementReleasedError was added for in fix(orchestrator): reap a lifecycle that took a batch slot and never placed an agent (#303) #304.
  • a multi-spec dispatch stays within the overall ceiling rather than multiplying by spec count.

Context

Follow-up to #307 (fixes #306). Related: #303, #304.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions