[factory] Spawn-ack timeout cannot fire: the deadline guards the poll loop, not the calls inside it — sweep hung 62min with 0 failures

[khaliqgant]

Summary

The readiness sweep can hang indefinitely on the agent-placement path. A 5-minute spawn-ack timeout is configured and never fires, because the deadline is checked only between calls rather than applied to the calls themselves.

Production has been observed with a single sweep in flight for 62 minutes while consecutiveFailures stayed at 0. Nothing throws, nothing retries, and the occupied batch slots are never released, so dispatch stops completely.

Observed in production (0.1.66)

From the daemon's own health block:

readinessReconcile: state=stalled  consecutiveFailures=0  failureThreshold=3  intervalMs=60000
                    lastStartedAtMs=1787280884216
                    lastCompletedAtMs=1787280824210   <-- EARLIER than lastStarted
                    inFlightMs=3742257                <-- 62 minutes, monotonically climbing
                    missedPasses=62
dispatchCapacity:   state=stalled  batchSize=2  active=2  waiting=4
                    longestWaitMs=3082193             <-- 51 minutes
                    agentlessOccupants=1
                    agentlessHoldTimeoutMs=1800000
fleetControlPlane:  state=closed (healthy)  consecutiveFailures=0
degradedSubsystems: ["readinessReconcile","dispatchCapacity"]

lastStartedAtMs > lastCompletedAtMs with a climbing inFlightMs is the signature of a pass that started and never returned — a hang, not an error. A sweep that does complete costs ~738ms, so this is not slowness.

The decisive fact: 62 minutes elapsed against a 5-minute ack timeout with zero recorded failures. Whatever the layers underneath do, nothing on this path is bounded in practice.

Mechanism (hypothesis, code cited)

src/fleet/relay-fleet-client.ts — the client used when running --backend relay.

1. The deadline guards the loop, not the calls inside it.

async #awaitInvocation(actionName, ack) {
  const deadline = Date.now() + this.#spawnAckTimeoutMs;   // DEFAULT_SPAWN_ACK_TIMEOUT_MS = 5 * 60_000
  while (!terminalStatuses.has(status)) {
    if (Date.now() > deadline) throw ...                   // only reachable BETWEEN iterations
    await this.#sleep(this.#pollIntervalMs);
    invocation = await messaging.commands.getInvocation(actionName, ack.invocationId);  // unbounded
    status = invocation.status || 'pending';
  }
}

If getInvocation never settles, control never returns to the deadline check. The timeout is real, enforced, and unreachable against precisely the failure it exists for. A deadline observable only between calls does not bound a call.

2. An earlier await has no deadline at all.

async spawn(input) {
  const messaging = await this.#ensureMessaging();        // unbounded
  await this.#ensureLifecycleAction(messaging);           // unbounded
  const ack = await messaging.placement.spawn({ ... });   // unbounded, BEFORE any deadline exists
  const invocation = await this.#awaitInvocation(...);

placement.spawn runs before #awaitInvocation computes a deadline, so a hang there is unbounded by construction.

Why this client specifically

• src/fleet/relay-fleet-client.ts: no AbortSignal, no .kill(, no per-call timeout.
• src/fleet/internal-fleet-client.ts: 13 timeout references and a .kill().
• src/mount/github-api-issue-read.ts: bounds its fetch with AbortSignal.timeout(30_000).

The codebase knows how to bound calls. This client is the outlier, and it is the one production runs.

Why it is worse than a stall

agentlessOccupants=1 with active=2/batchSize=2: the hung placements hold their slots, four units queue behind them, and the pass never settles to release them. #303's reaper clears agent-less lifecycles after agentlessHoldTimeoutMs, but they re-form on the next pass, so the system oscillates between reaped and wedged and never dispatches. #303 fixed the symptom; this is the cause.

Proposed fix

Bound each individual call, not just the loop:

1. Give messaging.placement.spawn, messaging.commands.getInvocation, #ensureMessaging and #ensureLifecycleAction their own per-call deadline (AbortSignal.timeout or Promise.race against the remaining budget).
2. Derive each per-call budget from the remaining overall deadline so total time stays bounded even across many polls.
3. On expiry, throw a classified error so the pass records a failure and releases the slot, rather than holding it silently. Note consecutiveFailures=0 throughout a 62-minute hang — the health signal cannot currently distinguish "working" from "wedged forever" on this path.

Tests worth demanding

• must-fire: a getInvocation that never resolves causes #awaitInvocation to reject within the ack budget. This must fail before the change.
• must-not-fire: a slow-but-completing sequence of polls that stays within the overall budget still succeeds, and is not killed by an over-eager per-call timeout.
• a hung placement.spawn (before any deadline exists) also rejects within a bounded time.
• after expiry, the batch slot is released and the failure is counted.

[khaliqgant]

Update: confirmed, and there are three stacked defects — this issue described only the second

An independent sampler (61 samples over 31 minutes) confirms the mechanism and found the layer beneath it. Revised causation:

(1) ROOT CAUSE — an ack does not mean anything launched

Placement confirmation is opt-in via input.confirm, and Factory never sets it. messaging.placement.spawn(...) can therefore return a valid ack when nothing was actually launched, and Factory cannot distinguish that from a real spawn. It then waits for an invocation that will never reach a terminal status, because no worker exists behind it.

This is why the sweep hangs at all. Everything else is consequence.

(2) Why it presents as an indefinite hang — the original body of this issue

#awaitInvocation checks its deadline at the loop top around an unbounded commands.getInvocation, so the 5-minute bound is unreachable.

Confirming evidence. Across all 61 samples spanning 31 minutes, lastStartedAtMs and lastCompletedAtMs are byte-identical:

03:02:34Z  inFlightMs=456256    started=1787280884216  completed=1787280824210
03:15:22Z  inFlightMs=1236509   started=1787280884216  completed=1787280824210
03:33:17Z  inFlightMs=2301832   started=1787280884216  completed=1787280824210

One sweep started 02:54:44Z and was still in flight at 03:33:17Z — 38.4 minutes, climbing linearly with wall clock, never settling. A reachable 5-minute deadline cannot produce that, regardless of how many spawns the pass contains.

Note on a weaker signal: consecutiveFailures counts sweep failures, not dispatch failures, so 0 is not by itself evidence of a hang. Combined with a sweep that has not settled in 38 minutes it is corroborating — nothing threw because nothing returned.

(3) Why it never self-heals — the agentless reaper did not reap

03:07:41Z  active=2 waiting=4  agentlessOccupants=None
03:08:12Z  active=1 waiting=4  agentlessOccupants=1     <- past its reap deadline
03:33:17Z  active=2 waiting=4  agentlessOccupants=1     <- still 1, 25 minutes later

agentlessOccupants counts an occupied slot that never placed an agent and is already past the deadline that should have reaped it. It survived 25 minutes across 50 consecutive samples with agentlessHoldTimeoutMs=1800000. So #304's reaper is either not firing on this occupant or firing and failing to clear it. "The reaper handles it" is currently false, and that compensating control cannot be relied on.

Meanwhile longestWaitMs climbed monotonically to 1,641,768 ms — a real work unit waited 27 minutes behind a slot held by a lifecycle that never placed an agent.

Retraction

I previously cited state-document PATCH traffic dropping from 152/100s to 0/65s after the 0.1.66 deploy as evidence the #303 reaper worked. Finding (3) undercuts that reading: zero writes is equally explained by "the sweep hung and nothing is progressing." That measurement should not be treated as evidence the reaper works.

Fix ordering

1. (1) stops the wedge forming — the only one that actually restores dispatch.
2. (2) stops it being invisible and infinite; converts it to a bounded, classified, countable failure that releases the slot.
3. (3) stops it being permanent.

All three are worth having. (1) and (2) are two halves of the same failure and belong together; (3) may warrant its own change since it is a gap in work merged in #304.

Caveat on (1): input.confirm should not simply be switched on without establishing what it costs — an extra round trip, added latency, or a capability nodes must support. That needs to be stated before it lands, not assumed.