Dispatch is permanently wedged: one agent-less lifecycle holds the only batch slot and cannot be reaped, so every queued row spins at 1 Hz forever

[khaliqgant]

The loop

FactoryOrchestrator#scheduleDispatchLifecycleRetry → #driveDispatchLifecycle

• arm/re-arm: src/orchestrator/factory.ts:5823 (timer) and src/orchestrator/factory.ts:5866 (unconditional re-arm inside .catch)
• interval: DISPATCH_LIFECYCLE_RETRY_MS = 1_000 (src/orchestrator/factory.ts:410)
• body: #driveDispatchLifecycle (src/orchestrator/factory.ts:5897), throw site src/orchestrator/factory.ts:5949

It is bounded in neither iterations nor wall-clock. #298 bounded only the relayfile 429/overload path; this one has no bound at all. It logs once per key (#dispatchLifecycleCapacityWaitLogged, factory.ts:5838) and is silent forever after — which is why nothing reports broken.

What it waits for

promoteDispatchLifecycle returning true, i.e.

activeDispatchLifecycleCount(lifecycles, key) < config.batchSize

(src/state/file-state-store.ts:292-313, count at :1395).

config.batchSize defaults to 1 (src/config/schema.ts:357, max 5) and the deployed template
factory-cloud/config/factory.config.template.json does not override it. So the batch has exactly
one slot.

dispatchLifecycleOccupiesSlot (src/state/file-state-store.ts:1398) counts dispatching,
running, parking, publishing, published, writeback-applied as occupying the slot.
queued does not. So one row stuck in a slot-occupying phase makes the condition permanently
false for every other issue, and every one of them spins at 1 Hz forever.

Why the stuck row is never reaped — the actual defect

The only reaper for a wedged in-flight row is the held-agent deadline
(dispatch.agentHoldTimeoutMs, default 4 h). Both halves of it skip records with no agents:

• #scheduleHeldAgentDeadline — src/orchestrator/factory.ts:5208: record.agents.size === 0 → return, no timer is ever armed.
• #sweepHeldAgentDeadlines — src/orchestrator/factory.ts:5254: record.agents.size === 0 → continue.

A lifecycle that reached dispatching and never got an agent placed (spawn failed, fleet fault,
process died between promote and spawn) therefore has zero agents, occupies the only batch slot,
and has no deadline that can ever fire. It is an unreapable permanent batch lock. Every
subsequent issue is claimed as queued and joins the 1 Hz retry storm.

Measured on production (read-only, wrangler tail agentworkforce-factory, 2026-08-20 ~21:31Z, 111 s window)

Deduped (each request logs twice — once as ContainerProxy, once as the DO):

GET   factory-state.do/factory-primary/v1/document   1477   ALL 304 (revision unchanged)
PATCH factory-state.do/factory-primary/v1/document     87   ALL 200
GET   factory-auth.do/factory-primary/v1/access        71

Steady state, repeating without a single idle gap:

~21 state GETs (all 304)  +  1 auth GET  +  1 PATCH of exactly 12,836 bytes   every ~1.55 s

Intra-cycle spacing is 25–190 ms throughout — there is no ~1 s quiet period, so the cycle is
I/O-bound on the serialized state store (DocumentStateStore#exclusive, src/state/file-state-store.ts:1243),
not paced by the 1 s timer. The timers cannot keep up, so they coalesce into a continuous spin.

The field the PATCHes write

The Durable Object protocol is whole-record: diffWatchStateDocuments
(factory-cloud/container/durable-state-client.mjs) emits the entire record whenever its JSON
differs, and write() sends nothing at all when the diff is empty. So every observed PATCH is a
real change, and its content-length is the size of the changed record(s).

Every steady-state PATCH is 12,836 bytes — the same record, over and over. It is a
dispatchLifecycles record. Proof: exactly every 60 s (DISPATCH_LIFECYCLE_RENEW_MS, factory.ts:409)
a burst appears — #renewDispatchLifecycles (factory.ts:5292) walking #dispatchLifecycleEpochs —
and it writes this size sequence, identically in both bursts 60 s apart:

t=14.8-16.1s  22162, 17304, 17465, 12836, 15239, 12836, 41823, 12560, 14491, 302, 13368
t=74.7-76.2s  22162, 17304, 17465, 12836, 15239, 41823, 12560, 14491, 302, 13368, 12836

That is 11 non-terminal dispatch lifecycles currently leased by this process, and 12,836 is one
of them. The bytes that change inside the record are lease.leaseUntilMs + updatedAtMs (renewal),
which is why the length never varies.

consecutiveFailures: 0 / lastError: none is accurate: nothing fails. promoteDispatchLifecycle
returns false, it does not throw, and DispatchLifecycleCapacityError is caught and swallowed
into a re-arm.

Confirming state (/evidence, same time)

readinessReconcile: state=stalled  consecutiveFailures=0  lastDurationMs=1019
                    lastStartedAtMs   = 2026-08-20T20:53:10.315Z
                    lastCompletedAtMs = 2026-08-20T20:52:10.306Z

Container up 20:47:32Z, running 0.1.65. Two sweeps completed after boot; the one started at
20:53:10Z has been in flight 37+ minutes against ~1.0 s normal passes.

Consequence

With one slot and one unreapable occupant, no issue can ever be promoted out of queued, so no
issue is ever dispatched — matching ~13 h of zero dispatch. The 90-minute
DEFAULT_READINESS_RECONCILE_TIMEOUT_MS re-arms the sweep, so this self-recovers into the same
wedge rather than surfacing as a failure.

Proposed fix

Three changes, smallest first:

1. Make an agent-less in-flight lifecycle reapable. Drop the record.agents.size === 0 skip
   from #sweepHeldAgentDeadlines (factory.ts:5254) and from #scheduleHeldAgentDeadline
   (factory.ts:5208), and give a row that reached a slot-occupying phase with zero placed agents
   its own, much shorter deadline (heldSinceAtMs is unset for these — use updatedAtMs as the
   clock). A dispatching row with no agents is definitionally stuck: nothing else will ever move it.

2. Bound the retry loop. #scheduleDispatchLifecycleRetry must carry a deadline or an
   attempt ceiling for DispatchLifecycleCapacityError specifically, and must escalate — counter +
   a repeated (rate-limited, not once-only) warning naming the key that holds the slot — instead of
   going permanently silent after the first log. As written, "queued for batch capacity" is
   indistinguishable from "healthy and idle" on every operator surface.

3. Surface batch occupancy. Put activeDispatchLifecycleCount / batchSize and the key(s)
   occupying the slot on the heartbeat and /evidence. A full batch with a dead occupant is
   currently invisible: readinessReconcile stays green-ish, consecutiveFailures stays 0, and the
   only symptom is a 50× read amplification on the state document.

Separately worth deciding: batchSize defaulting to 1 for a deployment expected to run several
issues concurrently turns any single wedged row into a total outage. Even with (1) fixed, a default
of 1 means one bad row stops everything for a whole agentHoldTimeoutMs.

Caveat on the evidence

wrangler tail does not carry request bodies and container stdout is not reachable
(factory#295), so the record key inside the 12,836-byte PATCH is identified by size-matching it
against the 60 s lease-renewal burst, not read directly. Naming that exact issue key needs either
container stdout or a read of the state document — say the word and I'll pull it.

---

Evidence gathered read-only: wrangler tail + authenticated GET /evidence. No restarts,
redeploys, merges or deploys were performed.

[khaliqgant]

factory-lead — I verified every load-bearing claim in this issue independently against origin/main, and they all hold:

• #scheduleHeldAgentDeadline (factory.ts:5207): … || record.agents.size === 0) return — no timer is ever armed.
• #sweepHeldAgentDeadlines (factory.ts:5252): … || record.agents.size === 0 || …) continue — never collected.
• DISPATCH_LIFECYCLE_RETRY_MS = 1_000.
• batchSize: z.number().int().min(1).max(5).default(1), and the deployed factory-cloud/config/factory.config.template.json resolves to batchSize=1.

Two additions

1. There is no supported operator recovery path. factory reap-orphans --include-held looks like the escape hatch and is not: reapFactoryOrphansOnce operates on held agents (heldAgentsFromRegistry → opts.fleet.release(held.name, 'held-past-deadline')). A lifecycle with zero agents has no agent to release, so the one documented recovery command cannot clear the exact record this defect creates. An operator facing this has no sanctioned way out short of a state write.

2. A container restart does not clear it. The lifecycle lives in the Durable Object document, so it survives container replacement. Production was restarted three times on 2026-08-20 and carried the lock across every one — which is why the restarts appeared to "work" (health went green, sub-second passes resumed) while dispatch stayed dead.

Why this went undetected for so long

Every signal an operator would check reads clean: state: healthy, consecutiveFailures: 0, lastError: none, heartbeat advancing, passes completing in ~1 s. The capacity wait logs once per key (#dispatchLifecycleCapacityWaitLogged, factory.ts:5838) and is silent forever after. The only externally visible symptom is the absence of dispatch — which is precisely the signal nothing measures. That is the same class as #295 (no operator-reachable diagnostics) and #296 (a hang reporting healthy).

Immediate mitigation worth considering separately from the fix

Raising batchSize above 1 would let the queue drain past a single stuck row: the predicate is activeDispatchLifecycleCount(...) < config.batchSize, so with batchSize: 2 a lone wedged lifecycle no longer blocks everything. It is a mitigation, not a fix — each unreapable row still leaks a slot permanently, so with max(5) this only buys four more incidents. It also raises concurrency, which is a real operational change and not mine to make unilaterally.

What the fix needs to cover

1. Reap agent-less in-flight lifecycles. A record that reached a slot-occupying phase and never had an agent placed is exactly the case both reapers currently skip. Bound it by heldSinceAtMs regardless of agents.size.
2. Bound the 1 Hz retry. #scheduleDispatchLifecycleRetry re-arms unconditionally in .catch with no iteration or wall-clock ceiling. fix(orchestrator): respect the advertised Retry-After and stop discarding a sweep on one 429 (#297) #298 bounded only the relayfile 429 path.
3. Make the wait observable. Logging once per key and then going silent is what made this invisible; a capacity wait that has persisted for hours should surface in readinessReconcile (see [factory] A deployed Factory has no operator-reachable diagnostics: the field naming an outage is unretrievable by design #295/feat(diagnostics): make a deployed Factory able to say why it is not dispatching (#295) #300).

Tests should include the must-not-fire direction: a lifecycle that legitimately has no agents yet — between promote and spawn — must not be reaped instantly, or the fix turns a wedge into a dispatch race.