fix(agents-server): dropped parent wake on parallel sub-agent spawn#4720
fix(agents-server): dropped parent wake on parallel sub-agent spawn#4720andresberrios wants to merge 3 commits into
Conversation
Each child's runFinished wake is registered from two paths (spawn + manifest-sync) keyed by the same manifestKey. The second insert hits the uq_wake_registration unique constraint and takes the conflict branch, which called loadRegistrations() — a full clear-and-rebuild of the in-memory registration cache from a snapshot read across an await. Under parallel spawn several of these reloads interleave. A stale snapshot whose continuation lands last clears the whole cache and rebuilds it without a sibling's newer registration, evicting it. When that sibling later finishes, evaluate() finds no cache entry and the wake is silently dropped (no error path). Sequential spawn never overlaps the reloads, so only the parallel fan-out reproduced it. Re-read only the single conflicting row and cache just that entry, leaving sibling registrations untouched. Add a regression test that registers two siblings for the same subscriber with the second hitting the conflict branch against a stale full-table snapshot, and asserts both still evaluate.
When a parent spawns 2+ sub-agents in one turn, only some children's runFinished wakes reached the parent; the rest were silently dropped. Root cause is in ElectricAgentsTenantRuntime.syncManifestWakes: for each manifest `upsert` it ran `unregisterByManifestKey()` then `await register()`. That non-atomically removes the wake registration from the in-memory cache and only re-adds it after a DB round-trip. Every spawned child re-syncs a registration identical to the one the spawn already created, so each child's registration churns through a remove -> re-add window; a sibling whose run finishes inside that window evaluates against a cache with no matching registration and its wake is lost. The earlier parallel-spawn-clobber fix guarded register()'s on-conflict path, which distinct-source children never exercise, so it did not cover this case. Fix: WakeRegistry.reconcileManifestRegistration registers the desired registration first (register() now returns the dbId it left in the cache: the fresh insert, or the pre-existing row re-read on conflict), then deletes only the other rows anchored to that manifest key. The registration is continuously present, so no in-flight wake falls through a gap. Pruning by the exact kept id (never a re-derived field key) guarantees the row just kept is never removed. Verified with a new pull-wake e2e (parent spawns 6 workers in parallel via the real runner + mock LLM): every child's wake reaches the parent's stream across 15/15 runs; it failed ~1-in-1..1-in-3 before. Adds two deterministic unit tests for the reconcile contract. Existing wake-registry suites unchanged (36/36). Note: the pull-wake runner legitimately coalesces multiple wake events into one handler invocation, so the assertion is at the production layer (one wake event per child on the parent stream), not handler-invocation count. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…bhook-source sites upsertCronSchedule and upsertWebhookSourceSubscription used the same unregister-then-register sequence that dropped wakes on parallel sub-agent spawn: the manifest-anchored wake registration is briefly absent from the in-memory cache across the register()'s DB round-trip, so a cron tick or webhook event arriving in that window would be missed. These are single API calls rather than concurrent bursts, so the race is far narrower than the spawn path, but the gap is the same class of bug — and both also race against the manifest-sync reconcile triggered by their own writeManifestEntry. Route both through WakeRegistry.reconcileManifestRegistration, which registers the desired reg first (never leaving the cache empty) and then prunes only stale rows for the manifest key. Delete-only sites (deleteSchedule, deleteWebhookSourceSubscription, deletePgSyncObservation, upsertFutureSendSchedule) have no re-register and so no gap; left as-is. Cron/webhook/manifest suites pass (30/30). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Follow-up: pushed two commits that fix a different parallel-spawn wake drop than the original clobber this PR addressed. The original fix guards
Verified with a new pull-wake e2e (parent spawns 6 workers in parallel via the real runner + mock LLM): every child's wake now reaches the parent's stream across 15/15 runs; it dropped ~1-in-1..1-in-3 before. Two deterministic unit tests added; existing wake-registry suites (36) unchanged. Note: the pull-wake runner legitimately coalesces multiple wake events into one handler invocation, so the guarantee asserted is at the production layer (one wake event per child on the parent stream), not handler-invocation count. |
Problem
When a parent spawns two (or more) sub-agents in parallel — both
spawn_worker-style tool calls issued in the same turn, as a model naturally does when fanning out — only one of the children ever wakes the parent on completion. The other child'srunFinishedis silently dropped: no error, no warning. Spawning the same children sequentially always works.Root cause
Each child's
runFinishedwake is registered from two independent paths, both keyed by the samemanifestKey:entity-manager.tsspawnInner(if (req.wake) wakeRegistry.register(...))runtime.tssyncManifestWakes→wakeRegistry.register(...), when the parent's manifest entry for the child lands on the parent's stream.register()inserts with.onConflictDoNothing()againstuq_wake_registration. The second path to run gets an empty result and previously took this branch:loadRegistrations()snapshots all rows across anawait, thenresetCachedRegistrations()clears the entire in-memory cache and rebuilds from that snapshot. Parallel spawn issues up to 4register()calls (2 children × {spawn, manifest-sync}); several land in the conflict branch and their reloads interleave. If a stale snapshot's continuation runs last, it clears the cache and rebuilds without a sibling's newer registration — evicting it even though the DB row exists. When that sibling then finishes,evaluate()reads only the in-memory cache, finds no match, returns[], and the wake is dropped with no error path hit.Sequential spawn never overlaps the reloads (each child fully registers and delivers before the next spawns), which is exactly why only the parallel fan-out reproduces.
Fix
In
register()'s conflict branch, re-read only the single conflicting row (by itsuq_wake_registrationcolumns) andupsertCachedRegistration()just that entry. No global cache reset → no cross-registration clobber.Test
Adds a regression test in
test/wake-registry.test.ts: registers two siblings for the same subscriber where the second hits the conflict branch against a stale full-table snapshot (missing the first), then asserts both stillevaluate()to a match. Fails onmain(first sibling evicted →evaluatereturns[]), passes with the fix. Fullwake-registry.test.tssuite (34 tests) green.