Skip to content

fix(agents-server): dropped parent wake on parallel sub-agent spawn#4720

Open
andresberrios wants to merge 3 commits into
electric-sql:mainfrom
andresberrios:fix/wake-registry-parallel-spawn-clobber
Open

fix(agents-server): dropped parent wake on parallel sub-agent spawn#4720
andresberrios wants to merge 3 commits into
electric-sql:mainfrom
andresberrios:fix/wake-registry-parallel-spawn-clobber

Conversation

@andresberrios

Copy link
Copy Markdown

Problem

When a parent spawns two (or more) sub-agents in parallel — both spawn_worker-style tool calls issued in the same turn, as a model naturally does when fanning out — only one of the children ever wakes the parent on completion. The other child's runFinished is silently dropped: no error, no warning. Spawning the same children sequentially always works.

Root cause

Each child's runFinished wake is registered from two independent paths, both keyed by the same manifestKey:

  • spawn pathentity-manager.ts spawnInner (if (req.wake) wakeRegistry.register(...))
  • manifest-sync pathruntime.ts syncManifestWakeswakeRegistry.register(...), when the parent's manifest entry for the child lands on the parent's stream.

register() inserts with .onConflictDoNothing() against uq_wake_registration. The second path to run gets an empty result and previously took this branch:

if (result.length === 0) {
  await this.loadRegistrations()  // full clear-and-rebuild of the cache
  return
}

loadRegistrations() snapshots all rows across an await, then resetCachedRegistrations() clears the entire in-memory cache and rebuilds from that snapshot. Parallel spawn issues up to 4 register() calls (2 children × {spawn, manifest-sync}); several land in the conflict branch and their reloads interleave. If a stale snapshot's continuation runs last, it clears the cache and rebuilds without a sibling's newer registration — evicting it even though the DB row exists. When that sibling then finishes, evaluate() reads only the in-memory cache, finds no match, returns [], and the wake is dropped with no error path hit.

Sequential spawn never overlaps the reloads (each child fully registers and delivers before the next spawns), which is exactly why only the parallel fan-out reproduces.

Fix

In register()'s conflict branch, re-read only the single conflicting row (by its uq_wake_registration columns) and upsertCachedRegistration() just that entry. No global cache reset → no cross-registration clobber.

Test

Adds a regression test in test/wake-registry.test.ts: registers two siblings for the same subscriber where the second hits the conflict branch against a stale full-table snapshot (missing the first), then asserts both still evaluate() to a match. Fails on main (first sibling evicted → evaluate returns []), passes with the fix. Full wake-registry.test.ts suite (34 tests) green.

andresberrios and others added 3 commits July 15, 2026 18:34
Each child's runFinished wake is registered from two paths (spawn +
manifest-sync) keyed by the same manifestKey. The second insert hits the
uq_wake_registration unique constraint and takes the conflict branch,
which called loadRegistrations() — a full clear-and-rebuild of the
in-memory registration cache from a snapshot read across an await.

Under parallel spawn several of these reloads interleave. A stale
snapshot whose continuation lands last clears the whole cache and
rebuilds it without a sibling's newer registration, evicting it. When
that sibling later finishes, evaluate() finds no cache entry and the
wake is silently dropped (no error path). Sequential spawn never
overlaps the reloads, so only the parallel fan-out reproduced it.

Re-read only the single conflicting row and cache just that entry,
leaving sibling registrations untouched. Add a regression test that
registers two siblings for the same subscriber with the second hitting
the conflict branch against a stale full-table snapshot, and asserts
both still evaluate.
When a parent spawns 2+ sub-agents in one turn, only some children's
runFinished wakes reached the parent; the rest were silently dropped.

Root cause is in ElectricAgentsTenantRuntime.syncManifestWakes: for each
manifest `upsert` it ran `unregisterByManifestKey()` then `await register()`.
That non-atomically removes the wake registration from the in-memory cache and
only re-adds it after a DB round-trip. Every spawned child re-syncs a
registration identical to the one the spawn already created, so each child's
registration churns through a remove -> re-add window; a sibling whose run
finishes inside that window evaluates against a cache with no matching
registration and its wake is lost. The earlier parallel-spawn-clobber fix
guarded register()'s on-conflict path, which distinct-source children never
exercise, so it did not cover this case.

Fix: WakeRegistry.reconcileManifestRegistration registers the desired
registration first (register() now returns the dbId it left in the cache: the
fresh insert, or the pre-existing row re-read on conflict), then deletes only
the other rows anchored to that manifest key. The registration is continuously
present, so no in-flight wake falls through a gap. Pruning by the exact kept id
(never a re-derived field key) guarantees the row just kept is never removed.

Verified with a new pull-wake e2e (parent spawns 6 workers in parallel via the
real runner + mock LLM): every child's wake reaches the parent's stream across
15/15 runs; it failed ~1-in-1..1-in-3 before. Adds two deterministic unit tests
for the reconcile contract. Existing wake-registry suites unchanged (36/36).

Note: the pull-wake runner legitimately coalesces multiple wake events into one
handler invocation, so the assertion is at the production layer (one wake event
per child on the parent stream), not handler-invocation count.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…bhook-source sites

upsertCronSchedule and upsertWebhookSourceSubscription used the same
unregister-then-register sequence that dropped wakes on parallel sub-agent
spawn: the manifest-anchored wake registration is briefly absent from the
in-memory cache across the register()'s DB round-trip, so a cron tick or webhook
event arriving in that window would be missed. These are single API calls rather
than concurrent bursts, so the race is far narrower than the spawn path, but the
gap is the same class of bug — and both also race against the manifest-sync
reconcile triggered by their own writeManifestEntry.

Route both through WakeRegistry.reconcileManifestRegistration, which registers
the desired reg first (never leaving the cache empty) and then prunes only stale
rows for the manifest key. Delete-only sites (deleteSchedule,
deleteWebhookSourceSubscription, deletePgSyncObservation, upsertFutureSendSchedule)
have no re-register and so no gap; left as-is. Cron/webhook/manifest suites pass
(30/30).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@andresberrios

Copy link
Copy Markdown
Author

Follow-up: pushed two commits that fix a different parallel-spawn wake drop than the original clobber this PR addressed.

The original fix guards register()'s on-conflict loadRegistrations() path. But sibling sub-agents spawned in one turn register distinct sources, so they never hit that conflict path — and a patched build still dropped wakes. Root cause turned out to be ElectricAgentsTenantRuntime.syncManifestWakes: for each child's manifest entry it did unregisterByManifestKey()await register(), non-atomically removing then re-adding a registration that's identical to the one the spawn already created. During the await the registration is absent from the in-memory cache; a sibling finishing in that window has its runFinished wake silently lost.

  • deb8c53 adds WakeRegistry.reconcileManifestRegistration (register-first — register() now returns the kept dbId — then prune only stale rows by that exact id) and routes syncManifestWakes through it. No gap.
  • 781c9eb routes the same latent unregister→register pattern at upsertCronSchedule and upsertWebhookSourceSubscription through the same gap-free helper.

Verified with a new pull-wake e2e (parent spawns 6 workers in parallel via the real runner + mock LLM): every child's wake now reaches the parent's stream across 15/15 runs; it dropped ~1-in-1..1-in-3 before. Two deterministic unit tests added; existing wake-registry suites (36) unchanged.

Note: the pull-wake runner legitimately coalesces multiple wake events into one handler invocation, so the guarantee asserted is at the production layer (one wake event per child on the parent stream), not handler-invocation count.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant