feat(ocap-kernel): launch subcluster vats in parallel by grypez · Pull Request #983 · MetaMask/ocap-kernel

grypez · 2026-07-27T13:57:20Z

Explanation

Subcluster startup was O(sum of vat init times) because #launchVatsForSubcluster used a serial for...of loop — each vat's initVat RPC handshake had to complete before the next one began.

This PR changes startup to O(max vat init time) by launching all vats concurrently via Promise.all. The mechanism:

Before any await, one kernel promise (kp<N>) is pre-allocated per vat and marked as kernel-decided.
The bootstrap message is queued immediately, targeting the bootstrap vat's unresolved kp<N>. KernelRouter parks the send on the promise via enqueuePromiseMessage.
All vats launch in parallel. As each vat's initVat handshake completes, its root kernel promise is resolved via resolvePromises('kernel', ...), and the run loop forwards any queued messages.
The bootstrap vat receives kernel promise KRefs for all peer vats in its bootstrap(roots, services) call — it can pipeline calls to peers while they are still initializing.

Changes

SubclusterManager.ts — rewrote #launchVatsForSubcluster to pre-allocate kernel promises, queue bootstrap immediately, and launch all vats with Promise.all.
SubclusterManager.test.ts — updated mocks (initKernelPromise, setPromiseDecider, resolvePromises) and assertions to match the new flow.
Kernel.test.ts — added resolvePromises = vi.fn() to the KernelQueue mock class.

Checklist

Tests pass (yarn workspace @metamask/ocap-kernel test:dev:quiet)
Build passes (yarn workspace @metamask/ocap-kernel build)
Changelog updated

Note

Medium Risk
Touches core launchSubcluster sequencing, failure propagation, and vat termination refcount logic; behavior changes for multi-vat startup and error handling.

Overview
Subcluster vats now launch in parallel via Promise.allSettled in #launchVatsForSubcluster, so startup time tracks the slowest vat instead of the sum of serial inits. Bootstrap still runs after all launch attempts finish, using the bootstrap vat’s real ko<N> root.

When a non-bootstrap vat fails, its entry in bootstrap(roots, …) is a rejected kernel promise (VAT_TERMINATED) so the bootstrap vat can observe the failure through pipelined E(peer) calls; launchSubcluster then rejects with the peer error after bootstrap returns. Failed launches roll back by terminating any vats that did start (#terminateVatQuietly), then tearing down IO and the subcluster record as before.

SubclusterLaunchResult adds vatRootKrefs (name → root kref for successful vats). Integration tests stop assuming fixed ko4/ko5 ordering and use returned refs instead.

A small vat cleanup fix skips the export baseline refcount decrement when GC has already zeroed reachability. Changelog and unit/integration coverage cover peer rejection and cleanup paths.

^{Reviewed by Cursor Bugbot for commit ddcb114. Bugbot is set up for automated code reviews on this repo. Configure here.}

Previously #launchVatsForSubcluster awaited each vat's initVat handshake serially, making startup O(sum of vat init times). Now all vats launch concurrently via Promise.all. Each vat gets a kernel promise pre-allocated for its root object; the bootstrap message is queued immediately targeting the bootstrap vat's (unresolved) promise. KernelRouter parks the send until that promise resolves. As each vat's handshake completes its root promise is resolved via resolvePromises('kernel', ...), making startup O(max). The bootstrap vat receives kernel promises for all other vats' roots in its bootstrap() call, so it can pipeline calls to those vats while they are still initializing. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

grypez · 2026-07-27T13:57:36Z

Benchmark: serial vs parallel subcluster startup

Measured on real worker processes (NodejsPlatformServices), 5 iterations each. Single-vat launch ≈ 865 ms (worker spawn + initVat handshake).

Vats	Serial (before)	Parallel (after)	Speedup
1	865 ms	868 ms	1.0×
3	2599 ms	880 ms	3.0×
5	4398 ms	987 ms	4.5×

Serial scaled linearly (~865 ms × N). Parallel is bounded by the slowest vat; the ~120 ms overhead above 1-vat for the 5-vat case is kernel promise allocation and resolution bookkeeping.

Raw numbers

Serial (HEAD^)

[bench] 1-vat: avg=865ms  min=851ms  max=889ms  (851, 889, 865, 866, 857ms)
[bench] 3-vat: avg=2599ms  min=2581ms  max=2636ms  (2582, 2602, 2592, 2581, 2636ms)
[bench] 5-vat: avg=4398ms  min=4363ms  max=4424ms  (4424, 4411, 4363, 4376, 4416ms)

Parallel (this PR)

[bench] 1-vat: avg=868ms  min=852ms  max=892ms  (881, 892, 853, 852, 862ms)
[bench] 3-vat: avg=880ms  min=852ms  max=906ms  (906, 863, 852, 872, 905ms)
[bench] 5-vat: avg=987ms  min=974ms  max=1002ms  (979, 987, 1002, 994, 974ms)

…nchVatsForSubcluster Both loops iterate vatEntries identically. Building roots[vatName] from the locally-scoped kpid in the same pass eliminates the second loop, the ! non-null assertion, and the eslint-disable comment that accompanied it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ise.all result Instead of mutating an outer variable via a .then() side-effect, collect the resolved root refs as the return value of Promise.all and extract the bootstrap vat's ref by index. Promise.all preserves insertion order, so the index is stable. config.bootstrap is guaranteed present (validated before this method is called), so the as-KRef cast is correct. Also removes the dead-code guard that followed — if Promise.all resolves, every vat (including bootstrap) launched successfully, so rootRefs[idx] is always defined. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

When a vat in a multi-vat subcluster fails to launch, its pre-allocated kernel promise was left unresolved. Any bootstrap message pipelined through that promise (E(roots.failingPeer).method()) would park forever. Add a .catch() to the launchVat chain that calls resolvePromises with rejected=true and a VAT_TERMINATED kernel error, then re-throws so the outer Promise.all still rejects and launchSubcluster surfaces the error. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…r rejection Add a peer-rejection-bootstrap vat that calls E(peer).ping() and logs whether the call resolves or rejects, making peer-vat-launch failures observable from the bootstrap vat. Add an integration test that launches a two-vat cluster where the peer vat (error-build-throw) always fails to build its root object. The test asserts that launchSubcluster rejects and that, after the kernel run loop drains, the bootstrap vat has logged a rejection message containing 'VAT_TERMINATED' — confirming that the rejected kernel promise propagates to the bootstrap vat via E() pipelining. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…omise] failures Switch #launchVatsForSubcluster from Promise.all with pre-allocated kp<N> refs to Promise.allSettled. Wait for all vats to settle, then send bootstrap with real ko<N> refs for succeeded vats and immediately-rejected kp<N> for failed peers. The previous approach sent bootstrap before any vat launched, so bootstrap received pending kp<N> refs. If bootstrap returned a value pipelined through those refs, the result CapData contained kp<N> slots, and kunser() would fail with "value is not durable: '[Promise]'" when trying to deserialize them. The new approach preserves concurrent vat launch (allSettled, not allSettled- then-serial) while ensuring bootstrap always gets concrete refs. Failed peer vats still get an immediately-rejected kernel promise so bootstrap can observe the failure via E(roots.peer).method() pipelining. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…t launch With concurrent subcluster startup (PR #983), ko<N> refs are assigned in the order vat workers respond — not the order vats are declared. Tests that hardcoded ko4/ko5/ko6 for alice/bob/carol broke because those refs are now non-deterministic. - Add `vatRootKrefs: Record<string, KRef>` to `SubclusterLaunchResult` so callers can look up the root KRef of any vat by name - persistence.test.ts: use `rootKref` from `launchSubcluster` instead of hardcoded ko4 for the multi-vat coordinator test - resume.test.ts: use `vatRootKrefs.alice/bob/carol` instead of ko4/ko5/ko6 - rejection.test.ts: sort before comparing `getVatIds()` output; Map insertion order is non-deterministic with concurrent launch Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The local Fail tag is declared as returning Error (not never) to work around a TypeScript control-flow analysis bug. Using `x ?? Fail` therefore widens the type to `T | Error`, breaking tsc on CI even though ts-jest's transpile-only mode accepts it. Replace each occurrence with an explicit undefined guard and throw. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…edVat When BOYD (bringOutYourDead) runs for an importing vat before terminateAllVats, dropImports/retireImports decrements the exported object's refcount to (0,0). cleanupTerminatedVat then tries to decrement the baseline (1,1) that was set at object creation, causing an underflow. Guard: check the current reachable count before the baseline decrement and skip it if reachable is already zero. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

github-actions · 2026-07-28T16:25:41Z

Coverage Report

Status	Category	Percentage	Covered / Total
🔵	Lines	71.79% ⬆️ +0.04%	9157 / 12754
🔵	Statements	71.63% ⬆️ +0.04%	9307 / 12992
🔵	Functions	72.65% ⬇️ -0.07%	2192 / 3017
🔵	Branches	65.35% ⬆️ +0.04%	3684 / 5637

File Coverage

File	Stmts	Branches	Functions	Lines	Uncovered Lines
Changed Files
packages/kernel-test/src/vats/peer-rejection-bootstrap.ts	0%	0%	0%	0%	14-24
packages/ocap-kernel/src/types.ts	100% 🟰 ±0%	100% 🟰 ±0%	100% 🟰 ±0%	100% 🟰 ±0%
packages/ocap-kernel/src/store/methods/vat.ts	97.29% ⬆️ +0.03%	81.81% ⬇️ -1.52%	100% 🟰 ±0%	97.27% ⬆️ +0.03%	224, 297, 304-305
packages/ocap-kernel/src/vats/SubclusterManager.ts	96.19% ⬇️ -0.43%	90.12% ⬇️ -2.18%	100% 🟰 ±0%	96.13% ⬇️ -0.44%	155-158, 236-239, 293, 368, 388, 405

Generated in workflow #4618 for commit ddcb114 by the Vitest Coverage Report Action

…ction vatName and service names come from ClusterConfig (user-provided). Using a plain {} as the accumulator allowed prototype pollution if a name like '__proto__' or 'constructor' was supplied. Object.create(null) removes the prototype chain, eliminating the risk. vatRootKrefs is spread into a plain {} at the return site so callers that use toStrictEqual continue to work. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Validate that no vat name shadows Object.prototype built-ins (e.g. __proto__, constructor) before using vatName as a property key on the roots/vatRootKrefs maps. Adds // lgtm[js/remote-property-injection] suppressions on the three write sites as a belt-and-suspenders measure for CodeQL, which does not track the null-prototype path through the TypeScript cast. Object.create(null) was considered but rejected: @endo/marshal requires Object.prototype-chained objects for CopyRecord serialization. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

GitHub Code Scanning requires '// lgtm [rule]' with a space, not '// lgtm[rule]' without one. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Accumulate roots and vatRootKrefs into entry arrays and call Object.fromEntries at the end, eliminating the obj[taintedKey] = value write pattern that CodeQL's js/remote-property-injection rule flags. @endo/marshal requires Object.prototype-chained CopyRecords, so Object.create(null) was not viable. The // lgtm suppression syntax is also not active on this repo. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…e arg - Move the Object.prototype name check into the main loop body, eliminating a separate pre-pass over vatEntries. - Drop the 'vatRoot' iface arg from kslot(kpid): kslot ignores iface for kp-prefixed refs (returns makeStandinPromise early), so the arg was dead code that implied it had an effect. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

grypez · 2026-07-30T17:31:14Z

Design note: `Promise.allSettled` vs. early-bootstrap with peer kernel promises

During review we considered an alternative to the current Promise.allSettled approach: start bootstrap as soon as its own launch resolves, passing a kernel promise (kpid) for each peer vat and resolving/rejecting it when that peer's launch settles. This would let bootstrap overlap with slow peer startups.

Resolved-promise inlining: verified absent. To assess the per-call cost of passing kpid instead of ko<N>, we traced the kernel's message routing. KernelQueue.enqueueSend never checks promise state — routing happens only in KernelRouter.#routeMessage at dequeue time. So even for an already-resolved kpid, a send targeting it re-enters the run queue and takes one extra crank before reaching ko<N>. There is no inlining shortcut.

But the extra crank only hurts awaited calls. The kpid approach also changes the natural programming style: bootstrap has no reason to await E(peer).method() — it can fire-and-forget, letting the kernel queue those messages against the still-unresolved kpid and deliver them once the peer is up. For that pattern the extra crank is essentially free. The cost is real only when code needs a return value from a peer, and it would apply to all callers of those peer refs, not just bootstrap.

Current choice: Promise.allSettled is simpler, has zero per-call overhead for successful peers, and startup latency hasn't been a measured bottleneck. Worth revisiting if that changes.

The parallel launch path built kslot(result.value) without an iface argument, so vat root remotes were branded 'Alleged: undefined' instead of 'Alleged: vatRoot'. Pass 'vatRoot' as the second argument to match the previous serial behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

When a non-bootstrap vat failed to launch, the catch block in launchSubcluster only destroyed IO channels and removed the subcluster record from the store, leaving any successfully-started vat workers running as orphans with no subcluster membership. The fix iterates getSubclusterVats and calls terminateVat/collectGarbage for each vatId where hasVat returns true (i.e., the worker is alive). Vats whose launch failed are not in the VatManager's live map so hasVat returns false and they are skipped. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…cluster-startup # Conflicts: # packages/ocap-kernel/CHANGELOG.md # packages/ocap-kernel/src/vats/SubclusterManager.test.ts

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1ff7a6a. Configure here.}

The launch rollback terminated vats in a loop wrapped in a single try/catch, so one vat that failed to die abandoned every vat after it while deleteSubcluster still removed the record — leaving vats running with no subcluster handle to terminate them by. Extract the per-vat teardown into #terminateVatQuietly so each vat's cleanup is independently best-effort. Terminating one vat at a time is also the shape the eventual retry path needs, where a failed vat is reaped on its own and the subcluster survives. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

github-advanced-security AI found potential problems Jul 27, 2026

View reviewed changes

Comment thread packages/ocap-kernel/src/vats/SubclusterManager.ts Fixed

Comment thread packages/ocap-kernel/src/vats/SubclusterManager.ts Fixed

grypez and others added 4 commits July 27, 2026 10:55

github-advanced-security AI found potential problems Jul 27, 2026

View reviewed changes

Comment thread packages/ocap-kernel/src/vats/SubclusterManager.ts Fixed

Comment thread packages/ocap-kernel/src/vats/SubclusterManager.ts Fixed

grypez and others added 2 commits July 27, 2026 12:55

github-advanced-security AI found potential problems Jul 27, 2026

View reviewed changes

Comment thread packages/ocap-kernel/src/vats/SubclusterManager.ts Fixed

Comment thread packages/ocap-kernel/src/vats/SubclusterManager.ts Fixed

Comment thread packages/ocap-kernel/src/vats/SubclusterManager.ts Fixed

grypez and others added 3 commits July 28, 2026 11:01

fix(ocap-kernel): flatten nested changelog bullet rejected by validator

cff762a

grypez and others added 2 commits July 28, 2026 12:50

github-advanced-security AI found potential problems Jul 28, 2026

View reviewed changes

Comment thread packages/ocap-kernel/src/vats/SubclusterManager.ts Fixed

Comment thread packages/ocap-kernel/src/vats/SubclusterManager.ts Fixed

Comment thread packages/ocap-kernel/src/vats/SubclusterManager.ts Fixed

grypez and others added 3 commits July 28, 2026 13:10

fix: correct lgtm suppression comment format (space before bracket)

79efe8d

GitHub Code Scanning requires '// lgtm [rule]' with a space, not '// lgtm[rule]' without one. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

grypez marked this pull request as ready for review July 30, 2026 18:13

grypez requested a review from a team as a code owner July 30, 2026 18:13

cursor Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread packages/ocap-kernel/src/vats/SubclusterManager.ts

Comment thread packages/ocap-kernel/src/vats/SubclusterManager.ts

grypez and others added 2 commits August 3, 2026 10:31

grypez enabled auto-merge August 3, 2026 14:51

grypez mentioned this pull request Aug 4, 2026

C-list import accounting is asymmetric: the refcount increment on import is missing #1006

Open

8 tasks

Merge remote-tracking branch 'origin/main' into grypez/concurrent-sub…

1ff7a6a

…cluster-startup # Conflicts: # packages/ocap-kernel/CHANGELOG.md # packages/ocap-kernel/src/vats/SubclusterManager.test.ts

cursor Bot reviewed Aug 10, 2026

View reviewed changes

Comment thread packages/ocap-kernel/src/vats/SubclusterManager.ts

Navigation Menu

Uh oh!

feat(ocap-kernel): launch subcluster vats in parallel - #983

feat(ocap-kernel): launch subcluster vats in parallel#983
grypez wants to merge 19 commits into
mainfrom
grypez/concurrent-subcluster-startup

grypez commented Jul 27, 2026 •

edited by cursor Bot

Loading

Uh oh!

grypez commented Jul 27, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

grypez commented Jul 30, 2026

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

grypez commented Jul 27, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Explanation

Changes

Checklist

Uh oh!

grypez commented Jul 27, 2026

Benchmark: serial vs parallel subcluster startup

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Coverage Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

grypez commented Jul 30, 2026

Design note: Promise.allSettled vs. early-bootstrap with peer kernel promises

Uh oh!

Uh oh!

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

grypez commented Jul 27, 2026 •

edited by cursor Bot

Loading

github-actions Bot commented Jul 28, 2026 •

edited

Loading

Design note: `Promise.allSettled` vs. early-bootstrap with peer kernel promises