docs(decisions): ADR-021 propose Lambda MicroVMs compute backend (#645)

[dreamorosi]

Records the design decision for adding AWS Lambda MicroVMs as a third opt-in ComputeStrategy backend (compute_type: 'lambda-microvm'), fulfilling the ADR/RFC acceptance criterion of #645. Docs-only: the ADR (status proposed) plus its generated Starlight mirror; implementation follows in separate PRs per the phased rollout the ADR defines.

Lifecycle model resolves real substrate tensions

MicroVM idle detection is inbound-traffic-based while the ABCA agent is outbound-only, and MicroVMs cannot self-suspend — so the ADR mandates auto-suspend disabled at RunMicrovm and makes suspend/resume an orchestrator-owned decision, targeted at HITL approval waits (compute billing stops while memory/disk state, including the cloned repo and warm caches, is preserved).

Interface widening follows the codebase idiom

suspendSession/resumeSession are added as mandatory methods returning typed unsupported results on agentcore/ecs (no optional interface methods exist in the codebase; AgentCoreComputeStrategy.pollSession is already a mandatory explicit stub), landing in P3 across all three strategies in one commit.

Regional availability is enforced in layers

Synth-time static constant with a context-flag escape hatch, live probes (list-managed-microvm-images) at onboarding and in bgagent platform doctor, and orchestration-time error classification as defense in depth — live probes wherever possible so the platform self-heals as AWS expands beyond the 5 launch regions.

Scope corrections vs the original issue text: AgentCore Memory parity is in scope (P2) since Memory is already consumed cross-substrate on ECS via IAM grant + MEMORY_ID; only the Runtime-coupled workload-access-token injection path is deferred (#249/ADR-016). Gateway integration is orthogonal (ADR-019/#641, substrate-portable by design). Numbered ADR-021: 018/019 are claimed by open PRs #548/#663 (banner in the ADR covers renumbering).

Refs #645

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@aa3372e). Learn more about missing BASE report.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #688   +/-   ##
=======================================
  Coverage        ?   90.13%           
=======================================
  Files           ?      241           
  Lines           ?    59008           
  Branches        ?     5749           
=======================================
  Hits            ?    53188           
  Misses          ?     5820           
  Partials        ?        0           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[krokoko]

ADR review — thanks for this

Really solid proposed ADR. The lifecycle diagnosis (inbound idle detection vs an outbound-only agent, no self-suspend, snapshot uniqueness, short JWE TTL) is accurate, the interface widening matches the codebase idiom (pollSession stub / exhaustive never), and the scope corrections vs #645 (Memory in P2, WAT deferred to #249/ADR-016, Gateway orthogonal) are well reasoned. Numbering banner, EARS requirements, layered regional availability, and the P1–P3 rollout all look right for a docs-only decision record.

Verdict: approve-with-changes before moving to accepted — the HITL suspend model needs to be reconciled with today’s agent-owned approval poll, and a few API/IAM details should be pinned so P1/P3 implementers don’t have to rediscover them.

Blocking / please resolve before accepted

1. HITL suspend vs agent-owned approval poll

Today (CEDAR_HITL_GATES.md): soft-deny → agent writes AWAITING_APPROVAL → agent _poll_for_decision on DDB (monotonic timeout, default 300s) → approve-task writes DDB → agent discovers approval and resumes the tool call. The container stays alive; concurrency slot is held for that reason (decision #7).

Sub-decision 2 currently says the orchestrator suspends on HITL wait and resumes “before delivering the approval outcome.” That delivery path doesn’t exist today — the agent discovers approval itself.

Could you make the intended handshake explicit? For example:

Question	Needs a decision
Who triggers suspend?	Orchestrator sees AWAITING_APPROVAL after grace → suspendSession
Who triggers resume?	approve/deny Lambda? Orchestrator poll? Event?
Timeout while suspended	time.monotonic() freezes → Cedar gate timeout effectively pauses. Intentional? Or does wall-clock / stranded reconciler still win?
suspendedDurationSeconds	Must cover the approval window / stranded timeout (2h), or an intentional suspend gets auto-terminated
Concurrency slots	Decision #7 rationale (“container consuming memory/CPU”) changes under suspend — slot still held?

Without that, P3 will fight Cedar semantics.

2. /run is a fast hook, not the agent pipeline

Runtime /run timeout is 1–60s and the lifecycle docs / toolkit skill say not to do long work there. Suggest a normative line that /run validates the payload / kicks off the agent asynchronously and returns 200; clone→PR stays off the hook path.

3. Pin “auto-suspend disabled”

The normative mechanism is omit idlePolicy on RunMicrovm (if the block is present, all three fields are required). “Disabled” alone is ambiguous for implementers and for the disabled-auto-suspend invariant test.

Should-fix (fine for proposed, needed before P1/P3)

4. IAM gaps. lambda:RunMicrovm / Suspend / Resume / Terminate / CreateMicrovmAuthToken look correct per the IAM reference. Please also call out lambda:PassNetworkConnector (required even for default connectors) and typically lambda:GetMicrovm for pollSession.

5. JWE refresh has no decision. Tension #4 notes the ≤60 min TTL; Decision never says who refreshes, for which calls, or that P1 may need no orchestrator→agent HTTP if work is /run-driven + agent egress only.

6. Security-bar deltas. #645’s acceptance criteria ask to meet the same security bar or list deltas. Workload-token and no SHELL_INGRESS are mentioned; a short table (egress VPC, session-role admit, secrets-at-/run, public HTTPS endpoint + JWE, snapshot uniqueness) would close that AC cleanly.

7. pollSession: SUSPENDED → running. Fine for intentional HITL suspend — please note the orchestrator must not treat AWAITING_APPROVAL + SUSPENDED as unhealthy / fail-fast the way a silent AgentCore crash would.

Nits

• Consequence on the 8h cap including suspended time is good; worth adding that suspendedDurationSeconds is a second kill switch for intentional suspends.
• /suspend “flushes progress events” — keep that ≤60s; prefer durable writes before returning 200.
• Disk “32 GB hard ceiling” — memory/vCPU are well documented; confirm disk against current quotas when COMPUTE.md is updated in the implementing PR.
• Starlight mirror looks sync’d — thanks.

Suggested minimal ADR edits

1. Rewrite sub-decision 2 around detect AWAITING_APPROVAL → suspend → on approval row terminal → resume → agent poll continues; drop “orchestrator delivers approval outcome.”
2. Add EARS requirements for omit-idlePolicy, async /run, suspendedDurationSeconds ≥ stranded/approval bound, and PassNetworkConnector.
3. One paragraph on approval-timeout semantics under freeze (monotonic vs wall/stranded).
4. Short security-delta table for the #645 AC.

Happy to re-review once those land. Refs #645.

[dreamorosi]

Thanks @krokoko — all points addressed in de599b6. Point-by-point:

Blocking 1 (HITL handshake): You were right that "orchestrator delivers the approval outcome" doesn't exist — the agent discovers decisions itself via _poll_for_decision. Sub-decision 2 is rewritten around your table:

Question	Decision
Suspend trigger	Orchestrator durable poll sees AWAITING_APPROVAL → grace → suspendSession, only when the gate's remaining window exceeds grace + resume overhead
Resume trigger	Inline best-effort in ApproveTaskFn after the transactional write commits, orchestrator poll reconciles (decision row + still SUSPENDED → retry)
Timeout under freeze	Orchestrator owns the wall clock: resumes at gate deadline − margin so the agent's monotonic timer fires the deny agent-side — Cedar keeps deny authority, the orchestrator never writes decisions
suspendedDurationSeconds	Backstop only, ≥ gate ceiling (1 h) + margin — never the timeout mechanism
Concurrency slot	Held. Decision #7's "container consuming memory" rationale weakens, but AWS counts SUSPENDED MicroVMs toward the account memory quota, so releasing the slot frees nothing — it would only admit tasks the substrate can't start. P3 updates the decision #7 rationale note

On the resume-trigger tiebreak specifically, we went looking for codebase precedent rather than deciding on taste: cancel-task.ts already resolves this exact shape — a user-initiated, latency-sensitive action whose purpose is an immediate compute side effect — by invoking StopTask/StopRuntimeSession inline from the API plane, best-effort (warn on failure, state transition stands), with a task_cancel_compute_orphan event for the reconciler and conditional IAM in task-api.ts. Resume-on-approve mirrors that pattern 1:1 (including the orphan event + poll backstop), rather than paying up to a poll interval of latency on every approval for a purity argument that was already litigated and declined for cancel. The resume call sits after the transaction commits so it cannot affect the decision outcome — relevant given the security-critical ownership comparison in approve-task.ts (finding #6).

Blocking 2: EARS requirement added — /run validates and starts the pipeline asynchronously, returns 200 within the hook budget; clone→PR never on the hook path (matches the existing background-thread-behind-/ping design).

Blocking 3: Pinned as omit idlePolicy entirely on RunMicrovm; the invariant test asserts omission.

4 (IAM): Added lambda:PassNetworkConnector and lambda:GetMicrovm; also dropped lambda:CreateMicrovmAuthToken from every role — see next point.

5 (JWE): Decided explicitly: P1–P3 have zero orchestrator→agent HTTP (payload via runHookPayload/S3-pointer, agent work is outbound-only), so no tokens are minted at all and the TTL-refresh problem is deferred until a real consumer exists (#391 shell access). endpoint stays in the handle as genuinely per-session state for that day.

6: Security-delta table added under sub-decision 4 (egress, session-role admit, secrets-at-/run, JWE-gated endpoint with no minted tokens, snapshot uniqueness, WAT posture shared with ECS, no SHELL_INGRESS).

7: pollSession now maps SUSPENDED → running iff task status is AWAITING_APPROVAL; any other combination is surfaced as an anomaly, not fail-fasted.

Nits: second-kill-switch note added; /suspend hook constrained to durable writes within the 60 s budget; disk ceiling flagged for quota confirmation in the implementing COMPUTE.md PR. Also corrected our own economics claim: under the 1 h gate ceiling (decision #6) the "weekend suspend" scenario can't occur — savings are bounded at ~1 h/gate, with suspend as the enabler for any future off-hours ceiling extension (§14.8).

Ready for re-review.

[krokoko]

Re-review of de599b6 — thanks @dreamorosi

Really thorough response to the first review. The HITL handshake rewrite, cancel-task precedent for inline best-effort resume, omit-idlePolicy pin, async /run, IAM (PassNetworkConnector / GetMicrovm, no CreateMicrovmAuthToken in P1–P3), JWE deferral, security-delta table, and SUSPENDED-iff-AWAITING_APPROVAL poll mapping all look solid. The economics correction to the 1 h Cedar gate ceiling is appreciated too.

Happy to approve once the two blockers below are addressed.

Blockers

1. Timeout-under-freeze accounting

Resume-at-wall-deadline does not make the agent deny at that deadline.

Monotonic freezes across suspend. After a grace suspend on a 300 s gate (~30 s elapsed), ~270 s remains on the agent timer. Resume at wall deadline − margin still leaves those ~270 s; deny fires roughly that long after resume — often minutes past the user’s gate window.

The EARS line (“resume so the monotonic timeout can fire”) understates that. Please pick one and write it down, e.g.:

• On /resume (or poll wakeup), the agent re-bases the gate on wall clock (created_at + timeout_s), or
• Document that wall “deadline” only starts resume, deny is resume_wall + mono_remaining, and stranded reconciler / UX must match, or
• Another path that keeps deny authority agent-side without the orchestrator writing decisions

Without that, P3 ships a Cedar timeout hole.

2. omit idlePolicy vs suspendedDurationSeconds (internal contradiction)

suspendedDurationSeconds lives inside idlePolicy. Omitting the block disables traffic auto-suspend but also means you cannot set that field.

EARS currently requires both omit-idlePolicy and suspendedDurationSeconds ≥ gate ceiling + margin. Could you resolve by something like:

• P1: omit idlePolicy; P3: pass idlePolicy with maxIdleDurationSeconds ≈ maximumDurationInSeconds (or 28800), autoResumeEnabled: false, and the sized suspendedDurationSeconds, or
• Drop the suspendedDurationSeconds requirement and rely on maximumDurationInSeconds + orchestrator TerminateMicrovm

Nits (non-blocking)

• Prose names ApproveTaskFn for resume; EARS says approve/deny — worth naming DenyTaskFn in the prose too.
• Sub-decision 1 still lists create-microvm-auth-token among lifecycle APIs while P1–P3 mint none — soft leftover.
• Confirm whether manual SuspendMicrovm without idlePolicy has any platform default suspend TTL (docs are thin); that feeds blocker 2.

Happy to re-review / approve as soon as the two blockers land. Refs #645.

[dreamorosi]

Thanks @krokoko — both blockers addressed in ca3e9e6. You were right on both counts: our round-1 timeout fix didn't do the arithmetic (the frozen monotonic budget survives resume), and we mandated omitting the very block suspendedDurationSeconds lives in.

Blocker 1 — your first option, grounded in two existing mechanisms: gate expiry is now min(monotonic budget, created_at + timeout_s), evaluated on each poll iteration and on /resume; the agent denies on expiry. We chose wall-clock re-base over documenting the late deny because the late deny doesn't just hurt UX — it races the approval row TTL (created_at + timeout_s + 120s) and would trigger the "row reaped → stranded" fallback on a healthy gate. The re-base extends patterns already in the doc rather than adding one: decision #6 is already min-wins for timeouts, and §13.12 already makes the durable row authoritative over the agent's local timer (the conditional TIMED_OUT write + ConsistentRead re-read is untouched). Since the agent writes created_at itself, both clocks are in the agent's own domain — no skew. The orchestrator's deadline − margin resume is now explicitly wake-up-only, with no correctness role.

Blocker 2 — your second option: idlePolicy is omitted in every phase, and the suspendedDurationSeconds requirement is dropped. The suspended-state bound is maximumDurationInSeconds (already mandatory, task-budget-derived, ≤ 8 h — and it bounds running and suspended time) + the orchestrator's finalization terminate-microvm + the stranded-approval reconciler. Two reasons we preferred this over the P1/P3 idlePolicy flip: it keeps one invariant meaning across phases (the invariant test always asserts omission), and it matches where this codebase puts lifecycle policy — in the orchestrator and reconcilers (stranded-approval reconciler, task_cancel_compute_orphan), not substrate config; ECS likewise has no substrate auto-stop. The blocker-1 fix also removes the main dangling-suspend scenario the TTL would have guarded: the orchestrator resumes at the wall deadline regardless of decision, so a long-dangling suspend requires a broken orchestrator — exactly what maximumDurationInSeconds bounds. A tighter substrate suspended-TTL is noted as an additive idlePolicy follow-up if operators want it, and the consequences section states the trade honestly.

Nits: DenyTaskFn now named alongside ApproveTaskFn (both handlers exist); create-microvm-auth-token annotated as not called in P1–P3 in the handle rationale; your manual-suspend default-TTL question is recorded as a P1 verification item in Testing — with maximumDurationInSeconds always set, the design is safe under either answer, so it's an empirical note for COMPUTE.md rather than a design dependency.

Ready for re-review.

[dreamorosi]

Round 3 (85ee85d): we ran an independent adversarial review of the ADR against the codebase before re-review. It found two more unimplementable normative statements that rounds 1–2 (ours and review) missed, plus several tightenings. All applied:

Fixed — would have blocked P1/P3 implementers:

1. pollSession can't see task state. The interface takes only the SessionHandle, so the SUSPENDED-iff-AWAITING_APPROVAL mapping was unimplementable where it was specified. Resolved with a reports/interprets split: SessionStatus gains a 'suspended' variant, the strategy maps GetMicrovm state mechanically, and the orchestrator cross-references against the task row — the same division of labor finalPollState (ECS substrate × DDB) and pollTaskStatus (agentcore heartbeats) already use. New EARS requirements cover both halves.
2. "maximumDurationInSeconds derived from the task budget" was undefined — platform budgets are max_turns/max_budget_usd; no wall-clock budget exists. Pinned at 28 800 s: parity with AgentCore's 8 h session cap (not new policy), inside the orchestrator's ~8.5 h safety-net poll window; Blueprint override deferred until a real need appears.

Also applied:

3. Corrected our cancel-precedent description: task_cancel_compute_orphan fires only on missing_runtime_handle, not on failed stops (failed stops warn only). The resume path intentionally goes one step further — orphan event on failed resume too — since a failed resume strands a suspended VM awaiting a decision.
4. Specified handle plumbing: startSession → compute_metadata (microvmId, endpoint; the field cancel-task.ts already reads ECS handles from); approve/deny do a post-commit strongly-consistent GetItem before the best-effort resume.
5. Specified the S3-pointer payload path per the ecs-agent-cluster.ts pattern (platform payload bucket, execution-role read-only grant, lifecycle expiry, URI-only in runHookPayload).
6. Split all compound EARS bullets into atomic single-pattern requirements per ADR-020; rationale moved to prose.
7. Flagged the stack-level compute_type tag in main.ts as incorrect with three backends; added a cost-allocation-tags requirement (per feat(compute): add AWS Lambda MicroVMs as a ComputeStrategy backend #645's attribution AC).
8. Marked the Lambda MicroVMs service facts as externally sourced and expanded the P1 empirical verification list (manual-suspend TTL, disk quota, runHookPayload limit, IAM action names, region probe, quota treatment of SUSPENDED).
9. Testing section split by rollout phase (P1/P2/P3) so P1 doesn't inherit suspend/resume test obligations.

Mirror re-synced. Ready for re-review.