feat(inference): let gateway mint avatar token - #2152
feat(inference): let gateway mint avatar token#2152rosetta-livekit-bot[bot] wants to merge 1 commit into
Conversation
🦋 Changeset detectedLatest commit: b6e5c09 The changes in this PR will be included in the next version bump. This PR includes changesets to release 39 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
shawnfeldman
left a comment
shawnfeldman
left a comment
There was a problem hiding this comment.
Reviewed at b6e5c094. No Critical or Important issues — approving. The port is faithful and the wire contract is right; everything below is a suggestion.
What I verified
Checked out the head SHA and ran it rather than trusting the description:
pnpm --filter @livekit/agents build— clean, including thetsc --declarationemit (so this typechecks).vitest run src/inference/avatar.test.ts— 36/36 pass.- Full package suite — 111 files, 1504 passed, 5 skipped, matching your numbers exactly.
eslinton both changed files — 0 problems. (The repo-wide141 problemsare pre-existing; the two files you touched are clean.)
Wire contract, field-by-field against the merged gateway (agent-gateway origin/main:pkg/handler/avatar.go, #1095 merged 17:54Z): the payload is exactly avatarRequest. Every field the gateway requires non-empty is guarded locally first — livekit_url at avatar.ts:199, room_name at :209, agent_identity at :229, and avatar_identity by the constructor default. avatar_name lines up with the gateway's name == "" → AvatarIdentity fallback, so the two agree in both directions. livekit_token is gone on both sides.
Attribute parity survives the move, which is the part worth being sure about: mintAvatarWorkerToken still stamps lk.publish_on_behalf and lk.avatar_provider and keeps kind=agent. That matters because room_io.ts:251 reads lk.publish_on_behalf to skip the avatar when picking the participant to listen to — a silent drop there would have the agent transcribing its own avatar.
Port fidelity to agents#6575 is 1:1 on the payload dict. Keeping livekitApiKey/livekitApiSecret where Python deleted them is the right call — Python has no RoomServiceClient SID fallback, so it never needed them — and the moved credential check is a strict relaxation: no configuration that used to succeed now fails.
Deleting the JWT-claims test is also correct. That coverage moved server-side into #1095's own tests, and the JS side keeps assertions on every input the gateway mints from. not.toHaveProperty('livekit_token') is a real regression guard.
Suggestions
1. The class TSDoc dropped the cross-project caveat that the Python counterpart added. agents/src/inference/avatar.ts:96-102
agents@0534f5e89's docstring ends with:
Because the gateway signs the worker token with the key it authenticated, the project that serves inference must be the project that owns the room — pointing
LIVEKIT_INFERENCE_*at a different project than the room is no longer supported.
That's a real consequence of this change: the SDK used to sign with the room's LIVEKIT_API_KEY, and the gateway now signs with whatever project it authenticated. A user with LIVEKIT_INFERENCE_* pointed at a different project than the room gets a worker token the room's server rejects, and the only symptom is waitForJoin() timing out. The JS class doc was rewritten in this PR but stops at "lets the gateway mint the avatar worker's room token," and the changeset doesn't mention it either. Since the endpoint is behind a default-off flag there's no installed base to break, so this is prospective guidance rather than a migration warning — but it's the one substantive thing the port left on the floor, and the "no source diff files were skipped" table reads as if it came across. Worth carrying that sentence (and ideally the lk.publish_on_behalf / lk.avatar_provider paragraph above it) into the TSDoc.
2. Nothing executes the code this PR moved. agents/src/inference/avatar.ts:527-533
The relocated credential check and the RoomServiceClient.listRooms fallback below it are both unreached by all 36 tests — the if (sid) return sid false branch is never taken. FakeConnectedRoom.getSid() returns 'RM_789', and the only fake returning '' is either disconnected (throws at :225 first) or has jobCtx.job.room.sid short-circuiting the || at :220. The listRooms half was already uncovered before this PR, but the credential throw is new, and it's the single place the JS diverges from the Python original.
Reachable cheaply via the job path: job: { room: { name: 'job-room' } } with no sid plus FakeJobRoom (inherited getSid() → ''), with LIVEKIT_API_KEY/LIVEKIT_API_SECRET deleted, asserting rejects.toThrow(/to resolve room sid/). A sibling test passing the keys and spying RoomServiceClient.prototype.listRooms closes the older gap too — the file already spies that prototype at avatar.test.ts:549.
3. avatar_identity normalization now differs between the SDK and the gateway. avatar.ts:156-158 vs the gateway's normalize()
The gateway does strings.TrimSpace(req.AvatarIdentity) and mints with the trimmed value; the SDK keeps its untrimmed copy for destinationIdentity (:267) and the join-wait. Before this PR both sides used the same untrimmed string because the SDK minted the token, so this is newly load-bearing. With whitespace in a configured avatarParticipantIdentity, the worker joins as bob while the SDK waits for ' bob ' — and waitForParticipant has no timeout, only an abort signal, so it hangs with no error and no log past a debug, while the provider session bills. Needs a pathological input, hence a suggestion, but it's zero-diagnosability when it does happen.
Cheap fix is .trim() in the constructor. The better one: the gateway already returns the authoritative avatar_identity in its 201 body, and CreateSessionResponse (:77-82) drops it — picking it up and warning on mismatch closes this whole class of drift, and would also cover suggestion 5.
4. livekit_url is asserted nowhere. avatar.test.ts:239-249
The string doesn't appear in the test file at all, only livekitUrl as an input. The gateway 400s without it, so a rename or drop of that payload key leaves the suite green and breaks every avatar create. Pre-existing, but you're rewriting exactly that toMatchObject block, so it's a one-line add.
5. Smaller things, all optional:
avatar.ts:184-186— the sync-guard comment's parenthetical still lists "token mint" as one of the awaits before the ids are assigned. The load-bearing claim is still correct; the example list is nowsuper.start()/ sid resolution / gateway create.avatar.ts:342-344— the new payload comment enumerates roomName / avatar identity / agent identity, but omitsprovider → lk.avatar_provider, which is precisely the attribute whose local setter this PR deletes. A reader can't tell it survived.avatar.ts:515-519—resolveRoomSidtakes the wholeAvatarSessionStartOptionsto read two members;Pick<AvatarSessionStartOptions, 'livekitApiKey' | 'livekitApiSecret'>narrows it without giving up the lazy env fallback.avatar.test.ts:413-425—'start without livekitUrl raises'passes nofetchoverride, so if that guard is ever reordered the test issues a real POST and fails as an opaque timeout instead of an assertion.expect(fetchMock).not.toHaveBeenCalled()pins it.avatar.ts:232-234— JS still hard-requiresroom_sidwhere the gateway marks itomitemptyand Python passes an empty string through. I'd leave this alone: the gateway forwardsRoomSIDinto LemonSlice'slivekit_session_idunconditionally and nothing confirms the provider tolerates empty. Worth a note to the gateway owner rather than a change here.
1 participant
Summary
Ports livekit/agents#6575 so the inference avatar gateway mints the avatar worker room token instead of the JS SDK signing it locally.
Changes
AvatarSession.start().room_name,avatar_identity,avatar_name,agent_identity) and omitlivekit_tokenfrom avatar session create payloads.livekitUrlfor start-time gateway session creation.@livekit/agents.Source diff coverage
livekit-agents/livekit/agents/inference/avatar.pyagents/src/inference/avatar.tsAccessTokenworker-token signing andlivekit_tokenpayload field; addedavatar_name; keptroom_name/avatar_identity/agent_identityas gateway mint inputs. JS-specific adaptation:livekitApiKey/livekitApiSecretstart options remain only for the existing@livekit/rtc-nodeRoomServiceClientroom-SID fallback whenroom.getSid()is empty; they no longer mint or sign the worker token.tests/test_inference_avatar.pyagents/src/inference/avatar.test.tslivekit_token, assert gateway mint inputs includingavatar_name, remove unnecessary room-project credentials from starts, and add the missing-livekitUrlfailure coverage.No source diff files were skipped.
Verification
pnpm test agentspassed: 111 files, 1504 tests passed, 5 skipped.pnpm --filter @livekit/agents buildpassed.pnpm --filter @livekit/agents lintcompleted with existing warnings.pnpm buildpassed.pnpm lintcompleted with existing warnings.git diff --checkpassed.Additional non-required check:
pnpm --filter @livekit/agents api:checkfails before evaluating this change because API Extractor 7.43.7 does not support the existingexport * as ___syntax inagents/dist/index.d.ts:10.Ported from livekit/agents#6575
Original PR description
What
inference.AvatarSessionminted the avatar worker's room token locally and posted it to the gateway aslivekit_token. The gateway now mints that token itself — livekit/agent-gateway#1095 — so this PR stops minting client-side and sends the fields the gateway mints from.These two PRs must land together. Against gateway #1095, a create that sends
livekit_tokenand omitsagent_identityis a 400. No live impact either way: the gateway endpoint is gated by the default-offavatar_lemonsliceflag, so no project can be provisioning avatars today.Why
Billing integrity. Server-side avatar-minutes metering keys on the
lk.avatar_providerparticipant attribute — and that attribute was being set by this code, in a token we signed. You cannot reconcile revenue against a value the customer's process controls. Same forroom_name/room_sid/avatar_identity: self-reported body fields with nothing tying them to the token actually used.Moving the mint server-side makes all of it gateway-authored. The gateway signs with the key it just authenticated the request on, so it grants nothing the caller could not already mint for itself.
The change
The gateway reproduces exactly that grant set:
kind=agent, the identity,roomJoinscoped to the one room, publish/subscribe left implicit, pluslk.publish_on_behalfandlk.avatar_provider.Payload changes:
livekit_tokenremoved;avatar_nameadded so a customavatar_participant_namestill reaches the minted token.room_name/avatar_identity/agent_identitywere already sent and are now load-bearing mint inputs rather than attribution.AvatarSession.start()no longer acceptslivekit_api_key/livekit_api_secret.AvatarSessionshipped in livekit-agents 1.6.7, so this is a break to a published surface, not an unreleased one.await avatar.start( session, room=ctx.room, livekit_url="wss://...", - livekit_api_key="...", - livekit_api_secret="...", )Nothing signs a token client-side any more, so only
livekit_urlremains — the provider still needs a URL to dial. Passing the removed arguments raisesTypeErrorat the call site.Dropping them rather than ignoring them is deliberate. The gateway signs with the inference project's key, so a room owned by a different project than
LIVEKIT_INFERENCE_*points at is no longer supported. That configuration used to work (we minted for the room project; the gateway just forwarded it). Silently ignoring the old arguments would turn it into an unexplained join failure minutes later; aTypeErrorsays what happened, where it happened.Testing
uv run pytest tests/test_inference_avatar.py --unit— 30 passed.ruff format/ruff checkclean;mypy -p livekit.agentsclean (205 files).test_start_mints_worker_token_with_expected_grantswithtest_start_sends_mint_inputs_and_no_token: assertslivekit_tokenis absent from the payload and that all four mint inputs (room_name,avatar_identity,avatar_name,agent_identity) are present, with a custom participant name flowing through asavatar_name.test_start_without_livekit_url_raises— the URL is still required; the room project's key/secret are not.start()call sites; removed the now-unusedjwtimport.Note for the reviewer (out of scope here)
examples/avatar/inference_agent.pycallsinference.AvatarSession("lemonslice", image_url=..., prompt=...), but those go insideextra_kwargs(LemonSliceOptions) — the example raisesTypeErroron the constructor. It's pre-existing (example and class landed together in #6492, so it was never run) and unrelated to this change, so I left it alone rather than widening the diff. Worth a separate one-line fix.🤖 Generated with Claude Code