feat(simulate): platform scenarios + LiveKit SIP transport + provider acceptance flows - #55
Open
azain-commits wants to merge 37 commits into
Open
Conversation
…iveKit SIP Prompt & simulation: - Add fi.simulate.simulation.voice_prompt with call-direction-aware, role-locked, personality/style-aware persona prompt policy adapted from platform. - Make LiveKit engine use the SDK prompt by default; treat simulator.instructions as a full override; drop the ad-hoc opener that could read scenario text aloud. - Retain SIP outbound path: sip_number/wait_until_answered/ringtone, PARTICIPANT_KIND_SIP subscription, typed sip_dial_failed and sip_inbound_no_participant failures. Platform scenario generation (opt-in, keyed): - Add fi.alk.studio._generate with PlatformScenarioRequest, GeneratedScenario, ScenarioGenerationError, ensure_platform_agent, generate_scenario, fetch_scenario. Idempotent stable-name Agent Definition create/reuse, scanned safe metadata upload, async Scenario poll + dataset-table pagination. - Upgrade fi.alk.studio._download with real dataset-table hydration (map_dataset_table_rows, fetch_dataset_rows, hydrate_platform_scenario) and a shared row parser; fail closed instead of fabricating rows. - Export new public API from fi.alk.studio. CLI & manifests: - Add `agent-learn scenario generate` command (local AgentDefinition or explicit platform IDs, JSON output). - Add scenario.platform block to fi.simulate.cli._build_scenario with local cache reuse; incompatible with source/dataset; runs generation off-thread. Runtime scaffolding & artifacts: - New fi.simulate.runtime, fi.simulate.artifacts, fi.simulate.environments, fi.simulate.evidence, fi.simulate.results supporting canonical local text, LiveKit, and cloud engines; refactor engines and recording accordingly. - Add examples/build_delivery_support_suite.py and .github/workflows/sdk-smoke.yml. Tests: - New tests/runtime/ suites (cli_smoke, livekit_engine, manifest_engine_dispatch, runtime_contracts, simulation_runner, delivery_support_suite). No new scenario-generation tests in this pass (tracked as follow-up debt).
…ades Follow-up to previous commit that only picked up the new files: - src/fi/alk/cli.py: add `agent-learn scenario generate` subcommand and wire it to studio.generate_scenario, including local AgentDefinition and explicit platform ID paths, JSON output writing, and structured refusal on failure. - src/fi/alk/studio/__init__.py: export the new generation public API (PlatformAgentReference, PlatformScenarioRequest, GeneratedScenario, ScenarioGenerationError, ensure_platform_agent, generate_scenario, fetch_scenario) alongside existing symbols. - src/fi/alk/studio/_download.py: real dataset-table pagination and hydration (map_dataset_table_rows, fetch_dataset_rows, hydrate_platform_scenario) plus a shared row parser reused by generate_scenario; pull_scenarios uses the actual platform contract and fails closed instead of fabricating rows. - src/fi/alk/simulate.py: expose scenario generation surface alongside existing simulate helpers. - src/fi/simulate/agent/definition.py: TelephonyTransport (webrtc / sip_outbound / sip_inbound) with E.164 validation and wait_until_answered semantics; AgentDefinition.transport field. - src/fi/simulate/cli.py: add scenario.platform block to _build_scenario with local cache reuse and off-thread execution; incompatible with scenario.source / scenario.dataset. - src/fi/simulate/manifest.py: ancillary manifest plumbing for the new scenario shape. - .gitignore/pyproject.toml/uv.lock: keep new package layout clean and pin dependency changes required by the runtime + livekit paths.
… Vertex switch
Land the missing pieces called out in the simulation SDK implementation plan and
the LiveKit Cloud provider work that follows from it:
- Protocols & media: endpoints/base.py, realtime/{media,events,session}, and
simulator/ define AgentEndpoint, RealtimeEndpoint, AudioFrame,
SimulatorPolicy, the canonical event vocabulary, and the media profile.
- Endpoint adapters: callable/http/websocket/livekit/retell + Vapi originator
helper in endpoints/vapi.py (POST /call, DELETE /call/{id}, env resolution).
- Evidence sources: providers/{vapi,retell} adapters plus caller_observed,
livekit_room, livekit_instrumentation, and otel evidence skeletons.
- Instrumentation: instrumentation/livekit/FutureAGIObserver skeleton.
- Result sink: results/futureagi.py serialises the plan's §11.2 route contract
locally when FUTURE_AGI_API_URL is unset.
- Matrix runner: simulation/matrix.py + matrix_cli.py drive provider×channel
sweeps and now surface each leg's SimulationReport status so evaluator score
no longer masks failed runs.
- Vertex/Gemini backend: livekit_models.py adds Google LLM/STT/TTS factories
(Vertex when GOOGLE_APPLICATION_CREDENTIALS + GOOGLE_CLOUD_PROJECT are set,
Gemini API when GEMINI_API_KEY is set). Split speech creds from LLM Vertex
kwargs; default TTS voice moves to en-US-Chirp3-HD-Kore for streaming.
- Package plumbing: livekit extras include livekit-plugins-google; public
re-exports for endpoints/realtime/simulator/results/evidence added.
- Manifest tests exercise the SIP transport + Vapi/Retell evidence contracts.
… failures Previously the LiveKit engine could not run against a real LiveKit Cloud project: outbound dial hit a spurious ServerError on premature RoomService CreateRoom, inbound routed the SIP participant into a randomly-suffixed room that the local simulator was not in, the AgentSession subscribed to the simulator's own participant instead of the SIP caller, cleanup deleted the room before the dispatch rule, and the failure surface collapsed every setup error into a single opaque code. Fixes: - Outbound: skip CreateRoom for sip_outbound (LiveKit auto-creates on join); connect the simulator RTC first, start its session, then create the SIP participant. Split answer_timeout_seconds from connect_timeout so PSTN answer latency stops surfacing as generic connect timeouts. - Inbound: _ensure_sip_inbound_dispatch now creates a per-case SIPDispatchRuleDirect(room_name=<exact case room>) bound to LIVEKIT_INBOUND_TRUNK_ID and validates a reused named rule by trunk + direct-rule type + destination room. Adds sip_inbound_route_conflict. - Participant subscription: _TestRunnerAgent.start_session accepts participant_kinds/participant_identity; for SIP the engine passes [PARTICIPANT_KIND_SIP] and the caller identity so the AgentSession locks onto the actual caller, not the simulator itself. - Vapi call originator: new inbound_call_originator="vapi" on TelephonyTransport plus call_id_source="originator_response" on ProviderEvidenceConfig. Engine drives VapiCallOriginator after the direct dispatch and session are ready, passes the returned Vapi call ID into _collect_provider_evidence as the explicit hint, and cancels the Vapi call on cleanup. - Cleanup order: delete SDK-owned dispatch rule before the room; retain not-found tolerance. - Failures: typed sanitized codes for room create, SIP dial, SIP dispatch, and Vapi call start (livekit_room_create_failed, sip_dial_failed, sip_answer_timeout, sip_inbound_dispatch_failed, sip_inbound_no_participant, vapi_call_start_failed, vapi_call_start_timeout). _safe_provider_error_details keeps only exception type, provider code, HTTP status. - Grade natural target hang-ups: if the target disconnected and both roles spoke, treat the case as completed instead of target_disconnected. - AgentDefinition._check_transport validates origin-only ws/wss URL scheme and rejects SIP transports with room_mode="external". Focused tests cover new SIP behaviours, Vapi originator request shape, and sanitized error propagation.
…+ port Platform voice prompt
Split AgentDefinition into a target-agent shape (Vapi/Retell provider
config, non-secret assistant/agent id, api_key_env) and a new
LiveKitSimulatorRuntime that owns FutureAGI's LiveKit URL, room, and
key/secret env names. Legacy AgentDefinition.url/room_name/room_mode
stay as a compatibility boundary. Provider bridges and evidence
adapters now read credentials from the target's api_key_env and the
target's API base URL instead of hard-coded VAPI_/RETELL_ globals.
Ported the complete production voice persona prompt and execution
rules from the Platform (test_executor + ee.voice guides) into
fi.simulate.simulation.voice_prompt, and made
SimulatorAgentDefinition.instructions additive on top of the
scenario-derived customer prompt instead of replacing it.
Platform registration for a direct target now records the actual
provider ("vapi" or "retell") with its non-secret id and sets
scenario_generation_only, so no provider secrets or FutureAGI LiveKit
runtime credentials are uploaded during scenario generation. The
Platform description is the target agent's system prompt only.
Renamed the primary Platform credential env vars to FI_API_KEY,
FI_SECRET_KEY, and FI_BASE_URL. FUTURE_AGI_* and AGENT_LEARNING_*
remain as compatibility aliases.
CLI, manifest builder, examples, and tests updated to consume the
explicit target/runtime shape.
…ning Simulator engine now honors LiveKitSimulatorRuntime.room_name_verbatim so runs can join a pre-existing dispatch rule bound to a fixed room, which is what unblocks the 10-slot inbound-simulator DID pool used by the production LiveKit matrix. Multi-persona runs guard against verbatim reuse. trigger_livekit_outbound short-circuits room discovery when the matrix runner provides an override. Runner-side hardening: split simulator utterances are merged before turn-count evaluation, endCall refuses to fire before both speakers have participated (min_turn_messages becomes a validation floor rather than a stop trigger), and empty / silent / short conversations now surface as failures instead of passing. Matrix runner returns non-zero on behavioural evaluation failure. Provider fixes bundled: Cartesia STT/TTS support, Google endpoint + STT + LINEAR16 TTS corrections, authenticated Bearer retrieval for Vapi private recordings with signed-URL redirects, Vapi tool arg/result capture, corrected recording paths and manifest metadata, migration off deprecated LiveKit room/turn options, and unique worker names to avoid stale registrations.
Replaces the deferred submission stub with a working HTTP client that posts a finished simulation run to the FutureAGI platform's ALK ingestion endpoints: start a test execution, allocate call executions, upload each recording, then PATCH the per-call result. The submitted payload carries only what the SDK directly observed — transcript (with per-message speech timing so the platform can recompute WPM, talk-ratio and interruptions), recording, provider call data, terminal status. Start/end and duration are derived from the observed speech timestamps when the engine does not stamp case-level times. Recordings are streamed to the platform as a multipart upload. Submission is env-gated (FI_BASE_URL / FI_API_KEY / FI_SECRET_KEY / FI_RUN_TEST_ID); with any missing it records not_configured and no HTTP is attempted, so local runs are unaffected. Adds oss/simulation-acceptance/run_platform_voice_case.py to run one voice acceptance case end-to-end and submit its report through the sink.
Extends the FutureAGI sink to pull the agent-under-test's provider-reported usage from case evidence and fold it into provider_call_data under the normalized usage.llm shape the platform reads, plus costs.cost_cents. Provider-agnostic: dispatches to a per-provider extractor since each reports differently — Vapi costBreakdown (llmPrompt/CompletionTokens, dollar cost), Retell call_cost + llm_token_usage (combined_cost in cents, total-only or split tokens), LiveKit normalized usage. This is the target agent's real usage, not the FutureAGI simulator's. Timing and transcript already flow for every target because they come from the shared LiveKit session, not the provider evidence.
Author
|
Pushed two commits extending this branch for the platform-submission path (ALK → FutureAGI voice ingestion):
Backend counterpart: future-agi#1976. |
Requirement-gated metrics (browser/multi-agent/orchestration/voice/etc.
coverage + quality) early-return a vacuous 1.0 when a case configures no
requirement for them, inflating the flat-mean aggregate toward ~0.94
regardless of real agent quality.
Add an 'applicable' flag to AgentReportMetricResult, classify these
unconfigured metrics as not-applicable (reason ends 'provided.' /
'configured.' / 'not required.'), and score only applicable metrics in
_weighted_average. Genuine safety passes ('No secret-like output
detected.', 'No unsafe memory writes.') stay applicable and counted.
…ites Eval-suite providers previously supported only offline stubs (echo, scripted, artifact, python_callable). Add a litellm-backed provider so suites can call a real LLM directly: type 'vertex'/'gemini' (bare model, auto-prefixed vertex_ai/) for Vertex AI, or 'litellm' with a fully-qualified model string for any other provider. Wired at the single _provider_output choke point, so both 'agent-learn eval' and 'agent-learn optimize-eval' pick it up. Vertex auth via GOOGLE_APPLICATION_CREDENTIALS; routing via vertex_project/vertex_location provider fields or VERTEXAI_* env.
MetaPrompt, ProTeGi, and PromptWizard hardcoded the task generator (the model that runs candidate prompts while scoring) to gpt-4o-mini/gpt-5-mini, forcing an OpenAI key even when the teacher generator was another provider. Add a task_model constructor arg (default None) that falls back to the teacher generator's model, so passing e.g. a Vertex teacher makes the whole optimizer run on Vertex. Backward-compatible: pass task_model to override.
Evals: add `fi_eval` suite assertion type that scores case output with any hosted FutureAGI eval template via fi.evals.evaluate (platform/turing engine, FI_* creds), pass = score >= threshold. Thread case vars into assertions and expose public evaluate_assertions(). Optimizers: register curriculum/pareto/feedback tokens in _optimizer_cls, and add a generative eval-suite bridge (generative_suite.py) so gepa/protegi/ metaprompt/promptwizard/random_search/bayesian_search run real LLM prompt rewriting from an eval suite, scored against the suite's own assertions. optimize_eval_suite routes generative tokens before the deterministic target.
- FutureAGIResultSink: emit tool_calls/tool_call_result transcript
segments (assistant tool-call turns carry empty content and were
dropped) plus per-turn latency
- chat environment: stamp wall-clock agent latency per turn so the
platform's avg_latency_ms populates for every target type
- HTTPAgentWrapper._openai_tool_spec: accept OpenAI-nested tool specs
({"function": {...}}); flat-only parsing handed the model a tool named
"tool", breaking mock/tool matching
…nd canon mirror - one runtime dispatch: environment/target/simulator resolved through registries (register_environment / register_simulator / endpoint profiles) instead of hardcoded branches; planner validates against the registered vocabulary - world_kind is a faithful mirror of the frozen SIMULATION_WORLD_KINDS canon (conversation/tool_api/browser/computer_use/code_exec/voice_telephony); admission label, not an engine selector - voice enters the same SimulationSpec spine (legacy LiveKit engine wrapped) - facade exports (fi.alk.simulate) surface the gym-model types
…argets) The SDK side of the platform-triggered runner: the child the backend spawns. - fi/simulate/hosted: StartRunnerJob schema, child_entrypoint (reads the job, builds target/voice spec, runs SimulationRunner + FutureAGIResultSink, heartbeats + graceful SIGTERM), targets.resolve_chat_target (deny caller code in hosted; http/websocket only, operator default behind an explicit opt-in) - FutureAGIResultSink already accepts a pre-created test_execution_id, so the child submits into the execution the platform created - tests: offline child chat run + pre-created-execution routing - oss run_e2e_refactor.py: end-to-end demo over the refactored path
Persona 'voice'/'voice_id' overrides the simulator TTS voice (Deepgram -> aura model; other providers -> voice field), falling back to the global default.
Hosted-runner result submission was authenticating only with the tenant x-api-key/x-secret-key, which 403s when the runner's FI key doesn't match the pre-created TestExecution's org. When an internal secret is present (FI_INTERNAL_SUBMIT_SECRET / ALK_RUNNER_INTERNAL_SECRET / INTERNAL_API_SECRET) the sink now also sends Authorization: Bearer <secret>, which the platform's InternalServiceAuthentication accepts and authorizes against the execution's org. The FI keys still travel with the request for downstream tenant context. This wiring previously existed only as an uncommitted wheel patch baked into the dev runner image; a clean rebuild from source reverted it. Committing it so the built wheel carries it.
The simulator's post-endpoint LLM generation drove a 3-5s inter-turn gap (measured: 3.07s first reply, of which VAD/endpointing was only 0.55s; the LLM TTFT ~1.12s ran entirely after end-of-turn). Enabling LiveKit preemptive generation overlaps the LLM with the speech tail, cutting the same path to ~0.88s and turning silence-failed runs into sustained multi-turn exchanges. This flips the engine default; _resolve_preemptive_generation plumbing already exists so a caller can still override per-run.
…pacity The LiveKit engine ran dataset cases strictly sequentially, so an N-case run took N x one-case wall-clock. Run them concurrently up to max_concurrency (default 1; SIP legs forced serial via profile.is_sip since a run leases one DID). gather preserves dataset order, so report.results keeps the positional contract the FutureAGI sink relies on; a crashing case yields a dense failed result in its own slot rather than a hole. Threaded through run_voice_simulation.
…ishes The hosted runner sink PATCHed every CallExecution at run end, so the platform showed no progress until the whole run completed. Stream each case the moment it finishes instead — rows land one-by-one mid-run, and a killed or timed-out job keeps every completed case instead of submitting nothing. SDK-only: the ingestion API already rolls up + notifies per PATCH and only marks the TestExecution terminal once all calls are terminal. - report: extract SimulationTestCaseResult.from_legacy_case so a streamed case is byte-identical to the finalized one - engine: fire on_case_complete(index, case) outside the case semaphore, on both the success and dense-FAILED exits - thread the callback engine -> run_test -> run_voice_simulation -> voice plugin (attaches per-case goal_machine) -> runner - runner: _begin_streaming binds the sink callback (from_legacy_case + to_thread) and passes on_case_complete to plugin.run; chat/base accept + ignore it - sink: begin_stream (hosted-only gate, allocate rows up front, persistent thread-safe client) / submit_case (PATCH by index, records failures) / finalize_stream (reconcile un-streamed; status=failed only when none landed) - local and chat runs keep the batch-at-end path unchanged Tests: engine callback per index incl. the failed slot; sink lifecycle incl. reconcile-of-missed, all-failed->failed, and the runner streaming closure.
Two hosted-voice fixes surfaced by a live run. 1. Conversation no longer cuts off at the message floor. The old `minimum_messages_reached` backstop ended a call once it hit `min_turn_messages` plus a 5s lull — shorter than a normal voice turn-gap (STT endpoint + LLM + TTS), so it truncated conversations that were not finished. Removed it. A call now ends on the simulator's `endCall`, a disconnect, `max_seconds`, or a genuine 60s stretch of mutual silence (`_wait_for_conversation_silence`, count-independent, resets on every turn and while either side speaks). New ended_reason `conversation_settled` classifies COMPLETED. `min_turn_messages` is kept only as the endCall-eligibility and too-short-failure floor. 2. Eval role reversal (agent <-> customer). The transcript is stored correctly (assistant = tested agent, user = simulator), but a black-box LiveKit target surfaces no usage evidence, so `provider_call_data` was empty and the platform resolver fell back to VAPI + inbound, which swaps the labels. Stamp a truthy `provider_call_data["livekit"]` marker on every LiveKit-engine run so the platform detects LiveKit, whose role map is direction-independent and correct. Tests: silence backstop fires on quiet regardless of count and holds while speaking; the end race returns `conversation_settled`; that reason classifies COMPLETED; a LiveKit run stamps the provider marker.
…by default A target agent built from a LiveKit template branches on ctx.job.metadata: any non-empty payload flips it into an outbound/no-greet job, so it never publishes an audio track and the simulator's readiness wait times out (agent_unavailable). The engine was always injecting simulation context (run id, test-case id, simulator identity, target_instructions) into the target's dispatch, which broke every real third-party agent while the FutureAGI reference agent (built to read that metadata) still passed. Default the dispatch metadata to an empty string via _dispatch_metadata_json; add an opt-in AgentDefinition.dispatch_metadata for targets that are built to consume a payload. Nothing reads target_instructions in-repo.
…nversation Hosted runs force record_audio=True, so the recorder joins the room before the simulator's AgentSession. LiveKit RoomIO auto-links to the first participant (the silent recorder) and never re-considers the target, so the simulator's STT heard nothing: only the simulator's opening turn was recorded and every run died as insufficient_conversation. After readiness selects the target, relink RoomIO to it, and consume the target agent's authoritative lk.transcription stream — feeding each utterance into the simulator via generate_reply(user_input=...) as a user turn, and disabling the now-redundant simulator STT so it cannot emit duplicate turns. Live-verified: one-sided 1-turn runs became full 10-23 turn conversations that complete.
In agent_first the target agent greets as soon as it joins, but the lk.transcription handler was only registered after _wait_for_target_audio. LiveKit RTC discards a text-stream header that arrives with no registered handler (no replay), so the greeting was lost, the simulator never received a turn, and conversations stalled at 1-5 messages -> insufficient_conversation. Register the transcription handler right after room.connect() for agent_first, defer the managed target dispatch until the handler exists, and buffer any stream that arrives before session/target readiness (drained once ready). The simulator_first path keeps its original dispatch + registration + open order.
…t greeting" This reverts commit 1fc2a40.
In agent_first the target greets on join, but the lk.transcription handler was registered only after readiness, and the LiveKit client drops a text-stream header that arrives with no handler — so the greeting was lost, the simulator never got a turn, and the call dead-aired. For managed external-room agent_first only: register an early buffer handler right after room.connect, defer the target dispatch until the buffer + session are live, and drain the buffered greeting through the (unchanged, unconditional) main handler once the target is selected. The buffer handler disables the simulator STT on the first target-attributable stream to kill the duplicate-reply race at the source; the buffer is bounded; the unregister→ register swap has no await between it. simulator_first + Vapi/Retell web-bridge paths are unchanged.
hadarishav
requested changes
Aug 13, 2026
hadarishav
left a comment
hadarishav
left a comment
Contributor
There was a problem hiding this comment.
Two blocking items
Reviewed at b93f2c9. The architecture holds up — I checked the decoupling goals explicitly and all four pass: the SDK runs standalone (the sink records not_configured and returns cleanly with no HTTP when platform keys are absent), there are no backend imports anywhere in src/fi/, the platform spawns the SDK as a child rather than reimplementing it, and hosted runs write the same local artifacts as a laptop run. That was the point of the re-architecture and it landed.
Blocking on two things, both small relative to the PR:
1. Five of this PR's own engine tests fail at HEAD. tests/runtime/test_livekit_engine.py is new here (1,915 lines) and covers the headline feature. The trend is in your favour — 10 failing at d1208ce, down to 5 now — but the suite has been red for six commits.
2. CI cannot see it. This PR adds .github/workflows/sdk-smoke.yml, which is a real improvement over having no CI at all. But it installs the package and runs one text simulation — no pytest, no lint. That is how a suite stays red for six commits without anyone noticing, and it means the ~3,000 lines of tests added here never execute automatically.
Details inline. Non-blocking observations (stale PR body — it still describes "nine commits / ~19k lines / 111 files" against an actual 27 / 26,652 / 146; PR size; cli.py +3,587 on an already-33k-line module) I'll leave out of the blocking set.
| calls = [] | ||
| audio_kind = livekit.rtc.TrackKind.KIND_AUDIO | ||
|
|
||
| class FakeRoom: |
Contributor
There was a problem hiding this comment.
FakeRoom is missing a method the engine now calls, failing 5 tests.
Un-redacting the error (the redacted_exc_info wrapper hides it, so I patched it out locally):
AttributeError: 'FakeRoom' object has no attribute 'register_text_stream_handler'
at src/fi/simulate/simulation/engines/livekit.py:953
Failing:
test_managed_case_dispatches_waits_and_cleans_uptest_sip_outbound_dials_per_case_room_and_identitytest_web_bridge_joins_as_target_without_sip(all 3 params)
I confirmed the real rtc.Room does have register_text_stream_handler (livekit-rtc 1.1.14), so this is a stale double rather than a production bug — b93f2c9 added the call and the fake wasn't updated. Low risk to runtime, but it means the engine suite currently validates nothing about the engine.
Stubbing register_text_stream_handler and unregister_text_stream_handler (used at livekit.py:1021) on FakeRoom should take it green.
| vapi_originator = VapiCallOriginator.from_env() | ||
| vapi_call = await asyncio.wait_for( | ||
| vapi_originator.start(), timeout=connect_timeout | ||
| ) |
Contributor
There was a problem hiding this comment.
This is the call the test double doesn't implement — the source of the 5 engine-test failures (see the comment on FakeRoom).
Worth noting the fix this line belongs to is the strongest thing in the PR: identifying that forcing record_audio=True makes the recorder join first, RoomIO auto-links to that silent participant and never re-considers the target, so the simulator's STT hears nothing and runs die as insufficient_conversation. That matches exactly what we were seeing from the outside — one-sided transcripts and stalled runs. Good catch, and the relink-plus-consume-lk.transcription approach is the right fix.
Only the test double needs to catch up.
| run: >- | ||
| python -c "import json, pathlib; | ||
| report=json.loads(pathlib.Path('smoke-report.json').read_text()); | ||
| assert report['status'] == 'ran' and report['report']['results']" |
Contributor
There was a problem hiding this comment.
The workflow doesn't run the tests this PR adds.
Credit first: this file is new in this PR, so it takes the repo from no CI to some CI. That's the right direction.
But as written it installs the package, runs agent-learn doctor, and executes one local text simulation. No pytest, no lint. So roughly 3,000 lines of new tests — test_livekit_engine.py (1,915), test_manifest_engine_dispatch.py (521), test_acceptance_regressions.py (490) — never execute in CI.
That's the mechanism behind the five failures above surviving six commits: nothing was watching.
A step like:
- name: Run tests
run: |
python -m pip install pytest pytest-asyncio
python -m pytest tests/ -qwould close it. Two caveats from running the suite locally: tests/test_config_and_facades.py hangs (>3 min, also on the base branch — pre-existing, worth an --ignore or a timeout), and tests/test_phase7_persona_studio.py has 2 network-dependent failures that also reproduce on base. Neither is introduced here, but both need handling before a pytest gate goes green.
Produce a 2-channel WAV (ch0 simulator/customer, ch1 target/assistant) alongside the mono mix, upload it, and set stereo_recording_url. mix_recordings_stereo mirrors mix_recordings (same-rate enforce, zero-pad, int16 clip). The platform serializer + UI already split the stereo file into assistant/customer channels.
…ngual) StreamingGeminiTTS subclasses the beta gemini_tts plugin but consumes generate_content_stream, pushing frames as they arrive (~1.35s first-byte) over the Vertex genai endpoint. Registered as the gemini/gemini_tts TTS provider; forces the global endpoint for gemini-3 and guards the model string. Cloud TTS needs an API the Vertex SA lacks, deepgram has no Arabic, and openai is unavailable — Gemini streaming TTS is multilingual (incl Arabic) on the creds we already have. Avoids the whole-stack livekit-agents 1.6.x bump that the upstream streaming plugin requires.
… emit per-channel recordings
Voice-sim transcript + recording fidelity for the LiveKit engine:
- Record the target agent's final utterance. Its transcription stream is
often still in flight when the call ends; the snapshot ran first and the
finally-block then cancelled it, dropping the last turn. Now signal a
conversation-ended flag, drain the pending target-transcription tasks
(bounded), and commit the final turn via session.history.add_message so it
lands in the transcript WITHOUT triggering another simulator reply.
- Truncate interrupted turns to what was actually spoken. text_output was
False, which stopped RoomIO building the TranscriptSynchronizer that aligns
the transcript to audio playback; an interrupted turn therefore showed the
full LLM text. Enable text_output so the synchronizer truncates on
interruption (playback-timing estimate). Deliberately NOT using
use_tts_aligned_transcript: neither Deepgram nor the Gemini voice emits word
timing, so it would drop turns instead of truncating.
- Emit per-channel assistant/customer recordings. The engine already produced
the per-speaker mono WAVs (audio_input=customer, audio_output=assistant);
upload them and attach to provider_call_data.livekit.recording.{customer,
assistant} so evals mapped to call.assistant_recording /
call.customer_recording resolve (previously only combined + stereo existed).
…s captured The target agent's final turn (its closing after the customer is already done) was dropped from the transcript even though its audio was recorded. Target turns were only recorded as a side effect of session.generate_reply(user_input=), which drops the turn (or raises 'AgentSession is closing') when the simulator won't/can't reply. Record every target turn straight onto session.history via add_message first, then elicit a simulator reply only while the conversation is live. Guarantees trailing target turns land in the transcript.
… draining session
The target's closing (delivered after the customer is done) was recorded only
by feeding it through the simulator AgentSession. But once the session starts
draining it rejects new input ('speech scheduling is paused'), so the closing
was dropped from the transcript while still recorded in the audio.
Capture every target utterance straight off its transcription stream into an
independent list, drain those tasks after the end condition until the target
stays quiet (bounded 30s) so a late/long closing completes, then merge any
trailing target turns into the report (deduped). The target's final turn now
lands in the transcript regardless of session state.
…apshotting A LiveKit assistant turn only commits to history once its TTS playback finishes. When the simulator ends the call (endCall), the wait returns before that playback completes, so the simulator's own last turn was snapshotted away. Extend the end-of-conversation drain to also wait while the session is still speaking, not just while target transcriptions are in flight.
…ly ends The room was never deleted when the conversation ended, so after one side hung up the other agent kept monologuing into a dead room — its audio was recorded by the room recorder but couldn't be transcribed (the simulator had already disconnected), leaving audio and transcript inconsistent. On end, wait briefly for the party that just spoke to commit its own final turn, then delete the room (kicking the target). The recording now ends when the call ends and matches the transcript.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Nine commits that take fi.simulate from local-only, LiveKit-hard-coded runs to a full provider-agnostic acceptance surface (LiveKit + Vapi + Retell) with platform-driven scenario generation and prod-hardened SIP support. ~19k lines added / 2.3k removed across 111 files.
What lands
ee69932):fi.simulate.simulation.voice_promptwith call-direction-aware, role-locked persona policy;fi.alk.studio._generate(idempotent Agent Definition create/reuse, safe metadata upload, async Scenario poll + dataset-table pagination); scenario cache; runtime/artifacts/environments/evidence/results scaffolding.67033fa):agent-learn scenario generate, real dataset-table hydration in_download,TelephonyTransportwith E.164 validation andwait_until_answeredsemantics.c063404): endpoints/realtime/simulator/results namespaces, Vapi + Retell evidence adapters, LiveKit instrumentation skeleton,simulation/matrix.py+matrix_cli.py(surfaces per-leg status so evaluator score no longer masks failed runs), Vertex + Gemini backends.e16933a): skip prematureCreateRoomforsip_outbound; per-caseSIPDispatchRuleDirectbound toLIVEKIT_INBOUND_TRUNK_ID;AgentSessionlocks onto SIP caller viaparticipant_kinds=[SIP]+ identity; typed sanitized failure codes; grade natural target hang-ups as completed when both roles spoke.3cf8c8f): direct-target flow support.963a399): splitAgentDefinitioninto target-agent shape andLiveKitSimulatorRuntime; renamed primary Platform creds toFI_API_KEY/FI_SECRET_KEY/FI_BASE_URL(legacy aliases retained); scenario-generation-only platform registration excludes provider secrets and runtime creds.35d6525): studio fix.d2b1420): full sweep flow wiring.b53e4a7):LiveKitSimulatorRuntime.room_name_verbatimhonored by_resolve_room_nameso runs join a pre-existing dispatch rule bound to a fixed room (unblocks livekit-infra DID pool). Multi-persona runs guard against verbatim reuse.trigger_livekit_outboundshort-circuits room discovery onACCEPTANCE_ROOM_NAME_OVERRIDE.min_turn_messagesbecomes validation floor; empty/silent/short conversations now fail correctly; matrix runner non-zero on eval failure; split simulator utterances merged before turn-count.Test plan
pytest tests/runtime/test_livekit_engine.py tests/test_acceptance_regressions.py tests/test_voice_simulation.py tests/test_acceptance_run_voice_case.py tests/test_acceptance_trigger.py tests/test_acceptance_voice_cases.py— 73 passed.run_prod_livekit_matrix.py --case 1.1.1 --case 1.2.1 --use-pool) — both cases pass against production GKE LiveKit.agent-learn scenario generateagainst staging Platform (idempotent Agent Definition path).FI_API_KEY/FI_SECRET_KEYalias fallbacks (FUTURE_AGI_*,AGENT_LEARNING_*) still resolve for existing tooling.