From 171959e7ce941ac873af5f19d8710472db5f6166 Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Wed, 29 Jul 2026 00:00:22 -0400 Subject: [PATCH 01/11] docs: design spec for EVA aai assistant server Additive integration: a shared WebSocketBridgeAssistantServer base (Twilio framing, 20ms pacing, buffer sync, latency/turn logging) plus an aai host-mode server that injects the domain prompt + tool schemas via config.host and relays tool_call -> tool_result so EVA's ToolExecutor mutates the scenario database. Existing five servers untouched. Co-Authored-By: Claude Opus 5 (1M context) --- ...6-07-28-eva-aai-assistant-server-design.md | 242 ++++++++++++++++++ 1 file changed, 242 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-28-eva-aai-assistant-server-design.md diff --git a/docs/superpowers/specs/2026-07-28-eva-aai-assistant-server-design.md b/docs/superpowers/specs/2026-07-28-eva-aai-assistant-server-design.md new file mode 100644 index 00000000..1ed1226f --- /dev/null +++ b/docs/superpowers/specs/2026-07-28-eva-aai-assistant-server-design.md @@ -0,0 +1,242 @@ +# EVA assistant server for the aai voice-agent framework + +**Date:** 2026-07-28 +**Status:** Approved — ready for implementation planning +**Repos:** `~/Code/eva` (this change), `~/Code/aai` (host mode, already implemented) +**Prior art:** `~/Code/tau2-bench` `aai` audio-native provider + +## Goal + +Evaluate the **aai** voice-agent framework on EVA, scoring it on both EVA-A (accuracy) +and EVA-X (experience) across EVA's existing domains and scenarios. + +EVA integrates a system under test as an **assistant server**: a process that exposes a +Twilio-framed WebSocket which EVA's user simulator calls into, and that writes a fixed +set of output files the metrics read. `docs/assistant_server_contract.md` is the +normative reference. This spec adds such a server for aai, plus a reusable base that a +second WebSocket-backed integration can sit on later. + +## Why aai host mode is the integration point + +aai agents are normally defined in TypeScript and execute their tools **server-side in a +sandbox** against `ctx.kv`. EVA requires the inverse: the agent runs the *domain's* +prompt and tool *schemas*, while **EVA's `ToolExecutor` executes the tools** against the +scenario database. + +This matters because EVA's accuracy metrics diff `initial_scenario_db.json` against +`final_scenario_db.json`. If aai executed tools internally, the scenario database would +never mutate and every task-completion metric would read as "nothing happened". + +aai's **host mode** (`packages/aai/host/host-mode.ts`) bridges this: the client's first +`config` frame carries `host{systemPrompt, greeting, tools}`, aai builds a fresh +single-use `Runtime` from it, and each tool call is relayed to the client as a +`tool_call` frame that resolves when the matching `tool_result` arrives. This is the same +mechanism the tau2 `aai` provider drives. + +## Confirmed facts (from code inspection) + +**aai side** (`~/Code/aai/agent`): +- WS endpoint `ws://localhost:3000/websocket`; host mode selected by `?host=1` + (`wantsHostMode(rawUrl)` in `packages/aai-server/orchestrator.ts`). +- Host mode is gated by `AAI_ALLOW_HOST` (`isHostAllowed`, `packages/aai/host/host-mode.ts`). +- Handshake: client sends `config` with `host{systemPrompt, greeting?, tools[]}` plus + `audioFormat:"pcm16"`, `sampleRate`, `ttsSampleRate`; schema in `packages/aai/sdk/protocol.ts`. +- Client→server `tool_result{toolCallId, result, error?}`; `result` is a string on the wire + and a JSON string is unwrapped before reaching the model. +- Audio: binary PCM16 frames, **16000 Hz in / 24000 Hz out** by default. +- Server→client JSON events: `config`, `speech_started`, `speech_stopped`, + `user_transcript`, `agent_transcript`, `tool_call`, `tool_call_done`, `reply_done`, + `audio_done`, `cancelled`, `reset`, `idle_timeout`, `error`, `custom_event`. +- Host agents default to `maxSteps` 30, so multi-tool turns are supported. +- **No usage/token events exist.** + +**EVA side** (`~/Code/eva`): +- `AbstractAssistantServer` (`src/eva/assistant/base_server.py`, 368 lines) provides + `audit_log`, `tool_handler`, `execute_tool()`, the three audio buffers, and + `save_outputs()`. Subclasses implement `start()` and `stop()`. +- The user simulator speaks Twilio Media Streams framing: μ-law 8 kHz, plus + `user_speech_start` / `user_speech_stop` events carrying wall-clock timestamps. +- `src/eva/assistant/audio_bridge.py` already provides `mulaw_8k_to_pcm16_16k` and + `pcm16_24k_to_mulaw_8k` — exactly aai's rates — along with `sync_buffer_to_position`, + `FrameworkLogWriter`, and `MetricsLogWriter`. +- Framework selection is a `Literal` at `src/eva/models/config.py:506`; the server + registry is `_get_server_class()` at `src/eva/orchestrator/worker.py:26`. +- Prompt and tool definitions come from `AgentConfig` (`configs/agents/*.yaml`), so no + per-domain work is needed — airline, itsm, and medical_hr all work from one server. +- The existing five servers each hand-roll the Twilio bridge and pacing loop + (600–850 lines apiece). +- **No `.env` is present locally**, so a live end-to-end run cannot be performed as part + of this work. + +## Architecture + +``` +user simulator ──Twilio μ-law 8k──▶ AAIAssistantServer ──PCM16 16k───▶ aai host +(ElevenLabs / ◀──Twilio μ-law 8k── localhost:{port}/ws ◀──PCM16 24k── :3000/websocket?host=1 + OpenAI caller) │ + execute_tool() + ▼ + scenario database +``` + +Two WebSocket hops with the server in the middle. Audio converts at each boundary; tool +calls travel inward from aai and resolve against EVA's scenario database. + +**Note the inversion from tau2.** In tau2 the aai code is a *client* pulled along by a +discrete-time tick loop. In EVA it is a *server* between two live sockets, responsible +for pacing its own output at wall-clock 20 ms / 160-byte cadence — the user simulator +infers turn boundaries from arrival timing, so pacing errors corrupt the evaluation +rather than merely sounding wrong. What ports from tau2 is the protocol layer, not the +driving loop. + +### aai host provisioning + +EVA connects out to an already-running host at `AAI_WS_URL` +(default `ws://localhost:3000/websocket?host=1`), started separately with +`AAI_ALLOW_HOST=1`. Because host mode builds a fresh single-use `Runtime` per +connection, EVA's `max_concurrent_conversations` maps to N parallel host sessions +without further coordination. + +Rejected: having EVA spawn the aai server as a subprocess, or as a compose service. +Both place Node lifecycle and port allocation inside a Python eval worker for no +evaluation benefit. + +## Components + +All work is **additive**. The five existing servers are left untouched so that +leaderboard-validated pipelines cannot regress. + +### New modules + +| File | Purpose | +|---|---| +| `src/eva/assistant/ws_bridge_server.py` | `WebSocketBridgeAssistantServer` — shared base: `start`/`stop`, Twilio parse/encode, 20 ms pacing task, buffer sync, turn bookkeeping, latency and audit writes. Protocol-agnostic. | +| `src/eva/assistant/aai_events.py` | aai protocol event models + `parse_aai_event`, ported from tau2's `events.py`. | +| `src/eva/assistant/aai_session.py` | `AAIHostSession` — WS client implementing the backend seam: host handshake, binary audio, `tool_result` relay. | +| `src/eva/assistant/aai_server.py` | `AAIAssistantServer` — builds the system prompt from `AgentConfig` and the flat aai tool schemas. Thin. | + +Flat modules rather than a package, matching EVA's existing `*_server.py` convention. + +### Edits + +| File | Change | +|---|---| +| `src/eva/models/config.py:506` | Add `"aai"` to the `framework` Literal and its description. | +| `src/eva/orchestrator/worker.py` | Add a lazy-import branch to `_get_server_class()` and update the `Supported:` error string. | +| `.env.example` | Document `AAI_WS_URL` and the `AAI_ALLOW_HOST` requirement. | +| `docs/aai_integration.md` *(new)* | Setup, run command, and known gaps. | + +No changes to `configs/agents/*.yaml`. + +## The backend seam + +```python +class VoiceBackendSession(Protocol): + backend_input_rate: int # 16000 for aai + backend_output_rate: int # 24000 for aai + + async def send_audio(self, pcm: bytes) -> None: ... + async def send_tool_result(self, call_id: str, result: dict) -> None: ... + def events(self) -> AsyncIterator[BridgeEvent]: ... + async def aclose(self) -> None: ... +``` + +`BridgeEvent` is a union of `AudioChunk`, `AssistantTranscript`, `UserTranscript`, +`ToolCall`, `SpeechStarted`, `TurnDone`, and `BackendError`. + +The base selects audio converters from the declared sample rates and owns every +contract-shaped concern; a subclass owns only its wire protocol. Adding a second +WebSocket-backed integration is therefore one new `*_session.py` plus a small server +subclass, with no base changes. + +Each unit is independently testable: `aai_events` is pure parsing, `aai_session` is +protocol framing over a socket, `ws_bridge_server` is the EVA contract over an abstract +session, and `aai_server` is prompt and schema construction. + +## Event mapping + +| aai event | Bridge event | Contract action | +|---|---|---| +| binary frame | `AudioChunk` | On first chunk of a turn: `fw_log.turn_start()` and `write_latency("model_response", …)` (bounds-checked `0 < ms < 30_000`). Convert 24 k → μ-law, enqueue for paced send. `sync_buffer_to_position(user_audio_buffer, …)` then extend `assistant_audio_buffer`. | +| `agent_transcript` | `AssistantTranscript` | Accumulate with full-text-overwrite semantics, as tau2 does. | +| `user_transcript` | `UserTranscript` | `append_user_input(text, timestamp_ms=)`. | +| `tool_call` | `ToolCall` | `await self.execute_tool(name, args)` → `send_tool_result`. | +| `speech_started` | `SpeechStarted` | Barge-in: drain the pacing queue, `turn_end(was_interrupted=True)`, record the partial transcript suffixed `[interrupted]`. | +| `reply_done` / `audio_done` | `TurnDone` | `append_assistant_output(text)`, `fw_log.llm_response(text)`, `turn_end(False)`. | +| `error` | `BackendError` | Log. | +| `idle_timeout` | `BackendError` | Log and end the session. | +| `tool_call_done`, `cancelled`, `reset`, `custom_event`, `speech_stopped` | — | Parsed, no contract action. | + +### Timestamps and VAD + +Two VAD systems coexist: the simulator's (`user_speech_start` / `user_speech_stop`, with +wall-clock timestamps) and aai's (`speech_started` / `speech_stopped`). The contract +specifies the simulator's `user_speech_stop` as the latency denominator, so that remains +the sole source for `model_response` latency. aai's events drive barge-in only. Mixing +the two would silently skew `model_response_latency`. + +### Greeting + +`host.greeting` is set to `get_initial_message(language)`, so aai opens the call. This +matches EVA's expectation that the assistant speaks first. + +### Tools + +`self.agent.tools` maps to aai's flat `ToolSchema` shape — +`{name, description, parameters}` with `parameters` as JSON Schema. Analogous to +`_build_realtime_tools()` in `openai_realtime_server.py`, but flat rather than nested. + +## Decisions and known gaps + +- **No token usage.** aai emits no usage events, so `write_token_usage()` is omitted. + `pipecat_metrics.jsonl` is still written with latency entries, so + `ConversationResult.model_response_latency` populates normally. Documented as a gap. +- **Model identity.** The contract requires `s2s_params["model"]`. Configuration is + `s2s: aai-host` with `s2s_params: {model: "aai-host", ws_url: …}`. +- **Relay tool timeout.** aai rejects a relayed call after + `DEFAULT_RELAY_TOOL_TIMEOUT_MS`. EVA tool execution is local and fast, so the default + stands; a timeout surfaces as a tool error and the turn continues. + +## Error handling + +- **Handshake failure** (host mode disabled, bad config): fail `start()` loudly with the + server's rejection reason. A silent fallback would produce a scored-but-meaningless run. +- **Backend disconnect mid-conversation**: end the session, still call `save_outputs()` + so partial artifacts and the scenario database are written for inspection. +- **Malformed events**: `parse_aai_event` yields an unknown-event model that is logged + and skipped, never raised — one unrecognized frame must not abort a conversation. +- **Tool execution failure**: `execute_tool()` already records the error result in the + audit log; relay it to aai as a `tool_result` error so the model can recover. + +## Testing + +Unit tests in `tests/unit/assistant/`: + +- `test_aai_events.py` — event parsing, ported from tau2. +- `test_aai_session.py` — handshake frame contents, `tool_result` framing, relay timeout. +- `test_ws_bridge_server.py` — Twilio round-trip, pacing cadence, buffer drift ≤ 500 ms, + latency sanity bounds, barge-in handling. +- `test_aai_server.py` — prompt and tool-schema construction, and registration assertions + covering both the `framework` Literal and `_get_server_class("aai")`. + +Plus an **in-process fake aai backend** speaking the real protocol, giving a full-session +test with no paid APIs and no Node process. This is the verification driven to green. + +**Not verifiable here:** the live end-to-end run, which needs API keys absent from this +checkout plus a running aai host: + +```bash +AAI_ALLOW_HOST=1 EVA_FRAMEWORK=aai EVA_MODEL__S2S=aai-host \ +EVA_USER_SIMULATOR__PROVIDER=openai_realtime EVA_DOMAIN=itsm \ +EVA_RECORD_IDS=15 EVA_MAX_CONCURRENT_CONVERSATIONS=1 eva +``` + +This command is documented in `docs/aai_integration.md` for the user to run. + +## Non-goals + +- Migrating the five existing servers onto the new base. +- Building the second (AssemblyAI-hosted) integration. The base is shaped to accept one; + no speculative hooks are added for it. +- Perturbation-suite or leaderboard-submission plumbing for aai. +- Any change to aai's sandbox or tool-execution model for non-host agents. From 4dce64bd5b1b5346be9736c8c6e1af1b410a6a03 Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Wed, 29 Jul 2026 00:09:43 -0400 Subject: [PATCH 02/11] docs: implementation plan for the EVA aai assistant server Six TDD tasks: aai event models, the shared WebSocket bridge base + seam, the aai host session client, the server + framework registration, a full-session test against an in-process fake aai host, and docs. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-07-29-eva-aai-assistant-server.md | 2408 +++++++++++++++++ 1 file changed, 2408 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-29-eva-aai-assistant-server.md diff --git a/docs/superpowers/plans/2026-07-29-eva-aai-assistant-server.md b/docs/superpowers/plans/2026-07-29-eva-aai-assistant-server.md new file mode 100644 index 00000000..41ab0cb8 --- /dev/null +++ b/docs/superpowers/plans/2026-07-29-eva-aai-assistant-server.md @@ -0,0 +1,2408 @@ +# EVA aai Assistant Server Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Let EVA evaluate the aai voice-agent framework by adding an assistant server that bridges EVA's user simulator to an aai host-mode WebSocket, executing all tool calls inside EVA so the scenario database mutates. + +**Architecture:** A new `WebSocketBridgeAssistantServer` base owns everything the EVA assistant-server contract requires (Twilio framing, 20 ms output pacing, time-aligned recording buffers, turn bookkeeping, audit and metrics writes). A subclass supplies only a `VoiceBackendSession` — for aai, a WebSocket client that performs the `config.host` handshake and relays `tool_call` → `tool_result`. All five existing servers are left untouched. + +**Tech Stack:** Python 3.11–3.13, FastAPI + uvicorn (WebSocket server), `websockets` (client, already a dependency), pydantic v2, pytest with `asyncio_mode = "auto"`. + +**Spec:** `docs/superpowers/specs/2026-07-28-eva-aai-assistant-server-design.md` + +## Global Constraints + +- **Additive only.** Do not modify `pipecat_server.py`, `openai_realtime_server.py`, `gemini_live_server.py`, `elevenlabs_server.py`, `grok_voice_server.py`, or `base_server.py`. Their numbers are on the published leaderboard. +- **`_shutdown()`, not `stop()`.** `AbstractAssistantServer.stop()` is a concrete template method (`base_server.py:138`); `_shutdown()` is the abstract hook (`base_server.py:197`). The contract doc at `docs/assistant_server_contract.md` is stale on this point — follow the code. +- **`_build_system_prompt()` already exists** on the base class (`base_server.py:341`). Do not reimplement it. +- **Recording sample rate is 24000 Hz** for both audio buffers regardless of backend wire rates. Convert inbound μ-law twice — once for the wire, once for the recording buffer — exactly as `gemini_live_server.py:405-410` does. +- **Never call `self.tool_handler.execute()` directly.** Always `await self.execute_tool(...)`, which writes the audit-log entries. +- **Do not write token usage.** aai emits no usage events. +- **Logging** uses `from eva.utils.logging import get_logger`, never `loguru` (tau2's source files use loguru; swap it on port). +- **Line length 120**, matching the existing codebase. Run `uv run ruff check` and `uv run ruff format` before each commit. +- Tests need no `@pytest.mark.asyncio` decorator — `asyncio_mode = "auto"` is set in `pyproject.toml:89`. + +--- + +## File Structure + +| File | Responsibility | +|---|---| +| `src/eva/assistant/bridge_events.py` *(new)* | Backend-agnostic event dataclasses + the `VoiceBackendSession` protocol. No I/O, no EVA imports. | +| `src/eva/assistant/ws_bridge_server.py` *(new)* | `WebSocketBridgeAssistantServer` — the EVA contract over an abstract backend session. | +| `src/eva/assistant/aai_events.py` *(new)* | aai wire-event pydantic models + `parse_aai_event`. Pure parsing. | +| `src/eva/assistant/aai_session.py` *(new)* | `AAIHostSession` — host handshake, audio frames, `tool_result` relay, wire→bridge event mapping. | +| `src/eva/assistant/aai_server.py` *(new)* | `AAIAssistantServer` — tool-schema construction and backend wiring. Thin. | +| `src/eva/models/config.py:506` *(modify)* | Add `"aai"` to the `framework` Literal. | +| `src/eva/orchestrator/worker.py:26-56` *(modify)* | Register the server in `_get_server_class()`. | +| `.env.example` *(modify)* | Document `AAI_WS_URL`. | +| `docs/aai_integration.md` *(new)* | Setup, run command, known gaps. | +| `tests/unit/assistant/test_aai_events.py` *(new)* | Event parsing. | +| `tests/unit/assistant/test_ws_bridge_server.py` *(new)* | Base-class turn/latency/barge-in/pacing behavior. | +| `tests/unit/assistant/test_aai_session.py` *(new)* | Handshake framing, tool_result framing, event mapping. | +| `tests/unit/assistant/test_aai_server.py` *(new)* | Tool schemas + registration. | +| `tests/integration/test_aai_fake_backend.py` *(new)* | Full session against an in-process fake aai server. | + +This splits the seam types into their own module (`bridge_events.py`), a refinement of the spec's four-module list: it keeps `ws_bridge_server.py` focused and lets a future backend import the seam without importing the server. + +--- + +### Task 1: aai wire-event models + +**Files:** +- Create: `src/eva/assistant/aai_events.py` +- Test: `tests/unit/assistant/test_aai_events.py` + +**Interfaces:** +- Consumes: nothing. +- Produces: `parse_aai_event(data: dict) -> BaseAAIEvent`, and the models + `AAIConfigEvent`, `AAISpeechStartedEvent`, `AAISpeechStoppedEvent`, + `AAIUserTranscriptEvent(text, turn_order)`, `AAIAgentTranscriptEvent(text)`, + `AAIToolCallEvent(tool_call_id, tool_name, args)`, + `AAIToolCallDoneEvent(tool_call_id, result)`, `AAIReplyDoneEvent`, + `AAIAudioDoneEvent`, `AAICancelledEvent`, `AAIResetEvent`, + `AAIIdleTimeoutEvent`, `AAIErrorEvent(code, message)`, + `AAICustomEvent(event, data)`, `AAIUnknownEvent(type, raw)`. + +This is a port of `~/Code/tau2-bench/src/tau2/voice/audio_native/aai/events.py` with three changes: loguru → `eva.utils.logging`, drop `AAITimeoutEvent` and `AAIAudioChunkEvent` (both were artifacts of tau2's tick loop; binary audio is handled out-of-band here), and drop `_prepare_log_data` (no base64 fields exist on this protocol — aai audio is binary frames). + +- [ ] **Step 1: Write the failing test** + +Create `tests/unit/assistant/test_aai_events.py`: + +```python +"""Tests for aai wire-event parsing.""" + +from eva.assistant.aai_events import ( + AAIAgentTranscriptEvent, + AAIErrorEvent, + AAIIdleTimeoutEvent, + AAIReplyDoneEvent, + AAISpeechStartedEvent, + AAIToolCallEvent, + AAIUnknownEvent, + AAIUserTranscriptEvent, + parse_aai_event, +) + + +class TestParseAaiEvent: + def test_parses_tool_call_camel_case_aliases(self): + event = parse_aai_event( + { + "type": "tool_call", + "toolCallId": "call_abc", + "toolName": "get_reservation", + "args": {"confirmation_number": "DJ3LPO"}, + } + ) + assert isinstance(event, AAIToolCallEvent) + assert event.tool_call_id == "call_abc" + assert event.tool_name == "get_reservation" + assert event.args == {"confirmation_number": "DJ3LPO"} + + def test_tool_call_args_default_to_empty_dict(self): + event = parse_aai_event({"type": "tool_call", "toolCallId": "c1", "toolName": "list_flights"}) + assert isinstance(event, AAIToolCallEvent) + assert event.args == {} + + def test_parses_agent_transcript(self): + event = parse_aai_event({"type": "agent_transcript", "text": "How can I help?"}) + assert isinstance(event, AAIAgentTranscriptEvent) + assert event.text == "How can I help?" + + def test_parses_user_transcript_with_turn_order(self): + event = parse_aai_event({"type": "user_transcript", "text": "hello", "turnOrder": 3}) + assert isinstance(event, AAIUserTranscriptEvent) + assert event.text == "hello" + assert event.turn_order == 3 + + def test_parses_zero_field_events(self): + assert isinstance(parse_aai_event({"type": "speech_started"}), AAISpeechStartedEvent) + assert isinstance(parse_aai_event({"type": "reply_done"}), AAIReplyDoneEvent) + assert isinstance(parse_aai_event({"type": "idle_timeout"}), AAIIdleTimeoutEvent) + + def test_parses_error_with_code_and_message(self): + event = parse_aai_event({"type": "error", "code": "host_disabled", "message": "AAI_ALLOW_HOST"}) + assert isinstance(event, AAIErrorEvent) + assert event.code == "host_disabled" + assert event.message == "AAI_ALLOW_HOST" + + def test_ignores_unknown_extra_fields(self): + event = parse_aai_event({"type": "agent_transcript", "text": "hi", "futureField": 1}) + assert isinstance(event, AAIAgentTranscriptEvent) + + def test_unrecognized_type_returns_unknown_event(self): + event = parse_aai_event({"type": "brand_new_thing", "x": 1}) + assert isinstance(event, AAIUnknownEvent) + assert event.type == "brand_new_thing" + assert event.raw == {"type": "brand_new_thing", "x": 1} + + def test_missing_type_returns_unknown_event(self): + event = parse_aai_event({"text": "orphan"}) + assert isinstance(event, AAIUnknownEvent) + assert event.type == "unknown" + + def test_malformed_known_event_returns_unknown_rather_than_raising(self): + # `text` is required on agent_transcript; a malformed frame must never abort a conversation. + event = parse_aai_event({"type": "agent_transcript"}) + assert isinstance(event, AAIUnknownEvent) + assert event.type == "agent_transcript" +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest tests/unit/assistant/test_aai_events.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'eva.assistant.aai_events'` + +- [ ] **Step 3: Write the implementation** + +Create `src/eva/assistant/aai_events.py`: + +```python +"""Pydantic models for aai voice-agent wire events. + +The aai host sends camelCase JSON frames over the session WebSocket; binary +frames carry PCM16 audio and are handled by the session, not here. + +Ported from the tau2-bench aai provider. Parsing never raises: an unrecognized +or malformed frame becomes an ``AAIUnknownEvent`` so a single bad frame cannot +abort a conversation mid-evaluation. +""" + +from typing import Any, Literal + +from pydantic import BaseModel, ConfigDict, Field + +from eva.utils.logging import get_logger + +logger = get_logger(__name__) + + +class BaseAAIEvent(BaseModel): + """Base class for all aai events. + + ``extra="ignore"`` so that new server-side fields never break parsing. + """ + + model_config = ConfigDict(extra="ignore", populate_by_name=True) + + type: str + + +class AAIConfigEvent(BaseAAIEvent): + """Server acknowledgment of the client's config frame — the handshake completing.""" + + type: Literal["config"] = "config" + + +class AAISpeechStartedEvent(BaseAAIEvent): + """aai's VAD detected the start of user speech.""" + + type: Literal["speech_started"] = "speech_started" + + +class AAISpeechStoppedEvent(BaseAAIEvent): + """aai's VAD detected the end of user speech.""" + + type: Literal["speech_stopped"] = "speech_stopped" + + +class AAIUserTranscriptEvent(BaseAAIEvent): + """Transcript of a user turn, as heard by aai.""" + + type: Literal["user_transcript"] = "user_transcript" + text: str + turn_order: int | None = Field(default=None, alias="turnOrder") + + +class AAIAgentTranscriptEvent(BaseAAIEvent): + """Transcript of the agent's reply. Carries the full text, not a delta.""" + + type: Literal["agent_transcript"] = "agent_transcript" + text: str + + +class AAIToolCallEvent(BaseAAIEvent): + """A relayed tool call awaiting a ``tool_result`` from this client.""" + + type: Literal["tool_call"] = "tool_call" + tool_call_id: str = Field(alias="toolCallId") + tool_name: str = Field(alias="toolName") + args: dict = Field(default_factory=dict) + + +class AAIToolCallDoneEvent(BaseAAIEvent): + """Acknowledgment that a relayed tool call was resolved.""" + + type: Literal["tool_call_done"] = "tool_call_done" + tool_call_id: str = Field(alias="toolCallId") + result: str = "" + + +class AAIReplyDoneEvent(BaseAAIEvent): + """The agent's reply is complete.""" + + type: Literal["reply_done"] = "reply_done" + + +class AAIAudioDoneEvent(BaseAAIEvent): + """The agent's audio output is complete.""" + + type: Literal["audio_done"] = "audio_done" + + +class AAICancelledEvent(BaseAAIEvent): + """An in-flight reply was cancelled.""" + + type: Literal["cancelled"] = "cancelled" + + +class AAIResetEvent(BaseAAIEvent): + """Session state was reset.""" + + type: Literal["reset"] = "reset" + + +class AAIIdleTimeoutEvent(BaseAAIEvent): + """The session was closed for inactivity.""" + + type: Literal["idle_timeout"] = "idle_timeout" + + +class AAIErrorEvent(BaseAAIEvent): + """An error reported by the aai host.""" + + type: Literal["error"] = "error" + code: str | None = None + message: str | None = None + + +class AAICustomEvent(BaseAAIEvent): + """An application-defined event emitted by the agent.""" + + type: Literal["custom_event"] = "custom_event" + event: str + data: Any | None = None + + +class AAIUnknownEvent(BaseAAIEvent): + """An unrecognized or unparseable frame, preserved for logging.""" + + type: str + raw: dict | None = None + + +_EVENT_TYPE_MAP: dict[str, type[BaseAAIEvent]] = { + "config": AAIConfigEvent, + "speech_started": AAISpeechStartedEvent, + "speech_stopped": AAISpeechStoppedEvent, + "user_transcript": AAIUserTranscriptEvent, + "agent_transcript": AAIAgentTranscriptEvent, + "tool_call": AAIToolCallEvent, + "tool_call_done": AAIToolCallDoneEvent, + "reply_done": AAIReplyDoneEvent, + "audio_done": AAIAudioDoneEvent, + "cancelled": AAICancelledEvent, + "reset": AAIResetEvent, + "idle_timeout": AAIIdleTimeoutEvent, + "error": AAIErrorEvent, + "custom_event": AAICustomEvent, +} + + +def parse_aai_event(data: dict) -> BaseAAIEvent: + """Parse a raw aai frame into a typed event. + + Never raises. Unknown types and validation failures both yield + ``AAIUnknownEvent`` with the original payload attached. + """ + event_type = data.get("type", "unknown") + event_class = _EVENT_TYPE_MAP.get(event_type) + + if event_class is None: + return AAIUnknownEvent(type=event_type, raw=data) + + try: + return event_class.model_validate(data) + except Exception as e: + logger.warning(f"Failed to parse aai event {event_type}: {e}") + return AAIUnknownEvent(type=event_type, raw=data) +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `uv run pytest tests/unit/assistant/test_aai_events.py -v` +Expected: PASS (11 tests) + +- [ ] **Step 5: Lint and commit** + +```bash +cd ~/Code/eva +uv run ruff format src/eva/assistant/aai_events.py tests/unit/assistant/test_aai_events.py +uv run ruff check src/eva/assistant/aai_events.py tests/unit/assistant/test_aai_events.py +git add src/eva/assistant/aai_events.py tests/unit/assistant/test_aai_events.py +git commit -m "feat(aai): wire-event models and tolerant parser" +``` + +--- + +### Task 2: Backend seam + WebSocket bridge base + +**Files:** +- Create: `src/eva/assistant/bridge_events.py` +- Create: `src/eva/assistant/ws_bridge_server.py` +- Test: `tests/unit/assistant/test_ws_bridge_server.py` + +**Interfaces:** +- Consumes: `AbstractAssistantServer` (`base_server.py`), the `audio_bridge` helpers. +- Produces: + - `bridge_events.AudioChunk(pcm: bytes)`, `AssistantTranscript(text: str)`, + `UserTranscript(text: str)`, `ToolCall(call_id: str, name: str, arguments: dict)`, + `SpeechStarted()`, `TurnDone()`, `BackendError(message: str, fatal: bool = False)`, + the `BridgeEvent` union, and the `VoiceBackendSession` protocol. + - `ws_bridge_server.WebSocketBridgeAssistantServer`, an abstract subclass of + `AbstractAssistantServer` requiring `model_name: str` (property) and + `async _open_backend(self) -> VoiceBackendSession`. + +- [ ] **Step 1: Write the failing test** + +Create `tests/unit/assistant/test_ws_bridge_server.py`: + +```python +"""Tests for WebSocketBridgeAssistantServer turn, latency, and pacing behavior. + +The server is built with ``object.__new__`` and hand-set attributes (the pattern +used by test_openai_realtime_server.py) so tests need no ToolExecutor, no +scenario database, and no live sockets. +""" + +import asyncio +from unittest.mock import MagicMock + +from eva.assistant.bridge_events import ( + AssistantTranscript, + AudioChunk, + BackendError, + SpeechStarted, + ToolCall, + TurnDone, + UserTranscript, +) +from eva.assistant.ws_bridge_server import WebSocketBridgeAssistantServer + + +class _FakeSession: + """In-memory VoiceBackendSession.""" + + backend_input_rate = 16000 + backend_output_rate = 24000 + + def __init__(self, events=()): + self._events = list(events) + self.sent_audio: list[bytes] = [] + self.tool_results: list[tuple[str, object]] = [] + self.closed = False + + async def send_audio(self, pcm: bytes) -> None: + self.sent_audio.append(pcm) + + async def send_tool_result(self, call_id: str, result: object) -> None: + self.tool_results.append((call_id, result)) + + async def events(self): + for event in self._events: + yield event + + async def aclose(self) -> None: + self.closed = True + + +class _Server(WebSocketBridgeAssistantServer): + @property + def model_name(self) -> str: + return "test-model" + + async def _open_backend(self): + return _FakeSession() + + +def _bare_server(session: _FakeSession | None = None) -> _Server: + srv = object.__new__(_Server) + srv._session = session or _FakeSession() + srv._fw_log = MagicMock() + srv._metrics_log = MagicMock() + srv.audit_log = MagicMock() + srv._audio_out = asyncio.Queue() + srv._stream_sid = "conv-1" + srv._running = True + srv._user_speech_start_ms = None + srv._user_speech_stop_ms = None + srv._user_speaking = False + srv._assistant_speaking = False + srv._turn_first_audio_ms = None + srv._assistant_text = "" + srv._tasks = [] + srv.user_audio_buffer = bytearray() + srv.assistant_audio_buffer = bytearray() + srv._audio_sample_rate = 24000 + srv._to_backend = None + srv._from_backend = lambda pcm: b"\xff" * (len(pcm) // 6) # 24k PCM16 -> 8k mulaw + return srv + + +class TestModelResponseLatency: + def test_writes_latency_on_first_audio_chunk_of_turn(self): + srv = _bare_server() + srv._user_speech_stop_ms = 1_000_000 + srv._now_ms = lambda: 1_000_450 + + srv._on_audio_chunk(b"\x00\x00" * 240) + + srv._metrics_log.write_latency.assert_called_once_with("model_response", 0.45, "test-model") + + def test_writes_latency_only_once_per_turn(self): + srv = _bare_server() + srv._user_speech_stop_ms = 1_000_000 + srv._now_ms = lambda: 1_000_450 + + srv._on_audio_chunk(b"\x00\x00" * 240) + srv._on_audio_chunk(b"\x00\x00" * 240) + + assert srv._metrics_log.write_latency.call_count == 1 + + def test_skips_latency_when_no_user_speech_stop(self): + """The opening greeting is model-initiated: there is no user turn to measure from.""" + srv = _bare_server() + srv._now_ms = lambda: 1_000_450 + + srv._on_audio_chunk(b"\x00\x00" * 240) + + srv._metrics_log.write_latency.assert_not_called() + srv._fw_log.turn_start.assert_called_once() + + def test_skips_implausible_latency(self): + srv = _bare_server() + srv._user_speech_stop_ms = 1_000_000 + srv._now_ms = lambda: 1_000_000 + 45_000 # 45s, over the 30s ceiling + + srv._on_audio_chunk(b"\x00\x00" * 240) + + srv._metrics_log.write_latency.assert_not_called() + + def test_skips_negative_latency(self): + srv = _bare_server() + srv._user_speech_stop_ms = 1_000_000 + srv._now_ms = lambda: 999_000 + + srv._on_audio_chunk(b"\x00\x00" * 240) + + srv._metrics_log.write_latency.assert_not_called() + + +class TestAudioBuffers: + def test_assistant_audio_appended_and_queued_for_output(self): + srv = _bare_server() + srv._now_ms = lambda: 1_000 + + srv._on_audio_chunk(b"\x01\x02" * 240) + + assert len(srv.assistant_audio_buffer) == 480 + assert srv._audio_out.qsize() == 1 + + def test_assistant_audio_pads_user_track_to_stay_aligned(self): + srv = _bare_server() + srv._now_ms = lambda: 1_000 + srv.assistant_audio_buffer.extend(b"\x00" * 960) + + srv._on_audio_chunk(b"\x01\x02" * 240) + + # User track padded up to where the assistant track started. + assert len(srv.user_audio_buffer) == 960 + + def test_no_padding_while_user_is_speaking(self): + """During overlap both tracks advance on their own; padding would double-count.""" + srv = _bare_server() + srv._now_ms = lambda: 1_000 + srv._user_speaking = True + srv.assistant_audio_buffer.extend(b"\x00" * 960) + + srv._on_audio_chunk(b"\x01\x02" * 240) + + assert len(srv.user_audio_buffer) == 0 + + +class TestTurnCompletion: + def test_finish_turn_writes_transcript_and_turn_end(self): + srv = _bare_server() + srv._now_ms = lambda: 5_000 + srv._turn_first_audio_ms = 4_000 + srv._assistant_text = "Your flight is confirmed." + + srv._finish_turn(interrupted=False) + + srv.audit_log.append_assistant_output.assert_called_once_with( + "Your flight is confirmed.", timestamp_ms="4000" + ) + srv._fw_log.llm_response.assert_called_once_with("Your flight is confirmed.") + srv._fw_log.turn_end.assert_called_once_with(was_interrupted=False) + + def test_finish_turn_resets_state_for_next_turn(self): + srv = _bare_server() + srv._now_ms = lambda: 5_000 + srv._turn_first_audio_ms = 4_000 + srv._assistant_text = "text" + + srv._finish_turn(interrupted=False) + + assert srv._assistant_text == "" + assert srv._turn_first_audio_ms is None + assert srv._assistant_speaking is False + + def test_finish_turn_is_idempotent(self): + """reply_done and audio_done both map to TurnDone; the second must be a no-op.""" + srv = _bare_server() + srv._now_ms = lambda: 5_000 + srv._turn_first_audio_ms = 4_000 + srv._assistant_text = "text" + + srv._finish_turn(interrupted=False) + srv._finish_turn(interrupted=False) + + assert srv.audit_log.append_assistant_output.call_count == 1 + assert srv._fw_log.turn_end.call_count == 1 + + +class TestBargeIn: + def test_barge_in_drains_queued_audio_and_marks_turn_interrupted(self): + srv = _bare_server() + srv._now_ms = lambda: 5_000 + srv._turn_first_audio_ms = 4_000 + srv._assistant_text = "As I was saying" + srv._audio_out.put_nowait(b"\xff" * 160) + srv._audio_out.put_nowait(b"\xff" * 160) + + srv._handle_barge_in() + + assert srv._audio_out.qsize() == 0 + srv._fw_log.turn_end.assert_called_once_with(was_interrupted=True) + srv.audit_log.append_assistant_output.assert_called_once_with( + "As I was saying [interrupted]", timestamp_ms="4000" + ) + + def test_speech_started_while_assistant_silent_is_not_barge_in(self): + srv = _bare_server() + srv._now_ms = lambda: 5_000 + + srv._handle_barge_in() + + srv._fw_log.turn_end.assert_not_called() + + +class TestToolCalls: + async def test_tool_result_relayed_to_backend(self): + session = _FakeSession() + srv = _bare_server(session) + srv.execute_tool = MagicMock(return_value=_async_value({"status": "ok", "seat": "12A"})) + + await srv._handle_tool_call(ToolCall(call_id="c1", name="assign_seat", arguments={"seat": "12A"})) + + srv.execute_tool.assert_called_once_with("assign_seat", {"seat": "12A"}) + assert session.tool_results == [("c1", {"status": "ok", "seat": "12A"})] + + async def test_tool_failure_relays_error_result_so_the_turn_continues(self): + session = _FakeSession() + srv = _bare_server(session) + srv.execute_tool = MagicMock(side_effect=RuntimeError("db offline")) + + await srv._handle_tool_call(ToolCall(call_id="c2", name="broken", arguments={})) + + assert len(session.tool_results) == 1 + call_id, result = session.tool_results[0] + assert call_id == "c2" + assert result["status"] == "error" + assert "db offline" in result["message"] + + +class TestEventDispatch: + async def test_dispatches_each_event_kind(self): + session = _FakeSession( + [ + UserTranscript(text="I need to rebook"), + AudioChunk(pcm=b"\x01\x02" * 240), + AssistantTranscript(text="Sure, let me look."), + TurnDone(), + ] + ) + srv = _bare_server(session) + srv._now_ms = lambda: 7_000 + srv._user_speech_start_ms = 6_000 + + await srv._process_backend_events() + + srv.audit_log.append_user_input.assert_called_once_with("I need to rebook", timestamp_ms="6000") + srv.audit_log.append_assistant_output.assert_called_once_with("Sure, let me look.", timestamp_ms="7000") + + async def test_assistant_transcript_overwrites_rather_than_appends(self): + """aai sends full text per frame, not deltas.""" + session = _FakeSession([AssistantTranscript(text="Hello"), AssistantTranscript(text="Hello there")]) + srv = _bare_server(session) + srv._now_ms = lambda: 7_000 + + await srv._process_backend_events() + + assert srv._assistant_text == "Hello there" + + async def test_fatal_backend_error_stops_dispatch(self): + session = _FakeSession([BackendError(message="idle_timeout", fatal=True), AssistantTranscript(text="never")]) + srv = _bare_server(session) + srv._now_ms = lambda: 7_000 + + await srv._process_backend_events() + + assert srv._assistant_text == "" + + +class TestPacing: + async def test_rechunks_output_to_160_byte_frames(self): + """Backend audio does not arrive in 160-byte multiples; the simulator requires 20ms frames.""" + srv = _bare_server() + sent: list[str] = [] + + class _WS: + async def send_text(self, msg: str) -> None: + sent.append(msg) + + srv._audio_out.put_nowait(b"\xff" * 400) + task = asyncio.create_task(srv._pace_audio_output(_WS())) + await asyncio.sleep(0.08) + srv._running = False + task.cancel() + try: + await task + except asyncio.CancelledError: + pass + + # 400 bytes -> two full 160-byte frames, 80 bytes held back for the next chunk. + assert len(sent) == 2 + + +def _async_value(value): + async def _coro(): + return value + + return _coro() +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest tests/unit/assistant/test_ws_bridge_server.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'eva.assistant.bridge_events'` + +- [ ] **Step 3: Write the seam types** + +Create `src/eva/assistant/bridge_events.py`: + +```python +"""Backend-agnostic events and the session protocol used by the bridge server. + +A voice backend (aai host mode, a hosted realtime API, …) speaks its own wire +protocol. ``WebSocketBridgeAssistantServer`` never sees that protocol: an +adapter translates it into the small event vocabulary below, and the bridge +turns those events into what EVA's evaluation contract requires. + +Keeping this module free of EVA imports means a new backend adapter can be +written and unit-tested without pulling in the server. +""" + +from dataclasses import dataclass +from typing import Any, AsyncIterator, Protocol, runtime_checkable + + +@dataclass(frozen=True) +class AudioChunk: + """PCM16 audio from the backend, at its declared ``backend_output_rate``.""" + + pcm: bytes + + +@dataclass(frozen=True) +class AssistantTranscript: + """The assistant's reply text. + + Full-text semantics: each event supersedes the previous one for the current + turn rather than appending to it. + """ + + text: str + + +@dataclass(frozen=True) +class UserTranscript: + """A user turn as transcribed by the backend.""" + + text: str + + +@dataclass(frozen=True) +class ToolCall: + """A tool the backend wants executed. The bridge runs it and replies.""" + + call_id: str + name: str + arguments: dict + + +@dataclass(frozen=True) +class SpeechStarted: + """Backend VAD detected user speech. Triggers barge-in if the assistant is speaking.""" + + +@dataclass(frozen=True) +class TurnDone: + """The assistant's turn is complete. Safe to deliver more than once per turn.""" + + +@dataclass(frozen=True) +class BackendError: + """A backend-reported problem. ``fatal`` ends the session.""" + + message: str + fatal: bool = False + + +BridgeEvent = ( + AudioChunk | AssistantTranscript | UserTranscript | ToolCall | SpeechStarted | TurnDone | BackendError +) + + +@runtime_checkable +class VoiceBackendSession(Protocol): + """One live conversation with a voice backend. + + Implementations own their wire protocol and nothing else: no Twilio framing, + no recording buffers, no audit logging. + """ + + #: Sample rate of the PCM16 audio this session expects from ``send_audio``. + backend_input_rate: int + #: Sample rate of the PCM16 audio this session emits in ``AudioChunk``. + backend_output_rate: int + + async def send_audio(self, pcm: bytes) -> None: + """Send PCM16 user audio at ``backend_input_rate``.""" + ... + + async def send_tool_result(self, call_id: str, result: Any) -> None: + """Resolve the relayed tool call identified by ``call_id``.""" + ... + + def events(self) -> AsyncIterator[BridgeEvent]: + """Yield events until the backend closes the session.""" + ... + + async def aclose(self) -> None: + """Close the session. Must be safe to call more than once.""" + ... +``` + +- [ ] **Step 4: Write the bridge server** + +Create `src/eva/assistant/ws_bridge_server.py`: + +```python +"""Shared base for assistant servers that bridge EVA to a WebSocket voice backend. + +EVA's user simulator dials in over a Twilio-framed WebSocket; a voice backend +speaks its own protocol on its own socket. Every such integration needs the same +middle layer, and getting any part of it wrong corrupts the evaluation rather +than merely degrading audio: + +* **Output pacing.** The simulator infers turn boundaries from arrival timing, + so assistant audio must leave at wall-clock 20 ms / 160-byte cadence. +* **Track alignment.** The two recording buffers form a shared timeline; each + must be padded to the other's position before it advances, or the mixed WAV + is skewed and every audio judge metric reads the wrong thing. +* **Recording rate.** Both buffers are written at 24 kHz regardless of the + backend's wire rates, so inbound mu-law is converted twice: once for the wire, + once for the recording buffer. + +Subclasses supply only a ``VoiceBackendSession`` and a model identifier. + +See docs/assistant_server_contract.md for the contract this satisfies. Note that +the doc is stale on shutdown: ``AbstractAssistantServer.stop()`` is a concrete +template method and ``_shutdown()`` is the hook implemented here. +""" + +import asyncio +import contextlib +import json +import time +from abc import abstractmethod +from typing import Callable + +import uvicorn +from fastapi import FastAPI, WebSocket + +from eva.assistant.audio_bridge import ( + FrameworkLogWriter, + MetricsLogWriter, + create_twilio_media_message, + mulaw_8k_to_pcm16_16k, + mulaw_8k_to_pcm16_24k, + parse_twilio_media_message, + pcm16_24k_to_mulaw_8k, + sync_buffer_to_position, +) +from eva.assistant.base_server import AbstractAssistantServer +from eva.assistant.bridge_events import ( + AssistantTranscript, + AudioChunk, + BackendError, + SpeechStarted, + ToolCall, + TurnDone, + UserTranscript, + VoiceBackendSession, +) +from eva.utils.logging import get_logger + +logger = get_logger(__name__) + +#: Both recording buffers are written at this rate, whatever the backend uses. +RECORDING_SAMPLE_RATE = 24000 + +#: mu-law bytes per 20 ms frame at 8 kHz — one Twilio media message. +MULAW_FRAME_BYTES = 160 +FRAME_INTERVAL_S = 0.02 + +#: A model_response latency outside this bound indicates a bookkeeping bug, not a slow model. +MAX_PLAUSIBLE_LATENCY_MS = 30_000 + +#: mu-law 8 kHz -> PCM16 at the backend's input rate. +_TO_BACKEND: dict[int, Callable[[bytes], bytes]] = { + 16000: mulaw_8k_to_pcm16_16k, + 24000: mulaw_8k_to_pcm16_24k, +} + +#: PCM16 at the backend's output rate -> mu-law 8 kHz. +_FROM_BACKEND: dict[int, Callable[[bytes], bytes]] = { + 24000: pcm16_24k_to_mulaw_8k, +} + + +class WebSocketBridgeAssistantServer(AbstractAssistantServer): + """Bridges the user simulator's Twilio socket to a ``VoiceBackendSession``.""" + + def __init__(self, **kwargs): + super().__init__(**kwargs) + self._audio_sample_rate = RECORDING_SAMPLE_RATE + + self._session: VoiceBackendSession | None = None + self._to_backend: Callable[[bytes], bytes] | None = None + self._from_backend: Callable[[bytes], bytes] | None = None + + self._stream_sid: str = self.conversation_id + self._audio_out: asyncio.Queue[bytes] = asyncio.Queue() + self._tasks: list[asyncio.Task] = [] + + # Turn bookkeeping + self._user_speech_start_ms: int | None = None + self._user_speech_stop_ms: int | None = None + self._user_speaking = False + self._assistant_speaking = False + self._turn_first_audio_ms: int | None = None + self._assistant_text = "" + + # ── Subclass contract ───────────────────────────────────────────── + + @property + @abstractmethod + def model_name(self) -> str: + """Model identifier recorded in the metrics log.""" + ... + + @abstractmethod + async def _open_backend(self) -> VoiceBackendSession: + """Open a session with the backend. Raise to abort the conversation.""" + ... + + # ── Lifecycle ───────────────────────────────────────────────────── + + async def start(self) -> None: + self.output_dir.mkdir(parents=True, exist_ok=True) + self._fw_log = FrameworkLogWriter(self.output_dir) + self._metrics_log = MetricsLogWriter(self.output_dir) + + self._app = FastAPI() + + @self._app.websocket("/ws") + async def ws_endpoint(websocket: WebSocket): + await websocket.accept() + await self._handle_session(websocket) + + @self._app.websocket("/") + async def ws_root(websocket: WebSocket): + await websocket.accept() + await self._handle_session(websocket) + + config = uvicorn.Config(self._app, host="0.0.0.0", port=self.port, log_level="warning") + self._server = uvicorn.Server(config) + self._server_task = asyncio.create_task(self._server.serve()) + while not self._server.started: + await asyncio.sleep(0.05) + self._running = True + logger.info(f"{type(self).__name__} listening on ws://localhost:{self.port}/ws") + + async def _shutdown(self) -> None: + self._running = False + + for task in self._tasks: + task.cancel() + for task in self._tasks: + with contextlib.suppress(asyncio.CancelledError, Exception): + await task + self._tasks.clear() + + if self._session is not None: + with contextlib.suppress(Exception): + await self._session.aclose() + self._session = None + + if self._server is not None: + self._server.should_exit = True + if self._server_task is not None: + self._server_task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await self._server_task + + # ── Session ─────────────────────────────────────────────────────── + + async def _handle_session(self, websocket: WebSocket) -> None: + """Run one conversation: simulator socket in, backend socket out.""" + try: + self._session = await self._open_backend() + except Exception as e: + # Loud failure: a silent fallback would yield a scored but meaningless run. + logger.error(f"Failed to open backend session, aborting conversation: {e}", exc_info=True) + with contextlib.suppress(Exception): + await websocket.close() + return + + self._to_backend = self._converter(_TO_BACKEND, self._session.backend_input_rate, "input") + self._from_backend = self._converter(_FROM_BACKEND, self._session.backend_output_rate, "output") + + self._tasks = [ + asyncio.create_task(self._forward_user_audio(websocket)), + asyncio.create_task(self._process_backend_events()), + asyncio.create_task(self._pace_audio_output(websocket)), + ] + done, pending = await asyncio.wait(self._tasks, return_when=asyncio.FIRST_COMPLETED) + for task in pending: + task.cancel() + for task in done: + if task.exception() is not None: + logger.error(f"Bridge task failed: {task.exception()}") + + @staticmethod + def _converter(table: dict[int, Callable[[bytes], bytes]], rate: int, direction: str) -> Callable[[bytes], bytes]: + converter = table.get(rate) + if converter is None: + raise ValueError( + f"No mu-law converter for backend {direction} rate {rate} Hz " + f"(available: {sorted(table)}). Add one to audio_bridge.py." + ) + return converter + + async def _forward_user_audio(self, websocket: WebSocket) -> None: + """Simulator -> backend, plus the user recording track.""" + while self._running: + try: + raw = await websocket.receive_text() + except Exception: + break + + try: + msg = json.loads(raw) + except json.JSONDecodeError: + continue + event = msg.get("event") + + if event == "start": + self._stream_sid = msg.get("start", {}).get("streamSid", self._stream_sid) + elif event == "stop": + break + elif event == "user_speech_start": + self._user_speech_start_ms = _as_int_ms(msg.get("timestamp_ms")) + self._user_speaking = True + elif event == "user_speech_stop": + self._user_speech_stop_ms = _as_int_ms(msg.get("timestamp_ms")) + self._user_speaking = False + elif event == "media": + mulaw = parse_twilio_media_message(raw) + if not mulaw: + continue + await self._session.send_audio(self._to_backend(mulaw)) + self._record_user_audio(mulaw) + + def _record_user_audio(self, mulaw: bytes) -> None: + """Append to the user track at the recording rate, keeping tracks aligned.""" + pcm = mulaw_8k_to_pcm16_24k(mulaw) + if not self._assistant_speaking: + sync_buffer_to_position(self.assistant_audio_buffer, len(self.user_audio_buffer)) + self.user_audio_buffer.extend(pcm) + + async def _process_backend_events(self) -> None: + """Backend -> audit log, metrics, recording track, and output queue.""" + async for event in self._session.events(): + if isinstance(event, AudioChunk): + self._on_audio_chunk(event.pcm) + elif isinstance(event, AssistantTranscript): + self._assistant_text = event.text + elif isinstance(event, UserTranscript): + timestamp = str(self._user_speech_start_ms or self._now_ms()) + self.audit_log.append_user_input(event.text, timestamp_ms=timestamp) + elif isinstance(event, ToolCall): + await self._handle_tool_call(event) + elif isinstance(event, SpeechStarted): + self._handle_barge_in() + elif isinstance(event, TurnDone): + self._finish_turn(interrupted=False) + elif isinstance(event, BackendError): + logger.error(f"Backend error: {event.message}") + if event.fatal: + break + + def _on_audio_chunk(self, pcm: bytes) -> None: + """Record assistant audio, start the turn if needed, and queue for paced send.""" + if self._turn_first_audio_ms is None: + self._turn_first_audio_ms = self._now_ms() + self._fw_log.turn_start(timestamp_ms=self._turn_first_audio_ms) + self._write_model_response_latency(self._turn_first_audio_ms) + + self._assistant_speaking = True + if not self._user_speaking: + sync_buffer_to_position(self.user_audio_buffer, len(self.assistant_audio_buffer)) + self.assistant_audio_buffer.extend(pcm) + self._audio_out.put_nowait(self._from_backend(pcm)) + + def _write_model_response_latency(self, first_audio_ms: int) -> None: + """Time from the simulator's user_speech_stop to our first audio byte. + + The simulator's VAD is the contract-specified source. The backend's own + speech events drive barge-in only; mixing the two skews this metric. + """ + if self._user_speech_stop_ms is None: + return # Model-initiated turn, e.g. the opening greeting. + + latency_ms = first_audio_ms - self._user_speech_stop_ms + if 0 < latency_ms < MAX_PLAUSIBLE_LATENCY_MS: + self._metrics_log.write_latency("model_response", latency_ms / 1000, self.model_name) + else: + logger.warning(f"Discarding implausible model_response latency: {latency_ms} ms") + self._user_speech_stop_ms = None + + def _finish_turn(self, interrupted: bool) -> None: + """Close out the assistant turn. Idempotent — both reply_done and audio_done arrive.""" + if self._turn_first_audio_ms is None and not self._assistant_text: + return + + text = self._assistant_text + if interrupted and text: + text = f"{text} [interrupted]" + + if text: + timestamp = str(self._turn_first_audio_ms or self._now_ms()) + self.audit_log.append_assistant_output(text, timestamp_ms=timestamp) + self._fw_log.llm_response(text) + self._fw_log.s2s_transcript(text) + self._fw_log.turn_end(was_interrupted=interrupted) + + self._assistant_text = "" + self._turn_first_audio_ms = None + self._assistant_speaking = False + + def _handle_barge_in(self) -> None: + """Drop undelivered assistant audio and close the turn as interrupted.""" + if self._turn_first_audio_ms is None and not self._assistant_text: + return # Assistant was not speaking: an ordinary user turn starting. + + dropped = 0 + while not self._audio_out.empty(): + self._audio_out.get_nowait() + dropped += 1 + if dropped: + logger.debug(f"Barge-in: dropped {dropped} queued audio frames") + self._finish_turn(interrupted=True) + + async def _handle_tool_call(self, event: ToolCall) -> None: + """Execute against the scenario database and relay the result back.""" + try: + result = await self.execute_tool(event.name, event.arguments) + except Exception as e: + logger.error(f"Tool {event.name} raised: {e}", exc_info=True) + result = {"status": "error", "message": str(e)} + await self._session.send_tool_result(event.call_id, result) + + async def _pace_audio_output(self, websocket: WebSocket) -> None: + """Drain the output queue at 20 ms per 160-byte frame. + + Backend audio does not arrive in 160-byte multiples once converted, so + frames are re-chunked here. Sending faster or slower than real time makes + the simulator misjudge turn boundaries. + """ + pending = bytearray() + next_send = time.monotonic() + + while self._running: + chunk = await self._audio_out.get() + pending.extend(chunk) + + while len(pending) >= MULAW_FRAME_BYTES: + frame = bytes(pending[:MULAW_FRAME_BYTES]) + del pending[:MULAW_FRAME_BYTES] + try: + await websocket.send_text(create_twilio_media_message(self._stream_sid, frame)) + except Exception: + return + + next_send += FRAME_INTERVAL_S + sleep_s = next_send - time.monotonic() + if sleep_s > 0: + await asyncio.sleep(sleep_s) + else: + # Fell behind (or the queue was idle): resync rather than burst. + next_send = time.monotonic() + + @staticmethod + def _now_ms() -> int: + """Wall-clock milliseconds. Overridden in tests.""" + return int(time.time() * 1000) + + +def _as_int_ms(value: object) -> int | None: + """Coerce a simulator timestamp to int milliseconds, tolerating str or float.""" + if value is None: + return None + try: + return int(float(value)) + except (TypeError, ValueError): + return None +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `uv run pytest tests/unit/assistant/test_ws_bridge_server.py -v` +Expected: PASS (17 tests) + +- [ ] **Step 6: Lint and commit** + +```bash +cd ~/Code/eva +uv run ruff format src/eva/assistant/bridge_events.py src/eva/assistant/ws_bridge_server.py tests/unit/assistant/test_ws_bridge_server.py +uv run ruff check src/eva/assistant/bridge_events.py src/eva/assistant/ws_bridge_server.py tests/unit/assistant/test_ws_bridge_server.py +git add src/eva/assistant/bridge_events.py src/eva/assistant/ws_bridge_server.py tests/unit/assistant/test_ws_bridge_server.py +git commit -m "feat(assistant): shared WebSocket bridge base for voice backends" +``` + +--- + +### Task 3: aai host session + +**Files:** +- Create: `src/eva/assistant/aai_session.py` +- Test: `tests/unit/assistant/test_aai_session.py` + +**Interfaces:** +- Consumes: `parse_aai_event` and the event models from Task 1; the bridge event + dataclasses from Task 2. +- Produces: + - `DEFAULT_AAI_WS_URL = "ws://localhost:3000/websocket"`, + `DEFAULT_AAI_INPUT_SAMPLE_RATE = 16000`, `DEFAULT_AAI_OUTPUT_SAMPLE_RATE = 24000`, + `DEFAULT_AAI_MODEL = "aai-host"`. + - `with_host_flag(url: str) -> str` + - `build_host_config_message(*, system_prompt: str, tools: list[dict], greeting: str | None, input_rate: int, output_rate: int) -> dict` + - `map_aai_event(event: BaseAAIEvent) -> BridgeEvent | None` + - `AAIHostSessionError(Exception)` + - `AAIHostSession` with `connect(...)` classmethod, satisfying `VoiceBackendSession`. + +- [ ] **Step 1: Write the failing test** + +Create `tests/unit/assistant/test_aai_session.py`: + +```python +"""Tests for the aai host-mode session: framing, handshake, and event mapping.""" + +import json + +import pytest + +from eva.assistant.aai_events import ( + AAIAgentTranscriptEvent, + AAIAudioDoneEvent, + AAICustomEvent, + AAIErrorEvent, + AAIIdleTimeoutEvent, + AAIReplyDoneEvent, + AAISpeechStartedEvent, + AAISpeechStoppedEvent, + AAIToolCallEvent, + AAIUserTranscriptEvent, +) +from eva.assistant.aai_session import ( + AAIHostSession, + build_host_config_message, + map_aai_event, + with_host_flag, +) +from eva.assistant.bridge_events import ( + AssistantTranscript, + BackendError, + SpeechStarted, + ToolCall, + TurnDone, + UserTranscript, +) + + +class TestWithHostFlag: + def test_appends_host_flag(self): + assert with_host_flag("ws://localhost:3000/websocket") == "ws://localhost:3000/websocket?host=1" + + def test_merges_with_existing_query_params(self): + result = with_host_flag("ws://localhost:3000/websocket?sessionId=abc") + assert "sessionId=abc" in result + assert "host=1" in result + + def test_overwrites_existing_host_param(self): + assert with_host_flag("ws://h/websocket?host=0").endswith("host=1") + + +class TestBuildHostConfigMessage: + def test_builds_handshake_frame(self): + msg = build_host_config_message( + system_prompt="You are an airline agent.", + tools=[{"type": "function", "name": "t", "description": "d", "parameters": {"type": "object"}}], + greeting="Thank you for calling.", + input_rate=16000, + output_rate=24000, + ) + assert msg["type"] == "config" + assert msg["audioFormat"] == "pcm16" + assert msg["sampleRate"] == 16000 + assert msg["ttsSampleRate"] == 24000 + assert msg["host"]["systemPrompt"] == "You are an airline agent." + assert msg["host"]["greeting"] == "Thank you for calling." + assert len(msg["host"]["tools"]) == 1 + + def test_omits_greeting_when_empty(self): + msg = build_host_config_message( + system_prompt="p", tools=[], greeting=None, input_rate=16000, output_rate=24000 + ) + assert "greeting" not in msg["host"] + + def test_is_json_serializable(self): + msg = build_host_config_message( + system_prompt="p", tools=[], greeting="g", input_rate=16000, output_rate=24000 + ) + assert json.loads(json.dumps(msg)) == msg + + +class TestMapAaiEvent: + def test_maps_agent_transcript(self): + assert map_aai_event(AAIAgentTranscriptEvent(text="hi")) == AssistantTranscript(text="hi") + + def test_maps_user_transcript(self): + assert map_aai_event(AAIUserTranscriptEvent(text="hello")) == UserTranscript(text="hello") + + def test_maps_tool_call(self): + event = AAIToolCallEvent(toolCallId="c1", toolName="get_reservation", args={"x": 1}) + assert map_aai_event(event) == ToolCall(call_id="c1", name="get_reservation", arguments={"x": 1}) + + def test_maps_speech_started_to_barge_in_signal(self): + assert map_aai_event(AAISpeechStartedEvent()) == SpeechStarted() + + def test_maps_both_turn_end_signals(self): + assert map_aai_event(AAIReplyDoneEvent()) == TurnDone() + assert map_aai_event(AAIAudioDoneEvent()) == TurnDone() + + def test_maps_error_with_code_and_message(self): + result = map_aai_event(AAIErrorEvent(code="bad_config", message="nope")) + assert isinstance(result, BackendError) + assert "bad_config" in result.message + assert "nope" in result.message + assert result.fatal is False + + def test_idle_timeout_is_fatal(self): + result = map_aai_event(AAIIdleTimeoutEvent()) + assert isinstance(result, BackendError) + assert result.fatal is True + + def test_unmapped_events_return_none(self): + assert map_aai_event(AAISpeechStoppedEvent()) is None + assert map_aai_event(AAICustomEvent(event="metric", data={"a": 1})) is None + + +class _FakeWebSocket: + """Stands in for a websockets client connection.""" + + def __init__(self, inbound=()): + self.sent: list[object] = [] + self._inbound = list(inbound) + self.closed = False + + async def send(self, data) -> None: + self.sent.append(data) + + async def recv(self): + if not self._inbound: + raise AssertionError("recv() called with no inbound frames left") + return self._inbound.pop(0) + + def __aiter__(self): + return self + + async def __anext__(self): + if not self._inbound: + raise StopAsyncIteration + return self._inbound.pop(0) + + async def close(self) -> None: + self.closed = True + + +class TestSessionFraming: + async def test_send_audio_sends_raw_binary(self): + ws = _FakeWebSocket() + session = AAIHostSession(ws, input_rate=16000, output_rate=24000) + + await session.send_audio(b"\x01\x02\x03\x04") + + assert ws.sent == [b"\x01\x02\x03\x04"] + + async def test_send_tool_result_json_encodes_the_result(self): + ws = _FakeWebSocket() + session = AAIHostSession(ws, input_rate=16000, output_rate=24000) + + await session.send_tool_result("call_1", {"status": "ok", "seat": "12A"}) + + frame = json.loads(ws.sent[0]) + assert frame["type"] == "tool_result" + assert frame["toolCallId"] == "call_1" + # aai unwraps a JSON string on the wire, so result must be a string. + assert json.loads(frame["result"]) == {"status": "ok", "seat": "12A"} + + async def test_send_tool_result_survives_non_serializable_values(self): + ws = _FakeWebSocket() + session = AAIHostSession(ws, input_rate=16000, output_rate=24000) + + await session.send_tool_result("call_2", {"when": object()}) + + frame = json.loads(ws.sent[0]) + assert frame["toolCallId"] == "call_2" + + async def test_events_yields_audio_for_binary_frames(self): + ws = _FakeWebSocket([b"\x01\x02", json.dumps({"type": "reply_done"})]) + session = AAIHostSession(ws, input_rate=16000, output_rate=24000) + + events = [event async for event in session.events()] + + assert events[0].pcm == b"\x01\x02" + assert events[1] == TurnDone() + + async def test_events_skips_unmapped_and_malformed_frames(self): + ws = _FakeWebSocket( + [ + "not json at all", + json.dumps({"type": "speech_stopped"}), + json.dumps({"type": "agent_transcript", "text": "ok"}), + ] + ) + session = AAIHostSession(ws, input_rate=16000, output_rate=24000) + + events = [event async for event in session.events()] + + assert events == [AssistantTranscript(text="ok")] + + async def test_aclose_closes_the_socket(self): + ws = _FakeWebSocket() + session = AAIHostSession(ws, input_rate=16000, output_rate=24000) + + await session.aclose() + + assert ws.closed is True + + def test_declares_backend_rates(self): + session = AAIHostSession(_FakeWebSocket(), input_rate=16000, output_rate=24000) + assert session.backend_input_rate == 16000 + assert session.backend_output_rate == 24000 + + +class TestHandshake: + async def test_await_config_ack_accepts_config_event(self): + ws = _FakeWebSocket([json.dumps({"type": "config"})]) + await AAIHostSession._await_config_ack(ws, timeout_s=1.0) + + async def test_await_config_ack_skips_binary_frames(self): + ws = _FakeWebSocket([b"\x00\x00", json.dumps({"type": "config"})]) + await AAIHostSession._await_config_ack(ws, timeout_s=1.0) + + async def test_await_config_ack_raises_on_error_event(self): + from eva.assistant.aai_session import AAIHostSessionError + + ws = _FakeWebSocket([json.dumps({"type": "error", "code": "host_disabled", "message": "AAI_ALLOW_HOST"})]) + + with pytest.raises(AAIHostSessionError) as excinfo: + await AAIHostSession._await_config_ack(ws, timeout_s=1.0) + + assert "AAI_ALLOW_HOST" in str(excinfo.value) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest tests/unit/assistant/test_aai_session.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'eva.assistant.aai_session'` + +- [ ] **Step 3: Write the implementation** + +Create `src/eva/assistant/aai_session.py`: + +```python +"""aai host-mode session: the wire protocol behind AAIAssistantServer. + +aai agents normally execute their tools server-side in a sandbox. Host mode +inverts that: this client supplies the system prompt, greeting, and tool +*schemas* in its first ``config`` frame, and aai relays every tool *call* back +as a ``tool_call`` frame that resolves when the matching ``tool_result`` +arrives. That inversion is what lets EVA's ToolExecutor mutate the scenario +database — without it the accuracy metrics would score an unchanged database. + +Host mode is selected by the ``?host=1`` query parameter and gated server-side +by ``AAI_ALLOW_HOST``. +""" + +import asyncio +import contextlib +import json +from typing import Any, AsyncIterator +from urllib.parse import parse_qs, urlencode, urlparse, urlunparse + +import websockets + +from eva.assistant.aai_events import ( + AAIAgentTranscriptEvent, + AAIAudioDoneEvent, + AAIConfigEvent, + AAIErrorEvent, + AAIIdleTimeoutEvent, + AAIReplyDoneEvent, + AAISpeechStartedEvent, + AAIToolCallEvent, + AAIUserTranscriptEvent, + BaseAAIEvent, + parse_aai_event, +) +from eva.assistant.bridge_events import ( + AssistantTranscript, + AudioChunk, + BackendError, + BridgeEvent, + SpeechStarted, + ToolCall, + TurnDone, + UserTranscript, +) +from eva.utils.logging import get_logger + +logger = get_logger(__name__) + +DEFAULT_AAI_WS_URL = "ws://localhost:3000/websocket" +DEFAULT_AAI_INPUT_SAMPLE_RATE = 16000 +DEFAULT_AAI_OUTPUT_SAMPLE_RATE = 24000 + +#: aai host mode is endpoint-determined: the model is whatever the host runs. +DEFAULT_AAI_MODEL = "aai-host" + +CONFIG_ACK_TIMEOUT_S = 15.0 + + +class AAIHostSessionError(Exception): + """The aai host refused or failed the host-mode handshake.""" + + +def with_host_flag(url: str) -> str: + """Return *url* with ``host=1``, activating host mode on the aai server.""" + parsed = urlparse(url) + params = parse_qs(parsed.query) + params["host"] = ["1"] + return urlunparse(parsed._replace(query=urlencode(params, doseq=True))) + + +def build_host_config_message( + *, + system_prompt: str, + tools: list[dict], + greeting: str | None, + input_rate: int, + output_rate: int, +) -> dict: + """Build the host-mode handshake frame. + + Mirrors ``HostConfigMessageSchema`` in the aai SDK: the audio negotiation + fields ride alongside the ``host`` block in a single ``config`` frame. + """ + host: dict[str, Any] = {"systemPrompt": system_prompt, "tools": list(tools)} + if greeting: + host["greeting"] = greeting + return { + "type": "config", + "audioFormat": "pcm16", + "sampleRate": input_rate, + "ttsSampleRate": output_rate, + "host": host, + } + + +def map_aai_event(event: BaseAAIEvent) -> BridgeEvent | None: + """Translate an aai wire event into a bridge event, or None to ignore it. + + Both ``reply_done`` and ``audio_done`` map to ``TurnDone``: audio completion + is the truer end-of-turn signal for a voice call, but a text-only reply + produces no ``audio_done``, so honoring both avoids a stuck turn. The + bridge's turn completion is idempotent, making the duplicate harmless. + """ + if isinstance(event, AAIAgentTranscriptEvent): + return AssistantTranscript(text=event.text) + if isinstance(event, AAIUserTranscriptEvent): + return UserTranscript(text=event.text) + if isinstance(event, AAIToolCallEvent): + return ToolCall(call_id=event.tool_call_id, name=event.tool_name, arguments=event.args) + if isinstance(event, AAISpeechStartedEvent): + return SpeechStarted() + if isinstance(event, (AAIReplyDoneEvent, AAIAudioDoneEvent)): + return TurnDone() + if isinstance(event, AAIIdleTimeoutEvent): + return BackendError(message="aai session idle timeout", fatal=True) + if isinstance(event, AAIErrorEvent): + return BackendError(message=f"{event.code or 'error'}: {event.message or ''}".strip()) + return None + + +class AAIHostSession: + """One host-mode conversation with an aai voice agent. + + Satisfies ``VoiceBackendSession``. Audio is raw binary PCM16 frames; + everything else is JSON. + """ + + def __init__(self, ws, input_rate: int, output_rate: int): + self._ws = ws + self.backend_input_rate = input_rate + self.backend_output_rate = output_rate + + @classmethod + async def connect( + cls, + *, + ws_url: str, + system_prompt: str, + tools: list[dict], + greeting: str | None = None, + input_rate: int = DEFAULT_AAI_INPUT_SAMPLE_RATE, + output_rate: int = DEFAULT_AAI_OUTPUT_SAMPLE_RATE, + ) -> "AAIHostSession": + """Open a host-mode session and complete the handshake. + + Raises: + AAIHostSessionError: the host rejected the handshake, or did not + acknowledge it within ``CONFIG_ACK_TIMEOUT_S``. + """ + url = with_host_flag(ws_url) + logger.info(f"Connecting to aai host at {url} ({len(tools)} tools)") + try: + # max_size=None: TTS audio frames can exceed the 1 MiB default. + ws = await websockets.connect(url, max_size=None) + except Exception as e: + raise AAIHostSessionError(f"Could not connect to aai host at {url}: {e}") from e + + try: + await ws.send( + json.dumps( + build_host_config_message( + system_prompt=system_prompt, + tools=tools, + greeting=greeting, + input_rate=input_rate, + output_rate=output_rate, + ) + ) + ) + await cls._await_config_ack(ws, timeout_s=CONFIG_ACK_TIMEOUT_S) + except Exception: + with contextlib.suppress(Exception): + await ws.close() + raise + + logger.info("aai host-mode handshake complete") + return cls(ws, input_rate=input_rate, output_rate=output_rate) + + @staticmethod + async def _await_config_ack(ws, timeout_s: float) -> None: + """Block until the host acknowledges the config frame.""" + while True: + try: + raw = await asyncio.wait_for(ws.recv(), timeout=timeout_s) + except asyncio.TimeoutError as e: + raise AAIHostSessionError( + f"aai host did not acknowledge the host-mode config within {timeout_s}s " + f"(is AAI_ALLOW_HOST set?)" + ) from e + + if isinstance(raw, (bytes, bytearray)): + continue # Audio can precede the ack; ignore it. + + try: + event = parse_aai_event(json.loads(raw)) + except json.JSONDecodeError: + continue + + if isinstance(event, AAIConfigEvent): + return + if isinstance(event, AAIErrorEvent): + raise AAIHostSessionError(f"aai host rejected host mode: {event.code}: {event.message}") + + async def send_audio(self, pcm: bytes) -> None: + await self._ws.send(pcm) + + async def send_tool_result(self, call_id: str, result: Any) -> None: + """Relay a tool result. ``result`` travels as a JSON string, which aai unwraps.""" + await self._ws.send( + json.dumps( + { + "type": "tool_result", + "toolCallId": call_id, + "result": json.dumps(result, default=str), + } + ) + ) + + async def events(self) -> AsyncIterator[BridgeEvent]: + async for raw in self._ws: + if isinstance(raw, (bytes, bytearray)): + yield AudioChunk(pcm=bytes(raw)) + continue + try: + data = json.loads(raw) + except json.JSONDecodeError: + logger.warning("Discarding non-JSON text frame from aai host") + continue + mapped = map_aai_event(parse_aai_event(data)) + if mapped is not None: + yield mapped + + async def aclose(self) -> None: + with contextlib.suppress(Exception): + await self._ws.close() +``` + +- [ ] **Step 4: Run test to verify it passes** + +Run: `uv run pytest tests/unit/assistant/test_aai_session.py -v` +Expected: PASS (22 tests) + +- [ ] **Step 5: Lint and commit** + +```bash +cd ~/Code/eva +uv run ruff format src/eva/assistant/aai_session.py tests/unit/assistant/test_aai_session.py +uv run ruff check src/eva/assistant/aai_session.py tests/unit/assistant/test_aai_session.py +git add src/eva/assistant/aai_session.py tests/unit/assistant/test_aai_session.py +git commit -m "feat(aai): host-mode session client with tool relay" +``` + +--- + +### Task 4: aai assistant server and registration + +**Files:** +- Create: `src/eva/assistant/aai_server.py` +- Modify: `src/eva/models/config.py:506` +- Modify: `src/eva/orchestrator/worker.py:32-56` +- Test: `tests/unit/assistant/test_aai_server.py` + +**Interfaces:** +- Consumes: `WebSocketBridgeAssistantServer` (Task 2), `AAIHostSession` and the + `DEFAULT_AAI_*` constants (Task 3), `AgentConfig.tools` items which expose + `function_name`, `name`, `description`, `get_parameter_properties()`, and + `get_required_param_names()`. +- Produces: `AAIAssistantServer`, selected by `framework: "aai"`. + +- [ ] **Step 1: Write the failing test** + +Create `tests/unit/assistant/test_aai_server.py`: + +```python +"""Tests for AAIAssistantServer configuration, tool schemas, and registration.""" + +from unittest.mock import MagicMock + +from eva.assistant.aai_server import AAIAssistantServer +from eva.assistant.aai_session import DEFAULT_AAI_MODEL, DEFAULT_AAI_WS_URL + + +def _bare_server(s2s_params: dict | None = None) -> AAIAssistantServer: + """Construct without __init__ (skips PromptManager and ToolExecutor setup).""" + srv = object.__new__(AAIAssistantServer) + srv.pipeline_config = MagicMock() + srv.pipeline_config.s2s_params = s2s_params if s2s_params is not None else {} + srv.agent = MagicMock() + srv.agent.tools = [] + srv._model = (s2s_params or {}).get("model", DEFAULT_AAI_MODEL) + srv._ws_url = (s2s_params or {}).get("ws_url", DEFAULT_AAI_WS_URL) + return srv + + +def _tool(function_name: str, name: str, description: str, properties: dict, required: list[str]) -> MagicMock: + tool = MagicMock() + tool.function_name = function_name + tool.name = name + tool.description = description + tool.get_parameter_properties.return_value = properties + tool.get_required_param_names.return_value = required + return tool + + +class TestModelName: + def test_defaults_to_aai_host(self): + assert _bare_server().model_name == DEFAULT_AAI_MODEL + + def test_honors_configured_model(self): + assert _bare_server({"model": "aai-host-custom"}).model_name == "aai-host-custom" + + +class TestBuildAaiTools: + def test_returns_empty_list_when_agent_has_no_tools(self): + assert _bare_server()._build_aai_tools() == [] + + def test_builds_flat_function_schema(self): + srv = _bare_server() + srv.agent.tools = [ + _tool( + "get_reservation", + "Get Reservation", + "Look up a booking", + {"confirmation_number": {"type": "string"}}, + ["confirmation_number"], + ) + ] + + tools = srv._build_aai_tools() + + assert tools == [ + { + "type": "function", + "name": "get_reservation", + "description": "Get Reservation: Look up a booking", + "parameters": { + "type": "object", + "properties": {"confirmation_number": {"type": "string"}}, + "required": ["confirmation_number"], + }, + } + ] + + def test_description_is_never_empty(self): + """aai's ToolSchema requires a description of at least one character.""" + srv = _bare_server() + srv.agent.tools = [_tool("f", "", "", {}, [])] + + description = srv._build_aai_tools()[0]["description"] + + assert len(description) >= 1 + + +class TestWebSocketUrl: + def test_defaults_to_localhost(self): + assert _bare_server()._ws_url == DEFAULT_AAI_WS_URL + + def test_honors_configured_url(self): + assert _bare_server({"ws_url": "ws://aai.internal/websocket"})._ws_url == "ws://aai.internal/websocket" + + +class TestRegistration: + def test_framework_literal_accepts_aai(self): + from eva.models.config import RunConfig + + assert "aai" in RunConfig.model_fields["framework"].annotation.__args__ + + def test_worker_resolves_the_aai_server_class(self): + from eva.orchestrator.worker import _get_server_class + + assert _get_server_class("aai") is AAIAssistantServer + + def test_unknown_framework_error_lists_aai(self): + from eva.orchestrator.worker import _get_server_class + + try: + _get_server_class("nope") + except ValueError as e: + assert "aai" in str(e) + else: + raise AssertionError("expected ValueError") +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `uv run pytest tests/unit/assistant/test_aai_server.py -v` +Expected: FAIL — `ModuleNotFoundError: No module named 'eva.assistant.aai_server'` + +- [ ] **Step 3: Write the server** + +Create `src/eva/assistant/aai_server.py`: + +```python +"""EVA assistant server for the aai voice-agent framework. + +Bridges EVA's user simulator to an aai host-mode session. All the contract +plumbing lives in ``WebSocketBridgeAssistantServer``; this class only builds the +per-session agent definition — system prompt, greeting, and tool schemas — and +opens the backend. + +Requires a running aai host with ``AAI_ALLOW_HOST`` enabled. See +docs/aai_integration.md. +""" + +import os + +from eva.assistant.aai_session import ( + DEFAULT_AAI_INPUT_SAMPLE_RATE, + DEFAULT_AAI_MODEL, + DEFAULT_AAI_OUTPUT_SAMPLE_RATE, + DEFAULT_AAI_WS_URL, + AAIHostSession, +) +from eva.assistant.bridge_events import VoiceBackendSession +from eva.assistant.ws_bridge_server import WebSocketBridgeAssistantServer +from eva.utils.logging import get_logger + +logger = get_logger(__name__) + + +class AAIAssistantServer(WebSocketBridgeAssistantServer): + """Runs an EVA conversation against an aai host-mode agent.""" + + def __init__(self, **kwargs): + super().__init__(**kwargs) + s2s_params = self.pipeline_config.s2s_params or {} + self._model: str = s2s_params.get("model") or DEFAULT_AAI_MODEL + self._ws_url: str = s2s_params.get("ws_url") or os.environ.get("AAI_WS_URL") or DEFAULT_AAI_WS_URL + self._input_rate: int = int(s2s_params.get("input_sample_rate") or DEFAULT_AAI_INPUT_SAMPLE_RATE) + self._output_rate: int = int(s2s_params.get("output_sample_rate") or DEFAULT_AAI_OUTPUT_SAMPLE_RATE) + + @property + def model_name(self) -> str: + return self._model + + def _build_aai_tools(self) -> list[dict]: + """Convert the agent's tools to aai's flat ToolSchema shape. + + aai validates ``{type, name, description, parameters}`` with a non-empty + description and a JSON Schema object for parameters. + """ + tools: list[dict] = [] + for tool in self.agent.tools or []: + description = f"{tool.name}: {tool.description}".strip(": ").strip() or tool.function_name + tools.append( + { + "type": "function", + "name": tool.function_name, + "description": description, + "parameters": { + "type": "object", + "properties": tool.get_parameter_properties(), + "required": tool.get_required_param_names(), + }, + } + ) + return tools + + async def _open_backend(self) -> VoiceBackendSession: + return await AAIHostSession.connect( + ws_url=self._ws_url, + system_prompt=self._build_system_prompt(), + tools=self._build_aai_tools(), + greeting=self.initial_message, + input_rate=self._input_rate, + output_rate=self._output_rate, + ) +``` + +- [ ] **Step 4: Register the framework** + +In `src/eva/models/config.py`, replace the `framework` field at line 506: + +```python + framework: Literal["pipecat", "openai_realtime", "gemini_live", "elevenlabs", "grok_voice", "aai"] = Field( + "pipecat", + description=( + "Agent framework to use for the assistant server." + "'pipecat' (default): Pipecat pipeline." + "'openai_realtime': OpenAI Realtime API directly." + "'gemini_live': Gemini Live API via google-genai." + "'elevenlabs': ElevenLabs Conversational AI API." + "'grok_voice': xAI Grok voice realtime API." + "'aai': aai voice-agent framework in host mode." + ), +``` + +In `src/eva/orchestrator/worker.py`, add a branch before the `else` in `_get_server_class()` and update the error message: + +```python + elif framework == "aai": + from eva.assistant.aai_server import AAIAssistantServer + + return AAIAssistantServer + else: + raise ValueError( + f"Unknown framework: {framework!r}. " + "Supported: pipecat, openai_realtime, gemini_live, elevenlabs, grok_voice, aai" + ) +``` + +- [ ] **Step 5: Run test to verify it passes** + +Run: `uv run pytest tests/unit/assistant/test_aai_server.py -v` +Expected: PASS (10 tests) + +- [ ] **Step 6: Verify nothing else regressed** + +Run: `uv run pytest tests/unit -q` +Expected: PASS — no pre-existing test changes behavior. + +- [ ] **Step 7: Lint and commit** + +```bash +cd ~/Code/eva +uv run ruff format src/eva/assistant/aai_server.py tests/unit/assistant/test_aai_server.py +uv run ruff check src/eva/assistant/aai_server.py src/eva/models/config.py src/eva/orchestrator/worker.py +git add src/eva/assistant/aai_server.py src/eva/models/config.py src/eva/orchestrator/worker.py tests/unit/assistant/test_aai_server.py +git commit -m "feat(aai): assistant server and framework registration" +``` + +--- + +### Task 5: Full-session test against a fake aai backend + +**Files:** +- Test: `tests/integration/test_aai_fake_backend.py` + +**Interfaces:** +- Consumes: everything from Tasks 1–4. +- Produces: no source changes. This is the verification gate — it proves the two + hops connect, a tool call round-trips, and audio flows both ways, with no paid + APIs and no Node process. + +- [ ] **Step 1: Write the failing test** + +Create `tests/integration/test_aai_fake_backend.py`: + +```python +"""End-to-end bridge test against an in-process fake aai host. + +Exercises both WebSocket hops without any paid API or Node process: +a fake user simulator sends Twilio-framed mu-law audio to AAIAssistantServer, +which speaks host mode to a fake aai server that requests a tool and replies +with audio. +""" + +import asyncio +import audioop +import json +from unittest.mock import MagicMock + +import pytest +import websockets + +from eva.assistant.aai_server import AAIAssistantServer +from eva.assistant.audio_bridge import create_twilio_media_message + +FAKE_AAI_PORT = 38999 +BRIDGE_PORT = 38998 + + +class FakeAaiHost: + """Minimal aai host: completes the handshake, calls one tool, speaks, ends the turn.""" + + def __init__(self): + self.handshake: dict | None = None + self.tool_results: list[dict] = [] + self.received_audio_bytes = 0 + self._server = None + + async def start(self) -> None: + self._server = await websockets.serve(self._handle, "localhost", FAKE_AAI_PORT) + + async def stop(self) -> None: + if self._server is not None: + self._server.close() + await self._server.wait_closed() + + async def _handle(self, ws) -> None: + # 1. Handshake: first text frame carries the host config. + raw = await ws.recv() + self.handshake = json.loads(raw) + await ws.send(json.dumps({"type": "config"})) + + # 2. Ask the client to run a tool. + await ws.send( + json.dumps( + { + "type": "tool_call", + "toolCallId": "call_1", + "toolName": "probe_tool", + "args": {"query": "status"}, + } + ) + ) + + # 3. Wait for the relayed result, tolerating interleaved user audio. + while True: + frame = await ws.recv() + if isinstance(frame, (bytes, bytearray)): + self.received_audio_bytes += len(frame) + continue + message = json.loads(frame) + if message.get("type") == "tool_result": + self.tool_results.append(message) + break + + # 4. Speak: transcript, 100 ms of 24 kHz PCM16, then end the turn. + await ws.send(json.dumps({"type": "agent_transcript", "text": "Your status is green."})) + await ws.send(b"\x00\x01" * 2400) + await ws.send(json.dumps({"type": "reply_done"})) + await ws.send(json.dumps({"type": "audio_done"})) + + # Keep the socket open so the bridge can finish draining. + await asyncio.sleep(1.0) + + +def _make_server() -> AAIAssistantServer: + """Build a server with the heavy collaborators stubbed.""" + srv = object.__new__(AAIAssistantServer) + + srv.pipeline_config = MagicMock() + srv.pipeline_config.s2s_params = { + "model": "aai-host", + "ws_url": f"ws://localhost:{FAKE_AAI_PORT}/websocket", + } + srv.agent = MagicMock() + srv.agent.tools = [] + srv.conversation_id = "conv-test" + srv.port = BRIDGE_PORT + srv.initial_message = "Thank you for calling." + srv.audit_log = MagicMock() + srv.tool_handler = MagicMock() + + async def _fake_execute(name, args): + return {"status": "ok", "echo": args} + + srv.execute_tool = _fake_execute + srv._build_system_prompt = lambda: "You are a test agent." + + # Attributes normally set by the two __init__ methods. + srv._model = "aai-host" + srv._ws_url = f"ws://localhost:{FAKE_AAI_PORT}/websocket" + srv._input_rate = 16000 + srv._output_rate = 24000 + srv._session = None + srv._to_backend = None + srv._from_backend = None + srv._stream_sid = "conv-test" + srv._audio_out = asyncio.Queue() + srv._tasks = [] + srv._user_speech_start_ms = None + srv._user_speech_stop_ms = None + srv._user_speaking = False + srv._assistant_speaking = False + srv._turn_first_audio_ms = None + srv._assistant_text = "" + srv.user_audio_buffer = bytearray() + srv.assistant_audio_buffer = bytearray() + srv._audio_buffer = bytearray() + srv._audio_sample_rate = 24000 + srv._app = None + srv._server = None + srv._server_task = None + srv._running = False + return srv + + +@pytest.fixture +async def fake_host(): + host = FakeAaiHost() + await host.start() + yield host + await host.stop() + + +async def test_full_session_round_trip(fake_host, tmp_path): + srv = _make_server() + srv.output_dir = tmp_path + await srv.start() + + received_audio = bytearray() + try: + async with websockets.connect(f"ws://localhost:{BRIDGE_PORT}/ws") as sim: + await sim.send(json.dumps({"event": "start", "start": {"streamSid": "stream-1"}})) + + # 200 ms of silence as mu-law 8 kHz, in 20 ms Twilio frames. + silence_mulaw = audioop.lin2ulaw(b"\x00\x00" * 1600, 2) + for offset in range(0, len(silence_mulaw), 160): + await sim.send(create_twilio_media_message("stream-1", silence_mulaw[offset : offset + 160])) + await asyncio.sleep(0.005) + + await sim.send(json.dumps({"event": "user_speech_stop", "timestamp_ms": str(_now_ms())})) + + # Collect whatever the bridge paces back. + deadline = asyncio.get_running_loop().time() + 3.0 + while asyncio.get_running_loop().time() < deadline: + try: + raw = await asyncio.wait_for(sim.recv(), timeout=0.3) + except asyncio.TimeoutError: + if received_audio: + break + continue + message = json.loads(raw) + if message.get("event") == "media": + import base64 + + received_audio.extend(base64.b64decode(message["media"]["payload"])) + finally: + await srv.stop() + + # Handshake carried the injected agent definition. + assert fake_host.handshake is not None + assert fake_host.handshake["type"] == "config" + assert fake_host.handshake["audioFormat"] == "pcm16" + assert fake_host.handshake["sampleRate"] == 16000 + assert fake_host.handshake["ttsSampleRate"] == 24000 + assert fake_host.handshake["host"]["systemPrompt"] == "You are a test agent." + assert fake_host.handshake["host"]["greeting"] == "Thank you for calling." + + # User audio reached the backend. + assert fake_host.received_audio_bytes > 0 + + # The tool round-tripped through EVA, not the backend's sandbox. + assert len(fake_host.tool_results) == 1 + assert fake_host.tool_results[0]["toolCallId"] == "call_1" + assert json.loads(fake_host.tool_results[0]["result"])["status"] == "ok" + + # Assistant audio came back as 160-byte mu-law frames. + assert len(received_audio) > 0 + assert len(received_audio) % 160 == 0 + + # The turn was recorded. + srv.audit_log.append_assistant_output.assert_called() + assert srv.audit_log.append_assistant_output.call_args[0][0] == "Your status is green." + + +def _now_ms() -> int: + import time + + return int(time.time() * 1000) +``` + +- [ ] **Step 2: Run test to verify it fails or reveals integration bugs** + +Run: `uv run pytest tests/integration/test_aai_fake_backend.py -v` +Expected: This is the first time all four modules run together. If it fails, +the failure is a real integration bug — fix the source, not the test. The most +likely causes, in order: + +1. `stop()` calling `_save_scenario_dbs()` against a `MagicMock` tool handler — + if `save_outputs` raises, stub `srv.get_initial_scenario_db` and + `srv.get_final_scenario_db` to return `{}` in `_make_server()`. +2. The pacing task exiting before draining, because `_running` flipped false — + confirm `_shutdown()` cancels tasks only after the assertion window. +3. `websockets.serve` handler signature differing across versions — if the + installed `websockets` passes `(ws, path)`, accept a second optional arg in + `FakeAaiHost._handle`. + +- [ ] **Step 3: Run the whole suite** + +Run: `uv run pytest tests/unit tests/integration/test_aai_fake_backend.py -q` +Expected: PASS + +- [ ] **Step 4: Lint and commit** + +```bash +cd ~/Code/eva +uv run ruff format tests/integration/test_aai_fake_backend.py +uv run ruff check tests/integration/test_aai_fake_backend.py +git add tests/integration/test_aai_fake_backend.py +git commit -m "test(aai): full-session bridge test against a fake aai host" +``` + +--- + +### Task 6: Documentation + +**Files:** +- Create: `docs/aai_integration.md` +- Modify: `.env.example` + +**Interfaces:** +- Consumes: the finished integration. +- Produces: the run instructions the user needs for the live end-to-end pass. + +- [ ] **Step 1: Write the integration doc** + +Create `docs/aai_integration.md`: + +```markdown +# Evaluating the aai Voice Agent + +EVA can evaluate the [aai](https://github.com/alexkroman/aai) voice-agent +framework via aai's **host mode**, in which EVA supplies the domain system +prompt, greeting, and tool schemas per session and aai relays each tool call +back to EVA for execution. + +Tool execution must happen inside EVA: the accuracy metrics diff +`initial_scenario_db.json` against `final_scenario_db.json`, so a backend that +ran tools internally would score as if nothing happened. + +## Prerequisites + +1. **A running aai host with host mode enabled.** From your aai checkout: + + ```bash + AAI_ALLOW_HOST=1 pnpm dev + ``` + + The default endpoint is `ws://localhost:3000/websocket`; EVA appends + `?host=1` to select host mode. Without `AAI_ALLOW_HOST`, the host rejects the + handshake and EVA aborts the conversation with a logged error. + +2. **EVA's usual keys** in `.env` — a user simulator (ElevenLabs or OpenAI + Realtime) plus the judge-model credentials the metrics need. See the main + README. + +## Configuration + +| Setting | Default | Purpose | +|---|---|---| +| `EVA_FRAMEWORK=aai` | — | Selects this server. | +| `EVA_MODEL__S2S=aai-host` | — | Marks the run as speech-to-speech. | +| `AAI_WS_URL` | `ws://localhost:3000/websocket` | aai host endpoint. | +| `s2s_params.ws_url` | — | Per-run override, takes precedence over `AAI_WS_URL`. | +| `s2s_params.model` | `aai-host` | Label recorded in the metrics log. | +| `s2s_params.input_sample_rate` | `16000` | PCM16 rate sent to aai. | +| `s2s_params.output_sample_rate` | `24000` | PCM16 rate received from aai. | + +## Running + +```bash +AAI_ALLOW_HOST=1 EVA_FRAMEWORK=aai EVA_MODEL__S2S=aai-host \ +EVA_USER_SIMULATOR__PROVIDER=openai_realtime EVA_DOMAIN=itsm \ +EVA_RECORD_IDS=15 EVA_MAX_CONCURRENT_CONVERSATIONS=1 eva +``` + +`EVA_MAX_CONCURRENT_CONVERSATIONS` above 1 works: host mode builds a fresh +single-use runtime per connection, so each conversation gets its own session. + +## Architecture + +``` +user simulator ──Twilio μ-law 8k──▶ AAIAssistantServer ──PCM16 16k───▶ aai host + ◀──Twilio μ-law 8k── localhost:{port}/ws ◀──PCM16 24k── ?host=1 + │ + execute_tool() + ▼ + scenario database +``` + +| Module | Responsibility | +|---|---| +| `assistant/ws_bridge_server.py` | Shared bridge: Twilio framing, 20 ms pacing, recording buffers, turn bookkeeping, metrics. | +| `assistant/bridge_events.py` | Backend-agnostic event vocabulary and the session protocol. | +| `assistant/aai_session.py` | aai host handshake, audio frames, `tool_result` relay. | +| `assistant/aai_events.py` | aai wire-event models. | +| `assistant/aai_server.py` | Prompt and tool-schema construction. | + +Adding another WebSocket voice backend means writing one session adapter plus a +thin server subclass — no changes to the bridge. + +## Known gaps + +- **No token usage.** aai emits no usage events, so `pipecat_metrics.jsonl` + contains latency entries only. `model_response_latency` is unaffected; + token-based cost reporting is unavailable for aai runs. +- **Backend output must be 24 kHz.** Only a 24 kHz → μ-law converter exists in + `audio_bridge.py`. Another rate raises at session start with a clear message + rather than producing distorted audio. + +## Troubleshooting + +| Symptom | Cause | +|---|---| +| `aai host did not acknowledge the host-mode config` | `AAI_ALLOW_HOST` not set, or the host is not listening. | +| `aai host rejected host mode` | Host mode disabled server-side, or the tool schemas failed validation (each tool needs a non-empty description). | +| `Could not connect to aai host` | Wrong `AAI_WS_URL`, or the host is not running. | +| Empty transcript, conversation scored as failure | Check the log for `Failed to open backend session` — the handshake aborted. | +``` + +- [ ] **Step 2: Document the environment variable** + +Append to `.env.example`: + +```bash +# ── aai framework (EVA_FRAMEWORK=aai) ──────────────────────────────── +# WebSocket endpoint of a running aai host. EVA appends ?host=1 to select +# host mode; the host must be started with AAI_ALLOW_HOST=1. +# See docs/aai_integration.md +AAI_WS_URL=ws://localhost:3000/websocket +``` + +- [ ] **Step 3: Verify the docs match reality** + +Run: `uv run pytest tests/unit/assistant/test_aai_server.py tests/unit/assistant/test_aai_session.py -q` + +Then confirm by inspection that every default named in the doc table matches the +constants in `aai_session.py` and the `s2s_params` keys read in +`aai_server.py.__init__`. A doc that drifts from the code is worse than no doc. + +- [ ] **Step 4: Commit** + +```bash +cd ~/Code/eva +git add docs/aai_integration.md .env.example +git commit -m "docs(aai): integration setup, configuration, and known gaps" +``` + +--- + +## Final verification + +- [ ] **Run the full suite** + +Run: `uv run pytest tests/unit tests/integration/test_aai_fake_backend.py -q` +Expected: PASS, with no pre-existing test newly failing. + +- [ ] **Confirm the additive constraint held** + +Run: `git diff --stat main -- src/eva/assistant/` +Expected: only `aai_events.py`, `aai_server.py`, `aai_session.py`, +`bridge_events.py`, and `ws_bridge_server.py` appear. If any of the five +existing servers or `base_server.py` shows up, revert that change. + +- [ ] **Lint the whole change** + +Run: `uv run ruff check src/eva tests` and `uv run ruff format --check src/eva tests` +Expected: clean. + +- [ ] **Report honestly** + +State plainly that the live end-to-end run was not performed, that it requires +`.env` credentials plus a running aai host, and give the command from +`docs/aai_integration.md`. Do not describe the integration as verified +end-to-end on the strength of the fake-backend test alone. From 9e8540069ed86e367b0229246859448bbf45c5fd Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Wed, 29 Jul 2026 00:12:44 -0400 Subject: [PATCH 03/11] feat(aai): wire-event models and tolerant parser Ported from the tau2-bench aai provider, with loguru swapped for eva's logger and the tick-loop-only event types dropped. Parsing never raises: unknown and malformed frames become AAIUnknownEvent so one bad frame cannot abort a conversation mid-evaluation. Co-Authored-By: Claude Opus 5 (1M context) --- src/eva/assistant/aai_events.py | 168 ++++++++++++++++++++++++ tests/unit/assistant/test_aai_events.py | 77 +++++++++++ 2 files changed, 245 insertions(+) create mode 100644 src/eva/assistant/aai_events.py create mode 100644 tests/unit/assistant/test_aai_events.py diff --git a/src/eva/assistant/aai_events.py b/src/eva/assistant/aai_events.py new file mode 100644 index 00000000..073acf99 --- /dev/null +++ b/src/eva/assistant/aai_events.py @@ -0,0 +1,168 @@ +"""Pydantic models for aai voice-agent wire events. + +The aai host sends camelCase JSON frames over the session WebSocket; binary +frames carry PCM16 audio and are handled by the session, not here. + +Ported from the tau2-bench aai provider. Parsing never raises: an unrecognized +or malformed frame becomes an ``AAIUnknownEvent`` so a single bad frame cannot +abort a conversation mid-evaluation. +""" + +from typing import Any, Literal + +from pydantic import BaseModel, ConfigDict, Field + +from eva.utils.logging import get_logger + +logger = get_logger(__name__) + + +class BaseAAIEvent(BaseModel): + """Base class for all aai events. + + ``extra="ignore"`` so that new server-side fields never break parsing. + """ + + model_config = ConfigDict(extra="ignore", populate_by_name=True) + + type: str + + +class AAIConfigEvent(BaseAAIEvent): + """Server acknowledgment of the client's config frame — the handshake completing.""" + + type: Literal["config"] = "config" + + +class AAISpeechStartedEvent(BaseAAIEvent): + """aai's VAD detected the start of user speech.""" + + type: Literal["speech_started"] = "speech_started" + + +class AAISpeechStoppedEvent(BaseAAIEvent): + """aai's VAD detected the end of user speech.""" + + type: Literal["speech_stopped"] = "speech_stopped" + + +class AAIUserTranscriptEvent(BaseAAIEvent): + """Transcript of a user turn, as heard by aai.""" + + type: Literal["user_transcript"] = "user_transcript" + text: str + turn_order: int | None = Field(default=None, alias="turnOrder") + + +class AAIAgentTranscriptEvent(BaseAAIEvent): + """Transcript of the agent's reply. Carries the full text, not a delta.""" + + type: Literal["agent_transcript"] = "agent_transcript" + text: str + + +class AAIToolCallEvent(BaseAAIEvent): + """A relayed tool call awaiting a ``tool_result`` from this client.""" + + type: Literal["tool_call"] = "tool_call" + tool_call_id: str = Field(alias="toolCallId") + tool_name: str = Field(alias="toolName") + args: dict = Field(default_factory=dict) + + +class AAIToolCallDoneEvent(BaseAAIEvent): + """Acknowledgment that a relayed tool call was resolved.""" + + type: Literal["tool_call_done"] = "tool_call_done" + tool_call_id: str = Field(alias="toolCallId") + result: str = "" + + +class AAIReplyDoneEvent(BaseAAIEvent): + """The agent's reply is complete.""" + + type: Literal["reply_done"] = "reply_done" + + +class AAIAudioDoneEvent(BaseAAIEvent): + """The agent's audio output is complete.""" + + type: Literal["audio_done"] = "audio_done" + + +class AAICancelledEvent(BaseAAIEvent): + """An in-flight reply was cancelled.""" + + type: Literal["cancelled"] = "cancelled" + + +class AAIResetEvent(BaseAAIEvent): + """Session state was reset.""" + + type: Literal["reset"] = "reset" + + +class AAIIdleTimeoutEvent(BaseAAIEvent): + """The session was closed for inactivity.""" + + type: Literal["idle_timeout"] = "idle_timeout" + + +class AAIErrorEvent(BaseAAIEvent): + """An error reported by the aai host.""" + + type: Literal["error"] = "error" + code: str | None = None + message: str | None = None + + +class AAICustomEvent(BaseAAIEvent): + """An application-defined event emitted by the agent.""" + + type: Literal["custom_event"] = "custom_event" + event: str + data: Any | None = None + + +class AAIUnknownEvent(BaseAAIEvent): + """An unrecognized or unparseable frame, preserved for logging.""" + + type: str + raw: dict | None = None + + +_EVENT_TYPE_MAP: dict[str, type[BaseAAIEvent]] = { + "config": AAIConfigEvent, + "speech_started": AAISpeechStartedEvent, + "speech_stopped": AAISpeechStoppedEvent, + "user_transcript": AAIUserTranscriptEvent, + "agent_transcript": AAIAgentTranscriptEvent, + "tool_call": AAIToolCallEvent, + "tool_call_done": AAIToolCallDoneEvent, + "reply_done": AAIReplyDoneEvent, + "audio_done": AAIAudioDoneEvent, + "cancelled": AAICancelledEvent, + "reset": AAIResetEvent, + "idle_timeout": AAIIdleTimeoutEvent, + "error": AAIErrorEvent, + "custom_event": AAICustomEvent, +} + + +def parse_aai_event(data: dict) -> BaseAAIEvent: + """Parse a raw aai frame into a typed event. + + Never raises. Unknown types and validation failures both yield + ``AAIUnknownEvent`` with the original payload attached. + """ + event_type = data.get("type", "unknown") + event_class = _EVENT_TYPE_MAP.get(event_type) + + if event_class is None: + return AAIUnknownEvent(type=event_type, raw=data) + + try: + return event_class.model_validate(data) + except Exception as e: + logger.warning(f"Failed to parse aai event {event_type}: {e}") + return AAIUnknownEvent(type=event_type, raw=data) diff --git a/tests/unit/assistant/test_aai_events.py b/tests/unit/assistant/test_aai_events.py new file mode 100644 index 00000000..a9b7c9b6 --- /dev/null +++ b/tests/unit/assistant/test_aai_events.py @@ -0,0 +1,77 @@ +"""Tests for aai wire-event parsing.""" + +from eva.assistant.aai_events import ( + AAIAgentTranscriptEvent, + AAIErrorEvent, + AAIIdleTimeoutEvent, + AAIReplyDoneEvent, + AAISpeechStartedEvent, + AAIToolCallEvent, + AAIUnknownEvent, + AAIUserTranscriptEvent, + parse_aai_event, +) + + +class TestParseAaiEvent: + def test_parses_tool_call_camel_case_aliases(self): + event = parse_aai_event( + { + "type": "tool_call", + "toolCallId": "call_abc", + "toolName": "get_reservation", + "args": {"confirmation_number": "DJ3LPO"}, + } + ) + assert isinstance(event, AAIToolCallEvent) + assert event.tool_call_id == "call_abc" + assert event.tool_name == "get_reservation" + assert event.args == {"confirmation_number": "DJ3LPO"} + + def test_tool_call_args_default_to_empty_dict(self): + event = parse_aai_event({"type": "tool_call", "toolCallId": "c1", "toolName": "list_flights"}) + assert isinstance(event, AAIToolCallEvent) + assert event.args == {} + + def test_parses_agent_transcript(self): + event = parse_aai_event({"type": "agent_transcript", "text": "How can I help?"}) + assert isinstance(event, AAIAgentTranscriptEvent) + assert event.text == "How can I help?" + + def test_parses_user_transcript_with_turn_order(self): + event = parse_aai_event({"type": "user_transcript", "text": "hello", "turnOrder": 3}) + assert isinstance(event, AAIUserTranscriptEvent) + assert event.text == "hello" + assert event.turn_order == 3 + + def test_parses_zero_field_events(self): + assert isinstance(parse_aai_event({"type": "speech_started"}), AAISpeechStartedEvent) + assert isinstance(parse_aai_event({"type": "reply_done"}), AAIReplyDoneEvent) + assert isinstance(parse_aai_event({"type": "idle_timeout"}), AAIIdleTimeoutEvent) + + def test_parses_error_with_code_and_message(self): + event = parse_aai_event({"type": "error", "code": "host_disabled", "message": "AAI_ALLOW_HOST"}) + assert isinstance(event, AAIErrorEvent) + assert event.code == "host_disabled" + assert event.message == "AAI_ALLOW_HOST" + + def test_ignores_unknown_extra_fields(self): + event = parse_aai_event({"type": "agent_transcript", "text": "hi", "futureField": 1}) + assert isinstance(event, AAIAgentTranscriptEvent) + + def test_unrecognized_type_returns_unknown_event(self): + event = parse_aai_event({"type": "brand_new_thing", "x": 1}) + assert isinstance(event, AAIUnknownEvent) + assert event.type == "brand_new_thing" + assert event.raw == {"type": "brand_new_thing", "x": 1} + + def test_missing_type_returns_unknown_event(self): + event = parse_aai_event({"text": "orphan"}) + assert isinstance(event, AAIUnknownEvent) + assert event.type == "unknown" + + def test_malformed_known_event_returns_unknown_rather_than_raising(self): + # `text` is required on agent_transcript; a malformed frame must never abort a conversation. + event = parse_aai_event({"type": "agent_transcript"}) + assert isinstance(event, AAIUnknownEvent) + assert event.type == "agent_transcript" From a7c703c2a12fec73dbea297b1a646d82b67c953e Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Wed, 29 Jul 2026 00:15:37 -0400 Subject: [PATCH 04/11] feat(assistant): shared WebSocket bridge base for voice backends WebSocketBridgeAssistantServer owns everything the assistant-server contract requires -- Twilio framing, 20ms output pacing, time-aligned recording buffers at a fixed 24kHz, turn bookkeeping, latency and audit writes -- leaving a backend to supply only a VoiceBackendSession. Existing servers are untouched; this sits alongside them. Co-Authored-By: Claude Opus 5 (1M context) --- src/eva/assistant/bridge_events.py | 99 +++++ src/eva/assistant/ws_bridge_server.py | 377 ++++++++++++++++++ tests/unit/assistant/test_ws_bridge_server.py | 331 +++++++++++++++ 3 files changed, 807 insertions(+) create mode 100644 src/eva/assistant/bridge_events.py create mode 100644 src/eva/assistant/ws_bridge_server.py create mode 100644 tests/unit/assistant/test_ws_bridge_server.py diff --git a/src/eva/assistant/bridge_events.py b/src/eva/assistant/bridge_events.py new file mode 100644 index 00000000..5224d115 --- /dev/null +++ b/src/eva/assistant/bridge_events.py @@ -0,0 +1,99 @@ +"""Backend-agnostic events and the session protocol used by the bridge server. + +A voice backend (aai host mode, a hosted realtime API, ...) speaks its own wire +protocol. ``WebSocketBridgeAssistantServer`` never sees that protocol: an +adapter translates it into the small event vocabulary below, and the bridge +turns those events into what EVA's evaluation contract requires. + +Keeping this module free of EVA imports means a new backend adapter can be +written and unit-tested without pulling in the server. +""" + +from collections.abc import AsyncIterator +from dataclasses import dataclass +from typing import Any, Protocol, runtime_checkable + + +@dataclass(frozen=True) +class AudioChunk: + """PCM16 audio from the backend, at its declared ``backend_output_rate``.""" + + pcm: bytes + + +@dataclass(frozen=True) +class AssistantTranscript: + """The assistant's reply text. + + Full-text semantics: each event supersedes the previous one for the current + turn rather than appending to it. + """ + + text: str + + +@dataclass(frozen=True) +class UserTranscript: + """A user turn as transcribed by the backend.""" + + text: str + + +@dataclass(frozen=True) +class ToolCall: + """A tool the backend wants executed. The bridge runs it and replies.""" + + call_id: str + name: str + arguments: dict + + +@dataclass(frozen=True) +class SpeechStarted: + """Backend VAD detected user speech. Triggers barge-in if the assistant is speaking.""" + + +@dataclass(frozen=True) +class TurnDone: + """The assistant's turn is complete. Safe to deliver more than once per turn.""" + + +@dataclass(frozen=True) +class BackendError: + """A backend-reported problem. ``fatal`` ends the session.""" + + message: str + fatal: bool = False + + +BridgeEvent = AudioChunk | AssistantTranscript | UserTranscript | ToolCall | SpeechStarted | TurnDone | BackendError + + +@runtime_checkable +class VoiceBackendSession(Protocol): + """One live conversation with a voice backend. + + Implementations own their wire protocol and nothing else: no Twilio framing, + no recording buffers, no audit logging. + """ + + #: Sample rate of the PCM16 audio this session expects from ``send_audio``. + backend_input_rate: int + #: Sample rate of the PCM16 audio this session emits in ``AudioChunk``. + backend_output_rate: int + + async def send_audio(self, pcm: bytes) -> None: + """Send PCM16 user audio at ``backend_input_rate``.""" + ... + + async def send_tool_result(self, call_id: str, result: Any) -> None: + """Resolve the relayed tool call identified by ``call_id``.""" + ... + + def events(self) -> AsyncIterator[BridgeEvent]: + """Yield events until the backend closes the session.""" + ... + + async def aclose(self) -> None: + """Close the session. Must be safe to call more than once.""" + ... diff --git a/src/eva/assistant/ws_bridge_server.py b/src/eva/assistant/ws_bridge_server.py new file mode 100644 index 00000000..014eb66e --- /dev/null +++ b/src/eva/assistant/ws_bridge_server.py @@ -0,0 +1,377 @@ +"""Shared base for assistant servers that bridge EVA to a WebSocket voice backend. + +EVA's user simulator dials in over a Twilio-framed WebSocket; a voice backend +speaks its own protocol on its own socket. Every such integration needs the same +middle layer, and getting any part of it wrong corrupts the evaluation rather +than merely degrading audio: + +* **Output pacing.** The simulator infers turn boundaries from arrival timing, + so assistant audio must leave at wall-clock 20 ms / 160-byte cadence. +* **Track alignment.** The two recording buffers form a shared timeline; each + must be padded to the other's position before it advances, or the mixed WAV + is skewed and every audio judge metric reads the wrong thing. +* **Recording rate.** Both buffers are written at 24 kHz regardless of the + backend's wire rates, so inbound mu-law is converted twice: once for the wire, + once for the recording buffer. + +Subclasses supply only a ``VoiceBackendSession`` and a model identifier. + +See docs/assistant_server_contract.md for the contract this satisfies. Note that +the doc is stale on shutdown: ``AbstractAssistantServer.stop()`` is a concrete +template method and ``_shutdown()`` is the hook implemented here. +""" + +import asyncio +import contextlib +import json +import time +from abc import abstractmethod +from collections.abc import Callable + +import uvicorn +from fastapi import FastAPI, WebSocket + +from eva.assistant.audio_bridge import ( + FrameworkLogWriter, + MetricsLogWriter, + create_twilio_media_message, + mulaw_8k_to_pcm16_16k, + mulaw_8k_to_pcm16_24k, + parse_twilio_media_message, + pcm16_24k_to_mulaw_8k, + sync_buffer_to_position, +) +from eva.assistant.base_server import AbstractAssistantServer +from eva.assistant.bridge_events import ( + AssistantTranscript, + AudioChunk, + BackendError, + SpeechStarted, + ToolCall, + TurnDone, + UserTranscript, + VoiceBackendSession, +) +from eva.utils.logging import get_logger + +logger = get_logger(__name__) + +#: Both recording buffers are written at this rate, whatever the backend uses. +RECORDING_SAMPLE_RATE = 24000 + +#: mu-law bytes per 20 ms frame at 8 kHz — one Twilio media message. +MULAW_FRAME_BYTES = 160 +FRAME_INTERVAL_S = 0.02 + +#: A model_response latency outside this bound indicates a bookkeeping bug, not a slow model. +MAX_PLAUSIBLE_LATENCY_MS = 30_000 + +#: mu-law 8 kHz -> PCM16 at the backend's input rate. +_TO_BACKEND: dict[int, Callable[[bytes], bytes]] = { + 16000: mulaw_8k_to_pcm16_16k, + 24000: mulaw_8k_to_pcm16_24k, +} + +#: PCM16 at the backend's output rate -> mu-law 8 kHz. +_FROM_BACKEND: dict[int, Callable[[bytes], bytes]] = { + 24000: pcm16_24k_to_mulaw_8k, +} + + +class WebSocketBridgeAssistantServer(AbstractAssistantServer): + """Bridges the user simulator's Twilio socket to a ``VoiceBackendSession``.""" + + def __init__(self, **kwargs): + super().__init__(**kwargs) + self._audio_sample_rate = RECORDING_SAMPLE_RATE + + self._session: VoiceBackendSession | None = None + self._to_backend: Callable[[bytes], bytes] | None = None + self._from_backend: Callable[[bytes], bytes] | None = None + + self._stream_sid: str = self.conversation_id + self._audio_out: asyncio.Queue[bytes] = asyncio.Queue() + self._tasks: list[asyncio.Task] = [] + + # Turn bookkeeping + self._user_speech_start_ms: int | None = None + self._user_speech_stop_ms: int | None = None + self._user_speaking = False + self._assistant_speaking = False + self._turn_first_audio_ms: int | None = None + self._assistant_text = "" + + # ── Subclass contract ───────────────────────────────────────────── + + @property + @abstractmethod + def model_name(self) -> str: + """Model identifier recorded in the metrics log.""" + ... + + @abstractmethod + async def _open_backend(self) -> VoiceBackendSession: + """Open a session with the backend. Raise to abort the conversation.""" + ... + + # ── Lifecycle ───────────────────────────────────────────────────── + + async def start(self) -> None: + self.output_dir.mkdir(parents=True, exist_ok=True) + self._fw_log = FrameworkLogWriter(self.output_dir) + self._metrics_log = MetricsLogWriter(self.output_dir) + + self._app = FastAPI() + + @self._app.websocket("/ws") + async def ws_endpoint(websocket: WebSocket): + await websocket.accept() + await self._handle_session(websocket) + + @self._app.websocket("/") + async def ws_root(websocket: WebSocket): + await websocket.accept() + await self._handle_session(websocket) + + config = uvicorn.Config(self._app, host="0.0.0.0", port=self.port, log_level="warning") + self._server = uvicorn.Server(config) + self._server_task = asyncio.create_task(self._server.serve()) + while not self._server.started: + await asyncio.sleep(0.05) + self._running = True + logger.info(f"{type(self).__name__} listening on ws://localhost:{self.port}/ws") + + async def _shutdown(self) -> None: + self._running = False + + for task in self._tasks: + task.cancel() + for task in self._tasks: + with contextlib.suppress(asyncio.CancelledError, Exception): + await task + self._tasks.clear() + + if self._session is not None: + with contextlib.suppress(Exception): + await self._session.aclose() + self._session = None + + if self._server is not None: + self._server.should_exit = True + if self._server_task is not None: + self._server_task.cancel() + with contextlib.suppress(asyncio.CancelledError): + await self._server_task + + # ── Session ─────────────────────────────────────────────────────── + + async def _handle_session(self, websocket: WebSocket) -> None: + """Run one conversation: simulator socket in, backend socket out.""" + try: + self._session = await self._open_backend() + except Exception as e: + # Loud failure: a silent fallback would yield a scored but meaningless run. + logger.error(f"Failed to open backend session, aborting conversation: {e}", exc_info=True) + with contextlib.suppress(Exception): + await websocket.close() + return + + self._to_backend = self._converter(_TO_BACKEND, self._session.backend_input_rate, "input") + self._from_backend = self._converter(_FROM_BACKEND, self._session.backend_output_rate, "output") + + self._tasks = [ + asyncio.create_task(self._forward_user_audio(websocket)), + asyncio.create_task(self._process_backend_events()), + asyncio.create_task(self._pace_audio_output(websocket)), + ] + done, pending = await asyncio.wait(self._tasks, return_when=asyncio.FIRST_COMPLETED) + for task in pending: + task.cancel() + for task in done: + if task.exception() is not None: + logger.error(f"Bridge task failed: {task.exception()}") + + @staticmethod + def _converter(table: dict[int, Callable[[bytes], bytes]], rate: int, direction: str) -> Callable[[bytes], bytes]: + converter = table.get(rate) + if converter is None: + raise ValueError( + f"No mu-law converter for backend {direction} rate {rate} Hz " + f"(available: {sorted(table)}). Add one to audio_bridge.py." + ) + return converter + + async def _forward_user_audio(self, websocket: WebSocket) -> None: + """Simulator -> backend, plus the user recording track.""" + while self._running: + try: + raw = await websocket.receive_text() + except Exception: + break + + try: + msg = json.loads(raw) + except json.JSONDecodeError: + continue + event = msg.get("event") + + if event == "start": + self._stream_sid = msg.get("start", {}).get("streamSid", self._stream_sid) + elif event == "stop": + break + elif event == "user_speech_start": + self._user_speech_start_ms = _as_int_ms(msg.get("timestamp_ms")) + self._user_speaking = True + elif event == "user_speech_stop": + self._user_speech_stop_ms = _as_int_ms(msg.get("timestamp_ms")) + self._user_speaking = False + elif event == "media": + mulaw = parse_twilio_media_message(raw) + if not mulaw: + continue + await self._session.send_audio(self._to_backend(mulaw)) + self._record_user_audio(mulaw) + + def _record_user_audio(self, mulaw: bytes) -> None: + """Append to the user track at the recording rate, keeping tracks aligned.""" + pcm = mulaw_8k_to_pcm16_24k(mulaw) + if not self._assistant_speaking: + sync_buffer_to_position(self.assistant_audio_buffer, len(self.user_audio_buffer)) + self.user_audio_buffer.extend(pcm) + + async def _process_backend_events(self) -> None: + """Backend -> audit log, metrics, recording track, and output queue.""" + async for event in self._session.events(): + if isinstance(event, AudioChunk): + self._on_audio_chunk(event.pcm) + elif isinstance(event, AssistantTranscript): + self._assistant_text = event.text + elif isinstance(event, UserTranscript): + timestamp = str(self._user_speech_start_ms or self._now_ms()) + self.audit_log.append_user_input(event.text, timestamp_ms=timestamp) + elif isinstance(event, ToolCall): + await self._handle_tool_call(event) + elif isinstance(event, SpeechStarted): + self._handle_barge_in() + elif isinstance(event, TurnDone): + self._finish_turn(interrupted=False) + elif isinstance(event, BackendError): + logger.error(f"Backend error: {event.message}") + if event.fatal: + break + + def _on_audio_chunk(self, pcm: bytes) -> None: + """Record assistant audio, start the turn if needed, and queue for paced send.""" + if self._turn_first_audio_ms is None: + self._turn_first_audio_ms = self._now_ms() + self._fw_log.turn_start(timestamp_ms=self._turn_first_audio_ms) + self._write_model_response_latency(self._turn_first_audio_ms) + + self._assistant_speaking = True + if not self._user_speaking: + sync_buffer_to_position(self.user_audio_buffer, len(self.assistant_audio_buffer)) + self.assistant_audio_buffer.extend(pcm) + self._audio_out.put_nowait(self._from_backend(pcm)) + + def _write_model_response_latency(self, first_audio_ms: int) -> None: + """Time from the simulator's user_speech_stop to our first audio byte. + + The simulator's VAD is the contract-specified source. The backend's own + speech events drive barge-in only; mixing the two skews this metric. + """ + if self._user_speech_stop_ms is None: + return # Model-initiated turn, e.g. the opening greeting. + + latency_ms = first_audio_ms - self._user_speech_stop_ms + if 0 < latency_ms < MAX_PLAUSIBLE_LATENCY_MS: + self._metrics_log.write_latency("model_response", latency_ms / 1000, self.model_name) + else: + logger.warning(f"Discarding implausible model_response latency: {latency_ms} ms") + self._user_speech_stop_ms = None + + def _finish_turn(self, interrupted: bool) -> None: + """Close out the assistant turn. Idempotent — both reply_done and audio_done arrive.""" + if self._turn_first_audio_ms is None and not self._assistant_text: + return + + text = self._assistant_text + if interrupted and text: + text = f"{text} [interrupted]" + + if text: + timestamp = str(self._turn_first_audio_ms or self._now_ms()) + self.audit_log.append_assistant_output(text, timestamp_ms=timestamp) + self._fw_log.llm_response(text) + self._fw_log.s2s_transcript(text) + self._fw_log.turn_end(was_interrupted=interrupted) + + self._assistant_text = "" + self._turn_first_audio_ms = None + self._assistant_speaking = False + + def _handle_barge_in(self) -> None: + """Drop undelivered assistant audio and close the turn as interrupted.""" + if self._turn_first_audio_ms is None and not self._assistant_text: + return # Assistant was not speaking: an ordinary user turn starting. + + dropped = 0 + while not self._audio_out.empty(): + self._audio_out.get_nowait() + dropped += 1 + if dropped: + logger.debug(f"Barge-in: dropped {dropped} queued audio frames") + self._finish_turn(interrupted=True) + + async def _handle_tool_call(self, event: ToolCall) -> None: + """Execute against the scenario database and relay the result back.""" + try: + result = await self.execute_tool(event.name, event.arguments) + except Exception as e: + logger.error(f"Tool {event.name} raised: {e}", exc_info=True) + result = {"status": "error", "message": str(e)} + await self._session.send_tool_result(event.call_id, result) + + async def _pace_audio_output(self, websocket: WebSocket) -> None: + """Drain the output queue at 20 ms per 160-byte frame. + + Backend audio does not arrive in 160-byte multiples once converted, so + frames are re-chunked here. Sending faster or slower than real time makes + the simulator misjudge turn boundaries. + """ + pending = bytearray() + next_send = time.monotonic() + + while self._running: + chunk = await self._audio_out.get() + pending.extend(chunk) + + while len(pending) >= MULAW_FRAME_BYTES: + frame = bytes(pending[:MULAW_FRAME_BYTES]) + del pending[:MULAW_FRAME_BYTES] + try: + await websocket.send_text(create_twilio_media_message(self._stream_sid, frame)) + except Exception: + return + + next_send += FRAME_INTERVAL_S + sleep_s = next_send - time.monotonic() + if sleep_s > 0: + await asyncio.sleep(sleep_s) + else: + # Fell behind (or the queue was idle): resync rather than burst. + next_send = time.monotonic() + + @staticmethod + def _now_ms() -> int: + """Wall-clock milliseconds. Overridden in tests.""" + return int(time.time() * 1000) + + +def _as_int_ms(value: object) -> int | None: + """Coerce a simulator timestamp to int milliseconds, tolerating str or float.""" + if value is None: + return None + try: + return int(float(value)) + except (TypeError, ValueError): + return None diff --git a/tests/unit/assistant/test_ws_bridge_server.py b/tests/unit/assistant/test_ws_bridge_server.py new file mode 100644 index 00000000..6ba82acd --- /dev/null +++ b/tests/unit/assistant/test_ws_bridge_server.py @@ -0,0 +1,331 @@ +"""Tests for WebSocketBridgeAssistantServer turn, latency, and pacing behavior. + +The server is built with ``object.__new__`` and hand-set attributes (the pattern +used by test_openai_realtime_server.py) so tests need no ToolExecutor, no +scenario database, and no live sockets. +""" + +import asyncio +from unittest.mock import MagicMock + +from eva.assistant.bridge_events import ( + AssistantTranscript, + AudioChunk, + BackendError, + SpeechStarted, + ToolCall, + TurnDone, + UserTranscript, +) +from eva.assistant.ws_bridge_server import WebSocketBridgeAssistantServer + + +class _FakeSession: + """In-memory VoiceBackendSession.""" + + backend_input_rate = 16000 + backend_output_rate = 24000 + + def __init__(self, events=()): + self._events = list(events) + self.sent_audio: list[bytes] = [] + self.tool_results: list[tuple[str, object]] = [] + self.closed = False + + async def send_audio(self, pcm: bytes) -> None: + self.sent_audio.append(pcm) + + async def send_tool_result(self, call_id: str, result: object) -> None: + self.tool_results.append((call_id, result)) + + async def events(self): + for event in self._events: + yield event + + async def aclose(self) -> None: + self.closed = True + + +class _Server(WebSocketBridgeAssistantServer): + @property + def model_name(self) -> str: + return "test-model" + + async def _open_backend(self): + return _FakeSession() + + +def _bare_server(session: _FakeSession | None = None) -> _Server: + srv = object.__new__(_Server) + srv._session = session or _FakeSession() + srv._fw_log = MagicMock() + srv._metrics_log = MagicMock() + srv.audit_log = MagicMock() + srv._audio_out = asyncio.Queue() + srv._stream_sid = "conv-1" + srv._running = True + srv._user_speech_start_ms = None + srv._user_speech_stop_ms = None + srv._user_speaking = False + srv._assistant_speaking = False + srv._turn_first_audio_ms = None + srv._assistant_text = "" + srv._tasks = [] + srv.user_audio_buffer = bytearray() + srv.assistant_audio_buffer = bytearray() + srv._audio_sample_rate = 24000 + srv._to_backend = None + srv._from_backend = lambda pcm: b"\xff" * (len(pcm) // 6) # 24k PCM16 -> 8k mulaw + return srv + + +class TestModelResponseLatency: + def test_writes_latency_on_first_audio_chunk_of_turn(self): + srv = _bare_server() + srv._user_speech_stop_ms = 1_000_000 + srv._now_ms = lambda: 1_000_450 + + srv._on_audio_chunk(b"\x00\x00" * 240) + + srv._metrics_log.write_latency.assert_called_once_with("model_response", 0.45, "test-model") + + def test_writes_latency_only_once_per_turn(self): + srv = _bare_server() + srv._user_speech_stop_ms = 1_000_000 + srv._now_ms = lambda: 1_000_450 + + srv._on_audio_chunk(b"\x00\x00" * 240) + srv._on_audio_chunk(b"\x00\x00" * 240) + + assert srv._metrics_log.write_latency.call_count == 1 + + def test_skips_latency_when_no_user_speech_stop(self): + """The opening greeting is model-initiated: there is no user turn to measure from.""" + srv = _bare_server() + srv._now_ms = lambda: 1_000_450 + + srv._on_audio_chunk(b"\x00\x00" * 240) + + srv._metrics_log.write_latency.assert_not_called() + srv._fw_log.turn_start.assert_called_once() + + def test_skips_implausible_latency(self): + srv = _bare_server() + srv._user_speech_stop_ms = 1_000_000 + srv._now_ms = lambda: 1_000_000 + 45_000 # 45s, over the 30s ceiling + + srv._on_audio_chunk(b"\x00\x00" * 240) + + srv._metrics_log.write_latency.assert_not_called() + + def test_skips_negative_latency(self): + srv = _bare_server() + srv._user_speech_stop_ms = 1_000_000 + srv._now_ms = lambda: 999_000 + + srv._on_audio_chunk(b"\x00\x00" * 240) + + srv._metrics_log.write_latency.assert_not_called() + + +class TestAudioBuffers: + def test_assistant_audio_appended_and_queued_for_output(self): + srv = _bare_server() + srv._now_ms = lambda: 1_000 + + srv._on_audio_chunk(b"\x01\x02" * 240) + + assert len(srv.assistant_audio_buffer) == 480 + assert srv._audio_out.qsize() == 1 + + def test_assistant_audio_pads_user_track_to_stay_aligned(self): + srv = _bare_server() + srv._now_ms = lambda: 1_000 + srv.assistant_audio_buffer.extend(b"\x00" * 960) + + srv._on_audio_chunk(b"\x01\x02" * 240) + + # User track padded up to where the assistant track started. + assert len(srv.user_audio_buffer) == 960 + + def test_no_padding_while_user_is_speaking(self): + """During overlap both tracks advance on their own; padding would double-count.""" + srv = _bare_server() + srv._now_ms = lambda: 1_000 + srv._user_speaking = True + srv.assistant_audio_buffer.extend(b"\x00" * 960) + + srv._on_audio_chunk(b"\x01\x02" * 240) + + assert len(srv.user_audio_buffer) == 0 + + +class TestTurnCompletion: + def test_finish_turn_writes_transcript_and_turn_end(self): + srv = _bare_server() + srv._now_ms = lambda: 5_000 + srv._turn_first_audio_ms = 4_000 + srv._assistant_text = "Your flight is confirmed." + + srv._finish_turn(interrupted=False) + + srv.audit_log.append_assistant_output.assert_called_once_with("Your flight is confirmed.", timestamp_ms="4000") + srv._fw_log.llm_response.assert_called_once_with("Your flight is confirmed.") + srv._fw_log.turn_end.assert_called_once_with(was_interrupted=False) + + def test_finish_turn_resets_state_for_next_turn(self): + srv = _bare_server() + srv._now_ms = lambda: 5_000 + srv._turn_first_audio_ms = 4_000 + srv._assistant_text = "text" + + srv._finish_turn(interrupted=False) + + assert srv._assistant_text == "" + assert srv._turn_first_audio_ms is None + assert srv._assistant_speaking is False + + def test_finish_turn_is_idempotent(self): + """reply_done and audio_done both map to TurnDone; the second must be a no-op.""" + srv = _bare_server() + srv._now_ms = lambda: 5_000 + srv._turn_first_audio_ms = 4_000 + srv._assistant_text = "text" + + srv._finish_turn(interrupted=False) + srv._finish_turn(interrupted=False) + + assert srv.audit_log.append_assistant_output.call_count == 1 + assert srv._fw_log.turn_end.call_count == 1 + + +class TestBargeIn: + def test_barge_in_drains_queued_audio_and_marks_turn_interrupted(self): + srv = _bare_server() + srv._now_ms = lambda: 5_000 + srv._turn_first_audio_ms = 4_000 + srv._assistant_text = "As I was saying" + srv._audio_out.put_nowait(b"\xff" * 160) + srv._audio_out.put_nowait(b"\xff" * 160) + + srv._handle_barge_in() + + assert srv._audio_out.qsize() == 0 + srv._fw_log.turn_end.assert_called_once_with(was_interrupted=True) + srv.audit_log.append_assistant_output.assert_called_once_with( + "As I was saying [interrupted]", timestamp_ms="4000" + ) + + def test_speech_started_while_assistant_silent_is_not_barge_in(self): + srv = _bare_server() + srv._now_ms = lambda: 5_000 + + srv._handle_barge_in() + + srv._fw_log.turn_end.assert_not_called() + + +class TestToolCalls: + async def test_tool_result_relayed_to_backend(self): + session = _FakeSession() + srv = _bare_server(session) + srv.execute_tool = MagicMock(return_value=_async_value({"status": "ok", "seat": "12A"})) + + await srv._handle_tool_call(ToolCall(call_id="c1", name="assign_seat", arguments={"seat": "12A"})) + + srv.execute_tool.assert_called_once_with("assign_seat", {"seat": "12A"}) + assert session.tool_results == [("c1", {"status": "ok", "seat": "12A"})] + + async def test_tool_failure_relays_error_result_so_the_turn_continues(self): + session = _FakeSession() + srv = _bare_server(session) + srv.execute_tool = MagicMock(side_effect=RuntimeError("db offline")) + + await srv._handle_tool_call(ToolCall(call_id="c2", name="broken", arguments={})) + + assert len(session.tool_results) == 1 + call_id, result = session.tool_results[0] + assert call_id == "c2" + assert result["status"] == "error" + assert "db offline" in result["message"] + + +class TestEventDispatch: + async def test_dispatches_each_event_kind(self): + session = _FakeSession( + [ + UserTranscript(text="I need to rebook"), + AudioChunk(pcm=b"\x01\x02" * 240), + AssistantTranscript(text="Sure, let me look."), + TurnDone(), + ] + ) + srv = _bare_server(session) + srv._now_ms = lambda: 7_000 + srv._user_speech_start_ms = 6_000 + + await srv._process_backend_events() + + srv.audit_log.append_user_input.assert_called_once_with("I need to rebook", timestamp_ms="6000") + srv.audit_log.append_assistant_output.assert_called_once_with("Sure, let me look.", timestamp_ms="7000") + + async def test_assistant_transcript_overwrites_rather_than_appends(self): + """The aai backend sends full text per frame, not deltas.""" + session = _FakeSession([AssistantTranscript(text="Hello"), AssistantTranscript(text="Hello there")]) + srv = _bare_server(session) + srv._now_ms = lambda: 7_000 + + await srv._process_backend_events() + + assert srv._assistant_text == "Hello there" + + async def test_speech_started_event_triggers_barge_in(self): + session = _FakeSession([SpeechStarted()]) + srv = _bare_server(session) + srv._now_ms = lambda: 7_000 + srv._turn_first_audio_ms = 6_000 + srv._assistant_text = "mid sentence" + + await srv._process_backend_events() + + srv._fw_log.turn_end.assert_called_once_with(was_interrupted=True) + + async def test_fatal_backend_error_stops_dispatch(self): + session = _FakeSession([BackendError(message="idle_timeout", fatal=True), AssistantTranscript(text="never")]) + srv = _bare_server(session) + srv._now_ms = lambda: 7_000 + + await srv._process_backend_events() + + assert srv._assistant_text == "" + + +class TestPacing: + async def test_rechunks_output_to_160_byte_frames(self): + """Backend audio does not arrive in 160-byte multiples; the simulator requires 20ms frames.""" + srv = _bare_server() + sent: list[str] = [] + + class _WS: + async def send_text(self, msg: str) -> None: + sent.append(msg) + + srv._audio_out.put_nowait(b"\xff" * 400) + task = asyncio.create_task(srv._pace_audio_output(_WS())) + await asyncio.sleep(0.08) + srv._running = False + task.cancel() + try: + await task + except asyncio.CancelledError: + pass + + # 400 bytes -> two full 160-byte frames, 80 bytes held back for the next chunk. + assert len(sent) == 2 + + +def _async_value(value): + async def _coro(): + return value + + return _coro() From 458090475d12689356fadc63ced3809ae88b8974 Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Wed, 29 Jul 2026 00:16:59 -0400 Subject: [PATCH 05/11] feat(aai): host-mode session client with tool relay Performs the ?host=1 handshake (system prompt, greeting, and tool schemas injected per session), streams PCM16 audio as binary frames, and relays tool_call -> tool_result so tools execute in EVA rather than aai's sandbox. Co-Authored-By: Claude Opus 5 (1M context) --- src/eva/assistant/aai_session.py | 235 +++++++++++++++++++++++ tests/unit/assistant/test_aai_session.py | 220 +++++++++++++++++++++ 2 files changed, 455 insertions(+) create mode 100644 src/eva/assistant/aai_session.py create mode 100644 tests/unit/assistant/test_aai_session.py diff --git a/src/eva/assistant/aai_session.py b/src/eva/assistant/aai_session.py new file mode 100644 index 00000000..14d25db5 --- /dev/null +++ b/src/eva/assistant/aai_session.py @@ -0,0 +1,235 @@ +"""aai host-mode session: the wire protocol behind AAIAssistantServer. + +aai agents normally execute their tools server-side in a sandbox. Host mode +inverts that: this client supplies the system prompt, greeting, and tool +*schemas* in its first ``config`` frame, and aai relays every tool *call* back +as a ``tool_call`` frame that resolves when the matching ``tool_result`` +arrives. That inversion is what lets EVA's ToolExecutor mutate the scenario +database — without it the accuracy metrics would score an unchanged database. + +Host mode is selected by the ``?host=1`` query parameter and gated server-side +by ``AAI_ALLOW_HOST``. +""" + +import asyncio +import contextlib +import json +from collections.abc import AsyncIterator +from typing import Any +from urllib.parse import parse_qs, urlencode, urlparse, urlunparse + +import websockets + +from eva.assistant.aai_events import ( + AAIAgentTranscriptEvent, + AAIAudioDoneEvent, + AAIConfigEvent, + AAIErrorEvent, + AAIIdleTimeoutEvent, + AAIReplyDoneEvent, + AAISpeechStartedEvent, + AAIToolCallEvent, + AAIUserTranscriptEvent, + BaseAAIEvent, + parse_aai_event, +) +from eva.assistant.bridge_events import ( + AssistantTranscript, + AudioChunk, + BackendError, + BridgeEvent, + SpeechStarted, + ToolCall, + TurnDone, + UserTranscript, +) +from eva.utils.logging import get_logger + +logger = get_logger(__name__) + +DEFAULT_AAI_WS_URL = "ws://localhost:3000/websocket" +DEFAULT_AAI_INPUT_SAMPLE_RATE = 16000 +DEFAULT_AAI_OUTPUT_SAMPLE_RATE = 24000 + +#: aai host mode is endpoint-determined: the model is whatever the host runs. +DEFAULT_AAI_MODEL = "aai-host" + +CONFIG_ACK_TIMEOUT_S = 15.0 + + +class AAIHostSessionError(Exception): + """The aai host refused or failed the host-mode handshake.""" + + +def with_host_flag(url: str) -> str: + """Return *url* with ``host=1``, activating host mode on the aai server.""" + parsed = urlparse(url) + params = parse_qs(parsed.query) + params["host"] = ["1"] + return urlunparse(parsed._replace(query=urlencode(params, doseq=True))) + + +def build_host_config_message( + *, + system_prompt: str, + tools: list[dict], + greeting: str | None, + input_rate: int, + output_rate: int, +) -> dict: + """Build the host-mode handshake frame. + + Mirrors ``HostConfigMessageSchema`` in the aai SDK: the audio negotiation + fields ride alongside the ``host`` block in a single ``config`` frame. + """ + host: dict[str, Any] = {"systemPrompt": system_prompt, "tools": list(tools)} + if greeting: + host["greeting"] = greeting + return { + "type": "config", + "audioFormat": "pcm16", + "sampleRate": input_rate, + "ttsSampleRate": output_rate, + "host": host, + } + + +def map_aai_event(event: BaseAAIEvent) -> BridgeEvent | None: + """Translate an aai wire event into a bridge event, or None to ignore it. + + Both ``reply_done`` and ``audio_done`` map to ``TurnDone``: audio completion + is the truer end-of-turn signal for a voice call, but a text-only reply + produces no ``audio_done``, so honoring both avoids a stuck turn. The + bridge's turn completion is idempotent, making the duplicate harmless. + """ + if isinstance(event, AAIAgentTranscriptEvent): + return AssistantTranscript(text=event.text) + if isinstance(event, AAIUserTranscriptEvent): + return UserTranscript(text=event.text) + if isinstance(event, AAIToolCallEvent): + return ToolCall(call_id=event.tool_call_id, name=event.tool_name, arguments=event.args) + if isinstance(event, AAISpeechStartedEvent): + return SpeechStarted() + if isinstance(event, AAIReplyDoneEvent | AAIAudioDoneEvent): + return TurnDone() + if isinstance(event, AAIIdleTimeoutEvent): + return BackendError(message="aai session idle timeout", fatal=True) + if isinstance(event, AAIErrorEvent): + return BackendError(message=f"{event.code or 'error'}: {event.message or ''}".strip()) + return None + + +class AAIHostSession: + """One host-mode conversation with an aai voice agent. + + Satisfies ``VoiceBackendSession``. Audio is raw binary PCM16 frames; + everything else is JSON. + """ + + def __init__(self, ws, input_rate: int, output_rate: int): + self._ws = ws + self.backend_input_rate = input_rate + self.backend_output_rate = output_rate + + @classmethod + async def connect( + cls, + *, + ws_url: str, + system_prompt: str, + tools: list[dict], + greeting: str | None = None, + input_rate: int = DEFAULT_AAI_INPUT_SAMPLE_RATE, + output_rate: int = DEFAULT_AAI_OUTPUT_SAMPLE_RATE, + ) -> "AAIHostSession": + """Open a host-mode session and complete the handshake. + + Raises: + AAIHostSessionError: the host rejected the handshake, or did not + acknowledge it within ``CONFIG_ACK_TIMEOUT_S``. + """ + url = with_host_flag(ws_url) + logger.info(f"Connecting to aai host at {url} ({len(tools)} tools)") + try: + # max_size=None: TTS audio frames can exceed the 1 MiB default. + ws = await websockets.connect(url, max_size=None) + except Exception as e: + raise AAIHostSessionError(f"Could not connect to aai host at {url}: {e}") from e + + try: + await ws.send( + json.dumps( + build_host_config_message( + system_prompt=system_prompt, + tools=tools, + greeting=greeting, + input_rate=input_rate, + output_rate=output_rate, + ) + ) + ) + await cls._await_config_ack(ws, timeout_s=CONFIG_ACK_TIMEOUT_S) + except Exception: + with contextlib.suppress(Exception): + await ws.close() + raise + + logger.info("aai host-mode handshake complete") + return cls(ws, input_rate=input_rate, output_rate=output_rate) + + @staticmethod + async def _await_config_ack(ws, timeout_s: float) -> None: + """Block until the host acknowledges the config frame.""" + while True: + try: + raw = await asyncio.wait_for(ws.recv(), timeout=timeout_s) + except TimeoutError as e: + raise AAIHostSessionError( + f"aai host did not acknowledge the host-mode config within {timeout_s}s (is AAI_ALLOW_HOST set?)" + ) from e + + if isinstance(raw, bytes | bytearray): + continue # Audio can precede the ack; ignore it. + + try: + event = parse_aai_event(json.loads(raw)) + except json.JSONDecodeError: + continue + + if isinstance(event, AAIConfigEvent): + return + if isinstance(event, AAIErrorEvent): + raise AAIHostSessionError(f"aai host rejected host mode: {event.code}: {event.message}") + + async def send_audio(self, pcm: bytes) -> None: + await self._ws.send(pcm) + + async def send_tool_result(self, call_id: str, result: Any) -> None: + """Relay a tool result. ``result`` travels as a JSON string, which aai unwraps.""" + await self._ws.send( + json.dumps( + { + "type": "tool_result", + "toolCallId": call_id, + "result": json.dumps(result, default=str), + } + ) + ) + + async def events(self) -> AsyncIterator[BridgeEvent]: + async for raw in self._ws: + if isinstance(raw, bytes | bytearray): + yield AudioChunk(pcm=bytes(raw)) + continue + try: + data = json.loads(raw) + except json.JSONDecodeError: + logger.warning("Discarding non-JSON text frame from aai host") + continue + mapped = map_aai_event(parse_aai_event(data)) + if mapped is not None: + yield mapped + + async def aclose(self) -> None: + with contextlib.suppress(Exception): + await self._ws.close() diff --git a/tests/unit/assistant/test_aai_session.py b/tests/unit/assistant/test_aai_session.py new file mode 100644 index 00000000..49562eeb --- /dev/null +++ b/tests/unit/assistant/test_aai_session.py @@ -0,0 +1,220 @@ +"""Tests for the aai host-mode session: framing, handshake, and event mapping.""" + +import json + +import pytest + +from eva.assistant.aai_events import ( + AAIAgentTranscriptEvent, + AAIAudioDoneEvent, + AAICustomEvent, + AAIErrorEvent, + AAIIdleTimeoutEvent, + AAIReplyDoneEvent, + AAISpeechStartedEvent, + AAISpeechStoppedEvent, + AAIToolCallEvent, + AAIUserTranscriptEvent, +) +from eva.assistant.aai_session import ( + AAIHostSession, + AAIHostSessionError, + build_host_config_message, + map_aai_event, + with_host_flag, +) +from eva.assistant.bridge_events import ( + AssistantTranscript, + BackendError, + SpeechStarted, + ToolCall, + TurnDone, + UserTranscript, +) + + +class TestWithHostFlag: + def test_appends_host_flag(self): + assert with_host_flag("ws://localhost:3000/websocket") == "ws://localhost:3000/websocket?host=1" + + def test_merges_with_existing_query_params(self): + result = with_host_flag("ws://localhost:3000/websocket?sessionId=abc") + assert "sessionId=abc" in result + assert "host=1" in result + + def test_overwrites_existing_host_param(self): + assert with_host_flag("ws://h/websocket?host=0").endswith("host=1") + + +class TestBuildHostConfigMessage: + def test_builds_handshake_frame(self): + msg = build_host_config_message( + system_prompt="You are an airline agent.", + tools=[{"type": "function", "name": "t", "description": "d", "parameters": {"type": "object"}}], + greeting="Thank you for calling.", + input_rate=16000, + output_rate=24000, + ) + assert msg["type"] == "config" + assert msg["audioFormat"] == "pcm16" + assert msg["sampleRate"] == 16000 + assert msg["ttsSampleRate"] == 24000 + assert msg["host"]["systemPrompt"] == "You are an airline agent." + assert msg["host"]["greeting"] == "Thank you for calling." + assert len(msg["host"]["tools"]) == 1 + + def test_omits_greeting_when_empty(self): + msg = build_host_config_message(system_prompt="p", tools=[], greeting=None, input_rate=16000, output_rate=24000) + assert "greeting" not in msg["host"] + + def test_is_json_serializable(self): + msg = build_host_config_message(system_prompt="p", tools=[], greeting="g", input_rate=16000, output_rate=24000) + assert json.loads(json.dumps(msg)) == msg + + +class TestMapAaiEvent: + def test_maps_agent_transcript(self): + assert map_aai_event(AAIAgentTranscriptEvent(text="hi")) == AssistantTranscript(text="hi") + + def test_maps_user_transcript(self): + assert map_aai_event(AAIUserTranscriptEvent(text="hello")) == UserTranscript(text="hello") + + def test_maps_tool_call(self): + event = AAIToolCallEvent(toolCallId="c1", toolName="get_reservation", args={"x": 1}) + assert map_aai_event(event) == ToolCall(call_id="c1", name="get_reservation", arguments={"x": 1}) + + def test_maps_speech_started_to_barge_in_signal(self): + assert map_aai_event(AAISpeechStartedEvent()) == SpeechStarted() + + def test_maps_both_turn_end_signals(self): + assert map_aai_event(AAIReplyDoneEvent()) == TurnDone() + assert map_aai_event(AAIAudioDoneEvent()) == TurnDone() + + def test_maps_error_with_code_and_message(self): + result = map_aai_event(AAIErrorEvent(code="bad_config", message="nope")) + assert isinstance(result, BackendError) + assert "bad_config" in result.message + assert "nope" in result.message + assert result.fatal is False + + def test_idle_timeout_is_fatal(self): + result = map_aai_event(AAIIdleTimeoutEvent()) + assert isinstance(result, BackendError) + assert result.fatal is True + + def test_unmapped_events_return_none(self): + assert map_aai_event(AAISpeechStoppedEvent()) is None + assert map_aai_event(AAICustomEvent(event="metric", data={"a": 1})) is None + + +class _FakeWebSocket: + """Stands in for a websockets client connection.""" + + def __init__(self, inbound=()): + self.sent: list[object] = [] + self._inbound = list(inbound) + self.closed = False + + async def send(self, data) -> None: + self.sent.append(data) + + async def recv(self): + if not self._inbound: + raise AssertionError("recv() called with no inbound frames left") + return self._inbound.pop(0) + + def __aiter__(self): + return self + + async def __anext__(self): + if not self._inbound: + raise StopAsyncIteration + return self._inbound.pop(0) + + async def close(self) -> None: + self.closed = True + + +class TestSessionFraming: + async def test_send_audio_sends_raw_binary(self): + ws = _FakeWebSocket() + session = AAIHostSession(ws, input_rate=16000, output_rate=24000) + + await session.send_audio(b"\x01\x02\x03\x04") + + assert ws.sent == [b"\x01\x02\x03\x04"] + + async def test_send_tool_result_json_encodes_the_result(self): + ws = _FakeWebSocket() + session = AAIHostSession(ws, input_rate=16000, output_rate=24000) + + await session.send_tool_result("call_1", {"status": "ok", "seat": "12A"}) + + frame = json.loads(ws.sent[0]) + assert frame["type"] == "tool_result" + assert frame["toolCallId"] == "call_1" + # aai unwraps a JSON string on the wire, so result must be a string. + assert json.loads(frame["result"]) == {"status": "ok", "seat": "12A"} + + async def test_send_tool_result_survives_non_serializable_values(self): + ws = _FakeWebSocket() + session = AAIHostSession(ws, input_rate=16000, output_rate=24000) + + await session.send_tool_result("call_2", {"when": object()}) + + frame = json.loads(ws.sent[0]) + assert frame["toolCallId"] == "call_2" + + async def test_events_yields_audio_for_binary_frames(self): + ws = _FakeWebSocket([b"\x01\x02", json.dumps({"type": "reply_done"})]) + session = AAIHostSession(ws, input_rate=16000, output_rate=24000) + + events = [event async for event in session.events()] + + assert events[0].pcm == b"\x01\x02" + assert events[1] == TurnDone() + + async def test_events_skips_unmapped_and_malformed_frames(self): + ws = _FakeWebSocket( + [ + "not json at all", + json.dumps({"type": "speech_stopped"}), + json.dumps({"type": "agent_transcript", "text": "ok"}), + ] + ) + session = AAIHostSession(ws, input_rate=16000, output_rate=24000) + + events = [event async for event in session.events()] + + assert events == [AssistantTranscript(text="ok")] + + async def test_aclose_closes_the_socket(self): + ws = _FakeWebSocket() + session = AAIHostSession(ws, input_rate=16000, output_rate=24000) + + await session.aclose() + + assert ws.closed is True + + def test_declares_backend_rates(self): + session = AAIHostSession(_FakeWebSocket(), input_rate=16000, output_rate=24000) + assert session.backend_input_rate == 16000 + assert session.backend_output_rate == 24000 + + +class TestHandshake: + async def test_await_config_ack_accepts_config_event(self): + ws = _FakeWebSocket([json.dumps({"type": "config"})]) + await AAIHostSession._await_config_ack(ws, timeout_s=1.0) + + async def test_await_config_ack_skips_binary_frames(self): + ws = _FakeWebSocket([b"\x00\x00", json.dumps({"type": "config"})]) + await AAIHostSession._await_config_ack(ws, timeout_s=1.0) + + async def test_await_config_ack_raises_on_error_event(self): + ws = _FakeWebSocket([json.dumps({"type": "error", "code": "host_disabled", "message": "AAI_ALLOW_HOST"})]) + + with pytest.raises(AAIHostSessionError) as excinfo: + await AAIHostSession._await_config_ack(ws, timeout_s=1.0) + + assert "AAI_ALLOW_HOST" in str(excinfo.value) From cad28df72f5f2a64075e39615396c1fc29b2d18e Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Wed, 29 Jul 2026 00:18:32 -0400 Subject: [PATCH 06/11] feat(aai): assistant server and framework registration AAIAssistantServer builds the per-session agent definition -- system prompt via the base class, greeting from the run's initial message, and tool schemas in aai's flat ToolSchema shape -- then opens a host session. Selected with framework: aai. Co-Authored-By: Claude Opus 5 (1M context) --- src/eva/assistant/aai_server.py | 74 ++++++++++++++++ src/eva/models/config.py | 3 +- src/eva/orchestrator/worker.py | 6 +- tests/unit/assistant/test_aai_server.py | 107 ++++++++++++++++++++++++ 4 files changed, 188 insertions(+), 2 deletions(-) create mode 100644 src/eva/assistant/aai_server.py create mode 100644 tests/unit/assistant/test_aai_server.py diff --git a/src/eva/assistant/aai_server.py b/src/eva/assistant/aai_server.py new file mode 100644 index 00000000..71566095 --- /dev/null +++ b/src/eva/assistant/aai_server.py @@ -0,0 +1,74 @@ +"""EVA assistant server for the aai voice-agent framework. + +Bridges EVA's user simulator to an aai host-mode session. All the contract +plumbing lives in ``WebSocketBridgeAssistantServer``; this class only builds the +per-session agent definition — system prompt, greeting, and tool schemas — and +opens the backend. + +Requires a running aai host with ``AAI_ALLOW_HOST`` enabled. See +docs/aai_integration.md. +""" + +import os + +from eva.assistant.aai_session import ( + DEFAULT_AAI_INPUT_SAMPLE_RATE, + DEFAULT_AAI_MODEL, + DEFAULT_AAI_OUTPUT_SAMPLE_RATE, + DEFAULT_AAI_WS_URL, + AAIHostSession, +) +from eva.assistant.bridge_events import VoiceBackendSession +from eva.assistant.ws_bridge_server import WebSocketBridgeAssistantServer +from eva.utils.logging import get_logger + +logger = get_logger(__name__) + + +class AAIAssistantServer(WebSocketBridgeAssistantServer): + """Runs an EVA conversation against an aai host-mode agent.""" + + def __init__(self, **kwargs): + super().__init__(**kwargs) + s2s_params = self.pipeline_config.s2s_params or {} + self._model: str = s2s_params.get("model") or DEFAULT_AAI_MODEL + self._ws_url: str = s2s_params.get("ws_url") or os.environ.get("AAI_WS_URL") or DEFAULT_AAI_WS_URL + self._input_rate: int = int(s2s_params.get("input_sample_rate") or DEFAULT_AAI_INPUT_SAMPLE_RATE) + self._output_rate: int = int(s2s_params.get("output_sample_rate") or DEFAULT_AAI_OUTPUT_SAMPLE_RATE) + + @property + def model_name(self) -> str: + return self._model + + def _build_aai_tools(self) -> list[dict]: + """Convert the agent's tools to aai's flat ToolSchema shape. + + aai validates ``{type, name, description, parameters}`` with a non-empty + description and a JSON Schema object for parameters. + """ + tools: list[dict] = [] + for tool in self.agent.tools or []: + description = f"{tool.name}: {tool.description}".strip(": ").strip() or tool.function_name + tools.append( + { + "type": "function", + "name": tool.function_name, + "description": description, + "parameters": { + "type": "object", + "properties": tool.get_parameter_properties(), + "required": tool.get_required_param_names(), + }, + } + ) + return tools + + async def _open_backend(self) -> VoiceBackendSession: + return await AAIHostSession.connect( + ws_url=self._ws_url, + system_prompt=self._build_system_prompt(), + tools=self._build_aai_tools(), + greeting=self.initial_message, + input_rate=self._input_rate, + output_rate=self._output_rate, + ) diff --git a/src/eva/models/config.py b/src/eva/models/config.py index c2c0b6fc..c51e5195 100644 --- a/src/eva/models/config.py +++ b/src/eva/models/config.py @@ -503,7 +503,7 @@ class ModelDeployment(DeploymentTypedDict): ) # Framework selection - framework: Literal["pipecat", "openai_realtime", "gemini_live", "elevenlabs", "grok_voice"] = Field( + framework: Literal["pipecat", "openai_realtime", "gemini_live", "elevenlabs", "grok_voice", "aai"] = Field( "pipecat", description=( "Agent framework to use for the assistant server." @@ -512,6 +512,7 @@ class ModelDeployment(DeploymentTypedDict): "'gemini_live': Gemini Live API via google-genai." "'elevenlabs': ElevenLabs Conversational AI API." "'grok_voice': xAI Grok voice realtime API." + "'aai': aai voice-agent framework in host mode." ), ) diff --git a/src/eva/orchestrator/worker.py b/src/eva/orchestrator/worker.py index ef5519c4..2bda505e 100644 --- a/src/eva/orchestrator/worker.py +++ b/src/eva/orchestrator/worker.py @@ -49,10 +49,14 @@ def _get_server_class(framework: str) -> type[AbstractAssistantServer]: from eva.assistant.grok_voice_server import GrokVoiceAssistantServer return GrokVoiceAssistantServer + elif framework == "aai": + from eva.assistant.aai_server import AAIAssistantServer + + return AAIAssistantServer else: raise ValueError( f"Unknown framework: {framework!r}. " - "Supported: pipecat, openai_realtime, gemini_live, elevenlabs, grok_voice" + "Supported: pipecat, openai_realtime, gemini_live, elevenlabs, grok_voice, aai" ) diff --git a/tests/unit/assistant/test_aai_server.py b/tests/unit/assistant/test_aai_server.py new file mode 100644 index 00000000..6c0cc744 --- /dev/null +++ b/tests/unit/assistant/test_aai_server.py @@ -0,0 +1,107 @@ +"""Tests for AAIAssistantServer configuration, tool schemas, and registration.""" + +from unittest.mock import MagicMock + +from eva.assistant.aai_server import AAIAssistantServer +from eva.assistant.aai_session import DEFAULT_AAI_MODEL, DEFAULT_AAI_WS_URL + + +def _bare_server(s2s_params: dict | None = None) -> AAIAssistantServer: + """Construct without __init__ (skips PromptManager and ToolExecutor setup).""" + srv = object.__new__(AAIAssistantServer) + srv.pipeline_config = MagicMock() + srv.pipeline_config.s2s_params = s2s_params if s2s_params is not None else {} + srv.agent = MagicMock() + srv.agent.tools = [] + srv._model = (s2s_params or {}).get("model", DEFAULT_AAI_MODEL) + srv._ws_url = (s2s_params or {}).get("ws_url", DEFAULT_AAI_WS_URL) + return srv + + +def _tool(function_name: str, name: str, description: str, properties: dict, required: list[str]) -> MagicMock: + tool = MagicMock() + tool.function_name = function_name + tool.name = name + tool.description = description + tool.get_parameter_properties.return_value = properties + tool.get_required_param_names.return_value = required + return tool + + +class TestModelName: + def test_defaults_to_aai_host(self): + assert _bare_server().model_name == DEFAULT_AAI_MODEL + + def test_honors_configured_model(self): + assert _bare_server({"model": "aai-host-custom"}).model_name == "aai-host-custom" + + +class TestBuildAaiTools: + def test_returns_empty_list_when_agent_has_no_tools(self): + assert _bare_server()._build_aai_tools() == [] + + def test_builds_flat_function_schema(self): + srv = _bare_server() + srv.agent.tools = [ + _tool( + "get_reservation", + "Get Reservation", + "Look up a booking", + {"confirmation_number": {"type": "string"}}, + ["confirmation_number"], + ) + ] + + tools = srv._build_aai_tools() + + assert tools == [ + { + "type": "function", + "name": "get_reservation", + "description": "Get Reservation: Look up a booking", + "parameters": { + "type": "object", + "properties": {"confirmation_number": {"type": "string"}}, + "required": ["confirmation_number"], + }, + } + ] + + def test_description_is_never_empty(self): + """The aai ToolSchema requires a description of at least one character.""" + srv = _bare_server() + srv.agent.tools = [_tool("f", "", "", {}, [])] + + description = srv._build_aai_tools()[0]["description"] + + assert len(description) >= 1 + + +class TestWebSocketUrl: + def test_defaults_to_localhost(self): + assert _bare_server()._ws_url == DEFAULT_AAI_WS_URL + + def test_honors_configured_url(self): + assert _bare_server({"ws_url": "ws://aai.internal/websocket"})._ws_url == "ws://aai.internal/websocket" + + +class TestRegistration: + def test_framework_literal_accepts_aai(self): + from eva.models.config import RunConfig + + assert "aai" in RunConfig.model_fields["framework"].annotation.__args__ + + def test_worker_resolves_the_aai_server_class(self): + from eva.orchestrator.worker import _get_server_class + + assert _get_server_class("aai") is AAIAssistantServer + + def test_unknown_framework_error_lists_aai(self): + from eva.orchestrator.worker import _get_server_class + + try: + _get_server_class("nope") + except ValueError as e: + assert "aai" in str(e) + else: + raise AssertionError("expected ValueError") From 36429ecfd867e2e8cee86b8514f7522524ed5bce Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Wed, 29 Jul 2026 00:19:41 -0400 Subject: [PATCH 07/11] test(aai): full-session bridge test against a fake aai host Drives both WebSocket hops in-process: a fake simulator sends Twilio-framed mu-law audio, the bridge speaks host mode to a fake aai server that requests a tool and replies with audio. Verifies the injected prompt and greeting, the tool round-trip through EVA, and 160-byte mu-law output framing -- with no paid APIs and no Node process. Co-Authored-By: Claude Opus 5 (1M context) --- tests/integration/test_aai_fake_backend.py | 203 +++++++++++++++++++++ 1 file changed, 203 insertions(+) create mode 100644 tests/integration/test_aai_fake_backend.py diff --git a/tests/integration/test_aai_fake_backend.py b/tests/integration/test_aai_fake_backend.py new file mode 100644 index 00000000..0d109198 --- /dev/null +++ b/tests/integration/test_aai_fake_backend.py @@ -0,0 +1,203 @@ +"""End-to-end bridge test against an in-process fake aai host. + +Exercises both WebSocket hops without any paid API or Node process: +a fake user simulator sends Twilio-framed mu-law audio to AAIAssistantServer, +which speaks host mode to a fake aai server that requests a tool and replies +with audio. +""" + +import asyncio +import audioop +import base64 +import json +import time +from unittest.mock import MagicMock + +import pytest +import websockets + +from eva.assistant.aai_server import AAIAssistantServer +from eva.assistant.audio_bridge import create_twilio_media_message + +FAKE_AAI_PORT = 38999 +BRIDGE_PORT = 38998 + + +class FakeAaiHost: + """Minimal aai host: completes the handshake, calls one tool, speaks, ends the turn.""" + + def __init__(self): + self.handshake: dict | None = None + self.tool_results: list[dict] = [] + self.received_audio_bytes = 0 + self._server = None + + async def start(self) -> None: + self._server = await websockets.serve(self._handle, "localhost", FAKE_AAI_PORT) + + async def stop(self) -> None: + if self._server is not None: + self._server.close() + await self._server.wait_closed() + + async def _handle(self, ws) -> None: + # 1. Handshake: first text frame carries the host config. + raw = await ws.recv() + self.handshake = json.loads(raw) + await ws.send(json.dumps({"type": "config"})) + + # 2. Ask the client to run a tool. + await ws.send( + json.dumps( + { + "type": "tool_call", + "toolCallId": "call_1", + "toolName": "probe_tool", + "args": {"query": "status"}, + } + ) + ) + + # 3. Wait for the relayed result, tolerating interleaved user audio. + while True: + frame = await ws.recv() + if isinstance(frame, bytes | bytearray): + self.received_audio_bytes += len(frame) + continue + message = json.loads(frame) + if message.get("type") == "tool_result": + self.tool_results.append(message) + break + + # 4. Speak: transcript, 100 ms of 24 kHz PCM16, then end the turn. + await ws.send(json.dumps({"type": "agent_transcript", "text": "Your status is green."})) + await ws.send(b"\x00\x01" * 2400) + await ws.send(json.dumps({"type": "reply_done"})) + await ws.send(json.dumps({"type": "audio_done"})) + + # Keep the socket open so the bridge can finish draining. + await asyncio.sleep(1.0) + + +def _make_server() -> AAIAssistantServer: + """Build a server with the heavy collaborators stubbed.""" + srv = object.__new__(AAIAssistantServer) + + srv.pipeline_config = MagicMock() + srv.pipeline_config.s2s_params = { + "model": "aai-host", + "ws_url": f"ws://localhost:{FAKE_AAI_PORT}/websocket", + } + srv.agent = MagicMock() + srv.agent.tools = [] + srv.conversation_id = "conv-test" + srv.port = BRIDGE_PORT + srv.initial_message = "Thank you for calling." + srv.audit_log = MagicMock() + srv.tool_handler = MagicMock() + + async def _fake_execute(name, args): + return {"status": "ok", "echo": args} + + srv.execute_tool = _fake_execute + srv._build_system_prompt = lambda: "You are a test agent." + srv.get_initial_scenario_db = lambda: {} + srv.get_final_scenario_db = lambda: {} + + # Attributes normally set by the two __init__ methods. + srv._model = "aai-host" + srv._ws_url = f"ws://localhost:{FAKE_AAI_PORT}/websocket" + srv._input_rate = 16000 + srv._output_rate = 24000 + srv._session = None + srv._to_backend = None + srv._from_backend = None + srv._stream_sid = "conv-test" + srv._audio_out = asyncio.Queue() + srv._tasks = [] + srv._user_speech_start_ms = None + srv._user_speech_stop_ms = None + srv._user_speaking = False + srv._assistant_speaking = False + srv._turn_first_audio_ms = None + srv._assistant_text = "" + srv.user_audio_buffer = bytearray() + srv.assistant_audio_buffer = bytearray() + srv._audio_buffer = bytearray() + srv._audio_sample_rate = 24000 + srv._app = None + srv._server = None + srv._server_task = None + srv._running = False + return srv + + +@pytest.fixture +async def fake_host(): + host = FakeAaiHost() + await host.start() + yield host + await host.stop() + + +async def test_full_session_round_trip(fake_host, tmp_path): + srv = _make_server() + srv.output_dir = tmp_path + await srv.start() + + received_audio = bytearray() + try: + async with websockets.connect(f"ws://localhost:{BRIDGE_PORT}/ws") as sim: + await sim.send(json.dumps({"event": "start", "start": {"streamSid": "stream-1"}})) + + # 200 ms of silence as mu-law 8 kHz, in 20 ms Twilio frames. + silence_mulaw = audioop.lin2ulaw(b"\x00\x00" * 1600, 2) + for offset in range(0, len(silence_mulaw), 160): + await sim.send(create_twilio_media_message("stream-1", silence_mulaw[offset : offset + 160])) + await asyncio.sleep(0.005) + + await sim.send(json.dumps({"event": "user_speech_stop", "timestamp_ms": str(_now_ms())})) + + # Collect whatever the bridge paces back. + deadline = asyncio.get_running_loop().time() + 3.0 + while asyncio.get_running_loop().time() < deadline: + try: + raw = await asyncio.wait_for(sim.recv(), timeout=0.3) + except TimeoutError: + if received_audio: + break + continue + message = json.loads(raw) + if message.get("event") == "media": + received_audio.extend(base64.b64decode(message["media"]["payload"])) + finally: + await srv.stop() + + # Handshake carried the injected agent definition. + assert fake_host.handshake is not None + assert fake_host.handshake["type"] == "config" + assert fake_host.handshake["audioFormat"] == "pcm16" + assert fake_host.handshake["sampleRate"] == 16000 + assert fake_host.handshake["ttsSampleRate"] == 24000 + assert fake_host.handshake["host"]["systemPrompt"] == "You are a test agent." + assert fake_host.handshake["host"]["greeting"] == "Thank you for calling." + + # User audio reached the backend. + assert fake_host.received_audio_bytes > 0 + + # The tool round-tripped through EVA, not the backend's sandbox. + assert len(fake_host.tool_results) == 1 + assert fake_host.tool_results[0]["toolCallId"] == "call_1" + assert json.loads(fake_host.tool_results[0]["result"])["status"] == "ok" + + # Assistant audio came back as 160-byte mu-law frames. + assert len(received_audio) > 0 + assert len(received_audio) % 160 == 0 + + # The turn was recorded. + srv.audit_log.append_assistant_output.assert_called() + assert srv.audit_log.append_assistant_output.call_args[0][0] == "Your status is green." + + +def _now_ms() -> int: + return int(time.time() * 1000) From 032a105c66ac502ab77ad9e0e23fb3ce99d9a1c0 Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Wed, 29 Jul 2026 00:21:46 -0400 Subject: [PATCH 08/11] docs(aai): integration setup, configuration, and known gaps Adds docs/aai_integration.md (prerequisites, config table, run command, architecture, troubleshooting), indexes it in docs/README.md, and registers AAI_WS_URL plus the aai framework option in .env.example so the config editor offers them. Co-Authored-By: Claude Opus 5 (1M context) --- .env.example | 8 ++- docs/README.md | 1 + docs/aai_integration.md | 115 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 123 insertions(+), 1 deletion(-) create mode 100644 docs/aai_integration.md diff --git a/.env.example b/.env.example index d6ca357b..e7d5842f 100644 --- a/.env.example +++ b/.env.example @@ -139,9 +139,15 @@ EVA_MODEL__TTS_PARAMS='{"api_key": "your_cartesia_api_key", "model": "sonic"}' # --- Framework (S2S / AudioLLM) --- #i Base framework for S2S or AudioLLM pipelines. #d enum -#e pipecat,openai_realtime,gemini_live,elevenlabs,grok_voice +#e pipecat,openai_realtime,gemini_live,elevenlabs,grok_voice,aai #v EVA_FRAMEWORK=openai_realtime +#i WebSocket endpoint of a running aai host. EVA appends ?host=1 to select host +#i mode; start the host with AAI_ALLOW_HOST=1. See docs/aai_integration.md. +#d string +#x EVA_FRAMEWORK=aai +#v AAI_WS_URL=ws://localhost:3000/websocket + # ============================================== # LiteLLM Deployments # ============================================== diff --git a/docs/README.md b/docs/README.md index 0bd7fec2..77c52763 100644 --- a/docs/README.md +++ b/docs/README.md @@ -3,6 +3,7 @@ | Document | Description | |---|---| | [Assistant Server Contract](assistant_server_contract.md) | How to implement a new framework integration (s2s model or custom pipeline) | +| [aai Integration](aai_integration.md) | Evaluating the aai voice-agent framework via host mode | | [Metrics](metrics/README.md) | Metric definitions, scoring rubrics, and judge prompts | | [MetricContext](metric_context.md) | Data structures passed to metrics during evaluation | | [LLM Configuration](llm_configuration.md) | LLM provider setup and LiteLLM routing | diff --git a/docs/aai_integration.md b/docs/aai_integration.md new file mode 100644 index 00000000..474d42b6 --- /dev/null +++ b/docs/aai_integration.md @@ -0,0 +1,115 @@ +# Evaluating the aai Voice Agent + +EVA can evaluate the **aai** voice-agent framework via aai's **host mode**, in +which EVA supplies the domain system prompt, greeting, and tool schemas per +session and aai relays each tool call back to EVA for execution. + +Tool execution must happen inside EVA: the accuracy metrics diff +`initial_scenario_db.json` against `final_scenario_db.json`, so a backend that +ran tools internally would score as if nothing happened. + +## Prerequisites + +1. **A running aai host with host mode enabled.** From your aai checkout: + + ```bash + AAI_ALLOW_HOST=1 pnpm dev + ``` + + The default endpoint is `ws://localhost:3000/websocket`; EVA appends + `?host=1` to select host mode. Without `AAI_ALLOW_HOST`, the host rejects the + handshake and EVA aborts the conversation with a logged error. + +2. **EVA's usual keys** in `.env` — a user simulator (ElevenLabs or OpenAI + Realtime) plus the judge-model credentials the metrics need. See the main + README. + +## Configuration + +| Setting | Default | Purpose | +|---|---|---| +| `EVA_FRAMEWORK=aai` | — | Selects this server. | +| `EVA_MODEL__S2S=aai-host` | — | Marks the run as speech-to-speech. | +| `AAI_WS_URL` | `ws://localhost:3000/websocket` | aai host endpoint. | +| `s2s_params.ws_url` | — | Per-run override, takes precedence over `AAI_WS_URL`. | +| `s2s_params.model` | `aai-host` | Label recorded in the metrics log. | +| `s2s_params.input_sample_rate` | `16000` | PCM16 rate sent to aai. | +| `s2s_params.output_sample_rate` | `24000` | PCM16 rate received from aai. | + +## Running + +```bash +AAI_ALLOW_HOST=1 EVA_FRAMEWORK=aai EVA_MODEL__S2S=aai-host \ +EVA_USER_SIMULATOR__PROVIDER=openai_realtime EVA_DOMAIN=itsm \ +EVA_RECORD_IDS=15 EVA_MAX_CONCURRENT_CONVERSATIONS=1 eva +``` + +`EVA_MAX_CONCURRENT_CONVERSATIONS` above 1 works: host mode builds a fresh +single-use runtime per connection, so each conversation gets its own session. + +All three domains (airline, itsm, medical_hr) work without further setup — the +system prompt and tool schemas come from `configs/agents/*.yaml`. + +## Architecture + +``` +user simulator ──Twilio μ-law 8k──▶ AAIAssistantServer ──PCM16 16k───▶ aai host + ◀──Twilio μ-law 8k── localhost:{port}/ws ◀──PCM16 24k── ?host=1 + │ + execute_tool() + ▼ + scenario database +``` + +| Module | Responsibility | +|---|---| +| `assistant/ws_bridge_server.py` | Shared bridge: Twilio framing, 20 ms pacing, recording buffers, turn bookkeeping, metrics. | +| `assistant/bridge_events.py` | Backend-agnostic event vocabulary and the session protocol. | +| `assistant/aai_session.py` | aai host handshake, audio frames, `tool_result` relay. | +| `assistant/aai_events.py` | aai wire-event models. | +| `assistant/aai_server.py` | Prompt and tool-schema construction. | + +Adding another WebSocket voice backend means writing one session adapter plus a +thin server subclass — no changes to the bridge. + +### Turn taking and latency + +Two VAD systems are in play. The user simulator's `user_speech_start` / +`user_speech_stop` events carry wall-clock timestamps and are the only source +used for `model_response` latency, as the assistant-server contract specifies. +aai's own `speech_started` drives barge-in. Mixing the two would silently skew +the latency metric. + +## Known gaps + +- **No token usage.** aai emits no usage events, so `pipecat_metrics.jsonl` + contains latency entries only. `model_response_latency` is unaffected; + token-based cost reporting is unavailable for aai runs. +- **Backend output must be 24 kHz.** Only a 24 kHz → μ-law converter exists in + `audio_bridge.py`. Another rate raises at session start with a clear message + rather than producing distorted audio. + +## Troubleshooting + +| Symptom | Cause | +|---|---| +| `aai host did not acknowledge the host-mode config` | `AAI_ALLOW_HOST` not set, or the host is not listening. | +| `aai host rejected host mode` | Host mode disabled server-side, or the tool schemas failed validation (each tool needs a non-empty description). | +| `Could not connect to aai host` | Wrong `AAI_WS_URL`, or the host is not running. | +| Empty transcript, conversation scored as failure | Check the log for `Failed to open backend session` — the handshake aborted. | + +## Tests + +```bash +uv run python -m pytest tests/unit/assistant/test_aai_events.py \ + tests/unit/assistant/test_aai_session.py \ + tests/unit/assistant/test_aai_server.py \ + tests/unit/assistant/test_ws_bridge_server.py \ + tests/integration/test_aai_fake_backend.py +``` + +The integration test drives both WebSocket hops against an in-process fake aai +host, so it needs no API keys and no Node process. + +Note: use `python -m pytest` rather than `uv run pytest`, which can resolve to a +pytest outside the project virtualenv. From b06a411283a82a676209d261d6a079fa2654abd9 Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Wed, 29 Jul 2026 14:38:55 -0400 Subject: [PATCH 09/11] feat(aai): authenticate host mode against deployed agents MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A deployed agent's WebSocket is deliberately unauthenticated, so the platform requires the owner's API key as a bearer token before honoring host-mode prompt and tool overrides — without that check, host mode would turn any deployed agent into an open LLM proxy billed to its owner. A local `aai dev` host gates on AAI_ALLOW_HOST instead and needs no key. AAIHostSession.connect() therefore takes an api_key and sends it as Authorization: Bearer on the upgrade. Header only, never a query parameter: a URL travels through proxy logs and Referer headers, and this token is the caller's whole platform credential. The key is read from s2s_params.api_key or AAI_API_KEY. RunConfig already requires api_key to be present for any S2S provider, so this gives that field a real meaning for aai rather than a placeholder. Also maps the two rejections a host-mode upgrade can hit onto actionable messages — 401 (no bearer token) and 403 (key does not own the slug) otherwise surface as an opaque InvalidStatus. Verified against a deployed agent: handshake completes, 59 tool schemas accepted, audio flows both ways. Co-Authored-By: Claude Opus 5 (1M context) --- src/eva/__init__.py | 2 +- src/eva/assistant/aai_server.py | 3 + src/eva/assistant/aai_session.py | 37 +++++++++++- tests/integration/test_aai_fake_backend.py | 1 + tests/unit/assistant/test_aai_session.py | 70 ++++++++++++++++++++++ 5 files changed, 109 insertions(+), 4 deletions(-) diff --git a/src/eva/__init__.py b/src/eva/__init__.py index eb2eaa6a..d64fa6b4 100644 --- a/src/eva/__init__.py +++ b/src/eva/__init__.py @@ -7,7 +7,7 @@ # Bump simulation_version when changes affect benchmark outputs (agent code, # user simulator, orchestrator, simulation prompts, agent configs, tool mocks). -simulation_version = "2.0.1" +simulation_version = "2.1.0" # Bump metrics_version when changes affect metric computation (metrics code, # judge prompts, pricing tables, postprocessor). diff --git a/src/eva/assistant/aai_server.py b/src/eva/assistant/aai_server.py index 71566095..ef0cd4fe 100644 --- a/src/eva/assistant/aai_server.py +++ b/src/eva/assistant/aai_server.py @@ -35,6 +35,8 @@ def __init__(self, **kwargs): self._ws_url: str = s2s_params.get("ws_url") or os.environ.get("AAI_WS_URL") or DEFAULT_AAI_WS_URL self._input_rate: int = int(s2s_params.get("input_sample_rate") or DEFAULT_AAI_INPUT_SAMPLE_RATE) self._output_rate: int = int(s2s_params.get("output_sample_rate") or DEFAULT_AAI_OUTPUT_SAMPLE_RATE) + # Only a deployed agent needs this; a local `aai dev` host takes no key. + self._api_key: str | None = s2s_params.get("api_key") or os.environ.get("AAI_API_KEY") @property def model_name(self) -> str: @@ -71,4 +73,5 @@ async def _open_backend(self) -> VoiceBackendSession: greeting=self.initial_message, input_rate=self._input_rate, output_rate=self._output_rate, + api_key=self._api_key, ) diff --git a/src/eva/assistant/aai_session.py b/src/eva/assistant/aai_session.py index 14d25db5..69643772 100644 --- a/src/eva/assistant/aai_session.py +++ b/src/eva/assistant/aai_session.py @@ -69,6 +69,26 @@ def with_host_flag(url: str) -> str: return urlunparse(parsed._replace(query=urlencode(params, doseq=True))) +def _explain_connect_error(exc: Exception) -> str: + """Append a cause to the HTTP rejections a host-mode upgrade can hit. + + The platform answers a host-mode upgrade with 401 when the bearer token is + absent and 403 when it is valid but does not own the slug; both arrive as an + opaque ``InvalidStatus`` otherwise. + """ + status = getattr(getattr(exc, "response", None), "status_code", None) + hints = { + 401: ( + "host mode on a deployed agent requires the owner's API key — set " + 'EVA_MODEL__S2S_PARAMS \'{"api_key": "..."}\' or AAI_API_KEY ' + "(a local `aai dev` host uses AAI_ALLOW_HOST instead)" + ), + 403: "the API key is valid but does not own this agent slug", + } + hint = hints.get(status) if isinstance(status, int) else None + return f"{exc} — {hint}" if hint else str(exc) + + def build_host_config_message( *, system_prompt: str, @@ -141,20 +161,31 @@ async def connect( greeting: str | None = None, input_rate: int = DEFAULT_AAI_INPUT_SAMPLE_RATE, output_rate: int = DEFAULT_AAI_OUTPUT_SAMPLE_RATE, + api_key: str | None = None, ) -> "AAIHostSession": """Open a host-mode session and complete the handshake. + *api_key* is the agent owner's platform API key, sent as a bearer token + on the upgrade. A deployed agent's WebSocket is otherwise + unauthenticated, so the platform requires proof of slug ownership before + honoring prompt and tool overrides — without it, host mode would turn + any deployed agent into an open LLM proxy billed to its owner. A local + ``aai dev`` host gates on ``AAI_ALLOW_HOST`` instead and needs no key. + Raises: AAIHostSessionError: the host rejected the handshake, or did not acknowledge it within ``CONFIG_ACK_TIMEOUT_S``. """ url = with_host_flag(ws_url) - logger.info(f"Connecting to aai host at {url} ({len(tools)} tools)") + logger.info(f"Connecting to aai host at {url} ({len(tools)} tools, api_key={'yes' if api_key else 'no'})") + # Header only: a query parameter would leak the caller's whole platform + # credential into proxy logs and Referer headers. + headers = {"Authorization": f"Bearer {api_key}"} if api_key else None try: # max_size=None: TTS audio frames can exceed the 1 MiB default. - ws = await websockets.connect(url, max_size=None) + ws = await websockets.connect(url, max_size=None, additional_headers=headers) except Exception as e: - raise AAIHostSessionError(f"Could not connect to aai host at {url}: {e}") from e + raise AAIHostSessionError(f"Could not connect to aai host at {url}: {_explain_connect_error(e)}") from e try: await ws.send( diff --git a/tests/integration/test_aai_fake_backend.py b/tests/integration/test_aai_fake_backend.py index 0d109198..cd2b1fc9 100644 --- a/tests/integration/test_aai_fake_backend.py +++ b/tests/integration/test_aai_fake_backend.py @@ -109,6 +109,7 @@ async def _fake_execute(name, args): srv._ws_url = f"ws://localhost:{FAKE_AAI_PORT}/websocket" srv._input_rate = 16000 srv._output_rate = 24000 + srv._api_key = None # The fake host is unauthenticated, like `aai dev`. srv._session = None srv._to_backend = None srv._from_backend = None diff --git a/tests/unit/assistant/test_aai_session.py b/tests/unit/assistant/test_aai_session.py index 49562eeb..f946e81f 100644 --- a/tests/unit/assistant/test_aai_session.py +++ b/tests/unit/assistant/test_aai_session.py @@ -218,3 +218,73 @@ async def test_await_config_ack_raises_on_error_event(self): await AAIHostSession._await_config_ack(ws, timeout_s=1.0) assert "AAI_ALLOW_HOST" in str(excinfo.value) + + +class _FakeResponse: + """Carries the status code websockets attaches to an InvalidStatus.""" + + def __init__(self, status_code: int): + self.status_code = status_code + + +class TestConnectAuth: + """Host mode on a deployed agent authenticates with the owner's API key.""" + + async def _connect(self, monkeypatch, *, api_key=None, raises=None): + captured: dict = {} + + async def fake_connect(url, **kwargs): + captured["url"] = url + captured.update(kwargs) + if raises is not None: + raise raises + return _FakeWebSocket([json.dumps({"type": "config"})]) + + monkeypatch.setattr("eva.assistant.aai_session.websockets.connect", fake_connect) + session = await AAIHostSession.connect( + ws_url="wss://host.example/agent/websocket", + system_prompt="be helpful", + tools=[], + api_key=api_key, + ) + return session, captured + + async def test_api_key_is_sent_as_a_bearer_header(self, monkeypatch): + _, captured = await self._connect(monkeypatch, api_key="secret-key") + + assert captured["additional_headers"] == {"Authorization": "Bearer secret-key"} + + async def test_api_key_never_travels_in_the_url(self, monkeypatch): + _, captured = await self._connect(monkeypatch, api_key="secret-key") + + assert "secret-key" not in captured["url"] + + async def test_no_api_key_sends_no_headers(self, monkeypatch): + """A local `aai dev` host gates on AAI_ALLOW_HOST and takes no key.""" + _, captured = await self._connect(monkeypatch) + + assert captured["additional_headers"] is None + + async def test_401_explains_the_missing_owner_key(self, monkeypatch): + error = Exception("server rejected WebSocket connection: HTTP 401") + error.response = _FakeResponse(401) # type: ignore[attr-defined] + + with pytest.raises(AAIHostSessionError) as excinfo: + await self._connect(monkeypatch, raises=error) + + assert "owner's API key" in str(excinfo.value) + + async def test_403_explains_the_key_does_not_own_the_slug(self, monkeypatch): + error = Exception("server rejected WebSocket connection: HTTP 403") + error.response = _FakeResponse(403) # type: ignore[attr-defined] + + with pytest.raises(AAIHostSessionError) as excinfo: + await self._connect(monkeypatch, api_key="wrong-owner", raises=error) + + assert "does not own this agent slug" in str(excinfo.value) + + async def test_other_connect_errors_pass_through_unembellished(self, monkeypatch): + with pytest.raises(AAIHostSessionError) as excinfo: + await self._connect(monkeypatch, raises=OSError("connection refused")) + + assert "connection refused" in str(excinfo.value) From 62db5a6c3561f2b21333f7390fe788f310c9157d Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Wed, 29 Jul 2026 14:39:18 -0400 Subject: [PATCH 10/11] fix(user-simulator): send wall-clock user_speech_stop timestamps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The send loop's `current_time` is the event loop's monotonic clock, which shares no origin with wall time. It was going out on the wire as user_speech_stop's timestamp_ms, while assistant servers subtract that stamp from their own time.time() reading to compute model_response latency. Every latency therefore came out as the gap between the two clocks — ~1.8e12 ms — and was discarded as implausible, so the metric was never recorded at all. user_speech_start was already wall-clock (time.time()), so the two events were not even in the same domain as each other. The monotonic-to-wall conversion already existed one block above, but was scoped inside `if self.event_logger:`. Hoisting it fixes the wire value and keeps the existing semantics: the stamp is anchored to the last real audio chunk (~600 ms before detection), not to detection time. This also repairs gemini_live_server, whose bare `0 < latency_ms < 30_000` guard has been silently dropping every value for the same reason. Verified against a deployed aai agent: 9 model_response samples recorded across one conversation (2.9-3.6 s, one 11.7 s outlier) where the previous code recorded none, and correctly no sample for the model-initiated greeting turn. Co-Authored-By: Claude Opus 5 (1M context) --- src/eva/__init__.py | 2 +- src/eva/user_simulator/audio_bridge.py | 20 +++-- .../unit/user_simulator/test_audio_bridge.py | 78 +++++++++++++++++++ 3 files changed, 92 insertions(+), 8 deletions(-) diff --git a/src/eva/__init__.py b/src/eva/__init__.py index d64fa6b4..f38f4aa1 100644 --- a/src/eva/__init__.py +++ b/src/eva/__init__.py @@ -7,7 +7,7 @@ # Bump simulation_version when changes affect benchmark outputs (agent code, # user simulator, orchestrator, simulation prompts, agent configs, tool mocks). -simulation_version = "2.1.0" +simulation_version = "2.1.1" # Bump metrics_version when changes affect metric computation (metrics code, # judge prompts, pricing tables, postprocessor). diff --git a/src/eva/user_simulator/audio_bridge.py b/src/eva/user_simulator/audio_bridge.py index d1e75415..b3caa038 100644 --- a/src/eva/user_simulator/audio_bridge.py +++ b/src/eva/user_simulator/audio_bridge.py @@ -453,13 +453,19 @@ async def _on_user_audio_end(self, current_time: float) -> None: self._user_audio_active = False # Consume the end-of-turn cue so it doesn't carry into the next turn. self._user_turn_complete = False + # *current_time* is the event loop's monotonic clock, which shares no + # origin with wall time. Convert once here: assistant servers subtract + # this stamp from a ``time.time()`` reading to get model_response + # latency, and ``user_speech_start`` is already wall-clock, so emitting + # the raw monotonic value made every latency ~the clock gap (~1.8e12 ms) + # and get discarded as implausible. + # Detection lags the real end by ~600ms; stamp at the last real chunk. + real_end_unix = ( + time.time() - (current_time - self._last_user_audio_send_time) + if self._last_user_audio_send_time is not None + else time.time() + ) if self.event_logger: - # Detection lags the real end by ~600ms; stamp at the last real chunk. - real_end_unix = ( - time.time() - (current_time - self._last_user_audio_send_time) - if self._last_user_audio_send_time is not None - else time.time() - ) self.event_logger.log_audio_end("simulated_user", real_end_unix) logger.info("🎤 User audio END") @@ -471,7 +477,7 @@ async def _on_user_audio_end(self, current_time: float) -> None: { "event": "user_speech_stop", "conversation_id": self.conversation_id, - "timestamp_ms": str(int(round(current_time * 1000))), + "timestamp_ms": str(int(round(real_end_unix * 1000))), } ) ) diff --git a/tests/unit/user_simulator/test_audio_bridge.py b/tests/unit/user_simulator/test_audio_bridge.py index d416ced9..c955df16 100644 --- a/tests/unit/user_simulator/test_audio_bridge.py +++ b/tests/unit/user_simulator/test_audio_bridge.py @@ -8,9 +8,11 @@ import asyncio import base64 import json +import time from unittest.mock import AsyncMock, MagicMock, patch import pytest +from websockets.protocol import State as WebSocketState from eva.user_simulator.audio_bridge import ( ASSISTANT_SAMPLE_RATE, @@ -393,3 +395,79 @@ async def test_handles_no_websocket_or_tasks(self): iface = _make_interface() await iface.stop_async() assert iface._stopping is True + + +class TestSpeechEventClockDomain: + """user_speech_start/stop must share one clock. + + Assistant servers diff these stamps against ``time.time()`` to compute + model_response latency. + """ + + @staticmethod + def _open_ws(): + ws = AsyncMock() + ws.state = WebSocketState.OPEN + return ws + + @staticmethod + def _sent_events(ws) -> dict: + return { + frame["event"]: frame + for frame in (json.loads(call.args[0]) for call in ws.send.call_args_list) + if "event" in frame + } + + @pytest.mark.asyncio + async def test_user_speech_stop_timestamp_is_wall_clock(self): + """A monotonic loop time must not leak onto the wire.""" + iface = _make_interface() + iface.websocket = self._open_ws() + iface._user_audio_active = True + # A plausible monotonic reading: seconds since boot, nowhere near epoch. + monotonic_now = 159_381.0 + iface._last_user_audio_send_time = monotonic_now - 0.6 + + before_ms = time.time() * 1000 + await iface._on_user_audio_end(monotonic_now) + after_ms = time.time() * 1000 + + stamp = int(self._sent_events(iface.websocket)["user_speech_stop"]["timestamp_ms"]) + # Stamped at the last real chunk, ~600ms before now. + assert before_ms - 2000 <= stamp <= after_ms + assert stamp != int(round(monotonic_now * 1000)) + + @pytest.mark.asyncio + async def test_start_and_stop_are_in_the_same_clock_domain(self): + """The two stamps must be differenceable — a real latency, not a clock gap.""" + iface = _make_interface() + iface.websocket = self._open_ws() + + with patch("eva.user_simulator.audio_bridge.asyncio.get_event_loop") as mock_loop: + mock_loop.return_value.time.return_value = 159_380.0 + await iface._on_user_audio_start() + iface._user_audio_active = True + iface._last_user_audio_send_time = 159_381.0 + await iface._on_user_audio_end(159_381.0) + + events = self._sent_events(iface.websocket) + start = int(events["user_speech_start"]["timestamp_ms"]) + stop = int(events["user_speech_stop"]["timestamp_ms"]) + # Same domain => the gap is a real utterance duration, not ~1.8e12 ms. + assert 0 <= stop - start < 30_000 + + @pytest.mark.asyncio + async def test_stop_timestamp_survives_a_missing_send_time(self): + """No prior chunk send: fall back to now rather than emitting monotonic.""" + iface = _make_interface() + iface.websocket = self._open_ws() + iface._user_audio_active = True + iface._last_user_audio_send_time = None + + before_ms = time.time() * 1000 + await iface._on_user_audio_end(159_381.0) + after_ms = time.time() * 1000 + + stamp = int(self._sent_events(iface.websocket)["user_speech_stop"]["timestamp_ms"]) + # 1ms slack: the stamp is rounded to whole milliseconds. + assert before_ms - 1 <= stamp <= after_ms + 1 From e1736c7ec7527915302b161503df0e981223e880 Mon Sep 17 00:00:00 2001 From: Alex Kroman Date: Wed, 29 Jul 2026 14:39:28 -0400 Subject: [PATCH 11/11] docs(aai): deployed-agent auth, and correct three wrong claims MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Documents running against a deployed agent: the bearer-token requirement, api_key/AAI_API_KEY in the config table, and 401/403 plus the agent's own LLM failures in troubleshooting. Corrects three things that were wrong: - The run command carried AAI_ALLOW_HOST, which EVA never reads. It gates host mode server-side on the aai host process only. - The turn-taking section claimed the simulator's speech events carry wall-clock timestamps. They did not, which is what the preceding commit fixes; the note now explains the conversion and why it matters. - The test command omitted --extra dev, so it failed on a default `uv sync` — pytest lives in an optional extra. Co-Authored-By: Claude Opus 5 (1M context) --- docs/aai_integration.md | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/docs/aai_integration.md b/docs/aai_integration.md index 474d42b6..01b3050e 100644 --- a/docs/aai_integration.md +++ b/docs/aai_integration.md @@ -20,6 +20,15 @@ ran tools internally would score as if nothing happened. `?host=1` to select host mode. Without `AAI_ALLOW_HOST`, the host rejects the handshake and EVA aborts the conversation with a logged error. + **Or a deployed agent**, at `wss:////websocket`. The platform + gates host mode differently: an agent's WebSocket is deliberately + unauthenticated, so allowing prompt and tool overrides on that footing would + turn any deployed agent into an open LLM proxy billed to its owner. It + therefore requires the owner's platform API key as a bearer token on the + upgrade — set `s2s_params.api_key` or `AAI_API_KEY`. `401` means no key was + sent; `403` means the key does not own that slug. No `AAI_ALLOW_HOST` + involved. + 2. **EVA's usual keys** in `.env` — a user simulator (ElevenLabs or OpenAI Realtime) plus the judge-model credentials the metrics need. See the main README. @@ -32,6 +41,7 @@ ran tools internally would score as if nothing happened. | `EVA_MODEL__S2S=aai-host` | — | Marks the run as speech-to-speech. | | `AAI_WS_URL` | `ws://localhost:3000/websocket` | aai host endpoint. | | `s2s_params.ws_url` | — | Per-run override, takes precedence over `AAI_WS_URL`. | +| `s2s_params.api_key` / `AAI_API_KEY` | — | Agent owner's platform key. Required for a **deployed** agent, ignored by `aai dev`. | | `s2s_params.model` | `aai-host` | Label recorded in the metrics log. | | `s2s_params.input_sample_rate` | `16000` | PCM16 rate sent to aai. | | `s2s_params.output_sample_rate` | `24000` | PCM16 rate received from aai. | @@ -39,11 +49,13 @@ ran tools internally would score as if nothing happened. ## Running ```bash -AAI_ALLOW_HOST=1 EVA_FRAMEWORK=aai EVA_MODEL__S2S=aai-host \ +EVA_FRAMEWORK=aai EVA_MODEL__S2S=aai-host \ EVA_USER_SIMULATOR__PROVIDER=openai_realtime EVA_DOMAIN=itsm \ EVA_RECORD_IDS=15 EVA_MAX_CONCURRENT_CONVERSATIONS=1 eva ``` +`AAI_ALLOW_HOST` belongs on the *host* process, not here — EVA never reads it. + `EVA_MAX_CONCURRENT_CONVERSATIONS` above 1 works: host mode builds a fresh single-use runtime per connection, so each conversation gets its own session. @@ -75,8 +87,13 @@ thin server subclass — no changes to the bridge. ### Turn taking and latency Two VAD systems are in play. The user simulator's `user_speech_start` / -`user_speech_stop` events carry wall-clock timestamps and are the only source -used for `model_response` latency, as the assistant-server contract specifies. +`user_speech_stop` events are the only source used for `model_response` latency, +as the assistant-server contract specifies. Both carry wall-clock (`time.time()`) +milliseconds; the bridge subtracts them from its own `time.time()` reading, so +the simulator converts its monotonic send-loop clock before emitting +(`user_simulator/audio_bridge.py:_on_user_audio_end`). Emitting the raw +monotonic value makes every latency register as the clock gap (~1.8e12 ms) and +get discarded. aai's own `speech_started` drives barge-in. Mixing the two would silently skew the latency metric. @@ -94,6 +111,9 @@ the latency metric. | Symptom | Cause | |---|---| | `aai host did not acknowledge the host-mode config` | `AAI_ALLOW_HOST` not set, or the host is not listening. | +| `HTTP 401` on connect | Deployed agent, no API key sent — set `s2s_params.api_key` or `AAI_API_KEY`. | +| `HTTP 403` on connect | The API key is valid but does not own that agent slug. | +| `llm: Failed after 3 attempts … Internal Server Error` | The *agent's own* LLM provider is failing. Host mode runs on the deployed agent's credentials and pipeline; nothing to fix in EVA. | | `aai host rejected host mode` | Host mode disabled server-side, or the tool schemas failed validation (each tool needs a non-empty description). | | `Could not connect to aai host` | Wrong `AAI_WS_URL`, or the host is not running. | | Empty transcript, conversation scored as failure | Check the log for `Failed to open backend session` — the handshake aborted. | @@ -101,7 +121,7 @@ the latency metric. ## Tests ```bash -uv run python -m pytest tests/unit/assistant/test_aai_events.py \ +uv run --extra dev python -m pytest tests/unit/assistant/test_aai_events.py \ tests/unit/assistant/test_aai_session.py \ tests/unit/assistant/test_aai_server.py \ tests/unit/assistant/test_ws_bridge_server.py \ @@ -111,5 +131,7 @@ uv run python -m pytest tests/unit/assistant/test_aai_events.py \ The integration test drives both WebSocket hops against an in-process fake aai host, so it needs no API keys and no Node process. -Note: use `python -m pytest` rather than `uv run pytest`, which can resolve to a -pytest outside the project virtualenv. +Note: `pytest` lives in the `dev` extra, which `uv sync` skips by default — hence +`--extra dev` (or run `uv sync --all-extras` once). Use `python -m pytest` rather +than `uv run pytest`, which silently resolves to a pytest outside the project +virtualenv.