Evaluate the aai voice-agent framework via host mode

[alexkroman]

Adds an assistant server so EVA can benchmark the aai voice-agent framework through aai's host mode, where EVA supplies the domain system prompt, greeting, and tool schemas per session and aai relays each tool call back to EVA for execution.

Tool execution has to happen inside EVA: the accuracy metrics diff initial_scenario_db.json against final_scenario_db.json, so a backend that ran tools internally would score as if nothing happened.

Structure

The reusable part is split from the aai-specific part, so adding another WebSocket voice backend means writing one session adapter plus a thin server subclass — no changes to the bridge:

Module	Responsibility
assistant/ws_bridge_server.py	Shared bridge: Twilio framing, 20 ms pacing, recording buffers, turn bookkeeping, metrics
assistant/bridge_events.py	Backend-agnostic event vocabulary and the session protocol
assistant/aai_session.py	aai host handshake, audio frames, tool_result relay
assistant/aai_events.py	aai wire-event models, tolerant parser
assistant/aai_server.py	Prompt and tool-schema construction

All three domains (airline, itsm, medical_hr) work with no extra setup — prompts and tool schemas come from configs/agents/*.yaml. EVA_MAX_CONCURRENT_CONVERSATIONS above 1 is fine: host mode builds a fresh single-use runtime per connection.

Two fixes found by running it end-to-end

Deployed-agent authentication. A deployed aai agent's WebSocket is deliberately unauthenticated, so the platform requires the owner's API key as a bearer token before honoring host-mode prompt and tool overrides — otherwise host mode would turn any deployed agent into an open LLM proxy billed to its owner. connect() now sends Authorization: Bearer, header-only (a URL travels through proxy logs and Referer headers). 401 and 403 get actionable messages instead of an opaque InvalidStatus. A local aai dev host gates on AAI_ALLOW_HOST and needs no key.

model_response latency was never recorded — for any backend. The user simulator's send loop uses asyncio.get_event_loop().time(), a monotonic clock, and that value was going out as user_speech_stop's timestamp_ms. Assistant servers subtract it from their own time.time() reading, so every latency came out as the gap between the two clocks (~1.8e12 ms) and was discarded as implausible. user_speech_start was already wall-clock, so the two events were not even in the same domain as each other.

The conversion already existed one block above but was scoped inside if self.event_logger:; hoisting it fixes the wire value while keeping the stamp anchored to the last real audio chunk (~600 ms before detection) rather than to detection time.

This also repairs gemini_live_server, whose bare 0 < latency_ms < 30_000 guard has been silently dropping every value for the same reason. elevenlabs_server prefers ElevenLabs' own end-of-turn timestamp and only hit this on fallback.

Verified against a deployed agent: 9 model_response samples across one conversation (2.9–3.6 s, one 11.7 s outlier) where the previous code recorded none — and correctly no sample for the model-initiated greeting turn.

Tests

• tests/unit/assistant/ — wire events, session framing and handshake, bearer auth, prompt/tool construction, bridge pacing and turn bookkeeping
• tests/integration/test_aai_fake_backend.py — drives both WebSocket hops against an in-process fake aai host, so it needs no API keys and no Node process
• tests/unit/user_simulator/test_audio_bridge.py — pins that a monotonic timestamp never reaches the wire and that the two speech events stay in one clock domain

Full suite: 1997 passed, 52 skipped, 3 xfailed. ruff check clean.

Note: run tests with uv run --extra dev python -m pytest — pytest lives in an optional extra that uv sync skips by default, and bare uv run pytest silently resolves to a pytest outside the project virtualenv.

Known gaps

• No token usage. aai emits no usage events, so pipecat_metrics.jsonl holds latency entries only. model_response_latency is unaffected; token-based cost reporting is unavailable for aai runs.
• Backend output must be 24 kHz. Only a 24 kHz → μ-law converter exists in audio_bridge.py; another rate raises at session start with a clear message rather than producing distorted audio.

Setup, configuration, and troubleshooting are documented in docs/aai_integration.md.

🤖 Generated with Claude Code