[Do not merge][Debug] Preview for voiceagent - #48484
Open
xitzhang (xitzhang) wants to merge 5 commits into
Open
Conversation
xitzhang (xitzhang)
requested review from
Renhe Li (lirenhe) and
Jeff Fisher (xirzec)
as code owners
August 6, 2026 21:05
xitzhang (xitzhang)
changed the title
|
Azure Pipelines: Successfully started running 1 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
This PR introduces an initial preview of the azure-ai-voiceagents Python SDK, including generated management clients, a hand-written async realtime (WebSocket) client, and supporting samples/tests for validation and guidance.
Changes:
- Adds the new
azure-ai-voiceagentspackage skeleton (generated clients/models + packaging metadata). - Implements an async realtime WebSocket client (
aio/_realtime.py) and an aiohttp transport workaround for brotli negotiation. - Adds unit tests, recorded tests (with sanitization), live smoke test, and scenario-based samples + sample README.
Reviewed changes
Copilot reviewed 59 out of 63 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| sdk/voiceagents/azure-ai-voiceagents/tsp-location.yaml | Tracks source spec location for generation/regeneration. |
| sdk/voiceagents/azure-ai-voiceagents/tests/unit/test_configuration.py | Unit tests for configuration defaults/required params. |
| sdk/voiceagents/azure-ai-voiceagents/tests/unit/test_client_construction.py | Unit tests for sync/async client construction and operation groups. |
| sdk/voiceagents/azure-ai-voiceagents/tests/unit/test_brotli_workaround.py | Unit tests validating async brotli/aiohttp transport workaround behavior. |
| sdk/voiceagents/azure-ai-voiceagents/tests/unit/conftest.py | Disables recorded-test proxy startup for unit-test runs. |
| sdk/voiceagents/azure-ai-voiceagents/tests/recording/test_voice_agents_client_async.py | Recorded async functional tests for read-only operations. |
| sdk/voiceagents/azure-ai-voiceagents/tests/recording/test_voice_agents_client.py | Recorded sync functional tests for read-only operations. |
| sdk/voiceagents/azure-ai-voiceagents/tests/recording/conftest.py | Adds extra URI sanitization for project name in recordings. |
| sdk/voiceagents/azure-ai-voiceagents/tests/recording/_preparer.py | Shared preparer/sanitized env var loader for recorded/live tests. |
| sdk/voiceagents/azure-ai-voiceagents/tests/live/test_smoke_live.py | Always-live smoke test to catch auth/wire/serialization issues. |
| sdk/voiceagents/azure-ai-voiceagents/tests/live/conftest.py | Disables recorded-test proxy startup for live-test runs. |
| sdk/voiceagents/azure-ai-voiceagents/tests/conftest.py | Starts the test-proxy for recorded tests. |
| sdk/voiceagents/azure-ai-voiceagents/samples/quickstart/sample_quickstart_async.py | End-to-end quickstart: create temp agent + realtime audio session + cleanup. |
| sdk/voiceagents/azure-ai-voiceagents/samples/management/sample_read_conversation_audio.py | Demonstrates reading and streaming persisted conversation audio to WAV. |
| sdk/voiceagents/azure-ai-voiceagents/samples/management/sample_read_conversation.py | Demonstrates reading persisted conversation envelope/items/responses. |
| sdk/voiceagents/azure-ai-voiceagents/samples/management/sample_manage_voice_agent_versions.py | Demonstrates creating/listing/fetching voice agent versions (incl drafts). |
| sdk/voiceagents/azure-ai-voiceagents/samples/management/sample_generate_voice_agent.py | Demonstrates guided-authoring agent generation and cleanup. |
| sdk/voiceagents/azure-ai-voiceagents/samples/management/sample_create_voice_agent_with_tools.py | Demonstrates richer agent definitions (tools + audio config + BYOM). |
| sdk/voiceagents/azure-ai-voiceagents/samples/management/sample_create_and_manage_voice_agent_async.py | Async management lifecycle sample: create/get/list/delete. |
| sdk/voiceagents/azure-ai-voiceagents/samples/management/sample_create_and_manage_voice_agent.py | Sync management lifecycle sample incl update and enable/disable. |
| sdk/voiceagents/azure-ai-voiceagents/samples/live/sample_live_text_conversation_async.py | Live typed conversation + optional audio playback + readback. |
| sdk/voiceagents/azure-ai-voiceagents/samples/live/sample_live_audio_conversation_async.py | Live mic/speaker conversation + barge-in + readback. |
| sdk/voiceagents/azure-ai-voiceagents/samples/README.md | Scenario-organized samples index + setup/troubleshooting guidance. |
| sdk/voiceagents/azure-ai-voiceagents/pytest.ini | Configures pytest asyncio integration (asyncio_mode=auto). |
| sdk/voiceagents/azure-ai-voiceagents/pyrightconfig.json | Pyright configuration for local development/type-checking. |
| sdk/voiceagents/azure-ai-voiceagents/pyproject.toml | Package metadata and dependencies for azure-ai-voiceagents. |
| sdk/voiceagents/azure-ai-voiceagents/dev_requirements.txt | Dev dependencies including local azure-core/azure-identity and aiohttp. |
| sdk/voiceagents/azure-ai-voiceagents/azure_ai_voiceagents-1.0.0b1/azure/ai/voiceagents/_configuration.py | Snapshot of generated configuration under versioned folder. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/py.typed | PEP 561 typing marker. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/operations/_patch.py | Customization hook for generated operations (sync). |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/operations/init.py | Exposes sync operation groups and patch hook. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/models/_patch.py | Customization hook for generated models. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/models/_enums.py | Generated enums supporting voice agents + realtime schemas. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/models/init.py | Public models/enums surface exports. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/operations/_patch.py | Customization hook for generated operations (async). |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/operations/init.py | Exposes async operation groups and patch hook. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/_realtime.py | Hand-written async realtime WebSocket client and event typing. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/_patch.py | Async client patch: adds realtime namespace + brotli workaround. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/_configuration.py | Generated async configuration. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/_client.py | Generated async HTTP client and operation group wiring. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/init.py | Async package exports and patch hook. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/_version.py | Package version constant. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/_utils/init.py | Generated utils package init. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/_unions.py | Generated public type aliases/unions. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/_patch.py | Sync customization hook placeholder. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/_configuration.py | Generated sync configuration. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/_client.py | Generated sync HTTP client and operation group wiring. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/init.py | Sync package exports and patch hook + version. |
| sdk/voiceagents/azure-ai-voiceagents/azure/ai/init.py | Namespace package marker for azure.ai. |
| sdk/voiceagents/azure-ai-voiceagents/azure/init.py | Namespace package marker for azure. |
| sdk/voiceagents/azure-ai-voiceagents/assets.json | Azure SDK assets tag configuration for CI. |
| sdk/voiceagents/azure-ai-voiceagents/apiview-properties.json | APIView cross-language mapping metadata. |
| sdk/voiceagents/azure-ai-voiceagents/_metadata.json | Tracks API version metadata for generation. |
| sdk/voiceagents/azure-ai-voiceagents/README.md | Package README scaffold. |
| sdk/voiceagents/azure-ai-voiceagents/MANIFEST.in | Source distribution inclusions (tests/samples/py.typed, etc.). |
| sdk/voiceagents/azure-ai-voiceagents/LICENSE | MIT license text. |
| sdk/voiceagents/azure-ai-voiceagents/CHANGELOG.md | Initial changelog entry for version b1. |
Suppressed comments (3)
sdk/voiceagents/azure-ai-voiceagents/tsp-location.yaml:1
- The PR description is still the template (missing a real summary of changes, links to relevant issues/spec PRs, and checklist updates). Please replace the placeholder description with a concise overview of what this PR adds (new SDK package, realtime WebSocket client, samples/tests) and link the spec change PR/commit referenced in
tsp-location.yaml.
sdk/voiceagents/azure-ai-voiceagents/tests/unit/test_configuration.py:1 - This test doesn’t actually validate overriding behavior because it passes the same value as the default (
\"v1\"). Consider changing it to pass a different value (orNoneif supported by the client) so the test can detect regressions where the override is ignored.
sdk/voiceagents/azure-ai-voiceagents/samples/management/sample_create_and_manage_voice_agent.py:1 - Using a fixed, non-namespaced
agent_namerisks collisions across repeated runs and (more importantly) can inadvertently delete/modify a real agent if a user already has an agent with that name. Safer pattern for samples is to generate a unique name (e.g., including a UUID suffix) and only delete the resource that the sample created.
Contributor
There was a problem hiding this comment.
Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.
Note
This error may be related to your runner configuration. You can now configure runners for Copilot code review separately from Copilot cloud agent by creating a copilot-code-review.yml file with your setup steps. Read the docs for details.
xitzhang (xitzhang)
requested review from
Daniel Jurek (danieljurek) and
Yuchao Yan (msyyc)
as code owners
August 6, 2026 23:30
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 63 out of 67 changed files in this pull request and generated 2 comments.
Suppressed comments (8)
sdk/voiceagents/ci.yml:1
- The PR description is still the template text (it asks to 'add an informative description' and link issues), but the change set is substantial (new SDK + CI + tests + samples). Please update the PR description to summarize the intent/scope and link any relevant spec/issue/PR references so reviewers can validate the changes against requirements.
sdk/voiceagents/azure-ai-voiceagents/pyproject.toml:1 - Excluding the
azureandazure.aipackages from package discovery is likely to produce an invalid wheel where the namespace package initializers (azure/__init__.py,azure/ai/__init__.py) are not installed, breaking imports in environments that rely on pkgutil-style namespace packaging. Recommended fix: remove\"azure\"and\"azure.ai\"from the exclude list (or switch to an explicit include pattern forazure*packages consistent with other Azure SDK Python packages).
sdk/voiceagents/azure-ai-voiceagents/samples/live/sample_live_audio_conversation_async.py:1 - The callback can return fewer than
wantedbytes when it hits the end-of-stream marker (if not data: break). pyaudio callbacks must return exactly the requested frame size; otherwise playback can glitch or error. Instead ofbreak, pad the remainder with silence (or returnpaCompletewith a correctly-sized buffer) to satisfy pyaudio’s size contract.
sdk/voiceagents/azure-ai-voiceagents/tests/unit/test_configuration.py:1 - This test name claims the API version is 'overridden', but it sets the value to the default (
\"v1\"), so it doesn’t actually verify override behavior. Either rename the test to reflect what it asserts (e.g., thatapi_versionis stored), or setapi_versionto a non-default value (if supported by the client) to validate that overriding works.
sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/_realtime.py:648 - Passing the WebSocket subprotocol by manually setting
Sec-WebSocket-Protocolis fragile because many WS clients (including aiohttp) expect to manage/validate this header via theprotocols=[...]argument. Preferws_connect(..., protocols=[\"realtime\"], ...)and remove the manual header to avoid handshake issues and ensure the negotiated subprotocol is tracked correctly by the client.
headers: Dict[str, str] = {
"Authorization": f"Bearer {token.token}",
"Foundry-Features": str(foundry_features),
"Sec-WebSocket-Protocol": "realtime",
}
sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/_realtime.py:655
- Passing the WebSocket subprotocol by manually setting
Sec-WebSocket-Protocolis fragile because many WS clients (including aiohttp) expect to manage/validate this header via theprotocols=[...]argument. Preferws_connect(..., protocols=[\"realtime\"], ...)and remove the manual header to avoid handshake issues and ensure the negotiated subprotocol is tracked correctly by the client.
connection = await session.ws_connect(url, headers=headers, params=params, **self._kwargs)
sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/_configuration.py:36
- The docstring says the default
api_versionisNone, but the implementation defaults it to\"v1\". Please align the docstring with the actual behavior (and consider applying the same fix in the async configuration file as well) to avoid confusing SDK consumers.
:keyword api_version: The API version to use for this operation. Known values are "v1" and
None. Default value is None. If not set, the operation's default API version will be used. Note
that overriding this default value may result in unsupported behavior.
sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/_configuration.py:41
- The docstring says the default
api_versionisNone, but the implementation defaults it to\"v1\". Please align the docstring with the actual behavior (and consider applying the same fix in the async configuration file as well) to avoid confusing SDK consumers.
def __init__(self, endpoint: str, credential: "TokenCredential", **kwargs: Any) -> None:
api_version: str = kwargs.pop("api_version", "v1")
Comment on lines
+535
to
+543
| msg = await self._connection.receive() | ||
| if msg.type in (aiohttp.WSMsgType.CLOSE, aiohttp.WSMsgType.CLOSING, aiohttp.WSMsgType.CLOSED): | ||
| raise ConnectionResetError("The realtime connection was closed.") | ||
| if msg.type == aiohttp.WSMsgType.ERROR: | ||
| raise ConnectionResetError( | ||
| "The realtime connection encountered an error." | ||
| ) from self._connection.exception() | ||
| raw = msg.data.decode("utf-8") if msg.type == aiohttp.WSMsgType.BINARY else msg.data | ||
| payload: Dict[str, Any] = json.loads(raw) |
Comment on lines
+279
to
+286
| async def avatar_connect(self, *, client_sdp: str, event_id: Optional[str] = None) -> None: | ||
| """Negotiate an avatar media session over WebRTC. | ||
|
|
||
| :keyword str client_sdp: The client's SDP offer for avatar media negotiation. | ||
| :keyword event_id: An optional client-generated event identifier. | ||
| :paramtype event_id: str or None | ||
| """ | ||
| await self._send(_models.VoiceAgentClientEventSessionAvatarConnect(client_sdp=client_sdp, event_id=event_id)) |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 63 out of 67 changed files in this pull request and generated 3 comments.
Suppressed comments (6)
sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/_realtime.py:51
- The exported connection's public methods and attributes use
ClientEvent,ServerEvent,ConversationItem, and the resource helper classes in their annotations, but this module andaio/_patch.pyexport only the three connection classes. Consumers therefore see public annotations that can only be imported from the private_realtimemodule. Re-export all types used by this public surface fromazure.ai.voiceagents.aio.
__all__ = [
"AsyncRealtime",
"AsyncRealtimeConnection",
"AsyncRealtimeConnectionManager",
]
sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/_patch.py:45
- Constructing
aiohttp.ClientSessionhere eagerly requires a running event loop and binds the client to that loop. This makes otherwise valid async-client construction outsideasyncio.run()fail before any request. Set the defaultAccept-Encodingthrough the headers policy instead soAioHttpTransportcan retain its lazy session creation.
kwargs["transport"] = AioHttpTransport(
session=aiohttp.ClientSession(auto_decompress=False, headers={"Accept-Encoding": "gzip, deflate"})
)
sdk/voiceagents/azure-ai-voiceagents/samples/live/sample_live_audio_conversation_async.py:138
- Barge-in does not discard a partially consumed chunk: each callback copies bytes from
remainingbefore checking that chunk's sequence against_playback_base. A large delta can therefore continue playing across callbacks afterskip_pending_audio(). Track the sequence associated withremainingand invalidate it before slicing whenever it is older than the playback base.
sdk/voiceagents/azure-ai-voiceagents/samples/quickstart/sample_quickstart_async.py:100 - Barge-in does not discard a partially consumed chunk: bytes are copied from
remainingbefore the callback checks sequence validity. If a delta spans multiple callback buffers, stale audio continues playing afterskip_pending_audio(). Associateremainingwith its sequence and clear it before slicing when that sequence predates_playback_base.
sdk/voiceagents/azure-ai-voiceagents/tests/unit/test_configuration.py:47 - This test passes the same value as the default, so it still passes if the
api_versionoverride is ignored. Use a distinct sentinel value to verify that the keyword is actually honored.
sdk/voiceagents/azure-ai-voiceagents/samples/live/sample_live_text_conversation_async.py:179 - After a timeout, the response remains active and its eventual events stay queued. On the next turn,
pump()can consume the previous response'sresponse.doneand return before the new response completes, or the service can reject the new response while the old one is active. Cancel the timed-out response and drain/match its terminal event before accepting another prompt.
|
|
||
| # ``connection_url`` fully overrides the computed route (scheme/host/path). This is the | ||
| # escape hatch used to reach a specific data-plane host/path directly. | ||
| url = self._connection_url or _to_ws_url(self._endpoint, self._agent_name) |
| build-backend = "setuptools.build_meta" | ||
|
|
||
| [project] | ||
| name = "azure-ai-voiceagents" |
Comment on lines
+10
to
+12
| and deletion are intentionally out of scope: at the time this suite was | ||
| written, the create (expects 201) and delete (expects 204) operations did not | ||
| match what the live test service actually returns (200), so recording those |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 63 out of 67 changed files in this pull request and generated 2 comments.
Suppressed comments (8)
sdk/voiceagents/azure-ai-voiceagents/pyproject.toml:13
- This new package is missing both
api.mdandapi.metadata.yml. The repository API consistency gate explicitly marks an affected package missing either file as a failure (.github/workflows/src/api-md-consistency/find_mismatches.js:76-92). Generate and commit both files withazpysdk apistub azure-ai-voiceagents.
sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/_patch.py:43 - Creating
aiohttp.ClientSessionin the client constructor makesVoiceAgentsClient(...)require a running event loop and binds the transport to that loop. This breaks the normal pattern of constructing an async SDK client beforeasyncio.run(...)(the new test even documents this limitation). Create/configure the session lazily when the transport opens instead.
kwargs["transport"] = AioHttpTransport(
session=aiohttp.ClientSession(auto_decompress=False, headers={"Accept-Encoding": "gzip, deflate"})
)
sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/_realtime.py:242
agent_nameis inserted into the URL as raw path text, unlike the generated operation which URL-serializes this path parameter. Reserved characters can therefore change the route instead of identifying the requested agent. Percent-encode the path segment before interpolation.
return f"{base}/agents/{agent_name}/endpoint/protocols/voice"
sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/_patch.py:65
- The public annotations/docstrings refer to
azure.ai.voiceagents.aio.ClientEventandServerEvent, but these aliases are defined only in the private_realtimemodule and are not re-exported here. Those documented imports therefore fail; include both aliases in the public aio surface.
__all__: list[str] = [
"VoiceAgentsClient",
"AsyncRealtime",
"AsyncRealtimeConnection",
"AsyncRealtimeConnectionManager",
] # Add all objects you want publicly available to users at this package level
sdk/voiceagents/azure-ai-voiceagents/samples/quickstart/sample_quickstart_async.py:115
- The callback consumes
remainingbefore checking its sequence against_playback_base. Afterskip_pending_audio()advances the base during barge-in, bytes left from the old response are still played on the next callback, so interruption does not actually drop all pending audio. Track the sequence associated withremainingand clear it before consuming stale bytes.
sdk/voiceagents/azure-ai-voiceagents/samples/live/sample_live_audio_conversation_async.py:153 - The callback consumes
remainingbefore checking its sequence against_playback_base. After barge-in advances the base, bytes left from the previous response are still played on the next callback, contradicting the promised immediate drop of pending audio. Track the sequence associated withremainingand discard stale bytes before playback.
sdk/voiceagents/azure-ai-voiceagents/tests/unit/test_configuration.py:47 - This test supplies the same
v1value as the default, so it still passes if the override is ignored. Use a non-default sentinel to exercise the behavior named by the test.
sdk/voiceagents/azure-ai-voiceagents/azure/ai/voiceagents/aio/_realtime.py:522 - The hand-written realtime implementation has no deterministic unit coverage for frame serialization/deserialization, typed-event mapping and unknown-event fallback, close/error handling, or resource helpers; the live smoke test covers only one service path. Add mocked WebSocket/session tests for these public behaviors so regressions do not require a gated live environment to detect.
async def recv(self) -> ServerEvent:
Comment on lines
+80
to
+82
| self.voice_agent_web_socket = VoiceAgentWebSocketOperations( | ||
| self._client, self._config, self._serialize, self._deserialize | ||
| ) |
Comment on lines
+81
to
+83
| self.voice_agent_web_socket = VoiceAgentWebSocketOperations( | ||
| self._client, self._config, self._serialize, self._deserialize | ||
| ) |
Contributor
[Pilot] PR Pipeline Failure AnalysisA CI pipeline failed on this pull request. Here is an automated analysis of what went wrong and how to get the build green. What failedThree categories of failures were found in the
Recommended next steps
Raw pipeline analysis (azsdk ci analyze)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Please add an informative description that covers that changes made by the pull request and link all relevant issues.
If an SDK is being regenerated based on a new API spec, a link to the pull request containing these API spec changes should be included above.
All SDK Contribution checklist:
General Guidelines and Best Practices
Testing Guidelines