Fix mlv2 test sleep gates and re-record cassettes#9934
Open
lavakumarrepala wants to merge 5 commits into
Open
Conversation
- Invert is_live sleep gates so sleeps run during live recording, not playback (test_feature_store_scenarios.py, test_workspace_scenarios.py, test_datastore_scenarios.py, test_deployment_template_scenarios.py). - Bump workspace mvnet test suffix to _mvnet2 to avoid soft-deleted conflict; bump live waits to 600s/300s. - Re-record affected cassettes against fresh resources.
❌Azure CLI Extensions Breaking Change Test
|
|
Hi @lavakumarrepala, |
Collaborator
|
mlv2 |
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR updates the Azure ML CLI v2 package/repo artifacts for a new release and refreshes scenario tests + VCR recordings to align with updated live-recording behavior.
Changes:
- Bump package version to 2.44.0 and update
azure-ai-mldependency. - Adjust scenario tests to wait only in live mode for async resource propagation, and add live-only registry setup/teardown for deployment-template scenarios.
- Refresh multiple test recordings to match new live runs / API responses.
Reviewed changes
Copilot reviewed 22 out of 24 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/machinelearningservices/setup.py | Bumps package version to 2.44.0. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_workspace_scenarios.py | Updates live-mode sleeps and workspace naming for managed network tests. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_feature_store_scenarios.py | Switches sleep condition to live-mode and changes resource group used by tests. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_deployment_template_scenarios.py | Adds live-only registry setup/teardown using subprocess to avoid VCR capture. |
| src/machinelearningservices/azext_mlv2/tests/latest/test_datastore_scenarios.py | Changes RG/WS used in command strings for datastore tests. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_datastore_file.yaml | Updates datastore-file VCR recording to new live response payloads/headers. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_datastore_blob_credentialless.yaml | Updates credentialless blob datastore recording. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_datastore_blob.yaml | Updates blob datastore recording. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_datastore_adlsgen2_credentialless.yaml | Updates credentialless ADLS Gen2 datastore recording. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_datastore_adlsgen2.yaml | Updates ADLS Gen2 datastore recording. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_datastore_adlsgen1_credentialless.yaml | Updates credentialless ADLS Gen1 datastore recording. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_datastore_adlsgen1.yaml | Updates ADLS Gen1 datastore recording. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_12_deployment_template_full_workflow_verification.yaml | Refreshes deployment-template full workflow recording (region/URIs/payloads). |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_10_deployment_template_update_multiple_tags.yaml | Refreshes deployment-template tag update recording. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_09_deployment_template_get_nonexistent.yaml | Refreshes “get nonexistent template” recording. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_08_deployment_template_list_specific.yaml | Refreshes list-specific deployment-template recording. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_06_deployment_template_restore.yaml | Refreshes restore deployment-template recording. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_04_deployment_template_update.yaml | Refreshes update deployment-template recording. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_03_deployment_template_get.yaml | Refreshes get deployment-template recording. |
| src/machinelearningservices/azext_mlv2/tests/latest/recordings/test_01_deployment_template_create.yaml | Refreshes create deployment-template recording. |
| src/machinelearningservices/azext_mlv2/manual/requirements.txt | Bumps azure-ai-ml pinned version. |
| src/machinelearningservices/CHANGELOG.rst | Adds a new 2.44.0 changelog header for 2026-06-11. |
Comment on lines
+59
to
+63
| result = subprocess.run( | ||
| ["az", "ml", "registry", "create", | ||
| "-g", cls.resource_group, "--file", registry_yaml_path], | ||
| capture_output=True, text=True, shell=True, | ||
| ) |
Comment on lines
+83
to
+87
| subprocess.run( | ||
| ["az", "ml", "registry", "delete", | ||
| "-g", cls.resource_group, "--name", cls.registry_name, "--yes"], | ||
| capture_output=True, text=True, shell=True, | ||
| ) |
Comment on lines
+18
to
+19
| def _is_live_mode() -> bool: | ||
| return os.environ.get("AZURE_TEST_RUN_LIVE", "").lower() == "true" |
| class FeatureStoreScenarioTest(MLBaseScenarioTest): | ||
| def test_featureStore(self) -> None: | ||
| self.kwargs["resourceGroup"] = "test-rg-000006" | ||
| self.kwargs["resourceGroup"] = "kchawla-rg" |
|
|
||
| def test_featurestore_managednetwork_provision(self) -> None: | ||
| self.kwargs["resourceGroup"] = "test-rg-000006" | ||
| self.kwargs["resourceGroup"] = "kchawla-rg" |
| if not suffix: | ||
| suffix = "" | ||
| return f"az ml datastore create --file ./src/machinelearningservices/azext_mlv2/tests/test_configs/datastore/{file_name} --name {{{var_name}}} {suffix} -g testrg -w testworkspace" | ||
| return f"az ml datastore create --file ./src/machinelearningservices/azext_mlv2/tests/test_configs/datastore/{file_name} --name {{{var_name}}} {suffix} -g kchawla-rg -w kchawla-ws" |
| from time import sleep | ||
|
|
||
| sleep(120) # This sleep is only required for fresh recording of cassette | ||
| sleep(600) # Wait for async create to propagate when recording live |
Comment on lines
+32
to
+34
| {"createdAt": "2026-06-11T23:40:51.9443397+00:00", "createdBy": "Lava Kumar | ||
| Repala", "createdByType": "User", "lastModifiedAt": "2026-06-11T23:40:52.2124838+00:00", | ||
| "lastModifiedBy": "Lava Kumar Repala", "lastModifiedByType": "User"}}' |
Member
|
/azp run |
|
Azure Pipelines successfully started running 2 pipeline(s). |
… subprocess args
- Revert hardcoded developer RG (kchawla-rg) and workspace (kchawla-ws) in
test_feature_store_scenarios.py and test_datastore_scenarios.py back to
the suite's standard test-rg-000006 / testrg / testworkspace values.
- Scrub user PII (createdBy / lastModifiedBy / userName values) from cassette
recordings to placeholders ("Test User", "Anonymous User").
- Replace shell=True with shell=False on subprocess.run calls that pass argv
as a list in test_deployment_template_scenarios.py registry create/delete
(shell=True with a list argv is silently broken on POSIX CI).
- Remove broken immutability check in _ml_deployment_template_update that
compared post-_to_dict() values and raised false ValueError on unchanged
fields. Mutation is already physically constrained to description/tags.
Member
|
/azp run |
|
Azure Pipelines successfully started running 2 pipeline(s). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This checklist is used to make sure that common guidelines for a pull request are followed.
Related command
General Guidelines
azdev style <YOUR_EXT>locally? (pip install azdevrequired)python scripts/ci/test_index.py -qlocally? (pip install wheel==0.30.0required)For new extensions:
About Extension Publish
There is a pipeline to automatically build, upload and publish extension wheels.
Once your pull request is merged into main branch, a new pull request will be created to update
src/index.jsonautomatically.You only need to update the version information in file setup.py and historical information in file HISTORY.rst in your PR but do not modify
src/index.json.