Enable CFSClean network isolation for Python pipelines - #48483
Open
Daniel Jurek (danieljurek) wants to merge 9 commits into
Open
Enable CFSClean network isolation for Python pipelines#48483Daniel Jurek (danieljurek) wants to merge 9 commits into
Daniel Jurek (danieljurek) wants to merge 9 commits into
Conversation
The default PIP_EXTRA_INDEX_URL was https://pypi.python.org/simple, which is applied by set_envvar_defaults() across ~18 azpysdk checks. Every pip invocation in those checks therefore probed public PyPI alongside the CFS feed, producing the bulk of the CFSClean network isolation violations observed in the "python - pullrequest" pipeline. Point both the shared default and the virtualenv seeding script at the azure-sdk-for-python Azure Artifacts feed, which has an upstream to PyPI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6f9d7432-df2e-4c67-8838-bde213908a16
|
Azure Pipelines: Successfully started running 1 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Validation-only commit. The PR otherwise touches eng/ only, so the diff-driven package resolution produced TargetingString=null and every package check (apistub, pyright, verifytypes, mypy, whl/sdist) was skipped -- which made the network isolation result a false negative. Revert before merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6f9d7432-df2e-4c67-8838-bde213908a16
… targeting" This reverts commit c3a62f7.
This reverts commit 4f72fc1.
Daniel Jurek (danieljurek)
marked this pull request as ready for review
August 7, 2026 01:03
Daniel Jurek (danieljurek)
requested a review
from Mike Harder (mikeharder)
as a code owner
August 7, 2026 01:03
Daniel Jurek (danieljurek)
requested a review
from Ben Broderick Phillips (benbp)
as a code owner
August 7, 2026 01:03
|
Azure Pipelines: Successfully started running 1 pipeline(s). 10 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Routes Python package resolution through the CFS-backed Azure DevOps feed and enables CFSClean pipeline enforcement.
Changes:
- Replaces public PyPI extra-index defaults with the CFS feed.
- Enables
Permissive, CFSCleannetwork isolation.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
eng/tools/azure-sdk-tools/ci_tools/variables.py |
Updates the default pip extra index. |
eng/scripts/seed-virtualenv-wheels.ps1 |
Seeds virtualenv wheels through CFS. |
eng/pipelines/templates/stages/1es-redirect.yml |
Enables CFSClean enforcement. |
| "VIRTUALENV_PIP": "24.0", | ||
| "VIRTUALENV_SETUPTOOLS": "75.3.2", | ||
| "PIP_EXTRA_INDEX_URL": "https://pypi.python.org/simple", | ||
| "PIP_EXTRA_INDEX_URL": "https://pkgs.dev.azure.com/azure-sdk/public/_packaging/azure-sdk-for-python/pypi/simple/", |
Daniel Jurek (danieljurek)
changed the title
Repointing PIP_EXTRA_INDEX_URL at the CFS feed made it redundant rather than correct. azpysdk.main already defaults PIP_INDEX_URL and UV_DEFAULT_INDEX to CFS_INDEX_URL when unset (main.py:198-201), and use-python-version.yml does the same at pipeline level -- both using "set only if unset" so PipAuthenticate@1's authenticated URL wins in CI. Keeping an extra index pointed at that same feed had three downsides: it was unauthenticated, it made pip query the same index twice per resolution, and it leaked the CFS feed into `azpysdk --pypi` runs, which explicitly set PIP_INDEX_URL to pypi.org and are meant to resolve from PyPI only. seed-virtualenv-wheels.ps1 runs virtualenv --download outside azpysdk, so it does not inherit that defaulting. It now prefers an already-set PIP_INDEX_URL and only falls back to the public CFS feed, matching the pattern in use-python-version.yml. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6f9d7432-df2e-4c67-8838-bde213908a16
Validation-only. Without a package file in the diff the resolver yields TargetingString=null and every package check is skipped, which makes the network isolation result a false negative. Revert before merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6f9d7432-df2e-4c67-8838-bde213908a16
Validation-only. Without a package file in the diff the resolver yields
TargetingString=null and every package check is skipped, making the
network isolation result a false negative.
Targets three deliberately different shapes:
- azure-keyvault-secrets: ordinary data-plane client
- azure-core: root dependency, rebuilt/revalidated widely
- azure-storage-extensions: the repo's only C-extension package, so it
exercises the cibuildwheel path (native build containers, and the
bundled nuget.exe that reaches api.nuget.org) that the other two do not
Revert before merge.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 6f9d7432-df2e-4c67-8838-bde213908a16
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (1)
sdk/keyvault/azure-keyvault-secrets/README.md:380
- This package-only whitespace change contradicts the PR description's statement that the temporary targeting touches were reverted, and it leaves an unrelated package in the final change/targeting set. Please remove the added blank line so the PR remains scoped to the three network-isolation files.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.
Suppressed comments (3)
sdk/storage/azure-storage-extensions/README.md:51
- The PR description says these validation-only package touches were reverted, but this marker is still present and explicitly says to revert it before merge. Remove it so the unrelated package README is not changed by this infrastructure PR.
<!-- Trivial change to exercise CI network-isolation validation. Revert before merge. -->
sdk/keyvault/azure-keyvault-secrets/README.md:381
- The PR description says these validation-only package touches were reverted, but this marker is still present and explicitly says to revert it before merge. Remove it so the unrelated package README is not changed by this infrastructure PR.
<!-- Trivial change to exercise CI network-isolation validation. Revert before merge. -->
sdk/core/azure-core/README.md:281
- The PR description says these validation-only package touches were reverted, but this marker is still present and explicitly says to revert it before merge. Remove it so the unrelated package README is not changed by this infrastructure PR.
<!-- Trivial change to exercise CI network-isolation validation. Revert before merge. -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bring
python - pullrequest(and friends) into CFSClean network-isolation compliancePart of the S360/SFI network-isolation effort. This enables the CFSClean policy and fixes the two things that were violating it.
Changes
eng/pipelines/templates/stages/1es-redirect.ymlnetworkIsolationPolicy: Permissive→Permissive, CFSCleaneng/tools/azure-sdk-tools/ci_tools/variables.pyPIP_EXTRA_INDEX_URL→ ADO feed instead ofpypi.python.orgeng/scripts/seed-virtualenv-wheels.ps1Root cause
PipAuthenticate@1sets onlyPIP_INDEX_URL.DEFAULT_ENVIRONMENT_VARIABLESis applied viaos.environ.setdefault, so the hardcodedPIP_EXTRA_INDEX_URL=https://pypi.python.org/simplesurvived authentication and was appended to every dependency resolution in every isolated venv (apistub,pyright,verifytypes,mypy,sphinx, …).Smoking gun from a baseline run:
Two pre-existing local workarounds corroborate this —
ci_tools/scenario/generation.pypassesPIP_EXTRA_INDEX_URL="", andeng/common/pipelines/templates/steps/python-auth-dev-feed.ymlclears it outright. This fixes it at the source instead.Validation
Baseline of 14 recent def-7050 builds: 9/14 NOT COMPLIANT for CFSClean, concentrated in
AnalyzeandBuild Extended.pypi.orgpypi.python.orgfiles.pythonhosted.orgFinal run 6670611 — succeeded, CFSClean COMPLIANT 13/13, with real package targeting (
azure-core, azure-keyvault-secrets, azure-template), so package checks genuinely executed rather than being skipped.pyright / verifytypes under enforcement
pyright-pythonself-checkshttps://pypi.org/pypi/pyright/jsonon startup to warn about new versions. Under CFSClean that request is now blocked — verified harmless:mureq.get(..., timeout=1)caps the wait at 1s; a bareexcept Exceptionswallows it and returnsNone, so no warning is printed and nothing raises.Run Pyrightsucceeded in 8.5s,Run verifytypesin 35.3s.WARNING: there is a new pyright version availableline present in baseline logs is absent here, confirming the call was blocked and handled gracefully.registry.npmjs.orgis never contacted either.Optionally,
PYRIGHT_PYTHON_IGNORE_WARNINGS=1would skip the request entirely (~1s saved per check), but it is not required for correctness.Out of scope (still non-compliant under other policies)
www.powershellgallery.comfrompwsh(eng/common/scripts/Helpers/PSModule-Helpers.ps1). Lives ineng/common, so it must be fixed inAzure/azure-sdk-toolsand synced.intersphinx_mappingreaching readthedocs-family domains (doc/sphinx/conf.py).Neither affects CFSClean.