Migrate dependency management from Pipenv to uv - #5417
Draft
letitz wants to merge 11 commits into
Draft
Conversation
letitz
commented
Aug 10, 2026
letitz
left a comment
letitz
left a comment
Collaborator
Author
There was a problem hiding this comment.
First pass at self-review.
| # Make pip3.11 the default so that pip install works. | ||
| mv /usr/local/bin/pip3.11 /usr/local/bin/pip && \ | ||
| python3.11 -m pipenv install --deploy --system | ||
| uv pip install --system --upgrade --no-deps -e /data |
Collaborator
Author
There was a problem hiding this comment.
Why --upgrade? And why --no-deps?
Collaborator
Author
There was a problem hiding this comment.
Add a comment explaining this change.
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 10, 2026
- pyproject.toml: Removed duplicate root dependencies already in vendor-core (cryptography, future, google-cloud-ndb, protobuf, psutil, crcmod); added conditional pywinauto dependency for win32 under dev group. - windows-init.ps1: Reverted redundant PATH modification (pipx ensurepath handles PATH); updated uv sync to use --frozen. - Dockerfiles: Removed redundant --upgrade and --no-deps flags from uv pip install -e /data. - local/install_deps_linux.bash: Made optional gcloud apt packages conditional on rodete and added b/414408644 and b/484368884 references. - local/install_deps_macos.bash: Removed redundant pipx check (handled by Brewfile); added --frozen to uv sync. - local/install_python_deps_linux.bash: Removed extra source .venv/bin/activate call after nodeenv. - src/local/butler/py_unittest.py: Removed unnecessary site.addsitedir call. - CI/Bot setup scripts: Added --frozen flag to all automated uv sync invocations. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
letitz
commented
Aug 10, 2026
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 10, 2026
- pyproject.toml: Moved setuptools==80.9.0 to [dependency-groups] dev. - local/tests/ci_tests.bash & setup scripts: Reverted --frozen flag from uv sync. - local/install_deps_linux.bash: Removed || true from apt-get install on rodete. - Dockerfile: Added explanatory comment for uv and python3.11 symlinks. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
letitz
commented
Aug 10, 2026
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 10, 2026
- local/README.md, AGENTS.md, kubernetes_e2e_test.bash, devcontainer.json: Removed redundant explicit pipx/uv installation steps as local/install_deps.bash handles uv setup. - pyproject.toml: Removed redundant packages from appengine optional dependencies group that are already included in core. Updated comment on platform-specific extras to clarify vendoring behavior. - uv.lock: Updated lockfile. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
letitz
force-pushed
the
uv
branch
2 times, most recently
from
August 10, 2026 20:38
98c2257 to
735f644
Compare
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- pyproject.toml: Removed duplicate root dependencies already in vendor-core (cryptography, future, google-cloud-ndb, protobuf, psutil, crcmod); added conditional pywinauto dependency for win32 under dev group. - windows-init.ps1: Reverted redundant PATH modification (pipx ensurepath handles PATH); updated uv sync to use --frozen. - Dockerfiles: Removed redundant --upgrade and --no-deps flags from uv pip install -e /data. - local/install_deps_linux.bash: Made optional gcloud apt packages conditional on rodete and added b/414408644 and b/484368884 references. - local/install_deps_macos.bash: Removed redundant pipx check (handled by Brewfile); added --frozen to uv sync. - local/install_python_deps_linux.bash: Removed extra source .venv/bin/activate call after nodeenv. - src/local/butler/py_unittest.py: Removed unnecessary site.addsitedir call. - CI/Bot setup scripts: Added --frozen flag to all automated uv sync invocations. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- pyproject.toml: Moved setuptools==80.9.0 to [dependency-groups] dev. - local/tests/ci_tests.bash & setup scripts: Reverted --frozen flag from uv sync. - local/install_deps_linux.bash: Removed || true from apt-get install on rodete. - Dockerfile: Added explanatory comment for uv and python3.11 symlinks. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- local/README.md, AGENTS.md, kubernetes_e2e_test.bash, devcontainer.json: Removed redundant explicit pipx/uv installation steps as local/install_deps.bash handles uv setup. - pyproject.toml: Removed redundant packages from appengine optional dependencies group that are already included in core. Updated comment on platform-specific extras to clarify vendoring behavior. - uv.lock: Updated lockfile. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- pyproject.toml: Removed duplicate root dependencies already in vendor-core (cryptography, future, google-cloud-ndb, protobuf, psutil, crcmod); added conditional pywinauto dependency for win32 under dev group. - windows-init.ps1: Reverted redundant PATH modification (pipx ensurepath handles PATH); updated uv sync to use --frozen. - Dockerfiles: Removed redundant --upgrade and --no-deps flags from uv pip install -e /data. - local/install_deps_linux.bash: Made optional gcloud apt packages conditional on rodete and added b/414408644 and b/484368884 references. - local/install_deps_macos.bash: Removed redundant pipx check (handled by Brewfile); added --frozen to uv sync. - local/install_python_deps_linux.bash: Removed extra source .venv/bin/activate call after nodeenv. - src/local/butler/py_unittest.py: Removed unnecessary site.addsitedir call. - CI/Bot setup scripts: Added --frozen flag to all automated uv sync invocations. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- pyproject.toml: Moved setuptools==80.9.0 to [dependency-groups] dev. - local/tests/ci_tests.bash & setup scripts: Reverted --frozen flag from uv sync. - local/install_deps_linux.bash: Removed || true from apt-get install on rodete. - Dockerfile: Added explanatory comment for uv and python3.11 symlinks. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- local/README.md, AGENTS.md, kubernetes_e2e_test.bash, devcontainer.json: Removed redundant explicit pipx/uv installation steps as local/install_deps.bash handles uv setup. - pyproject.toml: Removed redundant packages from appengine optional dependencies group that are already included in core. Updated comment on platform-specific extras to clarify vendoring behavior. - uv.lock: Updated lockfile. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- pyproject.toml: Removed duplicate root dependencies already in vendor-core (cryptography, future, google-cloud-ndb, protobuf, psutil, crcmod); added conditional pywinauto dependency for win32 under dev group. - windows-init.ps1: Reverted redundant PATH modification (pipx ensurepath handles PATH); updated uv sync to use --frozen. - Dockerfiles: Removed redundant --upgrade and --no-deps flags from uv pip install -e /data. - local/install_deps_linux.bash: Made optional gcloud apt packages conditional on rodete and added b/414408644 and b/484368884 references. - local/install_deps_macos.bash: Removed redundant pipx check (handled by Brewfile); added --frozen to uv sync. - local/install_python_deps_linux.bash: Removed extra source .venv/bin/activate call after nodeenv. - src/local/butler/py_unittest.py: Removed unnecessary site.addsitedir call. - CI/Bot setup scripts: Added --frozen flag to all automated uv sync invocations. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- pyproject.toml: Moved setuptools==80.9.0 to [dependency-groups] dev. - local/tests/ci_tests.bash & setup scripts: Reverted --frozen flag from uv sync. - local/install_deps_linux.bash: Removed || true from apt-get install on rodete. - Dockerfile: Added explanatory comment for uv and python3.11 symlinks. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- local/README.md, AGENTS.md, kubernetes_e2e_test.bash, devcontainer.json: Removed redundant explicit pipx/uv installation steps as local/install_deps.bash handles uv setup. - pyproject.toml: Removed redundant packages from appengine optional dependencies group that are already included in core. Updated comment on platform-specific extras to clarify vendoring behavior. - uv.lock: Updated lockfile. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- pyproject.toml: Removed duplicate root dependencies already in vendor-core (cryptography, future, google-cloud-ndb, protobuf, psutil, crcmod); added conditional pywinauto dependency for win32 under dev group. - windows-init.ps1: Reverted redundant PATH modification (pipx ensurepath handles PATH); updated uv sync to use --frozen. - Dockerfiles: Removed redundant --upgrade and --no-deps flags from uv pip install -e /data. - local/install_deps_linux.bash: Made optional gcloud apt packages conditional on rodete and added b/414408644 and b/484368884 references. - local/install_deps_macos.bash: Removed redundant pipx check (handled by Brewfile); added --frozen to uv sync. - local/install_python_deps_linux.bash: Removed extra source .venv/bin/activate call after nodeenv. - src/local/butler/py_unittest.py: Removed unnecessary site.addsitedir call. - CI/Bot setup scripts: Added --frozen flag to all automated uv sync invocations. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- pyproject.toml: Moved setuptools==80.9.0 to [dependency-groups] dev. - local/tests/ci_tests.bash & setup scripts: Reverted --frozen flag from uv sync. - local/install_deps_linux.bash: Removed || true from apt-get install on rodete. - Dockerfile: Added explanatory comment for uv and python3.11 symlinks. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- local/README.md, AGENTS.md, kubernetes_e2e_test.bash, devcontainer.json: Removed redundant explicit pipx/uv installation steps as local/install_deps.bash handles uv setup. - pyproject.toml: Removed redundant packages from appengine optional dependencies group that are already included in core. Updated comment on platform-specific extras to clarify vendoring behavior. - uv.lock: Updated lockfile. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
…lation conditional on rodete When running local environment setup (./local/install_deps.bash) on internal Google Linux workstations (rodete), apt-get fails when attempting to install several optional App Engine and Pub/Sub component packages that are not provided in internal apt repositories. Furthermore, attempting to add the docker-ce repository triggers interactive glogin authentication even when docker is already installed on the system. Specifically: - google-cloud-cli-app-engine-python, google-cloud-cli-app-engine-python-extras, google-cloud-cli-app-engine-go, and google-cloud-cli-pubsub-emulator apt packages are optional/unavailable on rodete (see internal references b/414408644 and b/484368884). - glogin and glinux-add-repo docker-ce-rodete should be skipped if docker is already installed on the machine. - Fix pipenv CLI invocation in local/install_python_deps_linux.bash. This commit updates local/install_deps_linux.bash to check $distro_codename and whether docker is installed. On rodete, only google-cloud-cli and google-cloud-cli-datastore-emulator are installed via apt-get, while non-rodete distributions retain the full component package list. Additionally adds local/tests/install_deps_test_linux.bash to test and verify fresh checkout setup on Linux.
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- pyproject.toml: Removed duplicate root dependencies already in vendor-core (cryptography, future, google-cloud-ndb, protobuf, psutil, crcmod); added conditional pywinauto dependency for win32 under dev group. - windows-init.ps1: Reverted redundant PATH modification (pipx ensurepath handles PATH); updated uv sync to use --frozen. - Dockerfiles: Removed redundant --upgrade and --no-deps flags from uv pip install -e /data. - local/install_deps_linux.bash: Made optional gcloud apt packages conditional on rodete and added b/414408644 and b/484368884 references. - local/install_deps_macos.bash: Removed redundant pipx check (handled by Brewfile); added --frozen to uv sync. - local/install_python_deps_linux.bash: Removed extra source .venv/bin/activate call after nodeenv. - src/local/butler/py_unittest.py: Removed unnecessary site.addsitedir call. - CI/Bot setup scripts: Added --frozen flag to all automated uv sync invocations. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- pyproject.toml: Moved setuptools==80.9.0 to [dependency-groups] dev. - local/tests/ci_tests.bash & setup scripts: Reverted --frozen flag from uv sync. - local/install_deps_linux.bash: Removed || true from apt-get install on rodete. - Dockerfile: Added explanatory comment for uv and python3.11 symlinks. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
letitz
added a commit
to letitz/clusterfuzz
that referenced
this pull request
Aug 11, 2026
- local/README.md, AGENTS.md, kubernetes_e2e_test.bash, devcontainer.json: Removed redundant explicit pipx/uv installation steps as local/install_deps.bash handles uv setup. - pyproject.toml: Removed redundant packages from appengine optional dependencies group that are already included in core. Updated comment on platform-specific extras to clarify vendoring behavior. - uv.lock: Updated lockfile. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
This commit completes the migration of ClusterFuzz's Python dependency management system from Pipenv to uv. pipx is used to install uv locally. Architecture & Configs: - Created root pyproject.toml with 5 dependency groups, centralizing dependency definitions in a single file (from two Pipfiles) and removing the need for the hack where appengine packages were modelled as dev dependencies in src/Pipfile. - Added setuptools==80.9.0 to project dependencies for pkg_resources API support. - Pinned isort~=5.13.2 and pylint~=2.17.7 in dev dependencies to support line length options. - Shared pinned Google client dependencies in vendor-appengine to avoid version mismatches with vendor-core. - Deleted legacy Pipfile* and requirements.txt files. Butler & Runtime Improvements: - Updated _install_third_party in src/local/butler/common.py to use uv and avoid materializing requirements.txt into the repo, which are duplicative of `uv.lock` contents anyway. - Updated src/local/butler/py_unittest.py to preserve src/third_party in sys.path while adding src/appengine/third_party, ensuring both App Engine and core libraries are importable during App Engine unit tests. This fixes an issue where the previous setup assumed third-party packages were not only vendored into third_party directories, but also installed into the virtual environment. - Updated virtual environment guards in src/local/butler/guard.py and CASP CLI wrapper in cli/casp/src/casp/utils/container.py. Scripts, CI, Docker & Docs: - Updated setup scripts in local/ to install pipx and uv and activate .venv. - Updated CI workflows to do the same. - Updated Docker build scripts and Dockerfiles similarly. - Updated bot setup scripts (configs/test/bot/setup/*.bash, windows-init.ps1). - Updated developer and agent documentation. - Updated local/tests/install_deps_test_linux.bash to verify fresh checkout environment setup with uv (.venv, third_party vendoring, bower components, and uv run butler.py lint / py_unittest).
TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
- pyproject.toml: Removed duplicate root dependencies already in vendor-core (cryptography, future, google-cloud-ndb, protobuf, psutil, crcmod); added conditional pywinauto dependency for win32 under dev group. - windows-init.ps1: Reverted redundant PATH modification (pipx ensurepath handles PATH); updated uv sync to use --frozen. - Dockerfiles: Removed redundant --upgrade and --no-deps flags from uv pip install -e /data. - local/install_deps_linux.bash: Made optional gcloud apt packages conditional on rodete and added b/414408644 and b/484368884 references. - local/install_deps_macos.bash: Removed redundant pipx check (handled by Brewfile); added --frozen to uv sync. - local/install_python_deps_linux.bash: Removed extra source .venv/bin/activate call after nodeenv. - src/local/butler/py_unittest.py: Removed unnecessary site.addsitedir call. - CI/Bot setup scripts: Added --frozen flag to all automated uv sync invocations. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
- pyproject.toml: Moved setuptools==80.9.0 to [dependency-groups] dev. - local/tests/ci_tests.bash & setup scripts: Reverted --frozen flag from uv sync. - local/install_deps_linux.bash: Removed || true from apt-get install on rodete. - Dockerfile: Added explanatory comment for uv and python3.11 symlinks. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
- pyproject.toml: Moved vendor-core, vendor-appengine, and vendor-platform-specific to [project.optional-dependencies] extras: core, appengine, platform-specific. Added comment explaining special handling for platform-specific binary wheel downloads via pip download in common.py _install_platform_pip. - src/local/butler/common.py: Updated _install_third_party and _install_platform_pip to use uv export --extra. TAG=agy CONV=e3e8364e-3aff-4a79-9c78-17378c583b06
- local/README.md, AGENTS.md, kubernetes_e2e_test.bash, devcontainer.json: Removed redundant explicit pipx/uv installation steps as local/install_deps.bash handles uv setup. - pyproject.toml: Removed redundant packages from appengine optional dependencies group that are already included in core. Updated comment on platform-specific extras to clarify vendoring behavior. - uv.lock: Updated lockfile. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
…ests.bash TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
- Configured [[tool.uv.index]] with PyPI (https://pypi.org/simple) as the default index in pyproject.toml. - Regenerated uv.lock so all package sources and wheel URLs point to PyPI/files.pythonhosted.org rather than Google internal pkg.dev. TAG=agy CONV=1906a716-4a06-47e6-b2fb-fc2038c78ef6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This commit completes the migration of ClusterFuzz's Python dependency management system from Pipenv to uv. pipx is used to install uv locally.
Architecture & Configs:
Butler & Runtime Improvements:
uv.lockcontents anyway.Scripts, CI, Docker & Docs:
Add integration test for fresh checkout environment installation on Linux