remove most reliance on pypi.nvidia.com in RAPIDS CI

[jameslamb]

Description

Many NVIDIA projects, including all of RAPIDS, publish Python sdists and wheels to an NVIDIA-maintained index, https://pypi.nvidia.com/

Today, most of RAPIDS and most of its dependencies also publish Python sdists and wheels to https://pypi.org/

Proposing that we try to eliminate unnecessary uses of pypi.nvidia.com in RAPIDS CI.

Benefits of this work

• improved CI stability
  • fewer network calls (no checking that index AND pypi.org is cached by our package proxy cache while pypi.nvidia.com is not)
  • faster environment solves
  • reduced risk of surprising environment solves
• release confidence
  • gives us a chance to detect "pip install cudf does not work with just pypi.org" types of problems in CI
• limits the number of sources of pip configuration when building RAPIDS (helpful for other build systems like RAPIDS DLFW builds)

Acceptance Criteria

• pypi.nvidia.com is not enabled in RAPIDS CI images by default
• 0 unnecessary uses of pypi.nvidia.com in RAPIDS projects' CI, build scripts, documentation, etc.

Approach

See results of this GitHub search: https://github.com/search?q=org%3Arapidsai+%22pypi.nvidia.com%22+AND+NOT+is%3Aarchived&type=code&p=1

Start by removing it from global pip configuration in CI images, to identify any places depending on that global configuration: https://github.com/rapidsai/ci-imgs/blob/28b8b5ac127bcf5cd6679cd8ee40a53ce7240421/context/pip.conf#L8 . Triggering all RAPIDS nightly CI (#238) may be one time-efficient way to test the effect of that.

For dependencies.yaml groups like this:

  depends_on_dask_cuda:
    common:
      ...
      - output_types: requirements
        packages:
          # pip recognizes the index as a global option for the requirements.txt file
          - --extra-index-url=https://pypi.nvidia.com
          - --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple
    specific:
      - output_types: [requirements, pyproject]
        matrices:
          - matrix:
              cuda: "12.*"
              cuda_suffixed: "true"
            packages:
              - dask-cuda[cu12]==26.2.*,>=0.0.0a0
          ...

(rapidsai/cudf - dependencies.yaml)

Remove that --extra-index-url pointing to pypi.nvidia.com.

For documentation sections like this:

# CUDA 13
pip install pylibraft-cu13 --extra-index-url=https://pypi.nvidia.com
pip install raft-dask-cu13 --extra-index-url=https://pypi.nvidia.com

(rapidsai/raft - README.md)

Remove that --extra-index-url if the project and all of its recursive dependencies can be installed from pypi.org.

Notes

Why not all repos?

Some RAPIDS libraries are not yet available on pypi.org as of this writing (for example, cugraph). Installing those with pip still requires pypi.nvidia.com.

That is also why we should not remove pypi.nvidia.com from the instructions at https://docs.rapids.ai/install/

Investigation

I think it's very likely that many RAPIDS repos do not need this pypi.nvidia.com. Looked at a recent wheel test CI job in cugraph, which ends up pulling in many RAPIDS libraries and their dependencies: https://github.com/rapidsai/cugraph/actions/runs/20773106650/job/59653038422

Only 18 packages were downloaded from pypi.nvidia.com:

Downloading https://pypi.nvidia.com/cuda-toolkit/cuda_toolkit-13.1.0-py2.py3-none-any.whl (2.4 kB)

Downloading https://pypi.nvidia.com/nvtx/nvtx-0.2.14-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (710 kB)

Downloading https://pypi.nvidia.com/nvidia-cuda-nvcc/nvidia_cuda_nvcc-13.1.80-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (34.7 MB)

Downloading https://pypi.nvidia.com/nvidia-cuda-nvrtc/nvidia_cuda_nvrtc-13.1.80-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (44.3 MB)

Downloading https://pypi.nvidia.com/nvidia-cufile/nvidia_cufile-1.16.0.49-py3-none-manylinux_2_27_aarch64.whl (1.1 MB)

Downloading https://pypi.nvidia.com/nvidia-nvjitlink/nvidia_nvjitlink-13.1.80-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (38.9 MB)

Downloading https://pypi.nvidia.com/nvidia-nvvm/nvidia_nvvm-13.1.80-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (61.0 MB)

Downloading https://pypi.nvidia.com/nvidia-libnvcomp-cu13/nvidia_libnvcomp_cu13-5.1.0.21-py3-none-manylinux_2_28_aarch64.whl (19.3 MB)

Downloading https://pypi.nvidia.com/nvidia-nccl-cu13/nvidia_nccl_cu13-2.29.2-py3-none-manylinux_2_18_aarch64.whl (201.0 MB)

Downloading https://pypi.nvidia.com/nvidia-cublas/nvidia_cublas-13.2.0.9-py3-none-manylinux_2_27_aarch64.whl (540.5 MB)

Downloading https://pypi.nvidia.com/nvidia-curand/nvidia_curand-10.4.1.34-py3-none-manylinux_2_27_aarch64.whl (62.4 MB)

Downloading https://pypi.nvidia.com/nvidia-cusolver/nvidia_cusolver-12.0.7.41-py3-none-manylinux_2_27_aarch64.whl (224.3 MB)

Downloading https://pypi.nvidia.com/nvidia-cusparse/nvidia_cusparse-12.7.2.19-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (168.6 MB)

Downloading https://pypi.nvidia.com/nvidia-cuda-runtime/nvidia_cuda_runtime-13.1.80-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (2.3 MB)

Downloading https://pypi.nvidia.com/nvidia-cuda-cccl/nvidia_cuda_cccl-13.1.78-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (3.5 MB)

Downloading https://pypi.nvidia.com/nvidia-cuda-crt/nvidia_cuda_crt-13.1.80-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (132 kB)

Downloading https://pypi.nvidia.com/libucx-cu13/libucx_cu13-1.19.0-py3-none-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl (27.0 MB)

I checked and every one of these packages, same versions and contents, are available on pypi.org. This environment could have been solved without pypi.nvidia.com, and there isn't any correctness or security reason that pip choose pypi.nvidia.com (it does not offer index priority, for example).

Task Lists

• add code to gha-tools for removing pypi.nvidia.com from the global pip config file (rapids-init-pip: add a migrator for removing pypi.nvidia.com gha-tools#241)
• update library repos
  • cucim (no changes required)
  • cudf (tighten wheel size limits, expand CI-skipping logic, other small build changes cudf#21220)
  • cugraph (tighten wheel size limits, expand CI-skipping logic, other small build changes cugraph#5413)
  • cugraph-gnn (tighten wheel size limits, expand CI-skipping logic, other small build changes cugraph-gnn#396)
  • cuml (tighten wheel size limits, expand CI-skipping logic, other small build changes cuml#7738)
  • cuopt (tighten wheel size limits, expand CI-skipping logic, other small build changes NVIDIA/cuopt#804
  • cuvs (tighten wheel size limits, expand CI-skipping logic, other small build changes cuvs#1751)
  • cuxfilter (tighten wheel size limits, expand CI-skipping logic, other small build changes cuxfilter#763
  • dask-cuda (no changes required)
  • jupyterlab-nvdashboard (no changes required)
  • kvikio (tighten wheel size limits, expand CI-skipping logic, other small build changes kvikio#911)
  • nx-cugraph (tighten wheel size limits, expand CI-skipping logic, other small build changes nx-cugraph#237)
  • raft (expand CI-skipping logic, other small build changes raft#2933)
  • rapids-build-backend (no changes required)
  • rapids-cli (no changes required)
  • rapids-dask-dependency (no changes required)
  • rapids-dependency-file-generator (no changes required)
  • rapids-logger (enforce wheel size limits, PEP 639 metadata rapids-logger#65)
  • rapids-metadata (no changes required)
  • rapids-reviser (no changes required)
  • rapidsmpf (tighten wheel size limits, expand CI-skipping logic, other small build changes rapidsmpf#830)
  • rmm (tighten wheel size limits, expand CI-skipping logic, other small build changes rmm#2236)
  • ucxx (tighten wheel size limits, expand CI-skipping logic, other small build changes ucxx#580)
  • ucx-wheels (no changes required)
  • (private repos)
• remove any other unnecessary references (GitHub search)
• remove pypi.nvidia.com in CI images (remove pypi.nvidia.com from default pip configuration ci-imgs#357)
• remove gha-tools migrator use and other workarounds in library repos
  • cucim (remove pip.conf migration code in CI scripts cucim#1023)
  • cudf (remove pip.conf migration code in CI scripts, update CI-skipping rules cudf#21303)
  • cugraph (remove pip.conf migration code in CI scripts, update CI-skipping rules cugraph#5415)
  • cugraph-gnn (remove pip.conf migration code in CI scripts, update CI-skipping rules cugraph-gnn#399)
  • cuml (remove pip.conf migration code in CI scripts, update CI-skipping rules cuml#7747)
  • cuopt (remove pip.conf migration code in CI scripts, fix pr-builder configuration NVIDIA/cuopt#821)
  • cuvs (remove pip.conf migration code in CI scripts, update CI-skipping rules cuvs#1760)
  • cuxfilter (remove pip.conf migration code in CI scripts cuxfilter#764)
  • dask-cuda (remove pip.conf migration code in CI scripts dask-cuda#1622)
  • jupyterlab-nvdashboard (remove pip.conf migration code in CI scripts jupyterlab-nvdashboard#252)
  • kvikio (remove pip.conf migration code in CI scripts, update CI-skipping rules kvikio#918)
  • nx-cugraph (remove pip.conf migration code in CI scripts nx-cugraph#239)
  • raft (remove pip.conf migration code in CI scripts, update CI-skipping rules raft#2938)
  • rapids-logger (remove pip.conf migration code in CI scripts rapids-logger#66
  • rapidsmpf (remove pip.conf migration code in CI scripts, update CI-skipping rules rapidsmpf#839)
  • rmm (remove pip.conf migration code in CI scripts, update CI-skipping rules rmm#2242)
  • ucxx (remove pip.conf migration code in CI scripts, update CI-skipping rules ucxx#583)
  • (private repos)
• revert migrator in gha-tools (Revert "rapids-init-pip: add a migrator for removing pypi.nvidia.com (#241)" gha-tools#242)