CI Test against the nightlies of our dependencies

[betatim]

Some of the projects that cuml depends on produce nightlies on a regular basis. These are used amongst PyData projects for regular testing (eg scikit-learn uses the numpy nightlies). We should take advantage of these nightlies as well to stay ahead of deprecations and other changes. This way we learn about them as they happen instead of close to the release date (when a RC might be available).

https://anaconda.org/scientific-python-nightly-wheels is where most packages upload nightlies. Instructions on how to use them.

We'd like to set up a job as part of the existing nightly of cuml that uses the wheels from the scientific-python-nightly-wheels index. Failures in this job should be reported, but probably shouldn't lead to the cuml nightly being counted as failed for the night (upstream packages do make mistakes or be incompatible with each other for a while).

Questions for the build team:

• what do you think of the plan of using the existing nightly infrastructure/setup?
• what is the best way of adding this to dependencies.yaml?
  • For example should we make a new matrix entry called nightly (right now there is "oldest", "latest" and "intermediate" (special case for scikit-learn))
  • or should we use a new file target that has the --extra-index-url lines added?

[jameslamb]

This sounds useful and I think it is totally doable with our existing tools / infrastructure!

We've been using those scientific-python-nightly-wheels nightlies to do exactly this in LightGBM for a while, and it's helped us to stay ahead of changes in scikit-learn and numpy:

• https://github.com/microsoft/LightGBM/blob/cfc145a526f85407e0d7b16e5b47d82bfef4f5c4/.ci/test-python-latest.sh
• https://github.com/microsoft/LightGBM/blob/cfc145a526f85407e0d7b16e5b47d82bfef4f5c4/.ci/pip-envs/requirements-latest.txt

what do you think of the plan of using the existing nightly infrastructure/setup?

You could add continue-on-error: true to make the job report

Yes I think this makes sense. You probably do not want to run this kind of thing on PR CI.

I recommend trying to do this just here in the cuML repo to prove out the concept and not trying to do something generic for all of RAPIDS. That could always come later.

The custom-job shared workflow would be a good candidate for this, so you don't have to worry about extending the DEPENDENCIES key in the wheel-testing matrix to more values or any of the matrix-filter stuff you had to do in #6988. (custom-job link)

Here's an example of that:

https://github.com/rapidsai/cudf/blob/a22534cb96339d4224b4afd352d717299faa98b8/.github/workflows/test.yaml#L140-L152

A diff similar to the following in dependencies.yaml in the cuML repo:

diff --git a/dependencies.yaml b/dependencies.yaml
index 63059cc01..8fb259c11 100644
--- a/dependencies.yaml
+++ b/dependencies.yaml
@@ -327,17 +327,34 @@ dependencies:
         packages:
           - joblib>=0.11
           - numba>=0.60.0,<0.62.0
-          - &numpy numpy>=1.23,<3.0
           - scipy>=1.11.0
           - packaging
           - rich
-          - &scikit_learn scikit-learn>=1.4
           - *treelite
+      - output_types: [requirements, pyproject]
+        packages:
+          - &numpy numpy>=1.23,<3.0
+          - &scikit_learn scikit-learn>=1.4
       - output_types: requirements
         packages:
           # pip recognizes the index as a global option for the requirements.txt file
           - --extra-index-url=https://pypi.nvidia.com
           - --extra-index-url=https://pypi.anaconda.org/rapidsai-wheels-nightly/simple
+    specific:
+      - output_types: [requirements]
+        matrices:
+          - matrix:
+              use-nightlies: "true"
+            packages:
+              - --extra-index-url https://pypi.anaconda.org/scientific-python-nightly-wheels/simple
+              # .dev0 specificier is important, to tell installers like 'pip' that it can pull pre-releases
+              # (it ignores them by default)
+              - numpy>=2.4.0.dev0
+              - scikit-learn>=1.8.dev0
+          - matrix:
+            packages:
+              - *numpy
+              - *scikit_learn
   cuda_version:
     specific:
       - output_types: conda

And you could do something like this to set up an environment in a script called by the CI workflow

source rapids-init-pip

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen "${RAPIDS_CUDA_VERSION}")"
CUML_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="cuml_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-github python)
LIBCUML_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="libcuml_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-github cpp)

rapids-dependency-file-generator \
    --output requirements \
    --file-key py_run_cuml \
    --matrix "use-nightlies=true;cuda_suffixed=true;cuda=13.0" \
| tee ./requirements.txt

python -m pip install \
  -r ./requirements.txt \
   "${LIBCUML_WHEELHOUSE}"/libcuml*.whl \
   "$(echo "${CUML_WHEELHOUSE}"/cuml*.whl)[test]"

requirements.txt (click me)

joblib>=0.11
libcuml-cu13==26.2.*,>=0.0.0a0
numba-cuda[cu13]>=0.19.1,<0.20.0
numba>=0.60.0,<0.62.0
numpy>=1.23,<3.0
numpy>=2.4.0.dev0
packaging
pylibraft-cu13==26.2.*,>=0.0.0a0
rich
rmm-cu13==26.2.*,>=0.0.0a0
scikit-learn>=1.4
scikit-learn>=1.8.dev0
scipy>=1.11.0
treelite>=4.6.1,<5.0.0

probably shouldn't lead to the cuml nightly being counted as failed for the night (upstream packages do make mistakes or be incompatible with each other for a while)

This is totally doable.

Something like GitHub's continue-on-error (docs) could work for that.