fix: pin CUDA_PATH to the CUDA 12.8 runtime wheel in CuPy CI

[hongyx11]

Problem

The self-hosted PyLops Testing with CuPy job has been failing on every run for the last few days (e.g. run 30707829236), dying in Test with pytest after ~35s with Process completed with exit code 2.

It's a pytest collection error, not a test failure:

ERROR collecting pytests/test_poststack.py
cupy.cuda.compiler.CompileException:
  .../cupy/_core/include/cupy/_cccl/libcudacxx/cuda/std/__floating_point/storage.h(137):
  error: incomplete type "__nv_fp8_e8m0" is not allowed
  ... 24 errors detected

Nothing is wrong with pylops or with test_poststack.py specifically — any CuPy kernel that pulls in CCCL headers fails to compile. test_poststack.py:31 is simply the first one to run, via cupyx.scipy.signal.filtfilt → convolve1d at import time.

Root cause

Mixed CUDA toolkit versions in the NVRTC include path. Dumping the real compile options from the failing kernel:

-I.../cupy/_core/include/cupy/_cccl/libcudacxx
-I.../cupy/_core/include
-I/usr/local/cuda/include                  <- CUDA 12.2, searched first
-I.../nvidia/cuda_runtime/include          <- CUDA 12.8, too late

• NVRTC is 12.8 (from the nvidia-cuda-nvrtc-cu12==12.8.93 wheel this workflow installs), so CuPy 14's bundled CCCL enables _CCCL_HAS_NVFP8_E8M0() — gated on _CCCL_CTK_AT_LEAST(12, 8) — and forward-declares __nv_fp8_e8m0.
• <cuda_fp8.h> resolves against the first matching -I, which is /usr/local/cuda/include. The system toolkit on the runner is CUDA 12.2.2, whose cuda_fp8.h predates __nv_fp8_e8m0 → incomplete type.

That -I/usr/local/cuda/include comes from cupy._environment.get_cuda_path(), which reads $CUDA_PATH ahead of everything else. The runner inherits CUDA_PATH=/usr/local/cuda from the shell it was launched in, so CuPy keeps pointing at the system 12.2 toolkit no matter what the workflow does.

This is an environment change rather than a dependency change — uv.lock has not moved (CuPy has been 14.0.1 throughout), and the job was green until the runner was re-registered.

Fix

The Configure CUDA NVVM from uv wheels step already rebuilds CUDA_HOME, LD_LIBRARY_PATH and NUMBA_CUDA_NVVM from the uv wheels — CUDA_PATH was the one that got missed. Pointing it at the 12.8 cuda_runtime wheel (which ships a complete include/ tree, __nv_fp8_e8m0 included) makes the header and NVRTC versions agree:

           echo "CUDA_HOME=$CUDA_HOME" >> "$GITHUB_ENV"
+          echo "CUDA_PATH=$CUDA_RUNTIME_ROOT" >> "$GITHUB_ENV"
           echo "CUDA_NVVM_LIB=$CUDA_HOME/nvvm/lib64" >> "$GITHUB_ENV"

$CUDA_RUNTIME_ROOT is already defined at the top of that step, so this needs no other changes.

Verification

Ran the job's exact command and environment on the self-hosted runner, with only CUDA_PATH corrected:

========= 2183 passed, 910 skipped, 486 warnings in 378.09s (0:06:18) ==========

Exit code 0, and the 6m18s runtime matches the last green runs of this workflow.

Worth noting separately: it would be more robust to also start the runner with a clean environment (env -u CUDA_PATH -u CUDA_HOME ./run.sh, or install it as a service via svc.sh) so host shell variables can't leak into jobs again. That's a runner-side change, not something this PR can do.

🤖 Generated with Claude Code

[codacy-production]

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}