Ship the CUDA delegate in the wheel

[shoumikhin]

The CUDA delegate runs a model on an NVIDIA GPU. It is built into the Python extension, so only
Python can use it. A C++ application has no way to link it, and nothing else can reuse it either.

There is a sharing problem too. A program may use more than one GPU backend at once, and they need
to agree on which CUDA stream (the queue the GPU runs work on) the caller chose. If each backend
carries its own copy of that state, work queued through one is invisible to the other.

Ship the CUDA delegate and a small stream helper as their own shared libraries, and name both as
CMake components. The stream helper is shared so a process has exactly one copy of the caller's
stream choice, which is what lets two backends agree on it.

find_package(executorch REQUIRED COMPONENTS backend_cuda)
target_link_libraries(my_app PRIVATE executorch::runtime
                                     executorch::backend_cuda)

A CUDA wheel does not bundle the CUDA runtime. It declares it as a dependency, the way the PyTorch
CUDA wheels do, so one copy is shared with torch rather than shipping a second one:

executorch/lib/libexecutorch_backend_cuda.so     the delegate
executorch/lib/libexecutorch_extension_cuda.so   the stream helper
executorch/backends/cuda/libaoti_cuda_shims.so   the GPU device code

Each library records a relative path to where pip installs the CUDA runtime, so it resolves without
the caller setting a library search path and without depending on a toolkit being installed.

The declared set includes the runtime compiler, because a shipped library links it to build kernels
at run time. Each declared package also needs its own directory recorded, since that is where the
loader looks. On the CUDA 12 packaging the compiler installs into its own directory, and omitting it
left that library unable to find the compiler even though the package was installed. The CUDA 13
packaging puts every component in one directory, so the same gap does not appear there.

The stream helper's header no longer includes cuda_runtime.h, which the wheel does not publish. It
only ever uses a CUDA stream as an opaque handle, so it declares that handle itself, and a consumer
can compile against the wheel with no CUDA toolkit installed.

Built a CUDA wheel, installed it into a clean environment, and:

• ran a GPU model from Python on an NVIDIA GPU, with output identical to eager PyTorch on the
  same weights and inputs (largest absolute difference 0).
• built a C++ application against the installed wheel alone and ran the same model, matching the
  same reference.
• confirmed one library defines the stream state and the GPU shims, not several. Extracting them
  into every consumer put three copies in one wheel, and a stream selected through one was invisible
  to the others.
• confirmed no shipped library records a CUDA toolkit path from the build machine, and every library
  that links the CUDA runtime has a relative path to it.
• confirmed a CPU wheel ships none of the CUDA libraries and no CUDA-only header.
• a row that names a CUDA train is built with the CUDA option on rather than left to autodetection,
  so a builder without a matching toolkit fails while configuring. Before this, such a row produced a
  wheel tagged for CUDA, carrying no CUDA library, that still declared the CUDA runtime packages: it
  installed cleanly and then reported the backend as unregistered when a model ran.
• a row whose major CUDA version does not match the installed toolkit now fails the build. The
  declared packages and the loader paths come from the row while the binaries come from the
  toolkit, and nothing compared the two, so a cu126 row built against a 13.0 toolkit attached
  CUDA 12 metadata to binaries needing libcudart.so.13. An unrecognised train fails too, instead
  of silently reporting whatever the builder happened to have. Detection reads the toolkit major
  directly, so the guard fires on any mismatch rather than only on the three exact (major, minor)
  pairs the supported list carries; on those three pairs it behaved correctly before, and on every
  other minor it saw an empty detection and skipped the check.
• the row classifier and the packaging read the row the same way now, so both agree on what a row
  spelled with an unsupported minor means. The shell classifier reduces the row to digits and
  matches against SUPPORTED_CUDA_VERSIONS. Packaging did the same shape on the outer decision
  and then took only the first two digits when picking runtime packages, so cu125 classified as
  CPU on one side and declared CUDA 12 on the other. Packaging now matches on the same digits and
  raises loudly on an unsupported train instead.
• whether a row is a CUDA row is decided by asking if it names a supported train, rather than by
  listing the spellings that mean "no CUDA". Checked 16 row values including cpu-aarch64, rocm6.2
  and cu118; the previous list-based form was wrong on several, and each wrong answer made a
  non-CUDA wheel declare the CUDA runtime.
• the CUDA components are required when the wheel's own version says it is a CUDA wheel. They were
  optional unconditionally, so a wheel tagged +cu126 with no CUDA library at all passed every check.
• the stream helper ships under either name it can be built with. The shim layer records it as a
  dependency whenever CUDA is on, while packaging named only the shared-build spelling, so a
  non-shared build shipped a shim whose dependency resolved to nothing.
• the relative hops between shipped libraries are sized by how deep the library sits in the package.
  A fixed pair was correct at one depth only: measured over every shipped location, 6 of 12 hops
  landed on a directory that does not exist, and the hop from lib/ climbed out of the package
  entirely, where an unrelated library with a matching soname could satisfy the dependency first.
• the stream helper no longer links or includes the CUDA toolkit. It uses a stream only as an
  opaque handle and calls no CUDA function, so it needs no toolkit include and no libcudart link.

Also fixed in this commit:

• The pre-build classifier resolves the Python interpreter (python3 or python, whichever
  exists) instead of assuming one name, and no longer discards stderr. Builders disagree on the
  name: Linux and macOS provide python3, while the Windows builder runs inside a conda
  environment that provides only python. Assuming either name breaks the other platform, and
  treating the failure as "not a CUDA row" silently rebuilt a CUDA row as a CPU row.
• CU_VERSION=cpu pip install . is handled explicitly instead of running the CUDA-train parser
  over it, which previously turned cpu into pu through a character-set strip and reached the
  unsupported-train error.

Ran end to end on H100, A100 and Jetson Thor, covering compute capabilities 9.0, 8.0 and 11.0.

Known gap, not introduced here: the Python Runtime.load_program path allocates activation memory
on the host, so a program exported to keep activations on the GPU fails there. The supported Python
loader and the C++ path both work. This is upstream in the Python bindings, which this change does
not touch.

[shoumikhin]

Stack from ghstack (oldest at bottom):

• Name the Python extension after what it contains #21816
• Ship linkable libraries in the macOS wheel #21771
• Ship the OpenVINO delegate as its own library #21770
• Build and publish CUDA wheels #21668
• -> Ship the CUDA delegate in the wheel #21645
• Ship the quantized kernels as their own library #21642
• Ship a C++ SDK in the wheel #21639
• Split the runtime out of the Python bindings extension #21610

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21645

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 31 New Failures, 1 Unrelated Failure, 31 Unclassified Failures

As of commit 80c9986 with merge base ed65b12 ():

NEW FAILURES - The following jobs have failed:

• Apple / build-benchmark-app / macos-job (gh)
  executorch
• Apple / build-demo-ios / macos-job (gh)
  executorch
• Apple / build-frameworks-ios / macos-job (gh)
  executorch
• Build Presets / apple (pybind) / build (gh)
  /Users/runner/work/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/sdk/program_path.h:103:84: error: use of typeid requires -frtti
• pull / test-qnn-delegate-linux / linux-job (gh)
  RuntimeError: Command docker exec -t f9a241d3c89ff3dfa41927857e076c43cc42223216c12530235e2e1b6152a3b2 /exec failed with exit code 3
• pull / test-qnn-passes-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 6a0a061d3d53e667dfb68e139680718e7b18a69cded84f2b924af48bbd946e19 /exec failed with exit code 2
• pull / test-qnn-python-imports-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 2af763f425413b09ca84e3e97fbd7ce1a6b001614b568121bd58d8f8c6ba06fd /exec failed with exit code 1
• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, models) / linux-job (gh)
  RuntimeError: Command docker exec -t 12c8a6c76fbddb58e6c0b778ca818e9148311f40ce365da2464909482e48bd6f /exec failed with exit code 1
• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, operators) / linux-job (gh)
  RuntimeError: Command docker exec -t 0a6074b93c3c2cdf55cc635b32c015f54f6fab9e5324cb4311b60e9639cec2f1 /exec failed with exit code 1
• pull / unittest / macos / macos-job (gh)
  executorch
• pull / unittest-buck / macos / macos-job (gh)
  executorch
• pull / unittest-editable / linux / linux-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest-editable / macos / macos-job (gh)
  /Users/ec2-user/runner/_work/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/sdk/program_path.h:103:84: error: use of typeid requires -frtti
• pull / unittest-nxp-neutron / linux-job (gh)
  RuntimeError: Command docker exec -t 92ba00dd4c7797ccb9b9435e419b49fe3240a11f0714d612c4c4318645c41f10 /exec failed with exit code 1
• Test CUDA Builds / export-model-cuda-artifact (SocialLocalMobile, Qwen3.5-35B-A3B-HQQ-INT4, quantized-int4-tile-packed) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh)
  RuntimeError: Command docker exec -t f399651fcd343a80f9d7f6bdacae920655eba7bcbcf624e3f8198d1962bfbbdc /exec failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
  RuntimeError: Command docker exec -t c8c9e05f3f71d75550a4f2ab3324062fb8f8d56a4c5ae51beeeb65464c810d14 /exec failed with exit code 1
• trunk / test-models-macos-coreml (dl3) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (edsr) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (efficient_sam) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (emformer_join) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (emformer_transcribe) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (ic3) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (ic4) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (mobilebert) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (mv2) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (mv3) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (resnet50) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (w2l) / macos-job (gh)
  executorch
• trunk / test-models-macos-mps / macos-job (gh)
  executorch
• trunk / unittest-release / macos / macos-job (gh)
  executorch

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Build macOS Wheels / pytorch/executorch / build-wheel-py3_10-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  /Users/runner/work/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/sdk/program_path.h:103:84: error: use of typeid requires -frtti
• Build macOS Wheels / pytorch/executorch / upload / upload-wheel-py3_10-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• pull / test-coreml-bc-macos (macos-m1-stable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• pull / test-coreml-bc-macos (macos-m2-stable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-coreml-delegate / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-custom-ops-macos (cmake) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-huggingface-transformers-macos (bert|coreml_fp32_gpu|--quantize) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-huggingface-transformers-macos (distilbert|coreml_fp32_gpu|--quantize) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-huggingface-transformers-macos (llama3.2-1b|coreml_fp32_gpu|--quantize) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-huggingface-transformers-macos (olmo-1b|coreml_fp32_gpu|--quantize) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-huggingface-transformers-macos (qwen3-0.6b|coreml_fp32_gpu|--quantize) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-huggingface-transformers-macos (smollm2-135m|coreml_fp32_gpu|--quantize) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-llama-runner-mac (fp32, xnnpack+custom+quantize_kv) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-llama-torchao-lowbit / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (efficient_sam, portable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (emformer_join, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (ic4, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (llama, portable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (llama3_2_vision_encoder, portable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (mobilebert, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (mv3, portable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (mv3, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (resnet50, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (vit, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (w2l, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-multimodal-macos (gemma3-4b) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-selective-build-macos (cmake) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-static-llama-ane / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch

FLAKY - The following job failed but was likely due to flakiness present on trunk:

• trunk / test-models-macos-coreml (vit) / macos-job (gh) (detected as infra flaky with no log or failing log classifier)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[digantdesai]

is this the only thing we need from the toolkit

[digantdesai]

can we also check for our local cuda kernels?

[digantdesai]

I guess this is usable only when toolkit is installed?

[digantdesai]

Thanks, skimmed it, sorry going slow..

[shoumikhin]

is this the only thing we need from the toolkit

Yes, and I checked rather than assumed. Here is the evidence.

The header uses exactly one CUDA name. Grepping every CUDA identifier in the file gives only
cudaStream_t (plus CUstream_st inside the typedef itself). The whole public surface is two
lines:

EXECUTORCH_EXTENSION_CUDA_API std::optional<cudaStream_t> getCallerStream();
static_assert(std::is_trivially_copyable_v<std::optional<cudaStream_t>>);

The stream is only stored and handed back, never dereferenced, so an opaque handle is all the
interface needs. The .cpp uses only cudaStream_t too.

It compiles with no toolkit at all, which is the point of the change: the wheel ships this
header and a consumer may not have CUDA installed.

g++ -fsyntax-only -std=c++17 -I<package-root> use_caller_stream.cpp
(no output)

The declaration matches CUDA's own exactly, so there is no risk of a subtly different type.
CUDA declares it in driver_types.h as:

typedef __device_builtin__ struct CUstream_st *cudaStream_t;

__device_builtin__ expands to nothing for host compilation (crt/host_defines.h), so the
underlying type is struct CUstream_st*, which is what this header declares. Proved it with the
real CUDA headers present:

#include <cuda_runtime.h>
#include <executorch/extension/cuda/caller_stream.h>
static_assert(std::is_same_v<cudaStream_t, struct CUstream_st*>);

That compiles, so the two declarations are the same type rather than merely similar.

Both include orders work. Repeating a typedef is well formed in C++, so a consumer that also
includes <cuda_runtime.h> is fine either way:

#include <cuda_runtime.h> then caller_stream.h  -> OK
caller_stream.h then #include <cuda_runtime.h>  -> OK

If this interface ever needs to touch a stream rather than pass it along, for example to
synchronise or query it, that would need real toolkit declarations and the include should come
back at that point. Today it does not.

[shoumikhin]

Two more from this round.

can we also check for our local cuda kernels?

Good catch, and yes. I had named four kernel shims, but the wheel actually exports nine, so five
were shipping unchecked. Read from the real built library rather than from the source list:

$ nm -D --defined-only libaoti_cuda_shims.so | grep -o 'aoti_torch_cuda_[a-z0-9_]*' | sort -u
aoti_torch_cuda__weight_int4pack_mm
aoti_torch_cuda_int4_plain_mm
aoti_torch_cuda_int5_plain_mm
aoti_torch_cuda_int6_plain_mm
aoti_torch_cuda_int8_plain_mm
aoti_torch_cuda_rand
aoti_torch_cuda_randint_low_out
aoti_torch_cuda_sort_stable
aoti_torch_cuda_guard_set_index

The list now names all nine. This matters more than it looks: the build drops every .cu file when
it finds no working CUDA compiler and still produces the library from its .cpp sources, so a
partially built library is exactly what this row exists to catch. A sample of four could pass while
the int5, int6 and int8 paths were missing. Verified each of the nine is present in a real wheel
before adding it, so the list is not aspirational.

I guess this is usable only when toolkit is installed?

No, and this is worth spelling out because it is the interesting part of the design: a consumer can
link both CUDA components with no CUDA toolkit installed at all.

extension_cuda needs nothing from CUDA:

$ readelf -d libexecutorch_extension_cuda.so | grep NEEDED
(no CUDA entries)

backend_cuda does need the CUDA runtime, but pip supplies it and the library already knows where
to find it:

$ readelf -d libexecutorch_backend_cuda.so
NEEDED   libcudart.so.12
RUNPATH  $ORIGIN:$ORIGIN/../../nvidia/cuda_runtime/lib:...

nvidia/cuda_runtime/lib is the pip-installed nvidia-cuda-runtime-cu12 package that the CUDA
wheel declares as a dependency, so installing the wheel installs the runtime it needs.

Proved it end to end by building a real consumer with the toolkit hidden completely (empty
environment, PATH=/usr/bin:/bin, CUDA_HOME and CUDACXX unset):

find_package(executorch REQUIRED COMPONENTS backend_cuda extension_cuda)
target_link_libraries(app PRIVATE executorch::runtime executorch::kernels_optimized
                                  executorch::backend_cuda executorch::extension_cuda)

$ ./app
caller stream set: no

It configures, links and runs. A toolkit is only needed to compile your own .cu files, not to use
these components.

The one real constraint is the wheel, not the toolkit: a CPU wheel defines neither component, so a
consumer asking for one is told at configure time rather than at link time.