Build and publish CUDA wheels - #21668
Open
shoumikhin wants to merge 31 commits into
Open
Conversation
Contributor
Author
shoumikhin
requested review from
abhinaykukkadapu,
digantdesai,
kirklandsign,
larryliu0820,
psiddh and
rascani
as code owners
August 7, 2026 17:55
This was referenced Aug 7, 2026
meta-cla
Bot
added
the
CLA Signed
shoumikhin
added a commit
that referenced
this pull request
Aug 7, 2026
## Why
The wheel now ships a CUDA delegate, but no release builds one. Every wheel workflow says:
with-cuda: disabled
So a user who wants GPU support still clones the repository and builds from source, which is
the thing the shipped libraries were supposed to remove. Nothing publishes them because
nothing builds them.
## What this change does
Adds two workflows, one per architecture, that build CUDA wheels the same way the CPU ones
build the default wheel:
build-wheels-cuda-linux.yml x86_64
build-wheels-cuda-aarch64-linux.yml aarch64
Both call the same shared matrix generator the CPU rows already use, with CUDA turned on
instead of off, then narrow the result.
## Which rows get built, and why not all of them
The generator emits every CUDA version it knows about. Publishing all of them would ship
wheels for combinations nothing can verify, and a GPU wheel that installs and then cannot run
is worse than one that does not exist: the failure appears when a model runs, and it looks
like a model problem rather than a packaging one.
So a row is kept only when all three of these hold:
| | |
| --- | --- |
| a GPU exists that the row's device code covers | otherwise the wheel installs and dies at the first kernel launch |
| a PyTorch build is published for that CUDA version and architecture | otherwise the dependency cannot be satisfied |
| a machine is available to run a model before release | otherwise nothing checks it |
That leaves:
| architecture | CUDA | Python |
| --- | --- | --- |
| x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |
| aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |
The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on,
rather than by what is convenient to verify here. A delegate built against one CUDA version
needs an ExecuTorch wheel for that same version, and a missing version leaves that consumer
with nothing to pair with, which fails for whoever installs the pair rather than for the row
that omitted it. 13.2 is published for that reason even though no machine on hand can execute
it: the packaging properties are checked on every row, and running a model is a release-gate
step on hardware with the matching GPU.
Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated
reason in the example requirements, so a GPU row would inherit a known-broken build. The
free-threaded builds are excluded because the CUDA dependencies are not published for them.
A pull request builds one representative row rather than the whole matrix, because a full
matrix on every push costs hours for little added signal.
## Jetson devices
Jetson needs its own row: a JetPack container, one Python version, one CUDA version. It cannot
take a generic aarch64 wheel, because the generic builds carry no device code for its GPU
architecture and no portable fallback either.
That row is present in the filter but deliberately empty. Published PyTorch stopped shipping
device code for those GPUs after 2.8.0, so a Jetson row today would produce a wheel whose
PyTorch dependency cannot execute on the device. The lists are there to be filled in when that
changes.
## What to expect
Nothing changes for a CPU user. These are additional rows, and the existing workflows are
untouched.
| | before | after |
| --- | --- | --- |
| GPU support from an install | build from source | a published wheel |
| CUDA runtime | not shipped | declared as a dependency |
The build asks for the delegate explicitly rather than letting the build detect a toolkit. A
detected build is fine locally, but a release row states what it is producing, and a row that
silently produced a CPU wheel because the toolkit was missing would publish under a CUDA name.
The environment script fails early for the same reason: without it, packaging looks for CUDA
libraries that were never built and reports a confusing missing-file error minutes later.
Test plan:
A smoke test for the CUDA rows, checking what an artifact can be held to on a builder that has
no GPU:
- the CUDA libraries are actually in the wheel, so a row named for CUDA cannot ship without a
delegate
- the CUDA runtime is declared, so a user has something to resolve it from
- each CUDA library reaches its runtime through a relative path, not through the builder's
toolkit directory, which resolves on the builder and nowhere else
Then everything a CPU wheel is already held to: one owner per component, no build-tree search
paths, and a C++ application outside the wheel able to link what it ships.
Each CUDA check was run against a wheel that should fail it as well as one that should pass,
because a check that cannot fail is worse than no check. All three correctly reject a CPU
wheel and all three accept a CUDA wheel built from source, which ships both libraries, declares
the runtime, and carries a relative path to it.
One of them did not fail on a CPU wheel at first: it looped over libraries that were not there
and reported a pass, having inspected nothing. It now requires at least one to be present.
The filter was exercised against a matrix shaped like the generator's output: 18 rows narrowed
to 8, a pull request narrowed to 1, the aarch64 rows given the newer builder image, and an
empty result treated as a failure rather than passed through, since a workflow with no build
job reads as a green check for a build that never happened.
ghstack-source-id: 10f5862
ghstack-comment-id: 5220374521
Pull-Request: #21668
shoumikhin
added a commit
that referenced
this pull request
Aug 7, 2026
## The problem
The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says:
```
with-cuda: disabled
```
So a user who wants GPU support still clones the repository and builds from source, which is the
thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds
them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on.
## The change
Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel,
calling the same shared matrix generator with CUDA turned on, then narrowing the result.
```
.github/workflows/build-wheels-cuda-linux.yml x86_64
.github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64
```
| architecture | CUDA | Python |
| --- | --- | --- |
| x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |
| aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |
The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather
than by what is convenient to verify. A delegate built against one CUDA version needs an
ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing
to pair with, which fails for whoever installs the pair rather than for the row that omitted it.
Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason,
so a GPU row would inherit a known-broken build. A pull request builds one representative row
rather than the whole matrix.
Jetson devices need their own row, with a different container and a single Python and CUDA
version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty:
published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a
wheel whose PyTorch dependency cannot execute on the device.
## Before and after
```
BEFORE AFTER
pip install executorch pip install executorch
CPU only, always CPU by default
GPU wheels published per CUDA version
GPU support means cloning the a published wheel carries the delegate
repository and building
a delegate built for CUDA 13.2 every consumer CUDA version has a
has no ExecuTorch wheel to pair with matching ExecuTorch wheel
```
## Test plan
A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU:
- the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate
- the CUDA runtime is declared, so a user has something to resolve it from
- each CUDA library reaches its runtime through a relative path, not the builder's toolkit
directory, which resolves on the builder and nowhere else
Then everything a CPU wheel is already held to: one owner per component, no build-tree search
paths, and a C++ application outside the wheel able to link what it ships.
Each check was run against a wheel that should fail it as well as one that should pass, because a
check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel
built from source. One did not fail at first: it looped over libraries that were not there and
reported a pass having inspected nothing, so it now requires at least one to be present.
The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to
24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty
result treated as a failure rather than passed through, since a workflow with no build job reads as
a green check for a build that never happened.
ghstack-source-id: bdf89eb
ghstack-comment-id: 5220374521
Pull-Request: #21668
shoumikhin
added a commit
that referenced
this pull request
Aug 7, 2026
## The problem
The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says:
```
with-cuda: disabled
```
So a user who wants GPU support still clones the repository and builds from source, which is the
thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds
them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on.
## The change
Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel,
calling the same shared matrix generator with CUDA turned on, then narrowing the result.
```
.github/workflows/build-wheels-cuda-linux.yml x86_64
.github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64
```
| architecture | CUDA | Python |
| --- | --- | --- |
| x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |
| aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |
The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather
than by what is convenient to verify. A delegate built against one CUDA version needs an
ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing
to pair with, which fails for whoever installs the pair rather than for the row that omitted it.
Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason,
so a GPU row would inherit a known-broken build. A pull request builds one representative row
rather than the whole matrix.
Jetson devices need their own row, with a different container and a single Python and CUDA
version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty:
published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a
wheel whose PyTorch dependency cannot execute on the device.
## Before and after
```
BEFORE AFTER
pip install executorch pip install executorch
CPU only, always CPU by default
GPU wheels published per CUDA version
GPU support means cloning the a published wheel carries the delegate
repository and building
a delegate built for CUDA 13.2 every consumer CUDA version has a
has no ExecuTorch wheel to pair with matching ExecuTorch wheel
```
## Test plan
A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU:
- the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate
- the CUDA runtime is declared, so a user has something to resolve it from
- each CUDA library reaches its runtime through a relative path, not the builder's toolkit
directory, which resolves on the builder and nowhere else
Then everything a CPU wheel is already held to: one owner per component, no build-tree search
paths, and a C++ application outside the wheel able to link what it ships.
Each check was run against a wheel that should fail it as well as one that should pass, because a
check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel
built from source. One did not fail at first: it looped over libraries that were not there and
reported a pass having inspected nothing, so it now requires at least one to be present.
The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is
treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed
run rather than a clear error. The checks now assert the accepted spelling.
The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to
24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty
result treated as a failure rather than passed through, since a workflow with no build job reads as
a green check for a build that never happened.
ghstack-source-id: df65c37
ghstack-comment-id: 5220374521
Pull-Request: #21668
shoumikhin
added a commit
that referenced
this pull request
Aug 7, 2026
## The problem
The wheel can now ship a CUDA delegate, but no release builds one. Every wheel workflow says:
```
with-cuda: disabled
```
So a user who wants GPU support still clones the repository and builds from source, which is the
thing the shipped libraries were supposed to remove. Nothing publishes them because nothing builds
them. Anything that wants to depend on an ExecuTorch GPU wheel has nothing to depend on.
## The change
Adds two workflows that build CUDA wheels the same way the existing ones build the default wheel,
calling the same shared matrix generator with CUDA turned on, then narrowing the result.
```
.github/workflows/build-wheels-cuda-linux.yml x86_64
.github/workflows/build-wheels-cuda-aarch64-linux.yml aarch64
```
| architecture | CUDA | Python |
| --- | --- | --- |
| x86_64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |
| aarch64 | 12.6, 13.0, 13.2 | 3.10 to 3.13 |
The CUDA versions are chosen so every accelerator consumer can find a wheel to depend on, rather
than by what is convenient to verify. A delegate built against one CUDA version needs an
ExecuTorch wheel for that same version, and a missing version leaves that consumer with nothing
to pair with, which fails for whoever installs the pair rather than for the row that omitted it.
Python 3.14 is excluded because the current CPU rows already fail on it for an unrelated reason,
so a GPU row would inherit a known-broken build. A pull request builds one representative row
rather than the whole matrix.
Jetson devices need their own row, with a different container and a single Python and CUDA
version, because they cannot take a generic aarch64 wheel. That row is in the filter but empty:
published PyTorch stopped shipping device code for those GPUs, so a Jetson row today would build a
wheel whose PyTorch dependency cannot execute on the device.
## Before and after
```
BEFORE AFTER
pip install executorch pip install executorch
CPU only, always CPU by default
GPU wheels published per CUDA version
GPU support means cloning the a published wheel carries the delegate
repository and building
a delegate built for CUDA 13.2 every consumer CUDA version has a
has no ExecuTorch wheel to pair with matching ExecuTorch wheel
```
## Test plan
A smoke test for the CUDA rows, checking what an artifact can be held to on a builder with no GPU:
- the CUDA libraries are in the wheel, so a row named for CUDA cannot ship without a delegate
- the CUDA runtime is declared, so a user has something to resolve it from
- each CUDA library reaches its runtime through a relative path, not the builder's toolkit
directory, which resolves on the builder and nowhere else
Then everything a CPU wheel is already held to: one owner per component, no build-tree search
paths, and a C++ application outside the wheel able to link what it ships.
Each check was run against a wheel that should fail it as well as one that should pass, because a
check that cannot fail is worse than no check. All three reject a CPU wheel and accept a CUDA wheel
built from source. One did not fail at first: it looped over libraries that were not there and
reported a pass having inspected nothing, so it now requires at least one to be present.
The rows keep whichever builder image the release provides. Substituting a plain manylinux image
broke the shared build setup, which expects conda to be present in that image.
The matrix generator accepts "enable" and "disable" for its build-type inputs. Anything else is
treated as not enabled, so a plausible-looking "enabled" produced an empty matrix and a failed
run rather than a clear error. The checks now assert the accepted spelling.
The filter was exercised against a matrix shaped like the generator's output: 36 rows narrowed to
24, a pull request narrowed to 1, the aarch64 rows given the correct builder image, and an empty
result treated as a failure rather than passed through, since a workflow with no build job reads as
a green check for a build that never happened.
ghstack-source-id: 0f17009
ghstack-comment-id: 5220374521
Pull-Request: #21668
A quantized model uses smaller numbers than a normal one, so the tensors take less memory.
Running one needs the quantized operator kernels.
The only copy the wheel shipped is the one torch loads to export a model, which a C++ application
cannot use. Such an application links the runtime, loads a quantized model, and the model fails at
run time with a missing operator, which looks like a model problem rather than a packaging one.
Build the quantized kernels as their own shared library and name it as a CMake component, the same
way the other kernel sets are named.
```cmake
find_package(executorch REQUIRED COMPONENTS kernels_quantized)
target_link_libraries(my_app PRIVATE executorch::runtime
executorch::kernels_quantized)
```
The wheel now ships `lib/libexecutorch_kernels_quantized.so`.
Note that the wheel also ships a second copy of these kernels, inside the library torch loads when
you export a model. That copy is built into the plugin rather than resolved from the shared library,
so a process holding both registers the same operators twice, and the runtime treats that as fatal:
```
Re-registering quantized_decomposed::add.out
```
This affects only a process that does both, for example an application that embeds a Python
interpreter. A plain C++ application can link the component freely.
Because of that, this is the one component `EXECUTORCH_LIBRARIES` does not include, so an
application that links whatever the package offers cannot end up in that position without asking.
A consumer that wants the quantized kernels names the component, or on CMake older than 3.28, where
no component targets exist, links `EXECUTORCH_QUANTIZED_KERNELS_LIBRARY` as well. That variable is
now populated on both CMake routes, so a consumer that adopts the older-CMake recipe and later
upgrades keeps the library on their link line instead of silently losing it.
Built the wheel, installed it into a clean environment, and:
- exported a quantized model and ran it from Python, matching eager PyTorch to within the
quantization step (measured worst difference 0.0048 against a tolerance of 0.02).
- built a C++ application that links `executorch::kernels_quantized`, ran the same program, and got
the same output as Python, byte for byte.
- confirmed the Python extension does not depend on the run-time copy, and that a process holding
the shipped library and the export plugin aborts in either load order.
- checked every shipped library the same way, to establish that this is the only pair that
collides: the CPU kernels, the delegate, the thread pool, the profiler and the runtime all
coexist with both the extension and the export plugin.
- an application linking only `EXECUTORCH_LIBRARIES` does not depend on the quantized library while
still depending on the CPU kernels, on CMake 3.28 and on real CMake 3.24. A new check asserts
this, and it fails on the previous behaviour.
- `EXECUTORCH_QUANTIZED_KERNELS_LIBRARY` resolves to the shipped library on both the modern-CMake
route (as the imported target) and the pre-3.28 route (as a file path).
- a missing quantized library now fails the checks instead of skipping them. The preset that builds
the wheel enables these kernels unconditionally, so their absence is a regression rather than a
configuration to tolerate, and both the ownership table and the C++ check previously treated it as
an acceptable state and reported coverage they had not run.
Ran on Linux x86_64 and aarch64.
ghstack-source-id: d5aa850
ghstack-comment-id: 5217087046
Pull-Request: #21642
The CUDA delegate runs a model on an NVIDIA GPU. It is built into the Python extension, so only
Python can use it. A C++ application has no way to link it, and nothing else can reuse it either.
There is a sharing problem too. A program may use more than one GPU backend at once, and they need
to agree on which CUDA stream (the queue the GPU runs work on) the caller chose. If each backend
carries its own copy of that state, work queued through one is invisible to the other.
Ship the CUDA delegate and a small stream helper as their own shared libraries, and name both as
CMake components. The stream helper is shared so a process has exactly one copy of the caller's
stream choice, which is what lets two backends agree on it.
```cmake
find_package(executorch REQUIRED COMPONENTS backend_cuda)
target_link_libraries(my_app PRIVATE executorch::runtime
executorch::backend_cuda)
```
A CUDA wheel does not bundle the CUDA runtime. It declares it as a dependency, the way the PyTorch
CUDA wheels do, so one copy is shared with torch rather than shipping a second one:
```
executorch/lib/libexecutorch_backend_cuda.so the delegate
executorch/lib/libexecutorch_extension_cuda.so the stream helper
executorch/backends/cuda/libaoti_cuda_shims.so the GPU device code
```
Each library records a relative path to where pip installs the CUDA runtime, so it resolves without
the caller setting a library search path and without depending on a toolkit being installed.
The declared set includes the runtime compiler, because a shipped library links it to build kernels
at run time. Each declared package also needs its own directory recorded, since that is where the
loader looks. On the CUDA 12 packaging the compiler installs into its own directory, and omitting it
left that library unable to find the compiler even though the package was installed. The CUDA 13
packaging puts every component in one directory, so the same gap does not appear there.
The stream helper's header no longer includes `cuda_runtime.h`, which the wheel does not publish. It
only ever uses a CUDA stream as an opaque handle, so it declares that handle itself, and a consumer
can compile against the wheel with no CUDA toolkit installed.
Built a CUDA wheel, installed it into a clean environment, and:
- ran a GPU model from Python on an NVIDIA GPU, with output identical to eager PyTorch on the
same weights and inputs (largest absolute difference 0).
- built a C++ application against the installed wheel alone and ran the same model, matching the
same reference.
- confirmed one library defines the stream state and the GPU shims, not several. Extracting them
into every consumer put three copies in one wheel, and a stream selected through one was invisible
to the others.
- confirmed no shipped library records a CUDA toolkit path from the build machine, and every library
that links the CUDA runtime has a relative path to it.
- confirmed a CPU wheel ships none of the CUDA libraries and no CUDA-only header.
- a row that names a CUDA train is built with the CUDA option on rather than left to autodetection,
so a builder without a matching toolkit fails while configuring. Before this, such a row produced a
wheel tagged for CUDA, carrying no CUDA library, that still declared the CUDA runtime packages: it
installed cleanly and then reported the backend as unregistered when a model ran.
- a row whose major CUDA version does not match the installed toolkit now fails the build. The
declared packages and the loader paths come from the row while the binaries come from the
toolkit, and nothing compared the two, so a `cu126` row built against a 13.0 toolkit attached
CUDA 12 metadata to binaries needing `libcudart.so.13`. An unrecognised train fails too, instead
of silently reporting whatever the builder happened to have. Detection reads the toolkit major
directly, so the guard fires on any mismatch rather than only on the three exact `(major, minor)`
pairs the supported list carries; on those three pairs it behaved correctly before, and on every
other minor it saw an empty detection and skipped the check.
- the row classifier and the packaging read the row the same way now, so both agree on what a row
spelled with an unsupported minor means. The shell classifier reduces the row to digits and
matches against `SUPPORTED_CUDA_VERSIONS`. Packaging did the same shape on the outer decision
and then took only the first two digits when picking runtime packages, so `cu125` classified as
CPU on one side and declared CUDA 12 on the other. Packaging now matches on the same digits and
raises loudly on an unsupported train instead.
- whether a row is a CUDA row is decided by asking if it names a supported train, rather than by
listing the spellings that mean "no CUDA". Checked 16 row values including `cpu-aarch64`, `rocm6.2`
and `cu118`; the previous list-based form was wrong on several, and each wrong answer made a
non-CUDA wheel declare the CUDA runtime.
- the CUDA components are required when the wheel's own version says it is a CUDA wheel. They were
optional unconditionally, so a wheel tagged `+cu126` with no CUDA library at all passed every check.
- the stream helper ships under either name it can be built with. The shim layer records it as a
dependency whenever CUDA is on, while packaging named only the shared-build spelling, so a
non-shared build shipped a shim whose dependency resolved to nothing.
- the relative hops between shipped libraries are sized by how deep the library sits in the package.
A fixed pair was correct at one depth only: measured over every shipped location, 6 of 12 hops
landed on a directory that does not exist, and the hop from `lib/` climbed out of the package
entirely, where an unrelated library with a matching soname could satisfy the dependency first.
- the stream helper no longer links or includes the CUDA toolkit. It uses a stream only as an
opaque handle and calls no CUDA function, so it needs no toolkit include and no libcudart link.
Also fixed in this commit:
- The pre-build classifier resolves the Python interpreter (`python3` or `python`, whichever
exists) instead of assuming one name, and no longer discards stderr. Builders disagree on the
name: Linux and macOS provide `python3`, while the Windows builder runs inside a conda
environment that provides only `python`. Assuming either name breaks the other platform, and
treating the failure as "not a CUDA row" silently rebuilt a CUDA row as a CPU row.
- `CU_VERSION=cpu pip install .` is handled explicitly instead of running the CUDA-train parser
over it, which previously turned `cpu` into `pu` through a character-set strip and reached the
unsupported-train error.
Ran end to end on H100, A100 and Jetson Thor, covering compute capabilities 9.0, 8.0 and 11.0.
Known gap, not introduced here: the Python `Runtime.load_program` path allocates activation memory
on the host, so a program exported to keep activations on the GPU fails there. The supported Python
loader and the C++ path both work. This is upstream in the Python bindings, which this change does
not touch.
ghstack-source-id: 59fa23e
ghstack-comment-id: 5219161655
Pull-Request: #21645
## The problem The wheel can carry the CUDA delegate, but no published wheel contains one: there is no CUDA row in any workflow, so a GPU user has to build from source. ## The change Add the workflows that build and publish CUDA wheels for Linux x86_64 and aarch64, and a smoke test that checks each wheel from the artifact itself. The build machines for these rows have no GPU, so the smoke test does not execute a model; it verifies the CUDA libraries are present, that the declared runtime matches the wheel's CUDA version, that nothing resolves through the build machine's toolkit, and that the shipped device code covers every GPU architecture the row claims. ``` executorch-1.5.0-cp312-cp312-manylinux_2_28_x86_64.whl +cu130 ``` A release publishes CUDA 12.6, 13.0 and 13.2, for Python 3.10 through 3.13. A pull request builds a single row instead of all twelve, because a full matrix costs hours for little extra signal. Which GPU architectures each row compiles for is chosen per row rather than detected on the builder. Detecting it would produce a wheel carrying device code for whatever machine happened to build it, which installs fine and then fails at the first GPU call. The aarch64 CUDA 12.6 row also compiles for compute capability 8.7, which is an embedded module. Every other row lists only the architectures the published PyTorch build for that train covers, and by that rule 8.7 would be left out, because the generic aarch64 build of this train carries 8.0 and 9.0 only. It is included because this is the only row whose CUDA major version matches what that module's software release ships, and because this wheel declares no PyTorch dependency: a user there supplies the build that carries their architecture. Leaving 8.7 out does not protect them from a bad pairing, it only removes the device code they need. Without it, a model reaching one of the shipped optional operators, quantized matrix multiply, sort or random number generation, fails at the first launch on that device. Two guards keep a release honest: - if the shared matrix generator stops offering a combination this policy advertises, the step fails instead of quietly publishing fewer wheels. A missing job is otherwise a green check for a wheel that was never built. - if a row reaches the architecture list with no CUDA version, the build refuses rather than falling back to the builder's GPU. A TORCH_CUDA_ARCH_LIST that holds only named GPU families PyTorch accepts, such as "Hopper", now fails to configure instead of quietly leaving CMAKE_CUDA_ARCHITECTURES unset and taking the compiler default. The three named forms CMake itself understands, "native", "all" and "all-major", are rejected before this logic runs: torch resolves the list with its own bundled CUDA architecture module, which does not know those names and stops the configure. That is upstream behaviour, not something this change introduces or can work around, so a caller has to name architectures explicitly. Windows CUDA is deliberately absent. The separate shared libraries this wheel exists to ship are Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not link. ## Test plan - built the full release matrix, twelve wheels, and confirmed each one's contents match the row it claims: the CUDA libraries present, the CUDA runtime declared, and device code for every GPU architecture the row advertises. - ran a GPU model end to end from a CI-built wheel on three NVIDIA GPUs covering three device architectures, with output identical to eager PyTorch on each (largest absolute difference 0), and inspected the wheel for a fourth device it cannot execute on. - ran the matrix filter over generated inputs, including incomplete and malformed ones, and confirmed it refuses rather than publishing a partial release: a missing CUDA version, a missing python, or a python present on rows this policy does not build are each reported by name. - confirmed a CPU row still produces a CPU wheel on a builder that happens to have a CUDA toolkit installed. - the newest architecture also ships in its portable form, so a GPU newer than any in the row can still run by having the driver compile it at load time. Checked with `cuobjdump --list-ptx`, since `--list-elf` prints identical output whether or not the portable form is present. - every library that carries GPU device code covers the whole row on its own. - the declared CUDA packages are compared against the expected set in BOTH directions. A one-way comparison accepted a wheel that omitted required packages, and a name-suffix comparison accepted cross-train names because for CUDA 13 the suffix is empty. - the python axis is an allowlist, matching the CUDA axis. Testing only the disabled list let any python not on it through: a 3.9 row was emitted successfully. - `install_utils.py` is in both CUDA workflows' path filters. It owns the supported CUDA train list and the toolkit detection, so a change there previously ran no CUDA wheel job. - requesting the JetPack rows fails with its own reason instead of the generic empty-matrix message, since both of its lists are deliberately empty and no workflow asks for them. - torchao keeps its CUDA channel where that channel exists. Falling back to the plain nightly index was needed only on aarch64, where the CUDA channel publishes nothing, and doing it everywhere changed which torchao an x86_64 install resolves. - the CUDA smoke test now asserts the QnnBackend and OpenvinoBackend registrations that a CPU Linux row asserts. The CUDA build enables OpenVINO on every Linux architecture and downloads the QNN SDK on x86_64, so a CUDA wheel carries both backends; a previous premise that "a CUDA row is not built with them" was false, and dropping the checks meant those two backends were unverified on every CUDA wheel. Known gap: no automated job runs a CUDA model on real hardware before publication. Running a model on real hardware is a separate release-time step that a person owns today, not an automated job wired into these workflows. ghstack-source-id: 95d67c7 ghstack-comment-id: 5220374521 Pull-Request: #21668
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
The wheel can carry the CUDA delegate, but no published wheel contains one: there is no CUDA
row in any workflow, so a GPU user has to build from source.
The change
Add the workflows that build and publish CUDA wheels for Linux x86_64 and aarch64, and a smoke test
that checks each wheel from the artifact itself. The aarch64 rows are validated on machines with no
GPU, so for those the smoke test cannot execute a model; it verifies the CUDA libraries are present, that the
declared runtime matches the wheel's CUDA version, that nothing resolves through the build
machine's toolkit, and that the shipped device code covers every GPU architecture the row claims.
A release publishes CUDA 12.6, 13.0 and 13.2, for Python 3.10 through 3.13. A pull request builds a
single row instead of all twelve, because a full matrix costs hours for little extra signal.
Which GPU architectures each row compiles for is chosen per row rather than detected on the builder.
Detecting it would produce a wheel carrying device code for whatever machine happened to build it,
which installs fine and then fails at the first GPU call.
The aarch64 CUDA 12.6 row also compiles for compute capability 8.7, which is an embedded module.
Every other row lists only the architectures the published PyTorch build for that train covers, and
by that rule 8.7 would be left out, because the generic aarch64 build of this train carries 8.0 and
9.0 only. It is included because this is the only row whose CUDA major version matches what that
module's software release ships, and because this wheel declares no PyTorch dependency: a user there
supplies the build that carries their architecture. Leaving 8.7 out does not protect them from a bad
pairing, it only removes the device code they need. Without it, a model reaching one of the shipped
optional operators, quantized matrix multiply, sort or random number generation, fails at the first
launch on that device.
Two guards keep a release honest:
instead of quietly publishing fewer wheels. A missing job is otherwise a green check for a wheel
that was never built.
back to the builder's GPU. A TORCH_CUDA_ARCH_LIST that holds only named GPU families PyTorch
accepts, such as "Hopper", now fails to configure instead of quietly leaving
CMAKE_CUDA_ARCHITECTURES unset and taking the compiler default. The three named forms CMake
itself understands, "native", "all" and "all-major", are rejected before this logic runs:
torch resolves the list with its own bundled CUDA architecture module, which does not know
those names and stops the configure. That is upstream behaviour, not something this change
introduces or can work around, so a caller has to name architectures explicitly.
Windows CUDA is deliberately absent. The separate shared libraries this wheel exists to ship are
Linux only today, so a Windows CUDA wheel would carry a delegate a C++ application still could not
link.
Test plan
claims: the CUDA libraries present, the CUDA runtime declared, and device code for every GPU
architecture the row advertises.
architectures, with output identical to eager PyTorch on each (largest absolute difference 0),
and inspected the wheel for a fourth device it cannot execute on.
confirmed it refuses rather than publishing a partial release: a missing CUDA version, a missing
python, or a python present on rows this policy does not build are each reported by name.
installed.
still run by having the driver compile it at load time. Checked with
cuobjdump --list-ptx, since--list-elfprints identical output whether or not the portable form is present.comparison accepted a wheel that omitted required packages, and a name-suffix comparison accepted
cross-train names because for CUDA 13 the suffix is empty.
python not on it through: a 3.9 row was emitted successfully.
install_utils.pyis in both CUDA workflows' path filters. It owns the supported CUDA train listand the toolkit detection, so a change there previously ran no CUDA wheel job.
since both of its lists are deliberately empty and no workflow asks for them.
was needed only on aarch64, where the CUDA channel publishes nothing, and doing it everywhere
changed which torchao an x86_64 install resolves.
row asserts. The CUDA build enables OpenVINO on every Linux architecture and downloads the QNN
SDK on x86_64, so a CUDA wheel carries both backends; a previous premise that "a CUDA row is not
built with them" was false, and dropping the checks meant those two backends were unverified on
every CUDA wheel.
Known gap: no automated job runs a CUDA model on real hardware before publication. The x86_64 rows
are validated on a machine with a GPU, so a model-execution check is possible there and its absence
is a gap rather than a limit. Only the aarch64 rows have no accelerator to run one on. Running a
model on real hardware is a separate release-time step that a person owns today, not an automated
job wired into these workflows.