Ship the quantized kernels as their own library

[shoumikhin]

A quantized model uses smaller numbers than a normal one, so the tensors take less memory.
Running one needs the quantized operator kernels.

The only copy the wheel shipped is the one torch loads to export a model, which a C++ application
cannot use. Such an application links the runtime, loads a quantized model, and the model fails at
run time with a missing operator, which looks like a model problem rather than a packaging one.

Build the quantized kernels as their own shared library and name it as a CMake component, the same
way the other kernel sets are named.

find_package(executorch REQUIRED COMPONENTS kernels_quantized)
target_link_libraries(my_app PRIVATE executorch::runtime
                                     executorch::kernels_quantized)

The wheel now ships lib/libexecutorch_kernels_quantized.so.

Note that the wheel also ships a second copy of these kernels, inside the library torch loads when
you export a model. That copy is built into the plugin rather than resolved from the shared library,
so a process holding both registers the same operators twice, and the runtime treats that as fatal:

Re-registering quantized_decomposed::add.out

This affects only a process that does both, for example an application that embeds a Python
interpreter. A plain C++ application can link the component freely.

Because of that, this is the one component EXECUTORCH_LIBRARIES does not include, so an
application that links whatever the package offers cannot end up in that position without asking.
A consumer that wants the quantized kernels names the component, or on CMake older than 3.28, where
no component targets exist, links EXECUTORCH_QUANTIZED_KERNELS_LIBRARY as well. That variable is
now populated on both CMake routes, so a consumer that adopts the older-CMake recipe and later
upgrades keeps the library on their link line instead of silently losing it.

Built the wheel, installed it into a clean environment, and:

• exported a quantized model and ran it from Python, matching eager PyTorch to within the
  quantization step (measured worst difference 0.0048 against a tolerance of 0.02).
• built a C++ application that links executorch::kernels_quantized, ran the same program, and got
  the same output as Python, byte for byte.
• confirmed the Python extension does not depend on the run-time copy, and that a process holding
  the shipped library and the export plugin aborts in either load order.
• checked every shipped library the same way, to establish that this is the only pair that
  collides: the CPU kernels, the delegate, the thread pool, the profiler and the runtime all
  coexist with both the extension and the export plugin.
• an application linking only EXECUTORCH_LIBRARIES does not depend on the quantized library while
  still depending on the CPU kernels, on CMake 3.28 and on real CMake 3.24. A new check asserts
  this, and it fails on the previous behaviour.
• EXECUTORCH_QUANTIZED_KERNELS_LIBRARY resolves to the shipped library on both the modern-CMake
  route (as the imported target) and the pre-3.28 route (as a file path).
• a missing quantized library now fails the checks instead of skipping them. The preset that builds
  the wheel enables these kernels unconditionally, so their absence is a regression rather than a
  configuration to tolerate, and both the ownership table and the C++ check previously treated it as
  an acceptable state and reported coverage they had not run.

Ran on Linux x86_64 and aarch64.

[shoumikhin]

Stack from ghstack (oldest at bottom):

• Name the Python extension after what it contains #21816
• Ship linkable libraries in the macOS wheel #21771
• Ship the OpenVINO delegate as its own library #21770
• Build and publish CUDA wheels #21668
• Ship the CUDA delegate in the wheel #21645
• -> Ship the quantized kernels as their own library #21642
• Ship a C++ SDK in the wheel #21639
• Split the runtime out of the Python bindings extension #21610

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21642

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 41 New Failures, 3 Unrelated Failures, 39 Unclassified Failures

As of commit 707e35f with merge base ed65b12 ():

NEW FAILURES - The following jobs have failed:

• Apple / build-benchmark-app / macos-job (gh)
  executorch
• Apple / build-demo-ios / macos-job (gh)
  executorch
• Apple / build-frameworks-ios / macos-job (gh)
  executorch
• Build Presets / apple (pybind) / build (gh)
  /Users/runner/work/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/sdk/program_path.h:103:84: error: use of typeid requires -frtti
• pull / test-qnn-delegate-linux / linux-job (gh)
  RuntimeError: Command docker exec -t b3003e5adcfcac49a441f11d202719c4b0456d2cc78cdb03bd268081cbb8bdd3 /exec failed with exit code 3
• pull / test-qnn-passes-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 97cec6bef67d5613e3e7085b7d8dc5736146c945d02862b09d7818ff8eb34e89 /exec failed with exit code 2
• pull / test-qnn-python-imports-linux / linux-job (gh)
  RuntimeError: Command docker exec -t a848ab5ad4df6966a57ca8aae3a39f7e1d0f95ea8961667a287fe33228928223 /exec failed with exit code 1
• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, models) / linux-job (gh)
  RuntimeError: Command docker exec -t 18e71d09ad7f5f9bd47bc739a3face9243ff0cf6e3106932f14e276dc9b96ac7 /exec failed with exit code 1
• pull / test-qnn-testsuite-linux / test-backend-linux (qnn, operators) / linux-job (gh)
  RuntimeError: Command docker exec -t 120333241af7a87446db023e41d6a6ddcbc91297eaf772a04df697ec8e06bc01 /exec failed with exit code 1
• pull / unittest / macos / macos-job (gh)
  executorch
• pull / unittest-buck / macos / macos-job (gh)
  executorch
• pull / unittest-editable / linux / linux-job (gh)
  backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe
• pull / unittest-editable / macos / macos-job (gh)
  /Users/ec2-user/runner/_work/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/sdk/program_path.h:103:84: error: use of typeid requires -frtti
• pull / unittest-nxp-neutron / linux-job (gh)
  RuntimeError: Command docker exec -t 13c71ba248aa03a2357b06d27db3ec26fdca55ff32453021fe4e58b6dbdb07d0 /exec failed with exit code 1
• Test CoreML Backend / test-coreml / test-backend-macos (coreml, models) / macos-job (gh)
  /Users/ec2-user/runner/_work/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/sdk/program_path.h:103:84: error: use of typeid requires -frtti
• Test CoreML Backend / test-coreml / test-backend-macos (coreml, operators) / macos-job (gh)
  /Users/ec2-user/runner/_work/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/sdk/program_path.h:103:84: error: use of typeid requires -frtti
• Test CUDA Builds / export-model-cuda-artifact (SocialLocalMobile, Qwen3.5-35B-A3B-HQQ-INT4, quantized-int4-tile-packed) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn_16a16w, models) / linux-job (gh)
  RuntimeError: Command docker exec -t 1a651646ec3859d8d0a14333f608e3c62540420ada2705362670643e1ff2057d /exec failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn_16a16w, operators) / linux-job (gh)
  RuntimeError: Command docker exec -t 8a238e80c77ac56fc8af54018a3180781582152bd5ddfe0e9e579ad79c62233d /exec failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn_16a4w_block, models) / linux-job (gh)
  RuntimeError: Command docker exec -t a567ddcd7aac159b9bc33db1751c7e8ec2da64927cdb44a4dbee796d328111c5 /exec failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn_16a4w_block, operators) / linux-job (gh)
  RuntimeError: Command docker exec -t 77aa2009557f3e1d2a0b39b4185f7ea11146f247d2c2230fafa6ee92e83f1257 /exec failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn_16a4w, models) / linux-job (gh)
  RuntimeError: Command docker exec -t 426d6584055ccf28ec6f024fe5ba608c0d8c6eefd6a2b73c38a002e0e277ba13 /exec failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn_16a4w, operators) / linux-job (gh)
  RuntimeError: Command docker exec -t 7d8b1c6ccbe8ae2dc69291bbff19a089439b039d500eba20982cc479be227316 /exec failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn_16a8w, models) / linux-job (gh)
  RuntimeError: Command docker exec -t c845dafdba596484aa0ce3e5b9017ea8283599ff57873257119388631c200b69 /exec failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn_16a8w, operators) / linux-job (gh)
  RuntimeError: Command docker exec -t 790b632f0024007ca803ecbe492dd7944d2588d2246cd9f185ac9a6a59282de7 /exec failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn_8a8w, models) / linux-job (gh)
  RuntimeError: Command docker exec -t 8b954a64d3f839ce47fbd380dedfb56b245b82502328dc1686bd240ee955c743 /exec failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn_8a8w, operators) / linux-job (gh)
  RuntimeError: Command docker exec -t 5dc275a270e9dcdfafa75c36d52fce2cecf09e9cb67df32c3fdc4a5e2afba179 /exec failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn, models) / linux-job (gh)
  RuntimeError: Command docker exec -t e0d7e72145dc65c2d7c0ce6a6899d4d127b80aec0e6d172ede5fbc56dcab93c1 /exec failed with exit code 1
• Test QNN Backend / test-qnn / test-backend-linux (qnn, operators) / linux-job (gh)
  RuntimeError: Command docker exec -t dbed8695a8830de000ed11a1b7df224a357a9f6a238226900d3d705eae650c19 /exec failed with exit code 1
• trunk / test-models-macos-coreml (edsr) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (efficient_sam) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (emformer_transcribe) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (ic3) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (ic4) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (mobilebert) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (mv2) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (mv3) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (vit) / macos-job (gh)
  executorch
• trunk / test-models-macos-coreml (w2l) / macos-job (gh)
  executorch
• trunk / test-models-macos-mps / macos-job (gh)
  executorch
• trunk / unittest-release / macos / macos-job (gh)
  executorch

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• Build macOS Wheels / pytorch/executorch / build-wheel-py3_10-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  /Users/runner/work/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/sdk/program_path.h:103:84: error: use of typeid requires -frtti
• Build macOS Wheels / pytorch/executorch / build-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  /Users/runner/work/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/sdk/program_path.h:103:84: error: use of typeid requires -frtti
• Build macOS Wheels / pytorch/executorch / build-wheel-py3_12-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  /Users/runner/work/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/sdk/program_path.h:103:84: error: use of typeid requires -frtti
• Build macOS Wheels / pytorch/executorch / build-wheel-py3_13-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  /Users/runner/work/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/sdk/program_path.h:103:84: error: use of typeid requires -frtti
• Build macOS Wheels / pytorch/executorch / build-wheel-py3_14-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  /Users/runner/work/executorch/executorch/pytorch/executorch/backends/apple/coreml/runtime/sdk/program_path.h:103:84: error: use of typeid requires -frtti
• Build macOS Wheels / pytorch/executorch / upload / upload-wheel-py3_10-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build macOS Wheels / pytorch/executorch / upload / upload-wheel-py3_11-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build macOS Wheels / pytorch/executorch / upload / upload-wheel-py3_12-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build macOS Wheels / pytorch/executorch / upload / upload-wheel-py3_13-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• Build macOS Wheels / pytorch/executorch / upload / upload-wheel-py3_14-cpu (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  Unable to download artifact(s): Artifact not found
• pull / test-coreml-bc-macos (macos-m1-stable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• pull / test-coreml-bc-macos (macos-m2-stable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-coreml-delegate / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-custom-ops-macos (cmake) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-huggingface-transformers-macos (bert|coreml_fp32_gpu|--quantize) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-huggingface-transformers-macos (distilbert|coreml_fp32_gpu|--quantize) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-huggingface-transformers-macos (llama3.2-1b|coreml_fp32_gpu|--quantize) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-huggingface-transformers-macos (olmo-1b|coreml_fp32_gpu|--quantize) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-huggingface-transformers-macos (qwen3-0.6b|coreml_fp32_gpu|--quantize) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-huggingface-transformers-macos (smollm2-135m|coreml_fp32_gpu|--quantize) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-llama-runner-mac (fp32, coreml) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-llama-runner-mac (fp32, mps) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-llama-runner-mac (fp32, xnnpack+custom+quantize_kv) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-llama-torchao-lowbit / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (efficient_sam, portable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (emformer_join, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (ic4, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (llama, portable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (llama2, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (llama3_2_vision_encoder, portable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (mobilebert, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (mv3, portable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (mv3, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (resnet50, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (vit, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-models-macos-cpu (w2l, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-multimodal-macos (gemma3-4b) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-selective-build-macos (cmake) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch
• trunk / test-static-llama-ane / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  executorch

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

• trunk / test-models-macos-coreml (dl3) / macos-job (gh) (detected as infra flaky with no log or failing log classifier)
• trunk / test-models-macos-coreml (emformer_join) / macos-job (gh) (detected as infra flaky with no log or failing log classifier)

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• trunk / test-models-macos-coreml (resnet50) / macos-job (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[digantdesai]

can we make this as clean as other .so files i.e. registered only once, and not part of the cpython extension lib?

[digantdesai]

yeah .. this.

[digantdesai]

nice!

[digantdesai]

ok with duplicate symbols but we should see if we can do something about it.

[shoumikhin]

can we make this as clean as other .so files i.e. registered only once, and not part of the
cpython extension lib?

Good question, and I dug into it rather than guessing. The short answer is that the duplicate is
real, but it is not in the cpython extension, and removing it is a bigger change than this PR.

Measured on the installed wheel, there are two separate libraries that both register the same
21 quantized operators:

kernels/quantized/libquantized_ops_aot_lib.so   21 quantized_decomposed:: entries
lib/libexecutorch_kernels_quantized.so          21 quantized_decomposed:: entries

They are not two copies of the same thing by accident. They serve different sides:

• libquantized_ops_aot_lib.so links torch and registers into the PyTorch dispatcher. It exists
  so quantized operators can run at export time, which is what executorch.kernels.quantized
  loads.
• libexecutorch_kernels_quantized.so is the runtime component this PR ships, registering into the
  ExecuTorch operator table for a C++ application.

So the kernels are already out of the cpython extension, which is the part your comment asks about.
What remains is that a single process which both imports the export-time plugin and links the
runtime component ends up registering the same operator names twice, and the runtime stops on the
second registration.

Making that "registered only once" means having the export-time plugin reuse the runtime component
instead of carrying its own kernels. That is a real cleanup and I think the right end state, but it
changes the AOT side, which no part of this stack touches, and it would need its own testing on the
export path. I would rather do it as a separate change than widen this one.

What I have done here is make the conflict impossible to hit by accident: the component is held out
of ${EXECUTORCH_LIBRARIES} so you never get it implicitly, and the docs say plainly that you name
it only when you want quantized operators without loading the Python plugin in the same process.