Skip to content

[docs] kernels #13139

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
stevhliu merged 2 commits into from
Mar 25, 2026
Merged

[docs] kernels #13139

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1f88f8f
kernels
stevhliu Feb 13, 2026
dd70a72
feedback
stevhliu Feb 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions docs/source/en/optimization/fp16.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -248,6 +248,24 @@ Refer to the [diffusers/benchmarks](https://huggingface.co/datasets/diffusers/be

The [diffusers-torchao](https://github.com/sayakpaul/diffusers-torchao#benchmarking-results) repository also contains benchmarking results for compiled versions of Flux and CogVideoX.

## Kernels

[Kernels](https://huggingface.co/docs/kernels/index) is a library for building, distributing, and loading optimized compute kernels on the [Hub](https://huggingface.co/kernels-community). It supports [attention](./attention_backends#set_attention_backend) kernels and custom CUDA kernels for operations like RMSNorm.

The [Diffusers Pipeline Integration](https://github.com/huggingface/kernels/blob/main/skills/cuda-kernels/references/diffusers-integration.md) guide shows how to integrate a kernel. Create a custom optimized attention processor, patch all modules in the model, and inject the kernel into the pipeline.
Copy link
Copy Markdown
Member

@sayakpaul sayakpaul Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The kernel skill basically lets users get an agent to write custom kernels for a model and hardware. It's not specific to the attention processor but also other modules as well such RMSNorm. Should we make it clearer?

Copy link
Copy Markdown
Member Author

@stevhliu stevhliu Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lmk if this is clearer!


> [!TIP]
> Install the [add cuda-kernels](https://github.com/huggingface/kernels/blob/main/skills/cuda-kernels/SKILL.md) skill to teach Claude or Codex how to write a kernel. The [Custom kernels for all from Codex and Claude](https://huggingface.co/blog/custom-cuda-kernels-agent-skills) blog post covers this in more detail.

For example, a custom RMSNorm kernel with [torch.compile](#torchcompile) speeds up LTX-Video generation 1.43x on an H100.
Copy link
Copy Markdown
Member

@sayakpaul sayakpaul Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It wasn't just RMSNorm but also other modules implemented with custom kernels.

Copy link
Copy Markdown
Member Author

@stevhliu stevhliu Feb 13, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i mention RMSNorm as an example only for the benchmark results below


<iframe
src="https://huggingface.co/datasets/docs-benchmarks/kernel-ltx-video/embed/viewer/default/train"
frameborder="0"
width="100%"
height="560px"
></iframe>

## Dynamic quantization

[Dynamic quantization](https://pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html) improves inference speed by reducing precision to enable faster math operations. This particular type of quantization determines how to scale the activations based on the data at runtime rather than using a fixed scaling factor. As a result, the scaling factor is more accurately aligned with the data.
Expand Down