docs: add AutoQuantize mixed-precision search blog#1979
Conversation
Signed-off-by: realAsma <akuriparambi@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
Signed-off-by: realAsma <akuriparambi@nvidia.com>
| AutoQuantize gradient is fast! | ||
| ============================== | ||
|
|
||
| **Speed.** The direct way to measure sensitivity — quantize one layer at a time and measure a downstream evaluation such as loss, accuracy — runs the whole model per measurement. The AutoQuantize gradient scores all layer × format combinations in one forward + backward sweep. On Qwen3.6-35B-A3B, with everything else identical, that's a ~51× difference (Table 1). |
There was a problem hiding this comment.
Should we add a brief line explaining what the KL divergence sensitivity score represents?
Since we already define the AutoQuantize Gradient-based sensitivity earlier, this would give readers the full context for the comparison.
There was a problem hiding this comment.
I added this, could you take a look at the latest?
Signed-off-by: realAsma <akuriparambi@nvidia.com>
|
|
||
| A mixed-precision assignment is only useful if the runtime can execute it. So AutoQuantize performs a deployment-aware search — runtime coupling constraints are folded into the search rather than patched up afterwards, meaning the searched model is deployable out of the box in vLLM, SGLang, TensorRT-LLM, and similar inference runtimes. Any restriction of the form "this group of operators takes one joint format decision" becomes a merged knapsack item with aggregated sensitivity and cost. | ||
|
|
||
| **1) Joint quantization for fused linear layers.** Inference runtimes often fuse linear operators, which imposes a shared quantization format across the fused group. This constraint is applied within each layer: that layer's Q, K, and V projections are fused and must share one format, so the fused QKV projection becomes a single decision variable. The naive score would just sum the three per-projection sensitivities — but that treats their Hessians as independent, when the three outputs actually interact through the attention operation. Instead, AutoQuantize quantizes all three projections jointly with format :math:`f` and measures the sensitivity at the attention output, so the metric naturally captures how the projections' quantization errors combine through attention: |
There was a problem hiding this comment.
Looks like there's no score rule to redirect QKV to an attention output, we only have that for MoEs: https://github.com/NVIDIA/Model-Optimizer/blob/main/modelopt/torch/quantization/algorithms.py#L1027
In my understanding the QKV will actually get a summed score
There was a problem hiding this comment.
Yes, @meenchen had recently implemented that and found that it was better. So I am keeping that here instead of making it confusing.
There was a problem hiding this comment.
I see, is it this PR:#1878? We should consider merging it if it works better :)
Signed-off-by: realAsma <akuriparambi@nvidia.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
| Results | ||
| ******* | ||
|
|
||
| .. image:: assets/autoquantize-qwen3-mmlu-effective-bits.png |
There was a problem hiding this comment.
BB: Can you get BF16 baselines for the models in the figure, regenerate this figure with BF16 baseline (plotted as a horizontal line) and send the figure to me in slack? Do not upload it here before my approval.
There was a problem hiding this comment.
🤖 Bot comment.
I’ve queued the BF16 baseline runs and a regenerated figure with horizontal BF16 reference lines. I’ll send the candidate privately in Slack and will not upload it to this PR before your approval.
3 participants
What does this PR do?
Type of change: documentation.
Adds the AutoQuantize technical blog to the announcements system introduced by #1971.
This is a stacked PR targeting the branch for #1971. After #1971 merges, this PR can be retargeted to main.
Usage
N/A; documentation only.
Testing
Before your PR is "Ready for review"
Additional Information
Depends on #1971.