Cortex-M: annotate and preserve in-place activations

[rascani]

Summary

A model that runs its activation in place emits aten.silu_ rather than aten.silu in the pre-dispatch graph. This is not a corner case: Ultralytics gives every Conv block the same class-level nn.SiLU instance, which initialize_weights() then flips to inplace=True, so one attribute decides the whole network. ACTIVATION_OP_PATTERNS listed only the functional overloads, so the quantizer never annotated those nodes; FoldAndAnnotateQParamsPass then declined to fold them, since it folds only into nodes carrying ArmAnnotationInfo, and each activation was left as an fp32 island between two quantized convolutions. On yolo11n that is 76 of them, visible only as a count of unannotated nodes in the quantizer report.

The other pattern dicts in this file already enumerate the in-place variants of relu, hardtanh, clamp and hardsigmoid, so this follows that convention rather than adding a normalization pass. aten.gelu_ is left out because no idiomatic model reaches it: there is no Tensor.gelu_, and neither nn.GELU nor F.gelu takes an inplace argument.

Annotating alone is not enough on the compiler path users actually run. aot_arm_compiler.py preserved hardsigmoid and hardswish through to_edge but not silu, so silu decomposed into sigmoid * mul, the mul carried qparams on only one input, and AtenToCortexMPass raised a KeyError. That made an annotated conv+SiLU model fail to compile where it had previously produced a working fp32 island. Adding silu to that preserve list, and to the same list in the overview doc, makes it lower to quantized_conv2d + quantized_activation instead. The test harness had silu preserved already, which is why no existing test saw this. That list is Cortex-M specific, and the Arm backend maps aten.silu to a TOSA TABLE op, so preserving it does not disturb the Ethos-U path.

Test plan

pytest backends/cortex_m/test/ops/test_activation_quant.py -- 48 cases, dialect and Corstone-300, green; 24 dialect cases also green on cortex-m0plus and cortex-m7. Full backends/cortex_m/test dialect run: 407 passed. Removing the three dict entries fails the three new in-place cases; removing the preserve entry reproduces the compile failure above on conv + nn.SiLU(inplace=True).

Authored with assistance from Claude Code.

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21819

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 1 Unrelated Failure, 3 Unclassified Failures

As of commit 55aab80 with merge base a56af1c ():

NEW FAILURES - The following jobs have failed:

• pull / test-arduino-library / run / arduino-library (gh)
  RuntimeError: Command docker exec -t b470f33601f1eba744b8411a81c53fa61faa1c31ef02b8d6d480e923de6be212 /exec failed with exit code 1
• trunk / test-models-linux-aarch64 (mv3, portable, linux.arm64.2xlarge) / linux-job (gh)
  RuntimeError: Command docker exec -t ffb68f74070d594558c82348294ab91e93cb44fdeb220a449840b7d163c96791 /exec failed with exit code 1

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

• pull / test-coreml-bc-macos (macos-m1-stable) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  API rate limit exceeded
• trunk / test-models-macos-coreml (mv3) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  API rate limit exceeded
• trunk / test-models-macos-cpu (w2l, xnnpack-quantization-delegation) / macos-job (gh) (this job did not run on the merge base, so DrCI cannot tell whether the failure is pre-existing)
  API rate limit exceeded

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Apple / build-demo-ios / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/runner/work/_temp/exec_script failed with exit code 65

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[Erik-Lundell]

LGTM. One question

[Erik-Lundell]

Is there a longer term solution for this - how does the end user know that silu should be supplied here to get correct handling of silu?

[Erik-Lundell]

Ok, I guess this answers my question about how to know. But still doesn't seem super scalable?

[rascani]

Agreed. I forgot to tag you on #21826. It was actually worse than just these 2 places.