Add dynamic shape explainer - #945
Conversation
|
A POC based on the current explainer is WIP. And it has been validated against some real-world Transformer, LLM and image generation models. |
fdwr
left a comment
fdwr
left a comment
There was a problem hiding this comment.
👍 I have some thoughts, but it's 95% 👌.
| }; | ||
| ``` | ||
|
|
||
| ### 2. `computeShapes()` |
There was a problem hiding this comment.
I love that computeShapes exists, because even though the primary intention is for returning the shape, it might also offer a means for the backends to precompute needed memory allocations, so that the later dispatch is immediate. Under the hood, model execution has a few key steps between knowing the shape (like planning memory and recompiling shaders) and actually executing, and unfortunately many ML libraries fail to expose that key stage -_- (and deferring all the way to dispatch is bad because then it means the first dispatch will be slow, and later ones will be fast, but also any shape changes will cause stutters).
One performance concern I have is the potential ping-pong of shape vacillation. If the model uses one shape the first iteration, and another shape the next 10 iterations, then every transition might require reallocating and replanning memory again. If however the caller could just say "build this graph, and return me two MLGraphs for these two input shapes, with all weights shared between them", the backends could execute more efficiently (no need to see the future or cache things behind your back).
Additionally, using computeShapes catches errors early rather than more confusingly during dispatch when it's too late to do much about it. So, there are a few good reasons to push shape evaluation earlier than dispatch.
(mostly just thinking aloud - resolve me)
| ## Goals | ||
| - Allow a single compiled `MLGraph` to execute across varying runtime input sizes, without rebuilding. | ||
|
|
||
| - Model dynamism the way the underlying runtimes already do: a dimension is either a **static size**, a **named dynamic** dimension (a symbolic name), or an **unnamed dynamic** dimension (fully unconstrained). |
There was a problem hiding this comment.
Do you think we even need unnamed dimensions? A uniquely named dynamic dimension and an unnamed dynamic dimension are identical after all, and it's preferable to have debuggable symbols. The only reason to support them would be because existing callers may have them (like ORTWeb calling WebNN), but we could always just synthesize a name on the fly like "inputTensorName2_axis3" 🤔. I mean, if I was debugging and hit a shape inference error, I'd rather see that than just null for a name. Alternately using "" instead of null could be less problematic (no need to check for null first before trying to use/print the string).
Actually, seeing generated names would probably help you too during WebNN/Chromium debugging, seeing where pass-through fails during shape inference.
| }; | ||
| ``` | ||
|
|
||
| Reading a shape back, `MLOperand.shape` is widened accordingly: its elements may now be strings, an unnamed dynamic dimension surfaces as `"?"` (a provisional representation — see [Open Questions](#fine-grained-shape-queries)), and the whole attribute is `null` for an **unranked** operand — one whose rank is not yet known (see [Dimension semantics](#dimension-semantics)): |
There was a problem hiding this comment.
one whose rank is not yet known
Interestingly this has also come up for MLOperandDataType dataType too, not just shape, when you want to infer the type based on the input tensor's type so you can reuse the same mini-model for float16 and float32.
(just comment, no action expected - resolve me)
|
|
||
| // Equal-split (scalar count) and explicit-splits (operand) forms. | ||
| sequence<MLOperand> splitDynamic(MLOperand input, [EnforceRange] unsigned long splits, optional MLSplitOptions options = {}); | ||
| sequence<MLOperand> splitDynamic(MLOperand input, MLOperand splits, optional MLSplitOptions options = {}); |
There was a problem hiding this comment.
Are we going with two separate functions rather than ([EnforceRange] unsigned long or sequence<[EnforceRange] unsigned long>) splits like normal split?
| The same predicate is applied uniformly across operators that impose cross-dimension constraints, e.g. `matmul`'s contraction dimension, concat's non-concatenated axes, `reshape`'s element-count product, broadcasting, and so on. | ||
|
|
||
| ### Shape folding at dispatch | ||
| *Shape folding* is the dispatch-time evaluation of a `shape()`-rooted chain down to the concrete values a shape parameter needs. It is deliberately narrow — the shape-calculation operations only: arithmetic and structural transforms on shape tensors (usually integer, sometimes float, e.g. reciprocal) — and never the data-producing operators such as `conv2d` or `matmul`, which do not sit on a shape chain. It is the value-computing counterpart to *shape inference* ([Deferred validation](#deferred-validation)), which propagates operand shapes across the whole graph. |
There was a problem hiding this comment.
and never the data-producing operators such as conv2d or matmul
It sounds like this is saying the output of matmul cannot be a shape input tileDynamic? If so, I don't think we should have an explicit list of blessed operators that can modify a shape, but rather, any WebNN operator can modify a shape tensor along the chain piping to a fooDynamic operator, so long as those ops are placed on the CPU EP/backend (because you would not want to read back from the GPU and stall). Plus, MatMul is really an arithmetic op (mul and add), and maybe a model wants to compute a shape using MatMul as a dot product (reducing two 1D vectors into a single scalar element count, equivalent to add and reduceProduct). I haven't seen that personally, but it seems plausible. 🤔 Though, I certainly could understand type restrictions (like only supporting integers), or dimension restrictions (like only working with 1D tensors).
Though, if the reason for this restriction is because we're doing these operators ourselves on the Chromium side for shape inference (rather than using the backend), then I could understand op restrictions. Still, it will be weird if WebNN can only use specific shape-compatible ops.
| ## Future Consideration | ||
|
|
||
| ### Bounded (min/max) dimensions | ||
| Currently, the proposed model is intentionally unbounded; a dynamic dimension is either provably static or deferred without a specific size range. As a future enhancement, we plan to allow dynamic dimensions to optionally declare a `minSize` and `maxSize` bound (and potentially an "optimal" size). For example: `{name: 'seqlen', minSize: 1, maxSize: 2048}`. |
There was a problem hiding this comment.
optionally declare a
minSizeandmaxSizebound
Another potentially useful constraint is a "multiple of", because models like StableDiffusion will fail if you try to bind an input whose width/height isn't a multiple of 8. (resolve me)
|
anssiko marked as non substantive for IPR from ash-nazg. |
|
I cleared the automatic IPR check for this PR, since explainer documents are consider non-substantive from the W3C Patent Policy point of view. DetailsThis IPR check is to ensure normative portions of the specification come from organizations who participate the WG. Furthermore, in this case, I can attest the authors of this PR are affiliated with Intel, and as such, any normative portions are reusable in the spec PR, as appropriate. |
4 participants
This is the initial draft to summarize the discussion on dynamic shapes #883 .
The explainer covers named/unnamed dynamic dimensions, deferred (dispatch-time) shape validation,
computeShapes()API, and a new family of shape-as-data (*Dynamic) operators.Open questions and considered alternatives are called out explicitly in the explainer and feedback on this doc would be very welcome.