From 32080a4844a8c91e052cabdd8d28f81ac4d39bc5 Mon Sep 17 00:00:00 2001 From: Miao Bin Date: Fri, 7 Aug 2026 17:28:39 +0800 Subject: [PATCH] Add dynamic shape explainer --- dynamic-shape-explainer.md | 361 +++++++++++++++++++++++++++++++++++++ 1 file changed, 361 insertions(+) create mode 100644 dynamic-shape-explainer.md diff --git a/dynamic-shape-explainer.md b/dynamic-shape-explainer.md new file mode 100644 index 00000000..b2fcc826 --- /dev/null +++ b/dynamic-shape-explainer.md @@ -0,0 +1,361 @@ +# Dynamic Shape Explainer + +## Authors +- [Bin Miao](mailto:bin.miao@intel.com), [Wanming Lin](mailto:wanming.lin@intel.com) (Intel) + +## Participate +- [Support flexible input sizes #883](https://github.com/webmachinelearning/webnn/issues/883) + +## Table of contents +1. [Introduction](#introduction) +2. [Goals](#goals) +3. [Non-goals](#non-goals) +4. [Use Cases](#non-goals) +5. [Proposed API](#proposed-api) +6. [Design Discussion](#design-discussion) +7. [Considered Alternatives](#considered-alternatives) +8. [Privacy & Security Considerations](#privacy--security-considerations) +9. [Future Consideration](#future-consideration) +10. [Open Questions](#fine-grained-shape-queries) +11. [References & Acknowledgements](#references--acknowledgements) + +## Introduction +Currently, all `MLOperand` instances within a WebNN graph are constrained to static shapes. Dimensions must be fully resolved during the [MLGraphBuilder.build()](https://www.w3.org/TR/webnn/#api-mlgraphbuilder-build) compilation phase. This constraint significantly limits the API's adaptability for modern machine learning workloads, where input dimensions often remain indeterminate until the point of inference: + +- **Transformer / LLM Decoding:** These workloads involve iterative execution of identical compute graphs with varying sequence lengths. Static shaping necessitates graph recompilation for each distinct sequence length, incurring prohibitive latency. + +- **Vision Encoders:** These architectures often process inputs of arbitrary resolutions, causing spatial dimensions to fluctuate dynamically throughout the network. + +- **Generative Image Models:** These models frequently compute intermediate tensor shapes at runtime (e.g., dynamic padding to block-size alignments), where padding requirements are strictly dependent on input dimension variance. + +Existing framework workarounds — such as frequent graph recompilation or falling back to slower, CPU-based execution environments (e.g., ONNX Runtime Web via WebAssembly) — introduce significant performance bottlenecks, particularly in latency-sensitive autoregressive decoding tasks. + +This proposal introduces **Dynamic Shape** for WebNN, enabling graph dimensions to remain unresolved during compilation and instead derive concrete values from input tensors at dispatch time. This mechanism permits a single compiled graph to accommodate a diverse range of runtime input sizes. This document details the design explored in the Chromium implementation and serves as a technical reference for the Working Group's resolution of [Issue #883, "Support flexible input sizes"](https://github.com/webmachinelearning/webnn/issues/883). + +## Goals +- Allow a single compiled `MLGraph` to execute across varying runtime input sizes, without rebuilding. + +- Model dynamism the way the underlying runtimes already do: a dimension is either a **static size**, a **named dynamic** dimension (a symbolic name), or an **unnamed dynamic** dimension (fully unconstrained). + +- Defer shape validation that cannot be decided at build time to the point where concrete input shapes are known, without weakening the checks that can be decided at build time. + +- Let a framework learn a graph's concrete output shapes for a given set of input shapes **before** dispatching it, so it can allocate output tensors of the right size. + +- Support models whose intermediate shapes are computed at runtime from other shapes (e.g. `padded_len = seq_len + (-seq_len % block)`), not just models that thread an input dimension straight through. + +## Non-goals +- **We do not resolve shapes from input tensor data.** Dynamism that depends on the *values* inside a tensor is out of scope. We resolve only *Symbolic sizes*: dimensions derivable by algebra from the input shapes and build-time constants. + +- **We do not redefine operator semantics.** The operators added here mirror the semantics of their existing static counterparts; they only move shape parameters from build-time attributes to runtime operands. + +## Use Cases + +### Variable sequence length +A web app runs an SLM decoder. The sequence length grows by one every step, so with static shapes the graph must be rebuilt (or a family of graphs pre-built) for each length. + +```js +// Before: the sequence length is baked into the graph. +const input = builder.input('attention_mask', {dataType: 'float32', shape: [1, 128]}); +// ...a different compiled graph is required for each sequence length. +``` + +With dynamic shapes, the sequence length is declared as a named dynamic dimension and the graph is compiled once: + +```js +// After: 'sequence_length' is a named dynamic dimension. +const input = builder.input('attention_mask', {dataType: 'float32', shape: [1, 'sequence_length']}); +const graph = await builder.build({output}); + +// The same graph runs at any sequence length. +mlContext.dispatch(graph, {'attention_mask': tensorWithLen(1)}, {'output': out1}); +mlContext.dispatch(graph, {'attention_mask': tensorWithLen(37)}, {'output': out2}); +``` + +Every operand whose shape depends on `sequence_length` carries the dynamism through the graph automatically. + +### Shapes computed at runtime +Some models contain genuine shape arithmetic that must run at inference time. For example, a "pad the sequence up to a multiple of the block size" step computes the padding amount from the (dynamic) sequence length: + +```js +padding_needed = (-seq_len) mod block // e.g. block = 32 +padded = pad(x, /*end=*/[padding_needed, 0]) // now a multiple of 32 +``` + +Expressing this requires reading a shape *as data*, doing arithmetic on it, and feeding the result back into an operator as a shape parameter: + +```js +const shape = builder.shape(x); // uint32 1-D tensor [seq_len, 2560] +const seqLen = builder.slice(shape, [0], [1]); // [seq_len] + +// (-seq_len) mod 32 +const pad = builder.modulusFloor(builder.neg(seqLen), builder.constant(..., [32])); +const zero = builder.constant(..., [0]); // no padding on the trailing axis +const padded = builder.padDynamic(x, builder.constant(..., [0, 0]), builder.concat([pad, zero], 0)); +``` + +This motivates the **shape-as-data operators** described below. + +### Getting output shapes before dispatch +A framework such as ONNX Runtime Web partitions a model and hands WebNN a subgraph. That subgraph's output can carry a dynamic dimension that is not present on any of its inputs (it is derived inside the subgraph). To allocate the output tensor, the framework needs the concrete output shape before it dispatches: + +```js +const outShapes = await graph.computeShapes({'attention_mask': [1, 37]}); +// => {'output': [1, 74]} + +// Allocate output MLTensors of the resolved size, then dispatch. +``` + +## Proposed API +The web-facing surface changes fall into three groups: 1) the dimension model and descriptors, 2) new `computeShapes()` methods and 3) new operators. The runtime behavior behind these — the two mechanisms that carry dynamism at dispatch, **shape inference** and **shape folding** — is described under [Design Discussion](#design-discussion). + +### 1. Dimension model and descriptors +A graph input's dimension may now be a number (static integer), a string (a **named dynamic** dimension mapped to a runtime symbolic name), or — as a `null` element of the shape sequence — an **unnamed dynamic** dimension. This is expressed with a new `MLDimension` type and a new `MLInputOperandDescriptor`: + +```webidl +// A single input dimension: a number is static; a string is a named dynamic +// dimension; a null element (below) is an unnamed dynamic dimension. +typedef ([EnforceRange] unsigned long or DOMString) MLDimension; + +dictionary MLInputOperandDescriptor { + required MLOperandDataType dataType; + required sequence shape; +}; +``` + +Reading a shape back, `MLOperand.shape` is widened accordingly: its elements may now be strings, an unnamed dynamic dimension surfaces as `"?"` (a provisional representation — see [Open Questions](#fine-grained-shape-queries)), and the whole attribute is `null` for an **unranked** operand — one whose rank is not yet known (see [Dimension semantics](#dimension-semantics)): + +```webidl +interface MLOperand { + readonly attribute MLOperandDataType dataType; + // Was: FrozenArray shape; + readonly attribute FrozenArray? shape; +}; +``` + +The plain `MLOperandDescriptor` is deliberately **unchanged** (static-only): it describes `constant()` data, and `computeShapes()` likewise returns fully concrete (static) shapes. Dynamism thus lives only on graph *inputs* and propagates from there. + +```webidl +// Unchanged — always concrete. +dictionary MLOperandDescriptor { + required MLOperandDataType dataType; + required sequence<[EnforceRange] unsigned long> shape; +}; +``` + +### 2. `computeShapes()` +`computeShapes()` runs the same shape inference and folding as dispatch, but early and without executing the graph, returning the concrete output shape for each output given concrete input shapes: + +```webidl +partial interface MLGraph { + record> computeShapes( + record> inputShapes); +}; +``` + +The programming model becomes **build → (optionally) computeShapes → dispatch**. + +This method enables frameworks to determine the output tensor sizes for dynamic subgraphs whose shapes are difficult to infer. Exposing it reduces redundant computation overhead. + +### 3. Shape-as-data operators +Threading a dynamic input dimension through the graph is not sufficient on its own; real models compute with shapes. This proposal adds a family of operators that treat a shape as a runtime tensor, plus dynamic variants of existing operators that take their shape parameters as **operands** rather than build-time attributes. + +```webidl +partial interface MLGraphBuilder { + // Read an operand's shape as a runtime uint32 1-D tensor. + MLOperand shape(MLOperand input, optional MLOperatorOptions options = {}); + + // Shape generators / arithmetic on shape tensors. + MLOperand range(MLOperand start, MLOperand limit, MLOperand delta, + optional MLOperatorOptions options = {}); + MLOperand modulusFloor(MLOperand a, MLOperand b, optional MLOperatorOptions options = {}); + MLOperand modulusTruncate(MLOperand a, MLOperand b, optional MLOperatorOptions options = {}); + + // Rank-changing operators (the seam where dynamic rank originates). + MLOperand squeeze(MLOperand input, optional MLSqueezeOptions options = {}); + MLOperand unsqueeze(MLOperand input, sequence<[EnforceRange] unsigned long> axes, + optional MLOperatorOptions options = {}); + MLOperand reshapeTo2d(MLOperand input, optional MLReshapeTo2dOptions options = {[EnforceRange] unsigned long axis = 1;}); + + // Dynamic variants: shape parameters are operands, evaluated at dispatch. + MLOperand reshapeDynamic(MLOperand input, MLOperand newShape, optional MLOperatorOptions options = {}); + MLOperand expandDynamic(MLOperand input, MLOperand newShape, optional MLOperatorOptions options = {}); + MLOperand sliceDynamic(MLOperand input, MLOperand starts, MLOperand sizes, optional MLSliceDynamicOptions options = {}); + MLOperand padDynamic(MLOperand input, MLOperand beginningPadding, MLOperand endingPadding, optional MLOperatorOptions options = {}); + + // Equal-split (scalar count) and explicit-splits (operand) forms. + sequence splitDynamic(MLOperand input, [EnforceRange] unsigned long splits, optional MLSplitOptions options = {}); + sequence splitDynamic(MLOperand input, MLOperand splits, optional MLSplitOptions options = {}); + MLOperand resample2dDynamic(MLOperand input, optional MLResample2dDynamicOptions options = {MLOperand sizes, ...}); + MLOperand tileDynamic(MLOperand input, MLOperand repetitions, optional MLOperatorOptions options = {}); +}; +``` + +Three notes on the design of this family, rather than a per-operator walkthrough: + +- **Each `*Dynamic` operator is a full dynamic mirror of its static counterpart.** `sliceDynamic` uses the same `starts + sizes (+ strides)` semantics as static `slice`; splitDynamic mirrors split's explicit and equal-split forms and its `axis` option. Keeping them one-to-one lets the static operators stay static-input-only and avoids divergent validation. + +- **`squeeze` / `unsqueeze` / `reshapeTo2d` are the origin of dynamic rank.** E.g. a no-axes `squeeze` removes every size-1 dimension, so its output rank depends on runtime data. Emulating this logic requires complex subgraph chains to calculate runtime shapes, resulting in significant graph bloat and excessive shape inference overhead. Native operator support provides a more efficient and direct representation of these rank-changing transformations. + +- **We do not adopt ONNX's -1 (auto-infer) / 0 (copy) reshape conventions.** These are framework-level conveniences that a framework can lower into a shape subgraph before calling WebNN, and not all runtimes support them; keeping them out of WebNN avoids baking one framework's convention into the platform. So frameworks must instead use a subgraph chain to calculate these dimensions dynamically: + +For example, when the target shape is a runtime operand (values unknown at build time), the framework resolves the dimensions element-wise across the entire shape vector: + +1. **Resolve 0:** `equal(targetShape, 0)` → `where(mask, shape(input), targetShape)` → `shapeNoZero`. +2. **Resolve -1:** + - `total = reduceProduct(shape(input))` + - `known = reduceProduct(where(equal(shapeNoZero, -1), 1, shapeNoZero))` + - `inferred = div(total, known)` +3. **Final Assembly:** `where(isNeg1, inferred, shapeNoZero)` → `cast(uint32)` → `reshapeDynamic`. + +*(Optimization: This chain may be skippable if the framework can prove the operand is already sentinel-free.)* + +## Design Discussion +This section describes the runtime behavior that gives it meaning — the two mechanisms that carry dynamism at dispatch, **shape inference** and **shape folding**, plus the dimension semantics they honor. + +### Dimension semantics +- **Named dimensions are constraints.** Two dynamic dimensions with the **same name** are guaranteed to take the **same** concrete value throughout the graph (e.g. "query and key sequence lengths are equal"). An unnamed dimension carries no such constraint, and a dynamic dimension has **no min/max bound** — anything not provably static simply defers. + +- **Derived dimensions are never given a synthesized name.** A named dimension survives only on a 1:1 pass-through; any dimension *computed* from a dynamic one becomes unnamed. This avoids inventing symbolic identities the runtime cannot honor. + +- **Unranked operands.** For example, a no-axes `squeeze` removes every size-1 dimension, so its output rank depends on runtime data — the operand is unranked (its `MLOperand.shape` is null) until `computeShapes()`/`dispatch()` recovers it. + +### Deferred validation +Validation splits across two phases: + +- **At build time**, we validate only what is knowable without concrete shapes: data-type compatibility, rank constraints (e.g. `conv2d` needs rank 4), same-name symbolic consistency, and any **definite static contradiction** (e.g. reshaping a static `[2, 3]` to `[7]`). + +- **At dispatch time** (and at `computeShapes()`), once concrete input shapes are known, we run **shape inference** — a forward propagation of each operand's concrete *shape* — over the whole graph, and validate the resulting concrete shapes against every constraint, including buffer sizes. + +Deferring this work is the inherent trade-off of dynamic shapes: the shape resolution and validation a static graph completes once at build time now runs at inference time — as a gatekeeper on every `dispatch`, before the graph executes. This adds per-inference overhead, so a user agent should optimize the common cases: because the result is a pure function of the input shapes, it can skip re-validation when a dispatch repeats a set of input shapes it has already validated. + +The build-time checks are expressed as **three-valued** dimension predicates. Instead of "equal / not-equal", a comparison is *provably-equal*, *provably-unequal*, or *unknown (defer)*. Only a provable contradiction is rejected at build time: + +```cpp +// Reject only when BOTH dims are static AND differ; otherwise defer to dispatch. +bool DimensionsAreDefinitelyUnequal(Dimension a, Dimension b) { + if (IsStatic(a) && IsStatic(b)) + return StaticValue(a) != StaticValue(b); + return false; // unknown -> defer +} +``` + +The same predicate is applied uniformly across operators that impose cross-dimension constraints, e.g. `matmul`'s contraction dimension, concat's non-concatenated axes, `reshape`'s element-count product, broadcasting, and so on. + +### Shape folding at dispatch +*Shape folding* is the dispatch-time evaluation of a `shape()`-rooted chain down to the concrete values a shape parameter needs. It is deliberately narrow — the shape-calculation operations only: arithmetic and structural transforms on shape tensors (usually integer, sometimes float, e.g. reciprocal) — and never the data-producing operators such as `conv2d` or `matmul`, which do not sit on a shape chain. It is the value-computing counterpart to *shape inference* ([Deferred validation](#deferred-validation)), which propagates operand shapes across the whole graph. + +A small chain makes this concrete. To reshape a `[1, 'seqlen', 512]` tensor into `[1, 'seqlen', 8, 64]` (splitting the static hidden size into 8 heads × 64) while keeping the dynamic sequence length: + +```js +const s = builder.shape(x); // uint32 1-D: the runtime dims of x +const batchSeq = builder.slice(s, [0], [2]); // first two dims → [1, seqlen] +const heads = builder.constant(/*uint32*/ ..., [8, 64]); // static tail +const newShape = builder.concat([batchSeq, heads], 0); // [1, seqlen, 8, 64] +const y = builder.reshapeDynamic(x, newShape); // y: [1, 'seqlen', 8, 64] +``` + +At dispatch with `seqlen = 37`, folding walks the `newShape` chain: `shape(x) `→` [1, 37, 512], slice `→` [1, 37], concat([1, 37], [8, 64]) `→` [1, 37, 8, 64]`. That folded value becomes reshapeDynamic's inferred output shape, `[1, 37, 8, 64]`. Note what it read: `x`'s **shape** and the **constant** `[8, 64]` — never `x`'s data. + +What folding may read is bounded by the root: `shape()` outputs and build-time constants only. A chain that reaches an input's tensor **data** is unresolvable by design and is rejected — the out-of-scope [Tensor-Derived](#fine-grained-shape-queries) case. + +### Backends mapping + +#### ORT +- **Named dynamic dimension** → ONNX symbolic name ([OrtApi::SetDimensions](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a6575872736b924b47a382deb97e2fc17)): + +```cpp +WebNN: shape=['batch', 512] +ONNX: shape=[dim_param:'batch', 512] +``` + +- **Unnamed dynamic dimension** → ONNX free dimension ([OrtApi::SetSymbolicDimensions](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#aa5b1654064d833a515f3acfcdcc5e81d)): + +```cpp +WebNN: shape=[null, 512] +ONNX: shape=[-1, 512] +``` + +#### LiteRT +- **Named/Unnamed dynamic dimension** → LiteRT unknown dimensions: + +```cpp +WebNN: shape=["batch", null, 512] +LiteRT: shape=[-1, -1, 512] +``` + +Represented in the LiteRT flatbuffer schema as: + +```cpp +const flatbuffers::Offset> dimensions +``` + +#### Core ML +- **Named/Unnamed dynamic dimension** → Core ML unbounded ranges dimensions: + +```cpp +WebNN: + shape=["batch", null, 512] +``` + +```cpp +// Core ML: +auto* size_range = shape_range->add_sizeranges(); +size_range->set_lowerbound(1); +// set upper bound to maximum long value +size_range->set_upperbound(std::numeric_limits::max()); +``` + +## Considered Alternatives + +### Per-operator build-time loosening +The first cut loosened each operator's build-time validator independently to tolerate dynamic dimensions. This scattered subtly different "is this okay?" logic across dozens of operators. We unified it behind the three-valued dimension predicates in [Deferred validation](#deferred-validation), so every operator defers the same definition of "unknown" and only rejects provable contradictions. + +### Bespoke dynamic operators +We considered giving each `*Dynamic` operator a shape signature tailored to its most common use, rather than mirroring the static operator. We rejected it for the reasons in [Shape-as-data operators](#3-shape-as-data-operators): faithful mirroring keeps the mental model small and lets the static operators stay static-input-only. + +## Privacy & Security Considerations +Dynamic shapes add no new fingerprinting or cross-origin surface: they expose no device or environment information a static graph does not, and folding and inference read only shapes the page itself supplied. The considerations below are therefore about security. + +The renderer is untrusted, so the service must remain safe on any graph a compromised renderer can construct, including ill-formed dynamic graphs: + +- **No dereference of an absent rank.** Unranked operands (e.g. from a no-axes squeeze) are handled uniformly by each shared validator — propagate, resolve, or cleanly reject — and unranked *graph inputs* are rejected at build time (a graph input always has a known rank). A dispatch-time exit gate fails cleanly if any operand is still unranked after inference, so no unranked operand ever reaches a backend. + +- **Folding never reads input data**, as described above, which also bounds what a graph can make the interpreter do. + +- **Bounded constant collection.** The interpreter seeds only from the shape operands of the dynamic operators and walks back along the shape-computation chain, copying only the constants actually on that chain (and skipping oversized constants). Weight tensors are never on a shape chain and are not copied, bounding both memory and folding work (a DoS/OOM guard). + +## Future Consideration + +### Bounded (min/max) dimensions +Currently, the proposed model is intentionally unbounded; a dynamic dimension is either provably static or deferred without a specific size range. As a future enhancement, we plan to allow dynamic dimensions to optionally declare a `minSize` and `maxSize` bound (and potentially an "optimal" size). For example: `{name: 'seqlen', minSize: 1, maxSize: 2048}`. + +Rather than introducing a new source of dynamism, these bounds will serve as critical implementation hints to underlying runtimes, unlocking ahead-of-time memory allocations and graph optimizations that an unbounded model cannot achieve. + +### Fine-grained Shape Queries +Currently, the `shape()` operator returns an operand's entire shape as a 1D tensor. To provide more fine-grained shape retrieval, we may consider introducing two additional operators: + +- `rank()` → Returns a 0D scalar tensor representing the operand's rank (similar to `tfl.rank`). + +- `dimension(axis)` → Returns a 0D scalar tensor representing the size of a specific dimension (similar to StableHLO's `get_dimension_size`). + +## Open Questions +- **Tensor-Derived sizes.** Whether, and how, to admit dimensions that depend on tensor *values*, which this proposal places out of scope. + +- **How to represent an unnamed dynamic dimension on readback.** On input it is a `null` element of the shape sequence, and the symmetric choice would be `null` on `MLOperand.shape` as well — but a `FrozenArray` cannot hold `null` elements, so the current implementation reports the string `"?"` instead. Whether to keep the `"?"` sentinel, change the readback type to admit `null`, or adopt another representation is left for the Working Group. + +- **Bidirectional broadcasting for Expand.** In dynamic shape scenarios, the shape operand of `expand` may contain flexible dimensions (e.g., `[1, 1, 1]`) that require bidirectional broadcasting against the input tensor. The original WebNN specification was limited to unidirectional broadcasting. Extending the `expand` operator to support bidirectional broadcasting aligns with the ONNX `Expand` operator and ensures correct handling of these dynamic shape cases. + +## References & Acknowledgements +- [Support flexible input sizes #883](https://github.com/webmachinelearning/webnn/issues/883) + +- [ORT API SetDimensions()](https://onnxruntime.ai/docs/api/c/struct_ort_api.html#a6575872736b924b47a382deb97e2fc17) + +- [TFLite Flatbuffer Tensor & TfLiteTensor](https://developers.google.com/edge/api/tflite/c/struct/tf-lite-tensor) + +- [Core ML Flexible Input Shapes](https://apple.github.io/coremltools/docs-guides/source/flexible-inputs.html) + +- Many thanks for valuable feedback and advice from: + + - [Ningxin Hu](mailto:ningxin.hu@intel.com) + + - [Dwayne Robinson](mailto:dwayner@microsoft.com)