Add low-precision floating point data types explainer

[mklimenko-nv]

This PR is a first step to add low-precision floating point data types to the WebNN specification. It's a follow-up to #930 with a conservative addition of bfloat16 and float8 types only, with a potential follow-up for microscaled data types.

[fdwr]

Tis a nice document, thanks for preparing, and I support adding low precision data types (I may just have "are we sure xyz" questions 🙂).

[fdwr]

parameters in 32-bit (float8)

Did you mean 32-bit (float32) here?

[mklimenko-nv]

Yes, that's a typo, thanks for noticing!

[fdwr]

How certain are we that this specific float8 is the float8 bit allocation that we want to codify into the API without any clarifying suffix? Will we need another float8 in the future, one that would need a distinguishing suffix (making it odd to have float8 and float8suffixed)? After all, we already determined that IEEE float16 (float16m10e5s1) was insufficient for all models, warranting float16m7e8s1.

Just to double check, the flavor of float8 proposed is this one ⭐, being IEEE-compliant, with full infinity, NaN, and distinct positive/negative zero?

float8m3e4s1_t ⭐     { uint8_t mantissa: 3; uint8_t exponent: 4; uint8_t sign: 1;} // infinities, NaNs
float8m2e5s1_t        { uint8_t mantissa: 2; uint8_t exponent: 5; uint8_t sign: 1;} // infinities, NaNs
float8m3e4s1fnuz_t    { uint8_t mantissa: 3; uint8_t exponent: 4; uint8_t sign: 1;} // no infinities, NaN as -0
float8m2e5s1fnuz_t    { uint8_t mantissa: 2; uint8_t exponent: 5; uint8_t sign: 1;} // no infinities, NaN as -0
float8m4e3s1_t        { uint8_t mantissa: 4; uint8_t exponent: 3; uint8_t sign: 1;} // infinities, NaN's
float8m0e8s0fn_t      { uint8_t mantissa: 8; uint8_t exponent: 0; uint8_t sign: 0;} // no infinities, NaN via all ones (purely range scaling)
float8m3e4b11s1fnuz_t { uint8_t mantissa: 3; uint8_t exponent: 4; uint8_t sign: 1;} // exp bias 11, no infinities, NaN as -0
float8m3e4s1fn_t      { uint8_t mantissa: 3; uint8_t exponent: 4; uint8_t sign: 1;} // infinities, NaNs

[mklimenko-nv]

I agree with the float16 vs bfloat16 use cases, but with float8 I find it important to maintain a balance between being explicit and user-friendly. We see that the e4m3 variant is used primarily in inference with e5m2 being more popular in training. For the sake of completeness, it is possible to specify the suffix upfront, I'll update the explainer.

On the _fnuz subvariants and other flavors my personal stance is that they're vendor-specific and would overcomplicate the specification.

[fdwr]

dequantizeLinear already supports blockwise scales? See emulated decomposition:

function dequantizeLinear(builder, input, scale, zeroPoint, options)
{
    // output = (input - zeroPoint) * scale
    const floatInput = builder.cast(input, scale.dataType);
    const floatZeroPoint = builder.cast(zeroPoint, scale.dataType);
    const upsampledScale = blockwiseExpand(builder, scale, input.shape);
    const upsampledZeroPoint = blockwiseExpand(builder, floatZeroPoint, input.shape);
    return builder.mul(builder.sub(floatInput, upsampledZeroPoint), upsampledScale);
}

I think had a Phi-3 WebNN demo (now Phi 4) running that used block scales.

If this was unclear from the documentation, we should clarify it.

[fdwr]

Aha, supporting links:

• Support block-wise quantization #779
• https://chromium-review.googlesource.com/c/chromium/src/+/5964816

[mklimenko-nv]

I missed it, thanks for pointing out

[fdwr]

We didn't add blockSize after becaming apparent that it was redundant from the tensor shapes, and given the ratio of the input tensor's shape and zero-point shape would have needed to be validated anyway, it's cleaner to just remove a potential failure point. Plus it's more flexible than a single block size since multiple shapes of dequantization exist (along rows, columns, square blocks, asymmetric...). Of course, performance is best when callers use shapes that align to whatever the underlying backend's have been optimized for (and it's actually worse performance if it's incompatible and you have to emulate things), but it's hard to know what backend you'll run on, and the current design doesn't lock us into so rigid a box when it changes later.

[fdwr]

There's a pending issue to make zeroPoint optional, which would be equivalent to symmetric, right?

[mklimenko-nv]

Yes, that's correct. I'd like to add to that that the 3% performance improvement is a conservative estimation based on my previous experiments.

[fdwr]

the low-precision data types are represented using non-native JavaScript data types

Note there's precedent for using a different underlying buffer type than the tensor's data type, as we had to use Uint16Array for float16 tensors, before Float16Array existed.

[mklimenko-nv]

I'll rephrase it for the next iteration of the explainer, thanks!