[ntuple] Fix kReal32Quant decoding for 32-bit values - #411
Conversation
Avoid 5-bit masking of JS shift counts, which made divisor 0 at nbits = 32. Cast res to Uint32, s.t. bit 31 is not interpreted as sign bit.
|
Can you provide small test file with such corner cases - from this and previous PR. I want to include this macro in jsroot-ci.yml |
|
real32quant.zip |
|
You are using Can you adjust reader.mjs that it works without it and instead writing some JSON file compare read values with some predefined values. Something like code from |
Avoid 5-bit masking of JS shift counts, which made divisor 0 at nbits = 32.
Cast res to Uint32, s.t. bit 31 is not interpreted as sign bit.