Skip to content

Clamp HDR endpoint spread before narrowing it to int#645

Merged
solidpixel merged 1 commit into
ARM-software:mainfrom
sahvx655-wq:hdr-submode-int-clamp
Jun 17, 2026
Merged

Clamp HDR endpoint spread before narrowing it to int#645
solidpixel merged 1 commit into
ARM-software:mainfrom
sahvx655-wq:hdr-submode-int-clamp

Conversation

@sahvx655-wq

Copy link
Copy Markdown
Contributor

The per-quant-level format search in compute_color_error_for_every_integer_count_and_quant_level estimates the likely RGB and RGBO submodes from the colour spread of the decoded endpoints, casting the three spread values bf, cf and df straight to int. The endpoints can fall a long way outside the input range, and an HDR image loaded through the fp16 path keeps any overexposed infinities or NaNs the source carried, so the spread is not bounded by INT_MAX. Feeding a crafted fp16 .ktx through -ch makes UBSan flag all three conversions with runtime error: <value> is outside the range of representable values of type 'int', and the integer that comes back is whatever the platform conversion happens to emit.

b, c and d are only read by the threshold tests that choose the submode and the largest threshold is 32768, so clamping each spread into [0, 65536] before the narrowing cast leaves every comparison result unchanged for in-range data while making the out-of-range case well defined. astc::clamp folds NaN to its low bound, so a non-finite spread drops into the small-value branch rather than into undefined behaviour. The bound lives at the conversion because that is the only place the value is narrowed; the endpoint maths upstream is left alone, and encoded output is byte-identical for valid images.

@solidpixel solidpixel merged commit 7d3a029 into ARM-software:main Jun 17, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants