Fix NaN bypass of AVIF_CLAMP in gain map pixel clamping

[jortles]

Fixes #3188

Summary

powf() in avifRGBImageComputeGainMap() can return NaN from degenerate gain map gamma fractions. AVIF_CLAMP(v, 0.0f, 1.0f) passes NaN through because IEEE 754 comparisons with NaN always return false. The NaN then triggers the assertion in avifSetRGBAPixel() (debug builds) or causes undefined behavior in the float-to-integer conversion (release builds).

Changes

Replace AVIF_CLAMP(x, 0.0f, 1.0f) with fminf(1.0f, fmaxf(0.0f, x)) at the three call sites in src/gainmap.c that feed pixel data into avifSetRGBAPixel():

1. Line 156 — avifRGBImageApplyGainMap(), identity copy path after color space conversion
2. Line 277 — avifRGBImageApplyGainMap(), tone-mapped pixel output
3. Line 770 — avifRGBImageComputeGainMap(), gain map value after powf() gamma remapping

Per C99 §7.12.12.2/§7.12.12.3, fmaxf/fminf return the non-NaN argument when one argument is NaN, which correctly clamps NaN to 0.0f.

Verification

• 81-byte minimized PoC (afl-tmin) that triggers the assertion on v1.4.1 and main — passes clean with this fix
• All 168 unique crash inputs from AFL++ fuzzing (12 instances, ~250M executions) pass clean with this fix
• No regressions: the fix is a strict behavioral superset of the original AVIF_CLAMP for non-NaN inputs

Discovery

Found via custom AFL++ fuzzer targeting the experimental gain map API (avifRGBImageComputeGainMap / avifRGBImageApplyGainMap). This code path is not covered by OSS-Fuzz as the build does not enable -DAVIF_ENABLE_EXPERIMENTAL_GAIN_MAP=ON.

[Copilot]

Pull request overview

This PR hardens the experimental gain map pipeline in src/gainmap.c so NaN values can no longer bypass the final [0,1] clamp before pixels are written. It fits into libavif’s gain map encode/apply flow by protecting the last stage of RGB output normalization in both tone mapping and gain map generation.

Changes:

• Replaces AVIF_CLAMP(..., 0.0f, 1.0f) with fminf(1.0f, fmaxf(0.0f, ...)) at the gain map code paths that feed pixel values into avifSetRGBAPixel().
• Adds inline comments explaining why comparison-based clamping is NaN-unsafe and why fminf/fmaxf is safer here.
• Covers both the apply path (avifRGBImageApplyGainMap) and the compute path (avifRGBImageComputeGainMap).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[wantehchang]

Anthony: Thank you for the pull request. Please add an entry for this fix to the "Changed since 1.4.1" section in CHANGELOG.md.

[maryla-uc]

Hi, thank you for the issue and pull request.

I'm mostly curious/confused what kind of "degenerate values" cause the powf to return NaN.

In the issue, you say that "A standalone reproducer is included in the linked PR." but I don't see it? And you say that your 81 byte poc is for a fuzzer that is not provided so I can't use it to reproduce.

I'm wondering if the root cause might be earlier in the function, making us pass bad values to powf.

[jortles]

You're right that the root cause is upstream of powf — it's receiving NaN, not producing it.

Apply path (the two sites at lines 156 and 277): when baseLinear + baseOffset is zero (e.g., a black pixel with baseOffset = {0, 1}), and the gain map log2 value is large enough for exp2f() to overflow to +Inf, the multiplication 0.0f * +Inf evaluates to NaN per IEEE 754. The NaN then passes through linearToGamma() and AVIF_CLAMP() unmodified, since the macro's ternary comparisons all return false for NaN operands.

A similar NaN path exists in the compute path (line 764): if base + baseOffset[c] is near-zero after color space conversion, the ratio at line 695 overflows to +Inf. AVIF_MAX(ratio, kEpsilon) only lower-bounds, so +Inf passes through into gainMapF. After avifFindMinMaxWithoutOutliers picks up +Inf as the channel max, the normalization (+Inf - min) / +Inf evaluates to NaN (indeterminate form).

The fminf/fmaxf fix is intentionally a safety net at the output boundary — it ensures NaN can't reach avifSetRGBAPixel() regardless of which upstream path produced it. If you'd prefer a root-cause fix as well (e.g., clamping the ratio or guarding exp2f overflow), I'm happy to add that.

Regarding the reproducer: apologies, that was incorrectly stated in the issue. The 81-byte PoC is consumed by a custom fuzzer harness that constructs avifImage objects and gain map metadata from raw bytes, then calls avifImageComputeGainMap() / avifImageApplyGainMap(). I'll push a standalone reproducer that triggers the apply-path NaN directly — black base pixel, baseOffset = 0, gainMapMax = 1000 so exp2f(1000) = +Inf.

[jortles]

Updated the reproducer in 608ca50. The original used sRGB output transfer, which masked the crash: avifToGammaSRGB() absorbs NaN because all branch conditions (< 0, < 0.003, < 1.0) are false for NaN, so it falls through to return 1.0f — turning NaN into silent data corruption (black pixels render as white) rather than triggering the assertion.

Switched to AVIF_TRANSFER_CHARACTERISTICS_LINEAR, whose transfer function (AVIF_CLAMP) also passes NaN through, allowing it to reach the assertion in avifSetRGBAPixel.

Verified:

• Without fix: Assertion 'rgbaPixel[0] >= 0.0f && rgbaPixel[0] <= 1.0f' failed (SIGABRT)
• With fix: PASS: no crash

This also highlights that the NaN issue affects correctness even with sRGB — the tone-mapped output is silently wrong (1.0 instead of the correct value).