Skip to content

Honor unaligned 16-bit RGB row strides - #3318

Closed
carrerasdarren-cell wants to merge 1 commit into
AOMediaCodec:mainfrom
carrerasdarren-cell:fix-unaligned-rgb16-strides
Closed

Honor unaligned 16-bit RGB row strides#3318
carrerasdarren-cell wants to merge 1 commit into
AOMediaCodec:mainfrom
carrerasdarren-cell:fix-unaligned-rgb16-strides

Conversation

@carrerasdarren-cell

Copy link
Copy Markdown

Problem

avifRGBImage allows caller-supplied buffers whose rowBytes is larger than the tight pixel width. For high-depth RGB, an odd padded stride makes successive rows alternate between aligned and unaligned addresses.

The scalar RGB and alpha paths currently access these samples through uint16_t *. UBSan reports misaligned stores in avifFillAlpha() and the 16-bit YUV-to-RGB writers. avifRGBImageToF16() also advances a uint16_t * by rowBytes >> 1, so it discards the low stride bit. On untouched main, a valid 9-byte-stride RGBA output changes when only the row-padding byte changes:

valid odd-stride output depends on padding: yes
end guard changed: no

That can turn caller-owned row padding into visible float output and is undefined behavior on strict-alignment targets.

Change

  • Use two-byte memcpy helpers for 16-bit samples in caller-owned RGB buffers while preserving native-endian behavior.
  • Address every F16 row through byte arithmetic so the exact stride is honored.
  • Have the libyuv F16 wrapper fall back when the typed base or byte stride is unaligned.
  • Cover unaligned odd-stride conversion in both directions and high-depth alpha multiply/unmultiply across all alpha-bearing formats.

The patched sanitizer run reports no alignment errors and the same reproducer becomes:

valid odd-stride output depends on padding: no
end guard changed: no

Verification

  • Normal build: avifrgbtest, avifalphapremtest, avifrgbtoyuvtest, and avifrgbtoyuvthreadingtest pass.
  • ASan/UBSan build: the same four suites pass.
  • libyuv 1949 build: avifrgbtest passes, including aligned F16 fast-path output versus the odd-stride scalar fallback.
  • Project clang-format and git diff --check pass.

Related work

PR #3312 touches avifFillAlpha() and same-depth avifReformatAlpha() to add an AArch64 NEON path, but it does not address odd row alignment. If it lands after this change, its 16-bit dispatch and scalar tails will need to retain the unaligned-row behavior fixed here.

@carrerasdarren-cell
carrerasdarren-cell force-pushed the fix-unaligned-rgb16-strides branch from a8916a1 to 72e07e3 Compare August 7, 2026 07:25
Caller-supplied avifRGBImage buffers may use padded byte strides. Use unaligned-safe 16-bit sample access throughout RGB conversion and alpha handling, address F16 rows with byte arithmetic, and fall back from libyuv when its typed buffers would be unaligned.

Add regression coverage for unaligned odd-stride conversion in both directions and for high-depth alpha multiply and unmultiply.
@carrerasdarren-cell
carrerasdarren-cell force-pushed the fix-unaligned-rgb16-strides branch from 72e07e3 to 433c33d Compare August 7, 2026 07:26
@wantehchang

Copy link
Copy Markdown
Member

Thank you for the pull request. Alignment of high bit depth buffers is an API contract. We don't need to accommodate for unaligned buffers or odd strides.

@wantehchang wantehchang closed this Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants