Skip to content

Core: Fix FixedByteBufferWriter to use writeFixed() for Avro fixed[N] fields - #17502

Open
DuanRuixiao wants to merge 2 commits into
apache:mainfrom
DuanRuixiao:fix-fixed-byte-buffer-writer
Open

Core: Fix FixedByteBufferWriter to use writeFixed() for Avro fixed[N] fields#17502
DuanRuixiao wants to merge 2 commits into
apache:mainfrom
DuanRuixiao:fix-fixed-byte-buffer-writer

Conversation

@DuanRuixiao

Copy link
Copy Markdown

Problem

FixedByteBufferWriter.write() was calling encoder.writeBytes(), which prepends a zigzag-encoded length prefix before the payload. For Avro fixed[N] fields this is wrong — the reader expects exactly N bytes with no prefix. The stray length byte spills into the next field in the record and corrupts it.

For example, writing a FIXED(3) partition value [AB CD EF] produced [06 AB CD EF] on disk (where 06 is zigzag(3)). The reader then consumed [06 AB CD] as the partition value, and the leftover [EF] corrupted the subsequent record_count field, decoding it as -568 instead of the actual row count.

The symmetric read path in InternalReader had the same issue: case FIXED: fell through to case BYTES: and used byteBuffers() (decoder.readBytes()), which reads a length-prefixed byte sequence instead of a fixed-size one.

Fix

  • ValueWriters: FixedByteBufferWriter.write() now extracts the bytes into a byte[] and calls encoder.writeFixed() (exact N bytes, no prefix), matching the existing FixedWriter for byte[].
  • ValueReaders: Add FixedByteBufferReader that calls decoder.readFixed(bytes, 0, length) / decoder.skipFixed(length). Expose it via ValueReaders.fixedBuffers(int length).
  • InternalReader: case FIXED: now returns ValueReaders.fixedBuffers(primitive.getFixedSize()) instead of falling through to the bytes reader.

Tests

TestFixedByteBufferWriter adds two regression tests:

  • testFixedWriterProducesExactBytes — directly verifies the encoder emits exactly N bytes with no length prefix.
  • testManifestRoundTripWithFixedPartition — writes and reads back a manifest with a FIXED(3) identity-partition column; asserts both the partition value and record_count survive the round-trip uncorrupted.

… fields

FixedByteBufferWriter.write() was calling encoder.writeBytes() which
prepends a zigzag-encoded length prefix before the data. For Avro
fixed[N] fields this is wrong — the reader consumes exactly N bytes with
no prefix, so the stray length byte spills into the next field and
corrupts it. For example, writing a FIXED(3) partition value [AB CD EF]
produced [06 AB CD EF] on disk; the reader consumed [06 AB CD] as the
partition value and the leftover [EF] corrupted the subsequent
record_count field, decoding it as -568 instead of the actual row count.

Fix FixedByteBufferWriter to extract the bytes and call
encoder.writeFixed() (exact N bytes, matching the sibling FixedWriter
for byte[]). Symmetrically fix InternalReader to use a new
FixedByteBufferReader that calls decoder.readFixed() for Avro fixed
fields instead of readBytes(), keeping write and read consistent.

Add TestFixedByteBufferWriter with two regression tests:
- testFixedWriterProducesExactBytes: verifies the encoder emits exactly
  N bytes with no length prefix
- testManifestRoundTripWithFixedPartition: verifies record_count and
  partition value survive a manifest write/read cycle for a FIXED(N)
  identity-partition column
@huaxingao

Copy link
Copy Markdown
Contributor

Thanks for tracking this down, @DuanRuixiao. The root cause is right: FixedByteBufferWriter uses writeBytes() for a fixed[N] field instead of writeFixed().

A couple of things:

1. Iceberg's own reader isn't affected. InternalReader's case FIXED falls through to readBytes(), which matches the writer's writeBytes(), so Iceberg reads its own manifests fine (a ManifestFiles round-trip returns record_count = 4). The corruption only shows up in spec-compliant external readers that do readFixed.

2. The fix flips the on-disk format both ways.

writer \ reader old reader new reader
old writer (existing manifests) 4 ❌ corrupted (-568 in the example)
new writer (new manifests) ❌ error 4

So the new reader misreads every existing manifest with a fixed field, and new manifests can't be read by older Iceberg. I think we need an approach that doesn't break existing data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants