Core: Fix FixedByteBufferWriter to use writeFixed() for Avro fixed[N] fields - #17502
Open
DuanRuixiao wants to merge 2 commits into
Open
Core: Fix FixedByteBufferWriter to use writeFixed() for Avro fixed[N] fields#17502DuanRuixiao wants to merge 2 commits into
DuanRuixiao wants to merge 2 commits into
Conversation
… fields FixedByteBufferWriter.write() was calling encoder.writeBytes() which prepends a zigzag-encoded length prefix before the data. For Avro fixed[N] fields this is wrong — the reader consumes exactly N bytes with no prefix, so the stray length byte spills into the next field and corrupts it. For example, writing a FIXED(3) partition value [AB CD EF] produced [06 AB CD EF] on disk; the reader consumed [06 AB CD] as the partition value and the leftover [EF] corrupted the subsequent record_count field, decoding it as -568 instead of the actual row count. Fix FixedByteBufferWriter to extract the bytes and call encoder.writeFixed() (exact N bytes, matching the sibling FixedWriter for byte[]). Symmetrically fix InternalReader to use a new FixedByteBufferReader that calls decoder.readFixed() for Avro fixed fields instead of readBytes(), keeping write and read consistent. Add TestFixedByteBufferWriter with two regression tests: - testFixedWriterProducesExactBytes: verifies the encoder emits exactly N bytes with no length prefix - testManifestRoundTripWithFixedPartition: verifies record_count and partition value survive a manifest write/read cycle for a FIXED(N) identity-partition column
3 tasks
Contributor
|
Thanks for tracking this down, @DuanRuixiao. The root cause is right: A couple of things: 1. Iceberg's own reader isn't affected. 2. The fix flips the on-disk format both ways.
So the new reader misreads every existing manifest with a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
FixedByteBufferWriter.write()was callingencoder.writeBytes(), which prepends a zigzag-encoded length prefix before the payload. For Avrofixed[N]fields this is wrong — the reader expects exactly N bytes with no prefix. The stray length byte spills into the next field in the record and corrupts it.For example, writing a
FIXED(3)partition value[AB CD EF]produced[06 AB CD EF]on disk (where06iszigzag(3)). The reader then consumed[06 AB CD]as the partition value, and the leftover[EF]corrupted the subsequentrecord_countfield, decoding it as-568instead of the actual row count.The symmetric read path in
InternalReaderhad the same issue:case FIXED:fell through tocase BYTES:and usedbyteBuffers()(decoder.readBytes()), which reads a length-prefixed byte sequence instead of a fixed-size one.Fix
ValueWriters:FixedByteBufferWriter.write()now extracts the bytes into abyte[]and callsencoder.writeFixed()(exact N bytes, no prefix), matching the existingFixedWriterforbyte[].ValueReaders: AddFixedByteBufferReaderthat callsdecoder.readFixed(bytes, 0, length)/decoder.skipFixed(length). Expose it viaValueReaders.fixedBuffers(int length).InternalReader:case FIXED:now returnsValueReaders.fixedBuffers(primitive.getFixedSize())instead of falling through to the bytes reader.Tests
TestFixedByteBufferWriteradds two regression tests:testFixedWriterProducesExactBytes— directly verifies the encoder emits exactly N bytes with no length prefix.testManifestRoundTripWithFixedPartition— writes and reads back a manifest with aFIXED(3)identity-partition column; asserts both the partition value andrecord_countsurvive the round-trip uncorrupted.