[vortex] Read the primary-key format's physical _SEQUENCE_NUMBER column (#8876) - #8877
Open
jordepic wants to merge 1 commit into
Open
[vortex] Read the primary-key format's physical _SEQUENCE_NUMBER column (#8876)#8877jordepic wants to merge 1 commit into
jordepic wants to merge 1 commit into
Conversation
…mn (apache#8876) The vortex reader classified any projected field named _SEQUENCE_NUMBER or _ROW_ID as a row-tracking field to synthesize, mapping its projection index to -1. But _SEQUENCE_NUMBER doubles as the primary-key file format's physical sequence column, so every vortex read of a primary-key table crashed with ArrayIndexOutOfBoundsException on the first key-value deserialization. Reserve synthesis for fields the file does not store physically: when the data schema carries the field (by id or name), map it to its physical column like any other.
Contributor
Author
|
cc @JingsongLi — companion find to #8874, from the same cross-implementation state-backend testing: vortex primary-key reads crash on the physical sequence column being reserved for row-tracking synthesis. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fixes #8876.
The vortex reader classified any projected field named
_SEQUENCE_NUMBERor_ROW_IDas a row-tracking field to synthesize, mapping its projection index to -1._SEQUENCE_NUMBERdoubles as the primary-key file format's physical sequence column, so every vortex read of a primary-key table crashed withArrayIndexOutOfBoundsExceptionat the first key-value deserialization. Reserve synthesis for fields the file does not store physically: when the data schema carries the field (by id or name), map it to its physical column like any other.Verified beyond the unit test with an external end-to-end setup: primary-key tables in vortex format under deletion-vector mode, written by paimon-rust and lookup-compacted by this repo's writer, round-trip cleanly with this fix (they crash without it).
Tests
VortexRecordsReaderTest#testPhysicalSequenceColumnIsReadNotSynthesized; the two existing schema-handling tests updated for the added data-schema parameter.API and Format
No changes.
Documentation
No changes.