[VL] test(iceberg): Fix disabled iceberg row group bytes test#12509
Open
infvg wants to merge 1 commit into
Open
[VL] test(iceberg): Fix disabled iceberg row group bytes test#12509infvg wants to merge 1 commit into
infvg wants to merge 1 commit into
Conversation
Member
|
related #12494 |
Contributor
|
This issue was introduced by a recent PR where the code writes the batch before checking if the buffer size exceeds bytesInRowGroup, causing the row group to exceed the limit. Unlike Velox, the parquet-java reference implementation strictly ensures the row group size never exceeds bytesInRowGroup. Since Velox's current behavior deviates from this, I've opened a community issue to discuss how to define this in Velox. We should wait for that decision before making any changes here, as a fix on the Velox side would be preferred over patching it in Gluten. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, the row group bytes value was a hard boundary of the uncompressed value. Now the implementation causes the buffer to reach or slightly exceed the threshold.