[fix](be) Preserve row groups when null count is missing#65499
Closed
Gabriel39 wants to merge 1 commit into
Closed
[fix](be) Preserve row groups when null count is missing#65499Gabriel39 wants to merge 1 commit into
Gabriel39 wants to merge 1 commit into
Conversation
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: FileScannerV2 treated a missing optional Parquet null_count as proof that a row group contained no NULL values. When min/max statistics were present, IS NULL predicates could therefore prune a row group that actually contained NULL rows. Treat unknown null counts conservatively as possibly containing NULL while retaining available min/max statistics, and cover the metadata combination with a unit test.
### Release note
Fix incorrect IS NULL results for Parquet files whose writers omit null_count statistics.
### Check List (For Author)
- Test: Unit Test
- ParquetStatistics* BE unit tests
- Behavior changed: Yes, row groups with unknown Parquet null counts are no longer discarded by IS NULL pruning
- Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
/review |
Contributor
Author
|
run buildall |
Contributor
There was a problem hiding this comment.
Automated review complete. I found no blocking correctness issue and no inline comments to submit.
Critical checkpoint conclusions:
- Goal and tests: The PR fixes FileScannerV2 Parquet row-group pruning when optional
null_countis absent by treating null presence as unknown/possible instead of false. The added BE unit test covers the key metadata shape: min/max retained whilehas_null_countis false. - Scope and focus: The change is narrow and focused: one statistics conversion line plus a targeted unit test.
- Correctness: The row-group zone-map path is conservative for
IS NULLafter the change. Row-level VExpr predicates still execute after metadata pruning. Page-index pruning remains separately conservative when page null counts are unavailable. - Parallel paths: The legacy Parquet reader disables its zone map when row-group
null_countis missing, so it does not repeat the false-prune behavior. Min/max aggregate pushdown only consumeshas_min_max, so the changed null flag does not alter aggregate min/max computation. - Concurrency, lifecycle, config, compatibility, persistence, and write paths: Not involved by this patch.
- Tests and style:
clang-format-16 --dry-run --Werrorpassed on the two changed files. I did not run the BE unit test because this review checkout lacksthirdparty/installedandthirdparty/installed/bin/protoc.
Subagent conclusions:
- optimizer-rewrite: no optimizer/rewrite, semantic-equivalence, parallel join, or aggregate candidates.
- tests-session-config: no tests/session/config, compatibility, expected-output, or CI/style candidates.
- Final convergence round 1 ended with both live subagents returning
NO_NEW_VALUABLE_FINDINGSfor the same empty proposed inline comment set.
User focus: No additional user-provided review focus was present.
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
Superseded by #65500, which consolidates and validates the four related fixes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: FileScannerV2 treated a missing optional Parquet
null_countas proof that a row group contained no NULL values. When min/max statistics were present,IS NULLpredicates could therefore prune a row group that actually contained NULL rows. This change treats unknown null counts conservatively as possibly containing NULL while retaining available min/max statistics, and covers the metadata combination with a unit test.Release note
Fix incorrect
IS NULLresults for Parquet files whose writers omitnull_countstatistics.Check List (For Author)
ParquetStatistics*BE unit tests (7 tests, 0 failures)build-support/check-format.shbuild-support/run-clang-tidy.sh --build-dir be/ut_build_ASANIS NULLpruning