[fix](be) Harden FileScannerV2 schema and reader edge cases#65548
Open
Gabriel39 wants to merge 1 commit into
Open
[fix](be) Harden FileScannerV2 schema and reader edge cases#65548Gabriel39 wants to merge 1 commit into
Gabriel39 wants to merge 1 commit into
Conversation
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: FileScannerV2 treated valid empty splits as scan failures, localized promoted nested struct predicates with mismatched operand types, retained a process-global Parquet page-cache range index, and rejected whole Parquet schemas when unprojected leaves used unsupported logical types. This change skips and counts empty splits, casts localized nested leaves back to table types, scopes range metadata to each file reader, and defers unsupported-type errors until projection. Regression fixtures and commented examples cover empty CSV input, mixed Parquet struct leaf types, repeated multi-file scans, and an unprojected TIME_MILLIS column.
### Release note
Fix FileScannerV2 compatibility for empty files, evolved nested predicates, and Parquet files containing unprojected unsupported columns; reduce Parquet page-cache range-index contention and lifetime.
### Check List (For Author)
- Test: Regression test / Unit Test
- 55 targeted BE ASAN unit tests
- test_file_scanner_v2_review_fixes regression suite
- Behavior changed: Yes (valid empty splits are skipped; unsupported Parquet logical leaves fail only when projected)
- Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
/review |
Contributor
Author
|
run buildall |
Contributor
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 20th, 2026 12:23 AM. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
Contributor
TPC-H: Total hot run time: 30262 ms |
Contributor
TPC-DS: Total hot run time: 180840 ms |
Contributor
ClickBench: Total hot run time: 24.99 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
/review |
Contributor
|
Codex automated review failed and did not complete. Error: You've hit your usage limit. Visit https://chatgpt.com/codex/settings/usage to purchase more credits or try again at Jul 20th, 2026 12:23 AM. Please inspect the workflow logs and rerun the review after the underlying issue is resolved. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: FileScannerV2 treated valid empty splits as scan failures, localized promoted nested struct predicates with mismatched operand types, retained a process-global Parquet page-cache range index, and rejected whole Parquet schemas when unprojected leaves used unsupported logical types. This change skips and counts empty splits, casts localized nested leaves back to table types, scopes range metadata to each file reader, and defers unsupported-type errors until projection. Regression fixtures and commented examples cover empty CSV input, mixed Parquet struct leaf types, repeated multi-file scans, and an unprojected TIME_MILLIS column.
Release note
Fix FileScannerV2 compatibility for empty files, evolved nested predicates, and Parquet files containing unprojected unsupported columns; reduce Parquet page-cache range-index contention and lifetime.
Check List (For Author)