[fix](be) Ignore NaN Parquet min max statistics

[Gabriel39]

What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:

FileScannerV2 treated Parquet FLOAT and DOUBLE min/max statistics containing NaN as valid. Doris floating-point comparisons can then interpret NaN as equal to predicate bounds, allowing range or inequality predicates to incorrectly prune row groups or pages that still contain matching finite values. The same invalid statistics could also affect min/max aggregate pushdown.

This change validates native Parquet min/max bounds before decoding them. If either floating-point bound is NaN, the footer or ColumnIndex min/max is ignored and pruning conservatively keeps the data. Null-count metadata remains available, finite FLOAT/DOUBLE statistics retain their existing behavior, and non-floating-point statistics are unchanged.

Release note

Fix missing rows when Parquet floating-point min/max statistics contain NaN.

Check List (For Author)

• Test: Unit Test
  • ParquetStatisticsTransformTest.IgnoresNaNFloatAndDoubleMinMax
  • ParquetStatisticsTransformTest.IgnoresNaNFloatAndDoubleColumnIndexMinMax
  • Existing finite min/max conversion regression test
• Behavior changed: Yes. NaN floating-point min/max statistics are ignored for pruning.
• Does this need documentation: No

Validation on the designated Linux build host:

• build-support/check-format.sh
• Targeted BE unit tests: 3 tests passed
• run-clang-tidy.sh was attempted; analysis is blocked by pre-existing diagnostics in the source file and jni-util.h toolchain static assertions unrelated to this change.

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Summary:
I found one issue that should be fixed before merge: the new ColumnIndex NaN handling aborts page-index pruning for an entire slot after one NaN page instead of marking only that page's min/max unusable.

Critical checkpoints:

• Goal/test: Footer statistics and MIN/MAX aggregate pushdown fall back conservatively when float/double min/max contains NaN. The added tests cover direct transform helpers, but the multi-page page-index behavior is not covered.
• Scope: The change is focused to Parquet statistics conversion plus BE unit tests.
• Concurrency/lifecycle/config/persistence/compatibility: No new concurrency, lifecycle, configuration, persistence, storage-format, or FE/BE protocol changes found.
• Parallel paths: Row-group pruning and aggregate pushdown share the fixed footer-stat path; page-index pruning has the granularity issue noted inline.
• Performance: A single invalid page now disables avoidable page-index pruning for later finite pages in the same slot.
• Testing: I could not run BE UTs in this runner because .worktree_initialized, thirdparty/installed, and thirdparty/installed/bin/protoc are missing; the PR body reports targeted BE UTs on the designated host.

Subagent conclusions:

• optimizer-rewrite found no additional candidates.
• tests-session-config proposed TSC-1, which I accepted as MAIN-1 and submitted inline.
• Convergence round 1 ended with both live subagents replying NO_NEW_VALUABLE_FINDINGS for the same MAIN-1-only final comment set.

User focus: No additional user-provided review focus was supplied.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Requesting changes for one remaining Parquet statistics pruning issue.

Critical checkpoint conclusions:

• Goal/test: the PR fixes NaN min/max being used for FLOAT/DOUBLE footer and page-index pruning, and tests cover NaN footer stats plus page-local ColumnIndex behavior. One null-count pruning path is still regressed.
• Scope: the code change is small and focused on Parquet statistics conversion and tests.
• Concurrency/lifecycle: no new concurrency, locks, thread lifecycle, or static initialization concerns found.
• Config/compatibility: no new config, storage format, or FE-BE protocol compatibility changes found.
• Parallel paths: footer stats and ColumnIndex stats were both reviewed; the existing thread for slot-wide page-index fallback is covered by the follow-up commit, but the remaining null-count issue affects both footer and page-index stats.
• Tests: added unit tests cover NaN min/max invalidation and finite page continuation, but not IS NULL pruning when NaN invalidates only min/max while null-count metadata is valid.
• Observability/transactions/persistence/data writes: not applicable for this change.
• Performance: the follow-up preserves page-index pruning for later finite pages; the remaining issue is unnecessary loss of valid null-count pruning.

Subagent conclusions: optimizer-rewrite found no additional valuable optimizer/rewrite or scan-pruning semantic candidates beyond the existing thread and MAIN-1; tests-session-config found no additional test/session/config/style candidates. Convergence round 1 ended with both live subagents replying NO_NEW_VALUABLE_FINDINGS for the same ledger/comment set after the accepted candidate update.

User focus: no additional user-provided review focus was present.

Validation: static review only in this runner. The base commit is absent in the shallow checkout, thirdparty/installed and thirdparty/installed/bin/protoc are missing, and the available local clang-format is not the required v16, so I did not run BE unit tests or official format checks.

[Gabriel39]

run buildall

[Gabriel39]

/review

[github-actions]

Reviewed current head 04aa972. I do not have any new inline findings for this round.

Checkpoint conclusions:

• Goal/test coverage: The change ignores NaN FLOAT/DOUBLE Parquet min/max statistics, keeps range pruning conservative, preserves null-count pruning, and adds targeted BE unit coverage for footer stats, page-index stats, mixed NaN/finite pages, and null-count preservation.
• Scope: The implementation is focused in parquet statistics transformation, zonemap creation, and tests.
• Concurrency/lifecycle: No new concurrency, thread lifecycle, persistence, transaction, or cross-version protocol surface found.
• Parallel paths: Row-group footer pruning, page-index pruning, and MIN/MAX aggregate pushdown were checked. Invalid min/max now falls back conservatively for pruning and aggregate pushdown still requires valid has_min_max.
• Configuration/session behavior: No new config item or session propagation path was added.
• Error handling/observability: Existing conservative unsupported-statistics behavior is preserved; no new logging or metrics requirement found for this narrow metadata-validation fix.
• Tests/results: clang-format-16 --dry-run --Werror passed locally for the three changed files. I could not run the targeted BE unit test in this checkout because thirdparty/installed and the test binary are absent. GitHub status still has BE/compile jobs pending, and the visible macOS BE UT failure is a runner JDK setup failure before changed code is built.

Existing threads:

• The earlier page-index NaN fallback thread is already known and the current head appears to address it by making invalid min/max page-local.
• The earlier null-count preservation thread is already known and the current head appears to address it with a pass_all zonemap that preserves has_null/has_not_null.

Subagent conclusions:

• optimizer-rewrite found no new valuable candidate beyond the existing NaN page-index and null-count threads.
• tests-session-config found no new valuable candidate, recorded both older concerns as duplicates of existing threads, and noted the same validation limits.
• Final convergence round 1 ended with both live subagents replying NO_NEW_VALUABLE_FINDINGS for the same ledger and empty proposed final comment set.

User focus: no additional user-provided review focus was present.

1

[hello-stephen]

BE UT Coverage Report

Increment line coverage 100.00% (35/35) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	56.90% (23545/41376)
Line Coverage	40.57% (230223/567461)
Region Coverage	36.47% (181889/498749)
Branch Coverage	37.55% (81205/216287)

[Gabriel39]

Superseded by #65500, which consolidates and validates the four related fixes.