Skip to content

[WIP] fix(parquet): preserve explicit mask with sparse pages#10288

Draft
hhhizzz wants to merge 2 commits into
apache:mainfrom
hhhizzz:fix/parquet-mask-sparse-pages
Draft

[WIP] fix(parquet): preserve explicit mask with sparse pages#10288
hhhizzz wants to merge 2 commits into
apache:mainfrom
hhhizzz:fix/parquet-mask-sparse-pages

Conversation

@hhhizzz

@hhhizzz hhhizzz commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Explicit RowSelectionPolicy::Mask now works when page pruning has loaded only sparse page ranges. Instead of decoding rows whose pages were intentionally skipped, the read plan carries loaded row ranges into a sparse mask cursor and the record batch reader decodes only backed selected segments.

Auto keeps its existing safety behavior by falling back to selectors for unsafe sparse page-loaded masks, so this stays scoped to the mask-preserving correctness path and does not import the broader reference branch's post-filter, cost-model, metrics, benchmark, CI, changelog, or dependency work.

Validation

  • cargo fmt --all --check
  • git diff --check
  • cargo test -p parquet --features arrow --lib selection
  • cargo test -p parquet --features arrow,async --test arrow_reader row_filter
  • cargo test -p parquet --features arrow,async --test arrow_reader predicate_cache

Post-Deploy Monitoring & Validation

No additional operational monitoring required. This is a parquet reader correctness fix covered by unit and row-filter integration regressions; downstream validation should come from normal CI and parquet reader test coverage.


Compound Engineering
GPT-5

@github-actions github-actions Bot added the parquet Changes to the parquet crate label Jul 5, 2026
@hhhizzz hhhizzz marked this pull request as draft July 5, 2026 15:43
@hhhizzz hhhizzz changed the title fix(parquet): preserve explicit mask with sparse pages [WIP] fix(parquet): preserve explicit mask with sparse pages Jul 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

parquet Changes to the parquet crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant