feat(pruning): expose IN-list rewrite size cap as a config option - #24074
Open
zhuqi-lucas wants to merge 4 commits into
Open
feat(pruning): expose IN-list rewrite size cap as a config option#24074zhuqi-lucas wants to merge 4 commits into
zhuqi-lucas wants to merge 4 commits into
Conversation
Issue: apache#24059 `PruningPredicate` rewrites `col IN (v1..vn)` into a chain of per-value min/max checks (via `build_predicate_expression`), but only when `n` is below a hardcoded `MAX_LIST_VALUE_SIZE_REWRITE = 20`. Beyond that, the IN branch falls through to `unhandled_hook`, which by default returns `TRUE` — so row-group / file-range statistics pruning does not fire at all for IN lists longer than 20. This is problematic for query patterns that pass a batch of identifiers as `col IN (...)` (REST endpoints filtering by a page of ~25-100 values, ORM-generated `WHERE id IN (25 items)` queries, batched crawlers). On a table sorted by `col`, the reader is forced to materialize the filter column across every row group instead of skipping row groups whose stats disagree with the IN set. ## Changes - Add `datafusion.execution.parquet.pruning_max_in_list_size: usize` (default `20`, preserving existing behaviour) to `TableParquetOptions` next to `max_predicate_cache_size`. - Add `PredicateRewriter::with_max_in_list_size(usize) -> Self` builder, mirroring the existing `with_unhandled_hook`. - Add `PruningPredicate::try_new_with_max_in_list_size` variant. - Add `build_pruning_predicate_with_max_in_list_size` variant of the public helper. - Make `MAX_LIST_VALUE_SIZE_REWRITE` `pub const` so callers can reference the historical default explicitly. - Wire the value through `datasource-parquet`: - `ParquetSource::pruning_max_in_list_size()` reads from `TableParquetOptions.global`. - `ParquetMorselizer` / `PreparedParquetOpen` / `RowGroupPruner` carry the value alongside `max_predicate_cache_size`. - `build_pruning_predicates` (opener) accepts the size and forwards to `build_pruning_predicate_with_max_in_list_size`. ## Backward compatibility - Public `PruningPredicate::try_new` and `build_pruning_predicate` are preserved as thin wrappers passing the historical default. - Internal `build_predicate_expression` takes a new `usize` parameter (crate-private). - Default value of the config option is `20`, so behaviour is unchanged unless the option is set explicitly. ## Tests - `row_group_predicate_in_list_rewritten_at_raised_cap`: `PredicateRewriter::with_max_in_list_size(32)` rewrites a 25-item IN into per-value min/max checks instead of falling through to `true`. - `row_group_predicate_in_list_disabled_at_zero_cap`: cap = 0 skips the IN rewrite even for small lists (opt-out path). - Existing `row_group_predicate_in_list_to_many_values` continues to pass, guarding the default-20 behaviour.
github-actions
Bot
added
common
Contributor
There was a problem hiding this comment.
Pull request overview
This PR makes the IN (...)-list rewrite cap used by PruningPredicate configurable (instead of a hardcoded 20), and plumbs that setting through the parquet datasource so row-group / file-range stats pruning can remain effective for larger IN lists when users opt in.
Changes:
- Add
datafusion.execution.parquet.pruning_max_in_list_size(default20) to parquet execution config and thread it throughParquetSource→ opener/morselizer → row-group pruner. - Expose the historical default as
pub const MAX_LIST_VALUE_SIZE_REWRITEand add API variants/builders to pass an explicit cap (with_max_in_list_size,try_new_with_max_in_list_size,build_pruning_predicate_with_max_in_list_size). - Add unit tests covering raised-cap rewrite behavior and cap=0 opt-out behavior.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| datafusion/pruning/src/pruning_predicate.rs | Adds configurable IN-list rewrite cap to predicate rewriting/pruning APIs and tests it. |
| datafusion/pruning/src/lib.rs | Re-exports the new const and helper function as part of the public pruning API. |
| datafusion/datasource-parquet/src/source.rs | Reads the new config from TableParquetOptions.global and propagates it into pruning predicate construction. |
| datafusion/datasource-parquet/src/push_decoder.rs | Stores and applies the cap when (re)building pruning predicates in RowGroupPruner. |
| datafusion/datasource-parquet/src/opener/mod.rs | Threads the cap through ParquetMorselizer/PreparedParquetOpen and uses the new helper to build pruning predicates. |
| datafusion/common/src/file_options/parquet_writer.rs | Updates writer options destructuring to account for the newly added parquet option field. |
| datafusion/common/src/config.rs | Introduces the pruning_max_in_list_size parquet execution config option with documentation. |
Suppressed comments (3)
datafusion/pruning/src/pruning_predicate.rs:493
- Docs reference
datafusion.execution.pruning_max_in_list_size, but the actual config option isdatafusion.execution.parquet.pruning_max_in_list_size. Update this reference so callers can find the right setting.
/// Same as [`PruningPredicate::try_new`] but with an explicit cap on the
/// size of `IN (...)` lists rewritten into per-value statistics checks.
/// Query engines typically pass
/// `datafusion.execution.pruning_max_in_list_size` here.
datafusion/pruning/src/pruning_predicate.rs:1404
- This doc comment points to
datafusion.execution.pruning_max_in_list_size, but the new option is namespaced underparquet(datafusion.execution.parquet.pruning_max_in_list_size). Fixing the key avoids confusion for users trying to set the default cap explicitly.
/// Default maximum number of entries in an `IN (...)` list that will be
/// rewritten into a chain of per-value min/max checks by
/// [`build_predicate_expression`]. Callers threading a [`PredicateRewriter`]
/// can override this via [`PredicateRewriter::with_max_in_list_size`], and
/// query engines can wire it from the
/// `datafusion.execution.pruning_max_in_list_size` config option.
pub const MAX_LIST_VALUE_SIZE_REWRITE: usize = 20;
datafusion/pruning/src/pruning_predicate.rs:1445
- This builder method's docs reference
datafusion.execution.pruning_max_in_list_size, but the config key isdatafusion.execution.parquet.pruning_max_in_list_size. Update the docs to match the actual option name.
/// The default (see [`MAX_LIST_VALUE_SIZE_REWRITE`]) preserves the
/// historical behaviour. Callers wiring config through can override via
/// `datafusion.execution.pruning_max_in_list_size`.
pub fn with_max_in_list_size(mut self, max_in_list_size: usize) -> Self {
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+399
to
+402
| /// Same as [`build_pruning_predicate`] but with an explicit cap on the size | ||
| /// of `IN (...)` lists rewritten into per-value statistics checks. Query | ||
| /// engines typically pass `datafusion.execution.pruning_max_in_list_size` | ||
| /// here. |
|
Thank you for opening this pull request! Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch). Details |
github-actions
Bot
added
auto detected api change
The new ParquetOptions::pruning_max_in_list_size field was not carried through the proto layer, so every explicit ParquetOptions initializer in datafusion-proto-common and datafusion-proto failed to compile (E0063), which also cascaded into the MSRV job. - Add uint64 pruning_max_in_list_size = 38 to the ParquetOptions proto message and regenerate proto-common (prost.rs, pbjson.rs) and proto-models (datafusion_proto_common.rs). - Map the field in proto-common from_proto/to_proto and in the proto crate's file_formats TryFromProto/IntoProto for TableParquetOptions. - Reorder the datafusion_pruning import (cargo fmt) and drop the now unused build_pruning_predicate import.
zhuqi-lucas
force-pushed
the
qizhu/config-pruning-in-list-rewrite-size
branch
from
August 4, 2026 06:32
038b3ee to
5f4cfa1
Compare
- cargo doc: the public MAX_LIST_VALUE_SIZE_REWRITE doc linked the private build_predicate_expression via an intra-doc link; demote it to a code span and correct the config path to datafusion.execution.parquet.pruning_max_in_list_size. - Regenerate configs.md for the new pruning_max_in_list_size option. - Add the two pruning_max_in_list_size rows to information_schema.slt (SHOW ALL and the df_settings description listing).
github-actions
Bot
added
documentation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24074 +/- ##
========================================
Coverage 80.88% 80.89%
========================================
Files 1102 1102
Lines 375892 376195 +303
Branches 375892 376195 +303
========================================
+ Hits 304047 304314 +267
- Misses 53740 53771 +31
- Partials 18105 18110 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
alamb
reviewed
Aug 4, 2026
alamb
left a comment
alamb
left a comment
Contributor
There was a problem hiding this comment.
Looks good @zhuqi-lucas . I had some suggestions on API design and naming, but the overall idea makes a lot of sense
| enable_row_group_stats_pruning: false, | ||
| coerce_int96: None, | ||
| max_predicate_cache_size: None, | ||
| pruning_max_in_list_size: MAX_LIST_VALUE_SIZE_REWRITE, |
Contributor
There was a problem hiding this comment.
it is strange to me that these names are not the same -- I would expect something like
pruning_max_in_list_size: PRUNING_MAX_IN_LIST_SIZE,| predicate, | ||
| file_schema, | ||
| predicate_creation_errors, | ||
| MAX_LIST_VALUE_SIZE_REWRITE, |
Contributor
There was a problem hiding this comment.
Why not just add the parameter to build_pruning_predicate ?
If we are going to introduce a new API, perhaps we can make one that is more future proof, like a builder
let pruning_predicate = PruningPredicaateBuilder::new()
.with_file_schema(file_schema)
.with_error_counter(predicate_creation_errors)
.build(predicate)?;That way if we add new parameters we have a place to put them
| /// before calling this method to make sure the expressions can be used for pruning. | ||
| pub fn try_new(mut expr: Arc<dyn PhysicalExpr>, schema: SchemaRef) -> Result<Self> { | ||
| pub fn try_new(expr: Arc<dyn PhysicalExpr>, schema: SchemaRef) -> Result<Self> { | ||
| Self::try_new_with_max_in_list_size(expr, schema, MAX_LIST_VALUE_SIZE_REWRITE) |
Contributor
There was a problem hiding this comment.
Same comment above related to simplifying this API via a builder rather than more methods
Comment on lines
+1192
to
+1205
| /// Maximum number of values in an `IN (...)` list for which the | ||
| /// pruning predicate will rewrite the list into a chain of per-value | ||
| /// statistics checks. Lists longer than this fall back to the | ||
| /// unhandled-predicate hook (defaulting to "keep the container"), | ||
| /// which effectively skips container-level pruning for large IN | ||
| /// lists. | ||
| /// | ||
| /// Higher values keep row-group / file-range statistics pruning | ||
| /// effective for larger IN lists (for example, REST endpoints that | ||
| /// filter by a batch of ~25-100 identifiers), at the cost of a | ||
| /// larger rewritten predicate expression evaluated for every | ||
| /// container. Set to 0 to disable the rewrite path entirely. | ||
| /// | ||
| /// The default of 20 preserves the previous hardcoded behaviour. |
Contributor
There was a problem hiding this comment.
I think we could rewrite this to focus more on the end user visible effects to make it clearer what was going on
Suggested change
| /// Maximum number of values in an `IN (...)` list for which the | |
| /// pruning predicate will rewrite the list into a chain of per-value | |
| /// statistics checks. Lists longer than this fall back to the | |
| /// unhandled-predicate hook (defaulting to "keep the container"), | |
| /// which effectively skips container-level pruning for large IN | |
| /// lists. | |
| /// | |
| /// Higher values keep row-group / file-range statistics pruning | |
| /// effective for larger IN lists (for example, REST endpoints that | |
| /// filter by a batch of ~25-100 identifiers), at the cost of a | |
| /// larger rewritten predicate expression evaluated for every | |
| /// container. Set to 0 to disable the rewrite path entirely. | |
| /// | |
| /// The default of 20 preserves the previous hardcoded behaviour. | |
| /// Maximum number of values in an `IN (...)` list for which pruning will | |
| /// occur. Longer lists will not be used to prune files, row groups, or | |
| /// data pages. | |
| /// | |
| /// Higher values help in cases such as a list of | |
| /// of ~25-100 identifiers, but also makes the predicate | |
| /// more expensive to evaluate. Set to 0 to disable the IN (..) list pruning entirely | |
| /// | |
| /// Defaults to 20. |
| /// container. Set to 0 to disable the rewrite path entirely. | ||
| /// | ||
| /// The default of 20 preserves the previous hardcoded behaviour. | ||
| pub pruning_max_in_list_size: usize, default = 20 |
Contributor
There was a problem hiding this comment.
Also I suggest changing this to be something more conisstent with the others names like max_predicate_cache_size
Perhaps something likemax_in_list_size or max_in_list_pruning_size
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale
PruningPredicaterewritescol IN (v1..vn)into a chain of per-value min/max checks (viabuild_predicate_expression), but only whenn <= MAX_LIST_VALUE_SIZE_REWRITE— currently a hardcoded20. Beyond that, the IN branch falls through tounhandled_hook, which by default returnsTRUE, so row-group and file-range statistics pruning does not fire at all for IN lists longer than 20.This is problematic for query patterns that pass a batch of identifiers as
col IN (...)— REST endpoints filtering by a page of ~25-100 values, ORM-generatedWHERE id IN (25 items)queries, batched crawlers. On a table sorted bycol, the reader is forced to materialize the filter column across every row group instead of skipping row groups whose stats disagree with the IN set.Full context in #24059.
What changes are included in this PR?
datafusion.execution.parquet.pruning_max_in_list_size: usize(default20, preserving existing behaviour), placed next tomax_predicate_cache_sizeonTableParquetOptions.global.MAX_LIST_VALUE_SIZE_REWRITEpromoted topub constso callers can reference the historical default explicitly.PredicateRewriter::with_max_in_list_size(usize) -> Selfbuilder, mirroring the existingwith_unhandled_hook.PruningPredicate::try_new_with_max_in_list_sizevariant.build_pruning_predicate_with_max_in_list_sizevariant of the public helper.datasource-parquet:ParquetSource::pruning_max_in_list_size()reads fromTableParquetOptions.global, propagates throughParquetMorselizer→PreparedParquetOpen→RowGroupPruner, then flows intobuild_pruning_predicatesat the opener andbuild_pruning_predicate_with_max_in_list_sizeinside the dynamic row-group pruner.Internal
build_predicate_expressiongains a newusizeparameter (crate-private).Backward compatibility
PruningPredicate::try_newandbuild_pruning_predicateare preserved as thin wrappers that pass the historicalMAX_LIST_VALUE_SIZE_REWRITEdefault. All existing callers continue to work with unchanged behaviour.20, so behaviour is unchanged unless the option is set explicitly.Are these changes tested?
Two new unit tests in
datafusion-pruning:row_group_predicate_in_list_rewritten_at_raised_cap:PredicateRewriter::with_max_in_list_size(32)rewrites a 25-item IN into per-value min/max checks OR'd together, instead of falling through totrue.row_group_predicate_in_list_disabled_at_zero_cap:cap = 0skips the IN rewrite even for small lists (opt-out path).The existing
row_group_predicate_in_list_to_many_valuescontinues to pass, guarding the default-20 behaviour.Are there any user-facing changes?
Yes — one new config option (
datafusion.execution.parquet.pruning_max_in_list_size, default20). Users who want row-group / file-range pruning for IN lists longer than 20 items can raise it (e.g.,SET datafusion.execution.parquet.pruning_max_in_list_size = 128).New public API on
datafusion-pruning:MAX_LIST_VALUE_SIZE_REWRITE: usize(re-exported)PredicateRewriter::with_max_in_list_size(usize) -> SelfPruningPredicate::try_new_with_max_in_list_size(expr, schema, size)build_pruning_predicate_with_max_in_list_size(predicate, schema, errors, size)