[SPARK-58315][SQL] Refactor V2 scan statistics handling#57487
Open
yyanyy wants to merge 1 commit into
Open
Conversation
Refactor how a DSv2 scan reports statistics so the work matches what the optimizer needs, and so source-reported statistics stay accurate after filter pushdown. Aligns the DSv2 path with the v1 (LogicalRelation / CatalogStatistics.toPlanStats) path. Benefits: - Stats are computed on demand: full stats only when CBO or plan stats are enabled, otherwise a cheaper size-only estimate via the new estimateSizeInBytes(). Previously the scan always fetched full stats. - Source stats stay accurate across pushdown: a connector whose stats do not reflect the fully pushed filters can set reflectsFullyPushedDownFilters() to false, and Spark re-adds those filters above the scan so FilterEstimation adjusts the stats. - V1 scans that report statistics now surface them through V1ScanWrapper (previously dropped). Changes: - computeStats gates on `cboEnabled || planStatsEnabled` (mirrors v1). - Add estimateSizeInBytes() and reflectsFullyPushedDownFilters() default methods to SupportsReportStatistics. - transformV2Stats infers sizeInBytes from numRows when no size is reported (matching v1 toPlanStats); drop the dead defaultRowCount parameter. - V1ScanWrapper implements SupportsReportStatistics. - Extract the shared statistics logic into V2StatisticsUtils (null-safe). Tests: CBO / plan-stats gating, size inference (including the zero-row case), null/empty statistics, V1ScanWrapper delegation, and post-pushdown filter re-adding (including recursion through stacked residual filters).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refactor how a DataSource V2 (DSv2) scan reports statistics, aligning it with
the v1 (
LogicalRelation/CatalogStatistics.toPlanStats) path.computeStatsgates oncboEnabled || planStatsEnabled: full statisticsonly when needed, otherwise a cheaper size-only estimate via the new
SupportsReportStatistics.estimateSizeInBytes()default method.SupportsReportStatistics.reflectsFullyPushedDownFilters()default method(
true). When a connector returnsfalse,V2ScanRelationPushDownre-addsthe fully pushed, deterministic filters above the scan so
FilterEstimationadjusts the stats. The rewrite is row-equivalent, idempotent, and best-effort
under column pruning (pruned-reference predicates are dropped).
transformV2StatsinferssizeInBytesfrom the row count when no size isreported (matching v1); the dead
defaultRowCountparameter is removed.V1ScanWrapperimplementsSupportsReportStatistics, delegating to thewrapped v1 scan so its stats are no longer dropped.
V2StatisticsUtils(null-safe).Both new methods have defaults that preserve existing behavior, so existing
connectors need no change.
Why are the changes needed?
To let DSv2 connectors produce statistics more cheaply and let Spark plan on
more accurate statistics, matching v1:
sizeInBytes, so a connector can skip the expensive row/column statistics.FilterEstimationinstead of planning on too-large estimates, improving plan choices.
now used instead of the default size, at no extra cost to the connector.
Does this PR introduce any user-facing change?
New API introduced but no behavior change by default except for the following cases:
default size was used)
SupportsReportStatisticsnow has its reported statistics surfaced (previously dropped).Both changes make the estimates more accurate and align with the v1 path.
How was this patch tested?
New unit and end-to-end tests in
DataSourceV2RelationSuiteandDataSourceV2Suitecover: the CBO / plan-stats gating, size inference(including the zero-row case), null/empty statistics handling,
V1ScanWrapperdelegation, and the post-pushdown filter re-adding (including the recursion
through stacked residual filters).
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code