Skip to content

Core: Add option to skip column stats in SnapshotChanges - #17490

Open
gtrettenero wants to merge 2 commits into
apache:mainfrom
gtrettenero:snapshot-changes-include-column-stats
Open

Core: Add option to skip column stats in SnapshotChanges#17490
gtrettenero wants to merge 2 commits into
apache:mainfrom
gtrettenero:snapshot-changes-include-column-stats

Conversation

@gtrettenero

Copy link
Copy Markdown
Contributor

SnapshotChanges reads manifests with no projection and copies added files with
entry.file().copy(), so the per-column statistics maps (column_sizes, value_counts,
null_value_counts, nan_value_counts, lower_bounds, upper_bounds) are always decoded and
retained. Callers that only need file identity — path, size, partition, spec — pay for stats they
never read, which on wide tables dominates the retained heap of the cached file lists. The removed
path already uses copyWithoutStats().

Adds SnapshotChanges.Builder#includeColumnStats(boolean), defaulting to true. When false,
manifests are read with BaseScan.scanColumns(manifest.content()) — the same curated projection
PartitionsTable and PartitionStatsHandler already use for direct manifest reads — and files are
copied without stats. Selecting by manifest content type keeps content, referenced_data_file,
content_offset, content_size_in_bytes and equality_ids on delete files; a single
caller-supplied column list would not be correct for both manifest types.

The default maps to ManifestReader.ALL_COLUMNS, which Schema#select short-circuits, so existing
callers are unaffected. The polarity is inverted relative to Scan#includeColumnStats(), which is
opt-in because scans default to dropping stats.

Testing

Four tests in TestSnapshotChanges cover the default, the opt-out, partition values on a
partitioned table, and content/equalityFieldIds on delete files.

@github-actions github-actions Bot added the core label Aug 3, 2026
private static final Map<Integer, ByteBuffer> UPPER_BOUNDS =
ImmutableMap.of(3, Conversions.toByteBuffer(Types.IntegerType.get(), 2));

private static final DataFile FILE_WITH_STATS =

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nan_value_counts is passed as null in the fixture, so testIncludeColumnStatsFalseDropsStats cannot detect a regression where a real non-null nan_value_counts is accidentally retained. Add NAN_VALUE_COUNTS = ImmutableMap.of(3, 1L) to the fixture and assert file.nanValueCounts().isNull() (and assertThat(file.nanValueCounts()).isNull() in the retain-default test). Adding explicit assertions on file_format and content in the same skip-test would further tighten coverage (both survive the projection by construction, but explicit assertions catch future breakage of that invariant).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added NAN_VALUE_COUNTS to the Metrics fixture with assertions in both the retain-default and skip tests. Also added the file_format and content assertions. The projection check for content is testIncludeColumnStatsFalseRetainsDeleteFileFields, which fails if delete manifests use the data projection.

@vishnuprakaz vishnuprakaz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

entry -> {
if (entry.status() == ManifestEntry.Status.ADDED) {
return Pair.of(ManifestEntry.Status.ADDED, entry.file().copy());
return Pair.of(ManifestEntry.Status.ADDED, entry.file().copy(columnStats));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this overlap with copyWithoutStats?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ContentFile.copy(boolean) is a default method: withStats ? copy() : copyWithoutStats(). So copy(false) is copyWithoutStats().

The idea for having it is that manifestColumns() delegates to BaseScan.scanColumns() and if a stats column were ever added there for scan-planning reasons, includeColumnStats(false) would silently start returning stats again. But as of today it I remove the boolean then nothing would change since the stats columns are never decoded in the first place when includeColumnStats(false)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants