Core: Add option to skip column stats in SnapshotChanges - #17490
Core: Add option to skip column stats in SnapshotChanges#17490gtrettenero wants to merge 2 commits into
Conversation
| private static final Map<Integer, ByteBuffer> UPPER_BOUNDS = | ||
| ImmutableMap.of(3, Conversions.toByteBuffer(Types.IntegerType.get(), 2)); | ||
|
|
||
| private static final DataFile FILE_WITH_STATS = |
There was a problem hiding this comment.
nan_value_counts is passed as null in the fixture, so testIncludeColumnStatsFalseDropsStats cannot detect a regression where a real non-null nan_value_counts is accidentally retained. Add NAN_VALUE_COUNTS = ImmutableMap.of(3, 1L) to the fixture and assert file.nanValueCounts().isNull() (and assertThat(file.nanValueCounts()).isNull() in the retain-default test). Adding explicit assertions on file_format and content in the same skip-test would further tighten coverage (both survive the projection by construction, but explicit assertions catch future breakage of that invariant).
There was a problem hiding this comment.
Added NAN_VALUE_COUNTS to the Metrics fixture with assertions in both the retain-default and skip tests. Also added the file_format and content assertions. The projection check for content is testIncludeColumnStatsFalseRetainsDeleteFileFields, which fails if delete manifests use the data projection.
| entry -> { | ||
| if (entry.status() == ManifestEntry.Status.ADDED) { | ||
| return Pair.of(ManifestEntry.Status.ADDED, entry.file().copy()); | ||
| return Pair.of(ManifestEntry.Status.ADDED, entry.file().copy(columnStats)); |
There was a problem hiding this comment.
Does this overlap with copyWithoutStats?
There was a problem hiding this comment.
ContentFile.copy(boolean) is a default method: withStats ? copy() : copyWithoutStats(). So copy(false) is copyWithoutStats().
The idea for having it is that manifestColumns() delegates to BaseScan.scanColumns() and if a stats column were ever added there for scan-planning reasons, includeColumnStats(false) would silently start returning stats again. But as of today it I remove the boolean then nothing would change since the stats columns are never decoded in the first place when includeColumnStats(false)
SnapshotChangesreads manifests with no projection and copies added files withentry.file().copy(), so the per-column statistics maps (column_sizes,value_counts,null_value_counts,nan_value_counts,lower_bounds,upper_bounds) are always decoded andretained. Callers that only need file identity — path, size, partition, spec — pay for stats they
never read, which on wide tables dominates the retained heap of the cached file lists. The removed
path already uses
copyWithoutStats().Adds
SnapshotChanges.Builder#includeColumnStats(boolean), defaulting totrue. Whenfalse,manifests are read with
BaseScan.scanColumns(manifest.content())— the same curated projectionPartitionsTableandPartitionStatsHandleralready use for direct manifest reads — and files arecopied without stats. Selecting by manifest content type keeps
content,referenced_data_file,content_offset,content_size_in_bytesandequality_idson delete files; a singlecaller-supplied column list would not be correct for both manifest types.
The default maps to
ManifestReader.ALL_COLUMNS, whichSchema#selectshort-circuits, so existingcallers are unaffected. The polarity is inverted relative to
Scan#includeColumnStats(), which isopt-in because scans default to dropping stats.
Testing
Four tests in
TestSnapshotChangescover the default, the opt-out, partition values on apartitioned table, and
content/equalityFieldIdson delete files.