Skip to content

Core: Fix geo bounds in v4 legacy stat maps - #17493

Open
vishnuprakaz wants to merge 4 commits into
apache:mainfrom
vishnuprakaz:v4-geo-bounds-legacy-maps
Open

Core: Fix geo bounds in v4 legacy stat maps#17493
vishnuprakaz wants to merge 4 commits into
apache:mainfrom
vishnuprakaz:v4-geo-bounds-legacy-maps

Conversation

@vishnuprakaz

@vishnuprakaz vishnuprakaz commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

While reading the v4 stats code noticed @dramaticlly's comment on #17322 suspecting that Conversions.toByteBuffer might not handle geometry bounds properly inside ContentStatsBackedMap.bound(). The suspicion is correct. For geometry and geography columns, the bound field in the stats schema is the bounding-box struct (x, y, z, m) built by StatsUtil.geoLowerBound, and Conversions.toByteBuffer has no struct case, so it throws.

A small repro against current main confirmed it:

  java.lang.UnsupportedOperationException: Cannot serialize type: STRUCT
      at org.apache.iceberg.types.Conversions.toByteBuffer(Conversions.java:145)
      at org.apache.iceberg.ContentStatsBackedMap.bound(ContentStatsBackedMap.java:169)
      at org.apache.iceberg.ContentStatsBackedMap.get(ContentStatsBackedMap.java:87)

Once a tracked file carries geo column stats, any access to lowerBounds() or upperBounds() that touches the geo column hits this a predicate on that column, or anything that copies or iterates the stats maps. Nothing reaches this in production today, but #17433 starts reading geo stats from v4 manifests, which puts the crash one call away from the read path.

The fix converts the bounding-box struct back to a GeospatialBound and serializes it with the spec's single-point bound encoding. That's the representation legacy consumers of these maps already expect: Conversions reads that encoding back for geo types, and v3 tables surface geo bounds through the same maps in the same form so the legacy view of a
v4 file stays consistent with v3 behavior, and the bounding-box storage format from the v4 stats design is untouched.

Tests cover geometry and geography columns next to a primitive column, the z-only and m-only encodings, a geo field with no lower bound, and a round-trip through Conversions.fromByteBuffer.

If this is already covered as part of the planned content-stats integration work, happy to fold it into that instead.

@vishnuprakaz

Copy link
Copy Markdown
Contributor Author

cc @dramaticlly @stevenzwu

@github-actions github-actions Bot added the core label Aug 3, 2026
Map.entry(10, GeospatialBound.createXY(5.0, 6.0).toByteBuffer()),
Map.entry(11, GeospatialBound.createXYZM(7.0, 8.0, 9.0, 10.0).toByteBuffer()));

// the encoding must round-trip through the geo conversion used by legacy readers

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Conversions.fromByteBuffer round-trip assertion covers only the XY (geometry lower) and XYZM (geography upper) variants; the XYZ (24-byte) and XYM (32-byte, NaN z-slot) encodings are asserted only at raw ByteBuffer byte-equality via .isEqualTo(GeospatialBound.createXYZ(...).toByteBuffer()), not round-tripped back through Conversions.fromByteBuffer. GeospatialBound.toByteBuffer/fromByteBuffer are separately tested so the risk is low, but XYM is the encoding most likely to harbor a NaN-slot/endianness edge case and is the exact integration point the fix depends on; adding fromByteBuffer round-trips for XYZ and XYM would close the integration coverage gap. This goes for testGeoBoundsUseSinglePointEncoding, testGeoBoundWithZOnlyAndMOnly, etc.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also cc @szehon-ho

@vishnuprakaz

Copy link
Copy Markdown
Contributor Author

Good point ! @uros-b ,Added fromByteBuffer round-trips for XYZ and XYM. Thanks for taking a look!

return null;
}

if (boundType.isStructType()) {

@szehon-ho szehon-ho Aug 5, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: rather than treating any struct bound as geo via isStructType(), can we check that this is a geometry/geography bound explicitly? That works today only because StatsUtil uses struct bounds solely for geo types, and an explicit check would avoid silently taking this path if another type ever gains struct bounds.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review @szehon-ho, Added StatsUtil.isGeoBoundType which checks the bounding-box shape explicitly with a direct test in TestStatsUtil.
So, a non-geo struct bound now falls through to Conversions.toByteBuffer and fails loudly instead of silently encoding as geo.

@vishnuprakaz
vishnuprakaz requested a review from szehon-ho August 5, 2026 05:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants