Core: Fix geo bounds in v4 legacy stat maps - #17493
Conversation
| Map.entry(10, GeospatialBound.createXY(5.0, 6.0).toByteBuffer()), | ||
| Map.entry(11, GeospatialBound.createXYZM(7.0, 8.0, 9.0, 10.0).toByteBuffer())); | ||
|
|
||
| // the encoding must round-trip through the geo conversion used by legacy readers |
There was a problem hiding this comment.
The Conversions.fromByteBuffer round-trip assertion covers only the XY (geometry lower) and XYZM (geography upper) variants; the XYZ (24-byte) and XYM (32-byte, NaN z-slot) encodings are asserted only at raw ByteBuffer byte-equality via .isEqualTo(GeospatialBound.createXYZ(...).toByteBuffer()), not round-tripped back through Conversions.fromByteBuffer. GeospatialBound.toByteBuffer/fromByteBuffer are separately tested so the risk is low, but XYM is the encoding most likely to harbor a NaN-slot/endianness edge case and is the exact integration point the fix depends on; adding fromByteBuffer round-trips for XYZ and XYM would close the integration coverage gap. This goes for testGeoBoundsUseSinglePointEncoding, testGeoBoundWithZOnlyAndMOnly, etc.
|
Good point ! @uros-b ,Added fromByteBuffer round-trips for XYZ and XYM. Thanks for taking a look! |
| return null; | ||
| } | ||
|
|
||
| if (boundType.isStructType()) { |
There was a problem hiding this comment.
Nit: rather than treating any struct bound as geo via isStructType(), can we check that this is a geometry/geography bound explicitly? That works today only because StatsUtil uses struct bounds solely for geo types, and an explicit check would avoid silently taking this path if another type ever gains struct bounds.
There was a problem hiding this comment.
Thanks for the review @szehon-ho, Added StatsUtil.isGeoBoundType which checks the bounding-box shape explicitly with a direct test in TestStatsUtil.
So, a non-geo struct bound now falls through to Conversions.toByteBuffer and fails loudly instead of silently encoding as geo.
While reading the v4 stats code noticed @dramaticlly's comment on #17322 suspecting that
Conversions.toByteBuffermight not handle geometry bounds properly insideContentStatsBackedMap.bound(). The suspicion is correct. For geometry and geography columns, the bound field in the stats schema is the bounding-box struct (x, y, z, m) built byStatsUtil.geoLowerBound, andConversions.toByteBufferhas no struct case, so it throws.A small repro against current main confirmed it:
Once a tracked file carries geo column stats, any access to
lowerBounds()orupperBounds()that touches the geo column hits this a predicate on that column, or anything that copies or iterates the stats maps. Nothing reaches this in production today, but #17433 starts reading geo stats from v4 manifests, which puts the crash one call away from the read path.The fix converts the bounding-box struct back to a
GeospatialBoundand serializes it with the spec's single-point bound encoding. That's the representation legacy consumers of these maps already expect:Conversionsreads that encoding back for geo types, and v3 tables surface geo bounds through the same maps in the same form so the legacy view of av4 file stays consistent with v3 behavior, and the bounding-box storage format from the v4 stats design is untouched.
Tests cover geometry and geography columns next to a primitive column, the z-only and m-only encodings, a geo field with no lower bound, and a round-trip through
Conversions.fromByteBuffer.If this is already covered as part of the planned content-stats integration work, happy to fold it into that instead.