fix BigTiff offset overflow - #3616
Conversation
Signed-off-by: Jan Van den bosch <jan@bossie.org>
Signed-off-by: Jan Van den bosch <jan@bossie.org>
| } | ||
| } | ||
|
|
||
| it("should handle offsets greater than 2^32 without overflowing") { |
There was a problem hiding this comment.
Unfoturnately this won't get through the CI (OOO); I'm fine with ignoring this one for now to make sure CI is happy; or if there is a better way to test the changes that'd be great ofc.
There was a problem hiding this comment.
Right, having a tile this large in memory is undesirable. I'm sure it can be improved, let me take a look.
As for the actual test: I currently do this on my machine by invoking tiffinfo -s on the generated BigTiff and taking a look at the actual tile offsets. The reason I do it this way is because:
- instantiating a
geotrellis.raster.io.geotiff.SinglebandGeoTifffrom it throws ajava.nio.BufferUnderflowException; - unless I'm mistaken, I don't think Geotrellis even exposes the actual offsets.
That being said, invoking something "independent" like tiffinfo to check things could actually be an advantage but this obviously needs to be available in the build container. Is it feasible to to install an additional package? I think it came with package libtiff-tools.
There was a problem hiding this comment.
🤔 hmmm, or to make it easier for you we can just make this test 'ignored' for now and that's probably better; to not introduce this complex dep on the tiff info
There was a problem hiding this comment.
Well, TBH at this point the heavy lifting has been done and now it's just a matter of installing a package in the test image. I favored tiffdump over tiffinfo because its output was easier to parse; it is also in the libtiff-tools package.
pomadchin
left a comment
pomadchin
left a comment
There was a problem hiding this comment.
LGTM; I'll take another look ofc.
But also the CI won't be happy unfortuantely due to OOO errs.
| val segments = geoTiff.imageData.segmentBytes | ||
| val segmentCount = segments.size | ||
| val segmentBytesCount = (0 until segmentCount).map(segments.getSegmentByteCount).sum | ||
| val segmentBytesCount = (0 until segmentCount).map(segments.getSegmentByteCount).foldLeft(0L) {_ + _} |
| val segments = geoTiff.imageData.segmentBytes | ||
| val segmentCount = segments.size | ||
| val segmentBytesCount = (0 until segmentCount).map(segments.getSegmentByteCount).sum | ||
| val segmentBytesCount = (0 until segmentCount).map(segments.getSegmentByteCount).foldLeft(0L) {_ + _} |
Signed-off-by: Jan Van den bosch <jan@bossie.org>
Signed-off-by: Jan Van den bosch <jan@bossie.org>
Signed-off-by: Jan Van den bosch <jan@bossie.org>
Signed-off-by: Jan Van den bosch <jan@bossie.org>
Signed-off-by: Jan Van den bosch <jan@bossie.org>
Signed-off-by: Jan Van den bosch <jan@bossie.org>
2 participants
Overview
Fixes offset overflow in full res BigTiffs. Manifests itself when the first overview already tips the image size over the 2^32 threshold.
Checklist
docsguides update, if necessary