Skip to content

Fill sparse COG tiles with nodata instead of crashing#1501

Merged
brendancol merged 1 commit intomainfrom
fix-sparse-cog-tile
May 6, 2026
Merged

Fill sparse COG tiles with nodata instead of crashing#1501
brendancol merged 1 commit intomainfrom
fix-sparse-cog-tile

Conversation

@brendancol
Copy link
Copy Markdown
Contributor

Summary

GDAL's SPARSE_OK=TRUE writes blocks consisting entirely of nodata with TileByteCounts (or StripByteCounts) == 0 and a matching offset of 0. The reader handled this badly:

  • Local mmap path tried to decompress the empty byte range and raised error -5 while decompressing data: incomplete or truncated stream.
  • COG HTTP path skipped the empty fetch but left the result as np.empty, so sparse regions returned uninitialised memory.
  • GPU path crashed inside the decode pipeline.

This branch detects sparse blocks before decode, pre-fills the result with the file's GDAL_NODATA value (zero when no nodata tag is present, matching GDAL's convention), and skips decode for those entries. The GPU dispatcher routes sparse files through the CPU reader and copies the result onto the device, since the GPU pipeline does not handle empty ranges.

Repro on main

import rasterio, numpy as np
from xrspatial.geotiff import open_geotiff

data = np.zeros((128, 128), dtype=np.uint16)
data[:64, :64] = 100
profile = {
    'driver': 'GTiff', 'dtype': 'uint16',
    'height': 128, 'width': 128, 'count': 1,
    'tiled': True, 'blockxsize': 64, 'blockysize': 64,
    'compress': 'DEFLATE', 'SPARSE_OK': 'TRUE', 'nodata': 0,
}
with rasterio.open('/tmp/sparse.tif', 'w', **profile) as dst:
    dst.write(data[:64, :64], 1, window=rasterio.windows.Window(0, 0, 64, 64))

open_geotiff('/tmp/sparse.tif')
# OSError: error -5 while decompressing data: incomplete or truncated stream

After the fix the call returns a (128, 128) float64 DataArray with the filled tile and NaN over the sparse region.

Test plan

  • pytest xrspatial/geotiff/tests/test_sparse_cog.py (5 new tests covering tiled/stripped, with/without nodata, raw + accessor reads, and GPU)
  • pytest xrspatial/geotiff/tests/ full suite (651 passed, 7 deselected — 3 pre-existing matplotlib palette RecursionError failures unrelated to this PR, also fail on origin/main)
  • Manual repro against GDAL SPARSE_OK=TRUE files for tiled and stripped layouts and CuPy GPU read

GDAL's SPARSE_OK=TRUE writes blocks consisting entirely of nodata with
TileByteCounts (or StripByteCounts) == 0 and a matching offset of 0.
The local mmap reader tried to decompress the empty range and raised
"incomplete or truncated stream"; the COG HTTP reader skipped the
fetch but left the result allocated as np.empty, returning whatever
happened to be in memory; the GPU reader crashed somewhere in the
decode pipeline.

Detect sparse entries up front, allocate the result with np.full(fill)
where fill is the file's GDAL_NODATA value (or 0 when unset), and skip
the decode for those entries. The GPU dispatcher routes sparse files
through the CPU reader and copies the result to device memory, since
the on-GPU pipeline does not handle empty tile ranges.

Tests cover tiled and stripped layouts with and without a nodata tag,
the raw read_to_array path (sparse becomes the sentinel value), the
public open_geotiff path (sparse becomes NaN via the existing nodata
promotion), and the GPU path.
@github-actions github-actions Bot added the performance PR touches performance-sensitive code label May 6, 2026
@brendancol brendancol merged commit 573d9ee into main May 6, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

performance PR touches performance-sensitive code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant