Memory-mapped read access with DictStore

[Karol-G]

Hey,

I want to store many large images as multi-resolution pyramid images using Blosc2 in the context of Machine Learning training. During the training, random Blosc2 images will be opened, and a single image patch from that image will be read. So, minimal file opening overhead and memory-mapped read access are important for this.

Reading the documentation, it seems a zipped DictStore seems to be the right approach here, as I want every multi-resolution image to be stored in a single file. However, it seems that DictStore only supports the mode argument, but not the mmap_mode (in contrast to Blosc2.open) argument. So my question is simply, can zipped DictStores be used with memory-mapping?

Best,
Karol

[FrancescAlted]

Hi. Good question. Short answer is that we implemented DictStores without thinking in providing support for mmap_mode, but agreed that can be useful. If you manage to contribute that support on a PR, happy to discuss about it. If you don't have time yourself, you can always buy support for that.