Add Blosc2 codec to zarr-python

[ruaridhg]

zarr-developers/zarr-python#1896

[K-Meech]

There has already been quite a lot of discussion about adding Blosc2 to zarr-python / numcodecs:

• issue on zarr-python
• issue on numcodecs
• image.sc post about blosc2 and zarr
• Draft PR adding Blosc2 to numcodecs by one of the core Blosc developers - from 2 years ago!
• Issue on zarr-python about supporting caterva - caterva has since been merged into Blosc2

This is complicated by the fact that Blosc2 added many additional features vs Blosc1. E.g. the caterva project was merged into it, leading to a new NDim / NDArray object in Blosc2, which supports multi-dimensional arrays with a numpy-like interface. It's debated how much of these new features to integrate.

Blosc2 (and Blosc1) are C libraries, with python wrappers. Currently, zarr-python (via numcodecs) is using the C libraries - but there are open issues / PRs to suggest using the python wrapper instead - for blosc1 + blosc2. There was some debate on the blosc1 PR about which to use - so it's not immediately clear which to integrate...

There is also already an open PR from 2 years ago from one of the core blosc developers to add integration for C-Blosc2. There has been little response to this.

[dstansby]

There are ongoing discussions at zarr-developers/zarr-python#3272 about where codecs should go in the future, so while that's ongoing myself and @K-Meech decided to not pursue implementing the codec for now.

Two things that could be done now, that might be helpful to push this forward:

1. Try implementing the codec against the Protocol proposed in add numcodec protocol zarr-developers/zarr-python#3318, and give feedback on that PR. The idea would be to try and create a simple-as-possible script that implements a custom blosc2 codec that zarr-python can use.
2. Open a request to have blosc2 added to the formal Zarr specification. This would be the start of the process of getting blosc2 to be accepted into zarr-python, as we are leaning towards just implementing codecs in the Zarr specification.