Behavioral issue changes within the same python version (libnetCDF/netcdf-c 4.10.0 bugs)

[jeremyleung521]

Hi,

Due to changes in libnetcdf (netcdf-c) 4.10.0, there appears to be behavioral changes even if staying on the same python version (1.7.4). This includes indexing and "diskless mode". The environments used for testing are all installs from conda-forge. Our CI failed miserably on the first example, so I'm assuming this is the behavior on all/most UNIX OSes. To resolve this I suppose releasing a post-version pinning libnetcdf <4.10.0 until the next python release seems apt? I'm not having trouble on the PyPI release.

1. Previously the code ignores any indices not present in the dataset (e.g. bigger than the size of the dataset) and only retrieve the rows that are possible. Now, it throws an IndexError. While I believe what is happening now (throwing an IndexError) is the "correct" behavior, it's a little rough having behavioral changes when we're technically in the same python version.

Minimal reproducible example with libnetcdf 4.10.0. Note that frame0.nc has 501 frames in the first dimension, so the slice had indexed something out of the shape.

import netCDF4
with netCDF4.Dataset('tests/data/frame0.nc', format='netCDF3') as file:
    coord = file.variables['coordinates'][slice(500, 510,2)]

Related:
mdtraj/mdtraj#2142
mdtraj/mdtraj#2143

2. While creating a self-contained example, I found out that diskless mode is also broken with libnetcdf 4.10.0. This is fine with libnetcdf 4.9.3.

In [4]: nc = netCDF4.Dataset('diskless_example.nc', 'w', diskless=True, persist=True)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[4], line 1
----> 1 nc = netCDF4.Dataset('diskless_example.nc', 'w', diskless=True, persist=True)

File src/netCDF4/_netCDF4.pyx:2415, in netCDF4._netCDF4.Dataset.__init__()

ValueError: diskless mode requires netcdf lib >= 4.2.1, you have 4.10.0

Thanks.

EDIT: 2 is fixed.

[ocefpaf]

2 is fixed in #1474

[jeremyleung521]

2 is fixed in #1474

Just saw those PRs and force-reinstalled netCDF4 from conda. I think it's now working... Thanks! 1 still persists though :)

[jswhit]

1 is due to changes in the netcdf-c library, not the python interface. We can't control what version of the C lib the python module is linked against, the only thing we could do is issue a warning or an exception when the python module is imported (and I don't think that's really a good idea).