Writing a file with `auto_complex=True` fails when using a dimension for the complex components

[ZedThree]

The following minimal example fails:

import netCDF4
import numpy as np

with netCDF4.Dataset("test_dim_autocomplex.nc", "w", auto_complex=True) as f:
    f.createDimension("x", size=2)
    f.createDimension("complex", size=2)
    c_dim = f.createVariable("data_dim", np.float64, ("x", "complex"))
    c_dim[:] = np.array([[1., 2.], [3., 4.]])

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File src/netCDF4/_netCDF4.pyx:5601, in netCDF4._netCDF4.Variable.__setitem__()

ValueError: cannot reshape array of size 4 into shape (2,)

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In[31], line 5
      3 f.createDimension("complex", size=2)
      4 c_dim = f.createVariable("data_dim", np.float64, ("x", "complex"))
----> 5 c_dim[:] = np.array([[1., 2.], [3., 4.]])

File src/netCDF4/_netCDF4.pyx:5603, in netCDF4._netCDF4.Variable.__setitem__()

File /tmp/mvce/.venv/lib/python3.13/site-packages/numpy/lib/_stride_tricks_impl.py:410, in broadcast_to(array, shape, subok)
    367 @array_function_dispatch(_broadcast_to_dispatcher, module='numpy')
    368 def broadcast_to(array, shape, subok=False):
    369     """Broadcast an array to a new shape.
    370 
    371     Parameters
   (...)    408            [1, 2, 3]])
    409     """
--> 410     return _broadcast_to(array, shape, subok=subok, readonly=True)

File /tmp/mvce/.venv/lib/python3.13/site-packages/numpy/lib/_stride_tricks_impl.py:349, in _broadcast_to(array, shape, subok, readonly)
    346     raise ValueError('all elements of broadcast shape must be non-'
    347                      'negative')
    348 extras = []
--> 349 it = np.nditer(
    350     (array,), flags=['multi_index', 'refs_ok', 'zerosize_ok'] + extras,
    351     op_flags=['readonly'], itershape=shape, order='C')
    352 with it:
    353     # never really has writebackifcopy semantics
    354     broadcast = it.itviews[0]

ValueError: input operand has more dimensions than allowed by the axis remapping

This does work without auto_complex=True, and because the dtype isn't actually complex, auto_complex should really have no effect here because we're writing. Writing without auto_complex and then reading with auto_complex=True does work as expected, returning a complex array of length 2.

I think this is happening because we create a variable in the netCDF first, detect that it matches a complex variable and so do our complex-variable fixing up, and then try to fill it with something not actually complex.

The line numbers in the error messages don't quite match up for me, but I think it's this bit:

        # reshape data array if needed to conform with start,count,stride.
        if data.ndim != len(datashape) or\
           (data.shape != datashape and data.ndim > 1): # issue #1083
            # create a view so shape in caller is not modified (issue 90)
            try: # if extra singleton dims, just reshape
                data = data.view()
                data.shape = tuple(datashape)
            except ValueError: # otherwise broadcast
                data = numpy.broadcast_to(data, datashape)

It also does work fine when writing the numpy struct version.

I'm not quite sure how to handle this off the top of my head. We probably need to do something like check if the variable dtype is complex and if it matches the passed in data, and if not, fall back to doing something else. Either writing it as the original dtype, or converting the passed in data to complex.

[ZedThree]

The problem is specifically that when auto_complex=True and a variable has a complex dimension, then we discard that from var.dimensions, which is now it gets the shape wrong in the above snippet.

In this particular case, auto_complex isn't useful, because we're expecting a complex dtype to be passed in. So we need at least some check that the dtypes match, and probably also to check that the dtype is complex if nc-complex says it's a complex variable.