Select Numpy C API version

[mattiapenati]

This PR introduces the build configuration to select the target Numpy C API version and adjusts struct definitions to match the user selected version.

The default configuration has been chosen to match the setting in the first release of Numpy v2.

The only potentially breaking change is the new definition of PyUFuncObject. The struct PyUFuncObject has been updated to match the definition given in Numpy source. The previous definition matched the definition contained in documentation. I think the former is more accurate.

I understand that this PR is not small and that not all changes can be easily reviewed, it wasn't easy to put everything together either. If the PR is too large, I could split it into smaller changes. Anyway, I hope it will help improve the FFI.

[Icxolu]

Thanks for the PR and for looking into this. As I understand this tries to add version specific condition compilation similar to pyo3 itself, right? rust-numpy currently does this dynamically by reading the API_VERSION at runtime and dispatching based on that. For this reason the struct layouts have to be compatible with all supported versions at all times and we can't include fields introduced in newer versions based on compile time flags. I think the two approaches, at least as currently implemented, at not compatible to each other. I also think that the dynamic approach has the really nice property of leaving the version choice to the consumer. For example I could build an abi3 wheel across a large range of Python versions, some of which only have numpy v1 wheels and others only v2 wheel available. Currently this just works, I think this is a property worth keeping.

Additionally Python¹² (and numpy as well) is slowly moving towards a new stable abi build around opaque PyObjects, so I'm also not sure if it would be wise now to introduce a stronger dependency on PyObject layouts when in the future we probably want to move away from that.

So currently I'm inclined to say we should keep the current design.

cc @davidhewitt @ngoldbaum Maybe you have some opinions/comments for this one as well.

Footnotes

1. https://peps.python.org/pep-0803 ↩

2. https://peps.python.org/pep-0809/ ↩

[ngoldbaum]

I've personally been responsible for adding new structs and functions to the NumPy C API and it would be a shame if those functions can't be used by rust-numpy.

That said, this does introduce a significant amount of complexity. I also don't think it makes much sense to try to support very old versions of NumPy.

I would instead try to support only NumPy 1.25 and NumPy 2.1 and newer. I would also tend not to care much about supporting NumPy 1.x but I understand if that's important for people.

[ngoldbaum]

This was changed to dict only relatively recently in 2024, see numpy/numpy#27735. It was first available in NumPy 2.2.

[ngoldbaum]

This was changed to vectorcall in numpy/numpy#15271 which I believe was first included in NumPy 1.20.0.

[ngoldbaum]

Separately from the discussion in this PR, it makes sense to fix these incorrect bindings. Can you fix the issues I pointed out above and send in a PR that doesn't depend on details that changed in later NumPy versions? I think that means wrapping the fields that changed as *mut c_void but maybe @Icxolu has a way they'd prefer to see it spelled.

[mattiapenati]

Thank you for the feedback! I've updated this PR based on @ngoldbaum's comments. The definition of PyUFuncObject now conditionally matches NumPy's definitions from version 1.15 through version 2.4.

One thought: I believe supporting newer API versions would have a greater beneficial impact than having a support of wider set of versions. Additionally, the code could be significantly simplified if only newer versions of NumPy were supported, like >=1.25, but this choice is out of my responsibility.

Sorry @ngoldbaum, but I'm not sure I understand your last comment. Could you clarify what you mean by "details that changed in later NumPy versions"? The definition of PyUFuncObject on the main branch already matches the one in NumPy v1.15. Are you suggesting a separate PR that uses the same conditional compilation approach to implement only the changes of least recent versions of numpy?

[ngoldbaum]

Could you clarify what you mean by "details that changed in later NumPy versions"?

e.g. following what @Icxolu was saying and not having any conditional compilation or version-dependent bindings.

Another thing that I realized after writing my earlier comments is that NumPy itself, as of NumPy 2.0, supports building binaries that work with arbitrary older or newer numpy versions found at runtime. This is tricky and it requires a lot of conditional compilation in NumPy's headers. If it's possible to replicate that and allow setting something analogous to the NPY_FEATURE_VERSION and NPY_RUNTIME_VERSION macros. I suspect that might be tricky though.

[ngoldbaum]

@seberg knows more about how numpy handles forward and backward compatibility in its headers and might be interested in this thread too.

[mattiapenati]

Now I have a clearer understanding of what you meant. The mechanism you're referring to is the backward compatibility built into the NumPy API. Each version of NumPy exposes an API that remains compatible with older versions, and users can override the default by defining the NPY_TARGET_VERSION macro and updating pyproject.toml file.

This PR aims to replicate that mechanism. By default, the FFI exposed by rust-numpy targets NumPy v1.25, which is the same default used by NumPy v2.0. This may not be the ideal choice, and I'd welcome your input on what would work best. A set of features named target-npy### has been introduced to mirror the role of the NPY_TARGET_VERSION macro in NumPy. Users can opt into a newer API version by enabling one of these features and updating their pyproject.toml accordingly, or they can disable the default feature to target an older API version.

Consistency is ensured by the fact that all conditional compilation instructions have been written to mirror the conditionals involving the NPY_FEATURE_VERSION macro in the NumPy API.

[ngoldbaum]

I think there's no reason to try to support any numpy version in rust-numpy older than 1.25, so I'd suggest simplifying things here and removing the targets for older versions.

It's also not clear to me from the diff that you are handling forward and backward compatibility for different runtime NumPy versions like I think you just claimed - can you explain how you handle that? I'd think you'd need some functions to emulate the macros and static inline functions in the numpy headers. How are you handling the fact that the descriptor object, ufunc object and array object have different layouts in different numpy versions at runtime?

[ngoldbaum]

I would think you'd need some sort of automated testing where you build with one numpy version and run with another to validate all this.

[seberg]

@seberg knows more about how numpy handles forward and backward compatibility in its headers and might be interested in this thread too.

Some of these fields are a bit "why would you ever need them", and I guess those may be sloppy in NumPy (i.e. the dict is probably OK as it was NULL before, but just like vectorcall, you should use the Python APIs...).

The thing I think seems missing here is the runtime check (I am sure this already exists for the major C version, but that check needs to be expanded).
I.e. the vital thing with NPY_TARGET_VERSION (translated to NPY_FEATURE_VERSION in the headers) is that NumPy injects code to prevent running on older NumPy versions at runtime where the feature/field would not be valid.
(Yes, that can get in the way because it is tedious to conditionally use new features at runtime.)

Validating all fields seems rather tedious, but it seems important to try and validate the runtime version check. (Tedious because most of them just shouldn't be used and many that may be used are still niche to the point that my guess is there is maybe a single external user.)

(E.g. a good general candidate is the descr elsize Which I think here requires NumPy 2 or higher because to support both you would need to add runtime logic. NumPy does that with static inline functions in the npy2_compat.h.)

[Icxolu]

I think there's no reason to try to support any numpy version in rust-numpy older than 1.25, so I'd suggest simplifying things here and removing the targets for older versions.

We currently support the same Python versions as PyO3 itself. The latest numpy version for 3.7 is 1.21 and for 3.8 is is 1.24.
So if we increase the numpy target we effectively also drop support for 3.7 and 3.8 (which at some point we should, but so should pyo3 itself, see also PyO3/pyo3#4581).

Some of these fields are a bit "why would you ever need them", and I guess those may be sloppy in NumPy (i.e. the dict is probably OK as it was NULL before, but just like vectorcall, you should use the Python APIs...).

This is also something that I thought about. Some motivation for APIs that we want to build on top of these would be great. It would maybe that outline a more specific path to a design we could move towards.

It's also not clear to me from the diff that you are handling forward and backward compatibility for different runtime NumPy versions like I think you just claimed - can you explain how you handle that?

This is also unclear to me.

One way that I think could maybe work if we want too only add a few APIs that require only a small number of specific fields is to basically define both layouts like so:

#[repr(C)]
struct PyArray_Descr {
    ... // Like currently
}

#[repr(C)]
struct PyArray_DescrV2 {
    base: PyArray_Descr,
    ... // additional V2 fields
}

After a runtime version check we could then perform a pointer cast to access the fields or return an error for older numpy versions. But this blows up quite quickly if need a big range of versions and fields, so would be feasible for a small number of combinations.