Tracking Issue: Tensor Extension Types

[connortsui20]

Tensors

We would like to add tensor types as extension types to Vortex.

Tensors are multi-dimensional (n-dimensional) arrays that generalize vectors (1D) and matrices (2D)
to arbitrary dimensions. They are quite common in ML/AI and scientific computing applications. To
name just a few examples:

• Image or video data stored as height x width x channels
• Multi-dimensional sensor or time-series data
• Embedding vectors from language models and recommendation systems

Tensor Types

There are different kinds of Tensors that we would like to add.

• FixedShapeTensor: a tensor with a fixed physical shape and dimensions
• VariableShapeTensor: a tensor with variable shape but fixed dimensions
• Matrix: a 2-dimensional tensor
• Vector: a 1-dimensional tensor (flat)
• UnitVec: a unit-normalized d-dimensional vector

Implementation of these types and related functionality like expressions (CosineSimilarity) will live in the vortex-tensor crate.

Prior Art

• Apache Arrow Canonical Extensions

Steps / History

• Extensions Types RFC
• Fixed-shape Tensor
  • RFC
  • Vortex Fixed-Shape Tensor #6812
  • move stuff around in vortex-tensor #6857
  • Logical shape cast
• Variable-size Tensor
• UnitVec: Tracking Issue: Unit-normalized Vectors #6855

Unresolved Questions

TODO