Vortex Fixed-Shape Tensor

[connortsui20]

Summary

Adds an experimental fixed-shape tensor extension type in a new vortex-tensor crate.

See https://vortex-data.github.io/rfcs/rfc/0024.html for info about the design of this tensor type.

Additionally adds a CosineSimilarity expression that takes 2 tensor arrays and computes the cosine similarity of tensors in the arrays (resulting in a PrimitiveArray).

Testing

Adds some very basic tests for cosine similarity and tensor metadata operations.

Future Work

I think this was a good way to see if our ExtVTable is not completely wrong, but at the same time this tells us nothing about what we might want to add for extension arrays on the ExtVTable because we as long as the storage DType is correct, any storage array is valid.

The more interesting expressions have not been implemented here. Those would include:

• Cast
• Index / Slice (and lazily get back another tensor array, potentially non-contiguous)
• Maybe others?

Additional work includes exporting to Arrow, NumPy, and PyTorch. Arrow will require a cheap translation from logical to physical shape, but other than that those conversions should be easy.

[gatesn]

Looks great! Maybe move cosine similarity into a scalar_fns module since we're going to add more.

You should also get public-api to work somehow?

[codspeed-hq]

Merging this PR will improve performance by 23.63%

⚡ 3 improved benchmarks
✅ 977 untouched benchmarks
🆕 20 new benchmarks
⏩ 1466 skipped benchmarks¹

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
⚡	Simulation	take_map[(0.1, 0.5)]	2.6 ms	2.1 ms	+23.63%
⚡	Simulation	take_map[(0.1, 0.1)]	1,007.5 µs	908.9 µs	+10.84%
⚡	Simulation	take_map[(0.1, 1.0)]	4.2 ms	3.5 ms	+20.81%
🆕	Simulation	sequence_compress_u32	N/A	5.2 ms	N/A
🆕	Simulation	sequence_decompress_u32	N/A	4.1 ms	N/A
🆕	Simulation	decompress_utf8[(1000, 16)]	N/A	31 µs	N/A
🆕	Simulation	decompress_utf8[(1000, 256)]	N/A	28.7 µs	N/A
🆕	Simulation	decompress_utf8[(10000, 16)]	N/A	136 µs	N/A
🆕	Simulation	decompress_utf8[(10000, 1024)]	N/A	112.6 µs	N/A
🆕	Simulation	decompress_utf8[(100000, 16)]	N/A	1.2 ms	N/A
🆕	Simulation	decompress_utf8[(10000, 256)]	N/A	113.3 µs	N/A
🆕	Simulation	decompress_utf8[(1000, 4)]	N/A	42.8 µs	N/A
🆕	Simulation	decompress_utf8[(10000, 4)]	N/A	209.9 µs	N/A
🆕	Simulation	decompress_utf8[(100000, 1024)]	N/A	946.9 µs	N/A
🆕	Simulation	decompress_utf8[(100000, 4096)]	N/A	944.7 µs	N/A
🆕	Simulation	decompress_utf8[(1000000, 256)]	N/A	9.4 ms	N/A
🆕	Simulation	decompress_utf8[(100000, 256)]	N/A	958.3 µs	N/A
🆕	Simulation	decompress_utf8[(1000000, 1024)]	N/A	9.3 ms	N/A
🆕	Simulation	decompress_utf8[(1000000, 4)]	N/A	19 ms	N/A
🆕	Simulation	decompress_utf8[(100000, 4)]	N/A	1.9 ms	N/A
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

---

_{Comparing ct/tensor-experiment (3da1787) with develop (5d6a3c8)²}

Footnotes

1. 1466 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

2. No successful run was found on develop (8eee959) during the generation of this report, so 5d6a3c8 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩