What is the problem the feature request solves?
Follow-up from #5233
spark_size for LargeList reuses Arrow's length kernel, which returns Int64, then cast_with_options(..., Int32, safe: false), then (when nulls are present) to_vec() to patch null slots to -1. That is three allocations on the hot path.
On the array_size bench after #5233, LargeList (10% null) is only ~1.3x faster than main (7.54 µs vs 9.70 µs), while no-null List is ~12x. The gap is the Int64 → cast → copy chain, not the kernel idea itself.
Describe the potential solution
Build an Int32 length array straight from the LargeList i64 offset buffer (offsets[i+1] - offsets[i], with a checked conversion that errors on overflow rather than wrapping or becoming -1), then apply the same null →
-1 rewrite as the List path (null_count == 0 fast path; otherwise into_parts + set_indices).
Avoid allocating the intermediate Int64 length output and the Int32 cast result when those are only used to derive i32 sizes
Additional context
What is the problem the feature request solves?
Follow-up from #5233
spark_sizeforLargeListreuses Arrow'slengthkernel, which returnsInt64, thencast_with_options(..., Int32, safe: false), then (when nulls are present)to_vec()to patch null slots to-1. That is three allocations on the hot path.On the
array_sizebench after #5233,LargeList (10% null)is only ~1.3x faster thanmain(7.54 µs vs 9.70 µs), while no-nullListis ~12x. The gap is the Int64 → cast → copy chain, not the kernel idea itself.Describe the potential solution
Build an
Int32length array straight from theLargeListi64 offset buffer (offsets[i+1] - offsets[i], with a checked conversion that errors on overflow rather than wrapping or becoming-1), then apply the same null →-1rewrite as the List path (null_count == 0fast path; otherwiseinto_parts+set_indices).Avoid allocating the intermediate Int64
lengthoutput and the Int32 cast result when those are only used to derive i32 sizesAdditional context
spark_size: LargeList (10% null)inbenches/array_size.rs. Consider adding a no-null LargeList shape for parity with the List production path.safe: false/ checked overflow semantics: a length that does not fit ini32must error, not become null then-1