You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Follow-up from #5233 spark_size for List / LargeList / FixedSizeList now reuses Arrow's length kernel.
The Map branch in native/spark-expr/src/array_funcs/size.rs is still a per-row loop that appends MapArray::value_length(i) into an Int32Builder. length does not accept MapArray, so Map cannot reuse the kernel. It was kept out of #5233 on purpose: different technique, and its own correctness and benchmark surface.
Describe the potential solution
MapArray::offsets() is the entry-range offset buffer; for each row, offsets[i+1] - offsets[i] is the entry count (same value value_length(i) returns one row at a time). Vectorize that and apply Spark's null → -1 rewrite with the same set_indices patch the list path uses
Additional context
Before merging:
Add a Map shape (with and without nulls) to benches/array_size.rs — there is no Map coverage today. The win is expected to be smaller than for List, since value_length already reads the offsets; only per-row dispatch and builder overhead go away. If the benchmark comes back flat, closing this as not worth the churn is fine.
What is the problem the feature request solves?
Follow-up from #5233
spark_sizeforList/LargeList/FixedSizeListnow reuses Arrow'slengthkernel.The
Mapbranch innative/spark-expr/src/array_funcs/size.rsis still a per-row loop that appendsMapArray::value_length(i)into anInt32Builder.lengthdoes not acceptMapArray, so Map cannot reuse the kernel. It was kept out of #5233 on purpose: different technique, and its own correctness and benchmark surface.Describe the potential solution
MapArray::offsets()is the entry-range offset buffer; for each row,offsets[i+1] - offsets[i]is the entry count (same valuevalue_length(i)returns one row at a time). Vectorize that and apply Spark's null →-1rewrite with the sameset_indicespatch the list path usesAdditional context
Before merging:
benches/array_size.rs— there is no Map coverage today. The win is expected to be smaller than for List, sincevalue_lengthalready reads the offsets; only per-row dispatch and builder overhead go away. If the benchmark comes back flat, closing this as not worth the churn is fine.MapArraytest, mirroring the sliced-List test added in perf: compute spark_size list lengths with Arrow length kernel #5233.MapArray::offsets()returns the sliced offsets sowindows(2)stays correct, but pin the invariant.