Skip to content

Use the offset buffer for Map in spark_size #5266

Description

@0lai0

What is the problem the feature request solves?

Follow-up from #5233
spark_size for List / LargeList / FixedSizeList now reuses Arrow's length kernel.
The Map branch in native/spark-expr/src/array_funcs/size.rs is still a per-row loop that appends MapArray::value_length(i) into an Int32Builder.
length does not accept MapArray, so Map cannot reuse the kernel. It was kept out of #5233 on purpose: different technique, and its own correctness and benchmark surface.

Describe the potential solution

MapArray::offsets() is the entry-range offset buffer; for each row, offsets[i+1] - offsets[i] is the entry count (same value value_length(i) returns one row at a time). Vectorize that and apply Spark's null → -1 rewrite with the same set_indices patch the list path uses

Additional context

Before merging:

  • Add a Map shape (with and without nulls) to benches/array_size.rs — there is no Map coverage today. The win is expected to be smaller than for List, since value_length already reads the offsets; only per-row dispatch and builder overhead go away. If the benchmark comes back flat, closing this as not worth the churn is fine.
  • Add a sliced-MapArray test, mirroring the sliced-List test added in perf: compute spark_size list lengths with Arrow length kernel #5233. MapArray::offsets() returns the sliced offsets so windows(2) stays correct, but pin the invariant.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions