Skip to content

perf: Optimize array_positions() for scalar needle#20770

Open
neilconway wants to merge 4 commits intoapache:mainfrom
neilconway:neilc/optimize-array-positions
Open

perf: Optimize array_positions() for scalar needle#20770
neilconway wants to merge 4 commits intoapache:mainfrom
neilconway:neilc/optimize-array-positions

Conversation

@neilconway
Copy link
Contributor

@neilconway neilconway commented Mar 7, 2026

Which issue does this PR close?

Rationale for this change

array_positions previously compared the needle against each row's sub-array individually. When the needle is a scalar (the common case), we can do a single bulk arrow_ord::cmp::not_distinct comparison against the entire flat values buffer and then walk the result bitmap, which is significantly faster: the speedup on the array_positions() microbenchmarks ranges from 5x to 40x, depending on the size of the array.

The same pattern has already been applied to array_position (#20532), and previously to other array UDFs.

What changes are included in this PR?

  • Add benchmarks for array_positions.
  • Implement bulk-comparison optimization
  • Refactor array_position's existing fast path slightly for consistency
  • Code cleanup to use "haystack" and "needle" consistently, not vague terms like "list_array" and "element"
  • Add unit tests for array_positions with sliced ListArrays, for peace of mind
  • Add unit tests for sliced lists and sliced lists with nulls for the new array_positions fast path.

Are these changes tested?

Yes.

Are there any user-facing changes?

No.

AI usage

Multiple AI tools were used to iterate on this PR. I have reviewed and understand the resulting code.

@github-actions github-actions bot added documentation Improvements or additions to documentation functions Changes to functions implementation labels Mar 7, 2026
@neilconway
Copy link
Contributor Author

Benchmarks:

  ┌────────────────┬────────┬────────┬─────────┐
  │   Benchmark    │ Before │ After  │ Speedup │
  ├────────────────┼────────┼────────┼─────────┤
  │ found_once/10  │ 1.46ms │ 39µs   │ 37x     │
  ├────────────────┼────────┼────────┼─────────┤
  │ found_many/10  │ 1.59ms │ 67µs   │ 24x     │
  ├────────────────┼────────┼────────┼─────────┤
  │ not_found/10   │ 1.28ms │ 32µs   │ 40x     │
  ├────────────────┼────────┼────────┼─────────┤
  │ found_once/100 │ 1.97ms │ 140µs  │ 14x     │
  ├────────────────┼────────┼────────┼─────────┤
  │ found_many/100 │ 4.06ms │ 501µs  │ 8x      │
  ├────────────────┼────────┼────────┼─────────┤
  │ not_found/100  │ 1.82ms │ 132µs  │ 14x     │
  ├────────────────┼────────┼────────┼─────────┤
  │ found_once/500 │ 4.05ms │ 601µs  │ 6.7x    │
  ├────────────────┼────────┼────────┼─────────┤
  │ found_many/500 │ 12.0ms │ 2.51ms │ 4.8x    │
  ├────────────────┼────────┼────────┼─────────┤
  │ not_found/500  │ 3.82ms │ 588µs  │ 6.5x    │
  └────────────────┴────────┴────────┴─────────┘

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation functions Changes to functions implementation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optimize array_positions() for scalar needle

1 participant