feat: Support Spark array_contains builtin function

[comphead]

Which issue does this PR close?

• Closes Create Spark builtin function array_contains #20611 .

Rationale for this change

What changes are included in this PR?

The Spark function is actual wrapper on top of array_has function. After result is being produced the nulls mask is set respectively for the output indices which correspond to input rows having nulls

Are these changes tested?

Are there any user-facing changes?

[mbutrovich]

I don't love the needle and haystack naming. I'm not sure if that idiom generalizes to all contributors.

[mbutrovich]

Is that what Spark does? It short circuits and returns true even if there are NULLs in the array? What if the match is after the NULL? Would it return NULL? I thought if any element of the array was NULL the outcome of the expression is NULL.

[comphead]

scala> spark.sql("select array_contains(array(1, null, 2), 2)").show(false)
+------------------------------------+
|array_contains(array(1, NULL, 2), 2)|
+------------------------------------+
|true                                |
+------------------------------------+


scala> spark.sql("select array_contains(array(1, 2, null), 2)").show(false)
+------------------------------------+
|array_contains(array(1, 2, NULL), 2)|
+------------------------------------+
|true                                |
+------------------------------------+


scala> spark.sql("select array_contains(array(1, null), 2)").show(false)
+---------------------------------+
|array_contains(array(1, NULL), 2)|
+---------------------------------+
|null                             |
+---------------------------------+

I think explanation in https://issues.apache.org/jira/browse/SPARK-55749 is very accurately explaining this behavior

[comphead]

on this data Spark run

=== Test Data ===
+----+----------------+----+                                                    
|col1|col2            |col3|
+----+----------------+----+
|1   |[1, 2, 3]       |10  |
|2   |[4, null, 6]    |5   |
|3   |[7, 8, 9]       |10  |
|4   |null            |1   |
|5   |[10, null, null]|10  |
+----+----------------+----+

=== array_contains(col2, col3) and array_contains(col2, 10) ===
+----+--------------------------+------------------------+
|col1|array_contains(col2, col3)|array_contains(col2, 10)|
+----+--------------------------+------------------------+
|1   |false                     |false                   |
|2   |null                      |null                    |
|3   |false                     |false                   |
|4   |null                      |null                    |
|5   |true                      |true                    |
+----+--------------------------+------------------------+

[comphead]

I don't love the needle and haystack naming. I'm not sure if that idiom generalizes to all contributors.

it is what currently used in array_has*, position and some other functions. But agree, not sure this is most idiomatic approach

[mbutrovich]

Thanks @comphead!

[Jefffrey]

Suggested change

	let haystack = match haystack_arg {
	ColumnarValue::Array(arr) => Arc::clone(arr),
	ColumnarValue::Scalar(s) => s.to_array_of_size(result.len())?,
	};
	let haystack = haystack_arg.to_array_of_size(result.len())?;

[Jefffrey]

Suggested change

	let old_validity = match result.nulls() {
	Some(n) => n.inner().clone(),
	None => BooleanBuffer::new_set(result.len()),
	};
	let new_validity = &old_validity & &(!&nullify_mask);
	let new_validity = match result.nulls() {
	Some(n) => n.inner() & &(!&nullify_mask),
	None => !&nullify_mask,
	};

[Jefffrey]

Probably better to just error/panic here to make it clear this None path is unreachable?

[comphead]

Thanks @Jefffrey and @mbutrovich for the review