fix: materialize ConstantColumnVector on Comet's serialize/export paths

[schenksj]

Which issue does this PR close?

Closes #4527.

Rationale for this change

Spark wraps file-source partition columns and other per-batch constants in ConstantColumnVector. When such a batch reaches Comet's serialization path (Utils.getBatchFieldVectors, used by broadcast/shuffle) or the FFI export path (NativeUtil.exportBatch), it was rejected with:

Comet execution only takes Arrow Arrays, but got ...ConstantColumnVector

This is a standalone fix; it was surfaced while working on the Delta Lake contrib integration (the OPTIMIZE / deletion-vector rewrite paths pull constants through a Comet operator), so prioritizing it helps that effort, but it applies to any plan that routes a constant column through a Comet operator.

What changes are included in this PR?

• ConstantColumnVectors.materialize (in the org.apache.spark.sql.comet.execution.arrow package) builds a fresh Arrow FieldVector holding the constant repeated numRows times. It reuses the existing per-type ArrowFieldWriters, so it covers every type -- scalars, decimal, timestamps, and complex struct/array/map -- and stays in sync with Spark's type handling, rather than a hand-rolled per-type switch.
• Utils.materializeConstantColumnVector exposes it to the serialization path.
• New match arms in Utils.getBatchFieldVectors and NativeUtil.exportBatch materialize a ConstantColumnVector instead of throwing. The existing CometVector path is untouched.

How are these changes tested?

New test in UtilsSuite round-trips a batch with a value ConstantColumnVector and a null ConstantColumnVector through serializeBatches / decodeBatches and asserts the materialized values (and nulls) survive. The test fails on main with the "only takes Arrow Arrays" exception and passes with this change. UtilsSuite (3/3) and CometExecSuite (126/0) pass. The FFI exportBatch arm shares the same materializeConstantColumnVector helper.

[schenksj]

Gentle ping — still looking for a first review on this one. Small, self-contained fix. Thanks!

[andygrove]

A few observations from a review pass. Acknowledging up front that this comment was drafted with help from AI (Claude Code).

The reuse of the existing per-type ArrowFieldWriters is exactly the right design choice — every Spark type comes along for free and the logic stays in sync with non-constant batches.

A few notes:

1. Hardcoded "UTC" timezone in Utils.materializeConstantColumnVector. The choice is defensible since Spark stores TimestampType as micros in UTC, but elsewhere in this file toArrowSchema requires callers to pass the session timezone explicitly. If a batch flows through getBatchFieldVectors containing both a ConstantColumnVector of TimestampType and a non-constant timestamp column, would they end up with mismatched Arrow timezone metadata? Worth either threading a timeZoneId parameter through getBatchFieldVectors (sourced from SQLConf.sessionLocalTimeZone), or adding a comment explaining why "UTC" is safe in this serialization context.

2. Test coverage in UtilsSuite. Nice round-trip test. Would you mind adding a case or two beyond IntegerType? A nullable StructType or ArrayType constant exercises a different ArrowFieldWriter code path (via getStruct(rowId) / getArray(rowId) on ConstantColumnVector), and a TimestampType constant would pin down the timezone choice in case anyone later changes it. The materializer claims to support every type and it'd be useful to prove it for at least the complex case.

3. FFI exportBatch arm. It shares materializeConstantColumnVector with getBatchFieldVectors, so the materializer itself is covered. The surrounding code (Data.exportVector + the allocator handoff) isn't exercised by the new test. If easy, a small smoke test that runs a ConstantColumnVector-containing batch through an actual Comet operator path would catch a regression in the FFI wiring. Optional.

4. CI flake. The one failing check (Spark 4.1 sql_core-3) annotates as The hosted runner lost communication with the server — infrastructure, not the PR. The matching sql_core-3 job on Spark 3.5 passes. A re-run should clear it.

[schenksj]

Thanks for the thorough review, @andygrove! All four points are addressed. (Disclosure: this PR's changes and this response were written with help from AI — Claude Code.)

1. Hardcoded "UTC" timezone. I went with the comment route, because on investigation "UTC" turns out to be the correct choice here rather than just a defensible one — and threading the session-local timezone would actively introduce the mismatch you were worried about. These constants are materialized alongside the batch's non-constant columns in the same VectorSchemaRoot, and Comet's non-constant TimestampType columns are Arrow vectors exported from native execution, where Comet always tags them Timestamp(µs, "UTC") (see native serde.rs). Spark itself stores TimestampType as micros in UTC, so the constant's value is already a UTC instant. Tagging the materialized constant "UTC" keeps its Arrow timezone metadata consistent with its sibling timestamp columns; threading SQLConf.sessionLocalTimeZone would make them diverge. I've expanded the comment in materializeConstantColumnVector to spell this out.

2. Test coverage in UtilsSuite. Added two cases beyond IntegerType:

• A nullable StructType constant (with a null nested field and a wholly-null struct) — exercises the getStruct(rowId) / getChild(ordinal) ArrowFieldWriter path.
• A TimestampType constant — round-trips the micros and pins down the "UTC" choice, guarding against anyone later swapping the zone argument.

3. FFI exportBatch arm. Done — added NativeUtilSuite, which runs a ConstantColumnVector-containing batch (scalar value, null, and a struct constant) through the actual Arrow C Data Interface via NativeUtil.exportBatchToAddresses + importVector — the same export/import round trip getNextBatch performs in production, so it covers Data.exportVector + the allocator handoff. I verified it's a real guard: removing the ConstantColumnVector arm in exportBatch makes it fail with Comet execution only takes Arrow Arrays.

4. CI flake. Agreed — the Spark 4.1 / sql_core-3 failure was the runner-setup infra flake. I've also merged latest main into the branch (it was out of date with base), so a fresh CI run picks all of this up.