fix(arrow): read null values from a FixedSizeList Parquet column

[ChaomingZhangCN]

Purpose

Follow-up to the VECTOR work in apache/paimon-cpp#198, kept separate because it changes the bundled Arrow build rather than Paimon code. It is based on codex/vector-parquet-mvp, so it can only merge after that branch lands.

A VECTOR column written by an engine that stores the Arrow schema, such as Paimon Rust or Python, is read back as Arrow FixedSizeList. Parquet stores a null list slot without any value, while Arrow 17's FixedSizeListReader::AssembleArray requires every slot, including a null one, to span exactly list_size values. Reading such a file therefore failed with:

Invalid: Expected all lists to be of size=3 but index 2 had size=0

There is no way to avoid this from the Paimon side: the reader materializes the column per the Arrow schema stored in the file, and parquet::FileMetaData exposes its key-value metadata read-only, so the stored FixedSizeList type cannot be bypassed.

This change backports the upstream fix (apache/arrow GH-50271) into cmake_modules/arrow.diff, which the project already uses to patch cpp/src/parquet/arrow/reader.cc:

• Walk the validity runs of the assembled data. A valid run reuses the decoded child slice; a null run materializes list_size null child values per slot, so the fixed-size layout is restored.
• Validate offsets per run: a valid slot must span list_size values, a null slot must span none.
• ListReader members become protected so FixedSizeListReader can reach the memory pool. The upstream code uses std::span, which is rewritten with plain pointer arithmetic to stay C++17.

Tests

ParquetVectorIoTest.ReadNullableRustFixtureIsUnsupported, which pinned the old failure against the rust_vector_nullable.parquet fixture (arrow-rs 58.4.0, FixedSizeList<Float32, 3> with a null vector), becomes ReadNullableRustFixture and now checks the values that are read back.

Validation, all against the patched Arrow:

cmake --build build --target paimon-parquet-format-test paimon-common-test paimon-core-test paimon-write-and-read-inte-test -j 4
./build/debug/paimon-parquet-format-test        # 194 passed
./build/debug/paimon-common-test                # 1443 passed
./build/debug/paimon-core-test                  # 1674 passed
./build/debug/paimon-write-and-read-inte-test   # 141 passed

The patch was also verified to apply to a pristine apache-arrow-17.0.0 tree with patch -s -N -p1 and to reproduce exactly the source that these runs were built from.

API and Format

API: No.

Schema protocol: No.

Storage format: No. Paimon C++ still writes VECTOR as a standard Parquet LIST; only the read path for files that record the column as FixedSizeList changes.

Documentation

The VECTOR section of docs/source/user_guide/data_types.rst no longer needs to warn that a FixedSizeList file with NULL vectors cannot be read, and the fixture notes record what the patch does.

Generative AI tooling

Generated-by: Cursor Cloud Agent (Claude Opus 5)