Add support for FixedSizeList to variant_to_arrow by rishvin · Pull Request #9663 · apache/arrow-rs

rishvin · 2026-04-06T02:17:21Z

Closes [Variant] Add variant_to_arrow FixedSizeList type support #9531

Rationale for this change

Add support for FixedSizeList when invoking variant_to_arrow.

What changes are included in this PR?

Introduces a new builder VariantToFixedSizeListArrowRowBuilder.
Adds test cases for shredding and getting variant by FixedSizeList.

Are these changes tested?

By adding few test cases.

Are there any user-facing changes?

N/A.

alamb · 2026-04-06T19:47:47Z

@sdf-jkl could you help review this PR?

sdf-jkl

Thanks @rishvin, I left some suggestions.

rishvin · 2026-04-11T21:25:08Z

Thanks @sdf-jkl for reviewing changes. I have addressed the comments, could you re-review ?

sdf-jkl

Thanks for your work on this @rishvin! Everything looks great, I left some nits.

sdf-jkl · 2026-04-13T14:31:53Z

 }

 impl<'a> ListElementBuilder<'a> {
+    fn append_null(&mut self) -> Result<()> {


Maybe add a comment mentioning this is only used for FixedSizeList builder?

Co-authored-by: Konstantin Tarasov <33369833+sdf-jkl@users.noreply.github.com>

rishvin · 2026-04-13T19:24:37Z

Thanks @sdf-jkl addressed new comments.

sdf-jkl

One tiny change.

Co-authored-by: Konstantin Tarasov <33369833+sdf-jkl@users.noreply.github.com>

rishvin · 2026-04-14T04:25:39Z

Thanks again @sdf-jkl ! Addressed it.

sdf-jkl

Thanks @rishvin! LGTM.

@scovich could you please check this when available? Thanks!

scovich · 2026-04-14T15:33:25Z

+                // With `safe` cast option set to false, appending list of wrong size to
+                // `typed_value_builder` of type `FixedSizeList` will result in an error. In such a
+                // case, the provided list should be appended to the `value_builder.


I'm not sure the variant shredding spec allows for shredding as a fixed size list, if the resulting layout differs physically from a normal list?

Arrays can be shredded by using a 3-level Parquet list for typed_value.

If the value is not an array, typed_value must be null. If the value is an array, value must be null.

It looks to me like any attempt to shred as fixed-sized list must either succeed (if the size is correct) or hard-fail (because value as fallback is not allowed).

It differs physically on the Arrow side, but once we write it to Parquet it'd be same as other ListLikeArrays. But this leads to further discussion on adding FixedSizeList support for VariantArray as well as implementing other types, currently not supported in spec.

We're keeping value because we consider this a cast from Variant to FixedSizeList. The extra len check is there because there is no Variant::FixedSizeList enum to match to. If len is incorrect we consider the cast failed and proceed following the safe cast option as if typed_value is Null.

When casting from variant to arrow, we can do whatever we want.

But this code here is about going from binary variant to shredded variant. And the variant shredding spec directly forbids value to contain a variant array, when shredding as array.

True. I think the core issue is that Parquet currently has only one logical LIST type. If Parquet had a dedicated logical type for FixedSizeList, the spec wording could be more explicit.

Btw, there’s ongoing work on this too: apache/parquet-format#241 (recently revived).

Given the current spec text:

Arrays can be shredded by using a 3-level Parquet list for typed_value. If the value is not an array, typed_value must be null. If the value is an array, value must be null.

I read “array” as "a value matching the specific list shape we’re shredding into". For List/LargeList/ListView it's List values, for FixedSizeList array it's a FixedSizeList value.

From what I understand, the variant spec neither knows nor cares about the intricacies of arrow array types (it also doesn't care about spark or SQL). If we're shredding to a 3-level parquet list, and we encounter a variant array value, the resulting value column entry must be null.

True, the use case should be the first thing we think about before implementing.

FSL could be useful for embeddings and tensors but the question is why would anyone put them inside a Variant for such a heavy compute work? Maybe coordinates inside a log struct?

The fact that we can't store FSL in Parquet is a big reason to not do it.

FixedSizeBinary actually does have a Parquet representation - FIXED_LEN_BYTE_ARRAY and is used for shredding into a UUID, but other than UUID, what could be a use case for storing inside a Variant?

Overall, I think we'd not gain much by including FSL because it's so frail and the use cause is not defined. Isn't the whole point of Variant is to host semi-structured data, if we have fixed-len-something it'd be better to keep a separate column for it. If fixed-length is a logical constraint rather than a physical one, enforcement belongs downstream.

Overall, I think we'd not gain much by including FSL because it's so frail and the use cause is not defined. Isn't the whole point of Variant is to host semi-structured data, if we have fixed-len-something it'd be better to keep a separate column for it. If fixed-length is a logical constraint rather than a physical one, enforcement belongs downstream.

Agree. I just realized that somebody could create a FSL of [shredded] variant, if the use case just needs flexible typing of N list members. But that also seems somewhat contrived, to have such strongly-typed structure but sufficient uncertainty about the list element type to merit using variant.

Should we then keep it hard fail and leave it to the user?

Catching up late on this. I'm aligned with keeping it a hard fail and leaving it to the user.

Thanks all for reviewing these changes and apologies for delays.
Could you please re-review.

…_arrow

sdf-jkl

LGTM, thanks @rishvin

alamb

Thank you @rishvin and @sdf-jkl -- looks good to me

scovich

LGTM!

scovich · 2026-05-20T20:37:02Z

+                self.typed_value_builder
+                    .append_value(&Variant::List(list))?;


Aside: a bit surprised fmt liked this line break?

sometimes fmt seems to give up for some reason 🤷

alamb · 2026-05-20T20:44:03Z

🚀

Add support for FixedSizeList to variant_to_arrow

808b1b5

github-actions Bot added the parquet-variant parquet-variant* crates label Apr 6, 2026

Merge branch 'main' into fixed_size_list_variant_to_arrow

1d55ade

rishvin and others added 2 commits April 6, 2026 13:45

Fixed test case

11ffd40

Merge branch 'main' into fixed_size_list_variant_to_arrow

4a84d12

sdf-jkl suggested changes Apr 7, 2026

View reviewed changes

rishvin and others added 2 commits April 11, 2026 09:22

Merge branch 'main' into fixed_size_list_variant_to_arrow

64bf4aa

Addressed comments.

99c6cc5

rishvin requested a review from sdf-jkl April 11, 2026 21:24

Merge branch 'main' into fixed_size_list_variant_to_arrow

47f99c5

sdf-jkl reviewed Apr 13, 2026

View reviewed changes

rishvin and others added 2 commits April 13, 2026 11:53

Apply suggestions from code review

384ed29

Co-authored-by: Konstantin Tarasov <33369833+sdf-jkl@users.noreply.github.com>

Addressed remanining comments

220427f

rishvin requested a review from sdf-jkl April 13, 2026 19:24

Merge branch 'main' into fixed_size_list_variant_to_arrow

acd294a

sdf-jkl reviewed Apr 14, 2026

View reviewed changes

Comment thread parquet-variant-compute/src/shred_variant.rs Outdated

Update parquet-variant-compute/src/shred_variant.rs

d2e4dd6

Co-authored-by: Konstantin Tarasov <33369833+sdf-jkl@users.noreply.github.com>

rishvin requested a review from sdf-jkl April 14, 2026 04:25

sdf-jkl approved these changes Apr 14, 2026

View reviewed changes

scovich reviewed Apr 14, 2026

View reviewed changes

rishvin added 2 commits May 18, 2026 09:31

Merge remote-tracking branch 'origin' into fixed_size_list_variant_to…

be5d40e

…_arrow

Hard fail when shredding with incorrect size

302b8d7

sdf-jkl approved these changes May 18, 2026

View reviewed changes

rishvin requested a review from scovich May 20, 2026 16:25

alamb approved these changes May 20, 2026

View reviewed changes

scovich approved these changes May 20, 2026

View reviewed changes

alamb merged commit dc821a9 into apache:main May 20, 2026
17 checks passed

rishvin deleted the fixed_size_list_variant_to_arrow branch May 20, 2026 20:48

		self.typed_value_builder
		.append_value(&Variant::List(list))?;

Conversation

rishvin commented Apr 6, 2026 • edited by alamb Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

alamb commented Apr 6, 2026

Uh oh!

sdf-jkl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rishvin commented Apr 11, 2026

Uh oh!

sdf-jkl left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rishvin commented Apr 13, 2026

Uh oh!

sdf-jkl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rishvin commented Apr 14, 2026

Uh oh!

sdf-jkl left a comment

Choose a reason for hiding this comment

Uh oh!

scovich Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdf-jkl Apr 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sdf-jkl left a comment

Choose a reason for hiding this comment

Uh oh!

alamb left a comment

Choose a reason for hiding this comment

Uh oh!

scovich left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rishvin commented Apr 6, 2026 •

edited by alamb

Loading

scovich Apr 14, 2026 •

edited

Loading

sdf-jkl Apr 14, 2026 •

edited

Loading