GH-430: Add vector logical type and a VECTOR repetition type

[rok]

Rationale for this change

As discussed in #430, fixed-size list / vector data in Parquet is currently encoded as standard LIST, even when every value has the same number of elements. This forces readers and writers through variable-length list machinery and repetition-level handling for data whose shape is known from the schema.

This is increasingly relevant for embeddings, tensors, images, and other fixed-shape arrays.

The topic was discussed on the mailing list and in a design doc. Several approaches were prototyped (A, B, C) and benchmarked.

What changes are included in this PR?

This PR adds a new VECTOR logical and repetition types for fixed-size lists.

Thrift additions:

• FieldRepetitionType.VECTOR = 3
• struct VectorType {}
• LogicalType.VECTOR = 19
• SchemaElement.vector_length = 11

The layout for vector data would look like so:

<required|optional> group emb (VECTOR) {
  vector group list [N] {
    <required|optional> <element-type> element;
  }
}

Key semantics:

• LogicalType.VECTOR annotates the outer group.
• The middle group has repetition_type = VECTOR and carries SchemaElement.vector_length.
• The VECTOR repetition level contributes no definition or repetition level.
• Nullable vectors are represented by an optional outer group.
• Nullable elements are represented by an optional element field.
• Each vector occurrence contributes exactly vector_length element slots / level entries.
• For each primitive leaf in the vector element subtree, num_values counts vector element slots / level entries, not necessarily physical values.
• Statistics remain element-level statistics.
• Encodings remain element-level encodings.
• Writers should not split one vector value across data pages.
• Readers that do not support FieldRepetitionType.VECTOR should reject the file rather than interpreting the field as another repetition type.

Do these changes have PoC implementations?

• feat(parquet): add experimental VECTOR repetition for Arrow FixedSizeList arrow-go#854

Closes #430

[adamreeve]

This wording is a bit confusing to me. What's the significance of "non-repeated" here? Does this mean that the vector repetition type cannot be used on a descendant of a node that is repeated? Or is this referring to the top-level group node in the 3-level schema being required or optional, and not repeated?

[pitrou]

Can you explain somewhere why the new logical type is useful? The new repetition type might be sufficient, no?

[adamreeve]

Can you explain somewhere why the new logical type is useful? The new repetition type might be sufficient, no?

This is required to support nullable vectors and nullable vector elements in a standardised way. Although both of those require definition levels so will come with some performance impact. This also makes vectors consistent with the existing list logical type.

The new repetition type alone would be sufficient for the common use case of required vectors and required elements.