GH-946: Add Variant extension type support #947

tmater · 2026-01-07T09:23:21Z

Summary

This PR adds support for the Variant extension type in Arrow Java, enabling storage and manipulation of semi-structured variant data with metadata and value buffers.

Changes

A new arrow-variant module introduces the Variant class for parsing and working with variant data. This module is separated from the core vector module to isolate the parquet-variant dependency, so users of the Arrow vector library don't have to depend on Parquet. This also maintains a clean API boundary between Arrow's core functionality and variant-specific parsing logic.

The core vector module gains VariantType as an extension type along with VariantVector for storing variant data as metadata/value buffer pairs. The implementation includes reader and writer support through VariantReaderImpl, VariantWriterImpl, and NullableVariantHolderReaderImpl, with corresponding holder classes for use in generated code paths.

Testing

Unit tests for VariantType, VariantVector, and Variant parsing
Integration tests with ListVector and MapVector
Extension type round-trip tests

Closes #946

lidavidm · 2026-01-07T11:13:53Z

We're about to overhaul the API for implementing extension writers so you may want to hold off

tmater · 2026-01-07T13:21:29Z

We're about to overhaul the API for implementing extension writers so you may want to hold off

Thanks @lidavidm, I will wait for that, then rebase. You are talking about the ExtensionTypeWriterFactory related changes, right?

gszadovszky · 2026-01-21T15:12:49Z

arrow-variant/src/main/java/org/apache/arrow/variant/extension/VariantVector.java

+  @Override
+  public Object getObject(int index) {
+    return getUnderlyingVector().getObject(index);
+  }


I'm not sure if it is useful this way. I don't have a strong opinion though what should it return instead. @lidavidm, do you have an idea?

As it stands, this returns the raw struct row? I suppose decoding to a Java object may be what's expected but that may be expensive. Or returning an explicit Variant object that has methods for inspecting/manipulating the data in that row

I'm wondering if the variant representation of parquet-java would be suitable here. That is a java object backed by nio ByteBuffers. It would be beneficial as arrow-java would not need to maintain/test the related code. Meanwhile, it does not support returning "rich" objects (e.g. timestamps) but the primitive representations only (e.g. long for a timestamp). I'm not sure how it would fit the current concept.
Alternatively, we would be able to wrap the parquet-java Variant object and extend it with those rich types the different Arrow vectors support (e.g. LocalDateTime for a timestamp nano value).

I think if we go this route, I'd rather have an Arrow Variant to avoid having to break APIs down the line if we decide to extend functionality

To be sure I understand you correctly, @lidavidm, do you agree with the following?

Add the new parquet-java dependency and use its variant implementation as a bases of a new Arrow Variant class to be returned by VariantVector.getObject. This class can return the corresponding java objects for the related primitives according to the java objects returned by their typed vectors. No parquet-java classes shall face the public API of Arrow.

I agree with arrow-parquet-variant approach. It cleanly isolated the dependency (to avoid any side effect in other parts of Arrow).

Sorry, I've misinterpreted @jbonofre's suggestion. I wanted to reference the parquet-java's parquet-variant module. That contains only 10 prod classes. But right, it has a dependency on other parquet-java modules as well. So, we have two options:

Create the separate arrow-variant module that contains any variant related classes, including the ones already included in this PR.

Duplicate the variant related code from parquet-java, and implement our own classes based on it in the existing vector module.

I do not have a strong opinion which one would be the best choice for Arrow.

Yes, that was my concern about parquet-variant: the "parent" and transitive dependencies. I don't think we should fetch a large part of Parquet in Arrow.

I have a preference to create arrow-variant module with clearly defined dependencies.

Thoughts ?

Sounds good to me. @tmater, could you update your PR accordingly?

Thanks for the discussion! I've updated the PR to introduce a separate arrow-variant module that depends on parquet-variant for the binary format handling. The Variant class wraps the Parquet implementation and is returned by VariantVector.getObject, keeping Parquet classes out of the public Arrow API.
Let me know what you think!

Implements VariantType extension type with VariantVector for storing variant data with metadata and value buffers. Includes reader/writer implementations and comprehensive test coverage.

gszadovszky

All variant related classes and code changes shall be placed in the new arrow-variant module so the user can decide if they need variant and accept the related transitive dependencies. No other arrow-java module shall depend on arrow-variant.

flight/flight-sql-jdbc-driver/pom.xml

arrow-variant/src/main/java/org/apache/arrow/variant/extension/VariantType.java

arrow-variant/src/main/java/org/apache/arrow/variant/extension/VariantVector.java

vector/src/main/java/module-info.java

vector/src/test/java/org/apache/arrow/vector/types/pojo/TestExtensionType.java

vector/src/test/java/org/apache/arrow/vector/TestListVector.java

vector/pom.xml

gszadovszky · 2026-01-28T09:28:01Z

arrow-variant/src/main/java/org/apache/arrow/variant/Variant.java

+  public ByteBuffer getValueBuffer() {
+    return delegate.getValueBuffer();
+  }
+
+  public ByteBuffer getMetadataBuffer() {
+    return delegate.getMetadataBuffer();
+  }
+


I don't think we should expose these to the user.

This follows how parquet-variant defines it, the ByteBuffer is read-only.
I’m happy to change it; I just wanted to call it out.

I'm not sure it is useful for the users, but do not have a strong opinion to remove it.

gszadovszky

Thank you for the changes, @tmater. It looks good to me.
@lidavidm, @jbonofre, could you check it out and also approve the awaiting workflows to validate the change?

jbonofre · 2026-01-28T14:46:37Z

@gszadovszky thanks ! I just ran the CI and fix the PR meta (milestone). I will do a new pass asap.

tmater requested review from jbonofre, laurentgo, lidavidm and wgtmac as code owners January 7, 2026 09:23

This comment has been minimized.

Sign in to view

tmater force-pushed the variant_logical branch from 2b45ca4 to a297cc6 Compare January 15, 2026 16:13

gszadovszky reviewed Jan 21, 2026

View reviewed changes

lidavidm added the enhancement PRs that add or improve features. label Jan 22, 2026

tmater added 3 commits January 28, 2026 09:48

apacheGH-946: Add Variant extension type support

479ea37

Implements VariantType extension type with VariantVector for storing variant data with metadata and value buffers. Includes reader/writer implementations and comprehensive test coverage.

rebase on Add ExtensionTypeWriterFactory to TransferPair

27ee846

rebase, introduce arrow-variant module, add complex tests

3a0d927

tmater force-pushed the variant_logical branch from a297cc6 to 3a0d927 Compare January 28, 2026 08:49

gszadovszky suggested changes Jan 28, 2026

View reviewed changes

tmater added 2 commits January 28, 2026 14:05

Move Variant classes from arrow-vector to arrow-variant module

b76a0ea

Spotless + Checkstyle

23549e6

tmater requested a review from gszadovszky January 28, 2026 13:26

gszadovszky approved these changes Jan 28, 2026

View reviewed changes

jbonofre added this to the 19.0.0 milestone Jan 28, 2026

GH-946: Add Variant extension type support #947

Are you sure you want to change the base?

GH-946: Add Variant extension type support #947

Uh oh!

Conversation

tmater commented Jan 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing

Uh oh!

This comment has been minimized.

lidavidm commented Jan 7, 2026

Uh oh!

tmater commented Jan 7, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tmater Jan 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gszadovszky left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gszadovszky left a comment

Choose a reason for hiding this comment

Uh oh!

jbonofre commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

tmater commented Jan 7, 2026 •

edited

Loading

tmater Jan 28, 2026 •

edited

Loading