[Parquet]: GH-563: Make path_in_schema optional

[etseidl]

Which issue does this PR close?

none

Rationale for this change

This is a proof of concept implementation for apache/parquet-format#563

What changes are included in this PR?

Since version 57.0.0, this crate has been tolerant of a missing path_in_schema. This PR adds options to cease writing the field as well. The option defaults to continuing to write the field.

See related discussion on parquet mailing list: https://lists.apache.org/thread/czm2bk45wwtkhhpqxqvmx9dk5wkwk1kt

Are these changes tested?

Yes

Are there any user-facing changes?

No, this only adds an optional behavior change that defaults to no change

Related PRs

• Make path_in_schema optional parquet-format#563
• GH-563: Make ColumnMetaData.path_in_schema optional parquet-format#564
• [Parquet]: GH-563: Make path_in_schema optional parquet-java#3470

[alamb]

less bloat for the win!

[jhorstmann]

A lot of other parquet implementations require this field, due to their generated thrift parser, even if they do not actually use the field for anything. I would be totally in favor of deprecating and skipping the field, but maybe a more compatible alternative would be to write the field as an empty list instead.

[etseidl]

A lot of other parquet implementations require this field, due to their generated thrift parser, even if they do not actually use the field for anything. I would be totally in favor of deprecating and skipping the field, but maybe a more compatible alternative would be to write the field as an empty list instead.

Well, parquet-java actually uses the field to populate its version of ColumnDescriptor, so an empty list will be just as damaging to an old version. The whole idea is to change the field to optional in the thrift definition, and give the ecosystem a few years for that change to percolate. After some reasonable time has passed we can default to not writing the field. But in the meantime, users who have data sensitive to metadata bloat and know they have up-to-date tooling can help themselves earlier.

[alamb]

A lot of other parquet implementations require this field, due to their generated thrift parser, even if they do not actually use the field for anything. I would be totally in favor of deprecating and skipping the field, but maybe a more compatible alternative would be to write the field as an empty list instead.

Well, parquet-java actually uses the field to populate its version of ColumnDescriptor, so an empty list will be just as damaging to an old version. The whole idea is to change the field to optional in the thrift definition, and give the ecosystem a few years for that change to percolate. After some reasonable time has passed we can default to not writing the field. But in the meantime, users who have data sensitive to metadata bloat and know they have up-to-date tooling can help themselves earlier.

See also related mailing list discussion:

• https://lists.apache.org/thread/czm2bk45wwtkhhpqxqvmx9dk5wkwk1kt

[alamb]

TLDR is I think this is a great idea. I also think it woudl be ok to merge this into arrow-rs even if there is not a broader consensus on the mailing list that we should do it in the format itself

My thinking is that some usecases are basically using parquet with the same read/writer and compatibility with older java based implementations is not important. This is the same thing for some of the newer encodings too

Letting users choose between "better/faster/stronger" and "more compatible" I think is very much a good idea

[alamb]

If we choose to go this way I think it would help to give some more context here on what types of readers would be affected (basically all the parquet-java based readers prior to late 2026)

We could also perhaps provide a link to the discussion: https://lists.apache.org/thread/czm2bk45wwtkhhpqxqvmx9dk5wkwk1kt

[alamb]

I think it is worth making it more apparently that this will cause incompatibilities with some older readers:

Suggested change

	/// Should the writer should emit the `path_in_schema` element of the
	/// `ColumnMetaData` Thrift struct.
	/// Should the writer should emit the `path_in_schema` element of the
	/// `ColumnMetaData` Thrift struct. WARNING: this will make the parquet
	/// file unreadable by some older parquet readers. See LINK HERE for details

[alamb]

I added a list of related PRs to this issue's description:

• Make path_in_schema optional parquet-format#563
• GH-563: Make ColumnMetaData.path_in_schema optional parquet-format#564
• [Parquet]: GH-563: Make path_in_schema optional parquet-java#3470

[alamb]

I wonder if we should just merge this into the Rust implementation of Parquet (with a caveat that this will make metadata smaller / more effiicent at the cost of incompatibility with older / java based readers)?

[etseidl]

I'm game. I don't see a reason not to since it's opt in

[alamb]

Thanks @etseidl -- I will try and review this more carefully shortly. I need to work on FSB stuff for a while first

[alamb]

Should we perhaps make a new benchmark so we can evaluate the performance both in default mode and wihtout this part of the metadata?

[etseidl]

Yes, that's probably worthwhile. I had made this modification for the purposes of the parquet-format submission.

[etseidl]

run benchmark metadata

[adriangbot]

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highcpu-48 (12 vCPU / 65 GiB) | Linux bench-c4361872261-1967-zrqkp 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  48
On-line CPU(s) list:                     0-47
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     48
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               3 MiB (48 instances)
L1i cache:                               3 MiB (48 instances)
L2 cache:                                96 MiB (48 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-47
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing deprecate_path_in_schema (feaae15) to f725bc9 (merge-base) diff
BENCH_NAME=metadata
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench metadata
BENCH_FILTER=
Results will be posted here when complete

---

File an issue against this benchmark runner

[adriangbot]

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highcpu-48 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  48
On-line CPU(s) list:                     0-47
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     48
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               3 MiB (48 instances)
L1i cache:                               3 MiB (48 instances)
L2 cache:                                96 MiB (48 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-47
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                               deprecate_path_in_schema               main
-----                                               ------------------------               ----
decode metadata (wide) with schema                  1.01     26.8±0.40ms        ? ?/sec    1.00     26.5±0.42ms        ? ?/sec
decode metadata (wide) with skip PES                1.00     26.3±0.37ms        ? ?/sec    1.00     26.2±0.43ms        ? ?/sec
decode metadata (wide) with skip all stats          1.00     28.2±0.31ms        ? ?/sec    1.00     28.3±0.29ms        ? ?/sec
decode metadata (wide) with skip column stats       1.01     27.6±0.38ms        ? ?/sec    1.00     27.3±0.36ms        ? ?/sec
decode metadata (wide) with skip size stats         1.00     30.4±0.37ms        ? ?/sec    1.00     30.4±0.37ms        ? ?/sec
decode metadata (wide) with stats mask              1.01     26.4±0.38ms        ? ?/sec    1.00     26.3±0.32ms        ? ?/sec
decode metadata with schema                         1.00      4.1±0.01µs        ? ?/sec    1.01      4.1±0.01µs        ? ?/sec
decode metadata with skip PES                       1.00      6.9±0.05µs        ? ?/sec    1.01      7.0±0.04µs        ? ?/sec
decode metadata with skip column stats              1.00      6.7±0.02µs        ? ?/sec    1.00      6.8±0.03µs        ? ?/sec
decode metadata with stats mask                     1.00      6.9±0.02µs        ? ?/sec    1.01      7.0±0.04µs        ? ?/sec
decode parquet metadata                             1.00      7.1±0.01µs        ? ?/sec    1.02      7.3±0.04µs        ? ?/sec
decode parquet metadata (wide)                      1.00     28.8±0.37ms        ? ?/sec    1.00     28.7±0.42ms        ? ?/sec
decode parquet metadata no path_in_schema (wide)    1.00     28.1±0.41ms        ? ?/sec  
decode parquet metadata w/ size stats (wide)        1.00     35.7±0.43ms        ? ?/sec    1.00     35.6±0.47ms        ? ?/sec
open(default)                                       1.00      7.7±0.02µs        ? ?/sec    1.02      7.8±0.02µs        ? ?/sec
open(page index)                                    1.00    132.7±0.35µs        ? ?/sec    1.00    132.7±0.37µs        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	145.0s
Peak memory	4.3 GiB
Avg memory	4.2 GiB
CPU user	141.8s
CPU sys	0.8s
Peak spill	0 B

branch

Metric	Value
Wall time	155.0s
Peak memory	4.3 GiB
Avg memory	4.2 GiB
CPU user	150.9s
CPU sys	0.2s
Peak spill	0 B

---

File an issue against this benchmark runner

[alamb]

I think this PR is ready except for:

1. Don't change the existing writer benchmark (otherwise we are not benchmarking with default settings)
2. Update the docs a bit more to explain the implications of not writing path_in_schema

[etseidl]

I think this PR is ready except for:

1. Don't change the existing writer benchmark (otherwise we are not benchmarking with default settings)

2. Update the docs a bit more to explain the implications of not writing path_in_schema

I tried to address these in 2a06837. PTAL

[alamb]

I had a few comments that github won't seem to let me post

It might help to explain what files it helps the most for:

    /// The `path_in_schema` field in the Thrift metadata is redundant and wastes a sometimes
    /// significant amount of space, especially for files with long column names or deeply nested
    /// schema. Parquet file footers can be made smaller and easier to
    /// parse by omitting this field.

It might help to be a little more specific about what readers this effects

    /// Setting this to `false` will break compatibility with Parquet readers that
    /// still expect this field to be present. Many older Parquet readers, such as parquet-java
    /// but with the exception
    /// of the one in this crate, expect this field to be present, and will terminate execution
    /// if it is not. This will continue to be the case unless/until the Parquet format
    /// specification is explicitly changed to allow this field to be missing. As a consquence,
    /// users should only set this to `false` if they have verified that any reader(s) they plan
    /// to use can tolerate the absence of this field.
    ///
    /// For more context, see [GH-563].

[etseidl]

Thanks @alamb. I added some more motivation for it's use. Please take another look 🙏

[etseidl]

I had a few comments that github won't seem to let me post

I've been having quite a few github problems lately :(

[alamb]

I had a few comments that github won't seem to let me post

I've been having quite a few github problems lately :(

Perhaps related to this: https://github.blog/news-insights/company-news/an-update-on-github-availability/

(aka everyone is having them 😆 )

[alamb]

Thanks @etseidl

[emkornfield]

TLDR is I think this is a great idea. I also think it woudl be ok to merge this into arrow-rs even if there is not a broader consensus on the mailing list that we should do it in the format itself My thinking is that some usecases are basically using parquet with the same read/writer and compatibility with older java based implementations is not important. This is the same thing for some of the newer encodings too

@etseidl @alamb This is true, but if you follow this logic to is conclusion, then each implementation could come up with new encodings themselves without broader consensus in the ecosystem? Is that the direction the arrow-rs implementation aims to take?

IIUC this would allow writing files that would fail parsing from other readers due to the field currently being required? If so, while it seems like there is general consensus on the parquet mailing list to transition to this, doing before it is adopted in parquet-format seems like a small risk of fragmenting the ecosystem?

CC @wgtmac

[alamb]

IIUC this would allow writing files that would fail parsing from other readers due to the field currently being required?

Well, some other readers based on older java would fail but some (e.g. DuckDB, polars) can read the files just fine.

If so, while it seems like there is general consensus on the parquet mailing list to transition to this, doing before it is adopted in parquet-format seems like a small risk of fragmenting the ecosystem?

This is true. though the same argument can be applied to V2 Data pages and other encodings like byte stream split that are not supported by other implementations.

In my mind there are basically two benefits to using the (Rust) implementation of parquet:

1. Interoperability with other systems
2. Reuse all the engineering that has gone into the implementation (though you don't plan to share the files)

I think there are a bunch of use cases where systems use parquet internally and either

• don't care about interoperability as the data never leaves there systems or
• have to rewrite the files for interoperability anyways (e.g. convert nanosecond --> millisecond timestamps)

Many of the early adopters of Vortex fall into this second category (as does InfluxData).

My goal with this setting is to cater to the second category and allow people to take advantage of the all the engineering in the Rust implementation

[etseidl]

@emkornfield I understand your concern, and would not normally be in favor of putting the cart this far in front of the horse. But in this specific instance, we have a situation where we do not internally depend on the field, and for the last two releases have not even parsed it. We felt that the potential gains were worth making available to downstream users as an experimental feature, albeit with very prominent warnings that use of the feature will break compatibility. Call me cautiously optimistic that the spec will follow soon.