Core: Basic fields and schemas for column files by gaborkaszab · Pull Request #16285 · apache/iceberg

gaborkaszab · 2026-05-11T13:52:03Z

This change introduces the interface for column files and also integrates it to the schema for TrackedFile.

gaborkaszab · 2026-05-11T16:02:14Z

First piece of the column update work: introducing the basic interface of the column updates files, aka column files
cc @anuragmantri @rdblue @pvary @RussellSpitzer @amogh-jahagirdar @anoopj @nastra

gaborkaszab · 2026-05-21T13:56:27Z

I opened a thread on dev@ to discuss the metadata structs for column files. Once that's finalized, I'll incorporate the changes here.

gaborkaszab · 2026-06-02T13:13:44Z

cc @amogh-jahagirdar @rdblue @anoopj

gaborkaszab · 2026-06-08T12:33:22Z

Adjusted field IDs because 157 is going to be allocated for writer_format_version in this PR.

gaborkaszab · 2026-06-12T14:39:56Z

+      this.status = EntryStatus.MODIFIED;
+    }
+    // Bumping 'dataSequenceNumber' to avoid having both equality deletes and column files.
+    this.dataSequenceNumber = null;


We discussed bumping the data sequence number when adding column files. We haven't mentioned file seq num, so I'm not bumping it here.
This works if the manifest owning this data file entry bumps its own seq num when adding column files. Let me know if there is any other way achieving this.

gaborkaszab · 2026-07-10T12:39:40Z

Rebased with main because there were overlapping activity.

Will re-think what fields should be part of column files. Coming back with an update soon

RussellSpitzer · 2026-07-14T15:28:24Z

+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/** Information about a column file. */


nit: Not a helpful javadoc, either needs more information about what a column file is or drop

Decided to drop the javadoc

RussellSpitzer · 2026-07-14T15:30:07Z

+  Types.NestedField FORMAT_VERSION =
+      Types.NestedField.required(
+          161, "format_version", Types.IntegerType.get(), "Format version of this column file");
+  Types.NestedField FIELD_IDS =


If we keep the field id's here, does that mean we need to update all column file entities whenever we add a new column file? I think that's probably alright but we should probably add some text on that requirement?

Yes, when we add a new column file, we have to remove its field IDs from other column files' field ID lists. I'm not sure we should document it here, this just the storage representation of such a structure. The "deduplication" of field IDs happen on a different level, maybe document this on the API where we add the new column files (non-existing as of now)?

RussellSpitzer · 2026-07-14T15:37:41Z

+interface ColumnFile {
+  Types.NestedField FORMAT_VERSION =
+      Types.NestedField.required(
+          161, "format_version", Types.IntegerType.get(), "Format version of this column file");


What format version? The Table format version when the file was written? Do we really need this?

I'd probably change this later, but the comment should probably be "Table Format version used to write this column file" or something

I followed the comment for the similar field in TrackedFile that says "Format version of this file".

RussellSpitzer · 2026-07-14T15:48:05Z

+
+    Builder formatVersion(int newFormatVersion) {
+      Preconditions.checkArgument(
+          newFormatVersion >= 0, "Invalid format version: %s (must be >= 0)", newFormatVersion);


If we are going to have this we should also block format version > MAX_FORMAT_VERSION

Not sure I get this, but isn't MAX_FORMAT_VERSION only used for tests? In real life, writers will provide their format version here, I'm not sure if we keep track what is the highest possible value we accept.

Maybe @stevenzwu can share if he has anything else in mind with this.

RussellSpitzer · 2026-07-14T15:48:57Z

+
+    Builder fieldIds(List<Integer> newFieldIds) {
+      Preconditions.checkArgument(newFieldIds != null, "Invalid field IDs: null");
+      Preconditions.checkArgument(!newFieldIds.isEmpty(), "Invalid field IDs: empty");


Does this mean we have to remove a column file if we override all the columns in it

Yes, this is what I have in mind. However, that's not a concern of this class, I figure we could have some sort of a builder for the TrackedFile's column file list, where we take care of that.

RussellSpitzer · 2026-07-14T16:15:27Z

+      Preconditions.checkArgument(!newFieldIds.isEmpty(), "Invalid field IDs: empty");
+      Preconditions.checkArgument(
+          Sets.newHashSet(newFieldIds).size() == newFieldIds.size(),
+          "Invalid field IDs: duplicateD IDs found: %s",


The message here makes it seem like it's about to display duplicate ids but it displays everything. I would either change the message a bit or actually list the duplicated ids.

Changed the message a bit

RussellSpitzer · 2026-07-14T16:25:13Z

+          160,
+          "latest_column_file_snapshot_id",
+          Types.LongType.get(),
+          "Snapshot ID where the latest column file was added; null if there are no column files");


I think this should mirror the SnapshotID

"Snapshot ID where the file was added, deleted, or replaced");

"Snapshot ID where the latest column file was added; deleted or replaced; null if there are no column files"

I'm not sure the same operations apply to column files as for the base file. A column file is added, but I don't think we want to delete or replace a column file as a separate operation (could be achieved by adding a new column file that eliminates one or more old ones). Just for additional context, this field is going to be used to return last_updated_sequence_number.

I think the same logic applies here as for the dv_snapshot_id where we say this:
"Snapshot ID where the DV was added; null if there is no DV"

RussellSpitzer · 2026-07-14T18:15:23Z

+          162,
+          "field_ids",
+          Types.ListType.ofRequired(163, Types.IntegerType.get()),
+          "Field IDs this column file contains");


"Live Field IDs in this column file"?

Sure, adjusted

gaborkaszab

Thank you for taking a look, @RussellSpitzer !
I made minor adjustments, mostly answered questions or explained concepts.

gaborkaszab · 2026-07-15T12:56:01Z

+import java.util.List;
+import org.apache.iceberg.types.Types;
+
+/** Information about a column file. */


Decided to drop the javadoc

gaborkaszab · 2026-07-15T12:57:51Z

+interface ColumnFile {
+  Types.NestedField FORMAT_VERSION =
+      Types.NestedField.required(
+          161, "format_version", Types.IntegerType.get(), "Format version of this column file");


I followed the comment for the similar field in TrackedFile that says "Format version of this file".

gaborkaszab · 2026-07-15T13:07:24Z

+  Types.NestedField FORMAT_VERSION =
+      Types.NestedField.required(
+          161, "format_version", Types.IntegerType.get(), "Format version of this column file");
+  Types.NestedField FIELD_IDS =


Yes, when we add a new column file, we have to remove its field IDs from other column files' field ID lists. I'm not sure we should document it here, this just the storage representation of such a structure. The "deduplication" of field IDs happen on a different level, maybe document this on the API where we add the new column files (non-existing as of now)?

gaborkaszab · 2026-07-15T13:15:54Z

+          162,
+          "field_ids",
+          Types.ListType.ofRequired(163, Types.IntegerType.get()),
+          "Field IDs this column file contains");


Sure, adjusted

gaborkaszab · 2026-07-15T13:24:18Z

+
+    Builder formatVersion(int newFormatVersion) {
+      Preconditions.checkArgument(
+          newFormatVersion >= 0, "Invalid format version: %s (must be >= 0)", newFormatVersion);


Not sure I get this, but isn't MAX_FORMAT_VERSION only used for tests? In real life, writers will provide their format version here, I'm not sure if we keep track what is the highest possible value we accept.

Maybe @stevenzwu can share if he has anything else in mind with this.

gaborkaszab · 2026-07-15T13:26:08Z

+
+    Builder fieldIds(List<Integer> newFieldIds) {
+      Preconditions.checkArgument(newFieldIds != null, "Invalid field IDs: null");
+      Preconditions.checkArgument(!newFieldIds.isEmpty(), "Invalid field IDs: empty");


Yes, this is what I have in mind. However, that's not a concern of this class, I figure we could have some sort of a builder for the TrackedFile's column file list, where we take care of that.

gaborkaszab · 2026-07-15T13:29:18Z

+      Preconditions.checkArgument(!newFieldIds.isEmpty(), "Invalid field IDs: empty");
+      Preconditions.checkArgument(
+          Sets.newHashSet(newFieldIds).size() == newFieldIds.size(),
+          "Invalid field IDs: duplicateD IDs found: %s",


Changed the message a bit

gaborkaszab · 2026-07-15T13:37:35Z

+          160,
+          "latest_column_file_snapshot_id",
+          Types.LongType.get(),
+          "Snapshot ID where the latest column file was added; null if there are no column files");


I'm not sure the same operations apply to column files as for the base file. A column file is added, but I don't think we want to delete or replace a column file as a separate operation (could be achieved by adding a new column file that eliminates one or more old ones). Just for additional context, this field is going to be used to return last_updated_sequence_number.

I think the same logic applies here as for the dv_snapshot_id where we say this:
"Snapshot ID where the DV was added; null if there is no DV"

gaborkaszab · 2026-07-23T13:49:18Z

Rebased with latest main to resolve git conflicts

amogh-jahagirdar

Still need to do a pass over tests, but had some comments on the changes.
The field IDs look reasonable to me. In terms of spec stuff on the entry structure, I remember there was an open quesiton around if we needed split offsets per column file or not. My take is yes, the files would have fundamentally different schemas and there may be more appropriate boundary split points in the file depending on that, but not sure if this had settled. Apologies if this had already reached a consensus and I missed it. @gaborkaszab @anuragmantri

amogh-jahagirdar · 2026-07-29T23:38:17Z

+  ByteBuffer keyMetadata();
+
+  /** Returns the list of recommended split locations for this column file, or null. */
+  List<Long> splitOffsets();


I remember there was a discussion around if we need split offsets per column file or not, what was the conclusion on that? I feel like it makes sense to have them, there may be different files with varying schemas and as a result there would be different boundaries that would be set appropriately for each file. It's additional complexity, but not much?

I brought this up once on the sync, we didn't have a deep dive but there were no objections either. Rational might be that when we project fields in a way that we don't have to read the base file, we can use the split offsets of one of the column files.
Now, with regular column updates, the column files usually contain a column or two and they have a single row group. However, later for column families we can have wider column files with more than one row groups where we can have split offsets.

amogh-jahagirdar · 2026-07-29T23:43:30Z

  }

  /** Copy constructor. */
+  @SuppressWarnings("CyclomaticComplexity")


Any reasonable way to tighten this, or is it just inherent complexity from all the null handling we need to do?

Yes, this is because the null checks. I didn't want to extract any of these into separate methods as that would have harmed readability.

amogh-jahagirdar · 2026-07-29T23:44:07Z

 class TrackingBuilder {
  private final long newSnapshotId;
  private final Long snapshotId;
-  private final Long dataSequenceNumber;


Does this need to move?

It's no longer final, I moved it to the non-final section.

amogh-jahagirdar · 2026-07-29T23:50:51Z

+      case 0 -> formatVersion;
+      case 1 -> fieldIds();
+      case 2 -> location;
+      case 3 -> fileFormat != null ? fileFormat.toString() : null;


I think I've noticed this on other PRs for TrackedFile for instance, when would fileFormat be null? It's required right?

I took this from TrackedFileStruct where we guard against fileFormat and contentType being null, however both of them are required. Maybe makes sense when we have a projected read and we don't project those fields?

amogh-jahagirdar · 2026-07-30T00:00:35Z

        toCopy.equalityIds != null
            ? Arrays.copyOf(toCopy.equalityIds, toCopy.equalityIds.length)
            : null;
+    this.columnFiles =


Minor: Could we express this without the Java stream? It's just a bit on the hot path when reading entries so all those allocations from the additional objects that streams create may add up.

Sure, done.

This change introduces the interface for column files and also integrates it to the schema for TrackedFile.

gaborkaszab · 2026-07-30T19:55:20Z

Rebased with main to resolve conflicts

gaborkaszab

Thanks for taking a look, @amogh-jahagirdar !

gaborkaszab · 2026-07-30T19:11:32Z

+  ByteBuffer keyMetadata();
+
+  /** Returns the list of recommended split locations for this column file, or null. */
+  List<Long> splitOffsets();


I brought this up once on the sync, we didn't have a deep dive but there were no objections either. Rational might be that when we project fields in a way that we don't have to read the base file, we can use the split offsets of one of the column files.
Now, with regular column updates, the column files usually contain a column or two and they have a single row group. However, later for column families we can have wider column files with more than one row groups where we can have split offsets.

gaborkaszab · 2026-07-30T19:14:58Z

+      case 0 -> formatVersion;
+      case 1 -> fieldIds();
+      case 2 -> location;
+      case 3 -> fileFormat != null ? fileFormat.toString() : null;


I took this from TrackedFileStruct where we guard against fileFormat and contentType being null, however both of them are required. Maybe makes sense when we have a projected read and we don't project those fields?

gaborkaszab · 2026-07-30T19:16:30Z

  }

  /** Copy constructor. */
+  @SuppressWarnings("CyclomaticComplexity")


Yes, this is because the null checks. I didn't want to extract any of these into separate methods as that would have harmed readability.

gaborkaszab · 2026-07-30T19:21:59Z

        toCopy.equalityIds != null
            ? Arrays.copyOf(toCopy.equalityIds, toCopy.equalityIds.length)
            : null;
+    this.columnFiles =


Sure, done.

gaborkaszab · 2026-07-30T19:23:04Z

 class TrackingBuilder {
  private final long newSnapshotId;
  private final Long snapshotId;
-  private final Long dataSequenceNumber;


It's no longer final, I moved it to the non-final section.

Defines the column_file element struct referenced by the column_files field (158) in the v4 content entry, matching the ColumnFile schema added in apache#16285. Co-authored-by: Gabor Kaszab <gaborkaszab@gmail.com> Co-authored-by: Anurag Mantripragada <amantripragada@apple.com>

Makes the column_files field type list<159: column_file> to match the inline element-id convention used by other list fields in the content entry, matching apache#16285. Co-authored-by: Gabor Kaszab <gaborkaszab@gmail.com> Co-authored-by: Anurag Mantripragada <amantripragada@apple.com>

github-actions Bot added the core label May 11, 2026

gaborkaszab commented May 11, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/ColumnFileInfo.java Outdated

anuragmantri added this to V4: Efficient Column Updates May 11, 2026

anuragmantri moved this to Backlog in V4: Efficient Column Updates May 11, 2026

anuragmantri moved this from Backlog to In progress in V4: Efficient Column Updates May 11, 2026

gaborkaszab force-pushed the main_column_file_interface branch from 630b00e to ca3259e Compare May 11, 2026 17:04

anuragmantri reviewed May 11, 2026

View reviewed changes

gaborkaszab force-pushed the main_column_file_interface branch from ca3259e to e6f7cf6 Compare May 12, 2026 09:35

gaborkaszab commented May 13, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/ColumnFileInfo.java Outdated

Comment thread core/src/main/java/org/apache/iceberg/ColumnFileInfo.java Outdated

gaborkaszab force-pushed the main_column_file_interface branch 3 times, most recently from 681633b to 813d5c0 Compare May 13, 2026 12:52

gaborkaszab force-pushed the main_column_file_interface branch 3 times, most recently from 596f6a4 to 6a1cbe9 Compare June 2, 2026 13:12

gaborkaszab changed the title ~~Core: Introduce interface for column files~~ Core: Basic fields and schemas for column files Jun 3, 2026

gaborkaszab force-pushed the main_column_file_interface branch from 6a1cbe9 to c683e72 Compare June 4, 2026 06:55

pvary moved this to In review in V4: metadata tree Jun 8, 2026

pvary added this to V4: metadata tree Jun 8, 2026

gaborkaszab force-pushed the main_column_file_interface branch from c683e72 to 6222fad Compare June 8, 2026 12:32

stevenzwu reviewed Jun 9, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/TrackedFileStruct.java

Comment thread core/src/main/java/org/apache/iceberg/TrackingStruct.java

gaborkaszab force-pushed the main_column_file_interface branch from 6222fad to 5c04f55 Compare June 11, 2026 09:27

gaborkaszab commented Jun 11, 2026

View reviewed changes

Comment thread core/src/main/java/org/apache/iceberg/TrackedFileStruct.java Outdated

Comment thread core/src/main/java/org/apache/iceberg/TrackingBuilder.java

gaborkaszab force-pushed the main_column_file_interface branch from 5c04f55 to b6ae446 Compare June 12, 2026 14:37

gaborkaszab commented Jun 12, 2026

View reviewed changes

gaborkaszab force-pushed the main_column_file_interface branch from b6ae446 to 0a252e8 Compare June 12, 2026 15:09

gaborkaszab force-pushed the main_column_file_interface branch from b4d6357 to c98ea62 Compare July 10, 2026 12:38

gaborkaszab force-pushed the main_column_file_interface branch 2 times, most recently from b207b61 to a51c0e9 Compare July 14, 2026 12:46

RussellSpitzer reviewed Jul 14, 2026

View reviewed changes

gaborkaszab force-pushed the main_column_file_interface branch from a51c0e9 to 6980283 Compare July 15, 2026 13:47

gaborkaszab commented Jul 15, 2026

View reviewed changes

gaborkaszab force-pushed the main_column_file_interface branch from 6980283 to 87d43a9 Compare July 15, 2026 14:11

RussellSpitzer added the Iceberg V4 Iceberg Table Format Version 4 label Jul 16, 2026

gaborkaszab force-pushed the main_column_file_interface branch from 87d43a9 to 56028ee Compare July 23, 2026 13:47

gaborkaszab force-pushed the main_column_file_interface branch from 56028ee to 30f42ac Compare July 23, 2026 14:40

amogh-jahagirdar reviewed Jul 30, 2026

View reviewed changes

gaborkaszab mentioned this pull request Jul 30, 2026

Core: Extend V4 DeletionVector with key_metadata field #17438

Open

gaborkaszab force-pushed the main_column_file_interface branch from 30f42ac to 870edbd Compare July 30, 2026 19:31

Core: Basic fields and schemas for column files

c303670

This change introduces the interface for column files and also integrates it to the schema for TrackedFile.

gaborkaszab force-pushed the main_column_file_interface branch from 870edbd to c303670 Compare July 30, 2026 19:54

gaborkaszab commented Jul 30, 2026

View reviewed changes

gaborkaszab requested review from RussellSpitzer and amogh-jahagirdar July 30, 2026 20:07

Uh oh!

Conversation

gaborkaszab commented May 11, 2026

Uh oh!

Uh oh!

gaborkaszab commented May 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gaborkaszab commented May 21, 2026

Uh oh!

gaborkaszab commented Jun 2, 2026

Uh oh!

gaborkaszab commented Jun 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaborkaszab commented Jul 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gaborkaszab left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amogh-jahagirdar left a comment •

edited

Loading