API, Core: Introduce foundational types for V4 manifest support #15049

anoopj · 2026-01-14T16:57:20Z

These types will be used for the v4 adaptive metadata tree and will be used by subsequent PRs for manifest reading/writing.

For now, we are adding these as package-private interfaces in core, and eventually we will move them into api.

Prototype PR: #14533

Changes

API:

Add DATA_MANIFEST and DELETE_MANIFEST to FileContent enum for V4 manifest entry types

Core:

TrackedFile - Unified V4 manifest entry representation (equivalent of ContentFile for V4)
TrackingInfo - Entry status, snapshot ID, sequence numbers, and first-row-id
ContentInfo - Metadata for externally stored content (deletion vectors)
ManifestStats - Manifest-level statistics (file/row counts, min sequence number)

anoopj · 2026-01-14T17:00:01Z

api/src/main/java/org/apache/iceberg/FileContent.java

  DATA(0),
  POSITION_DELETES(1),
-  EQUALITY_DELETES(2);
+  EQUALITY_DELETES(2),


I had to make this API change so that TrackingInfo can return the contentType. This is backward compatible since this is a new enum value. If we don't want to make any API changes, I will need to change TrackingInfo.contentType() to return an int instead.

amogh-jahagirdar

Did a cursory review, still need to review in more depth when I get a chance but thank you for publishing this @anoopj !

amogh-jahagirdar · 2026-01-14T22:31:30Z

core/src/main/java/org/apache/iceberg/TrackedFile.java

+   *
+   * <p>Used for applying manifest deletion vectors.
+   */
+  Long pos();


Should this be a primitive long? as There would always be some non-null ordinal position

Minor Preference: Would prefer to use the full name position()

I was trying to be consistent with the existing ContentFile.pos() API that returns a Long.

I'm not sure that we will always project this.

Also, this is in TrackingInfo so I don't think we need it here.

core/src/main/java/org/apache/iceberg/TrackedFile.java

amogh-jahagirdar · 2026-01-14T22:43:57Z

core/src/main/java/org/apache/iceberg/TrackedFile.java

+   * Returns the tracking information for this entry.
+   *
+   * <p>Contains status, snapshot ID, sequence numbers, and first-row-id. Optional - may be null if
+   * tracking info is inherited.
+   */
+  TrackingInfo trackingInfo();


Sorry if I missed the discussion on the original PR, so looks like TrackingInfo has a manifestLocation and ordinal as well. I think the intent there is so that TrackingInfo has enough information self contained on it's structure so it can stand alone?

Ah just saw the top level comment in TrackingInfo, cool!

amogh-jahagirdar · 2026-01-14T22:49:52Z

core/src/main/java/org/apache/iceberg/TrackedFile.java

+   * <p>Must be defined if location is defined.
+   */
+  Long fileSizeInBytes();


Maybe a miss on the proposal but this would be required always no? And as a result primitive long?

That makes sense. I changed the proposal (left it as a suggestion). I think we had it optional when we had inline manifest DVs as a separate record, which is not the case anymore.

I agree with long.

nastra · 2026-01-23T10:27:00Z

core/src/main/java/org/apache/iceberg/TrackedFile.java

+   * @return a DataFile representation
+   * @throws IllegalStateException if content_type is not DATA
+   */
+  DataFile asDataFile(PartitionSpec spec);


according to #14533 (comment) this should probably not have any arguments

rdblue · 2026-02-06T00:22:49Z

core/src/main/java/org/apache/iceberg/TrackedFile.java

+          "referenced_file",
+          Types.StringType.get(),
+          "Location of data file that a DV references if content_type is POSITION_DELETES. "
+              + "Location of affiliated data manifest if content_type is DELETE_MANIFEST, or null if unaffiliated");


These descriptions should be shorter. This explains what the field is, but it doesn't need to document all idiosyncrasies.

rdblue · 2026-02-06T00:25:54Z

core/src/main/java/org/apache/iceberg/TrackingInfo.java

+ */
+interface TrackingInfo {
+  /** Status of an entry in a tracked file */
+  enum Status {


We had to copy this because it was in ManifestEntry. It's probably better to make this top-level to avoid that problem in the future.

rdblue · 2026-02-06T00:31:57Z

core/src/main/java/org/apache/iceberg/TrackedFile.java

+   *
+   * <p>Must be defined if content_type is POSITION_DELETES, must be null otherwise.
+   */
+  ContentInfo contentInfo();


Not a problem: I'm debating whether to have a separate struct for this. Primarily motivated by how awkward the name "content info" is. It's nice to have an optional struct with required fields, but that is a very minor benefit.

I am neutral on this. Can change this if you feel strongly.

rdblue · 2026-02-06T00:33:26Z

core/src/main/java/org/apache/iceberg/TrackedFile.java

+   *
+   * <p>TODO: Define ContentStats structure per V4 proposal.
+   */
+  Object contentStats();


I think this is a really important part of the implementation, so we should focus on getting this ready.

Ack. @nastra is working on this.

ContentStats is already in core: https://github.com/apache/iceberg/blob/main/api/src/main/java/org/apache/iceberg/stats/ContentStats.java#L25
But in order to use it here we would have to move it into the api module

Thank you @nastra. Incorporated. We don't need to move it to api for now because this PR is currently against core.

ah got it, great. Somehow I thought we're in the api module. All good then

rdblue · 2026-02-06T00:41:56Z

Looking good overall, but I think we need to get ContentStats defined and then make sure the translation to DataFile and DeleteFile is reasonable.

These types follow the https://s.apache.org/iceberg-single-file-commit and will be used by subsequent PRs for manifest reading/writing. For now, we are adding these as package-private interfaces in core, and eventually we will move them into api. Changes API: - Add DATA_MANIFEST and DELETE_MANIFEST to FileContent enum for V4 manifest entry types Core: - TrackedFile - Unified V4 manifest entry representation (equivalent of ContentFile for V4) - TrackingInfo - Entry status, snapshot ID, sequence numbers, and first-row-id - ContentInfo - Metadata for externally stored content (deletion vectors) - ManifestStats - Manifest-level statistics (file/row counts, min sequence number)

github-actions bot added API core labels Jan 14, 2026

anoopj commented Jan 14, 2026

View reviewed changes

amogh-jahagirdar reviewed Jan 14, 2026

View reviewed changes

nastra changed the title ~~Introduce foundational types for V4 manifest support~~ API, Core: Introduce foundational types for V4 manifest support Jan 15, 2026

nastra reviewed Jan 23, 2026

View reviewed changes

rdblue reviewed Feb 6, 2026

View reviewed changes

anoopj added 4 commits February 9, 2026 07:27

Change asDataFile/asDeleteFile to no-argument API

cf2958c

Incorporate feedback from Ryan

3ab0b30

Update TrackedFile.contentStats() to return ContentStats

67c20b6

anoopj force-pushed the v4-foundational-types branch from 47542d3 to 67c20b6 Compare February 9, 2026 17:11

anoopj requested review from amogh-jahagirdar, nastra and rdblue February 9, 2026 17:18

API, Core: Introduce foundational types for V4 manifest support #15049

Are you sure you want to change the base?

API, Core: Introduce foundational types for V4 manifest support #15049

Conversation

anoopj commented Jan 14, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amogh-jahagirdar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rdblue commented Feb 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants