Skip to content

feat(writer): add PositionDeleteFileWriter - #2986

Open
laskoviymishka wants to merge 4 commits into
apache:mainfrom
laskoviymishka:feat/position-delete-writer
Open

feat(writer): add PositionDeleteFileWriter#2986
laskoviymishka wants to merge 4 commits into
apache:mainfrom
laskoviymishka:feat/position-delete-writer

Conversation

@laskoviymishka

@laskoviymishka laskoviymishka commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #340.

What changes are included in this PR?

This adds PositionDeleteFileWriter under writer/base_writer, next to the data
and equality-delete writers. We already had a writer for equality deletes but
nothing that writes position delete files, so I modelled this one on
equality_delete_writer.rs.

It writes the two required columns, file_path (string, field id 2147483546) and
pos (long, 2147483545), and sets the content type to PositionDeletes on close.
position_delete_schema() / position_delete_arrow_schema() hand back the
canonical schema, built from the field definitions already in metadata_columns
so the reserved ids aren't duplicated here.

write() checks the batch is exactly those two required, non-null, correctly
typed columns before handing it to the parquet writer, so a bad batch fails with a
clear message instead of a confusing error further down. close() carries the
partition key onto the data file and leaves sort_order_id null.

One thing worth calling out: this writer doesn't sort its input. Position deletes
have to be sorted by file_path then pos, and for now that's on the caller (the
write() docs say so). I'd rather land the plain writer first and put sorting on
top of it, so a sorting writer, referenced_data_file, and a higher-level
DeltaWriter (#2218) are follow-ups instead of being crammed in here.

Are these changes tested?

Yes, 12 unit tests: the schema shape, a parquet round trip for single and multiple
writes (write it, read it back, compare), partition and spec-id propagation on
close, and the rejection cases: wrong column count, wrong or missing field ids on
either column, wrong types (including LargeUtf8, which is what a lot of Arrow
producers give you), nullable columns, and writing after close. cargo test -p iceberg, clippy, and fmt are all clean.

AI assistance disclosure

Drafted with help from Claude Code, then reviewed and tested by me before opening.

@laskoviymishka
laskoviymishka force-pushed the feat/position-delete-writer branch from c242c13 to 054dc33 Compare August 11, 2026 16:26
Add PositionDeleteFileWriter and its builder under writer/base_writer. It writes
a position delete file with the two required columns file_path (string, field id
2147483546) and pos (long, 2147483545) and sets DataContentType::PositionDeletes
on close.

position_delete_schema() and position_delete_arrow_schema() return the canonical
schema, built from the existing metadata_columns field definitions. write()
checks that a batch has exactly those two required, non-nullable, correctly typed
columns before writing, and rejects a closed writer. close() propagates the
partition key and leaves sort_order_id null.

The writer does not sort its input; the write() docs note that the caller must
supply rows sorted by (file_path, pos) until a later writer enforces it. Setting
referenced_data_file and a higher-level DeltaWriter are separate follow-ups.

Tested: schema shape, a parquet round trip for single and multiple writes,
partition propagation, and the validation cases (wrong column count, wrong or
missing field ids, wrong types including LargeUtf8, nullable columns, and writes
after close).

Refs apache#340.
@laskoviymishka
laskoviymishka force-pushed the feat/position-delete-writer branch from 054dc33 to 6f76750 Compare August 11, 2026 19:13
@laskoviymishka
laskoviymishka marked this pull request as ready for review August 12, 2026 11:11

@anoopj anoopj left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I took a pass. Overall, the implementation looks mostly good to me. Some feedback:

//! [`RESERVED_FIELD_ID_DELETE_FILE_POS`]). The writer takes batches already shaped as
//! those two columns (see [`position_delete_schema`]) and sets
//! [`DataContentType::PositionDeletes`] on the output. It does not sort its input; see
//! [`PositionDeleteFileWriter::write`].

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a note that position delete files are a V2 construct? Per the spec they are deprecated in v3 and writers must not add new position delete files to v3 tables because v3s add DVs instead.. This base writer correctly has no format-version gate, so gating belongs at the transaction/commit layer.

Consider adding something like "position delete files apply to v2 tables; v3 replaces them with deletion vectors, so callers must not route v3 writes here".

// The rolling writer fills in file statistics.
assert!(data_file.file_size_in_bytes > 0);

// The written Parquet file round-trips back to the exact input rows.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a test/assertion that reads the written file's parquet schema and asserts the two column ids?

}

/// Returns the canonical Arrow schema of a position delete file.
pub fn position_delete_arrow_schema() -> ArrowSchemaRef {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be pub crate instead? position_delete_schema needs to be public though

Error::new(
ErrorKind::DataInvalid,
format!(
"Position delete column `{}` has an invalid field id: {e}",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider adding test coverage for this?

- Note in the module docs that position delete files are a v2 construct; v3
  replaces them with deletion vectors, so callers must not route v3 writes here.
  Gating stays at the transaction/commit layer, not this base writer.
- Make position_delete_arrow_schema crate-internal. Only the tests use it today,
  so gate it (and its backing static) under cfg(test); a later writer can widen
  it when it has a real caller.
- Add a test that reads the written Parquet schema and asserts the two reserved
  column field ids survive the write.
- Add a test for the field-id parse-error branch (metadata present but not an
  integer).
@laskoviymishka

Copy link
Copy Markdown
Contributor Author

Thanks for the review! Pushed a follow-up commit addressing all four:

  • Added a note to the module docs that position delete files are a v2 construct — v3 replaces them with deletion vectors and forbids new position delete files, so callers must not route v3 writes here, and format-version gating stays at the transaction/commit layer rather than this base writer.
  • Made position_delete_arrow_schema crate-internal. Since only the tests consume it today, I gated it (and its backing static) under #[cfg(test)] to avoid a dead-code warning; I'll widen it to pub(crate) when a higher-level writer actually needs the Arrow form. position_delete_schema stays public.
  • Added a test that reads the written file's Parquet schema and asserts the two reserved column field ids (2147483546 / 2147483545) survive the write.
  • Added a test for the field_id parse-error branch (metadata present but not an integer).

Also merged latest main in. Let me know if you'd prefer position_delete_arrow_schema kept as pub(crate) regardless — happy to switch.

@anoopj anoopj left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. There is a CI failure though:

  error: `unparseable` should be `unparsable`
      ╭▸ ./crates/iceberg/src/writer/base_writer/position_delete_writer.rs:593:50
      │
  593 │     async fn test_position_delete_writer_rejects_unparseable_field_id() -> Result<()> {

It's a bit weird that the checker thinks unparseable is a typo. (both are correct forms)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement the position delete writer

2 participants