feat(writer): add PositionDeleteFileWriter - #2986
Conversation
c242c13 to
054dc33
Compare
Add PositionDeleteFileWriter and its builder under writer/base_writer. It writes a position delete file with the two required columns file_path (string, field id 2147483546) and pos (long, 2147483545) and sets DataContentType::PositionDeletes on close. position_delete_schema() and position_delete_arrow_schema() return the canonical schema, built from the existing metadata_columns field definitions. write() checks that a batch has exactly those two required, non-nullable, correctly typed columns before writing, and rejects a closed writer. close() propagates the partition key and leaves sort_order_id null. The writer does not sort its input; the write() docs note that the caller must supply rows sorted by (file_path, pos) until a later writer enforces it. Setting referenced_data_file and a higher-level DeltaWriter are separate follow-ups. Tested: schema shape, a parquet round trip for single and multiple writes, partition propagation, and the validation cases (wrong column count, wrong or missing field ids, wrong types including LargeUtf8, nullable columns, and writes after close). Refs apache#340.
054dc33 to
6f76750
Compare
anoopj
left a comment
There was a problem hiding this comment.
I took a pass. Overall, the implementation looks mostly good to me. Some feedback:
| //! [`RESERVED_FIELD_ID_DELETE_FILE_POS`]). The writer takes batches already shaped as | ||
| //! those two columns (see [`position_delete_schema`]) and sets | ||
| //! [`DataContentType::PositionDeletes`] on the output. It does not sort its input; see | ||
| //! [`PositionDeleteFileWriter::write`]. |
There was a problem hiding this comment.
Could we add a note that position delete files are a V2 construct? Per the spec they are deprecated in v3 and writers must not add new position delete files to v3 tables because v3s add DVs instead.. This base writer correctly has no format-version gate, so gating belongs at the transaction/commit layer.
Consider adding something like "position delete files apply to v2 tables; v3 replaces them with deletion vectors, so callers must not route v3 writes here".
| // The rolling writer fills in file statistics. | ||
| assert!(data_file.file_size_in_bytes > 0); | ||
|
|
||
| // The written Parquet file round-trips back to the exact input rows. |
There was a problem hiding this comment.
Can we add a test/assertion that reads the written file's parquet schema and asserts the two column ids?
| } | ||
|
|
||
| /// Returns the canonical Arrow schema of a position delete file. | ||
| pub fn position_delete_arrow_schema() -> ArrowSchemaRef { |
There was a problem hiding this comment.
Can this be pub crate instead? position_delete_schema needs to be public though
| Error::new( | ||
| ErrorKind::DataInvalid, | ||
| format!( | ||
| "Position delete column `{}` has an invalid field id: {e}", |
There was a problem hiding this comment.
Consider adding test coverage for this?
- Note in the module docs that position delete files are a v2 construct; v3 replaces them with deletion vectors, so callers must not route v3 writes here. Gating stays at the transaction/commit layer, not this base writer. - Make position_delete_arrow_schema crate-internal. Only the tests use it today, so gate it (and its backing static) under cfg(test); a later writer can widen it when it has a real caller. - Add a test that reads the written Parquet schema and asserts the two reserved column field ids survive the write. - Add a test for the field-id parse-error branch (metadata present but not an integer).
|
Thanks for the review! Pushed a follow-up commit addressing all four:
Also merged latest |
anoopj
left a comment
There was a problem hiding this comment.
LGTM. There is a CI failure though:
error: `unparseable` should be `unparsable`
╭▸ ./crates/iceberg/src/writer/base_writer/position_delete_writer.rs:593:50
│
593 │ async fn test_position_delete_writer_rejects_unparseable_field_id() -> Result<()> {
It's a bit weird that the checker thinks unparseable is a typo. (both are correct forms)
Which issue does this PR close?
Closes #340.
What changes are included in this PR?
This adds
PositionDeleteFileWriterunderwriter/base_writer, next to the dataand equality-delete writers. We already had a writer for equality deletes but
nothing that writes position delete files, so I modelled this one on
equality_delete_writer.rs.It writes the two required columns,
file_path(string, field id 2147483546) andpos(long, 2147483545), and sets the content type toPositionDeleteson close.position_delete_schema()/position_delete_arrow_schema()hand back thecanonical schema, built from the field definitions already in
metadata_columnsso the reserved ids aren't duplicated here.
write()checks the batch is exactly those two required, non-null, correctlytyped columns before handing it to the parquet writer, so a bad batch fails with a
clear message instead of a confusing error further down.
close()carries thepartition key onto the data file and leaves
sort_order_idnull.One thing worth calling out: this writer doesn't sort its input. Position deletes
have to be sorted by
file_paththenpos, and for now that's on the caller (thewrite()docs say so). I'd rather land the plain writer first and put sorting ontop of it, so a sorting writer,
referenced_data_file, and a higher-levelDeltaWriter(#2218) are follow-ups instead of being crammed in here.Are these changes tested?
Yes, 12 unit tests: the schema shape, a parquet round trip for single and multiple
writes (write it, read it back, compare), partition and spec-id propagation on
close, and the rejection cases: wrong column count, wrong or missing field ids on
either column, wrong types (including LargeUtf8, which is what a lot of Arrow
producers give you), nullable columns, and writing after close.
cargo test -p iceberg,clippy, andfmtare all clean.AI assistance disclosure
Drafted with help from Claude Code, then reviewed and tested by me before opening.