refactor(rest): localize CreateTableRequest's partition-spec skip to REST#2702
Open
rahulsmahadev wants to merge 1 commit into
Open
refactor(rest): localize CreateTableRequest's partition-spec skip to REST#2702rahulsmahadev wants to merge 1 commit into
rahulsmahadev wants to merge 1 commit into
Conversation
…REST Follow-up to apache#2610. That PR added `#[serde(skip_serializing_if = "Option::is_none")]` to `UnboundPartitionField::field_id` and `UnboundPartitionSpec::spec_id` in `iceberg::spec::partition` so the REST `CreateTableRequest` payload would omit unset partition ids. The same core types are also serialized as part of `TableUpdate::AddSpec` on the commit path, where a missing `spec-id` would hard-fail a Java REST server (`PartitionSpecParser.fromJson` throws `IllegalArgumentException: Cannot parse missing int: spec-id`). The breakage is latent today only because `TableMetadataBuilder::add_partition_spec` always populates `spec_id` before pushing the `AddSpec` change. This change moves the skip behavior out of the core types and into the REST layer: - Drop `#[serde(skip_serializing_if = "Option::is_none")]` from `UnboundPartitionField::field_id` and `UnboundPartitionSpec::spec_id`. The core types again emit `spec-id` / `field-id` as `null` when unset, matching the pre-apache#2610 wire format and preserving the `TableUpdate::AddSpec` contract. - Add a `serialize_with` helper on `CreateTableRequest::partition_spec` that serializes through private shadow types whose fields skip the ids when `None`. This preserves the externally visible REST payload shape (`partition-spec` is omitted entirely when unset; inner `spec-id` / `field-id` are omitted when unset) without coupling the core spec type to it. Tests: - `test_add_spec_emits_spec_id_key`: regression guard that `TableUpdate::AddSpec` always emits the `spec-id` key, including when `spec_id` is `None`. - `test_unbound_partition_field_field_id_set_is_serialized`: positive test that a set `field_id` round-trips under the kebab-case `field-id` key (viirya's follow-up ask on apache#2610). - `test_create_table_request_partition_spec_skips_inner_none_ids`: verifies that the REST shadow serializer omits inner `spec-id` and `field-id` when unset, and emits them when set. - `test_unbound_partition_spec_serialization_emits_none_as_null` replaces the previous skip-expectation test and documents the new contract. Signed-off-by: rahulsmahadev <rahul.mahadev@databricks.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Follow-up to #2610. Addresses two review threads on that PR:
#[serde(skip_serializing_if = "Option::is_none")]off of the coreUnboundPartitionField/UnboundPartitionSpectypes and apply it only when these types are serialized as part ofCreateTableRequest.UnboundPartitionField::field_id(set case).What changes are included in this PR?
#2610 added
#[serde(skip_serializing_if = "Option::is_none")]toUnboundPartitionField::field_idandUnboundPartitionSpec::spec_idso that RESTCreateTableRequestpayloads would omit unset partition ids. The same core types are also serialized as part ofTableUpdate::AddSpecon the commit path, where a missingspec-idwould hard-fail a Java REST server (PartitionSpecParser.fromJsonthrowsIllegalArgumentException: Cannot parse missing int: spec-id). The breakage is latent today only becauseTableMetadataBuilder::add_partition_specalways populatesspec_idviawith_spec_idbefore pushing theAddSpecchange.This PR moves the skip behavior out of the core spec types and into the REST layer:
#[serde(skip_serializing_if = "Option::is_none")]fromUnboundPartitionField::field_idandUnboundPartitionSpec::spec_id. The core types again emitspec-id/field-idasnullwhen unset, matching the pre-fix(rest): skip serializing unset optional fields in CreateTableRequest #2610 wire format and preserving theTableUpdate::AddSpeccontract.serialize_withhelper onCreateTableRequest::partition_specthat serializes through private shadow types whose fields skip the ids whenNone. This preserves the externally visible REST payload shape (partition-specomitted entirely when unset; innerspec-id/field-idomitted when unset) without coupling the core spec type to it.The shadow-type approach was chosen over a public newtype wrapper to keep the change scope to the REST crate and avoid touching
iceberg::spec::partition's public surface beyond the revert.Are these changes tested?
Yes. Four new / updated tests:
crates/iceberg/src/catalog/mod.rs::test_add_spec_emits_spec_id_key: regression guard thatTableUpdate::AddSpecalways emits thespec-idkey, including whenspec_idisNone. This is the test that would have caught the latent bug if fix(rest): skip serializing unset optional fields in CreateTableRequest #2610 had been merged without theadd_partition_specalways-populates-spec_id invariant.crates/iceberg/src/spec/partition.rs::test_unbound_partition_field_field_id_set_is_serialized: positive test that a setfield_idround-trips under the kebab-casefield-idkey (viirya's follow-up ask).crates/catalog/rest/src/types.rs::test_create_table_request_partition_spec_skips_inner_none_ids: verifies that the REST shadow serializer omits innerspec-idandfield-idwhen unset, and emits them when set.crates/iceberg/src/spec/partition.rs::test_unbound_partition_spec_serialization_emits_none_as_nullreplaces the previous skip-expectation test and documents the new contract on the core type.All pre-existing tests in
iceberg(1361) andiceberg-catalog-rest(45) still pass.cargo fmt --checkandcargo clippy --all-targets --all-features -- -D warningsare clean on both crates.Test plan
cargo test -p iceberg --libcargo test -p iceberg-catalog-rest --libcargo fmt --check -p iceberg -p iceberg-catalog-restcargo clippy -p iceberg -p iceberg-catalog-rest --all-targets --all-features -- -D warnings