Skip to content

Fix incremental table compilation failing on metadata.extraProperties - #2236

Open
fuzmish wants to merge 1 commit into
dataform-co:mainfrom
fuzmish:fix/incremental-table-metadata-extra-properties
Open

Fix incremental table compilation failing on metadata.extraProperties#2236
fuzmish wants to merge 1 commit into
dataform-co:mainfrom
fuzmish:fix/incremental-table-metadata-extra-properties

Conversation

@fuzmish

@fuzmish fuzmish commented Jul 31, 2026

Copy link
Copy Markdown

Example

This fails to compile:

config {
  type: "incremental",
  onSchemaChange: "IGNORE",
  metadata: {
    extraProperties: {
      myField: "example"
    }
  }
}

SELECT 1
// -> Unexpected property "myField", or property value type of "string" is incorrect.

Cause

verifyObjectMatchesProto(protoType, object) in common/protos/index.ts:

const proto = protoType.create(object);          // shallow copy, no type conversion
const protoCastObject = protoType.toObject(proto);
checkFields(object, protoCastObject);             // throws on typeof mismatch at any key

create() copies extraProperties as-is, without converting it to the Struct format.
toObject() then can't read it as a Struct, so protoCastObject ends up with an empty extraProperties.
checkFields finds myField present in the raw object but missing from the round-tripped one, and throws.

Fix

Convert metadata.extraProperties into the Struct wire format in IncrementalTable.verifyConfig(), before calling verifyObjectMatchesProto, so both sides of the comparison already have matching shapes.
The conversion logic already existed; it is extracted into a named, exported function and incremental_table.ts call it directly.

Test

  • Added a test in core/actions/incremental_table_test.ts covering the example above.
  • bazel test $(bazel query 'kind(test, //core/...)') passed.

Related: #2119

@fuzmish
fuzmish requested a review from a team as a code owner July 31, 2026 21:34
@fuzmish
fuzmish requested review from krushangSk17 and removed request for a team July 31, 2026 21:34
@fuzmish fuzmish changed the title Fix metadata.extraProperties being dropped for incremental tables Fix incremental table compilation failing on metadata.extraProperties Jul 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant