Fix: Python models emitting parquet format for managed Iceberg (#1591)#1593
Merged
sd-db merged 5 commits intoJul 20, 2026
Merged
Conversation
Python models configured with table_format='iceberg' and the
use_managed_iceberg behavior flag failed to materialize with:
INVALID_PARAMETER_VALUE.MANAGED_TABLE_FORMAT: Only Delta is supported
for managed tables. Provided datasource format is PARQUET.
adapter.resolve_file_format returns the "parquet" sentinel for managed
Iceberg. That sentinel is only meaningful to the SQL path, where
file_format_clause overrides it to `using iceberg`. The PySpark writer
had no such override and emitted `.format("parquet")` verbatim into the
generated saveAsTable call, which Unity Catalog rejects for managed
tables.
Add a py_resolve_writer_format() macro that mirrors the SQL override:
it returns 'iceberg' when table_format='iceberg' and use_managed_iceberg
is enabled, and otherwise defers to resolve_file_format (unchanged
behavior for delta and all other formats). Use it in both the
py_get_writer_options and create_python_intermediate_table (incremental
staging) writer paths.
Verified end-to-end against a DBR 18.2 cluster: non-incremental and
incremental (build + merge) managed Iceberg Python models now create
MANAGED tables with UniForm Iceberg enabled; the equivalent SQL path is
unchanged.
Resolves databricks#1591
Signed-off-by: Divya Siruvuri <divyasiruvuri@gmail.com>
sd-db
reviewed
Jul 17, 2026
sd-db
approved these changes
Jul 17, 2026
sd-db
left a comment
sd-db
left a comment
Collaborator
There was a problem hiding this comment.
Added a functional test for live verification, overall PR looks good. Have added some minor commits, should be good to merge once resolved.
sd-db
changed the title
Coverage report
This PR does not seem to contain any modification to coverable code. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves #1591
Description
Python models configured with
table_format='iceberg'and theuse_managed_icebergbehavior flag failed to materialize with:Root cause:
adapter.resolve_file_formatreturns the"parquet"sentinel for managed Iceberg. That sentinel is only meaningful to the SQL path, wherefile_format_clauseoverrides it tousing iceberg. The PySpark writer inpython.sqlhad no such override and emitted.format("parquet")verbatim into the generatedsaveAsTable(...)call, which Unity Catalog rejects for managed tables.The equivalent SQL model (
table_format='iceberg'+use_managed_iceberg) succeeds, and a Python model withouttable_format='iceberg'(default Delta) also succeeds — only the Python + managed-Iceberg combination failed.Fix: Add a
py_resolve_writer_format()macro that mirrors the SQL override — it returns'iceberg'whentable_format='iceberg'anduse_managed_icebergis enabled, and otherwise defers toresolve_file_format(unchanged behavior for delta and all other formats). It is used in both writer paths:py_get_writer_options(non-incremental /table) andcreate_python_intermediate_table(incremental staging).resolve_file_formatitself is intentionally left unchanged, so the 15 SQL-path consumers of the sentinel are unaffected.Verification
Verified end-to-end against a DBR 18.2 all-purpose cluster on Unity Catalog (dbt-core 1.11.12, dbt-databricks from this branch):
tablemodel, managed IcebergMANAGED_TABLE_FORMATincrementalmodel — initial buildincrementalmodel — merge (staging table).format("iceberg")for__dbt_tmp, merge succeedsConfirmed the resulting table is
table_type=MANAGEDwithdelta.enablemanagedicebergtable=trueanddelta.universalFormat.enabledFormats=iceberg— matching the working SQL path.Unit tests:
tests/unit/macros/adapters/test_python_macros.py(2 new cases: managed-iceberg →iceberg, iceberg-without-flag →delta); full macro suite (237) passes.Checklist
CHANGELOG.mdand added information about my change to the "dbt-databricks next" section./dbt-databricks-pr-ready(AI agent skill in.claude/skills/) and addressed its merge-readiness feedbackThis pull request and its description were written by Isaac.