[SPARK-58311][SQL] Gate generated column values on write with a table capability

[szehon-ho]

What changes were proposed in this pull request?

This PR moves the gate that decides whether Spark auto-fills and enforces generated column values on a DSv2 write from a catalog-level capability to a table-level capability.

Specifically:

• Adds TableCapability.GENERATE_COLUMN_VALUES_ON_WRITE, checked via Table.capabilities().
• Removes TableCatalogCapability.SUPPORT_GENERATED_COLUMN_ON_WRITE (added by SPARK-57644 and still unreleased).
• GeneratedColumn.supportsGeneratedColumnsOnWrite now takes a Table and checks the table's capabilities instead of the TableCatalog's.
• Updates the four call sites to consult the table:
  • Analyzer (exposing generation-expression metadata to TableOutputResolver so missing values are auto-filled),
  • ResolveTableConstraints (adding the generated-column CheckInvariant constraints),
  • RewriteRowLevelCommand (blocking MERGE/UPDATE),
  • ResolveWriteToStream (blocking streaming writes).
• Test infra: the in-memory test table advertises the new capability, gated by a generate-column-values-on-write table property (default true), mirroring the existing accept-any-schema / auto-schema-evolution toggles.

Why are the changes needed?

Whether Spark should generate/enforce generated column values is a property of an individual table, not of the whole catalog. A single catalog can expose tables from formats or protocol versions with differing generated-column support, so a catalog-wide flag is too coarse.

This also aligns with how Delta Lake gates the feature: it is gated per-table on the table's protocol. GeneratedColumn.satisfyGeneratedColumnProtocol(protocol) returns protocol.isFeatureSupported(GeneratedColumnsTableFeature) -- i.e. it checks whether the GeneratedColumnsTableFeature is enabled in that specific table's protocol. A per-table TableCapability is the natural DSv2 analogue and matches the existing pattern for other per-table write behaviors (BATCH_WRITE, OVERWRITE_DYNAMIC, TRUNCATE, ...).

Does this PR introduce any user-facing change?

No change relative to a released version. The catalog capability being removed (SUPPORT_GENERATED_COLUMN_ON_WRITE) was introduced by SPARK-57644 and has not shipped in any release, so this only reshapes an unreleased API within the development branch. Connectors opt in by having their Table advertise TableCapability.GENERATE_COLUMN_VALUES_ON_WRITE. Auto-filling missing generated column values applies to by-name writes; ordinary by-position writes must still provide a value for every table column.

How was this patch tested?

GeneratedColumnWriteSuite (69 tests) passes. The existing "connector without the write capability does not auto-fill or enforce generated columns" coverage is retained (renamed to "table without write capability ..."), now driven by creating a table with TBLPROPERTIES ('generate-column-values-on-write' = 'false') instead of a dedicated no-capability catalog.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Cursor (Opus 4.8)

[cloud-fan]

0 blocking, 1 non-blocking, 0 nits.
The implementation is coherent, but the new public capability contract promises broader auto-fill behavior than the resolver currently provides.

Correctness (1)

• sql/catalyst/src/main/java/org/apache/spark/sql/connector/catalog/TableCapability.java:117: The capability contract overstates auto-fill behavior. Ordinary by-position writes reject a short input before resolving generated expressions, so this bullet should scope auto-computation to by-name writes (or the implementation should add equivalent by-position support). -- see inline

Verification

Traced both resolution modes in TableOutputResolver: by-name resolution substitutes a missing generated expression, while ordinary by-position resolution rejects a short input before column resolution.

PR description suggestions

• Clarify that missing generated values are currently auto-filled for by-name writes, while ordinary by-position writes still require matching arity.

[uros-b]

Thank you @szehon-ho and @cloud-fan!

[szehon-ho]

Merge Summary:

• merged into master 7865fb8
• merged into branch-4.x e10e949

Posted by merge_spark_pr.py

[szehon-ho]

Thanks @uros-b , @cloud-fan for review!