[spark] Add point-delete fast path for primary key DELETE to skip the target table scan

[ZZZxDong]

Purpose

Closes #8836.

Currently DELETE FROM on a primary-key table always scans the target table to locate matching rows, even when the condition is nothing but primary keys. In that case the matched keys are already fully described by the condition, so we can build the -D records directly and skip the scan.

This PR adds a fast path to DeleteFromPaimonTableCommand for the pk-upsert delete:

Condition	Path
pk = literal / pk IN (literals) covering all pk columns	build -D rows from the condition on the driver (capped by delete.point-delete.max-rows, default 1M, falls back to scan beyond that)
pk IN (subquery) covering all pk columns	use the subquery result as the key DataFrame, fully distributed — suits deleting tens of millions of keys given in a key table
anything else	existing scan-based path, unchanged

Keys absent from the table are harmless: their -D records simply merge away in compaction, which matches the semantics of the existing path.

Tests

DeletePointFastPathTest: literal IN / equality / composite pk, subquery with single and composite pk (including positionally-renamed columns), fallback cases (non-pk column in condition, pk not fully pinned), delete-then-reinsert. Also ran the existing DeleteFromTableTest suite (29 tests) — all pass, no behavior change.

API and Format

New connector option delete.point-delete.max-rows (default 1000000). No format change.

Documentation

N/A

[JingsongLi]

• For fast paths, only fill in the primary key (PK); set all non-PK fields to NULL; however, the validation also allows partial updates and aggregates.
• Non-null, non-PK fields will cause an error when the merge function initializes the DELETE row.
• partial-update.remove-record-on-sequence-group also requires values for the sequence-group field; leaving all fields blank will not preserve the original delete semantics.
• Additionally, the literal extraction does not exclude NULL values, posing a risk of generating delete rows with empty primary keys.

[ZZZxDong]

Good catch, all four points are valid — fixed. The fast path is now gated to DEDUPLICATE tables without NOT NULL non-pk columns (anything else falls back to the scan path), and NULL literals in the condition also fall back so SQL three-valued logic is preserved. Added tests for each case, and rebased onto master to resolve the conflict.

[ZZZxDong]

CI caught another case of the same pattern: tables with primary-key indexes (btree/bitmap etc.) build index entries from real field values, so NULL-filled -D rows would leave the deletion invisible to index lookups. Added a third gate — tables with any pk index definition also fall back to the scan path. All tests green now.

[ZZZxDong]

The remaining CI failure surfaced a real semantic difference: -D rows from the fast path carry NULL for non-key columns, so incremental / audit_log reads would not see the old field values of deleted rows (the TVF test asserts [-D, 1, 11] but got [-D, 1, null]). Since this affects the changelog contract for any deduplicate table, I made the fast path opt-in: it is now off by default and enabled via spark.paimon.delete.point-delete.enabled, with the trade-off documented on the option. Default behavior is completely unchanged. Happy to adjust if you prefer a different approach.

[ZZZxDong]

@JingsongLi Thanks for the review! Summarizing the current state since the comments above are scattered:

For fast paths, only fill in the primary key (PK); set all non-PK fields to NULL; however, the validation also allows partial updates and aggregates. / Non-null, non-PK fields will cause an error... / partial-update.remove-record-on-sequence-group also requires values... / the literal extraction does not exclude NULL values...

All four points are addressed, plus two more cases that CI surfaced along the way:

• The fast path is now gated to DEDUPLICATE tables with no NOT NULL non-pk columns and no pk index definitions (index entries are built from real field values, so NULL-filled -D rows would leave deletions invisible to index lookups). NULL literals in the condition also fall back to preserve SQL three-valued logic.
• CI further revealed that incremental / audit_log reads expect the old field values on -D rows, which the fast path cannot provide by design. Since this affects the changelog contract for any deduplicate table, the fast path is now opt-in: off by default, enabled via spark.paimon.delete.point-delete.enabled, with the trade-off documented on the option. Default behavior is completely unchanged.

Please review it again when you have time, thanks！