[python] Support VARIANT type in pypaimon

[chenghuichen]

Purpose

Background: #7655

This PR adds VARIANT read/write support to pypaimon, with a particular focus on shredded VARIANT.

• Write: when variant.shreddingSchema is configured on a table, VARIANT columns are written in shredded Parquet format according to the schema.
• Read: shredded VARIANT columns are automatically reassembled back into standard struct<value: binary, metadata: binary> form, transparent to the caller.

Shredded column pruning and predicate pushdown will be built on top of this PR.

Tests

• Unit tests
  • pypaimon/tests/variant_test.py
• E2E tests
  • run_java_variant_write_py_read_test
  • run_py_variant_write_java_read_test

[chenghuichen]

The PR is ready for review now.

[JingsongLi]

Thanks @chenghuichen , let me check this.

[JingsongLi]

Duplicated binary constants and helpers across generic_variant.py and variant_shredding.py
Both files define _PRIMITIVE, _SHORT_STR, _OBJECT, _ARRAY, _U8_MAX, _U32_SIZE, _VERSION_MASK, _read_unsigned, _get_int_size, _object_header, _array_header, etc. This is a maintenance risk — if the spec changes, both files need updating. Consider extracting shared constants/helpers into a small _variant_binary.py module.

[JingsongLi]

The comment explains that data may be laid out in a different order than the id table. The sorting-by-offset logic is correct but intricate. Consider adding a small inline example or diagram showing the case where insertion-order data differs from sorted-id order, to help future maintainers.