perf(read): reduce Arrow read-path overhead

[gripleaf]

Purpose

Linked issue: #240

Reduce avoidable CPU and memory-management overhead in the Avro and manifest read paths:

• Skip the redundant recursive Arrow array validation in release builds while retaining it in debug builds.
• Reuse a single ColumnarRow when scanning rows in the same manifest batch, updating only its row index.
• Reserve Avro Builder capacity based on the configured batch size.
• Reserve List and Map child Builder capacity using decoded Avro block sizes, including nested Struct builders.

These changes reduce repeated type fingerprint calculation, row-view construction and destruction, memory reallocations, and data copying.

Tests

• paimon-avro-format-test: 68 tests passed.
• ManifestFileTest.*: 8 tests passed.
• ColumnarRowTest.TestSimple: passed.
• Added coverage for:
  • Reusing ColumnarRow across different row indexes.
  • Recursive capacity reservation for Struct builders.
  • Correct capacity handling for nested List builders.
• pre-commit run --files <changed-files>: passed.
• git diff --check: passed.

API and Format

No public API changes under include/paimon/.

No storage format or protocol changes.

Documentation

No documentation changes. This is an internal performance optimization and does not introduce new user-facing functionality.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

[gripleaf]

Flame graph comparison

The following flame graphs were collected using the same workload and profiling configuration.

Before optimization:

After optimization:

[zjw1111]

Thanks for working on the read-path overhead here — the ColumnarRow reuse and the builder capacity reservation both look like the right direction, and I verified the row-reuse change is safe.

Two points I'd like to raise before this lands, left inline.

[zjw1111]

+1

[lxy-9602]

+1