perf: Replace Arrow IPC with more efficient shuffle format [experimental]

[andygrove]

Which issue does this PR close?

Performance improvements for native shuffle.

Rationale for this change

This PR contains two related performance improvements for native shuffle:

1. Shuffle Direct Read Simplification

Simplifies the shuffle direct read code path, removing unnecessary complexity (separate PR #3731 exists for this change).

2. Replace Arrow IPC with Raw Buffer Format

The shuffle write/read path previously used Arrow IPC to serialize RecordBatches. Arrow IPC embeds a full schema flatbuffer in every block (~200-800 bytes), plus message headers and alignment padding. Since the reader already knows the schema from protobuf, this overhead is unnecessary.

Benchmarking against Gluten (which uses raw columnar buffers for shuffle) showed this as a key performance differentiator. The new raw buffer format writes Arrow ArrayData buffers directly with minimal headers (~16 bytes per column vs ~200+ bytes for IPC).

What changes are included in this PR?

Shuffle direct read simplification:

• Simplified shuffle direct read code path
• Removed unused imports and doc files

New shuffle serialization format that replaces Arrow IPC with raw Arrow buffer writes:

• write_array_data / read_array_data: Recursively serialize/deserialize Arrow ArrayData (validity bitmap + value buffers + child data for nested types)
• write_raw_batch / read_shuffle_block: Top-level batch serialization with schema parameter
• Dictionary arrays are cast to their value type before writing (Arrow IPC handled this transparently; the raw format requires explicit handling)
• The outer block header (16-byte compressed_length + field_count) is unchanged for JVM compatibility
• JNI decodeShuffleBlock now accepts protobuf schema bytes so the native side can reconstruct the Arrow Schema for deserialization
• CometBlockStoreShuffleReader serializes the output schema as a protobuf ShuffleScan message and passes it to the native decoder

Format spec:

Outer header (unchanged):
  [8 bytes: compressed_length] [8 bytes: field_count]
  [4 bytes: codec tag] [compressed body...]

Inner body (new):
  [4 bytes: num_rows]
  For each column: array_data(...)

array_data (recursive):
  [4 bytes: null_count] [4 bytes: validity_len] [validity bytes]
  [4 bytes: num_buffers] (for each: [4 bytes: len] [bytes])
  [4 bytes: num_children] (for each: [4 bytes: child_rows] array_data(...))

How are these changes tested?

• 10 new Rust roundtrip tests covering all data types: primitives, strings, binary, boolean, null, decimal, date, timestamp, nested types (List, Struct, Map), dictionary cast to plain arrays, and empty batches. All 4 compression codecs tested (None, LZ4, Zstd, Snappy).
• All existing shuffle writer end-to-end tests updated and passing (156 Rust tests total)
• Full CometNativeShuffleSuite JVM integration tests pass (20 tests), exercising both the traditional JNI/FFI read path and the direct read path