feat: get_ranges 2x performance degradation

[comphead]

Which issue does this PR close?

Closes #7383

In Apache DataFusion Comet we figured out that HDFS access slow down dramatically, leading HDFS tasks performance to degrade 2x likely after #7192

Original issue apache/datafusion-comet#3926

Rationale for this change

Summary

• Concurrent range reads in get_ranges: Replace the sequential per-range read loop with concurrent reads using futures::stream::buffered(8), improving wall-clock latency for multi-range reads (e.g., Parquet rowgroups from HDFS)
• No stat() call: The existing get_ranges already avoids stat(); this preserves that by using a single reader created once per call
• Exact-sized fetches: Each range fetches only the requested bytes — no coalescing/over-fetching that would retain large backing buffers in memory

[comphead]

Putting this to draft, as adding more parallelism increases significantly the memory usage

[comphead]

The last revision seems returned the performance to what it was before, perhaps slightly faster because of not calling stats() and memory usage is moderate

@Xuanwo @kszucs PTAL

[comphead]

10 paralellism also used in coalesce_ranges which have been called in original object store get_ranges

[morristai]

I think the current change improves get_ranges latency by issuing per-range reads concurrently, but it still performs one backend read per requested range.

reader.read(range) still goes through OpenDAL's normal read path, so for HDFS this still means a fresh file open / read pipeline per range. In other words, this changes N sequential reads into up to N concurrent reads, but it doesn't coalesce nearby ranges or reduce the total number of HDFS opens / NameNode RPCs.

That seems different from the PR description, which suggests this should turn something like 50 range reads into ~5 merged reads. The earlier regression looks more like we lost object_store's default get_ranges coalescing when #7192 added the custom get_range / get_ranges path to avoid stat().

So this patch may still help wall-clock time, but I don't think it fixes the root cause if the goal is to actually reduce HDFS opens. That likely needs a merged-range path again, e.g. something based on OpenDAL's range-fetch/merge logic instead of buffering individual reader.read(range) calls.

[Xuanwo]

Maybe we just call fetch + concurrent here?

[comphead]

Thanks @morristai and @Xuanwo let me try to get back the coalesce_ranges as it was in original object_store implementation

[comphead]

Thanks @morristai and @Xuanwo let me try to get back the coalesce_ranges as it was in original object_store implementation

coalesce_ranges explodes memory usage at least twice

[comphead]

Rolled back to without using coalesce_ranges

object_store::coalesce_ranges merges nearby ranges (gap < 1MB) into larger fetches. For HDFS files with multiple rowgroups, this can merge separate rowgroups into a single ~full-file read. The returned Bytes::slice() views retain the full coalesced backing buffer via reference counting, causing ~2x
memory usage.

On our cluster with 1200 files × 140MB across 30 machines, this increased total memory from ~0.5T to ~1T. The simple per-range concurrent approach fetches exactly what's needed with no over-allocation.

This PR can act like a patch to fix the performance having relatively same memory utilization, and happy to participate in longer term solution in follow up PR if needed