POC: batch-granular parquet scan scheduling (and two row-group-granular alternatives) - #24086

Draft

adriangb wants to merge 9 commits into

apache:mainfrom

pydantic:claude/datafusion-rowgroup-buffering-ec9abc

Commits on Aug 4, 2026

POC: pipelined (overlapped) row-group prefetch with a byte budget in the parquet push decoder

adriangb
and
claude
committed
d38a4ee
POC: streaming (batch-granular) parquet scan via a persistent sync reader

adriangb
and
claude
committed
5ff7af6
streaming: fetch small files in a single wave

adriangb
and
claude
committed
bcbbda1
streaming: fill blocking waves and coalesce scattered ranges

adriangb
and
claude
committed
11f6867
streaming: derive the fetch plan from arrow-rs instead of reimplementing it

adriangb
and
claude
committed
34f28ea
streaming: load the page index when the streaming policy is active

adriangb
and
claude
committed
075e816
streaming: log fallbacks, install covered pages, lower the coalesce default

adriangb
and
claude
committed
ee1db7b
streaming: drop scheduler-side range coalescing

adriangb
and
claude
committed
7f3e448
docs: clarify that the streaming path does not drive the push decoder

adriangb
and
claude
committed
2400652