Commit 697d536
committed
feat(benchmark): add format-level Parquet read/write micro-benchmarks
Add paimon-parquet-format-benchmark, driving ParquetWriterBuilder and
ParquetFileBatchReader directly so a change confined to the format layer can
be attributed without the catalog lookup, split planning, merge/sort and
commit that the table-level read_write_benchmark includes.
Writer cases cover flat primitives, VARCHAR by cardinality with and without
file-level dictionary encoding, dictionary-encoded VARCHAR/INTEGER input
arrays against flat controls holding value and width fixed, DECIMAL at
precision 9/18/38, nested STRUCT/LIST/VECTOR/MAP, null density, rows per
AddBatch call, column count, row group size, the byte-triggered writer memory
threshold, and every codec Parquet accepts.
Reader cases cover full scan, projection, predicate-filtered reads with
page-index filtering on and off, skip-heavy reads driven by a strided
selection bitmap whose strides straddle the coalesce hole limit, null
density, DECIMAL, DOUBLE, dictionary against plain files, rows per NextBatch
call, and nested column reads.
Each case reports ns_per_row and bytes_per_row next to ns_per_input_row and
bytes_per_input_row, so settings that prune by different amounts stay
comparable, plus the reader's own row-group and batch counters. Read cases
assert on the rows they materialized and the process exits non-zero on any
SkipWithError, so a case that stops measuring what it claims fails instead of
looking fast.
Every axis is swept on its own rather than as a combined matrix, so each case
answers one question and a change can be attributed to it.
Also add paimon-parquet-format-benchmark-test, which guards the format-layer
assumptions the benchmark rests on. The benchmark only builds under
PAIMON_BUILD_BENCHMARKS, which CI does not set, so the test is what keeps
codec acceptance, dictionary round-trips, DECIMAL precision, nested types,
predicate and bitmap row counts, encoding agreement and the memory-threshold
flush covered.
Closes #2281 parent dbdce8f commit 697d536
4 files changed
Lines changed: 2120 additions & 4 deletions
File tree
- benchmark
- docs/source/examples
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
66 | 80 | | |
67 | 81 | | |
68 | 82 | | |
| |||
74 | 88 | | |
75 | 89 | | |
76 | 90 | | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
77 | 108 | | |
0 commit comments