Skip to content

Commit 697d536

Browse files
committed
feat(benchmark): add format-level Parquet read/write micro-benchmarks
Add paimon-parquet-format-benchmark, driving ParquetWriterBuilder and ParquetFileBatchReader directly so a change confined to the format layer can be attributed without the catalog lookup, split planning, merge/sort and commit that the table-level read_write_benchmark includes. Writer cases cover flat primitives, VARCHAR by cardinality with and without file-level dictionary encoding, dictionary-encoded VARCHAR/INTEGER input arrays against flat controls holding value and width fixed, DECIMAL at precision 9/18/38, nested STRUCT/LIST/VECTOR/MAP, null density, rows per AddBatch call, column count, row group size, the byte-triggered writer memory threshold, and every codec Parquet accepts. Reader cases cover full scan, projection, predicate-filtered reads with page-index filtering on and off, skip-heavy reads driven by a strided selection bitmap whose strides straddle the coalesce hole limit, null density, DECIMAL, DOUBLE, dictionary against plain files, rows per NextBatch call, and nested column reads. Each case reports ns_per_row and bytes_per_row next to ns_per_input_row and bytes_per_input_row, so settings that prune by different amounts stay comparable, plus the reader's own row-group and batch counters. Read cases assert on the rows they materialized and the process exits non-zero on any SkipWithError, so a case that stops measuring what it claims fails instead of looking fast. Every axis is swept on its own rather than as a combined matrix, so each case answers one question and a change can be attributed to it. Also add paimon-parquet-format-benchmark-test, which guards the format-layer assumptions the benchmark rests on. The benchmark only builds under PAIMON_BUILD_BENCHMARKS, which CI does not set, so the test is what keeps codec acceptance, dictionary round-trips, DECIMAL precision, nested types, predicate and bitmap row counts, encoding agreement and the memory-threshold flush covered. Closes #228
1 parent dbdce8f commit 697d536

4 files changed

Lines changed: 2120 additions & 4 deletions

File tree

benchmark/CMakeLists.txt

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,20 @@ if(PAIMON_BUILD_BENCHMARKS)
6363
${PAIMON_BENCHMARK_LINK_TOOLCHAIN}
6464
EXTRA_INCLUDES
6565
${CMAKE_SOURCE_DIR})
66+
67+
add_paimon_benchmark(parquet_format_benchmark
68+
SOURCES
69+
parquet_format_benchmark.cpp
70+
STATIC_LINK_LIBS
71+
arrow
72+
parquet
73+
${PAIMON_BENCHMARK_STATIC_LINK_LIBS}
74+
test_utils_static
75+
Threads::Threads
76+
${PAIMON_BENCHMARK_PLATFORM_LINK_LIBS}
77+
${PAIMON_BENCHMARK_LINK_TOOLCHAIN}
78+
EXTRA_INCLUDES
79+
${CMAKE_SOURCE_DIR})
6680
endif()
6781

6882
if(PAIMON_BUILD_TESTS)
@@ -74,4 +88,21 @@ if(PAIMON_BUILD_TESTS)
7488
STATIC_LINK_LIBS
7589
paimon_shared
7690
${GTEST_LINK_TOOLCHAIN})
91+
92+
# Guards the format-layer assumptions parquet_format_benchmark.cpp is built on. The benchmark
93+
# itself is only compiled under PAIMON_BUILD_BENCHMARKS, which CI does not set, so this test is
94+
# what keeps those assumptions covered.
95+
add_paimon_test(parquet_format_benchmark_test
96+
SOURCES
97+
parquet_format_benchmark_test.cpp
98+
EXTRA_INCLUDES
99+
${CMAKE_SOURCE_DIR}
100+
STATIC_LINK_LIBS
101+
arrow
102+
parquet
103+
paimon_shared
104+
${PAIMON_LOCAL_FILE_SYSTEM_SHARED_LINK_LIBS}
105+
${PAIMON_PARQUET_FILE_FORMAT_STATIC_LINK_LIBS}
106+
test_utils_static
107+
${GTEST_LINK_TOOLCHAIN})
77108
endif()

0 commit comments

Comments
 (0)