feat: add clickhouse-bench with auto-downloaded ClickHouse binary by fastio · Pull Request #6736 · vortex-data/vortex

fastio · 2026-03-02T10:14:09Z

Introduce a new clickhouse-bench benchmark crate that runs ClickBench queries against Parquet data via clickhouse-local, providing a baseline for comparing Vortex performance against ClickHouse.

Key design decisions:

build.rs auto-downloads the full ClickHouse binary (with Parquet support) into target/clickhouse-local/, similar to how vortex-duckdb downloads the DuckDB library. This eliminates manual install steps and avoids issues with slim/homebrew builds lacking Parquet support.
The binary path is baked in via CLICKHOUSE_BINARY env at compile time; CLICKHOUSE_LOCAL env var allows runtime override.
ClickHouse-dialect SQL queries are maintained in a separate clickbench_clickhouse_queries.sql file (43 queries).
CI workflows updated to include clickhouse:parquet target in ClickBench benchmarks and conditionally build clickhouse-bench.

#6425

joseph-isaacs · 2026-03-02T10:34:55Z

vortex-bench/clickbench_clickhouse_queries.sql

why do we need this file is it difference to the already included one?

Good catch! I have removed the duplicate clickbench_clickhouse_queries.sql and validated with cargo check -p vortex-bench.

myrrc

I don't think downloading untrusted binaries from internet via a build script is a good idea. We want first-class integration with duckdb thus we need to download its sources (although I'd not do it in build script as well), but we don't need such integration with Clickhouse yet.

My idea is to use clickhouse binary in CI (as it runs on Linux only) and require users to download it by hand if they want a local run. Benchmarking on MacOS doesn't make much sense anyway as vectorized instrustion set is different.

fastio · 2026-03-03T13:05:16Z

I don't think downloading untrusted binaries from internet via a build script is a good idea. We want first-class integration with duckdb thus we need to download its sources (although I'd not do it in build script as well), but we don't need such integration with Clickhouse yet.

My idea is to use clickhouse binary in CI (as it runs on Linux only) and require users to download it by hand if they want a local run. Benchmarking on MacOS doesn't make much sense anyway as vectorized instrustion set is different.

Agreed — removed the binary download from build.rs entirely. The clickhouse binary is now resolved at runtime: via CLICKHOUSE_BINARY env var or from $PATH. CI installs it via the official installer before building. Local users need to install it manually. No more untrusted binary downloads in the build script.

myrrc · 2026-03-03T17:27:07Z

.github/workflows/sql-benchmarks.yml

+      - name: Install ClickHouse
+        if: contains(matrix.targets, 'clickhouse:')
+        run: |
+          curl https://clickhouse.com/ | sh


Why not download the latest release file for our architecture from Github releases? We then don't need any installation and curl in general.

Good call — updated CI to download the static binary directly from GitHub Releases (pined ClickHouse to LTS release v25.8.18.1 from GitHub Releases), no curl | sh or installation needed.

myrrc · 2026-03-09T14:31:48Z

vortex-bench/src/clickbench/benchmark.rs

+            return query.to_string();
+        }
+
+        strip_simple_identifier_quotes(query)


Clickhouse does handle quoted identifiers correctly so I think we can pass them through to reduce this PR's diff.

myrrc · 2026-03-09T14:32:42Z

The changes look good to me conceptually, let's see what the CI run says.

Introduce a new clickhouse-bench benchmark crate that runs ClickBench queries against Parquet data via clickhouse-local, providing a baseline for comparing Vortex performance against ClickHouse. Key design decisions: - build.rs auto-downloads the full ClickHouse binary (with Parquet support) into target/clickhouse-local/, similar to how vortex-duckdb downloads the DuckDB library. This eliminates manual install steps and avoids issues with slim/homebrew builds lacking Parquet support. - The binary path is baked in via CLICKHOUSE_BINARY env at compile time; CLICKHOUSE_LOCAL env var allows runtime override. - ClickHouse-dialect SQL queries are maintained in a separate clickbench_clickhouse_queries.sql file (43 queries). - CI workflows updated to include clickhouse:parquet target in ClickBench benchmarks and conditionally build clickhouse-bench. Signed-off-by: fastio <pengjian.uestc@gmail.com>

…dling Signed-off-by: fastio <pengjian.uestc@gmail.com>

…use from PATH - Remove reqwest-based binary download from build.rs - Resolve clickhouse binary via CLICKHOUSE_BINARY env var or $PATH at runtime - Add CI step to install clickhouse before building when needed - Fail with clear error message if binary is not found locally Signed-off-by: fastio <pengjian.uestc@gmail.com>

- Pass subcommand arg to clickhouse-bench in run-sql-bench.sh for consistency - Use BenchmarkArg + create_benchmark() in main.rs like other engines - Replace `which` with `clickhouse local --version` for binary verification - Pin ClickHouse to LTS release v25.8.18.1 from GitHub Releases Signed-off-by: fastio <pengjian.uestc@gmail.com>

…identifier handling. Queries are now returned as-is without dialect-specific transformation. Signed-off-by: fastio <pengjian.uestc@gmail.com>

Signed-off-by: fastio <pengjian.uestc@gmail.com>

Signed-off-by: Peng Jian <pengjian.uestc@gmail.com>

codspeed-hq · 2026-03-10T14:19:37Z

Merging this PR will degrade performance by 20.89%

❌ 4 regressed benchmarks
✅ 996 untouched benchmarks
⏩ 1466 skipped benchmarks¹

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
❌	Simulation	`patched_take_200k_dispersed`	4.7 ms	5.6 ms	-16.59%
❌	Simulation	`take_200k_first_chunk_only`	3.3 ms	4.2 ms	-20.89%
❌	Simulation	`patched_take_200k_first_chunk_only`	4.8 ms	5.4 ms	-10.69%
❌	Simulation	`take_200k_dispersed`	3.6 ms	4.5 ms	-19.61%

_{Comparing fastio:integration-clickhouse-benchmark-baseline (fb28623) with develop (e477fa5)}

1466 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

connortsui20 · 2026-03-10T14:23:39Z

@fastio Feel free to ping us in the public slack channel if you want us to run CI for you! (Feel free to ping me here as well)

Edit: To fix the CI issues right now, could you update the lockfile with cargo check?

myrrc self-requested a review March 2, 2026 10:32

joseph-isaacs reviewed Mar 2, 2026

View reviewed changes

myrrc reviewed Mar 3, 2026

View reviewed changes

fastio requested review from joseph-isaacs and myrrc March 9, 2026 02:22

myrrc reviewed Mar 9, 2026

View reviewed changes

myrrc added changelog/feature A new feature changelog/ci labels Mar 9, 2026

myrrc self-assigned this Mar 9, 2026

fastio force-pushed the integration-clickhouse-benchmark-baseline branch from 82c11d1 to 60aa2d2 Compare March 10, 2026 13:13

fastio added 6 commits March 10, 2026 21:27

bench(clickbench): tighten ClickHouse query normalization and URL han…

38d2fe7

…dling Signed-off-by: fastio <pengjian.uestc@gmail.com>

Remove the and helpers that were only needed for ClickHouse unquoted …

7fffcf5

…identifier handling. Queries are now returned as-is without dialect-specific transformation. Signed-off-by: fastio <pengjian.uestc@gmail.com>

fix build errors

5aa201a

Signed-off-by: fastio <pengjian.uestc@gmail.com>

fastio force-pushed the integration-clickhouse-benchmark-baseline branch from 60aa2d2 to 5aa201a Compare March 10, 2026 13:28

Merge branch 'develop' into integration-clickhouse-benchmark-baseline

fb28623

Signed-off-by: Peng Jian <pengjian.uestc@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add clickhouse-bench with auto-downloaded ClickHouse binary#6736

feat: add clickhouse-bench with auto-downloaded ClickHouse binary#6736
fastio wants to merge 7 commits intovortex-data:developfrom
fastio:integration-clickhouse-benchmark-baseline

fastio commented Mar 2, 2026 •

edited

Loading

Uh oh!

joseph-isaacs Mar 2, 2026

Uh oh!

fastio Mar 3, 2026 •

edited

Loading

Uh oh!

myrrc left a comment

Uh oh!

fastio commented Mar 3, 2026 •

edited

Loading

Uh oh!

myrrc Mar 3, 2026

Uh oh!

fastio Mar 4, 2026

Uh oh!

myrrc Mar 9, 2026

Uh oh!

myrrc commented Mar 9, 2026 •

edited

Loading

Uh oh!

codspeed-hq bot commented Mar 10, 2026

Uh oh!

connortsui20 commented Mar 10, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

fastio commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joseph-isaacs Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

fastio Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

myrrc left a comment

Choose a reason for hiding this comment

Uh oh!

fastio commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

myrrc Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

fastio Mar 4, 2026

Choose a reason for hiding this comment

Uh oh!

myrrc Mar 9, 2026

Choose a reason for hiding this comment

Uh oh!

myrrc commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq bot commented Mar 10, 2026

Merging this PR will degrade performance by 20.89%

Performance Changes

Footnotes

Uh oh!

connortsui20 commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fastio commented Mar 2, 2026 •

edited

Loading

fastio Mar 3, 2026 •

edited

Loading

fastio commented Mar 3, 2026 •

edited

Loading

myrrc commented Mar 9, 2026 •

edited

Loading

connortsui20 commented Mar 10, 2026 •

edited

Loading