Optimize group by with limit#22009
Conversation
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing optimize-clickbench-q17-limit (dbd05b5) to 948cd09 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing optimize-clickbench-q17-limit (dbd05b5) to 948cd09 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing optimize-clickbench-q17-limit (dbd05b5) to 948cd09 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
run benchmarks |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing optimize-clickbench-q17-limit (d611742) to 948cd09 (merge-base) diff using: clickbench_partitioned File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing optimize-clickbench-q17-limit (d611742) to 948cd09 (merge-base) diff using: tpcds File an issue against this benchmark runner |
|
🤖 Benchmark running (GKE) | trigger CPU Details (lscpu)Comparing optimize-clickbench-q17-limit (d611742) to 948cd09 (merge-base) diff using: tpch File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpch — base (merge-base)
tpch — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usageclickbench_partitioned — base (merge-base)
clickbench_partitioned — branch
File an issue against this benchmark runner |
|
🤖 Benchmark completed (GKE) | trigger Instance: CPU Details (lscpu)Details
Resource Usagetpcds — base (merge-base)
tpcds — branch
File an issue against this benchmark runner |
|
@Dandandan , wow ! |
Yeah this is a pretty nice find 😄 |
Which issue does this PR close?
Rationale for this change
ClickBench q17 aggregates by
UserID, SearchPhraseand applies a small LIMIT. Building the full aggregate state can use substantially more memory than needed for queries where any limited set of grouping keys is sufficient.This PR preselects a limited set of grouping keys before running the original aggregate, allowing existing join dynamic filtering to push that key set into the second input scan.
What changes are included in this PR?
PushDownLimitto rewrite limited grouped aggregates with aggregate expressions whenenable_distinct_aggregation_soft_limitis enabled.LIMIT K (GROUP BY keys)as a key preselection plan, then right-semi joins the original input against those keys before the final aggregate.Are these changes tested?
Yes, existing tests
Are there any user-facing changes?
No API changes.