Draft
Conversation
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
github-actions
bot
removed
the
action/benchmark-sql
Merging this PR will degrade performance by 10.43%
Performance Changes
Comparing 
Footnotes
|
Contributor
Polar Signals Profiling ResultsLatest Run
Previous Runs (4)
Powered by Polar Signals Cloud |
Contributor
Benchmarks: PolarSignals ProfilingSummary
Detailed Results Table
|
Contributor
Benchmarks: FineWeb NVMeSummary
Detailed Results Table
|
Contributor
Benchmarks: TPC-H SF=1 on NVMESummary
Detailed Results Table
|
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
github-actions
bot
removed
the
action/benchmark-sql
Contributor
Benchmarks: TPC-DS SF=1 on NVMESummary
Detailed Results Table
|
Contributor
Benchmarks: TPC-H SF=10 on NVMESummary
Detailed Results Table
|
|
FWIW the morsels in apache/datafusion#20481 are very much an IO scheduler for Parquet scans... |
Contributor
Benchmarks: TPC-H SF=1 on S3Summary
Detailed Results Table
|
Contributor
Benchmarks: FineWeb S3Summary
Detailed Results Table
|
Contributor
Benchmarks: Statistical and Population GeneticsSummary
Detailed Results Table
|
Contributor
Benchmarks: Clickbench on NVMESummary
Detailed Results Table
|
Contributor
Benchmarks: TPC-H SF=10 on S3Summary
Detailed Results Table
|
Contributor
Author
|
@alamb I haven't read through the PR in a few days and it went through a lot of change since, this is mostly claude-driven because I was curious if there's any "free" perf here, I now see that the work stealing queue is enabled by |
github-actions
bot
removed
the
action/benchmark-sql
Contributor
Author
|
I guess if we can figure out some morsel size, there's also benefit in trying approach right? |
Absolutely -- I am also trying to sketch out a slightly different API (see details here apache/datafusion#20529 (comment)) for morsels. The idea of using this in Vortex is a great usecase -- I'll ping you on the PR for your feedback |
7 tasks
Based on some iterative benchmarking on Parquet, a "projected compressed size" seems a relatively good proxy to avoid splitting really narrow / small (e.g. all primitive) row groups of a number of MB (e.g. ~2-16MB). 100K rows (which is commonly used) seems to create lot's of small scans / too much overhead of e.g. one-column scans on primitive fields |
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
github-actions
bot
removed
the
action/benchmark-sql
Signed-off-by: Adam Gutglick <adam@spiraldb.com>
github-actions
bot
removed
the
action/benchmark-sql
|
Something that works quite good for Parquet on a couple of benchmarks it seems (on local file system):
This is with splitting both IO and decode a row group level (i.e. it does a bunch of IO a number of times) but seems not to hurt those benchmarks really. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Basically two changes smashed together: