perf: preallocate RowsGroupColumn buffers in take_n - #24070
Conversation
|
@kosiew - ready for review! |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24070 +/- ##
==========================================
+ Coverage 80.88% 80.89% +0.01%
==========================================
Files 1102 1102
Lines 375813 376184 +371
Branches 375813 376184 +371
==========================================
+ Hits 303971 304318 +347
- Misses 53741 53755 +14
- Partials 18101 18111 +10 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
@saadtajwar
Looks good overall. I left one non-blocking suggestion to make the preallocation behavior more directly exercised by the test.
| assert_eq!(g0, 20); | ||
| } | ||
|
|
||
| /// `take_n` preallocates the retained-row buffer from the known row |
There was a problem hiding this comment.
The test does a good job of confirming that rebuilding works correctly, but I do not think it distinguishes the new preallocation logic from the previous empty_rows(0, 0) path. The outer-null FixedSizeList has a shorter row encoding, but that row is emitted rather than retained. The retained inner-null and non-null Int32 rows appear to have the same encoded length.
Could we change or add retained rows with different encoded sizes, perhaps using FixedSizeList<Utf8> values with unequal payload lengths? That would exercise the lengths().skip(n).sum() preallocation calculation more directly.
There was a problem hiding this comment.
Thanks - pushed a change to exercise this directly! Please let me know your thoughts!
Which issue does this PR close?
Rationale for this change
Currently
RowsGroupColumn'stake_nmethod knows the remaining number of rows and bytes after taking the first n group rows, but when invokingempty_rowspasses in 0 for the row capacity and data capacity parameters instead of the known remainders, potentially causing unnecessary copying and allocatingWhat changes are included in this PR?
Using the known remaining number of rows and bytes in the invocation to
empty_rowsAre these changes tested?
Yes
Are there any user-facing changes?
No