IN LIST: add UInt16 bitmap filter#23012
Draft
geoffreyclaude wants to merge 5 commits into
Draft
Conversation
This was referenced Jun 18, 2026
55f3836 to
81ec379
Compare
This was referenced Jun 18, 2026
7043d4b to
2dbce01
Compare
This was referenced Jun 19, 2026
Replaces HashSet<u8> with a 32-byte stack-allocated bitmap. Provides O(1) membership testing via bit-shifting, significantly reducing memory overhead and improving cache locality. Triggers for UInt8 arrays.
Implements an 8 KB heap-allocated bitmap for UInt16. Maintains O(1) performance while handling the larger value space. Triggers for UInt16 arrays.
2dbce01 to
5514d78
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
INperformance with specialized implementations #19390.Rationale for this change
#23011 uses a bitmap checklist for
UInt8, where there are 256 possible values.UInt16is the same idea with a larger value range: 0 through 65,535.That is still small enough to represent directly. A
UInt16bitmap needs one bit for each possible value:Then a lookup is still simple: use the input value as the bit position and check whether that bit is set. For example, if the list contains
42, bit42is set, and every input row with value42can be recognized with one bit test.This PR keeps the scope narrow: it adds the unsigned 2-byte bitmap path as a concrete
UInt16filter. #23035 then unifies theUInt8andUInt16implementations, and #23013 uses that shared shape for signed same-width reinterpretation.What changes are included in this PR?
UInt16BitmapFilter, backed by a heap-allocated 65,536-bit bitmap.UInt16constant-list filtering to that bitmap path.IN/NOT INnull behavior as the generic path.UInt16boundary values, nulls, andNOT IN.Are these changes tested?
Yes.
cargo fmt --allcargo test -p datafusion-physical-expr bitmap_filter_u16 --libcargo test -p datafusion-physical-expr in_list_int_types --libcargo test -p datafusion-physical-expr test_in_list_from_array_type_combinations --libcargo test -p datafusion-physical-expr test_in_list_dictionary_types --libcargo clippy -p datafusion-physical-expr --all-targets --all-features -- -D warningsAre there any user-facing changes?
No. This is an internal performance optimization only.
Benchmark note
No local
in_list_strategynumbers are included for this PR because the benchmark harness does not currently include a directUInt16case. The availablei16rows measure the signed reinterpretation path added in #23013 after the bitmap unification in #23035, not this PR's unsignedUInt16bitmap filter.