You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This PR introduces the Python FieldRoaringBitmap32Agg implementations and aligns the Python roaring bitmap aggregation behavior with the Java implementation.
The changes include:
Add rbm32 field aggregation support.
Support roaring bitmap merge using serialized bytes representation.
Add VARBINARY type validation for roaring bitmap aggregation.
Add unit tests for aggregation and invalid input types.
rbm64 is serialized using pyroaring.BitMap64, but it is incompatible with the Java Roaring64Bitmap format. Bidirectional verification fails in both cases: Java reads an EOF when accessing the Python datagram, and Python cannot deserialize the Java datagram; this breaks cross-engine reading between Java and PyPaimon.
rbm64 is serialized using pyroaring.BitMap64, but it is incompatible with the Java Roaring64Bitmap format. Bidirectional verification fails in both cases: Java reads an EOF when accessing the Python datagram, and Python cannot deserialize the Java datagram; this breaks cross-engine reading between Java and PyPaimon.
Thanks for the feedback. I confirmed that pyroaring.BitMap64 is not compatible with Java's Roaring64Bitmap serialization format, so cross-engine interoperability cannot be guaranteed. I'll remove the rbm64 aggregator from this PR and revisit it when a compatible implementation is available.
PyRSA
changed the title
[python] Introduce rbm32/rbm64 aggregator function
[python] Introduce rbm32 aggregator function
Jul 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
This PR introduces the Python
FieldRoaringBitmap32Aggimplementations and aligns the Python roaring bitmap aggregation behavior with the Java implementation.The changes include:
rbm32field aggregation support.bytesrepresentation.VARBINARYtype validation for roaring bitmap aggregation.Tests
python -m pytest pypaimon/tests/test_field_aggregators.py::FieldRoaringBitmap32AggTest -qpython -m pytest pypaimon/tests/test_aggregation_e2e.py -qpython -m pytest pypaimon/tests/test_aggregation_merge_function.py -qgit diff --check