[docs] Add RoaringBitmap SQL functions documentation - #3823
Conversation
|
@polyzos @platinumhamburg Could you please take a look at this PR? |
| </TabItem> | ||
| </Tabs> | ||
|
|
||
| ## RoaringBitmap SQL Functions |
There was a problem hiding this comment.
These are Flink-side SQL functions registered by the Fluss catalog, not storage-level aggregate functions executed by the Fluss merge engine, so documenting them in the Aggregation Merge Engine page blurs the ownership and execution boundary. Could we move this section to a dedicated website/docs/engine-flink/sql-functions.md page titled SQL Functions, place it after Delta Joins in the Engine Flink sidebar, and leave only a short cross-link next to rbm32/rbm64 here?
For each function, please make the example self-contained and show the expected output. Prefer deterministic VALUES inputs so readers can immediately connect the inputs, output, and documented behavior. Also, please rename End-to-End Example to How to Use and move to the beginning of the page, and use that section to explain how to enable and use these Fluss-provided Flink functions, followed by the broader usage example.
There was a problem hiding this comment.
- Moved the entire section to a new dedicated page website/docs/engine-flink/sql-functions.md, placed after Delta Joins in the Flink sidebar
- Added a short cross-link in aggregation.md next to rbm32/rbm64 pointing to the new page
- Rewrote "End-to-End Example" as "How to Use" and moved it to the top of the page to explain catalog setup first
- All examples now use deterministic VALUES inputs and show explicit expected outputs
| Constructs a serialized `RoaringBitmap` from an array of integers within a single row. | ||
|
|
||
| - **Signature**: `rb_build(values ARRAY<INT>) -> BYTES` | ||
| - **Null Handling**: Null elements in the array are ignored; returns `NULL` if all elements are null |
There was a problem hiding this comment.
rb_build does not return NULL when all array elements are null. The implementation returns a serialized empty bitmap for any non-null array, and the existing test explicitly verifies that an all-null array produces a non-null empty bitmap. Therefore, rb_build(ARRAY[CAST(NULL AS INT)]) IS NULL is actually false, and rb_cardinality(...) returns 0. Please update the documentation to state that only a null array returns NULL; empty arrays and all-null arrays return an empty bitmap.
There was a problem hiding this comment.
Fixed rb_build null semantics: a NULL array returns NULL; an empty array or all-null array returns an empty bitmap
|
Thanks @wuchong for review, this looks more structured and well organized than before, PTAL another look when you have some time thanks! |
Purpose
Linked issue: Part of #3289
Adds user-facing documentation for the 12 RoaringBitmap SQL functions introduced by FIP-37 (PRs #3319, #3398, #3492, #3777). The functions are now registered natively in FlussCatalog and available after
USE CATALOG fluss_catalogwithout anyCREATE TEMPORARY FUNCTIONstatement.Brief change log
Modified
website/docs/table-design/merge-engines/aggregation.md:Added a new
## RoaringBitmap SQL Functionssection afterrbm64, covering all 12 functions across two subsections:Aggregate functions:
rb_build_agg,rb_or_agg,rb_and_agg,rb_xor_aggScalar functions:
rb_cardinality,rb_build,rb_contains,rb_to_array,rb_or,rb_and,rb_xor,rb_andnotEach function entry includes signature, behavior, null semantics, and a Flink SQL example. Functions without a server-side counterpart (
rb_and_agg,rb_xor_agg) include a note about compaction behavior. A note distinguishes these Flink-side SQL functions from the storage-levelrbm32/rbm64aggregators already documented in the file.Closes with an end-to-end example covering table creation, ingestion with
rb_build_agg, point query withrb_cardinality, and roll-up query withrb_or_agg, with a reference to the Real-Time UV Deduplication blog post for a full Docker-based tutorial.Tests
Manually verified by starting the website locally , The SQL examples have been verified against the implemented functions in PRs #3319, #3398, #3492, and #3777.
API and Format
This change does not affect any API or storage format. Documentation only.
Documentation
Yes, this PR adds the user-facing reference documentation for FIP-37 bitmap SQL functions, completing the documentation commitment made in PRs #3319, #3398, #3492, and #3777.