Skip to content

[docs] Add RoaringBitmap SQL functions documentation - #3823

Open
pbanakar wants to merge 2 commits into
apache:mainfrom
pbanakar:FIP37-docs
Open

[docs] Add RoaringBitmap SQL functions documentation#3823
pbanakar wants to merge 2 commits into
apache:mainfrom
pbanakar:FIP37-docs

Conversation

@pbanakar

@pbanakar pbanakar commented Aug 2, 2026

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: Part of #3289

Adds user-facing documentation for the 12 RoaringBitmap SQL functions introduced by FIP-37 (PRs #3319, #3398, #3492, #3777). The functions are now registered natively in FlussCatalog and available after USE CATALOG fluss_catalog without any CREATE TEMPORARY FUNCTION statement.

Brief change log

Modified website/docs/table-design/merge-engines/aggregation.md:

Added a new ## RoaringBitmap SQL Functions section after rbm64, covering all 12 functions across two subsections:

Aggregate functions: rb_build_agg, rb_or_agg, rb_and_agg, rb_xor_agg

Scalar functions: rb_cardinality, rb_build, rb_contains, rb_to_array, rb_or, rb_and, rb_xor, rb_andnot

Each function entry includes signature, behavior, null semantics, and a Flink SQL example. Functions without a server-side counterpart (rb_and_agg, rb_xor_agg) include a note about compaction behavior. A note distinguishes these Flink-side SQL functions from the storage-level rbm32/rbm64 aggregators already documented in the file.

Closes with an end-to-end example covering table creation, ingestion with rb_build_agg, point query with rb_cardinality, and roll-up query with rb_or_agg, with a reference to the Real-Time UV Deduplication blog post for a full Docker-based tutorial.

Tests

Manually verified by starting the website locally , The SQL examples have been verified against the implemented functions in PRs #3319, #3398, #3492, and #3777.

API and Format

This change does not affect any API or storage format. Documentation only.

Documentation

Yes, this PR adds the user-facing reference documentation for FIP-37 bitmap SQL functions, completing the documentation commitment made in PRs #3319, #3398, #3492, and #3777.

@pbanakar

pbanakar commented Aug 2, 2026

Copy link
Copy Markdown
Contributor Author

@polyzos @platinumhamburg Could you please take a look at this PR?

@wuchong wuchong left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @pbanakar , I left some comments.

</TabItem>
</Tabs>

## RoaringBitmap SQL Functions

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These are Flink-side SQL functions registered by the Fluss catalog, not storage-level aggregate functions executed by the Fluss merge engine, so documenting them in the Aggregation Merge Engine page blurs the ownership and execution boundary. Could we move this section to a dedicated website/docs/engine-flink/sql-functions.md page titled SQL Functions, place it after Delta Joins in the Engine Flink sidebar, and leave only a short cross-link next to rbm32/rbm64 here?

For each function, please make the example self-contained and show the expected output. Prefer deterministic VALUES inputs so readers can immediately connect the inputs, output, and documented behavior. Also, please rename End-to-End Example to How to Use and move to the beginning of the page, and use that section to explain how to enable and use these Fluss-provided Flink functions, followed by the broader usage example.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Moved the entire section to a new dedicated page website/docs/engine-flink/sql-functions.md, placed after Delta Joins in the Flink sidebar
  • Added a short cross-link in aggregation.md next to rbm32/rbm64 pointing to the new page
  • Rewrote "End-to-End Example" as "How to Use" and moved it to the top of the page to explain catalog setup first
  • All examples now use deterministic VALUES inputs and show explicit expected outputs

Constructs a serialized `RoaringBitmap` from an array of integers within a single row.

- **Signature**: `rb_build(values ARRAY<INT>) -> BYTES`
- **Null Handling**: Null elements in the array are ignored; returns `NULL` if all elements are null

@wuchong wuchong Aug 5, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rb_build does not return NULL when all array elements are null. The implementation returns a serialized empty bitmap for any non-null array, and the existing test explicitly verifies that an all-null array produces a non-null empty bitmap. Therefore, rb_build(ARRAY[CAST(NULL AS INT)]) IS NULL is actually false, and rb_cardinality(...) returns 0. Please update the documentation to state that only a null array returns NULL; empty arrays and all-null arrays return an empty bitmap.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed rb_build null semantics: a NULL array returns NULL; an empty array or all-null array returns an empty bitmap

@pbanakar

pbanakar commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @wuchong for review, this looks more structured and well organized than before, PTAL another look when you have some time thanks!

@pbanakar pbanakar changed the title [docs] Add RoaringBitmap SQL functions to aggregation documentation [docs] Add RoaringBitmap SQL functions documentation Aug 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants