[Umbrella] FIP-17: Support Full KV Scan for Primary Key Tables

[polyzos]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Fluss KV currently supports point lookups, but RocksDB allows iterators, range, and prefix scans.

For small tables (hundreds of rows or a few thousands), that can be used to create dashboards often need to periodically read all keys.

Goal: Assume I have a front-end and I want to create a dashboard to show < 1-2k values.. In this scenario, I could use the fluss client directly to periodically query the whole kv store

Solution

We can expose functionality to the api to allow a full scan (with snapshot consistency) to simplify such use cases.

---

An implementation can be found here and for easier reviews it can be broken down to the following individuals PRs:

[xx789633]

Hi @polyzos I can take this task. Please assign it to me.

[polyzos]

@xx789633 Thank you for your interest. Assigned 👍

[xx789633]

Hi @polyzos , I think we can reuse the existing KV snapshots for batch reading (#88) instead of invoking the Rocksdb scan iterator. This reduces the overhead on Rocksdb. What do you think?

[xx789633]

Currently, a workaround for full scan can be implemented like this:

            Scan scan = table.newScan().limit(limit).project(projectedFields);
            List<BatchScanner> scanners =
                    tableBuckets.stream()
                            .map(scan::createBatchScanner)
                            .collect(Collectors.toList());
            List<InternalRow> scannedRows = BatchScanUtils.collectLimitedRows(scanners, limit);

[polyzos]

@xx789633 I have some time and wanted to give this a spin.. If there is no PR, do you mind if I also give this a try?

[xx789633]

@polyzos I'm really sorry for the slow progress. I'm currently tied up with some other community tasks. ​​Please feel free to move forward.

[polyzos]

@xx789633 Nothing to do with slow progress and nothing urgent 😁 Based on our offline comms, I realise how busy you are, and because I have some extra time this week, I will be eager to give this a try.
I will keep you in the loop about any progress 🙏