-
Notifications
You must be signed in to change notification settings - Fork 37
[doc] Update fluss-rust readme markdown #415
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,25 +1,105 @@ | ||
| # Apache Fluss™ Rust Client (Incubating) | ||
| # Apache Fluss (Incubating) Official Rust Client | ||
|
|
||
| Rust client library for [Apache Fluss™](https://fluss.apache.org/). This crate provides the core client used by the fluss-rust workspace and by the Python and C++ bindings. | ||
| Official Rust client library for [Apache Fluss (Incubating)](https://fluss.apache.org/). | ||
|
|
||
| # Todo: move how to use to the first, and how to build to the last, https://github.com/apache/opendal/blob/main/core/README.md | ||
| # is a good reference | ||
| [](https://crates.io/crates/fluss-rs) | ||
| [](https://docs.rs/fluss-rs/) | ||
|
|
||
| ## Requirements | ||
| ## Usage | ||
|
|
||
| - Rust (see [rust-toolchain.toml](../../rust-toolchain.toml) at repo root) | ||
| - protobuf (for build) | ||
| The following example shows both **primary key (KV) tables** and **log tables** in one flow: connect, create a KV table (upsert + lookup), then create a log table (append + scan). | ||
|
|
||
| ## Build | ||
| ```rust | ||
| use fluss::client::EARLIEST_OFFSET; | ||
| use fluss::client::FlussConnection; | ||
| use fluss::config::Config; | ||
| use fluss::error::Result; | ||
| use fluss::metadata::{DataTypes, Schema, TableDescriptor, TablePath}; | ||
| use fluss::row::{GenericRow, InternalRow}; | ||
| use std::time::Duration; | ||
|
|
||
| From the repository root: | ||
| #[tokio::main] | ||
| async fn main() -> Result<()> { | ||
| let mut config = Config::default(); | ||
| config.bootstrap_servers = "127.0.0.1:9123".to_string(); | ||
| let connection = FlussConnection::new(config).await?; | ||
| let admin = connection.get_admin().await?; | ||
|
|
||
| ```bash | ||
| cargo build -p fluss-rs | ||
| // ---- Primary key (KV) table: upsert and lookup ---- | ||
| let kv_path = TablePath::new("fluss", "users"); | ||
| let mut kv_schema = Schema::builder() | ||
| .column("id", DataTypes::int()) | ||
| .column("name", DataTypes::string()) | ||
| .column("age", DataTypes::bigint()) | ||
| .primary_key(vec!["id"]); | ||
| let kv_descriptor = TableDescriptor::builder() | ||
| .schema(kv_schema.build()?) | ||
| .build()?; | ||
| admin.create_table(&kv_path, &kv_descriptor, false).await?; | ||
|
|
||
| let kv_table = connection.get_table(&kv_path).await?; | ||
| let upsert_writer = kv_table.new_upsert()?.create_writer()?; | ||
| let mut row = GenericRow::new(3); | ||
| row.set_field(0, 1i32); | ||
| row.set_field(1, "Alice"); | ||
| row.set_field(2, 30i64); | ||
| upsert_writer.upsert(&row)?; | ||
| upsert_writer.flush().await?; | ||
|
|
||
| let mut lookuper = kv_table.new_lookup()?.create_lookuper()?; | ||
| let mut key = GenericRow::new(1); | ||
| key.set_field(0, 1i32); | ||
| let result = lookuper.lookup(&key).await?; | ||
| if let Some(r) = result.get_single_row()? { | ||
| println!("KV lookup: id={}, name={}, age={}", | ||
| r.get_int(0)?, r.get_string(1)?, r.get_long(2)?); | ||
| } | ||
|
|
||
| // ---- Log table: append and scan ---- | ||
| let log_path = TablePath::new("fluss", "events"); | ||
| let log_schema = Schema::builder() | ||
| .column("ts", DataTypes::bigint()) | ||
| .column("message", DataTypes::string()) | ||
| .build()?; | ||
| let log_descriptor = TableDescriptor::builder() | ||
| .schema(log_schema) | ||
| .build()?; | ||
| admin.create_table(&log_path, &log_descriptor, false).await?; | ||
|
|
||
| let log_table = connection.get_table(&log_path).await?; | ||
| let append_writer = log_table.new_append()?.create_writer()?; | ||
| let mut event = GenericRow::new(2); | ||
| event.set_field(0, 1700000000i64); | ||
| event.set_field(1, "hello"); | ||
| append_writer.append(&event)?; | ||
| append_writer.flush().await?; | ||
|
|
||
| let scanner = log_table.new_scan().create_log_scanner()?; | ||
| scanner.subscribe(0, EARLIEST_OFFSET).await?; | ||
| let scan_records = scanner.poll(Duration::from_secs(1)).await?; | ||
| for record in scan_records { | ||
| let r = record.row(); | ||
| println!("Log scan: ts={}, message={}", r.get_long(0)?, r.get_string(1)?); | ||
luoyuxia marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| } | ||
|
|
||
| Ok(()) | ||
| } | ||
| ``` | ||
|
|
||
| ## Quick start and examples | ||
| ## Storage Support | ||
|
|
||
| The Fluss client reads remote data by accessing Fluss’s **remote files** (e.g. log segments and snapshots) directly. The following **remote file systems** are supported; enable the matching feature(s) for your deployment: | ||
|
|
||
| | Storage Backend | Feature Flag | Status | Description | | ||
| |----------------|--------------|--------|-------------| | ||
| | Local Filesystem | `storage-fs` | ✅ Stable | Local filesystem storage | | ||
| | Amazon S3 | `storage-s3` | ✅ Stable | Amazon S3 storage | | ||
| | Alibaba Cloud OSS | `storage-oss` | ✅ Stable | Alibaba Cloud Object Storage Service | | ||
|
|
||
| ## TODO | ||
| - [ ] Expand API documentation and usage examples in this README. | ||
| - [ ] Add more examples for table, log scan, and write flows. | ||
| You can enable all storage backends at once using the `storage-all` feature flag. | ||
|
|
||
| Example usage in Cargo.toml: | ||
| ```toml | ||
| [dependencies] | ||
| fluss-rs = { version = "0.x.x", features = ["storage-s3", "storage-fs"] } | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i wonder how easy it is to maintain this code snippet long-term in README.md, since it's not compiled, so API changes will silently make it stale. We already have examples in crates/examples/ that cover these flows.
Would we consider to keep short quickstart snippet to just give users a feel for API, and link example folder for detailed workflows. This was inspired by what OpenDAL does: https://github.com/apache/opendal/blob/main/core/README.md
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@charlesdong1991
This crossed my mind as well (I partially mitigate for python package docs by using generating it from other existing website documentations).
Generally there are duplications in test cases, examples, documentations and package readmes.
It sounds like there's an opportunity to improve this e.g., test cases as ground truth, scripts then populate docs code snippets from test cases. This way, there's less effort in separately maintaining doc as well as ensuring that documents are accurate. The work it saves increases as we have more language bindings. The script can even highlight gaps in testing /feature.
But this might be something that we consider in the mid/longer term.
It's certainly worth coming up with a proposal for it (there might be some other OS project which already solves this.) Happy to collaborate on proposing something.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the thoughtful discussion! I think we can keep the current approach for now —
the snippet is relatively small, and the API it demonstrates is stable enough that it's
unlikely to go stale anytime soon. Let's leave it as-is and revisit if it becomes a
real maintenance pain point down the road.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cool! Let's keep as is and revisit if becoming an issue long term! Thanks for the discussion!