feat(rest-server): add FileSystemCatalog-backed REST catalog server#371
Draft
TheR1sing3un wants to merge 2 commits into
Draft
feat(rest-server): add FileSystemCatalog-backed REST catalog server#371TheR1sing3un wants to merge 2 commits into
TheR1sing3un wants to merge 2 commits into
Conversation
Member
Author
|
@JingsongLi Hello, I'm not sure if the community needs this ability, so I'd like to ask for your opinion first. Internally, we mainly use rest catalogs. However, when conducting rest catalog tests on rust, it is often necessary to start the real rest catalog before we can test the behavior of real rest clients on the rust side. Therefore, I introduced a Rust version of the Rest Catalog that is directly based on the FileSystem wrapper. There is similar code in Java as well. Do you think this contribution to the community is reasonable? Or do I need to start it directly into the Test module instead of using it as a complete rest catalog server? |
TheR1sing3un
force-pushed
the
pr/rest-server
branch
from
5e813be to
c4d4a4d
Compare
Implement all column-level SchemaChange variants in TableSchema::apply_changes (add/rename/drop column, update column type/nullability/comment/position, update table comment), so FileSystemCatalog::alter_table and RESTCatalog::alter_table can evolve table schemas. Changes operate on top-level columns and reuse the existing ColumnAlreadyExist/ColumnNotExist errors; apply_changes keeps its single-argument signature (the catalog fills in the table name). Align the SchemaChange JSON wire format with Java Paimon: internally tagged by "action", with fieldNames arrays, comment / newDataType / keepNullability / newNullability / newComment fields, referenceFieldName move anchors, and FIRST/AFTER/BEFORE/LAST move types. Add AlterTableRequest and RESTApi::alter_table, and implement RESTCatalog::alter_table so the REST client can alter tables against a Paimon REST server.
Add a paimon-rest-server crate (library + binary, publish = false) that maps the Paimon REST protocol onto a real FileSystemCatalog, enabling end-to-end testing of the REST catalog without a Java server: config, database/table metadata CRUD, append write + commit (persisting the posted snapshot via SnapshotManager) + read back, and column-level alter table. The wire format mirrors Java Paimon so the existing RESTCatalog client can drive it and warehouses round-trip with Java. Add a CI job to run the crate's e2e tests, and a README describing usage and endpoints.
TheR1sing3un
force-pushed
the
pr/rest-server
branch
from
c4d4a4d to
c66c374
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: close #369
Note
Stacked on #370 (column-level alter table). This branch currently also
contains #370's commit, so the diff includes it until #370 merges. Kept as a
draft for now; once #370 is merged I will rebase and only the
paimon-rest-servercommit will remain.Add a real REST catalog server backed by
FileSystemCatalog, so theRESTCatalogclient can be tested end to end without a Java server.Brief change log
paimon-rest-servercrate (library + binary,publish = false) that maps the Paimon REST protocol onto a realFileSystemCatalog: config, database/table metadata CRUD, append write + commit (the commit endpoint persists the posted snapshot viaSnapshotManager) + read back, and column-level alter table.RESTCatalogclient drives it and warehouses round-trip with Java.README.Tests
tests/e2e.rs: metadata CRUD, write/commit/read round trip, and alter-table over REST. Wired into CI via a newcargo test -p paimon-rest-serverstep.API and Format
No change to
paimon's public API or storage format. The new crate introducesaxum/tower(MIT) dependencies;cargo deny check licensespasses without changes todeny.toml.Documentation
Crate
README.md(purpose, usage, endpoints).