Add seatunnel integration doc #464
There are no files selected for viewing
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| --- | ||
| title: "HugeGraph-SeaTunnel Connector Quick Start" | ||
| linkTitle: "使用 SeaTunnel Connector 同步数据" | ||
| weight: 5 | ||
| --- | ||
|
|
||
| ### 1 HugeGraph-SeaTunnel Connector 概述 | ||
|
|
||
| [Apache SeaTunnel](https://seatunnel.apache.org/) 是一个高性能、分布式的数据集成平台，支持海量数据的实时同步与批处理。 | ||
| HugeGraph 社区已向 Apache SeaTunnel 贡献了 Connector-V2 支持，用户可以通过 SeaTunnel 将 HugeGraph 与外部数据系统进行数据同步。 | ||
|
|
||
|  | ||
|
|
||
| ### 2 功能特性 | ||
|
|
||
| HugeGraph-SeaTunnel Connector 提供以下能力： | ||
|
|
||
| - **HugeGraph Sink Connector**（已发布）：将外部数据写入 HugeGraph，支持顶点和边的批量写入、更新与删除 | ||
| - **HugeGraph Source Connector**（开发中）：从 HugeGraph 读取图数据，支持顶点和边的批量读取 | ||
| - **顶点同步**：支持全量或增量同步顶点数据，支持多种 ID 策略 | ||
| - **边同步**：支持全量或增量同步边数据，自动关联源顶点与目标顶点 | ||
| - **Schema 自动管理**：支持自动创建 PropertyKey、VertexLabel、EdgeLabel（`CREATE_SCHEMA_WHEN_NOT_EXIST`） | ||
| - **数据迁移**：支持在不同 HugeGraph 实例之间迁移数据，或从其他数据源导入图数据 | ||
|
|
||
|  | ||
|
|
||
| ### 3 环境要求 | ||
|
|
||
| | 组件 | 版本要求 | 说明 | | ||
| |------|---------|------| | ||
| | Java | 8+ | SeaTunnel 运行环境要求 | | ||
| | Apache SeaTunnel | 2.3.12+ | Sink Connector 自此版本发布 | | ||
| | HugeGraph Server | 1.0.0+ | 建议使用最新稳定版 | | ||
|
|
||
| > **Source Connector 说明**：HugeGraph Source Connector 目前仅在 SeaTunnel 开发分支（Next/master）中可用， | ||
| > 尚未包含在正式 Release 中。如需使用 Source 功能，请从 SeaTunnel 源码编译或等待下一版本发布。 | ||
|
|
||
| ### 4 快速开始 | ||
|
|
||
| #### 4.1 安装 SeaTunnel | ||
|
|
||
| 请参考 [Apache SeaTunnel 部署指南](https://seatunnel.apache.org/docs/getting-started/) 完成 SeaTunnel 的安装部署。 | ||
|
|
||
| #### 4.2 使用 HugeGraph Sink Connector（写入数据） | ||
|
|
||
| 以下示例展示如何通过 SeaTunnel 将数据写入 HugeGraph： | ||
|
|
||
| **写入顶点数据：** | ||
|
|
||
| ```hocon | ||
| env { | ||
| job.mode = "BATCH" | ||
| } | ||
|
|
||
| source { | ||
| FakeSource { | ||
| plugin_output = "fake" | ||
| schema = { | ||
| fields = { | ||
| name = "string" | ||
| age = "int" | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| sink { | ||
| HugeGraph { | ||
| host = "127.0.0.1" | ||
| port = 8080 | ||
| graph_name = "hugegraph" | ||
| mappings = [ | ||
| { | ||
| type = "VERTEX" | ||
| label = "person" | ||
| idStrategy = "PRIMARY_KEY" | ||
| idFields = ["name"] | ||
| properties = ["name", "age"] | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| **写入边数据：** | ||
|
|
||
| ```hocon | ||
| sink { | ||
| HugeGraph { | ||
| host = "127.0.0.1" | ||
| port = 8080 | ||
| graph_name = "hugegraph" | ||
| mappings = [ | ||
| { | ||
| type = "EDGE" | ||
| label = "knows" | ||
| sourceConfig = { | ||
| label = "person" | ||
| idFields = ["person1_name"] | ||
| } | ||
| targetConfig = { | ||
| label = "person" | ||
| idFields = ["person2_name"] | ||
| } | ||
| properties = ["since"] | ||
| fieldMapping = { | ||
| person1_name = "name" | ||
| person2_name = "name" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| > **注意**：HugeGraph 的 Schema（PropertyKey、VertexLabel、EdgeLabel）需要在执行写入任务前预先创建， | ||
| > 或使用 `schema_save_mode = CREATE_SCHEMA_WHEN_NOT_EXIST`（默认行为）由 Connector 自动创建。 | ||
|
|
||
| #### 4.3 核心配置参数 | ||
|
|
||
| | 参数 | 类型 | 必填 | 默认值 | 说明 | | ||
| |------|------|------|--------|------| | ||
| | `host` | String | 是 | - | HugeGraph Server 地址 | | ||
| | `port` | Integer | 是 | - | HugeGraph Server 端口 | | ||
| | `graph_name` | String | 是 | - | 图名称 | | ||
| | `graph_space` | String | 否 | DEFAULT | 图空间名称 | | ||
| | `username` | String | 否 | - | 认证用户名 | | ||
| | `password` | String | 否 | - | 认证密码 | | ||
| | `protocol` | String | 否 | http | 协议，可选 `http` 或 `https` | | ||
| | `batch_size` | Integer | 否 | 500 | 每批写入的记录数 | | ||
| | `batch_interval_ms` | Integer | 否 | 5000 | 批次最大等待时间（毫秒） | | ||
| | `max_retries` | Integer | 否 | 3 | 写入失败最大重试次数 | | ||
| | `retry_backoff_ms` | Integer | 否 | 5000 | 重试间隔（毫秒） | | ||
|
|
||
| > 完整参数列表和详细说明请参考 [Apache SeaTunnel HugeGraph Connector 官方文档](https://seatunnel.apache.org/docs/connector-v2/sink/HugeGraph/)。 | ||
|
|
||
| ### 5 文档与资源 | ||
|
|
||
| - [Apache SeaTunnel 官方网站](https://seatunnel.apache.org/) | ||
| - [Apache SeaTunnel Connector-V2 文档](https://seatunnel.apache.org/docs/connector-v2/) | ||
| - [Apache SeaTunnel GitHub](https://github.com/apache/seatunnel) | ||
| - [HugeGraph GitHub](https://github.com/apache/hugegraph) | ||
|
|
||
| ### 6 许可证 | ||
|
|
||
| 与 HugeGraph 一样，HugeGraph-SeaTunnel Connector 采用 Apache 2.0 许可证。 |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,149 @@ | ||
| --- | ||
| title: "HugeGraph-SeaTunnel Connector Quick Start" | ||
| linkTitle: "Sync Data with SeaTunnel Connector" | ||
| weight: 5 | ||
| --- | ||
|
|
||
| ### 1 HugeGraph-SeaTunnel Connector Overview | ||
|
|
||
| [Apache SeaTunnel](https://seatunnel.apache.org/) is a high-performance, distributed data integration platform | ||
| that supports real-time synchronization and batch processing of massive data. The HugeGraph community has contributed | ||
| Connector-V2 support to Apache SeaTunnel, enabling users to synchronize data between HugeGraph and external data systems. | ||
|
|
||
|  | ||
|
|
||
| ### 2 Features | ||
|
|
||
| HugeGraph-SeaTunnel Connector provides the following capabilities: | ||
|
|
||
| - **HugeGraph Sink Connector** (Released): Write external data to HugeGraph, supporting batch write, update, and delete of vertices and edges | ||
| - **HugeGraph Source Connector** (In Development): Read graph data from HugeGraph, supporting batch reading of vertices and edges | ||
| - **Vertex Synchronization**: Support full or incremental vertex data synchronization with multiple ID strategies | ||
| - **Edge Synchronization**: Support full or incremental edge data synchronization, auto-resolving source and target vertices | ||
| - **Automatic Schema Management**: Support auto-creating PropertyKey, VertexLabel, EdgeLabel (`CREATE_SCHEMA_WHEN_NOT_EXIST`) | ||
| - **Data Migration**: Support data migration between different HugeGraph instances, or importing graph data from other data sources | ||
|
|
||
|  | ||
|
|
||
| ### 3 Environment Requirements | ||
|
|
||
| | Component | Version | Notes | | ||
| |-----------|---------|-------| | ||
| | Java | 8+ | SeaTunnel runtime requirement | | ||
| | Apache SeaTunnel | 2.3.12+ | Sink Connector released since this version | | ||
| | HugeGraph Server | 1.0.0+ | Latest stable version recommended | | ||
|
|
||
| > **Source Connector Note**: The HugeGraph Source Connector is currently only available in the SeaTunnel development | ||
| > branch (Next/master) and has not been included in an official release. If you need the Source feature, please build | ||
| > SeaTunnel from source or wait for the next release. | ||
|
|
||
| ### 4 Quick Start | ||
|
|
||
| #### 4.1 Install SeaTunnel | ||
|
|
||
| Please refer to the [Apache SeaTunnel Deployment Guide](https://seatunnel.apache.org/docs/getting-started/) to install and deploy SeaTunnel. | ||
|
|
||
| #### 4.2 Use HugeGraph Sink Connector (Write Data) | ||
|
|
||
| The following examples demonstrate how to write data to HugeGraph via SeaTunnel: | ||
|
|
||
| **Write vertices:** | ||
|
|
||
| ```hocon | ||
| env { | ||
| job.mode = "BATCH" | ||
| } | ||
|
|
||
| source { | ||
| FakeSource { | ||
| plugin_output = "fake" | ||
| schema = { | ||
| fields = { | ||
| name = "string" | ||
| age = "int" | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| sink { | ||
| HugeGraph { | ||
| host = "127.0.0.1" | ||
| port = 8080 | ||
| graph_name = "hugegraph" | ||
| mappings = [ | ||
| { | ||
| type = "VERTEX" | ||
| label = "person" | ||
| idStrategy = "PRIMARY_KEY" | ||
| idFields = ["name"] | ||
| properties = ["name", "age"] | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| **Write edges:** | ||
|
|
||
| ```hocon | ||
| sink { | ||
| HugeGraph { | ||
| host = "127.0.0.1" | ||
| port = 8080 | ||
| graph_name = "hugegraph" | ||
| mappings = [ | ||
| { | ||
| type = "EDGE" | ||
| label = "knows" | ||
| sourceConfig = { | ||
| label = "person" | ||
| idFields = ["person1_name"] | ||
| } | ||
| targetConfig = { | ||
| label = "person" | ||
| idFields = ["person2_name"] | ||
| } | ||
| properties = ["since"] | ||
| fieldMapping = { | ||
| person1_name = "name" | ||
| person2_name = "name" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| > **Note**: The HugeGraph schema (PropertyKey, VertexLabel, EdgeLabel) must be created before executing write tasks, | ||
| > or you can use `schema_save_mode = CREATE_SCHEMA_WHEN_NOT_EXIST` (default behavior) for the Connector to auto-create it. | ||
|
|
||
| #### 4.3 Core Configuration Parameters | ||
|
|
||
| | Parameter | Type | Required | Default | Description | | ||
| |-----------|------|----------|---------|-------------| | ||
| | `host` | String | Yes | - | HugeGraph Server host | | ||
| | `port` | Integer | Yes | - | HugeGraph Server port | | ||
| | `graph_name` | String | Yes | - | Graph name | | ||
| | `graph_space` | String | No | DEFAULT | Graph space name | | ||
| | `username` | String | No | - | Authentication username | | ||
| | `password` | String | No | - | Authentication password | | ||
| | `protocol` | String | No | http | Protocol, `http` or `https` | | ||
| | `batch_size` | Integer | No | 500 | Records per batch write | | ||
| | `batch_interval_ms` | Integer | No | 5000 | Max wait time per batch (ms) | | ||
| | `max_retries` | Integer | No | 3 | Max retries on write failure | | ||
| | `retry_backoff_ms` | Integer | No | 5000 | Retry backoff interval (ms) | | ||
|
|
||
| > For the complete parameter list and detailed documentation, please refer to the | ||
| > [Apache SeaTunnel HugeGraph Connector Official Docs](https://seatunnel.apache.org/docs/connector-v2/sink/HugeGraph/). | ||
|
|
||
| ### 5 Documentation & Resources | ||
|
|
||
| - [Apache SeaTunnel Official Website](https://seatunnel.apache.org/) | ||
| - [Apache SeaTunnel Connector-V2 Documentation](https://seatunnel.apache.org/docs/connector-v2/) | ||
| - [Apache SeaTunnel GitHub](https://github.com/apache/seatunnel) | ||
| - [HugeGraph GitHub](https://github.com/apache/hugegraph) | ||
|
|
||
| ### 6 License | ||
|
|
||
| Same as HugeGraph, HugeGraph-SeaTunnel Connector is licensed under Apache 2.0. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
\``text` opener is not closed anywhere in the file. The exact-head page contains five triple-backtick fences, so Hugo treats the remaining introduction, navigation, and contact content as one code block instead of rendering it as Markdown. Remove this stray opener (or add the intended matching fence) and verify the generated Chinese page.