diff --git a/TOC-tidb-cloud-premium.md b/TOC-tidb-cloud-premium.md index 93aeceaa0b66c..135b1e6bdbf86 100644 --- a/TOC-tidb-cloud-premium.md +++ b/TOC-tidb-cloud-premium.md @@ -140,6 +140,7 @@ - [Changefeed Overview](/tidb-cloud/changefeed-overview.md) - [To MySQL Sink](/tidb-cloud/changefeed-sink-to-mysql.md) - [To Kafka Sink](/tidb-cloud/changefeed-sink-to-apache-kafka.md) + - [To Cloud Storage](/tidb-cloud/changefeed-sink-to-cloud-storage.md) - Reference - [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md) - [Set Up Private Endpoint for Changefeeds](/tidb-cloud/premium/set-up-sink-private-endpoint-premium.md) diff --git a/tidb-cloud/changefeed-sink-to-cloud-storage.md b/tidb-cloud/changefeed-sink-to-cloud-storage.md index da0afce453b23..25d5c9f80816c 100644 --- a/tidb-cloud/changefeed-sink-to-cloud-storage.md +++ b/tidb-cloud/changefeed-sink-to-cloud-storage.md @@ -1,28 +1,54 @@ --- title: Sink to Cloud Storage -summary: This document explains how to create a changefeed to stream data from TiDB Cloud to Amazon S3, Google Cloud Storage (GCS), or Azure Blob Storage. It includes restrictions, configuration steps for the destination, replication, and specification, as well as starting the replication process. +summary: This document explains how to create a changefeed to stream data from TiDB Cloud to cloud storage. It includes restrictions, configuration steps for the destination, replication, and specification, as well as starting the replication process. --- # Sink to Cloud Storage -This document describes how to create a changefeed to stream data from TiDB Cloud to cloud storage. Currently, Amazon S3, Google Cloud Storage (GCS), and Azure Blob Storage are supported. + +This document describes how to create a changefeed to stream data from TiDB Cloud DedicatedTiDB Cloud Premium to cloud storage. + + + +> **Note:** +> +> - To stream data from {{{ .dedicated }}} to cloud storage, make sure that your TiDB cluster version is v7.1.1 or later. To upgrade your TiDB Cloud Dedicated cluster to v7.1.1 or later, [contact TiDB Cloud Support](/tidb-cloud/tidb-cloud-support.md). +> - For [{{{ .starter }}}](/tidb-cloud/select-cluster-tier.md#starter) instances, the changefeed feature is unavailable. +> - For [{{{ .essential }}}](/tidb-cloud/select-cluster-tier.md#essential) instances, the changefeed feature is only available upon request. For more information, see [Changefeed](/tidb-cloud/essential-changefeed-overview.md). +> - For [{{{ .premium }}}](/tidb-cloud/select-cluster-tier.md#premium) instances, see [Sink to Cloud Storage](https://docs.pingcap.com/tidbcloud/changefeed-sink-to-cloud-storage/?plan=premium). + + + + > **Note:** > -> - To stream data to cloud storage, make sure that your TiDB cluster version is v7.1.1 or later. To upgrade your TiDB Cloud Dedicated cluster to v7.1.1 or later, [contact TiDB Cloud Support](/tidb-cloud/tidb-cloud-support.md). > - For [{{{ .starter }}}](/tidb-cloud/select-cluster-tier.md#starter) instances, the changefeed feature is unavailable. > - For [{{{ .essential }}}](/tidb-cloud/select-cluster-tier.md#essential) instances, the changefeed feature is only available upon request. For more information, see [Changefeed](/tidb-cloud/essential-changefeed-overview.md). +> - For [{{{ .dedicated }}}](/tidb-cloud/select-cluster-tier.md#tidb-cloud-dedicated) clusters, see [Sink to Cloud Storage](https://docs.pingcap.com/tidbcloud/changefeed-sink-to-cloud-storage/). + + ## Restrictions -- For each TiDB Cloud Dedicated cluster, you can create up to 100 changefeeds. +- For each TiDB Cloud Dedicated clusterTiDB Cloud Premium instance, you can create up to 100 changefeeds. - Because TiDB Cloud uses TiCDC to establish changefeeds, it has the same [restrictions as TiCDC](https://docs.pingcap.com/tidb/stable/ticdc-overview#unsupported-scenarios). - If the table to be replicated does not have a primary key or a non-null unique index, the absence of a unique constraint during replication could result in duplicated data being inserted downstream in some retry scenarios. ## Step 1. Configure destination + + Navigate to the overview page of the target TiDB Cloud Dedicated cluster. Click **Data** > **Changefeed** in the left navigation pane, click **Create Changefeed** to go to the **Destination** page, and then select **Amazon S3**, **GCS**, or **Azure Blob Storage** as the destination, depending on the cloud provider on which your TiDB Cloud Dedicated cluster is hosted. The configuration process varies depending on the destination you choose. + + + + +Navigate to the overview page of the target TiDB Cloud Premium instance. Click **Data** > **Changefeed** in the left navigation pane, click **Create Changefeed** to go to the **Destination** page, and then select **Amazon S3** or **Alibaba Cloud OSS** as the destination, depending on the cloud provider on which your TiDB Cloud Premium instance is hosted. The configuration process varies depending on the destination you choose. + + +
@@ -32,7 +58,7 @@ For **Amazon S3**, you can use either **AWS Role ARN** or **AWS access key** for To use an IAM Role for authentication, follow these steps: -1. On the **Destination** page for Amazon S3, enter the **S3 URI**. Make sure that the S3 bucket is in the same AWS region as your TiDB cluster. +1. On the **Destination** page for Amazon S3, enter the **S3 URI**. Make sure that the S3 bucket is in the same AWS region as your TiDB Cloud Dedicated clusterTiDB Cloud Premium instance. 2. Under **Bucket Access**, select **AWS Role ARN**. 3. To create a new Role ARN, click **Click here to create new one with AWS CloudFormation**. This template automatically configures the required permissions. @@ -55,7 +81,7 @@ To use an IAM Role for authentication, follow these steps: To use an access key for authentication, follow these steps: -1. On the **Destination** page for Amazon S3, enter the **S3 URI**. Make sure that the S3 bucket is in the same AWS region as your TiDB cluster. +1. On the **Destination** page for Amazon S3, enter the **S3 URI**. Make sure that the S3 bucket is in the same AWS region as your TiDB Cloud Dedicated clusterTiDB Cloud Premium instance. 2. Under **Bucket Access**, select **AWS Access Key**. 3. Fill in the following fields: @@ -63,6 +89,9 @@ To use an access key for authentication, follow these steps: - **Secret Access Key**
+ + +
For **GCS**, before filling **GCS Endpoint**, you need to first grant the GCS bucket access. Take the following steps: @@ -118,6 +147,11 @@ For **GCS**, before filling **GCS Endpoint**, you need to first grant the GCS bu 7. In the TiDB Cloud console, go to the Changefeed's **Destination** page, and fill in the **bucket gsutil URI** field.
+ +
+ + +
For **Azure Blob Storage**, you must configure the container and get a SAS token in the Azure portal first. Take the following steps: @@ -158,9 +192,62 @@ For **Azure Blob Storage**, you must configure the container and get a SAS token - **SAS Token**: enter the generated SAS token obtained in step 3.
+ +
+ + + +
+ +For **Alibaba Cloud OSS**, follow these steps to configure the changefeed: + +1. In the [Alibaba Cloud console](https://www.alibabacloud.com/), perform the following prerequisite steps: + + 1. Create an OSS bucket in the same region as your TiDB cluster. For detailed instructions, see [Create a bucket](https://www.alibabacloud.com/help/en/oss/user-guide/create-a-bucket-4). + 2. Create a RAM user for the changefeed and generate an AccessKey pair. For detailed instructions, see [Create an AccessKey pair](https://www.alibabacloud.com/help/en/ram/user-guide/create-an-accesskey-pair). + 3. Create and attach a custom RAM policy to the RAM user to grant only the minimum permissions required by the changefeed. For more information, see [Control access to OSS resources with RAM policies](https://www.alibabacloud.com/help/en/oss/user-guide/ram-policy). + + - `oss:ListObjects` + - `oss:GetObject` + - `oss:PutObject` + - `oss:DeleteObject` + + The following JSON example shows a policy with the required permissions. Replace `` with the name of your OSS bucket. + + ```json + { + "Version": "1", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "oss:ListObjects", + "oss:GetObject", + "oss:PutObject", + "oss:DeleteObject" + ], + "Resource": [ + "acs:oss:*:*:", + "acs:oss:*:*:/*" + ] + } + ] + } + ``` + +2. On the **Destination** page for Alibaba Cloud OSS, fill in the following fields: + + - **Bucket URI**: enter the OSS URI in the format `oss:////`. + - **Access Key ID**: enter the AccessKey ID from the RAM user. + - **Access Key Secret**: enter the AccessKey Secret from the RAM user. + +
+ +
+
-Click **Next** to establish the connection from the TiDB Cloud Dedicated cluster to Amazon S3, GCS, or Azure Blob Storage. TiDB Cloud will automatically test and verify if the connection is successful. +Click **Next** to establish the connection from the TiDB Cloud Dedicated clusterTiDB Cloud Premium instance to your cloud storage. TiDB Cloud will automatically test and verify if the connection is successful. - If yes, you are directed to the next step of configuration. - If not, a connectivity error is displayed, and you need to handle the error. After the error is resolved, click **Next** to retry the connection.