diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 78d6a9346..d028481c2 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -333,7 +333,9 @@ * xref:sql:index.adoc[Redpanda SQL] ** xref:sql:get-started/index.adoc[Get Started] *** xref:sql:get-started/sql-quickstart.adoc[Quickstart] -*** xref:sql:get-started/deploy-sql-cluster.adoc[Enable Redpanda SQL] +*** xref:sql:get-started/deploy-sql-cluster.adoc[Enable Redpanda SQL on BYOC] +*** xref:sql:get-started/enable-sql-byovpc.adoc[Enable Redpanda SQL on BYOVPC] +**** xref:sql:get-started/enable-sql-byovpc-gcp.adoc[GCP] *** xref:sql:get-started/overview.adoc[] **** xref:sql:get-started/oltp-vs-olap.adoc[] **** xref:sql:get-started/redpanda-sql-vs-postgresql.adoc[] diff --git a/modules/sql/pages/get-started/enable-sql-byovpc-gcp.adoc b/modules/sql/pages/get-started/enable-sql-byovpc-gcp.adoc new file mode 100644 index 000000000..c8f7a49b8 --- /dev/null +++ b/modules/sql/pages/get-started/enable-sql-byovpc-gcp.adoc @@ -0,0 +1,180 @@ += Enable Redpanda SQL on a BYOVPC Cluster on GCP +:description: Provision SQL-specific GCP resources and enable the Redpanda SQL engine on a running BYOVPC cluster on GCP using the Cloud API. +:page-topic-type: how-to +:personas: platform_admin +:learning-objective-1: Provision the SQL-specific GCP customer-managed resources +:learning-objective-2: Enable Redpanda SQL on a running BYOVPC cluster with the Cloud API +:learning-objective-3: Verify that the SQL engine is running and ready to accept connections + +include::shared:partial$feature-flag-rpcn.adoc[] + +Enable Redpanda SQL on a running Bring Your Own Virtual Private Cloud (BYOVPC) cluster on GCP so you can query streaming data in Redpanda topics using standard PostgreSQL syntax. + +Unlike standard BYOC clusters, Redpanda does not create resources in your GCP project for BYOVPC clusters. You must provision the SQL-specific GCP resources yourself and supply them to the cluster as customer-managed resources (CMRs). On GCP, you enable Redpanda SQL as a day-2 operation with the Cloud API: the Cloud Console cannot supply the SQL customer-managed resources for a BYOVPC cluster. + +After reading this page, you will be able to: + +* [ ] {learning-objective-1} +* [ ] {learning-objective-2} +* [ ] {learning-objective-3} + +== Prerequisites + +* A running xref:get-started:cluster-types/byoc/gcp/vpc-byo-gcp.adoc[BYOVPC cluster on GCP], with its base customer-managed resources already provisioned. +* Admin permissions in your Redpanda Cloud organization on xref:billing:billing.adoc[usage-based billing], and a Redpanda Cloud service account (client ID and secret) with permission to update clusters. +* The https://cloud.google.com/sdk/docs/install[Google Cloud CLI^] (`gcloud`) with the beta component, authenticated with Application Default Credentials (`gcloud auth application-default login`) and your service project selected. +* To verify the engine after enabling it: `psql` version 16 or later on a host inside the cluster's VPC. A BYOVPC cluster is private, so the SQL endpoint is not reachable from outside the VPC. + +== Provision the SQL customer-managed resources + +Redpanda SQL on a GCP BYOVPC cluster requires four customer-managed resources that the base BYOVPC provisioning does not create: + +[cols="1,2",options="header"] +|=== +| Resource | Purpose +| A GCP service account for the SQL compute nodes | Reads and writes the SQL data-storage bucket. +| A GCP service account for the SQL API | Accesses Secret Manager secrets under the SQL prefix. +| A GCS bucket for SQL data storage | Stores the SQL engine's data. +| A Secret Manager prefix | A string namespace (not a resource) that reserves where Redpanda SQL Iceberg catalog credentials are stored. +|=== + +Create the two service accounts and the storage bucket, and choose a Secret Manager prefix. Replace `` and `` with your own values, and set the bucket location to your cluster's region. + +[,bash] +---- +# Service accounts +gcloud iam service-accounts create "redpanda-rpsql-" \ + --project "" --display-name "Redpanda SQL Service Account" +gcloud iam service-accounts create "redpanda-rpsql-api-" \ + --project "" --display-name "Redpanda SQL API Service Account" + +# SQL data-storage bucket +gcloud storage buckets create "gs://redpanda-rpsql-storage-" \ + --project "" --location "" --uniform-bucket-level-access +---- + +// TODO: Document the IAM roles the two SQL service accounts require. Pending SME confirmation. + +Grant each service account the permissions it needs. Confirm the exact roles with your Redpanda account team before you rely on this in production. + +The Secret Manager prefix is a string you choose (for example, `redpanda-rpsql-`); it is not a GCP resource you create. + +== Enable Redpanda SQL + +Enable the SQL engine with a Cloud API `PATCH` request. The Cloud Console does not support supplying the SQL customer-managed resources for a BYOVPC cluster. + +[IMPORTANT] +==== +The `PATCH` request must use an explicit leaf-path `update_mask` that lists each field you set, including each SQL customer-managed resource. A grouped mask (for example, `customer_managed_resources`) is silently dropped: the operation returns `STATE_COMPLETED`, but SQL stays disabled. +==== + +The SQL customer-managed resources `rpsql_service_account`, `rpsql_api_service_account`, and `rpsql_cloud_storage_bucket` are immutable while SQL is enabled. To change any of them, disable SQL first. You can change `rpsql_secret_manager_prefix` without disabling SQL, but any secrets mapped under the old prefix become invalid, and you must move them to the new prefix manually. + +. link:/api/doc/cloud-controlplane/authentication[Authenticate to the Cloud API] and obtain a bearer token. +. Make a link:/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster[`PATCH /v1/clusters/{cluster.id}`] request. Set the `update_mask` query parameter to the explicit leaf paths, and provide the `rpsql` block and the four SQL customer-managed resources in the body. Replace `{cluster-id}` and the placeholder values with your own: ++ +[,bash] +---- +curl -sX PATCH "https://api.redpanda.com/v1/clusters/{cluster-id}?update_mask=rpsql.enabled,rpsql.replicas,rpsql.zones,customer_managed_resources.gcp.rpsql_service_account.email,customer_managed_resources.gcp.rpsql_api_service_account.email,customer_managed_resources.gcp.rpsql_cloud_storage_bucket.name,customer_managed_resources.gcp.rpsql_secret_manager_prefix" \ + -H "Authorization: Bearer $AUTH_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "rpsql": { + "enabled": true, + "replicas": , + "zones": [""] + }, + "customer_managed_resources": { + "gcp": { + "rpsql_service_account": {"email": ""}, + "rpsql_api_service_account": {"email": ""}, + "rpsql_cloud_storage_bucket": {"name": ""}, + "rpsql_secret_manager_prefix": "" + } + } + }' +---- ++ +Replace the placeholders with your own values: ++ +* ``: Initial number of SQL compute nodes (minimum 1, maximum 9). +* `` (optional): One of the cluster's glossterm:availability zones[]. Even if the Redpanda cluster spans multiple AZs, the SQL engine always deploys to a single AZ. If omitted, Redpanda selects a zone automatically. The AZ is locked when SQL is first enabled and cannot be changed afterward. +* The four `rpsql_*` values: the service account emails, bucket name, and Secret Manager prefix from <>. ++ +A genuine enable returns a long-running operation that stays `STATE_IN_PROGRESS` for roughly 15 to 30 minutes while the SQL node group provisions. ++ +. Poll the link:/api/doc/cloud-controlplane/operation/operation-operationservice_getoperation[`GET /v1/operations/{operation.id}`] endpoint until the operation completes: ++ +[,bash] +---- +curl -s "https://api.redpanda.com/v1/operations/{operation-id}" \ + -H "Authorization: Bearer $AUTH_TOKEN" \ + -H "Content-Type: application/json" +---- + +== Verify the SQL engine is running + +A completed operation does not guarantee that SQL was enabled. Confirm the result by fetching the cluster and checking that the `rpsql` block and all four SQL customer-managed resources are populated: + +[,bash] +---- +curl -s "https://api.redpanda.com/v1/clusters/{cluster-id}" \ + -H "Authorization: Bearer $AUTH_TOKEN" +---- + +SQL is enabled when the response shows `rpsql.enabled` set to `true` with a non-empty `rpsql.url`, and the four `customer_managed_resources.gcp.rpsql_*` fields are populated (not null). For example: + +[,json,role="no-copy"] +---- +"rpsql": { + "enabled": true, + "replicas": 3, + "url": "oxla-..byoc..cloud.redpanda.com:5432", + "zones": [""] +} +---- + +A *Redpanda SQL* tile also appears in the cluster overview in the Cloud Console, under *Products*, showing engine status and node count. Provisioning can take up to 30 minutes. + +To connect from a host inside the cluster's VPC, use the SQL endpoint URL with `psql` version 16 or later and a bearer token: + +[,bash] +---- +PGPASSWORD="$AUTH_TOKEN" psql \ + "host= port=5432 dbname=oxla user=ignored options='-c auth_method=bearer' sslmode=require" \ + -c 'SHOW NODES;' +---- + +This example uses bearer-token authentication (`auth_method=bearer`), so Redpanda SQL takes the session identity from the token and ignores the `user` field. Redpanda SQL always connects to the `oxla` database regardless of the `dbname` value. For client-credentials and SCRAM authentication, where the `user` field carries the identity, see xref:sql:connect-to-sql/authenticate.adoc[Authenticate to Redpanda SQL]. + +// TODO: Confirm OIDC bearer-token authentication on GCP before publishing; cross-check authenticate.adoc. + +== Disable Redpanda SQL + +[WARNING] +==== +Disabling Redpanda SQL tears down the SQL compute engine and clears its catalog state (catalog metadata, table mappings, and role/grant data). In-flight queries fail when SQL is disabled. +==== + +Redpanda topic data, Schema Registry subjects, and any Iceberg-committed history for Iceberg-enabled topics are not affected. The Redpanda cluster itself continues to run normally, and only the SQL engine and its associated state are removed. + +Re-enabling SQL on the same cluster provisions a fresh engine. Redpanda does not restore prior catalog state, table mappings, or grants. You must re-create catalogs, tables, and grants after re-enabling. + +The SQL-specific GCP resources (service accounts and bucket) are not deleted when you disable SQL. They remain in your project and can be reused if you re-enable SQL. + +To disable SQL, make a link:/api/doc/cloud-controlplane/operation/operation-clusterservice_updatecluster[`PATCH /v1/clusters/{cluster.id}`] request with `rpsql.enabled` set to `false`: + +[,bash] +---- +curl -sX PATCH "https://api.redpanda.com/v1/clusters/{cluster-id}?update_mask=rpsql.enabled" \ + -H "Authorization: Bearer $AUTH_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"rpsql": {"enabled": false}}' +---- + +Poll the link:/api/doc/cloud-controlplane/operation/operation-operationservice_getoperation[`GET /v1/operations/{operation.id}`] endpoint until the operation completes. + +== Next steps + +* xref:sql:get-started/sql-quickstart.adoc[Quickstart]: Connect to Redpanda SQL with `psql` and run your first query. +* xref:sql:connect-to-sql/authenticate.adoc[Authenticate to Redpanda SQL]: Connect with OIDC client credentials or a SCRAM password.